YOUR BRIEF
What to work through
Work through chatGPT as a spoken system design interview. Start by clarifying the scope, sketch a working design and explain the trade-offs behind your choices.
- Send a prompt in a chat and get the answer streamed back token by token
- See past chats and continue one, with its earlier turns as context
- A free tier with limits, and paid tiers with more
- Stop an answer halfway
BEFORE YOU START
Constraints to discuss
- Time to first token under 200 ms: a blank screen feels broken
- About 20,000 prompts a second at peak
- About 120,000 answers streaming at the same time
- GPUs are scarce and cost about $2 an hour each, so GPU time is the budget
Need a starting point?
Watch for these common gaps as you prepare:
- Waiting for the whole answer before sending anything
- Tying the generation to one connection, so a reconnect loses the answer
- Treating GPUs like stateless web servers, ignoring the memory each chat holds
- No limits per tier, so a few heavy users take the pool
- Getting lost in model internals instead of the serving system around the model
Why this question is in the collection
These sources connect the problem to company interview reports or preparation guides. They do not guarantee what a company will ask you.
- OpenAI: PracHub lists a chat assistant inference service with streaming, free-tier limits and GPU scheduling, dated September 2026.
- Anthropic: Aced reports close variants: a token-generation service at 100,000 requests a second, and end-to-end batching of LLM queries.
- Google: CallSphere’s 2026 AI question list tags its ChatGPT-style conversational service with Google, beside OpenAI and Anthropic.