The context window
Working memory, not a hard drive
The context window is the maximum number of tokens the model can attend over in one shot: prompt, conversation history, pasted documents, and the response being written, all sharing one budget. GPT-2 lived in 1,024 tokens; current frontier models advertise 128k to a million-plus.
The part that reorganizes how you think about these systems: the model is stateless. Between API calls it retains NOTHING. Every "conversation" is an illusion maintained by re-sending the entire transcript with each message, and the model re-reading all of it to produce one more reply. There is no memory being consulted; there is a window being refilled.
Consequences fall out immediately: long chats cost more per message (you re-pay for the history every turn), and when the transcript outgrows the window, something has to fall out. Watch what that looks like from the inside.
Outgrow the window
A chat with a deliberately tiny 60-token window. Advance the conversation, watch the meter, and pay attention to what happens to the oldest messages when the budget runs out.
Why not just make the window infinite?
Three separate walls stand in the way, and knowing them explains a lot of product behavior:
- The n² bill from the transformer module: attention compares every token with every token. Long contexts also bloat the KV cache, eating GPU memory per token, per layer.
- Lost in the middle: on very long inputs, models retrieve information from the start and end far better than from the middle. A 200-page document dumped into a giant window is NOT reliably readable; measured retrieval sags exactly where your key fact probably sits.
- Attention dilution: softmax weights spread over 500,000 tokens leave less focus per token. Bigger windows trade sharpness for reach.
The engineering answer you will build in the AI Engineer protocol: stop stuffing and start retrieving. Store knowledge outside the model, embed it (module 2 paying rent again), and fetch only the relevant chunks into the window per request. RAG, in one sentence.
And with that, the protocol arc is complete: bytes → tokens → embeddings → attention → transformer → base model → assistant, with the window as the assistant's working desk. You now know, mechanically, what happens between pressing Enter and the first token appearing. The next protocols put that knowledge to work: building with these systems, and breaking them.
Check your understanding
Deep into a long chat, the assistant contradicts constraints you set in your first message. The most likely mechanical cause?
What to remember
- ✓The window is the model's entire working memory: system prompt, history, documents, and output share one token budget.
- ✓Models are stateless: each call re-reads the whole transcript. Long chats literally cost more per message.
- ✓Overflow means the oldest content ceases to exist for the model. Mid-chat amnesia is geometry, not mood.
- ✓Big windows have real limits: n² compute, KV-cache memory, and the measured lost-in-the-middle sag.
- ✓Retrieval (RAG) beats stuffing: fetch what matters into the window instead of shipping everything.
- ✓Protocol complete: from raw bytes to a working assistant, every stage now open. Two doors ahead: build with it, or break it.