Instruction tuning

The fix for the base model's answer-a-question-with-more-questions habit is almost anticlimactic. Collect some tens of thousands of examples of the job done right, each one a pair: an instruction ("Summarize this email", "Explain photosynthesis simply") and a high-quality response, written or curated by humans. Then keep training the base model on them, same next-token loss, nothing exotic.

This is supervised fine-tuning, SFT, and the budget tells you what kind of step it is: pretraining consumed trillions of tokens over months; SFT consumes millions over hours or days. A homeopathic dose of data, and the model comes out transformed: it answers instead of continuing.

The famous version of this move is what turned GPT-3 (2020, base model, a curiosity for practitioners) into something the public could use. The knowledge barely changed. The interface changed completely.

How do 50,000 examples redirect a model trained on 10,000,000,000,000 tokens? Because SFT is not teaching content. The base model has already read millions of well-answered questions, helpful explanations, and polite replies; assistant-like behavior is one of the countless personas living in its distribution. SFT amplifies that persona until it dominates. Selection, not instruction: the librarian was always in the building, SFT put them at the front desk.

Which is also the sharpest way to see its limits:

• SFT can only demonstrate, and demonstrations cap at what humans bother writing. You cannot write the ideal response to everything.
• It transfers no sense of BETTER and WORSE: the model sees one good answer per prompt, never a comparison between a great one and a mediocre one.
• Confidently wrong answers survive it: fluency was rewarded, calibration was not.

To go further you need to grade preferences, not copy examples. That grading machine is the next lesson, and it is the most consequential (and most contested) step in the whole pipeline.