Fine-tune, or engineer the context

01The expensive fork

Every applied LLM product reaches the same moment. The pilot shipped, real traffic arrived, and now one of three pressures is on the table: the model keeps getting domain questions wrong, the inference bill has become a line item someone reviews, or a stakeholder has said the words "our data is special." At that point the team faces the most expensive fork in applied LLM work: keep engineering the context, or fine-tune a model.

The fork is expensive because the two paths have different physics. Context engineering is a change to strings and pipelines. It deploys like software, rolls back like software, and ports across model vendors with an eval rerun. Fine-tuning is a program: a training data pipeline, training runs, an eval harness that can catch regressions you did not train for, safety testing, and a retraining clock you do not control, because a tuned model dies when the vendor retires its base. Choosing wrong here costs quarters, not sprints.

Our position, argued below from public evidence and current list prices: most teams should exhaust context engineering first. Fine-tuning earns its keep on form, latency, and unit cost at scale. It does not earn its keep as a way to teach a model facts, and teaching the model facts is the reason most teams reach for it.

02What a tune actually changes

Start with what the vendors themselves say. OpenAI's supervised fine-tuning guide lists the good use cases: classification, nuanced translation, generating content in a specific format, and correcting instruction-following failures. It recommends starting with about 50 well-crafted demonstrations and reports improvements from 50 to 100 examples.^[1] Notice what is on that list: behaviors. Notice what is not: "teach the model our domain."

The behavior story is real. Gorilla, a LLaMA-7B fine-tuned on API documentation, surpassed GPT-4 at writing correct API calls on the APIBench suite and substantially cut hallucinated arguments.^[2] A 7B model beating the then-frontier model on a narrow structured-output task is the cleanest demonstration of what gradient updates buy: form, schemas, and reliable adherence to an output contract.

The knowledge story is the opposite. Ovadia et al. ran the head-to-head on knowledge injection. On a current-events task, base Mistral-7B scored 0.481, the fine-tuned model 0.504, and the base model with retrieval 0.875. Retrieval stacked on the tune reached 0.810, worse than retrieval alone.^[3] Fine-tuning only began absorbing facts when each fact appeared in many paraphrased variants, with accuracy a monotonically increasing function of paraphrase count, and even with that augmentation the tuned model reached 0.588 against 0.875 for plain retrieval.^[3]

Worse than inefficient, teaching facts by gradient descent is actively harmful. Gekhman et al. showed in controlled closed-book QA that fine-tuning examples introducing new knowledge are learned significantly slower than examples consistent with what the model already knows, and that as those examples are finally learned, they linearly increase the model's tendency to hallucinate.^[4] Their conclusion matches our experience: pre-training supplies the facts, and fine-tuning teaches the model to use them. The effect is sharpest exactly where teams want the most help: for rare, tail-of-distribution entities, retrieval beats fine-tuning by a large margin.^[5]

Honesty requires the caveat: fine-tuning is not zero on knowledge. Microsoft's agriculture case study measured a gain of over 6 percentage points from fine-tuning, with retrieval adding 5 more on top, and fine-tuning lifted geographic answer similarity from 47% to 72%.^[6] Two things follow. The defensible claim is comparative, not absolute: retrieval wins on fresh and tail knowledge, and the combination often wins overall. And the gains compose, which matters later in this note.

03The unpriced liabilities

Three liabilities rarely appear in the fine-tuning business case.

Safety regression. Qi et al. showed in 2023 that fine-tuning gpt-3.5-turbo through the public API on just ten harmful examples, at a cost under twenty cents, moved its harmfulness rate from 1.8% to 88.8%.^[7] The finding that matters for ordinary teams is the benign case: one epoch on the plain Alpaca dataset, with no hostile intent anywhere, raised the harmfulness rate from 5.5% to 31.8%.^[7] Providers have deployed mitigations since, so treat these as 2023 results, not current attack rates. The structural lesson has not aged: alignment is shallow under gradient updates, and every tune, including a formatting tune on clean data, needs a safety regression eval before it ships.

Forgetting. Tuning on your task erodes what the base model could do elsewhere. The best-measured mitigation is also a confession: Biderman et al. found that LoRA substantially underperforms full fine-tuning on the target domain but better preserves base-model performance outside it, working as a stronger regularizer than weight decay or dropout, while full fine-tuning learns weight updates with a rank 10 to 100 times higher than typical LoRA configurations.^[8] You do not escape the tradeoff between learning and forgetting; you pick a point on it.

Mortality. A fine-tuned model dies with its base model, on the provider's schedule. OpenAI's deprecations page states that inference on fine-tuned models remains available only until the base model is deprecated, and lists gpt-4-0613 and gpt-4-turbo for shutdown on October 23, 2026 and the gpt-5-2025-08-07 snapshot for December 11, 2026.^[9] A system prompt and a retrieval corpus migrate to a successor model in an afternoon of eval reruns. A tune restarts the whole program: data, training, evals, safety. Note also that Anthropic, as of this writing, offers no general fine-tuning on its first-party API, so a tune-centric architecture forecloses an entire vendor.

04Context engineering, priced

The alternative path now has a name and a doctrine. Anthropic defines context engineering as "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference," and treats attention as a budget: a finite resource with diminishing marginal returns, where recall degrades as the window fills, a failure mode the post names context rot.^[10] The working vocabulary is compaction, just-in-time retrieval, and sub-agent context isolation. This is engineering, not prompt folklore.

The economics have moved decisively in its favor. Prompt caching makes the "put the knowledge in the context" strategy cheap on repeat traffic: Anthropic bills cache reads at one tenth of base input price and writes at 1.25 times for the default five minute cache, and states that such a cache pays off after a single read.^[11] OpenAI prices cached GPT-5 input at $0.125 per million tokens against $1.25 standard, a 90% discount that was 50% two model generations earlier on gpt-4o.^[12] Each deepening of that discount erodes the token savings that once justified a tune's shorter prompts. Long context is provider-dependent: Anthropic bills a 900,000-token request at the same per-token rate as a 9,000-token one, while Gemini 2.5 Pro doubles input pricing above 200K tokens and charges $4.50 per million tokens per hour to hold a cache.^[11][13]

The invoice has fine print. Context rot is a measured phenomenon, so dumping everything into a million-token window is not a strategy.^[10] Cache economics collapse if the prompt prefix churns, since any byte change invalidates it. "Exhaust context engineering first" therefore means discipline: stable prefixes, retrieval hygiene, eval-driven few-shot selection, and ruthless curation of what enters the window.

05Where fine-tuning wins

Fine-tuning has a home ground, and it is worth naming precisely: form, latency, and unit cost at volume.

Form. When the eval gap is schema compliance, tone, a house output format, or instruction-following that prompting cannot stabilize, tuning is the right tool, and the Gorilla result shows how far it goes: a tuned 7B outperforming GPT-4 on API-call generation.^[2] About 50 curated demonstrations is often enough to start.^[1]

Unit cost, via distillation down a tier. The honest economic win is not tuning your flagship model; that raises your price. On OpenAI's current list, fine-tuned gpt-4.1 runs $3.00 input and $12.00 output per million tokens, 1.5 times base gpt-4.1.^[12] The win is capturing flagship behavior in a smaller tier: fine-tuned gpt-4.1-mini at $0.80 and $3.20 sits 60% below base gpt-4.1 at both ends, though still at twice the price of untuned mini.^[12] The savings only materialize when the tier drop holds quality on your eval and volume amortizes the training run.

Capability transfer at the high end. Distillation is where tuning's capability story is strongest. DeepSeek's R1 distills, trained with supervised fine-tuning alone on roughly 800K reasoning traces from the large model, produced a 14B student scoring 69.7 on AIME 2024 against 63.6 for o1-mini, and a 7B scoring 92.8 on MATH-500 against 74.6 for GPT-4o.^[14] Small tuned models can beat much larger general models on a targeted skill. That is the pattern to copy: compress demonstrated behavior downward, not inject facts sideways.

Latency. A tuned smaller model with a short prompt cuts both time to first token and generation time, and removes a retrieval hop where the knowledge is stable enough to live in weights. For strict latency budgets and on-device deployments this can flip the whole decision, a point we return to below.

06The decision procedure

Here is the procedure we use in engagements, in order and without skipping steps.

1. Build the eval first. A scored, versioned eval on your real traffic. Without it, both paths are guesswork, and OpenAI's own guidance says to invest in fine-tuning only after evals exist.^[1]
2. Exhaust context engineering against that eval. System prompt revision, retrieval over your corpus, eval-selected few-shot examples, enforced structured outputs, and prompt caching behind a stable prefix. Most teams pass their eval somewhere in this step and stop, correctly.
3. Classify the residual gap. If the model lacks facts or serves stale ones, that is a retrieval problem; tuning will underperform retrieval and can raise hallucination.^[3][4] If the gap is form, schema, tone, or persistent instruction-following failure, tuning is a candidate. If the gap is cost or latency at volume, tuning via distillation is a candidate.
4. If tuning, tune cheap and low. Distill down a tier instead of tuning the flagship. Start near 50 curated demonstrations and grow the set only as the eval demands.^[1] Prefer parameter-efficient methods when general capability must survive.^[8] Keep retrieval in the serving path and include retrieved context in training examples, so the tune learns to use documents rather than replace them.^[2][6]
5. Regression-test beyond the target metric. Run safety evals and broad-capability evals on every tune, including tunes on visibly benign data.^[7][8]
6. Budget for mortality. Every base-model deprecation restarts the tune.^[9] Keep the context-engineered variant alive as your fallback and your portability hedge.

07Practice notes

Three notes from practice, and a closing position.

The fork is not exclusive. The strongest production systems combine both paths. Microsoft's agriculture study found tuning and retrieval gains additive,^[6] and Gorilla's retriever-aware training let it adapt to documentation changes at test time.^[2] When we tune at all, we increasingly tune the model's retrieval behavior, not its knowledge.

Small models flip the answer. Soudani et al. found fine-tuning most useful for small models,^[5] and the R1 distills show how much targeted skill fits in 7B or 14B weights.^[14] On-device, edge, and hard-latency deployments often lack the context budget and retrieval infrastructure that make context engineering the default. There, tuning moves from last resort to first tool.

Portability is an architectural property. A prompt, a corpus, and an eval suite are vendor-neutral assets; they follow you across model generations and providers. A tune is an asset with a decay schedule attached to someone else's roadmap. In any build that expects to run for years, we weight that heavily.

Our closing position fits in two sentences. Fine-tune to compress a solved problem into a cheaper, faster artifact. Engineer the context to solve the problem in the first place, and let your evals, not your enthusiasm, tell you when the first sentence applies.