MAROSI
From the lab
Field notes.
Engineering notes from production systems. Written for the people who build them: what we tried, what held, and what broke on the way.
-
Evals before agents
A convincing demo is one sample from a distribution. We lay out the public evidence that single-run success misleads, and the harness we build before any agent: real-failure task sets, end-state graders, pass^k with error bars, and a CI regression gate.
Read the note -
Retrieval is a data problem
When a RAG system answers wrong, teams swap the model. The public failure data says look upstream instead: parsing, chunking, index freshness, and permissions decide answer quality before the generator ever sees a token.
Read the note -
In praise of boring infrastructure
The systems that survive a decade are the ones nobody talks about. We lay out the public evidence for boring infrastructure and a procedure for spending innovation tokens only where the business differentiates.
Read the note -
Fine-tune, or engineer the context
Fine-tuning changes form, not knowledge, and it carries safety, forgetting, and deprecation liabilities most teams never price in. We walk through the public evidence and give a six-step procedure for choosing between tuning and context engineering.
Read the note -
One-way doors for machines
Bezos sorted decisions by whether you can walk back through the door. Agents need a third variable, measured error rate, and we turn the three into a working procedure: an exposure score, four autonomy levels, veto rules and a ratchet.
Read the note