Skip to content
Meir Dick
anon@meirdick
More in Ideas
~/ideas
Where next
~/ideas/next
The Adaptive Factory
~/ideas/the-adaptive-factory.md

What would you build today?

Ask it of any product you run. If you knew everything you know now, and had the team, the budget, the hardware and the models you have now, what would you build? For almost any product more than a few years old, the answer is not what you have.

That gap is normal. Every product is a bundle of decisions, each made with the knowledge, the tools, the data and the market of the day it was made. Then the world moves. A competitor ships something you could not have built when you started. A new model makes half your hand-written logic look like a workaround. New hardware arrives that your architecture cannot use. You learn which features nobody uses and which edge cases carry the business. The decisions stay where they were.

In a competitive market, the ability to move is the advantage. The better product wins only until the next shift; the team that can change its product as fast as the world changes wins the shift after that, and the one after that. So the question under every product strategy is this: how do you keep building without locking yourself into yesterday’s choices?

For twenty-five years the answer has been incremental development. Ship small, refactor at the edges, strangle the old system piece by piece, never rewrite. For most products it is the right answer, and it still is. It keeps the product live, keeps each risk small, and keeps every lesson the old code learned. But it has a ceiling. Incremental change improves the design you have. It cannot reach the design you would choose today, because each step starts where the last one stopped. In time the product becomes the best version of a design whose premises no longer hold.

The same thing happens to the company that built it. The people, the first architecture, the early customers and the tools set into habits. Those habits give the company its focus and its speed, and they also make it answer every new problem with its first answers. Dorothy Leonard-Barton named this in 1992: core capabilities become core rigidities. In plain terms, the choices that made you good are the choices that hold you in place.

The way out was a rewrite, and rewrites failed. Netscape scrapped its browser code in late 1998 and shipped nothing new until November 2000, by which time Internet Explorer had the market. Joel Spolsky drew the lesson and the industry agreed with him. But the case against rewrites was never that the new design would be worse. It was that writing it cost too much. The best engineers left the product for a year, every lesson in the old code had to be learned again, and nothing shipped in the meantime.

Writing is no longer the expensive part. Agents write. What stays expensive is knowing what to build and proving that the new thing is acceptable. So there is a new question. Is there a software factory for the age of AI that can build the answer to “what would you build today?”, and build it again next year, without the risk that made rewrites a bet nobody could afford? And what would we gain over the incremental way?

I think there is, and I call it the adaptive factory. One line serves the market: the champion. A second line, the challenger, rebuilds the product from everything learned since the champion was designed. The challenger runs in shadow against real traffic, which means it gets every input the champion gets but its outputs are never used, and it is promoted only on evidence. Tesla has run a version of this on its driving stack for a decade. Two things make it possible for everyone else: a ledger that records what the system must do and why, and a harness that proves the challenger is acceptable. The harness is what I am building.

What the factory gives you that incremental development cannot is a way off the old design when the world moves. New capabilities enter at the level of architecture, not as bolt-ons. Every decision is re-examined each cycle instead of inherited. And customers never see the first, worse version of the new design.

Where incremental is enough

Most software does not need this. A booking app, a marketing site, an internal tool: the requirements move slowly, the constraints are ordinary, and the best design today is close to the design you have. Keep shipping small. The factory costs a second stack, a ledger and a harness, and for a simple product that cost buys nothing.

The factory pays when the value of the product depends on constraints that move faster than code can follow:

  • Hardware. New sensors, chips or devices change what the product can do. A stack designed for the old hardware cannot use the new one well.
  • The AI frontier. Each model generation changes which parts of a system should be written and which should be learned or delegated. Logic written around last year’s model limits is this year’s dead weight.
  • Training data. When the data grows by orders of magnitude, approaches that were impossible become the best ones.
  • Codebase complexity. In a large system every change touches ten others, and the team spends more effort keeping it standing than improving it. Major refactors and platform migrations live here.
  • Changing requirements. In regulated, contested or fast-moving markets, what the system must do changes faster than the architecture can absorb.

Self-driving has all five. Most enterprise systems over ten years old have at least two. If your product has none, incremental development is the right answer and you can stop here.

Why good products get stuck

Richard Foster drew the picture in Innovation: The Attacker’s Advantage. Plot the effort put into a technology against what it returns and you get an S: slow at first, steep once the key knowledge falls into place, then flat as the approach nears its limits. The flat part is where most mature software lives. Every feature costs more than the last. The data model needs a workaround for each new case. A change in one place breaks three others. The team works as hard as it ever did and the product improves less each quarter. That is what a local maximum feels like from the inside: there is no better version of this architecture, only more expensive versions of it.

To go higher, jump to the next curve, and it starts lower

The champion is near the top of its curve; the challenger starts below it and climbs in shadow. The dashed line is the champion’s ceiling.

Getting higher means jumping to the next curve, and the next curve starts lower. The first version of a new architecture is worse than the mature version of the old one. That dip is why incumbents stay on the old curve too long, and why Foster gives the advantage to the attacker: whoever has no mature version to protect climbs the steep part of the new curve while the defender squeezes the last percent out of the flat one. The adaptive factory is a way to be your own attacker. The challenger climbs the new curve in shadow while the champion keeps serving, and no customer sees the dip.

The ceiling is set by the conditions under which the architecture was designed. Four inputs set it:

  • Environment. The users, the scale, the rules, the competitors. Requirements change because the world does.
  • Capabilities. The hardware, platforms, models and tooling that existed. Most architectures from 2018 are the best that could be built before there were agents.
  • Learnings. What is now known about the problem that was not known then: which features nobody used, which edge cases dominate, which assumptions were wrong.
  • Resources. The team, the budget and the time available at the start, which forced choices nobody would make again.

When the inputs change and the architecture does not, the champion becomes the best answer to a problem that no longer exists. It still works. It just can’t climb past a ceiling the old conditions set. These four inputs are also the four parts of the question in the title: what you know, what you have, what you can use, and what the world now asks for.

Writing got cheap

The large AI-assisted migrations of the last eighteen months show the cost curve has flipped, and they all point at the same bottleneck. Airbnb moved about 3,500 test files in six weeks on work estimated at a year and a half. Google had AI author roughly 80% of an int32-to-int64 migration in Ads and then throttled generation, because human review had become the constraint. Bun rewrote 535,000 lines from Zig to Rust with 64 parallel agents: eleven days to a mergeable branch, four months to a release. A test suite with over a million assertions served as the judge of correct behaviour, and nineteen semantic regressions (changes in what the code does that the tests didn’t catch) still slipped through. Thoughtworks cut reverse-engineering time on a 15-million-line COBOL estate by two thirds and found that subject-matter-expert time was the real limit. In October, InfoWorld ran a piece arguing it is time to rewrite all that old code.

Bun port to Rust: 11 days to write, about four months to ship

The agents finished their part in eleven days. Everything after that was proof.

Read together: writing is cheap. Three things are not. Domain knowledge is not, so the factory needs a ledger that holds it. Semantic regressions that pass tests are not, so it needs a harness that catches them on real traffic. Review is not, so it needs a team whose job is judgment. The rest of this essay is those three things.

Tesla already runs it

Tesla has rebuilt its driving stack from the ground up more than once, and each time the trigger was the same: new data, new hardware, or both made the old architecture the wrong answer.

The clearest case is FSD v12. In August 2023 Musk livestreamed it and said the v11 control stack held over 300,000 lines of C++ and v12 had almost none of it. Tesla’s release notes described city-street driving as a single end-to-end neural network trained on millions of video clips. It reached customers in January 2024. That was not a refactor of v11. It was a different kind of program, built on the premise that the fleet had produced enough video, and the company had bought enough training compute, that learning the behaviour now beat hand-coding it. v11 was the best stack that could be built before that data existed. v12 is the answer to “what would you build today?” asked in 2023, and what it bought was a stack that improves with data rather than with code: a different curve, with a different ceiling.

Hardware forces the same thing. HW4 shipped in January 2023 with 5-megapixel cameras against HW3’s 1.2 and twice the memory. For a year Tesla ported: as late as March 2024 all training targeted HW3 and HW4 ran the HW3 models in emulation. By v12.5 that summer it flipped: training went HW4-first with five times the parameters, and HW3 needed extra optimization work to follow. v13 shipped in December 2024 to HW4 only; HW3 got its own branch. In January 2025 Musk conceded that HW3 cars would need a computer upgrade for full autonomy, and repeated it this April. The stack did not grow to fit the new hardware. It forked, and the older fork is being retired.

The manufacturing side shows the same instinct with a longer record. Tesla has never had model years; changes land whenever they are validated, at a pace Musk once put at twenty a week on the Model S. Munro’s teardown counted thirteen design changes to one Model Y part in three months. Two cars built weeks apart can differ in parts, so each car is defined by what was built into it rather than by its model. The record-keeping itself is not unusual: aerospace has tracked as-built configuration per unit for decades. The cadence is. SpaceX’s Raptor engine went through three versions in which each successor was a new engine rather than a patched one: thrust rose from 185 to 280 tonnes while mass fell from 2,080 to 1,525 kg, and by the third version the external plumbing had been absorbed into the body.

Raptor 1, 2 and 3 side by side, with sea-level thrust of 185, 230 and 280 tonnes-force

Raptor 1, 2 and 3. Each was a new engine, not a patched one: thrust rose from 185 to 280 tonnes-force while the plumbing moved inside the body. Image: SpaceX.

Shadow mode started with a human champion

On 19 October 2016 Tesla announced that every new car carried the hardware for full self-driving, and that the software would at first run in shadow mode: computing what it would have done while the human drove, taking no action, and logging the moments where it would have acted differently. Musk told reporters the point was to gather “a statistically significant amount of data” before enabling anything.

Read that as a deployment architecture rather than a safety story. The human driver was the champion. The model was the challenger. Divergence was the signal: every time the car would have braked and the driver did not, or the reverse, a clip went to Tesla’s servers as a candidate for labelling. The promotion rule was explicit. In January 2017 Musk said the new Autopilot build had been pushed to cars in non-actuating mode to assess reliability, and would be allowed to act within the week if it held up.

By 2019 the roles had inverted. The model was the champion and the challengers were candidate networks. At Autonomy Day Tesla described the process: before a network takes control it runs in shadow mode across the fleet, disagreement with the driver or with the production model surfaces mispredictions, and alternatives are promoted to early adopters only if they prove safer. Karpathy’s 2021 CVPR keynote put numbers on it: 221 triggers, candidate models deployed to observe without acting, a dataset of a million ten-second clips, and the loop run seven times before the first release.

Tesla champion and challenger, 2016 to 2024

The champion changed from a person to a model in 2019. The loop around it did not change.

The dual-chip computer belongs here. The HW3 board, presented at the same 2019 event, carries two independent chips, each with its own power, memory and operating system. Each takes the full camera feed and computes its own plan, and the car acts only when the two agree. Tesla presented it as redundancy. Read it as architecture and it is more general than that: a dual configuration. Two independent computations run on the same live input, and their results are compared before anything acts. At Hot Chips that year Tesla said the chips could run either redundantly or separately, so the same board that checks a plan against its twin can run two different plans side by side. From late 2023 Tesla has reportedly used both chips as one, to fit networks that had outgrown a single chip. That was a response to the compute limits of the moment. It doesn’t change the design, and the design is the point. Tesla paid to put enough compute in every car to run two plans at once and compare them, and a board that can compare two plans can compare the production plan with a candidate’s. The same investment serves safety and evaluation, and a company that pays for one gets the other nearly free. Software needs the same dual configuration: production and its challenger, on the same inputs, with every divergence logged.

Tesla Autonomy Day slide titled Compare: the HW3 board with its two chips highlighted and arrows passing results between them

Tesla’s own slide for it was titled Compare: two chips on one board, each passing its plan to the other before the car acts. Image: Tesla, Autonomy Day 2019.

A dual configuration: on the board, and in software

The same shape at two scales. On the board, two chips plan from the same cameras. In the factory, two stacks answer the same traffic.

Tesla is not alone. Formula 1 teams began 2026 concept work in 2024 while racing the 2025 car, and the FIA had to ban 2026 wind-tunnel work before January 2025, because building this year’s car and next year’s at the same time is how the sport works. The challenger is always on the bench.

The loop

Here is the pattern with the car removed: a dual configuration for software. Two stacks exist at all times. The champion serves production. The challenger is a greenfield rebuild, not a refactor and not a migration, written from the ledger with everything known today. It receives the same inputs as the champion and is not allowed to act on them. The harness compares what each one did. Where they diverge, something adjudicates: a model judge for the bulk, a person for the cases that matter. The verdict (champion right, challenger right, both wrong, or legitimately different) goes into the ledger with a reason.

The challenger runs in shadow until it earns production

The same inputs reach both stacks; only the champion acts. Verdicts flow into the ledger, the ledger feeds the next build, and promotion swaps the challenger into the champion’s place.

Every cycle starts with a charter, written before any code. It answers one question: what changed? Not what the new version will do, but why the old one stopped being the right answer. The charter records what changed in the environment, the capabilities, the learnings and the resources; what limit the champion hit and how it showed up; what premise the challenger rests on; and what “better” means, as a measure named in advance. A rebuild with no stated reason is a preference. A rebuild with a charter is a hypothesis, and the shadow run either confirms it or refutes it.

Promotion is a rule, not a ceremony. The challenger becomes the champion when it reaches behavioural parity on everything that matters and beats the incumbent on the measure the charter named: cost, latency, maintainability, time-to-change. The day it is promoted, a new challenger starts from a ledger that now holds every verdict from the last cycle. The cadence is a choice. Formula 1 does it once a season. A software company might run quarterly, or keep a challenger in construction at all times.

The champion does not freeze while this happens. It keeps getting incremental fixes, because customers need them. Each fix lands in the ledger as a requirement, a decision or a verdict, and the challenger reads the ledger, not the champion’s code. So the fix reaches the challenger as knowledge rather than as a patch. Incremental development continues on the champion. The factory adds the rebuild beside it.

The ledger is the asset

If rebuilding is cheap, the code is no longer what you own. What you own is the record of what the system must do and why, good enough that someone with no history could build from it. Most companies do not have this. They have a codebase, a backlog, and the memories of whoever is still around.

A ledger needs four things a wiki does not have:

  • A charter per cycle. The reason for this rebuild, as above. The version bump is the charter’s title.
  • Attribution with an expiry. Every requirement carries the name of the person who set it and the condition under which it stops being true. Musk’s version of this rule, as Walter Isaacson records it (a requirement must come from a person, never a department), exists so a rebuild can ask someone rather than inherit a rule nobody can defend.
  • Decisions with their alternatives. Architecture decision records got this right in 2011: record the context, the options considered, the choice, and the consequences. The challenger needs the options you rejected as much as the one you took, because the reason you rejected them may no longer hold.
  • Verdicts. Every divergence that was adjudicated, with the ruling and the reason. Nobody keeps this today. It is what makes the second cycle smarter than the first.

Chesterton’s fence runs in reverse here. The classic rule says do not remove a fence until you know why it is there. A ledger that records why every fence was built tells the challenger which ones to pull. A fence with no entry is a fence to remove, and the harness will tell you if that was wrong.

At the deployment level the same discipline is a manifest, the software version of Tesla’s per-car record: which version of every component is live, so the challenger is compared against the exact champion it will replace, not an average of the last six releases.

The team judges; the agents write

Rewrites used to need a particular kind of engineer. A team that has built a system three times can build it a fourth time fast, because the requirements and the lessons are in their heads. That is why full rebuilds worked at Tesla and SpaceX and failed almost everywhere else: most teams have not built it three times, and the people who built it once have left.

Agents change the arithmetic. An agent working from a good ledger is doing it for the fourth time on its first attempt. The scarce skill moves from writing to judging: specifying what the system must do, ruling on divergences, deciding which fences to pull, deciding when the challenger has earned promotion. Karpathy described the same shift in 2017: engineers stopped writing the driving logic and started curating the data that trained it. Here they curate the ledger and the evals, the tests that score whether the system behaves as it should.

The team shape follows. A small group owns the ledger and the harness. Agents produce the challenger. People sit where the shadow-mode driver sat: they decide, and their decisions are recorded. An agent can rebuild from a ledger; it cannot rebuild from a department.

What we gain, and what it costs

Set against incremental development, the factory gains:

  • A new ceiling. Incremental work climbs the old curve. The challenger starts on the new one.
  • New capabilities at the level of architecture. v12 was not a feature added to v11. When the frontier, the hardware or the data moves, the challenger is built for it from the start.
  • No visible dip. The worse first version of the new design lives in shadow until it is better.
  • Decisions that expire. Lock-in becomes a recorded choice with an owner and a condition, re-examined every cycle.
  • Knowledge that outlives the team. The ledger holds what used to leave with the people who built the system.
  • Agility as a standing capability. When the world shifts, the response is the next cycle, not a crisis project. Each cycle is cheaper than the last, because the ledger is fuller and the judge has seen more verdicts.

And it costs:

  • A second stack running on real traffic, plus storage for the capture log.
  • Ledger discipline. Every requirement needs a name and an expiry. That is work, and most teams have never done it.
  • Judgment time. People must rule on the divergences the judge cannot.
  • An expensive first cycle. You pay for the ledger that should have existed and the harness nobody built.

That trade is worth it only where the constraints move. For a simple product, it is not.

Where this comes from

Every piece of this has a precedent. What is new is the combination, and the fact that the writing cost which made the combination unaffordable is gone.

PrecedentWhat it got rightWhere the adaptive factory departs
Brooks, The Mythical Man-Month (1975)Plan to throw one away; beware the second system that over-reachesThe ledger holds the second system to recorded requirements; the harness catches the over-reach before anyone depends on it
Spolsky, “Things You Should Never Do” (2000)A rewrite relearns every edge case the old code already knewThe edge cases live in the capture log and the ledger, not in the code
Fowler, Strangler Fig (2004)Never cut over all at once; retire the old system piece by pieceShadow mode gives the same safety without carrying the old seams into the new architecture
Nygard, Architecture Decision Records (2011)Record context, options, decision, consequences, in the repoAdds attribution, an expiry condition, and the verdicts from every cycle
FICO adaptive control (1980s), DataRobot (2020)Incumbent serves most traffic; challenger earns promotion on measured performanceThe whole stack rather than one model; shadow rather than a traffic slice
Twitter Diffy (2015), GitHub Scientist (2016)Run old and new side by side on real requests; log the mismatchesStateful, writes included; semantic comparison rather than response diffs
Karpathy, Software 2.0 (2017) and 3.0 (2025)Engineers curate the data, not the code; speed up the generate-and-verify loopThe curation shift applied to the whole application, with the ledger as the dataset
Thoughtworks test-first modernization (2025)Observed production I/O becomes the spec; the new component runs in parallel before the old one retiresThe closest precedent. The parallel run becomes permanent and its verdicts become a ledger

Spolsky was right for twenty-five years. The cost he was right about has changed.

The sharpest counter-argument comes from Brian Lee: the codebase is the most faithful spec of the product that exists, and regenerating from anything thinner loses behaviour. He is right about most codebases today, which is why the ledger and the capture log have to be built before the first challenger, not after. The SWE-Milestone benchmark makes the same point from the other side: agent success falls from above 80% on isolated tasks to 38% under continuous, dependent evolution, because regressions compound. That is an argument for shadow verification on every cycle, not against rebuilding.

As far as I can find, nobody has applied the champion/challenger name to whole-application rebuilds. The nearest work calls it a parallel run or a golden-master shadow. The framing is open.

The harness

Agents will write the challenger in days. Proving it is acceptable is still the whole job. That proof does not get cheaper with better models, and it is the part I am building.

The harness lets a challenger run against the champion’s real traffic without acting on it. It pairs what each stack did, flags where they diverge, adjudicates each divergence, and records every ruling in the ledger the next build reads. The stacks stay ordinary applications with a small SDK inside. The harness owns everything between them:

  • Capture and replay. Every production input is recorded, with personal data masked, and replayed to the challenger, or mirrored to it live. Synthetic data loses the long tail, and the long tail is where divergence lives.
  • A side-effect boundary. The challenger cannot send the email, charge the card, or call the partner API. The SDK replaces those drivers with recorders, so the harness compares what each stack meant to do. The boundary is checked when the challenger starts, not left to convention.
  • Semantic comparison. A rebuild changes the data model, so byte-level diffs are useless. Each stack exposes the same canonical views of its state, and the harness compares outputs, state and intended side effects on meaning, within stated tolerances.
  • Adjudication. A model judge rules on the bulk of divergences and sends the uncertain and the critical ones to a person. Every ruling becomes a verdict with a reason, and the judge is measured against the people who overrule it.
  • The ledger. Charter, requirements with names and expiries, decisions, verdicts, and a manifest of exactly which build was compared. Agents read it to build the challenger. Only people write requirements and charters.
  • A promotion gate. A scorecard tracks coverage, parity and the charter’s criteria, and refuses promotion while any critical divergence is open.
One tap in front of the champion; everything else runs beside it

One component, the tap, sits in front of the champion and records every input. Everything else runs beside production and can fail without the champion noticing.

None of these ideas is new on its own. Facebook dark-launched Chat in 2008. Twitter’s Diffy and GitHub’s Scientist ran old and new code side by side and logged the mismatches. All of them covered one service or one code path, and all of them assumed no side effects. The harness covers the whole stack, writes included. That extension is the hard part, and it is the part worth owning.

It is also useful before any challenger exists, which sets the order to build it in:

  1. Record the champion. Capture production inputs and write the ledger for what already exists: every requirement with a name and an expiry, every decision with the options it rejected. Where nobody can name the owner, record that too. It is a fence to pull.
  2. Run the harness champion against champion. Last release against this one. That gives a regression check on real traffic for every release, and it calibrates the comparison and the judge before a challenger exists to be judged.
  3. Write the charter, then let agents build the challenger from the ledger alone, with no access to the old code. Where the ledger is incomplete, the harness shows exactly where, and that gap is the first verdict of the first cycle.

The first cycle is the expensive one. Every cycle after it is cheaper. That is the difference between a rewrite and a loop: a rewrite is a bet you place once, a loop is a capability you own. I will write more about the build as it goes.

So, back to the question. If you knew everything you know now, and had what you have now, what would you build? For most of the history of software the honest answer was that it did not matter, because you could not afford to build it. That is no longer true. What remains is to prove the new one is better, every cycle, on real traffic, with a person deciding the cases that matter. That is what the harness is for.

Sources

The rewrite debate

Shadow deployment and champion/challenger

Tesla, SpaceX, Formula 1

AI-era rewrites and the spec-as-asset argument

Table of contents
~/ideas/contents
Contact
~/contact

Write to me about the problem you want to fix.

Write to me
~/mail/new
Ctrl+Enter sends.
1 – 5 switch workspace n b where next: the next step, or back Enter go to the menu ↑ ↓ Enter in the menu: move, open space / Ctrl K launcher: essays, builds, eras, themes, commands h l j k move focus: h l between columns, j k within one H L resize: move the split left or right J K resize: grow or shrink the focused side window f fullscreen the focused window w or q close the window under the mouse (reopen it from the launcher) t pick a theme m or c contact: write to me Esc leave a text box, the menu, fullscreen or a dialog Ctrl Enter send the message in the mail window ? this list