In December 2015, ten engineers from Google published a paper at NeurIPS with an unusual argument for a machine learning conference. Their subject was not accuracy. It was maintenance.
D. Sculley and his co-authors argued that ML systems accumulate technical debt faster than ordinary software, for reasons specific to machine learning. Their central observation is reproduced everywhere and understood rarely: in real-world ML systems, only a small fraction of the code is the machine learning. The rest is infrastructure around it.
The famous diagram from that paper shows a tiny black box labeled ML surrounded by much larger boxes — configuration, data collection, feature extraction, serving, monitoring. It appears in vendor decks as an argument for buying a platform.
That is a fair reading of the diagram and a poor reading of the paper, because the debts they describe are not solved by tooling.
The two that explain most of the rest
Entanglement, which they named the CACE principle: Changing Anything Changes Everything.
If a model uses a set of features and you change the input distribution of one, the importance and weights of all the others may shift. Adding a feature does the same. So does removing one. Their conclusion is blunt — no inputs are ever really independent. And it extends well past features, to hyperparameters, learning settings, sampling methods, convergence thresholds and data selection.
The operational consequence is severe. In ordinary software you can reason about a change locally. In an ML system you cannot, so every change requires end-to-end validation, and any process that assumes isolated improvements rests on a false premise.
Glue code.
Researchers build general-purpose packages. Getting your data in and out of one requires a large volume of supporting code, and that code freezes your system to the peculiarities of the package you picked. Testing an alternative becomes prohibitively expensive, so you stop testing alternatives — which means a mature package can inhibit the very improvements it was adopted to enable.
The rest of their list has the same character: hidden feedback loops, undeclared consumers, unstable data dependencies, configuration debt, and changes in the external world. Every one exists at the system level rather than inside any component, which is exactly why it is hard to see and impossible to purchase your way out of.
What follows for how you organize
Read as an operating brief rather than an engineering paper, the findings point at a small number of things that determine whether an ML capability survives production.
| What matters | Why, from the paper |
|---|---|
| One team owns the whole pipeline | Debt lives at the system level. Splitting model development from data engineering from serving guarantees nobody owns the place the problems are. |
| Every change is validated end to end | CACE makes local reasoning invalid. A gate that tests the component is testing the wrong thing. |
| Data dependencies are tracked like code dependencies | An upstream table changing its distribution is a breaking change with no compile error. Most organizations have no way to notice. |
| Consumers of model output are registered | Undeclared consumers build on your predictions without telling you, which turns any model change into a change to systems you do not know exist. |
| Configuration is reviewed like code | Configuration debt accumulates silently and is where a surprising share of production incidents begin. |
| Feedback loops are looked for deliberately | A model influencing the world that generates its next training data drifts in ways that look like decay and are not. |
| Retirement is planned | The framing is debt, and debt is repaid. A model nobody can retire is a permanent liability whose owner has probably left. |
What this means for buying a platform
Platforms address the parts of the diagram that are genuinely tooling problems: pipelines, registries, serving, monitoring infrastructure. That is real value. It is not the debt the paper describes.
Entanglement is a property of your model, not your platform. Undeclared consumers are an organizational discipline. Tracking data dependencies requires knowing who owns the upstream table, which is a conversation rather than a feature.
No platform will tell you that the marketing team is consuming your risk score in a dashboard you have never seen. Nothing will, except asking.
One question, and it is not about tooling
MLOps guidance has largely become tooling guidance, and the original diagnosis has been reduced to an argument for buying the boxes around the small black box.
The paper says something harder. These debts are structural to machine learning, they live between components rather than inside them, and organizations that manage them well do so through ownership boundaries and validation discipline rather than through purchases.
One question tests whether your ML practice is on solid ground, and it is not which platform you run. It is whether anyone can produce a list of every system consuming your models’ output. If they cannot, you cannot safely change the model — and a model you cannot change stops being an asset fairly quickly.
Source: D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo and Dan Dennison, "Hidden Technical Debt in Machine Learning Systems", Advances in Neural Information Processing Systems 28 (NeurIPS 2015), pp. 2503-2511. The paper states qualitatively that only a small fraction of code in real-world ML systems is ML code; specific percentages widely attributed to it do not appear in the paper and are not used here.
Related Reading
- MLOps Platforms
- AI/ML Platforms
- Data Quality & Observability
- LLM Observability & Evaluation
- An executive guide to open-weight models
- Most of your agent projects should be workflows
- Technical debt was never about bad code. That is the whole point of the metaphor.
- From Models to Systems: How Enterprise AI Actually Gets Built
- Building an Enterprise AI Governance Framework: Policy, Risk, and Oversight
Independent. No sponsorships. Unsubscribe anytime.