Knight Capital lost $460 million in 45 minutes because of code it stopped using in 2003

On 1 August 2012, Knight Capital Americas received 212 retail orders from its customers.

Its automated router, SMARS, turned those 212 orders into millions. In roughly 45 minutes it obtained over four million executions across 154 stocks, totaling more than 397 million shares. Knight ended the morning holding an unintended net long position of about $3.5 billion in 80 stocks and a net short position of about $3.15 billion in 74.

The firm lost more than $460 million. It had been handling around ten percent of all trading in US listed equities. Within months it was acquired.

The story is usually told as a deployment error, because there was one: a technician copied new code to seven of eight servers and missed the eighth, and no second technician reviewed the work because no written procedure required one.

That is true and it is the least interesting cause.

The code that should not have been there

The eighth server was still running a function called Power Peg. Knight had stopped using Power Peg in 2003. Nine years later it was still present on production servers and still callable.

Worse, it had been quietly broken in the interim. Power Peg relied on a cumulative quantity function that counted executed shares and told the code to stop sending orders once a parent order was filled. In 2005 Knight moved that counting function to an earlier point in the SMARS sequence. Nobody retested Power Peg afterwards to see whether it would still work if called — reasonably enough, since it was never supposed to be called again.

So the dormant code had lost its stopping condition seven years before it woke up.

The final detail is the one that turns this from bad luck into a pattern. The new code repurposed the very flag that used to activate Power Peg. On the seven correctly updated servers, that flag triggered the new functionality. On the eighth, it triggered a nine-year-old function with its brakes removed.

The alarms that were not alarms

Starting at about 8:01 that morning — an hour and a half before the market opened — Knight’s systems generated automated messages referencing SMARS and reporting an error described as Power Peg disabled. Ninety-seven of them went to a group of Knight personnel before the open.

They were not designed as alerts, and staff generally did not read them. The SEC’s finding is precise on this point: the messages were sent in real time, were caused by the deployment failure, and offered a chance to identify and fix the problem before the market opened. Nobody acted on them, and nobody used them to diagnose the problem afterwards either.

Then the response made it worse. Attempting to fix the problem in a live trading environment, Knight uninstalled the new code from the seven servers where it had been deployed correctly — which activated Power Peg on those seven as well.

The near-miss they had already had

This is the part that should be uncomfortable for anyone running a technology estate.

In October 2011, Knight ran a weekend disaster recovery test using test data. When trading resumed on Monday, one desk carried on using the test data to generate automated quotes. The firm lost nearly $7.5 million.

Knight responded. It limited the system to market hours, changed a control so the system would stop quoting after an execution, and added a check of test data to the disaster recovery checklist.

Every one of those fixes addresses precisely what happened, and nothing else. The SEC’s finding is blunt: Knight reacted narrowly, limiting its responses to changes designed to prevent the exact problem at hand from recurring. It did not ask the broader question — whether it had adequate controls to prevent erroneous orders regardless of which system produced them or why.

Ten months later, a different system produced erroneous orders for a different reason, and none of the October fixes applied.

The controls that existed and did not help

Knight was not uncontrolled. It had controls at the customer interface, in its order management system, and in its internal execution system. What it lacked was a control at the point immediately before orders left for the market — which is where SMARS sat.

It had position limits, including a $2 million gross limit on the account that accumulated the unwanted positions. That limit was not linked to anything that could stop orders being sent.

Its primary risk tool, PMON, watched positions after execution. It required a human to be looking, did not display the limits it was meant to enforce — so the person watching had to already know them — generated no automated alerts, and became inaccurate under exactly the high-volume conditions that would signal a crisis.

And the firm’s own compliance assessment had concluded its controls were satisfactory. That assessment worked by inventorying existing controls and confirming they functioned as intended. It never asked what would happen if SMARS malfunctioned.

That is the difference between auditing the controls you have and asking what could go wrong. Almost every compliance review in existence does the first.

What carries over

Two things carry over to organizations that will never route an equity order.

The first is that undecommissioned code is not inert. Power Peg cost nothing to leave in place for nine years, right up until it cost $460 million. Anything still deployed is still callable, and the safeguards around it decay silently because nobody maintains guardrails on a function nobody uses. Your estate has its own Power Peg. It is probably documented as retired.

The second is that the response to a near-miss reveals more than the near-miss does. Knight’s $7.5 million incident was a gift — a cheap demonstration of a class of failure. The firm fixed the instance. Ten months later the class returned wearing different clothes.

After your next incident review, read the remediation actions and ask whether any of them would have caught a different failure with the same shape. If the answer is no, you have not fixed anything. You have documented one specific thing that will not happen again.

Related Reading

• Observability & APM Platforms
• Incident Management & On-Call
• DevSecOps & Application Security Testing
• CI/CD Pipeline Platforms
• Technical debt was never about bad code. That is the whole point of the metaphor.
• Site Reliability Engineering: Implementing SRE Principles in the Enterprise
• API Monitoring in Modern Architectures: Reliability, Latency, and Governance