Testing the live system would have prevented 26 outages out of 40; more redundancy, 9

There is a number from 2003 that ought to appear in more board meetings than any other. Three Berkeley researchers — David Oppenheimer, Archana Ganapathi and David Patterson — took the internal outage reports of three large internet services, read them one by one, and then asked the question nobody asks after a postmortem: which technique, specifically, would have prevented this outage? For the service whose material was most complete, with forty outages analysed, the ranking came out like this: testing the running system would have prevented 26; exposing and monitoring failures better would have shortened detection or repair in more than ten cases; adding more redundancy would have prevented 9.

Twenty-three years later, the infrastructure budget of almost any Spanish company is built around the 9. The second node, the second power supply, the second line, the second array. And the 26 — actually testing what is already switched on — is still what gets done when there is spare time, which is never.

What they measured, and what they did not

It is worth bounding before using, because a number like that is easy to stretch. The study is called Why Do Internet Services Fail, and What Can Be Done About It? and it was presented at the USENIX USITS symposium in 2003. It analyses three anonymised services: Online ("a mature online service/Internet portal"), Content ("a bleeding-edge global content hosting service") and ReadMostly ("a mature read-mostly Internet service"). For the first two they had 296 and 205 component failure reports respectively, which translated into 40 and 56 service failures — the ones users notice. For the third, 21.

The ranking that opens this post is from the first service, over those forty outages. It is not a universal law nor an industry statistic: it is one organisation's count, done by people with the reports in front of them. And there is another precision worth making, because it changes what the word means. What the study calls online testing is not exactly breaking things on purpose. They define it thus: "pre-deployment testing prevents component faults in the deployed system, and online testing detects faulty components before they fail during normal operation". That is, checking that what is already running does what it claims to do. And they add a caveat that explains why almost nobody has it in place: to be useful, the test needs "application semantic information", their literal example being "posting a news article and checking to see that it showed up". A 443 port answering is not enough.

And a warning about the use we are about to make of it, so nobody takes away an idea the study does not support: the 26 is not the fault-injection figure. Breaking things on purpose appears in that same section as a different technique and with no number, carrying only the label "significant potential benefit". Everything we propose below — powering off a node, pulling a cable, revoking a credential — is our extrapolation from the 26, not a conclusion of theirs. We say it here rather than in the sources note.

The 9 can be bought; the 26 has to be done

The asymmetry between those two numbers is not technical, it is commercial. Redundancy comes with an invoice, a delivery note and a serial number: it gets approved in a committee, depreciated over three years and can be pointed at. Testing the live system has none of that. It is time from competent people doing something that, if it goes well, produces no visible deliverable — only confirmation that what you believed is still true. It is, literally, the hardest thing to sell and the hardest to defend when budgets get cut.

And yet the same study explains why redundancy yields less than it appears to. The number one cause of service failure in two of the three services was not the hardware: it was the operator. The authors say it plainly — "operator error is the leading cause of service failure in two of the three services" — and sharpen it further: "more than 50% (and in one case nearly 100%) of the operator errors that led to service failures were configuration errors", and those errors "arose when operators were making changes to the system, e.g., scaling or replacing hardware, or deploying or upgrading software".

Read it slowly: scaling or replacing hardware is on the list of activities that cause outages. Buying the second node is not free in risk terms; the day you install it, you touch things. Redundancy is not insurance you sign and forget, it is one more part somebody has to maintain, update and periodically replace. There is one more figure from the study that is rarely quoted and stings: operator errors "often take a significantly longer time to repair" than other kinds. A dead disk is fixed by a spare. A config file written badly at eight in the evening is fixed by someone who first has to realise that is what happened.

Here we have to say something that undercuts us, because it sits in the same section of the study and leaving it out would be cheating. The authors do not conclude that you should buy less redundancy. When they put cost into the equation they write the opposite: "we observe that the techniques of adding additional redundancy and better exposing and monitoring for failures offer the most significant 'bang for the buck' (...) while incurring relatively low cost". And they warn of something else: Online already used some redundancy, monitoring, isolation, proactive restart, and pre-deployment and online testing, so the table underestimates what adding those techniques would have achieved. What they nowhere say is that you can skip the 26.

In 2003 they already named the other half

The paragraph that closes that analysis is the one that really interests us, because it names something that did not yet exist as a discipline: "automatic sanity checking of configuration files, and online fault and load injection, also appear to offer significant potential benefit". Fault injection into the production system. In 2003. Nine years before Netflix open-sourced Chaos Monkey, on 30 July 2012, and thirteen before the team maintaining it wrote the paper that gave it an academic frame.

That paper — Chaos Engineering, by Basiri and six other authors, published in IEEE Software in 2016 — defines the idea in a sentence with nothing epic about it: "Many large tech organizations are using experimentation to verify the reliability of such systems. We use the term 'Chaos Engineering' to refer to this approach." Experimentation. Verify. These are laboratory words, not demolition words.

Chaos Monkey breaks things from nine to three, and ships switched off

The tool's fame has done more harm than good, because the name conjures a monkey loose in the data centre. Netflix's documentation, which is public and takes ten minutes to read, describes something considerably duller and considerably smarter. Every weekday, for each instance group, Chaos Monkey "flips a weighted coin" to decide whether to kill one; if it comes up heads, it schedules the termination "at a random time between 9AM and 3PM that day". Not at night. Not at weekends. During office hours.

It gets more cautious still. The configuration file, in the section the documentation itself titles "Defaults", opens with enabled = false; next comes my favourite, leashed = true, with this comment beside it: "if true, terminations are only simulated (logged only)"; and then schedule_enabled = false. The most famous tool for breaking things in production ships disabled, unscheduled and in dry-run mode. And it has one more part almost nobody mentions: an outage checker, whose documentation opens with "an outage checker is used to automatically disable Chaos Monkey during ongoing outages". That is: it breaks nothing if you are already broken.

The knob you turn, incidentally, is not brutality: it is frequency. You configure μ, the mean number of work days between terminations, and the daily probability follows from it (p = 1/μ). With μ=5, one working day in five. Chaos Monkey's aggressiveness is a number the service owner chooses, and by default it is zero.

You are not Netflix, and you do not need to be

A forty-person software company with three Proxmox nodes, a small Ceph, a backup server and Microsoft 365 does not need a daemon killing machines at random. It needs the other part of the discipline, the part that scales downwards: the method. The five principles published by the people who formalised it are to build a hypothesis around steady-state behaviour, vary real-world events, run experiments in production, automate, and minimise the blast radius. Of those five, the ones worth money in an SME are the first and the last.

The experiment card: six fields on half a sheet of paper

What separates a drill from an outage you caused yourself is not the tool: it is having written six things down before touching anything. If they are not written, it is not an experiment; it is a hunch with witnesses.

• Hypothesis. One falsifiable, concrete sentence, not a wish. "If I power off node 2, the VMs running on it start on another node and the ERP accepts logins again within five minutes." If the sentence cannot be disproved with a clock, it is not a hypothesis.
• Blast radius. What is in and what is out, in writing. One node, not the cluster. A test mailbox, not the finance director's. One site, not all four.
• Signal you watch. The business metric, not the machine one. Logins per minute, confirmed orders, mail delivered. The Berkeley study said as much: the test needs application semantics. A node answering a ping means nothing.
• Stop condition. The exact number and minute at which you abort and revert. Written beforehand, while nobody is in a hurry. In the moment it always looks like it will hold a little longer.
• Who can call it off. A name, not a job title. And that person in the room, not on call.
• What we learned. Two lines at the end, even — especially — if nothing happened. An experiment that confirms the hypothesis is also a result, and it is what turns the drill into a series rather than an anecdote.

Four experiments, from smaller to larger blast radius

This is what we would put on the table for a small, ordinary estate. The order matters: each one assumes the previous went well. You do not start at the third because it sounds braver.

Notice that the first one breaks nothing. That is deliberate. If experiment one is not passed, the other three do not happen: you fix the documentation and come back next month. We wrote this a few weeks ago in other words in the continuity plan nobody has rehearsed, and it remains the pattern we meet most often: the plan exists, it is well written and nobody has read it with a clock in hand.

Why during office hours and not at three in the morning

The industry habit is to touch things at night, and for a planned change that makes perfect sense: fewer users, lower cost if it goes wrong. For a drill, no. A drill happens when the people who know how to fix it are awake, which is exactly why Netflix schedules it from nine to three. At four in the morning nobody is watching the business metric, nobody notices that the order did not go through, and the finding is lost.

And there is an uncomfortable corollary. If the only possible answer to "can we power off a node on Thursday at eleven?" is "absolutely not", that is already the experiment's result. You have just discovered, without powering off anything, that your system cannot tolerate losing a part during working hours. Which is precisely when parts break.

When NOT to do any of this

We say this because it is the part enthusiastic articles leave out, and because we have run into all four situations. If any of these pieces is missing, a "chaos day" is not engineering: it is an outage you scheduled and paid for.

• You have no backup you have ever restored. Then the only experiment for you is number one in the table, and until it passes, the others do not exist. An untested backup is not a backup: it is a lucky charm.
• You have no one-step way back. If reverting means a reinstall, the blast radius is not what you wrote down: it is everything.
• You have no telemetry telling you whether you are fine. Without the business signal you cannot tell "it held" from "nobody was using it at that moment". The experiment yields no result: it yields a feeling.
• Nobody has the authority to stop it. If aborting depends on convincing somebody over the phone, the experiment lasts exactly as long as that call, and that call is always longer than the stop condition.

What usually goes wrong when it goes wrong

The most frequent failure in a drill is not that the part fails to recover. It is the second wave: everything connected to it starts retrying at once, and the retry load finishes off what the original failure had only grazed. We covered it in full in the retry storm, where the second outage is the one you cause, and it is the reason the stop condition is written beforehand: in the middle of a retry storm, everybody simultaneously proposes waiting a bit longer to see if it recovers.

What we measure, and what we do not promise

Our last timed full-recovery drill came out at 14 minutes. It is an internal figure and we bound it as plainly as we state it: it is evidence, not a promise, and it is not a contractual guarantee of anything. It measures our system, with our documentation and our people, on a specific date. A customer with a different architecture will get a different number, and the value of the exercise is not the number: it is that it exists and carries a date.

And it is worth remembering what this is measured against. 99% availability is 87.6 hours a year in the dark: more than two working weeks. That figure shows up in contracts as a reassuring ornament. Nobody meets it or misses it by luck, and certainly nobody checks it on the day it is signed. The only thing separating a promised number from a known number is having timed it at least once.

What we would do this quarter

First, experiment one: have someone who did not build the system restore a file from a two-day-old backup with the documentation in front of them, stopwatch in hand, on a Tuesday at eleven. Second, write the six-field card for experiment two before running it — and file it with a date, even if the result is "nothing happened". Third, put the result on the agenda of the monthly meeting, next to the spend, because that is where this survives or dies. And fourth, repeat. A one-off drill is an anecdote; four a year, dated, are a series that tells you whether you are getting better or worse.

Sources (checked 16 Oct 2026): the count of techniques over the 40 service failures at the Online service ("online testing would have helped the most, mitigating 26 service failures", "simply increasing redundancy would have mitigated 9 failures", the "more than 10 instances" for monitoring and the sentence on "online fault and load injection"), the definition of online testing, the 296 and 205 component failure reports, the 40, 56 and 21 service failures, the description of the three services, the sentence "operator error is the leading cause of service failure in two of the three services", the "more than 50% (and in one case nearly 100%)" of configuration errors, the change context (scaling or replacing hardware, deploying or upgrading software) the longer repair time for operator errors, the sentence "we observe that the techniques of adding additional redundancy and better exposing and monitoring for failures offer the most significant bang for the buck (...) while incurring relatively low cost" and the warning that Online already used some of those techniques, so the table underestimates their effect, all come from Oppenheimer, Ganapathi and Patterson, "Why Do Internet Services Fail, and What Can Be Done About It?", USENIX USITS 2003. The 9AM to 3PM window, the weighted coin, the geometric distribution and the μ work-day parameter come from Chaos Monkey's termination behaviour documentation; the defaults enabled = false, schedule_enabled = false, leashed = true and start_hour/end_hour, from its configuration file format; the outage checker, from its plugin documentation. The definition of the discipline and the experimentation quote come from Basiri et al., "Chaos Engineering", IEEE Software 33(3), 2016; the five principles, from principlesofchaos.org. The nines (87.6 h, 8.76 h and 52.6 min a year) are arithmetic. What is ours, and we bound it: reading the 26 against the 9 as a commercial asymmetry, the six-field card, the table of four experiments, the four situations in which NOT to do it, and the 14 minutes, which are an internal figure from a dated test and not a guarantee.