Circuit Breaker

Chapter OneLatency

You want to know who actually decides whether you get your money out?

Not the founder. The founder is on a stage in Lisbon. Not the exchange, because an exchange is not a thing, it is about nine hundred people and a great deal of borrowed infrastructure pretending very hard to be a building.

It’s a woman on the fourth floor of an office block in Singapore with a laptop and a red button that isn’t red and isn’t a button, and in May 2021 that was me.

My name is Meilin Chua. I ran site reliability for Titan, which at that point was the fourth largest crypto exchange in the world by volume and the second largest by the sort of volume you could actually believe. Nine hundred staff. Eleven million accounts. One matching engine, in a rack in a data centre in Tai Seng that I could reach in nineteen minutes on a normal day and had once reached in eleven.

My job was one sentence long. The system stays up.

The other half of my job, the half nobody wrote down, was that when the system could not stay up, I was the person who turned it off. Deliberately. Before it fell over on its own. Because a system you switch off comes back, and a system that falls over under load comes back with its ledger in pieces, and the difference between those two outcomes is about four billion dollars and everybody’s licence.

There is a word for it. Circuit breaker. It comes from equities, where it is a proud and respectable thing, a mechanism the regulator makes you have.

Let me explain what one actually is, because the phrase gets used by people who have never been near the switch.

An exchange is a queue. That is the whole secret. Every order anybody sends anywhere in the world arrives at one machine, in a rack, and that machine puts them in order and matches buyers to sellers one at a time, very fast, forever. Ours did about a hundred and forty thousand a second on a busy day.

A queue has exactly one failure mode. Things arrive faster than they leave.

When that happens, the queue grows. Orders sent at ten o’clock get matched at ten oh one against prices that stopped being true sixty seconds ago. Nothing has broken. Every single component is working perfectly. The system is simply arriving in the past, further behind with every second, and there is no amount of engineering on this earth that fixes it, because you cannot make a queue shorter by asking it nicely.

You can only stop putting things in.

In crypto that stop is the thing that happens to you at the exact moment you most need it not to, and every trader on earth believes we do it on purpose to hurt them.

Rafael Okonjo believed that too, on the record, for eight months, mostly to my face.

He was head of institutional coverage, which meant he owned the relationships with the market makers. Big firms in London and Chicago and Amsterdam who quoted prices on Titan all day and made the whole thing look like a market. Lagos-born, London-raised, LSE, three languages and a fourth he was rude about. He had the sort of composure you can hear on a phone line. Clients loved him. He wore it like a coat.

We had circled each other since the previous September. Two floors, eleven feet of glass, and a standing disagreement.

His position: every second of downtime is a client who never comes back.

My position: every second of uptime past capacity is a ledger event we cannot undo.

Neither of us was wrong. That was the entire problem.

It started in a post-incident review on the ninth of September, a small one, eleven minutes of degraded performance on a Tuesday, the sort of thing that generates a document and no consequences. I presented. He asked four questions. The fourth one was good enough that I had to go away and find out the answer, which had not happened to me in that building in two years, and I came back the next morning with it and put it on his desk without being asked.

He read it. Then he said, “You didn’t have to do that.”

“You asked a real question. Almost nobody in this building asks a real question, they ask a question that establishes their seniority.” I was already leaving. “You’re welcome.”

After that it was a habit. He came up to the fourth floor when something was wrong, which was reasonable, and then when nothing was wrong, which was not. He would stand at the end of the row with his hands in his pockets and ask me what the graphs meant, and I would tell him, and he would argue, and he was very good at arguing and completely wrong about eighty per cent of it, and I looked forward to it in a way I did not examine.

In February he was in Amsterdam for a week and I found that Thursday took approximately eleven hours to get through.

There was also a woman in Hong Kong that he saw until about January, and there was a man in my own life until the previous June, and neither of those facts was ever discussed by either of us at any point, which tells you everything about what we both understood was happening.

I told myself very firmly that the reason I checked the fourth-floor lifts at 08:40 was the lifts.

On the nineteenth of May 2021 I got in at 06:15 because the overnight had been strange.

Not bad. Strange. Order rate up forty per cent on a Wednesday with no news. Queue depth on the matching engine sitting at four hundred and climbing in slow steps rather than spikes, which is a different thing entirely and much worse, because a spike is somebody’s bot and a slow climb is everybody at once.

At 09:20 Xander Cole tweeted about the environmental cost of proof of work, and eleven million accounts opened at the same time.

I watched the graph do something I had only seen in the pre-production simulation we ran twice a year and had privately considered unrealistic. In the simulation it is a smooth curve, because the load generator is a program and programs are polite. In life it went up in a staircase, a flat second and then a step, a flat second and then a step, and each step was a country waking up to the same notification.

At 09:31 I put the incident channel into what we called amber, which is not a public state, it is nine engineers being told to cancel whatever they are doing today.

At 09:34 I sent the first of four messages to the founder’s chief of staff, none of which received a reply, which was normal and which I noted in the log anyway, because a log is not a record of what happened, a log is a record of what you told people and when.

At 09:38 my deputy asked whether we should pre-warn the market makers.

That is a real question with no good answer, and I want to set it out plainly because everything after it turns on it. If you warn the firms that provide the prices, they withdraw. That is not villainy, it is their job; a market maker who leaves quotes in a book he cannot get out of is a market maker who goes bust by lunchtime. But when they withdraw, the book empties, and an empty book means the next ordinary sell order falls a very long way before it finds a buyer.

So: warn them, and you cause the crash you are trying to survive. Don’t warn them, and you are holding two hundred million dollars of somebody else’s money in a burning building without telling them there is a fire.

I said, “That’s Rafa’s call, not ours.”

At 09:40 he came up without being asked, which he never did.

“How bad.”

“Queue depth eleven thousand.” I turned the monitor. “Ninety-second p99 on order acknowledgement. It was ninety milliseconds at breakfast.”

He looked at it for a while. To his credit he did not ask me to explain it, which most of that building would have.

“So a client who hits send now,” he said, “finds out in a minute and a half whether he got filled.”

“A minute and a half is optimistic. That’s the ninety-ninth percentile at the moment I measured it. The moment has moved.”

He took out his phone, and I watched him do a thing I had never seen him do, which was hesitate.

“I’ve got Amsterdam on hold,” he said. “Two hundred million of quoted liquidity. If I tell them what you’ve just told me, they pull it in nine seconds, and the book goes so thin that anything that hits it goes straight through the floor.”

“Then don’t tell them.”

“That’s what I was hoping you’d say.” He was already walking. “I hate that you said it.”

I want to be precise here, because a great many people have since been imprecise about it in print. He did not lie to Amsterdam. He told them latency was elevated and being managed, which was true, and he did not give them the number, which he was not obliged to.

He knew what he was doing. He held that liquidity in the book for another two hours and eleven minutes, and every one of those minutes was a decision he made in full possession of the facts, and there are people alive today with money in their accounts because of it, and there is at least one who is not.

At 11:51 the queue depth number on my screen went to a figure I did not believe.

I refreshed it. It was worse.

I picked up the phone to the data centre and said the sentence I had been employed for four years to be capable of saying, and had never said.

Chapter TwoHalt

Forty minutes.

11:58 to 12:38, Singapore time, the nineteenth of May 2021. Trading halted, API returning a maintenance code, withdrawals frozen, and eleven million people looking at a spinner.

That is the whole event. That is the entire thing that people wrote articles about for a year.

Here is what was happening inside it.

The matching engine did not fall over. That was the point. We took it down at ninety-four per cent of the load at which it fails, and it came back at 12:38 with a complete ledger and not one broken trade, which in the industry that year was a genuinely unusual outcome and which precisely nobody has ever thanked us for.

Rafa was in my ops room by 12:02. He had not asked. He had simply walked into a room he was not cleared for and shut the door.

“Bring it back.”

“No.”

“Meilin. Amsterdam is flat, Chicago is flat, and every retail account on this exchange is looking at a price that stopped four minutes ago while the rest of the world keeps trading.” His voice was completely level. That was the frightening thing. “The price outside is falling. When you open, they don’t open at the price they closed at. They open at the price the world got to while they were locked in.”

“I know.”

“They will be liquidated on a gap they had no way to trade out of.”

“I know exactly what happens.” I did not look away from the load graph. “I have known it since 06:15 and I have run it four ways and I am telling you the other outcome is worse.”

“How.”

“Because in the other outcome the engine goes down on its own at a hundred and four per cent and I cannot tell you which trades happened.” I turned round. “Not can’t tell you today. Can’t tell you ever. There is no rollback. Eleven million balances and I would be reconstructing them from logs for six months and every single one of them would be a legal claim. That is the exchange gone. Not embarrassed. Gone.”

He stood there.

“You’re right,” he said.

“I know.”

“I need you to understand that being right doesn’t make it not happen.” He put both hands flat on the desk. “Somewhere out there right now is a man with everything he has in a long position, and in thirty-six minutes we are going to open a hole under him and he is going to fall through it, and the reason will be a load graph he will never see, explained by a firm he will never speak to, and he is going to spend the rest of his life believing we did it on purpose. And he will be almost right. We did do it on purpose. We just did it for a reason he’d agree with if anyone ever told him.”

“Then tell him.”

“They won’t let me tell him.” He laughed once, without anything in it. “Comms will publish four sentences about elevated volumes. I’ve already seen the draft.”

Here is what forty minutes of a halted exchange is actually like, from the inside, since nobody has ever written it down properly.

It is quiet. That is the first thing. The graphs go flat, and the alerting stops, and a room that has been screaming at you since dawn simply stops, and there is nothing to do but watch a queue drain at a hundred and forty thousand a second and count.

Eleven minutes in, the founder came off the plane in Lisbon and started calling. I did not take the calls. That was not defiance; my hands were doing something else. My deputy took them and said the sentence we had agreed, which was that the engine was healthy, the halt was controlled, and the lead would brief at reopen.

Nineteen minutes in, someone in comms sent me a draft statement for approval that used the phrase scheduled maintenance, and I replied with one word, and it went out saying elevated volumes, which was at least not a lie.

Twenty-six minutes in, Rafa was still in the room. He had stopped arguing about eight minutes earlier. He sat on the edge of a desk with his phone face-down beside him, which for him was the equivalent of screaming, and he watched the drain graph with me, and neither of us said anything at all.

At about thirty-four minutes he said, “How long have you known you were going to do this?”

“Since 06:15.”

“You knew at breakfast.”

“I knew at breakfast that today was the day the thing I have been employed for four years to be capable of would probably have to happen.” I did not look away from the graph. “I did not know it would be at 11:51 and I did not know you would be standing there.”

He said, “I’m glad I was standing there,” and I did not answer, and that was the closest either of us had come in eight months.

At 12:36 the queue hit zero and held.

We opened at 12:38.

The gap was eighteen per cent.

I am not going to describe the next six hours in detail because I have described them under oath twice.

Two hundred and forty thousand accounts liquidated in the first ninety seconds. Not gradually. There is a moment when an exchange reopens and every stop-loss order that has been sitting in a queue for forty minutes discovers the world at once, and they all execute inside the same second and a half, and the sound in the ops room when it happened was a sort of collective inhale from twenty engineers watching a counter go up.

The support queue hit two hundred and eleven thousand open tickets, which was a number the support software could not display; it showed a negative figure for about an hour until someone patched it.

Somebody put our office address on Twitter at 14:20. By four there were people downstairs, perhaps thirty of them, and building security closed the lobby, and I went down at half four because it seemed to me that somebody from engineering ought to look at them rather than watch them on a camera.

I stood inside the glass for about a minute. A woman near the front was holding up a phone with a screenshot on it. She was not shouting. That was the thing I have never got past. She was just holding the phone up so that anyone inside could see it, patiently, the way you hold up a photograph of someone missing.

I went back upstairs and worked until ten.

At 22:00 there was an all-hands in the big room on six, and the founder dialled in from Lisbon, and the general counsel explained that the incident report would name a decision owner, because regulators require a named decision owner, and that this was procedure and not blame.

Everyone in that room knew whose name it was. It was on the phone log. 11:51, fourth floor, me.

Rafa stood up.

He said that he had held two hundred million of market-maker liquidity in the book for two hours and eleven minutes on the morning of the nineteenth with full knowledge of the latency figures, that this had materially deepened the book into which the halt released, that the decision had been his alone and was not made at the request of engineering, and that if the report needed a decision owner it should have the correct one.

Then he sat down.

It took me a full minute to understand what he had done, because the first thing I felt, sitting there in a room of ninety silent people, was not gratitude.

It was fury.

Because it was not true. It was not a lie either. Every single word of it was accurate, and he had assembled those accurate words into a shape that put his name in the box where mine belonged, and he had done it in public where I could not undo it without calling him a liar in front of the founder.

He knew that when he stood up. He must have known it.

I found him at half past one in the morning on the loading dock at the back of the building, where the smokers went, where nobody had smoked since the pandemic.

“Why,” I said.

He didn’t answer.

Chapter ThreeRestart

“Because you’d have been finished,” he said eventually. “And I won’t be.”

“That’s not a reason. That’s arithmetic.”

“It’s the only reason there is.” He was sitting on a stack of pallets with his tie in his pocket and his sleeves up and he looked, for the first time in eight months, like a man of thirty-one rather than an institution. “You’re the site reliability lead. Your name on an incident report is your entire career, because the whole product you sell is that when you say a system is safe, it’s safe. Mine is relationships. I’ll lose Amsterdam. I’ll get them back in eighteen months at a different firm, or I won’t, and either way I’ll still be a man who’s good on the phone.”

“You didn’t ask me.”

“No.”

“You made a decision about my life in a room, and you did it in a way I can’t reverse, and you did it because you decided you knew better.” My voice had gone somewhere I didn’t recognise. “Do you know what that is, Rafa? That’s the same thing they say we did to them today.”

He went very quiet.

“Yes,” he said. “I’ve been sitting out here for an hour working out how to say that to myself.”

The air on that dock at half one in the morning in May in Singapore is a physical object. It has weight. It sits on you.

“I’ve been managing you for eight months,” he said. “That’s the truthful version. Every argument on the fourth floor, every time I came up when I didn’t need to. I told myself it was the job, because the job explained it, and the job explains almost everything if you let it.”

“I know.”

“You know?”

“I check the lifts at twenty to nine.” I heard myself say it, and heard how long I had been not saying it. “I have checked the lifts at twenty to nine since September. There is nothing wrong with the lifts. There has never been anything wrong with the lifts. In February you were in Amsterdam for six days and I checked them every morning anyway.”

He looked up at me then, and something in his face came apart, and it was the first thing I had seen all day that was not a system failing.

“Say the rest of it,” he said.

“You first.”

“No.” He stood. He was close enough that I could feel the heat coming off him in air that was already far too hot to hold anything else. “I have spent today being the reasonable man in nine separate rooms. I told Amsterdam a version of the truth. I told the founder a version of the truth. I stood up in front of ninety people and told a version of the truth that put my name where yours was. I have nothing left, Meilin. Not one word. So it has to be you.”

So it was me.

I said it. Out loud, on a loading dock, in that heat, badly, in about eleven words that I have never been able to reconstruct afterwards, and which he can, and does, and enjoys doing at dinner parties.

Then I put my hand flat against his chest, which I had thought about doing since a Tuesday in September, and found that his heart was going like something in a net.

That is what undid me. Not anything he said. The discovery that all of the composure, the whole coat of it, the voice that a hundred market makers in four countries trusted more than they trusted their own risk systems, had been sitting on top of that for eight months, and that it was going like that because of me, and had probably been going like that in every one of those conversations at the end of my row.

“Meilin,” he said, and it came out wrong, cracked in the middle.

“Don’t be reasonable,” I said. “Not for the rest of tonight.”

He wasn’t.

I remember that the fire door into the service corridor would not latch and neither of us stopped for it, and that it stood ajar behind us the whole time, and that this did not occur to me as a problem until three days later.

I remember his hands actually shaking on the buttons of my shirt, which gave me a pleasure so specific I could not have named it before that night, because I had watched those hands stay perfectly steady on a phone for eleven hours while the entire building came apart, and here they were, defeated by a button.

I remember taking hold of his wrist and doing it myself, and the noise he made.

I remember laughing against his mouth at something neither of us had said, and then not laughing at all.

I remember the wall of that corridor, the cold of the painted breeze block through the back of my shirt, and his mouth going down my throat, and one of my hands finding a fistful of his hair and the other flat against the wall, holding on to nothing, and the extractor fan running somewhere above us because the extractor fans in that building never stop.

And I remember the thought. Very clear, very calm, arriving in the middle of all of it in the voice I use for post-incident reviews:

There is no rollback for this.

Everything I do professionally, I do so that it can be undone. Fail safe. Roll back. Restore from snapshot. Blue-green deploy, so that if the new thing is wrong you can be back on the old thing in four seconds and nobody outside ever knows. Nineteen years of building things that can be taken back.

This could not be taken back. There was no snapshot of the woman I had been at six o’clock that evening. And I did not want one, and that was the most reckless thought I have ever had, and I had it in a service corridor at two in the morning with my hands in his hair and a fire door standing open behind me.

His flat was eleven minutes away.

We did not make it there quickly, and when we did, we stood in his hallway and he stopped, actually stopped, with his forehead against mine, and said, “Tell me this isn’t today. Tell me you’d have done this in April.”

“I’d have done this in November,” I said.

That’s all you get.

The incident report went out on the second of June. Four thousand words, eleven pages, and a section headed Decision Ownership with his name in it.

I read it in the ops room with the door shut and I did not cry, because I do not, but it took me some time to be able to see the screen.

It is a good report. That is the strange thing. I wrote most of the technical sections myself and they are accurate and clear and I am proud of them, and they sit inside a document whose only function, in the end, was to put a name in a box.

There were three weeks between the halt and the report, and we spent them behaving with enormous discretion in the office and none whatsoever anywhere else, and I have never in my life been so tired or so happy at the same time, and I was aware every single day that the two things had the same cause and that this was not sustainable.

By then he had already resigned. He didn’t tell me first. He told me after, at his kitchen table, in the flat with the boxes in it, in exactly the tone he used with clients, and I let him have that because he needed it.

“Amsterdam won’t take my calls,” he said. “Chicago will, out of pity, which is worse.”

“You could stay. Legal cleared you.”

“Legal cleared me of the thing I was accused of. Nobody clears you of the thing everyone believes.” He turned his cup round. “There’s a firm in London. Prime brokerage. It’s a step down and the money’s worse and nobody there was in Singapore in May.”

“London.”

“Yes.”

I did the arithmetic. Everybody in this industry does arithmetic; it’s the only prayer we have.

“Twelve hours’ difference,” I said. “Seven in summer.”

“Meilin,”

“I’m doing the sums out loud. Let me finish.” I put my hand over his. “My handover is six weeks. There is exactly one other person on this earth who can run that engine and she is in Tallinn and she has already said yes twice. After that I have four years of on-call owed to me and a reputation that is, thanks to a man who did something stupid in a room, entirely intact.”

He didn’t say anything.

“You made a decision about my life without asking,” I said. “I’m making one about yours. It’s not the same, because I’m telling you before I do it and you’re allowed to say no.”

He said something in Yoruba that I did not understand and have never asked him to translate.

We were both wrong about the man, in the end.

There was a man. There were several hundred thousand, but there was one in particular, and I know his name because it is in a coroner’s file in Michigan and I have read it, and I am not going to put it in this.

That is what a circuit breaker is. It is a mechanism that produces the better of two catastrophes. There is no version where it produces a good outcome, and anyone who tells you otherwise has never had their finger on one.

I still run systems. I am very good at it. I am the person you want in the room, because I have been in the room on the worst day, and I know what the number costs.

But I keep one thing on my desk now that isn’t a monitor. It’s a printout of the p99 graph from the nineteenth of May, the whole day, the climb and the flat forty minutes and the ugly restart.

Rafa asked me once why I’d have that in the house.

“Because it’s the only decision I’ve ever made that I’d make again,” I said, “and I want to be able to look at what it did.”

He understood immediately, which is the entire reason I married him.