The rollback at 02:17

At 02:17, Bảo received the only notification that night without an exclamation mark.

I don’t know why yet.

It came from Mây, the agent Hạt Mưa’s team had recently assigned to a trial on-call shift. Mây could read logs and prepare a fix. It could not put that fix into the system serving customers.

Bảo looked at the message, then at seventeen green tests.

“Then why did you send a patch?”

Because the patch passes the tests we have. But the customer’s problem does not appear in those tests.

In the next window, a florist reported three missing orders. Not entirely missing: they still appeared in the end-of-day report. They had vanished from the screen used to prepare deliveries.

Kim joined the call with her camera off. Behind her voice, Bảo could hear water being poured into a cup.

“Let’s not start by finding who got it wrong,” she said. “Let’s find where the orders went.”

Bảo shared his screen. Mây opened a diagram with six boxes. Five were connected by solid lines. The last box, the seller sees the order, had a dotted line.

“What does the dotted line mean?” Kim asked.

Something I have not verified.

There was a test for creating an order. A test for calculating payment. A test for sending a notification. None opened the screen the florist actually used, with that shop’s time zone.

Bảo pulled his chair closer. Mây’s patch only changed the way the data was sorted. On the sample data, it looked so neat that he had almost approved it before the notification arrived.

“What if we roll back?”

Mây took longer than usual to answer.

New orders may need reconciliation. I have not checked enough to say that a rollback is safe.

Bảo wanted certainty. Kim wanted a list of what they did not know. That night, they could not have both.

They paused the job that was changing the data. Kim called the person responsible for the shop to explain the scope of the problem. Bảo compared the newly created orders with the records they had retained. Mây prepared a new test, one that went all the way to the seller’s screen.

At 03:06, the dotted line became solid.

Nobody reached for a celebration button. They still had to identify which orders needed recovery and check the result with the shop.

The next morning, Bảo opened the incident report. He typed its first line: I trusted the green.

Kim read it, deleted I, then stopped.

“Not to remove responsibility,” she said. “To describe precisely what in our way of working made it possible for someone to trust the wrong thing.”

Mây suggested a heading: The test ended before the user’s journey did.

Bảo kept it.

At the bottom of the document, the agent asked whether it should delete the failed experimental patches.

“No,” he replied. “Label them clearly. Next time, we should know which paths we have already taken.”

Background reading

The characters and events are fictional. These readings informed the technical setting:

1. Anthropic — Building effective agents, 19 December 2024. Background on agents, tools, feedback and stopping conditions. Accessed 15 October 2026.
2. Google SRE — Postmortem culture. Background on learning from incidents to improve systems. Accessed 15 October 2026.