What actually happens in a DR drill

In regulated payments you don't get to claim you can fail over. You have to demonstrate it. India's business-continuity expectations for payment systems mean a disaster-recovery drill is a recurring, audited exercise, not a document someone signs once and files.

And at this scale a small mistake is not a small mistake. PhonePe carries something like half of all UPI transactions in India. When it stops, people cannot buy groceries and merchants cannot take money; it is not an engineering inconvenience, it is on the news within the hour, and it has been: in May 2025, disaster-recovery drills themselves took UPI payments down for over an hour when peak traffic met a capacity ceiling nobody had found yet. The blast radius of anything you touch here is other people's lives.

I have taken part in several DR drills at PhonePe, and led them end to end. Most engineers never will. You can spend a career shipping features and never once be in the room when someone deliberately turns off a production data centre.

So this is an explainer, not a war story: what a drill actually is, what the two numbers mean, what fails over and how, what you do on the night, and a mistake that makes the whole exercise worthless. PhonePe's own engineering blog gives you a sense of what this looks like at their scale.

I owned an application, not the plumbing. To lead a drill credibly I had to learn the plumbing anyway. This is what I learnt, roughly in the order you need it.

Active-passive, RTO and RPO

Start with the shape. We ran active-passive: two data centres, but only one of them takes writes at a time. The other is a warm standby, kept up to date by replication, doing nothing until the day it has to do everything.

The alternative (active-active, both sites taking writes) sounds better and is much harder. The moment two data centres can both accept a write for the same account, you own a conflict resolution problem, and in payments the conflicts are somebody's money. Active-passive trades that away and buys you a much simpler failure story: everything is over there now.

The trade-off between the two is a well-worn one, and worth reading up on properly if this is new to you: this write-up on multi-region active-active vs active-passive lays out the shape of the decision. The short version: active-active buys you availability and costs you correctness guarantees. Which of those you can afford to lose is not an engineering preference. In payments, it is decided for you.

Two numbers describe how good your failover is, and they measure completely different things.

• RTO: recovery time objective. How long you are down. From the moment the primary is lost to the moment traffic is being served from the secondary. This is a time, and it is mostly a function of how much of your cutover is still manual.
• RPO: recovery point objective. How much data you lose. If the primary vanishes right now, how far behind is the standby? This is not a time you get to choose. It is handed to you by your slowest-replicating datastore, and no amount of planning changes it.

That distinction is the one people get wrong. RTO is an engineering problem: automate the steps and it comes down. RPO is a property of your storage. You can write any number you like in the document; what will actually be true on the day is your database's replication lag.

What we ran, and how it replicates

Every component in the stack fails over differently, and your RPO is the worst of them.

• MariaDB: the transactional store, and the one that matters. Active-passive with asynchronous replication, lagging the primary by a couple of minutes. This is the binding constraint. MariaDB's replication lag was our RPO; nothing else in the stack moved that number.
• Aerospike: run in AP mode with bidirectional cross-datacentre replication in the low hundreds of milliseconds. Effectively real time. Its contribution to data loss is negligible.
• RabbitMQ: asynchronous processing only. The durable state lives in the database, so messages in flight are reconstructable from persisted state. RMQ is not a data-loss surface on its own, which is a nice property to have designed in before you need it.
• ZooKeeper: coordination and leader election. Also what Ranger, the open-source service-discovery library, stores its service registry in. Router registry and Zeus: internal constructs on top of Ranger. Zeus is a layer-7 traffic-shaping service.
• Drove and drove-gateway: container orchestration, and the NGINX layer that routes traffic to wherever the containers actually are. Both are open source.

So the honest RPO was about two minutes, set entirely by MariaDB. Aerospike and RabbitMQ don't get a vote. If you want a better RPO you do not buy a better runbook; you change how the database replicates.

The steps of a drill

What actually happens, in order:

1. Stop the writes. Nothing new enters the primary. Until writes stop, the standby is chasing a moving target and you can never say it has caught up.
2. Verify there are no writes at the database. Not "we told the services to stop", look at the database and confirm nothing is landing. This step exists because the previous one is a request, and this one is evidence.
3. Mark the database read-only. Now it is enforced rather than agreed. A stray write from something you forgot about can no longer create a divergence you will discover a day later, in a reconciliation.
4. Bring up the other instance, in read-only mode. The standby comes up, but it cannot take writes yet, because two databases that both accept writes is the failure you are trying to avoid, not the one you are recovering from.
5. Redirect the API gateway to the secondary. Aim traffic at the standby now, while it is still read-only, rather than waiting until it can take writes. The edge is still closed, so nothing flows yet, but the gateway is pointed at the secondary before you open the door, not after.
6. Disable the API killer, if one is engaged. Traffic flows again, and it flows to the secondary. Only read traffic will work here: status checks, balance lookups, anything that doesn't write. The writes still fail, but they have been failing since step one, when the primary stopped taking them; a read-only secondary is no worse for them. What it buys you is reads back on a live endpoint early, and the redirect already off the critical path, so the one thing still gated on the final step is writes.
7. Give the database user read-write. This is the moment of failover: the moment writes come back. Traffic already points at the secondary, so the instant it can take them it is fully serving: the reads it was already handling, and now the writes that were failing a second ago. The old failure mode is a writable database nobody is talking to; here everybody is already talking to it, and you watch real requests land: served, verified, not just "the containers are up".

The order is the whole thing. Steps two and three look like bureaucracy and they are the reason you can state an RPO honestly instead of hoping for one: you stopped, you checked you had stopped, and then you made it impossible to start again.

And around the steps

• A screenshot after every step, timestamp visible: Grafana, the logs, a Swagger call. The drill is audited, and a regulator's expectation is not met by an engineer's memory of a good night.
• Then you fail back. The same steps in reverse, and the traffic with them. No screenshots this time: the evidence anyone wants is of the failover, not of the way home.
• Then you hand the screenshots to the external auditors, who verify the whole thing from their side. A drill is over not when the service comes up, but when somebody outside the company agrees that it did.

What a drill actually looks like

A drill is not a diagram; it is a room, and it is the middle of the night, because that is the only window in which you are allowed to take a payment system down on purpose.

On the call: engineers, an architect, SREs and their manager, compliance, and external auditors. Roughly a dozen people, most of whom do not report to you, all watching a clock. That is the part the runbook does not prepare you for: the failover is a technical problem, and everything around it is not.

We completed the drill on the correct path, with a measured recovery time. RTO came down from about an hour to under seven minutes, and it was still heading downwards: a lot of the cutover was manual and we were automating it.

And before any of it, somebody has to go round and ask everyone whether they are actually turning up. Nishant Jain made this, and it is the most accurate artefact of the whole exercise:

What I'd tell you it taught me

• A drill takes one service down. A real disaster takes the whole data centre, including, quite possibly, the very thing you were relying on to redirect traffic away from the failure. If your recovery plan is only exercised against a failure smaller than the one you're actually afraid of, you don't have a recovery plan. You have a rehearsal.
• You come away appreciating the architecture. You cannot lead a drill from inside your own service. You have to understand how data crosses data centres, how the gateway decides where a request goes, how a caller finds a service that has moved, and what each datastore does when the ground shifts under it. Most engineers spend their careers above that line and never once look under it. A drill drags you under it, at 4am, with auditors watching.
• The thing I appreciate most is not something I built. It is the foresight of the architects at PhonePe, who started taking disaster recovery seriously back in the early 2020s, years before there was any pressure to. Recent geopolitical conflict has made the whole industry realise how much this matters. By the time everyone else was realising it, PhonePe was already prepared.

That is what good architecture looks like from the inside: somebody, years ago, quietly doing the boring, expensive, unrewarded work for a day that hadn't come yet.