Epoch

PhonePe runs its containers on Drove, a distributed container orchestrator built in-house. Epoch is the piece that schedules work onto it: a time-based container job scheduler. If Drove is the thing that runs containers, Epoch is the cron that decides when.

I owned Epoch for online merchant services, the systems behind PhonePe's merchant-facing payments.

The shape of it is worth understanding, because it explains what the job actually is. A task is a unit of work packaged as a Docker image. A topology is that task plus a cron expression saying when to run it. Epoch holds the topologies and fires them; Drove runs them. One Epoch node is elected leader through ZooKeeper and does all the state management and scheduling, and the other nodes simply route requests to it, which is fine, because the expensive part (actually running a container) was never Epoch's job.

Standing it up

The org's scheduled work (reconciliation, reporting, cluster health, the operational scripts around our accounting and settlement service) ran on Apache Mesos, left over from an older era of orchestration. My job was to move all of it onto Epoch.

First it had to exist. I stood up my team's Epoch instance from nothing: onboarded onto the platform, built the Epoch infrastructure components, and got the Drove and auth permissions (along with every approval those required) before a single job could run. Then a dedicated staging environment, because you do not validate money-adjacent scheduled jobs by running them in production, and finally the production instance.

Then the migration: 20+ scheduled jobs, each owned by a different person. The infrastructure took about a week. The migration took three to four weeks, and the technical port was the small part of it; most of the time went into reaching every job's owner and moving their job across safely. That is the real shape of a migration inside a large organisation, and it is the part that gets under-planned.

Rewriting the framework instead of lifting and shifting

The scripts were written and operated largely by production-support people, not by specialists. The framework had grown fragile around that: hard-coded arguments, and no clean way to test a change before it hit production.

A straight port would have carried all of that forward. So I rewrote the scripting framework instead: dynamic refresh and parameterised arguments, so an operator changes behaviour by passing an argument rather than editing and redeploying code. I stood up a dedicated staging environment, because you do not validate money-adjacent scheduled jobs by running them in production. And I wired failures to raise a support ticket automatically, so they stopped being silent.

Operational overhead on those scripts dropped by roughly 80%, mostly because people could finally test end to end before shipping.

The documentation outgrew my team

I wrote the user guide and the documentation for it: how to run it, how to check cluster health, how to keep the repository sane. That was meant for the people on my team.

It got circulated widely across the org, and became the thing people used to onboard onto Epoch and to debug it. Other teams stood up their own Epoch instances off it, and I helped them do it. The setup I'd fought through once (the components, the permissions, the approvals) was written down well enough that nobody else had to fight through it from scratch.

The logs weren't being backed up

Mid-migration I went looking for why a failed job had left nothing behind to diagnose, and found that job logs weren't being persisted at all. Failures were effectively invisible after the fact.

That was an infrastructure problem, not my problem. But I was the one going around telling the org that failures were now visible, and that promise is worthless if the logs evaporate. I pulled in the architects and SRE, we root-caused it together, and it got fixed.

Which is also why I can't give you a clean incident count for the migration, and I think the absence is the interesting part: we had no way of knowing. Detection was reactive: someone raised an incident, and then we looked. That invisibility was the thing the whole exercise was trying to kill.

The unpopular quality gate

Anyone could merge into the scripts repository. For a body of money-adjacent scheduled jobs, that is not speed; it is speed until the first silent bad merge.

I protected the branch and restricted merge rights to a small set of owners genuinely accountable for the code. There was real backlash (it slows us down), and I didn't back off, but I also didn't just impose it and leave. I ran sessions walking people through why an unreviewed repo of jobs that touch money is a liability. The team came around, and it stuck.

I also wrote the reconciliation and reporting jobs: the ones that walk pending transactions, work out what actually settled, and close them out. In payments, a transaction stuck in limbo isn't a cosmetic problem. It's somebody's money.

Why the CLI existed at all

There was no programmatic way to drive Epoch. Everything was manual, and a manual cutover is a disaster-recovery risk, because it means human error and long recovery times at precisely the moment you can afford neither.

I didn't just go and build a tool. I took a one-page problem statement and a few candidate approaches to the chief architect, ran a design review on the trade-offs, and then built epoch-cli, from the first commit, in Python. We open-sourced it.

What went in:

• a topology plugin, and filtering on it, so you can see what's actually scheduled where;
• cluster management commands, including pause-all: the command you want to exist at 3am;
• skip and override, for the times a job should not run, or should run differently, right now;
• logs for runs, so a failed job can be diagnosed without going to find the container it died in;
• Poetry packaging and a developer setup, so the next person could actually contribute;
• naming made consistent with drove-cli, because two tools for one system should not have two vocabularies.

# what's scheduled, and where
$ epoch topology list
$ epoch topology get recon-nightly

# the command you want to exist at 3am
$ epoch cluster pause-all

# and the one that saved a datacentre cutover
$ epoch topology export --all > topologies.json
$ epoch topology create --file topologies.json --cluster secondary

I also stripped the PhonePe-specific constructs out of it. That's the work that turns an internal tool into something that can stand on its own, and it's why the CLI is public today.

It outgrew what I built it for. Moving a service from one data centre to another means moving its Epoch topologies too, and teams now do that cutover through epoch-cli. A tool I proposed to close a DR gap became the standard way people do the thing I was worried about.

The day it paid off

Later, a large cross-datacentre migration had to happen, not as a drill, and not on a comfortable timeline. Services had to come up in the secondary data centre, and every one of them that ran scheduled work needed its topologies to come with it.

epoch-cli already had import and export of topologies. Nobody had to write anything, and nobody had to move schedules by hand under time pressure. The capability was just there, because it had been built months earlier for a problem that hadn't happened yet.

I'd made the case that the manual-only path was a disaster-recovery risk months before anything went wrong. And then something went wrong, and the tool was already there.