Venmo · 2016 – 2019
Rebuilding ACH so it stopped waking people up
Led the rebuild of the ACH pipeline that moves money in and out — halving on-call load and closing a roughly $300k annual leak.
Sanitized- ~50%
- less downtime and on-call
- ~$300k
- annual losses prevented
The situation
ACH is how money actually enters and leaves the system. It is also a batch protocol with next-day settlement, bank-by-bank quirks, and failure modes that surface long after the moment a user pressed a button.
The existing pipeline treated those failures as exceptional. They were not exceptional — they were routine, and the cost of handling them by hand showed up as on-call pages and as money quietly written off.
The constraints
- Money movement cannot be paused for a rewrite, and it cannot be tested in production by trying things.
- Settlement is asynchronous and slow. A return can arrive days after the transaction it refers to, so correctness is a property of the whole timeline, not of a request.
- Unclaimed funds are regulated. Escheatment is not a feature you can decide to defer.
The call
Model returns and failures as ordinary expected states in an explicit lifecycle, rather than as exceptions to a happy path. Most of the reliability win came from this single reframing — once a late return was a normal transition, handling it was code rather than a page.
Make the pipeline distributed and idempotent so that reprocessing was always safe. A retry that cannot be trusted is not a retry; it is a second incident.
Build compliant escheatment workflows into the same lifecycle instead of bolting on a periodic sweep, so unclaimed funds were tracked continuously rather than discovered.
What happened
On-call load and downtime for the payments platform fell by roughly half.
The failure modes that had been silently writing off money were caught and handled, closing an annual leak of around $300k.
Platform reliability improved further through caching, database tuning, and distributed processing — but the lifecycle model is what made the rest tractable.
What I'd do differently
The lifecycle should have been documented as a state machine on a wall from day one. It existed correctly in the code well before it existed clearly in anyone's head, and every conversation with risk and ops paid the cost of that gap.