Systems design
Designing systems that handle money: an invariant checklist
Software that moves money fails differently from other software. A normal bug shows someone a wrong screen, they file a ticket, you fix it that week. A bug in a rating engine issues a wrong invoice. By the time you find it, that invoice has been paid, disputed, or filed with a regulator. The code was wrong for an hour. The consequence lasts a quarter.
I have spent most of thirty years in systems with that failure mode: telecom billing, interconnect settlement, call-centre platforms and trade execution. Below is the list I settle before writing code in this category. None of it is clever. All of it looks obvious in review and gets expensive when somebody left it implicit.
1. Every amount has a currency and a scale, and both live in the type
Money is a pair: a number and the unit you count it in. Most of the bugs on this list begin when something splits that pair up.
Floating point is the famous version, and the easiest one to avoid. Store minor units as integers, or use a decimal type your language and your database already agree on. The version that actually reaches production is quieter: two components hold the same number at different scales. A rating engine works in thousandths of a cent, because per-second telephony rates need that resolution. Invoicing works in cents. A conversion happens somewhere between them, and if it happens in three places, sooner or later it happens three different ways.
So the scale belongs in the type, and conversion is a named operation that exists once.
A function taking a bare integer called amount is taking an unexploded shell.
One related decision is worth making early, because finance will ask about it. When you allocate a discount across several items, or split a shared charge, a remainder is left over. One cent, every time. It has to land somewhere deliberate, and you want the answer written down: this item absorbs it, for this reason. “Rounding” is not an answer.
2. Rating is a pure function of the event, the rules as of then, and the state as of then
An engine that reads the current tariff table cannot reproduce last quarter’s invoice. That is the entire argument.
Rating takes three inputs, each carrying a time: the event, the rules in force at the moment of the event, and the account state at that moment. Tariffs, discounts, zone maps and plan assignments all need validity intervals, and editing one creates a new version rather than overwriting a row.
Versioning costs more up front. What it bought me was survival through a change I could not have anticipated: partway through eleven years of running a fixed-line operator’s billing system, the regulator replaced the interconnect settlement model, and the new rules applied to periods already invoiced. Against a current-state tariff table, that is a rewrite. Against versioned rules, it is data entry and a re-run.
You can test any billing system for this in an afternoon. Pick an invoice from a year ago and reproduce it to the cent from stored inputs. A system that manages it can also explain itself to a customer. A system that cannot will eventually lose an argument to somebody holding a spreadsheet.
3. Recalculation is an ordinary operation
Errors surface late. Somebody in finance finds a discrepancy from four months back, and one question decides what happens next: how much work is it to fix properly?
When re-running a period needs a restore, a maintenance window and the one engineer who understands the pipeline, nobody re-runs anything. They adjust the numbers by hand. From that day the system stops being the source of truth, and every report built on top of it carries a correction that left no trace of itself.
So recalculation ships on day one, before anyone asks: re-rate a subscriber, an account group or a full month, then diff the result against what you invoiced. The diff earns its keep as much as the re-run does. Nobody should send a corrected invoice without seeing the comparison first.
4. Ingestion is idempotent and reconciled against the source
Two failures, and only one of them is visible. Losing an event costs revenue quietly. Counting one twice reaches the customer.
At PeterStar, a day of CDRs from one switch went through collection twice, and collection billed roughly 400 subscribers for the same calls twice. Nothing in the rating path noticed, because nothing in the rating path was looking. Batch reconciliation against the switch’s own counters caught it, which is the only reason anyone saw it before the invoices went out.
The repair that mattered was structural. Idempotency moved into the record itself, keyed on origin plus sequence, so that re-running a batch does nothing the second time. Reconciliation stopped being a report somebody reads on Monday and became a gate the pipeline cannot pass.
Every ingestion path deserves those two questions. What happens if this runs twice? And which independent authority confirms the count was right? For a switch, its own counters. For a payment provider, their settlement file. Without a second source you are asking a pipeline to audit itself, and it will always agree with itself.
5. Rounding has exactly one canonical order of operations
Discount before tax or after. Round per event or per invoice. Each choice is defensible on its own, and having two of them in one system is not.
I shipped exactly that. One path applied discounts on one side of VAT, another applied them on the other, and rounding happened per call in one place and per invoice in another. It surfaced as a total finance could measure and nobody could attribute to a particular bill. That is the worst shape a money bug takes, because there is no single wrong row to point at.
Decide the order once and the rounding point once. Encode both in one shared routine, then keep every report calling it instead of doing arithmetic of its own.
6. The ledger is append-only, and corrections are entries
An UPDATE on a financial row destroys the only evidence of what you told the customer
last month. Corrections belong in the ledger as entries of their own: a credit note or a
reversal, each carrying a reason code and an author.
It is also the cheapest support feature you will ever ship. Somebody asks why their bill changed, and the answer is a query.
7. Time is explicit, and there is more of it than you expect
A single event carries several timestamps that people habitually treat as one: when it happened, when you received it, which billing period it belongs to, and the timezone each of those is written in.
A call starting at 23:58 on the last day of the month and running eleven minutes belongs to which period? Both answers defend themselves. Only one can be in the code, and the rating engine and the invoicing run had better hold the same one.
Switch clocks drift. Provider files arrive in local time with no offset. Daylight saving moves an hour that some rates depend on. The defence is dull and it works: store event time with an offset, store ingestion time separately, derive the billing period by a documented rule, and never compare two of them without a conversion somebody wrote on purpose.
8. Account state is a small enumerable set with a log
Active. Suspended for non-payment. Suspended at the customer’s request. Closed. Closed with debt outstanding. Write the list down, make it a real enumeration in the schema, and log every transition with a timestamp and a cause.
Tidiness is not the reason. Support gets asked why a line was barred on the 14th, and either has an answer or has a theory. A theory costs you an argument every time and a customer occasionally.
9. Every partial failure has an outcome you chose in advance
Any operation crossing a system boundary can half-succeed. You send a charge to a payment provider and lose the response. You commit a rated event and the invoice write fails. You hand a settlement file to another operator and never learn whether it landed.
Only a few answers exist, and choosing between them is design work rather than error handling: retry behind an idempotency key, park the operation in a pending state until a reconciliation loop resolves it, reverse it, or stop and refuse to continue. The one thing you cannot do is leave the outcome to whatever the code happens to do, because what code happens to do under these conditions is charge twice, sometimes.
For each boundary, ask what the customer sees if the process dies exactly there, and what the next run does about it. When nobody on the team can answer in a sentence, the real answer is “it depends on timing”. Timing is not a specification.
10. The data model outlives the code
I rewrote the PeterStar code repeatedly across eleven years. The schema only ever grew: new columns and new tables, never a removal. That is the actual reason the system lasted.
The spine ran subscriber, account, service, plan assignment, rated event, with the plan assignment carrying its own validity interval instead of sitting as a field on the subscriber. That single decision is what makes “re-rate last March” a query rather than an excavation.
Get the model wrong and refactoring will not rescue you afterwards. You will write compensating logic around the mistake for as long as the system runs, and every new requirement will cost more than the one before it.
What the list is actually testing
Nine of these ten items ask the same question in different places. Can the system explain, months later, exactly how it reached a number, and can it reach that number again? Money software that can do this survives auditors, regulators, angry customers and its own bugs. Money software that cannot gets replaced, usually right after an incident nobody wants written down.
All of it is cheap while the design is still on a whiteboard. Each item costs ten to a hundred times more once the wrong choice has shaped a year of production data.
If you are building or replacing something in this category and want a second read before you commit to it, that is what an architecture review is for. One or two weeks, fixed price, and you get this list applied to your system instead of mine.