Case study · Telecom billing

Billing for a fixed-line operator: eleven years in production

A rating and charging system built when there was nothing on the market to buy, which then had to survive a decade of tariff changes, a new regulatory settlement model, and a company that grew around it.

Client

PeterStar, a fixed-line operator in St Petersburg

Domain

Telecom billing

My role

Architect and lead developer

Period

1993–2004

Result

11 years in production without a rating rewrite

Context

A fixed-line operator with around 100,000 subscribers, running AT&T 5ESS switches, later rebadged Lucent on the same hardware, alongside Huawei equipment. Billing covered the full cycle: CDR collection from the switches, mediation, rating, invoicing, and interconnect settlement with other operators. Volume ran to several million CDRs a day once transit and interconnect traffic is counted, which is most of it; subscriber calls alone are the smaller half of what a fixed-line operator has to rate.

The people using the system daily were the finance department and customer support, not engineers. That shaped more of the design than any technical requirement did.

In 1993 there was no realistic option to buy. The vendors selling billing to Western operators did not price for this market and did not handle the local regulatory model at all. The operator was running a legacy system delivered from the UK by GPT, and in 1994 we replaced it. I wrote the requirements, ran the migration as project manager, and ran development on the result for the nine years that followed. The system itself was built to that specification by Peter-Service, a St Petersburg vendor that had written the billing for Delta Telecom in 1992 and later became Nexign.

The problem

Billing is not, mostly, a computation problem. Rating a call is arithmetic. The problem is that the arithmetic is attached to money, and every part of the company gets to change the rules:

• Marketing changes the tariff plans, and expects the change to apply from a date, not from a deployment.
• The regulator changes how interconnect is settled, retroactively.
• Finance discovers an error from four months ago and needs the corrected invoices to be reproducible and explicable, not just correct.
• Support needs to answer “why is this line on my bill” for a call that happened in a tariff period that no longer exists.

Any of these is easy in isolation. The design problem is all of them at once, forever, on a system that cannot be taken down at month end.

Constraints

This is the section that decided the architecture.

You cannot lose a CDR, and you cannot count one twice. Both failures are revenue errors and only one of them is visible to the customer. Everything in the collection path had to be idempotent and every batch had to reconcile against the switch’s own counters, because the switch is the only authority on what actually happened.

History is immutable, but tariffs are not. A bill issued in March must still be reproducible in December, after the tariff it used has been edited three times by marketing. That rules out the obvious design where the rating engine reads the current tariff table. Tariffs had to be versioned with validity intervals, and rating had to resolve the tariff as of the call, not as of now.

Recalculation had to be a normal operation. Errors are found late. If the only way to fix a month is to restore a backup and re-run everything, you will not do it, and instead you will patch numbers by hand, which is how billing systems start lying. So recalculation of an arbitrary period had to be a supported, repeatable operation from day one, and rating had to be a pure function of (CDR, tariff version, subscriber state at that time).

Non-engineers had to change tariffs. If every new plan needs a developer, marketing routes around you and the system stops being the source of truth. The tariff model had to be data, expressive enough for what marketing actually invents, and constrained enough that a wrong entry could not silently produce a wrong invoice.

Month-end had to run without a maintenance window. The heaviest load coincides with the least tolerance for downtime.

What I did

I specified rating as a pure function over versioned inputs, and required every stage of the pipeline to be restartable and reconcilable. These were the decisions I owned, first in the requirements and then across nine years of development on top of them.

Mediation as a separate, dumb stage. Collection from the switches did no interpretation: normalise the record, stamp it with its origin and sequence, persist it, reconcile the batch. Anything clever happened downstream where it could be re-run. Switch-specific quirks stayed in per-switch adapters rather than leaking into the rating logic, and there were plenty of them: the 5ESS and the Huawei switches each interpreted their own record format generously, and in different directions.

A tariff model with validity intervals rather than a current state. Every tariff, zone, discount and plan assignment carried a validity period. Rating took a timestamp and resolved the rules that were in force at that moment. Editing a tariff created a new version; it never mutated the old one. This is more work up front, and it is the reason the system did not need a rewrite when the regulator replaced the interconnect settlement model outright, a change that had to be applied to periods already invoiced.

The data model as the long-lived asset. The spine was subscriber → account → service → plan assignment → rated event, with the plan assignment carrying its own validity interval rather than living as a field on the subscriber. That one decision is what makes “re-rate last March” a query instead of an archaeology project. We rewrote the code repeatedly over the eleven years; the schema changed only by addition. That is the actual reason the system lasted. The durable thing in a billing system is its data model, and if you get that wrong no amount of refactoring later will save you.

Recalculation as a first-class operation. Re-rate a subscriber, an account group or a period, with the result diffable against what was previously invoiced before anything was issued to a customer.

What went wrong

A batch loaded twice. One day’s CDRs from one switch went through collection twice, billing roughly 400 subscribers for the same calls twice. Nothing in the rating path caught it. The batch reconciliation against the switch counters did, and that is the only reason the error surfaced before the invoices went out instead of afterward, from customers. The fix was not “be more careful with the loader”: idempotency moved into the collection stage as a property of the record itself, keyed on origin plus sequence, so that re-running a batch became a no-op instead of a duplicate. Reconciliation stopped being a report someone reads and became a gate the pipeline cannot pass without.

Discount before or after tax, and where rounding happens. One part of the code applied discounts on one side of VAT; another applied them on the other side. One path rounded per call, another rounded per invoice. Neither choice alone was wrong. Combined, they surfaced as a discrepancy finance could measure in aggregate, without anyone able to point at a single wrong bill: the worst shape a money bug can have. The fix was to define one canonical order of operations and one rounding point, then encode both in a single shared routine that every report had to use instead of doing its own arithmetic.

The tariff editor started too flexible. The first version let marketing express things the rating engine could interpret in more than one way. Constraining the model, with fewer degrees of freedom and validation at entry, removed a category of error that had been arriving as support tickets.

Result

• Eleven years in production, 1993 to 2004, through hundreds of tariff plans and a regulator-mandated replacement of the interconnect settlement model.
• Rating engine survived without a rewrite; the schema evolved by addition.
• Several million CDRs a day at peak, including transit and interconnect.
• Month-end close landed by the 5th, every month, without a maintenance window.
• Disputed invoices stayed in the fractions of a percent.

The number I would actually defend in a conversation is the first one. Eleven years is a claim that the invariants were right from the start, because there was no second chance to fix them.

Stack: C++ · Oracle · PL/SQL · Unix · CDR mediation · OSS/BSS integration