Trading systems
Fifteen ideas, zero orders: how to kill your own hypothesis cheaply
I built a six-layer crypto execution stack before establishing that there was anything to trade. Exchange drivers, an OMS, risk, strategies, telemetry, a front end. Months of work, and it was good work, aimed at a question I had not asked.
When I finally asked it, the answer took two weeks and cost nothing: the signal my strategy traded on was worth 0.5 to 1.5 basis points, and a round trip cost 11. Nothing survives a gap of sixteen times. I closed it and wrote up why.
Then I did the same thing to fourteen more ideas.
That is the actual number. Funding carry, trend following, candlestick patterns, token-unlock shorts, fixed-expiry basis, hourly imbalance, tick momentum, crash rebounds, crowding unwinds, listing events, relative funding between quote currencies, an options variance premium, and a few smaller probes. Every one closed on a threshold written down before the first run. Total orders placed with real money: zero.
People hear that number and assume the project failed. What it bought me was speed: fifteen answers in the time a single live experiment would have taken, and none of them paid for with a position.
The gate goes in writing, before the first run
A threshold you set after seeing results is not a threshold. It is a description of the results.
So the rule in my lab is that every direction begins with a written plan containing the number that kills it: minimum annualised return, maximum drawdown, minimum event count, minimum t-statistic, and the cost assumptions those are measured under. Then the code runs. The number does not move.
This sounds like bureaucracy until you watch yourself want to move it. My strongest experience of this was an order book signal that came back with a mean gross edge of +0.68 bp and a bootstrap lower bound above zero, which is what a discovery looks like. It had 48 events against a pre-declared minimum of 100. I had spent months on the platform underneath it. Every instinct said this one is real, collect more data, the gate was arbitrary anyway.
The gate was written two weeks earlier by a version of me who had no stake in the answer, and that is the entire reason it worked.
Compute the cost floor first, not last
Most retail strategy research has the arithmetic backwards. The signal gets studied for weeks and the fee schedule gets added at the end, as a haircut.
At a $10k account on a base fee tier, a round trip on one leg costs around 15 basis points taker or 21 maker, and a two-legged position costs upward of 30. That number is public, it is knowable on day one, and it is not improvable by better code.
Once it is on the wall, most ideas answer themselves. Every one of my closures hit the same wall from a different direction: gross signals of 1 to 5 bp at the fast end, 20 to 90 bp at the weekly end, against a floor that does not move. Which produced the filter I now apply before writing any analysis code at all. A new idea is worth building only if it promises more than 100 bp of gross effect per event, with enough events to measure, or a holding period of weeks with turnover under one. Anything shaped like a weak signal on a frequent rebalance has already been answered three times by independent routes, so the correct response is to point at the report rather than to run the backtest a fourth time and watch it land in the same place.
One out-of-sample boundary, opened once
The project has a single date splitting in-sample from out-of-sample, fixed across every hypothesis, and out-of-sample gets read once, after the configuration is chosen on the in-sample side.
Not once per idea. Once.
The reason is that the alternative is invisible. If you peek, adjust, and peek again, nothing in the output tells you it happened. The numbers look the same, the confidence intervals look the same, and the result is worthless in a way no statistic reports. The only defence is procedural.
Small samples produce beautiful lies
This is the one I would put on a poster.
A token-unlock short strategy passed its gate out of sample with a Sharpe ratio of 1.78 and a t-statistic of 3.38. Strong numbers. They rested on six events.
I went looking for more events, verified twelve additional unlock cliffs by hand, and re-ran the identical configuration on eighteen. Sharpe fell to 0.61. The t-statistic fell to 1.05. Nothing about the strategy changed. The only thing that changed was that the sample got large enough to stop flattering it.
The direction of that movement is the tell. Every verified event I added pushed the statistic toward zero, which is what noise does when you feed it more data. A result standing on fewer than twenty events, or on one market regime, is a sampling artifact until proven otherwise, and the proof is more events rather than a better argument.
Sanity-test the pipeline before you trust the pipeline
Before computing any statistic that matters, two checks run.
The first is a lookahead test: does the pipeline produce a result when the feature is shifted so it cannot possibly know the future. The second is a noise feature, a column of random numbers pushed through the whole apparatus. If random data scores well, the apparatus is broken, and finding that out after a week of interpreting results is expensive.
These caught two real bugs that would otherwise have silently changed
conclusions. One was a set of multiplier tickers, the 1000PEPE family, whose
contract multipliers poisoned basis calculations. The other was a data type bug
in the dataframe layer that quietly dropped the highest-volume days for the
most volatile instruments, which is to say precisely the observations that
carried the effect being measured.
Neither bug announced itself, and both would have produced numbers plausible enough to spend a week interpreting.
Stress the costs, then decide
A strategy that passes at exactly the modelled cost has not passed. Fee tiers change, spreads widen when you most want to trade, and the fill you modelled at the touch happens somewhere worse.
Every gate in the lab has to survive costs multiplied by two or three. When I apply that, the effect is not a marginal trim on the returns. Candidates that looked viable at modelled costs go negative, which tells you they were never strategies, they were rounding errors with a narrative attached.
Close the direction out loud
The last rule is the one that changed how much work I get done: a negative result is documented as thoroughly as a positive one, and the direction is closed explicitly, in a report, with its gate and its numbers.
Without that, a killed idea does not stay killed. It comes back in three months wearing a slightly different hat, and you re-run it, because all you remember is that it felt promising and something went wrong. The written closure means the next conversation starts with “what specifically is different this time”, which is usually a question with no answer.
Fifteen closures, each with a plan and a report. The reports are the asset. The code that produced them is reusable; the decisions are what stopped me spending another year.
This is not really about trading
I use the same shape in client work, where it is easier to sell and much less often applied.
Before building the thing, name the number that would prove it is not worth building, and name it in writing while nobody is invested. Compute the floor it has to clear before you optimise anything above that floor. Check your measurement apparatus against known-nothing input before you trust what it says about your real input. Decide how many observations you need before you look at the first one. And when something fails, write down that it failed, so the organisation stops paying for the same lesson.
The failure I keep meeting in this work is good engineering aimed at a question nobody checked was worth answering. The check is almost always cheaper than the build: mine cost two weeks against a platform that took months, and I would have paid ten times that to get the answer before I started rather than after.
Fifteen times now, the answer has been no. I have the reports, the pipeline and the money.