The instinct we bring from other domains
- Most engineers learn reliability where continuing is the safe choice. A browser skips the element it cannot render. A batch job logs the bad row and finishes the run. A web service falls back to a default and serves the page.
- In those systems the worst case of "keep going" is a degraded experience. Someone notices later, and nothing is lost that cannot be recomputed.
- Trading systems look similar from the inside — services, queues, retries — so the same reflex gets applied. It does not transfer.
There are only two kinds of surprise
- Expected. You thought about it before the market opened, decided what it means, and wrote a handler. A rejected order, a rate limit, a gap in the book at the open.
- Unexpected. Everything else. That is not a condition, it is a bug in your understanding of the system.
- Most codebases contain a third, unwritten category: unexpected, but let us guess and continue. That category is where money disappears.
The asymmetry that settles the argument
- Not trading costs opportunity. It is bounded, measurable, and boring.
- Trading on a state you do not understand costs capital, and it keeps costing until a human notices.
- A broken mechanism is worth less than no mechanism. You would not keep driving a car with an unknown noise in the steering because stopping is inconvenient.
Implicit behaviour is the actual defect
- The catch-all that swallows everything and returns to the main loop.
- Retries wrapped around an operation that is not idempotent, so one intent becomes three positions.
- A missing price that becomes
0. A NaN that becomes 0. A missing config key that becomes a default nobody chose. - A partial fill treated as a full fill because the code path for partials was never finished.
# The shape of the problem, not real code.
try:
price = feed.last_price(symbol)
except Exception:
price = 0.0 # now every risk check downstream is a lie
size = notional / price # and this is where it stops being funny
- Each of these is a small, reasonable-looking decision made by a developer who was not in the room when the position blew up.
What "stop and yell" means in practice
- Stop the strategy. Do not stop the supervisor, the monitoring, or the ability to reach the venue.
- Follow a shutdown policy decided in advance: cancel resting orders, flatten or hold, and know which one you chose. The exception handler is the worst place to make that decision.
- Alert loudly and to a human. Not a log line in a file nobody reads. Loud enough to be annoying, because the alternative is more annoying.
- Freeze the evidence: inputs, state, and the last few decisions. A halt you cannot explain afterwards will happen again.
Fail fast is not "no error handling"
- It is the opposite. It forces you to enumerate the cases you accept and to handle them explicitly, because everything you leave out becomes a halt.
- Over time the set of expected cases grows and halts become rare. That growth is the real work, and it only happens if the system stops making silent guesses on your behalf.
- Assertions and invariants are cheap. Check position against the venue, check that fills reconcile, check that the clock moves forward. Fail on the first thing that disagrees.
A short checklist
- Every
except names a specific error and a specific decision. - No default values for market data, positions, balances, or configuration.
- Reconciliation runs continuously, and a mismatch is a halt, not a warning.
- The halt path is tested as often as the happy path.
- One human is paged, and that human can actually act.
Closing
- An honest broken machine tells you it is broken. A polite broken machine keeps trading.
- We would rather be woken up at three in the morning by our own software than be told about it later by our accounting.
