Download CV

Proof  ·  Offline-First Field Operations Platform

Six suites were green. The system was still wrong

Six suites passing in isolation did not mean the system worked. Executing every documented step, in order, surfaced seven defects. The most instructive were the ones no test had reached.

Everything on this page is checkable against the source, which is public.

The nine-area learning roadmap that used to live on this page is now on the stack page →

Check it yourself

The resolver, the tests and the load-test harness are in the repository. Clone it and run the 48 resolver tests; they finish in under a second.

What running it found

7 defects  ·  all fixed and re-verified

Six of the seven are below: the ones with something to teach. None of them were found by a test suite. They were found by executing the documented steps of the system against itself.

01

OrderingNo test reached it

The flagship scenario, the one the whole architecture exists for, returned the wrong answer.

Root cause

The ownership check compared the current assignee against the caller before conflict detection ran, so a driver reassigned away while offline was rejected outright and never reached the resolver that should have ruled in his favour.

02

Ordering · hidden behind the firstNo test reached it

Fixing that exposed a second defect immediately behind it. Nothing had ever got far enough down the pipeline to hit it.

Root cause

The transition-legality check validates against the row current status, which is what the office conflicting edit had just set. A second, independent check was silently reimposing administrative precedence one step after the resolver had ruled the opposite.

03

Silent successNo test reached it

A multi-line order whose second line failed left the first line stock decrement applied.

Root cause

The insufficient-stock branch used return rather than throw, and the transaction helper only rolls back on a thrown error. A plain return looked like success, so the transaction committed, permanently applying every earlier decrement while reporting the order as rejected. One keyword.

04

UnobservableNo test reached it

The signature metric was structurally unreachable.

Root cause

It was observed only inside a worker with its own private registry, no HTTP server, and no scrape target. Confirmed by pushing a real delivery through the pipeline and watching the number stay at zero on the endpoint being scraped.

05

Unobservable · since the first sessionNo test reached it

A consumer had been dying roughly thirty seconds after every stack start on a transient broker race, with no restart policy.

Root cause

Which means half the event pipeline had been offline in every session since the first one, with no crash loop and no failing healthcheck to indicate it.

06

InstrumentationThe worst one

After adding distributed tracing, the API reported a mutation as applied with an incremented version while the row in Postgres never changed.

Root cause

Verified three times against a side-by-side uninstrumented container on a second port sharing the same database. That is worse than a crash, because a crash tells you: this was the API confidently reporting that a driver delivery was recorded when it was not, caused by the tooling added to make problems visible.

How it shipped

Shipped with database instrumentation disabled and the incompatibility documented, rather than trading correctness for a nicer trace.

What the tests are for

Fast tests get written, slow tests get skipped

The resolver has zero imports. No database, no logger, no clock; everything arrives as an argument. That is not architectural purity, it is a testing strategy with a measurable consequence: forty-eight tests in under a second, on the logic that decides whether a driver day of work is honoured.

Had it reached for a database import, every one of those cases would need Postgres to run, they would take minutes, and I would have written ten instead of forty-eight. The bug would have been in the thirty-eight I did not write.

Tests

115

six layers, none skipped

Resolver tests

48 in 0.7s

zero imports, so they stay fast

Defects tests caught

1

the resolver ordering bug, in 0.7s

Defects running it caught

7

with six suites already green

The one the tests did catch

I put the merge rule first, and shipped it into a test suite that immediately went red. The driver writes status and a delivery timestamp, the office writes an assignee. Those field sets do not overlap. So the merge rule fired, saw no disagreement, and applied both, producing a job that was simultaneously delivered and reassigned to a second driver.

That is precisely the failure the entire architecture exists to prevent, and I wrote it wrong. Four unit tests caught it in seven tenths of a second. The fix is one condition: a physical claim skips the merge path unconditionally.

A merge is only safe when neither side is asserting a fact about the world. That paragraph is now a comment above the check, so the next person who tries to simplify the ordering hits the reasoning before they hit the code.

And what is still not fixed

The ceiling moved 38%, and it is still a ceiling

A 300-device reconnect storm broke it: sync_failed 44.66%, p95 pinned at the 30s client timeout. The cause was five connection pools inheriting a 20-connection default against a max_connections of 100, which never throws because it queues. Rebalanced, sync_failed fell to 20.15% and throughput rose 38%.

Still not resolved, honestly

p95 remains pinned under the full 300-device storm. With connections no longer scarce, Postgres itself is the next constraint: contended row locks, WAL flushes, and lock waits across a sequential per-mutation-transaction design, on a single container on one laptop. The next lever is horizontal API replicas behind a load balancer, which is a larger change than a pool tweak, and it is not done.