115
Tests
six layers, none skipped
PRJ-001 · platform · 2026 · source public
Delivery drivers work for hours with no connectivity, then sync. The hard problem is not the sync, it is what happens when the field and the office disagree.
Source public
The repository holds the production conflict resolver, all 115 tests, and the write-ups this page draws on: CONFLICT_RESOLUTION.md, TEST_RESULTS.md and AWS.md.
115
Tests
six layers, none skipped
7
Defects found by running it
all fixed and re-verified
48 / 0.7s
Resolver tests
zero imports, so they stay fast
3
Protocols
REST, gRPC, GraphQL, each where it belongs
The problem
A driver loses signal at 13:40. At 14:02 he hands a parcel to a customer and taps delivered on a phone with no connectivity. At 14:30 the office, seeing no update for an hour and reasonably assuming he is stuck, reassigns that job to another driver. At 17:00 he reaches signal and the delivery syncs, three hours late, against a row that changed underneath him.
Last write wins
Hands it to the dispatcher, and the system then claims a parcel sitting in a customer kitchen is still out for delivery, so it dispatches a second van to collect it.
Server-authoritative
The same outcome wearing a nicer hat: the server state is not more true than the field, only more recent, and recency is not truth.
Trust device clocks
Builds consistency on a number any rooted phone can change by hand.
Ask a human
Produces forty conflicts on a bad day, and by the fifth nobody reads them, so the two that genuinely needed a person get buried in the thirty-eight that did not.
The approach
A delivery happened, and it is irreversible. A reassignment is an intention about what should happen. When a plan collides with an event, the event wins and the plan is told it was overtaken by reality. That is encoded as a rank: physical states outrank administrative ones, which outrank in-progress ones. A cancellation therefore cannot un-deliver a parcel.
Most apparent conflicts are not conflicts at all, so field ownership is split explicitly and disjoint edits merge silently, which is what keeps the conflict inbox readable. Equal rank with overlapping fields escalates to a human with both versions attached, and an escalation writes nothing, because a system that escalates and also acts has escalated nothing.
Architecture
One Postgres transaction per mutation covering six writes across five tables: lock the row, optimistic version check, guarded stock decrement, cash entry, ledger, outbox. Events are written to an outbox table inside that same transaction and relayed to Kafka by separate workers, so a broker outage becomes a delay rather than a data-loss event.
Sockets are a view, never a write path: if every socket dropped simultaneously, not one delivery would be lost.
Each with what it cost
A decision without a stated tradeoff is a preference. These are the seven that shaped the system, and the price each one carries.
01
Why
Conflict resolution is a business decision, not a technical one. The technology can detect the conflict; only the business can say who is right.
What it costs
The rule is domain-specific. It is correct for parcels and shop visits and would be wrong for a trading desk where a cancellation carries legal weight. Pretending it is universal would be the actual error. A rising escalation rate is the system being honest that reality contains a case the rules do not cover; the failure mode to fear is an escalation rate of zero, achieved by guessing.
02
Why
You cannot hold a database lock across nine hours of a driver being offline. The update carries the version the device last saw, and a zero-row result is the entire detection mechanism. One integer comparison, working identically whether the device was gone four seconds or four days.
What it costs
Conflicts are detected at sync time rather than prevented at write time, which is what makes arbitration necessary and is precisely the point. A stale write is not wrong, it is uninformed.
03
Why
A field phone clock drifts, gets set by hand, and jumps on a timezone change. Within a device its local counter is the truth about order, and it beats the timestamp even when the timestamp disagrees.
What it costs
Cross-device ordering falls back to the client timestamp with a deterministic tiebreak, because replay has to be reproducible. Applied in arrival order instead, a shuffled batch would attempt delivered before picked up, be rejected as an illegal transition, and lose a real delivery to a networking artefact.
04
Why
A database write and a broker publish cannot be made atomic. Writing the event to a table in the same transaction makes them inseparable, and a separate worker relays it afterwards. If the API published inline, a Kafka outage would stop drivers recording deliveries, taking down the most critical path in the system with a dependency it does not need in order to be correct.
What it costs
At-least-once delivery, so consumers must be idempotent, deduplicated on event id and consumer group so fan-out is preserved. Deduplicating on event id alone would silently stop the second consumer group ever seeing an event the first consumed. Exactly-once delivery is not a real thing; at-least-once plus idempotent consumers gives exactly-once effects.
05
Why
A 200-mutation batch from a device dark all afternoon contains some that succeed and some that conflict. One transaction per batch means a single stale item rolls back a whole day of legitimate field work.
What it costs
Partial success has to be a first-class response shape, with a per-mutation verdict rather than one status for the request. The device is told item by item exactly what landed.
06
Why
A driver reassigned away while legitimately offline is still authorized for what he did before the reassignment. Checking ownership against current state rejects the exact case the system exists to handle.
What it costs
Requires recording the previous assignee atomically with the reassignment rather than inferring it. The naive fix of skipping the ownership check during a conflict would let anyone claim any job by manufacturing one.
07
Why
A system that silently overrides a human decision is a system that human will fight, work around, and eventually disable. An automated decision nobody understands is worse than no automated decision, because now there is both a wrong answer and nobody watching for it.
What it costs
Every resolution path has to produce prose, which is more work than returning an enum. The reason string is the product; the resolution is just the mechanism.
Load testing
A reconnect storm: 300 devices returning from long offline periods simultaneously, 15% retrying the same batch to exercise idempotency under concurrency.
First the harness was wrong, and that mattered
It seeded one device session and shared it across all 300 virtual users. The rate limiter is keyed by device id, so 98% of traffic was throttled. That measured the rate limiter, not the sync path.
Then it broke properly
sync_failed
44.66%
sync_batch_latency
p95 30s
pinned at the client timeout
http_req_failed
20.76%
iterations
~2,000
of ~3,600
Diagnosis
6,616 slow-query warnings on a normally sub-5ms dedupe lookup, and zero Postgres errors. That absence is the clue: every service inherited a 20-connection pool default and none set an override, so five pools against Postgres's max_connections of 100 is exactly 100, with zero headroom.
And the driver's connection timeout bounds the initial TCP connect, not time spent waiting for a client in the pool's internal queue, so exhaustion never throws. It queues silently, latency climbs with nothing logged, clients time out, and their retries add more work on top.
Fix · pools rebalanced to actual demand, worst case 80 with 20 spare
sync_failed
44.66% → 20.15%
http_req_failed
20.76% → 10.53%
iterations
~2,000 → ~2,765 +38% throughput
Still not fully resolved, honestly
p95 remains pinned under the full 300-device storm. With connections no longer scarce, Postgres itself is the next constraint: contended row locks, WAL flushes, and lock waits across a sequential per-mutation-transaction design, on a single container on one laptop. The next lever is horizontal API replicas behind a load balancer, which is a larger change than a pool tweak, and it is not done.
Outcome
Built end to end and run end to end, which is the part that mattered. One of the seven was an API reporting a write as applied while the row never changed. All 115 tests pass across six layers with none skipped.
Load tested to failure, diagnosed, fixed, and re-measured, with the numbers published rather than summarised. The source is public, so the internals are not merely claimed.