Download CV
← all 25 systems

PRJ-001 · platform · 2026 · source public

Offline-First Field Operations Platform

Delivery drivers work for hours with no connectivity, then sync. The hard problem is not the sync, it is what happens when the field and the office disagree.

Source public

Read the resolver, the tests and the load-test harness yourself

The repository holds the production conflict resolver, all 115 tests, and the write-ups this page draws on: CONFLICT_RESOLUTION.md, TEST_RESULTS.md and AWS.md.

115

Tests

six layers, none skipped

7

Defects found by running it

all fixed and re-verified

48 / 0.7s

Resolver tests

zero imports, so they stay fast

3

Protocols

REST, gRPC, GraphQL, each where it belongs

The problem

A driver loses signal at 13:40. At 14:02 he hands a parcel to a customer and taps delivered on a phone with no connectivity. At 14:30 the office, seeing no update for an hour and reasonably assuming he is stuck, reassigns that job to another driver. At 17:00 he reaches signal and the delivery syncs, three hours late, against a row that changed underneath him.

Every naive answer is wrong

Last write wins

Hands it to the dispatcher, and the system then claims a parcel sitting in a customer kitchen is still out for delivery, so it dispatches a second van to collect it.

Server-authoritative

The same outcome wearing a nicer hat: the server state is not more true than the field, only more recent, and recency is not truth.

Trust device clocks

Builds consistency on a number any rooted phone can change by hand.

Ask a human

Produces forty conflicts on a bad day, and by the fifth nobody reads them, so the two that genuinely needed a person get buried in the thirty-eight that did not.

The approach

Physical truth outranks administrative intent

A delivery happened, and it is irreversible. A reassignment is an intention about what should happen. When a plan collides with an event, the event wins and the plan is told it was overtaken by reality. That is encoded as a rank: physical states outrank administrative ones, which outrank in-progress ones. A cancellation therefore cannot un-deliver a parcel.

Most apparent conflicts are not conflicts at all, so field ownership is split explicitly and disjoint edits merge silently, which is what keeps the conflict inbox readable. Equal rank with overlapping fields escalates to a human with both versions attached, and an escalation writes nothing, because a system that escalates and also acts has escalated nothing.

Architecture

One Postgres transaction per mutation covering six writes across five tables: lock the row, optimistic version check, guarded stock decrement, cash entry, ledger, outbox. Events are written to an outbox table inside that same transaction and relayed to Kafka by separate workers, so a broker outage becomes a delay rather than a data-loss event.

Sockets are a view, never a write path: if every socket dropped simultaneously, not one delivery would be lost.

Node.jsExpressPostgreSQLKafkaRedis

Seven decisions

Each with what it cost

A decision without a stated tradeoff is a preference. These are the seven that shaped the system, and the price each one carries.

01

Physical truth outranks administrative intent, and ties escalate

Why

Conflict resolution is a business decision, not a technical one. The technology can detect the conflict; only the business can say who is right.

What it costs

The rule is domain-specific. It is correct for parcels and shop visits and would be wrong for a trading desk where a cancellation carries legal weight. Pretending it is universal would be the actual error. A rising escalation rate is the system being honest that reality contains a case the rules do not cover; the failure mode to fear is an escalation rate of zero, achieved by guessing.

02

Optimistic concurrency on a row version, not locks

Why

You cannot hold a database lock across nine hours of a driver being offline. The update carries the version the device last saw, and a zero-row result is the entire detection mechanism. One integer comparison, working identically whether the device was gone four seconds or four days.

What it costs

Conflicts are detected at sync time rather than prevented at write time, which is what makes arbitration necessary and is precisely the point. A stale write is not wrong, it is uninformed.

03

Order by device sequence, never by timestamp

Why

A field phone clock drifts, gets set by hand, and jumps on a timezone change. Within a device its local counter is the truth about order, and it beats the timestamp even when the timestamp disagrees.

What it costs

Cross-device ordering falls back to the client timestamp with a deterministic tiebreak, because replay has to be reproducible. Applied in arrival order instead, a shuffled batch would attempt delivered before picked up, be rejected as an illegal transition, and lose a real delivery to a networking artefact.

04

The outbox pattern rather than publishing to the broker inline

Why

A database write and a broker publish cannot be made atomic. Writing the event to a table in the same transaction makes them inseparable, and a separate worker relays it afterwards. If the API published inline, a Kafka outage would stop drivers recording deliveries, taking down the most critical path in the system with a dependency it does not need in order to be correct.

What it costs

At-least-once delivery, so consumers must be idempotent, deduplicated on event id and consumer group so fan-out is preserved. Deduplicating on event id alone would silently stop the second consumer group ever seeing an event the first consumed. Exactly-once delivery is not a real thing; at-least-once plus idempotent consumers gives exactly-once effects.

05

One transaction per mutation, not per batch

Why

A 200-mutation batch from a device dark all afternoon contains some that succeed and some that conflict. One transaction per batch means a single stale item rolls back a whole day of legitimate field work.

What it costs

Partial success has to be a first-class response shape, with a per-mutation verdict rather than one status for the request. The device is told item by item exactly what landed.

06

Authorization evaluated against the version the device saw

Why

A driver reassigned away while legitimately offline is still authorized for what he did before the reassignment. Checking ownership against current state rejects the exact case the system exists to handle.

What it costs

Requires recording the previous assignee atomically with the reassignment rather than inferring it. The naive fix of skipping the ownership check during a conflict would let anyone claim any job by manufacturing one.

07

Every conflict carries a human-readable reason

Why

A system that silently overrides a human decision is a system that human will fight, work around, and eventually disable. An automated decision nobody understands is worse than no automated decision, because now there is both a wrong answer and nobody watching for it.

What it costs

Every resolution path has to produce prose, which is more work than returning an enum. The reason string is the product; the resolution is just the mechanism.

Load testing

Found the ceiling, then moved it

A reconnect storm: 300 devices returning from long offline periods simultaneously, 15% retrying the same batch to exercise idempotency under concurrency.

First the harness was wrong, and that mattered

It seeded one device session and shared it across all 300 virtual users. The rate limiter is keyed by device id, so 98% of traffic was throttled. That measured the rate limiter, not the sync path.

Then it broke properly

sync_failed

44.66%

sync_batch_latency

p95 30s

pinned at the client timeout

http_req_failed

20.76%

iterations

~2,000

of ~3,600

Diagnosis

6,616 slow-query warnings on a normally sub-5ms dedupe lookup, and zero Postgres errors. That absence is the clue: every service inherited a 20-connection pool default and none set an override, so five pools against Postgres's max_connections of 100 is exactly 100, with zero headroom.

And the driver's connection timeout bounds the initial TCP connect, not time spent waiting for a client in the pool's internal queue, so exhaustion never throws. It queues silently, latency climbs with nothing logged, clients time out, and their retries add more work on top.

Fix · pools rebalanced to actual demand, worst case 80 with 20 spare

sync_failed

44.66% → 20.15%

http_req_failed

20.76% → 10.53%

iterations

~2,000 → ~2,765 +38% throughput

Still not fully resolved, honestly

p95 remains pinned under the full 300-device storm. With connections no longer scarce, Postgres itself is the next constraint: contended row locks, WAL flushes, and lock waits across a sequential per-mutation-transaction design, on a single container on one laptop. The next lever is horizontal API replicas behind a load balancer, which is a larger change than a pool tweak, and it is not done.

Outcome

Running every documented step surfaced seven real defects

Built end to end and run end to end, which is the part that mattered. One of the seven was an API reporting a write as applied while the row never changed. All 115 tests pass across six layers with none skipped.

Load tested to failure, diagnosed, fixed, and re-measured, with the numbers published rather than summarised. The source is public, so the internals are not merely claimed.