Syncing a catalogue you do not own

Pulling a large supplier catalogue looks like a loop over pages. The naive version fetches them all in parallel, exhausts memory, gets rate limited, and fails somewhere in the middle with no way to resume. The version that works is a bounded worker pool drawing from one shared cursor, with connection reuse, because a thousand fresh TLS handshakes is its own bottleneck.

The subtle bug is time. Incremental sync means asking for everything changed since the last pass, and two systems never agree exactly on what time it is. Anything updated in the gap between your recorded cursor and the supplier clock silently never arrives. It does not error. The product just is not there, and nobody notices until a customer asks for it.

The fix is to rewind the cursor a short window on every pass and accept reprocessing some products, which idempotent upserts absorb for free. A small amount of duplicate work buys the guarantee that nothing is missed, and that is an easy trade.

The other thing worth building early is per-reason counters on whatever gates the catalogue. Thousands of rules decide what gets listed, and a supplier changing a field can start excluding everything. Counting exclusions by reason turns a mystery into a number you can watch. Without it, the first symptom of a schema change is an empty storefront.