Payments are experiencing issues due to temporary restrictions in Russia. If your payment does not go through, please submit a support request.Our support team is available 24/7 — we are always here to help with hosting and server issues.We are now accepting requests for dedicated server rental and colocation services in our data center.Reminder: we recommend enabling backups for additional data protection.A new VPS/VDS lineup with NVMe storage and improved performance is now available.Maintenance work on some servers has been completed. All services are operating normally.
Article11 min readViews2

Webhook Arrived Out of Order: How to Prevent an Order from Rolling Back to an Old Status

We examine why an event_id without long-term atomic persistence fails to protect against retries, and how versioning, gap detection, and reconciliation prevent late events from rolling back an order.

Comments 0

A woman in the light picking zone separates a delayed event card into a verification tray next to the package
In this article

If different webhooks for a single order can arrive out of order, deduplication alone is insufficient. A stable event_id from the source protects against exact repetition of local changes: the event log and the new local business state are recorded within a single atomic boundary. This local transaction alone does not make external effects atomic—such as a fiscal receipt, a refund, or a notification; for those, effect idempotency, outbox, a separate effect log, or compensation via the integration contract is required. Protection against processing a late, stale event for the recipient is provided by a separate version or another sequence tag, whose meaning is guaranteed by the source. If such a tag is absent, it is safer to treat the webhook as a signal and read the current state of the object via the API.

A practical criterion is this: the local version must not decrease, a skipped expected version must not be silently applied, a retry must not trigger the effect twice, and an unclear order must go to reconciliation. The request receipt time on your server is unsuitable for this: it describes delivery, not the sequence of business changes.

Before comparing the order, the receiver must verify the signature or another trust mechanism specified in the contract, the message schema, the version type, and the boundaries within which event_id remains unique. Otherwise, a forged event with a higher version can halt the stream or displace the correct state.

Duplicates and stale events are different cases

Events, deliveries, and orders have different identifiers. In the article about repeated webhooks and idempotency, we already covered the repetition of a single event_id. Here is a different failure: two events have different identifiers but describe different versions of the same order. Therefore, a unique constraint on event_id may work perfectly, yet the status still rolls back.

Input

What to compare

Model code

Business impact

New event_id, next version

version = stored + 1

applied

Apply once

Same event_id

Long-term event log

duplicate

Do not repeat

Different event_id, version ≤ stored

Object version

stale

Do not revert state

version > stored + 1 with continuous numbering

Expected and Received Versions

gap

Stop Application and Verify the Gap

The last statement is valid only when the contract guarantees sequential versions without gaps for a single object and the entire event scope relevant to your receiver. If the version is monotonic but may skip, or you subscribe only to a subset of events, a value of 3 following 1 does not by itself prove loss. In that case, accept the newer confirmed version according to the source rules, or first read the current object.

You must identify the version scope. One counter may order the entire order, while another orders only payment or shipment. With independent counters, an equal version paired with a different event_id may describe a separate valid action rather than an outdated event. The version ≤ stored rule applies only to the entity and flow for which the contract defines a complete order.

Which sequence marker can be used

In the official Shopify documentation for webhooks, it is explicitly stated that order is not guaranteed even within a single topic or between topics of the same resource. To organize events, the service points to the occurrence time and separately recommends periodically re-reading source objects and matching them against the local state. This is a Shopify property, not a permission to sort any webhook by a field with a similar name.

According to the official Google Cloud Pub/Sub documentation, ordering is guaranteed only under specified conditions. Messages must have a single non-empty ordering key; the subscription setting ordering must be explicitly enabled; and publishing must comply with regional requirements. There is no global ordering between messages with different ordering key values. The queue does not provide a retroactive guarantee: it must be explicitly obtained from the contract and configuration.

The official analysis from Stripe also warns that events may be delivered in any order and recommends fetching the current object before performing dependent actions. For integration, this is a critical boundary: a snapshot from an old event and the current state from the source may already differ.

The preferred selection order is as follows: a guaranteed sequence or aggregate version; then the documented timestamp of the business event with an equality resolution rule; then reading the current object from the source. The local time of request acceptance remains useful for diagnosing latency but does not become the business version.

The source defines the format for the version value. JavaScript Number is safe for sample values 1–3 but does not necessarily represent an arbitrary 64-bit integer value of the sequence field. Before implementation, verify the range and format: sometimes BigInt, a decimal string with consistent comparison, or the standard comparator SDK is required.

Minimal model: applied, duplicate, stale, and skipped

Below is a sample model, actually executed on November 16, 2026, on Node.js v24.22.1 in Linux x64. Before running, create a new empty directory on the test machine, save the following block as webhook-order-example.mjs, and run the command from that directory. All input events are defined within the file; the model receives a verified signature, schema, and field type, while event_id is considered globally unique in this sample stream. It does not access the network, write other files, or modify external data. After verification, delete only the created file and the now-empty test directory. The model assumes the source issues sequential versions without gaps for all tracked transitions for each order.

import assert from 'node:assert/strict'; function createReceiver() { const orders = new Map(); const processedEvents = new Set(); return { apply(event) { if (processedEvents.has(event.eventId)) { return { result: 'duplicate', current: orders.get(event.orderId) ?? null }; } const current = orders.get(event.orderId) ?? { version: 0, status: 'unknown' }; if (event.version <= current.version) { processedEvents.add(event.eventId); return { result: 'stale', current }; } if (event.version > current.version + 1) { return { result: 'gap', expectedVersion: current.version + 1, receivedVersion: event.version, }; } const next = { version: event.version, status: event.status }; orders.set(event.orderId, next); processedEvents.add(event.eventId); return { result: 'applied', current: next }; }, get(orderId) { return orders.get(orderId) ?? null; }, }; } const receiver = createReceiver(); assert.equal(receiver.apply({ eventId: 'evt-1', orderId: 'order-42', version: 1, status: 'created', }).result, 'applied'); assert.deepEqual(receiver.apply({ eventId: 'evt-3', orderId: 'order-42', version: 3, status: 'paid', }), { result: 'gap', expectedVersion: 2, receivedVersion: 3 }); assert.equal(receiver.apply({ eventId: 'evt-2', orderId: 'order-42', version: 2, status: 'awaiting_payment', }).result, 'applied'); assert.equal(receiver.apply({ eventId: 'evt-3', orderId: 'order-42', version: 3, status: 'paid', }).result, 'applied'); assert.equal(receiver.apply({ eventId: 'evt-2', orderId: 'order-42', version: 2, status: 'awaiting_payment', }).result, 'duplicate'); assert.equal(receiver.apply({ eventId: 'evt-old', orderId: 'order-42', version: 2, status: 'cancelled', }).result, 'stale'); assert.deepEqual(receiver.get('order-42'), { version: 3, status: 'paid' }); console.log(JSON.stringify({ ok: true, final: receiver.get('order-42') }));

node webhook-order-example.mjs

{"ok":true,"final":{"version":3,"status":"paid"}}

The command completed with code 0. Version 1 is applied first. Version 3 arrives before version 2 and receives code gap — "version skip", so the state does not change. After version 2, a re-delivery of version 3 is applied. The re-delivery evt-2 receives duplicate — "duplicate", while another event with the already outdated version 2 receives stale: its version is outdated, but the event's validity period is not evaluated here. The final result remains paid/version 3.

The check confirms only the logic of decisions within a single process. Set disappears on restart and does not protect against two parallel handlers. In the working database, four operations must constitute a single transaction or a conditional atomic write with locking according to the rules of the selected DBMS: a uniqueness check for event_id, version comparison, writing the new local state, and committing the processed event. For an external action within this boundary, the intent is preserved, for example by writing outbox or an effect log, while the action itself is performed with an idempotent key or a consistent compensation. These properties are not checked by the executed model.

The model also starts with a local version of 0 and expects version 1. The actual object may first arrive with version 42. Starting the receiver requires a separate initial step: fetch the current snapshot and save the original version, or accept the first event only if a documented full-snapshot contract exists. Versions 1–41 must not be automatically declared lost.

What to do when a version is skipped

A skipped version is not a reason to loop endlessly with an error. First, save the event's safe metadata and the reason for stopping, but do not execute any dependent action. Then use the source's prescribed method: request the missing event, read the current object, or run a reconciliation. After receiving the data, re-evaluate the decision based on the already saved version.

It is important to distinguish between restoring the current state and reproducing each effect. For projecting the status, obtaining a fresh snapshot of version 5 may suffice. However, if versions 3 and 4 triggered a cash (fiscal) receipt, return, shipment, or notification, the latest snapshot does not restore these transitions. In that case, you need backfill events, a separate effects log, or coordinated compensation — according to the specific integration contract.

The material on unknown API status addresses a related but different problem: the recipient does not know the field value. Here, the value is known, but it arrived too late. Neither case can be resolved by applying a default value.

Fallback scenario without a version

Do not infer order from the request number, the timestamp in your log, or the message position in the current batch. Treat the webhook as a signal of change, then request the current order, payment, or shipment from the source. If the read API is temporarily unavailable, save the state as pending_reconciliation — "pending reconciliation" — and perform a limited number of retry attempts according to the specific service rules.

Periodic reconciliation is needed even after successful delivery: a webhook may be missed due to a subscription error, processing failure, or downtime. Reconciliation compares objects by a stable identifier and a documented change indicator, not by simply repeating all side effects. This branch is based on provider documentation; the executed script does not verify it.

On the sender side, the transactional outbox reduces the risk of losing an event between local recording and sending. However, outbox does not define the order in which independent deliveries will be seen by a specific recipient, nor does it cancel version checking on their side.

Handler acceptance criteria

  1. Record exactly what event_id, object_id, and version/sequence identify; specify the version and uniqueness scope, and do not substitute one for another.

  2. Before business logic, verify the signature or other trust contract, schema, mandatory fields, and allowed version format.

  3. Define initial synchronization for an object that first appears not at version 1.

  4. Check the new transition, an exact repeat of the same event_id, and another event with an old version.

  5. Deliver version 3 after version 1. Only with guaranteed continuous numbering must the handler detect a gap and not silently apply the state.

  6. After resolving the gap, repeat version 3 and ensure the state moves only forward.

  7. Simulate a failure between the state write and the event_id mark in an isolated test instance of the same database and version, without production data or secrets; both writes must have a consistent boundary. After verification, remove the test records and the test environment according to a predefined procedure.

  8. Run two handlers for the same object in parallel and verify atomic version protection, not just the preliminary SELECT.

  9. Separate the fresh snapshot recovery test from the test of a missed irreversible effect.

  10. Disable the read API and verify the limited reconciliation retry, the observed queue, and the manual path without automatic status rollback.

  11. Compare metrics: age of pending_reconciliation, count of stale, duplicate, and gap, the share of successful reconciliations, and the oldest unprocessed object.

The handler is not ready just because all requests returned a green HTTP response. It is ready when the team can intentionally reorder different events, repeat one of them, create a gap, and prove that the business state did not roll back, the side effect did not repeat, and uncertainty remained visible until reconciliation.

Discussion 0

Share your experience and ask questions. Comments without links appear after editorial review.

No comments yet. Start the discussion.