Payments are experiencing issues due to temporary restrictions in Russia. If your payment does not go through, please submit a support request.Our support team is available 24/7 — we are always here to help with hosting and server issues.We are now accepting requests for dedicated server rental and colocation services in our data center.Reminder: we recommend enabling backups for additional data protection.A new VPS/VDS lineup with NVMe storage and improved performance is now available.Maintenance work on some servers has been completed. All services are operating normally.
Article5 min readViews1

Database Replica Repeated the Deletion: Where to Find the Pre-Error State

Using MySQL as an example, we examine the boundary between a second working database and saved history, as well as the recovery window for a lagging replica.

Comments 0

Two metal disks with identical notches and a separate intact disk behind them represent replicas and saved history.
In this article

The store has a second database server, and the erroneously deleted products have disappeared from it as well. Replication may have been functioning correctly: the change reached the second node and was applied. To return to the pre-error state, a saved past moment is needed, one that subsequent changes will not overwrite.

Let's examine this boundary using standard MySQL 8.4 replication. This explains the data protection mechanism, not a guide to stopping replication or restoring a live database. Specific commands depend on the topology and require a separate, validated plan. Documentation was verified on September 27, 2026; the emergency scenario on the test stand was not reproduced here.

Two servers can store the same error

Consider a training sequence. At 10:00, the primary database contains the required catalog. At 10:03, the application erroneously deletes part of the product catalog, and the transaction commits successfully. At 10:04, the replica applies this change. At 10:20, the manager notices the missing items.

At the moment of discovery, both nodes may be technically healthy and synchronized with each other. A check stating "server is available, replication has caught up to the source" does not confirm the catalog's correctness. It answers a different question: did the changes reach the second node? An erroneous application action is also a data modification.

Switching to the replica does not automatically restore the past. If it has already applied the deletion, switching only serves the same catalog to shoppers from a different server. Therefore, emergency plans must explicitly distinguish between hardware failure and logical errors. The former requires an available, functional instance; the latter requires a valid state prior to the incorrect operation.

What a separate backup adds

The MySQL documentation describes a replication scenario for backup that includes a distinct step: obtain a backup from the replica. Having a replica and having a completed backup are different outcomes. Using a second node can help organize the copying process, but it does not by itself define a retention period for past states.

Consider the same training accident. If a separate usable copy was made at 09:50, it still contains deleted products. However, after 09:50, real orders and correct price changes may have occurred. Rolling the entire database back to that time would also revert those changes. Therefore, "an old copy was found" marks the beginning of selecting a recovery method, not a license to immediately replace the live database.

The administrator must establish a boundary for the erroneous operation, access the history of changes, and be able to restore the required state in isolation. For a store, external payments, synchronization with 1C, and actions that have already occurred outside the database are verified separately. Restoring an order row does not cancel a real payment and should not trigger reprocessing without verification.

A lagging replica provides a limited window

MySQL 8.4 supports intentionally delayed replication: a transaction is applied no earlier than a specified interval after it is committed on the immediate source. By default, the additional delay is zero. The documentation explicitly names protection against user errors as one of the uses for this delay.

In the example, we set a window of 30 minutes. An erroneous change was recorded at 10:03, so it must not be applied to such a replica before 10:33. If the error is detected at 10:20, 13 minutes remain until that boundary. However, these minutes include diagnosis verification, the responsible party's decision, and execution of a pre-prepared procedure. Once the delay expires, the replica will continue applying transactions. To stop the application of an erroneous transaction, the administrator needs a pre-prepared procedure.

If the error is noticed at 11:00, one can no longer rely on the saved state: the minimum delay has passed. The actual lag may be greater for other reasons, but this is not a reliable time buffer. It is verified against the actual state of the replica. An intentional delay also means that this instance specifically serves older data; it cannot be unconditionally considered the best candidate for an urgent store failover.

What must be known before an incident

  • What task each database instance solves: reading, failover, creating backups, or delayed application of changes.

  • Where individual copies are stored, which past points are available, and when recovery from them was tested.

  • How long it typically takes the team to notice a data error and who makes the decision to restore.

  • How store data is reconciled with payments and integrations after reverting to a past state.

In this check, detection time and a specific recovery point are more valuable than the number of identical servers. If an incorrect import is noticed only the next day, a thirty-minute window does not fit this scenario. If previous copies are overwritten before detection, an additional current replica will not restore the lost history.

A good answer to the question "Can we restore the deleted catalog?" names the saved snapshot, how to verify it, and the fate of later correct changes. The answer "we have two servers" describes the infrastructure but lacks these three pieces of information.

Discussion 0

Share your experience and ask questions. Comments without links appear after editorial review.

No comments yet. Start the discussion.