Backup systems tend to report success from the writer’s point of view. Recovery is experienced by the reader: credentials must work, catalogs must be available, dependencies must exist, and the restored data must be internally consistent.
Name the recovery objectives
The recovery point objective describes how much recent data may be lost. The recovery time objective describes how long restoration may take. These are design inputs, not values to discover during an outage.
Separate failure domains
A second copy on the same host protects against some mistakes but not loss of the host. Copies should differ by location, credentials, and—where practical—administrative boundary. At least one copy should resist immediate modification by the production account.
Restore a representative slice
Frequent small restores reveal permission changes, missing encryption keys, expired credentials, incompatible formats, and incomplete runbooks. Periodically perform a larger recovery that exercises ordering and capacity constraints.
Integrity checks confirm stored bytes. Application checks confirm that those bytes form a usable system. Keep both.
Keep evidence of recovery
Record the backup identifier, restore environment, elapsed time, validation performed, and any manual step. Trends in restore time and repeated exceptions are useful capacity signals.
Small restore checklist
- Select a backup without using production-only metadata.
- Retrieve required keys through the documented path.
- Restore into an isolated destination.
- Run structural and application-level checks.
- Measure elapsed time against the recovery objective.
- Update the runbook while details are fresh.