--- title: "Disaster recovery planning: the parts that are not backup" description: "Per-system recovery objectives, the dependencies that break failover tests, what warm and pilot light really cost, and who may declare a disaster." url: "https://nixit.io/resources/disaster-recovery-plan/" lang: "en" --- # Disaster recovery planning: the parts that are not backup A tested backup is a prerequisite for disaster recovery, not a plan. It answers one question, where the data comes from, and the moment a data centre floods or a hypervisor cluster loses its storage, several harder questions arrive at once. In what order does everything come back. Who is allowed to decide that the secondary site takes over. How long will it actually take. What happens to the orders that arrive during the gap. Who talks to customers, and who files the regulatory notification. A disaster recovery plan is the written answer to those questions, produced in advance, because nobody produces good answers while a room full of people waits. If the underlying copies are not yet in order, [the 3-2-1 rule](/resources/3-2-1-backup-rule/) is the place to start, and everything below assumes that part already works. ## A restore is a task, a plan is a decision tree Restoring data is a technical operation with a measurable result. A disaster recovery plan is a sequence of decisions covering people, ordering, dependencies and communication, and most of it has nothing to do with storage at all. Startup order shows the difference most clearly. The data can be complete and correct, and the outage still triples in length if the application starts before its database, and the database before the identity service that authenticates the accounts it runs under. That order is written beforehand, from a map of what depends on what. ## Authority: who is allowed to declare a disaster This is the most frequently missing section and the most expensive one to omit. In real outages the first two hours often go to arguing about whether the situation is bad enough to fail over, because nobody wrote down who decides and on what basis. A usable plan has three elements. Observable criteria, phrased so that any two people reading them reach the same conclusion, for example that the primary system has been unreachable for more than 45 minutes with no identified cause. A named decision maker with at least two deputies, for the predictable case where the first person is on a flight. And a time-boxed escalation that transfers authority automatically if the named person cannot be reached within a stated window. Failover has a cost of its own, so the criteria protect against premature declaration as much as against hesitation. Write down who decides failback too. It happens under calmer conditions, and it is the step where data written to the secondary site most often gets lost. ## Objectives belong to systems, and they have to survive arithmetic Two numbers structure the plan. Recovery time objective is how long a system can be unavailable. Recovery point objective is how much work the organisation can afford to lose. Both are business decisions rather than technical ones, and the common mistake is setting them once for the whole estate. A single figure is wrong in two directions simultaneously. Four hours is expensive nonsense for a scanned document archive and dangerously generous for order intake. Real arithmetic helps. An order system taking 30 orders an hour through a 10 hour trading day, backed up once nightly, carries a recovery point objective of a full trading day. A failure at 16:00 loses roughly 200 orders, and the recovery work is not restoring a file, it is reconstructing those orders from payment processor records and email. If that is unacceptable, the fix is log shipping every 15 minutes and the cost that comes with it, not a stricter sentence in a policy document. Recovery time also has to pass a physics check. Two terabytes moved across a 200 Mbps link, about 25 MB per second at best, takes roughly 22 hours of pure transfer, before decompression, service startup, or any validation. A four hour recovery time objective written next to that architecture is an aspiration, not an objective. Either the architecture changes or the number does. ## The dependencies that break failover tests Failover tests rarely fail on the application being recovered. They fail on the same short list of supporting services. **DNS.** Cutting traffic over to a secondary site usually means changing a record. With a TTL of 3600 seconds, a portion of clients and intermediate resolvers keep the old address for an hour after the change, and TTLs of a day are common on records nobody has revisited. Lowering the value takes minutes, and it has to happen before the incident. **Identity.** Domain controllers, the federation service and multi-factor authentication are dependencies of everything else, including the administrative access needed to run the recovery. A plan that assumes engineers can authenticate while the service that validates them sits in the unavailable site contains a loop. **Licensing.** Licence servers, operating system activation and licences bound to a hardware identifier do not travel with a virtual machine. An application restored correctly but unable to reach its licence server produces the same outage as one never restored. **Certificates.** In many estates the private key for the public portal or the VPN concentrator exists only on the machine that has just been lost. Two more recur often enough to name. Administrative access routed through a VPN concentrator that lives in the affected site, and a password manager hosted on the infrastructure that stopped responding. Both are easy to solve and impossible to solve at the time. ## Cold, pilot light and warm, priced honestly These are not estate-wide choices. They are per-system choices, driven by that system's recovery time objective. **Cold** means the data and the procedure, with nothing running at the secondary site. Ongoing cost is storage. Recovery time is measured in days, because the environment has to be built before anything can be restored into it. For archives and internal tools that can be absent for a week, this is the correct answer rather than the cheap one. **Pilot light** keeps running only the small components that are slow to recreate: a domain controller, a database replica receiving logs, the configuration and the machine images. Everything else is created when the disaster is declared. Because compute dominates infrastructure bills and compute is almost absent between incidents, ongoing cost stays much closer to cold than to a duplicate environment. Recovery time is measured in hours. **Warm** means a working environment at reduced capacity, continuously updated, able to take traffic in minutes. Cost approaches a second production estate, which is why it is justified for one or two systems, not twenty. ## Communication during an outage Communication is part of recovery rather than something running alongside it. Left out, it generates a second incident made of phone calls landing on the people fixing the first one. The channel must not depend on the affected infrastructure. If email and the collaboration platform authenticate against the identity service that is down, the team loses its channel exactly when it needs one. An independent group chat and an offline list of phone numbers cover it. The rest is discipline. Message templates drafted in advance for the likely scenarios. A fixed update cadence honoured even when there is nothing new, because silence is read as loss of control. One authoritative source, so two departments do not publish contradictory versions. And a clear split between who speaks to customers, who to staff, and who to regulators. ## Runbooks written for the worst hour of the year A runbook is not written for its author. It is written for a colleague executing it for the first time, at three in the morning, while several people wait for an update. What separates a usable runbook from a document: numbered steps with exact commands rather than instructions to restore per standard procedure, the expected output after each step so a deviation is visible immediately, decision points marked explicitly with both branches described, a duration estimate per stage which doubles as the basis for status updates, and a stated path to credentials that does not assume the usual systems are answering. The quality test is straightforward. Someone who did not write the procedure executes it without asking questions, and every question that comes up marks a gap in the document. ## Tabletop exercises and real failovers find different faults A tabletop exercise takes about 90 minutes, touches nothing in production, and exposes the decision and communication gaps: unclear authority, stale contact details, two teams each assuming the other handles the same step. It is cheap enough to run quarterly. A real failover finds what discussion cannot. The IP address hardcoded in an application config. The firewall rule that exists in one site only. The expired certificate. The script that depends on a file share that is not there. Nobody raises these in a tabletop because nobody knows they exist. One real test a year in an announced window is a reasonable minimum for critical systems. If the business never grants the window, that is itself a finding worth recording in writing: the plan is unvalidated and the recovery time objective remains an estimate. ## What changes when a partner operates the estate If recovery is delivered by a managed provider or a nearshore team, the technical work is easier to buy and the authority question gets harder. Three points are worth settling in the contract rather than discovering during an incident. Delegated authority. A provider watching your systems overnight may detect the failure hours before anyone on your side is awake. Either they hold a written, bounded mandate to fail over specific systems against stated criteria, or the plan accepts the delay of waking someone up. Both are defensible; the undefined middle is not. Demonstrated versus contracted recovery time. A number in a service agreement is a commercial commitment. Ask for the last test report, the measured duration and the list of issues it found. A partner who tests will have that document. Regulatory scope. Organisations in scope of NIS2 owe staged reporting on significant incidents, with an initial notification within 24 hours and a fuller report within 72, and supervision sits with a national authority in each member state. In Romania that authority is the Directoratul Național de Securitate Cibernetică. The reporting duty stays with the in-scope organisation, so the plan needs a named person for it and an evidence trail from the provider that supports the filing. ## Where to start Take your single most important system and answer three questions in writing: who declares the disaster, what that system needs in order to start, and how long it actually takes to bring it up somewhere else. Any question without a written answer is the first piece of work. We design and test recovery plans as part of [backup and disaster recovery](/services/backup-disaster-recovery/), and mapping dependencies before setting objectives belongs to [infrastructure architecture](/services/infrastructure-architecture/).