The 2 AM Restore Test

Most small teams back up their database. A cron job runs mysqldump every night, compresses the output, and copies it somewhere safe. The backup exists, and it probably works. But nobody has actually restored it under pressure, which is a separate question from whether the file is being written.

So here’s the one worth asking: if your server disappeared right now, how long would it take to get the application running again? Most people have never measured it.

The optimistic version

You provision a new VPS. You install Nginx, PHP-FPM, and MySQL, pull the code from Git, restore the database from your latest mysqldump, and update DNS. An hour, you figure, maybe two.

What usually happens

Provisioning the server goes fine. Then you realize you don’t remember which PHP extensions the app needs, so you install them one at a time as the errors appear. The Nginx config isn’t in Git, because you edited it on the server six months ago. Your .env file has twelve values you don’t have memorized, and four of them are API keys for third-party services you’ll have to dig out of various dashboards. The MySQL restore works, but then you remember that user uploads live in /var/www/myapp/storage/uploads/ and aren’t in Git either. You go to rsync them from your backup, only the backup server is behind SSH too and you need to find that key first.

Any one of these is a ten-minute problem on its own. Stacked up at 2 AM with users waiting, they add up to hours.

Where recoveries break

mysqldump and rsync are both reliable. What tends to break a recovery is everything around the backup: the knowledge that only lives in your head, the config files that only exist on the server, and the sequence of steps you’ve never run start to finish.

Small teams are especially exposed here, because the person who set the server up is usually the same person who would rebuild it. If that person is unavailable, sick, or simply rattled at 2 AM, every undocumented step becomes a blocker.

Running the drill

Pick a Saturday. Provision a fresh VPS at a provider you already use, then rebuild your application from scratch using only your backups and whatever documentation you have. Time it.

You’ll find gaps. Maybe the Nginx config was never backed up. Maybe your crontab has four entries you’d forgotten about. Maybe the off-site backup takes 40 minutes to download because it’s 8 GB and you never noticed. Write down everything that slowed you down.

Then it’s worth doing three things with what you learned.

The first is to write a rebuild checklist: every step from “provision server” to “application responds in the browser.” Keep a copy on the server, one in your password manager, and one in the Git repo, and update it whenever you change the server setup.

The second is to back up the config files that tend to get left out, such as Nginx configs, .env, crontabs, and SSL certificates. They’re small and they rarely change, which is exactly why they’re easy to forget and costly to reconstruct.

The third is to test the database restore itself, not just whether the file decompresses. Import it into MySQL on a fresh server and confirm the application runs against it. A mysqldump that finishes without errors can still restore into something broken if the character sets or users don’t line up.

How long it should take

With a checklist and backups you can actually reach, a typical small PHP application on a single VPS should rebuild in under an hour. If it’s taking two or more, something in the backup or documentation process needs attention, and it’s worth running the drill again once you’ve fixed it.

A tested process and verified backups are what a good recovery comes down to, and the drill is how you get both.