The 2 AM Test

Most small teams back up their database. A cron job runs mysqldump every night, compresses the output, and copies it somewhere. The backup exists and it probably works, but nobody has tried restoring it under pressure, and that’s a different thing entirely.

If your server disappeared right now, how long until your application is running again? Have you ever measured that?

What you think will happen

You provision a new VPS. Install Nginx, PHP-FPM, MySQL. Pull the code from Git. Restore the database from your latest mysqldump. Update DNS. You figure an hour, maybe two.

What actually happens

You provision the server fine. Then you realize you don’t remember which PHP extensions your app needs, so you install them one at a time as errors appear. The Nginx config isn’t in Git because you edited it on the server six months ago. Your .env file has twelve values you don’t have memorized, and four of them are API keys for third-party services that you’ll need to look up in various dashboards. The MySQL restore works, but you forgot that user uploads live in /var/www/myapp/storage/uploads/ and those aren’t in Git either. You rsync them from your backup, except the backup server is also behind SSH and you need to find the key for that.

Each of these is a ten-minute problem in isolation. Stacked up at 2 AM when your users are waiting, they compound into hours.

The actual risk

mysqldump and rsync are both reliable. What breaks a recovery is everything around the backup: the knowledge in your head that hasn’t been written down, the config files that live only on the server, the sequence of steps you’ve never rehearsed.

Small teams are especially vulnerable here because the person who set up the server is usually the person who’d do the recovery. If that person is unavailable, sick, or just panicked at 2 AM, the undocumented steps become blockers.

Run the drill

Pick a Saturday. Provision a fresh VPS at a provider you already use. Then rebuild your application from scratch, using only your backups and whatever documentation you have. Time it.

You’ll discover gaps. Maybe your Nginx config isn’t backed up. Maybe your crontab has four entries you forgot about. Maybe your off-site backup takes 40 minutes to download because it’s 8 GB and you never noticed. Write down everything that slowed you down.

Then do three things:

Write a rebuild checklist. Every step from “provision server” to “application responds in browser.” Keep it on the server itself, in your password manager, and in your Git repo. When you change the server setup, update the checklist.

Back up your config files. Nginx configs, .env, crontabs, SSL certificates. These are small files that rarely change, and forgetting them costs disproportionate time during recovery.

Test your database restore. Not just “does the file decompress.” Actually import it into MySQL on a fresh server and verify the application works against it. A mysqldump that completes without errors can still produce a restore that fails if your character sets or users don’t match.

How long should it take

With a checklist and accessible backups, a typical small PHP application on a single VPS should take under an hour to rebuild. If it’s taking two or more, something in your backup or documentation process needs fixing. Run the drill again after you fix it.

A tested process and verified backups are enough for a good disaster recovery story. The drill is how you get both.