The Great Server Lockout of 2025

The Setup

I was debugging a database connection issue between two servers. The connection kept failing, and after digging through logs I traced it to Tailscale ACLs blocking traffic between the nodes. Simple fix. Just re-authorize Tailscale on the affected server.

I hit re-authorize.

My SSH session died instantly.

The server had no external access except through Tailscale. I had just killed the only way in.

The Panic

The server is on Hetzner. No public SSH port, no VPN fallback, no out-of-band management card. Just Tailscale, which I had just disconnected.

I tried the obvious things first:

• SSH via public IP — blocked, there is no open port
• Hetzner console VNC. Not available on this server type
• Firewall changes through the admin panel — unrelated to the problem

None of it worked. The server was up, healthy, and completely unreachable.

Rescue Mode, Take One

Hetzner has a rescue mode that boots the server into a minimal Linux environment from network. I enabled it, rebooted, and mounted the filesystem.

The plan: modify the firewall rules to open SSH on the public interface, reboot into the normal OS, reconnect.

I edited the firewall config. Rebooted. The config file was corrupted. The server came up with no firewall rules loading at all but also no SSH access because the service was now failing to start.

Worse than before.

Rescue Mode, Take Two

Back into rescue mode. This time I stopped trying to be surgical about it:

1. Disabled the firewall entirely
2. Copied my SSH public key into the server’s authorized_keys
3. Re-authorized Tailscale from the rescue environment
4. Rebooted

The server came up. SSH worked over the public IP. Tailscale reconnected. The original database issue was fixed because the re-authorization had cleared the ACL problem.

Total downtime: about 45 minutes of self-inflicted chaos.

What I Changed After

The immediate problem was obvious. I had a single point of access to a production-adjacent server. But the deeper issue was that I’d never tested what happens when Tailscale drops on a machine with no other access path.

What I did:

• Every server now has SSH accessible on the public interface, firewalled to specific IPs as a fallback. Tailscale is the primary access method, not the only one.
• I documented the Hetzner rescue mode procedure so I’m not figuring it out under pressure next time.
• Re-authorization on Tailscale gets tested on a non-critical machine first. I now know it terminates the session immediately. The docs don’t make that obvious.

The fix took 45 minutes. Setting up proper fallback access took 20. Should have done it first.