Chapter 11 · Operate

Runbooks & checklists

Procedures for the things you do rarely enough to forget. The build order in particular is not arbitrary — doing it in a different sequence is how people lock themselves out of a VPS.

Build order

Each step depends on the one before it working. The rule underneath all of them: establish the new path before removing the old one.

Order

$ ssh root@203.0.113.10
root@203.0.113.10's password: the only way in today

# admin console · device approval: on · tailnet lock: on
# nothing on any machine has changed yet

$ tailscale status
100.101.4.2 vps active · 100.117.202.65 macbook active

$ sudo ufw delete allow OpenSSH
Rule deleted — and nothing has replaced it yet

$ ssh yourname@100.101.4.2 'echo ok'
ok — the new path works before the old one closes

$ ssh yourname@100.101.4.2 'echo ok'
ssh: connect to host 100.101.4.2 port 22: Operation timed out

$ sudo ufw delete allow OpenSSH
Rule deleted — second session still connected

# not run — the VPS is unreachable
open the provider web console instead

$ mosh macbook -- tmux new -A -s phone
[detached] · roams from Wi-Fi to LTE without dropping

# not run — you never got past step 3
restore the firewall rule, then start again at 1

$ sudo sshd -T | grep -i passwordauthentication
passwordauthentication no

# not run — there is nothing left to remove
you removed it in step 2, before anything replaced it

Where you startThere is exactly one way into the VPS: port 22, open to the whole internet, with password authentication on. Everything below replaces that — and for most of it you are working through the very door you are about to close.
1 · Tailnet foundationsAccount, device approval, key expiry, tailnet lock. Pure policy, done in the admin console before a single machine changes. Nothing is exposed here and nothing is removed, which is why it is safe to go first.
2 · Install Tailscale everywhereMac, VPS, Ubuntu and phone, each confirmed online in tailscale status. You now have two ways into the VPS. Resist closing one — nothing has been tested end to end yet.
2 · …and you close public :22 while you are in thereIt feels efficient. Tailscale is installed and the box has a tailnet address, so the public rule looks redundant. It is not redundant — it is untested. A brand-new tailnet still allows everything by default, so this appears to work, which is exactly what makes it dangerous.
3 · Write the ACL policyDefault deny, then grant SSH only. Then prove it: ssh to the 100.x address and get a shell. That test is the whole reason this step sits before any hardening.
3 · The ACL shuts the only door left — stop hereDefault deny with no SSH grant yet, so the tailnet path dies the moment you save the policy. The public path went in step 2. Both closed in one move, and neither was ever tested. Recovery is the provider's web console — chapter 08 is where you checked that it works.
4 · Harden each host4a, 4b and 4c do not depend on each other, so take them in any order. This is where public :22 finally closes on the VPS — safely, because steps 1 to 3 are done and proven, and because you keep a second session open while you change the rule.
4 · Not runYou cannot reach the VPS to harden it. Get back in through the console, restore the firewall rule, and start again from step 1 — this time closing nothing until step 4.
5 · Phone: keys, Mosh, tmuxGenerate the key in the Secure Enclave, add the public half to every host, then layer Mosh and tmux on top. Plain SSH first, always — it fails with far clearer errors than Mosh does.
5 · Not runEverything above this point assumed a machine you can still log into. Fix the VPS, then re-run the order from the bottom.
6 · Verify, then remove old accessPassword authentication off, Remote Login off on the Mac, the old public rule gone for good. Every item here is a removal, which is precisely why they all live at the end.
6 · Not runThere is nothing left to remove. You removed it in step 2, before anything had replaced it — and that single reordering is the whole difference between these two runs.

Step 6 is last for a reason. Deleting the public SSH rule, disabling password auth, or switching off Remote Login before confirming the replacement path works is the most common way to lose a machine. Switch to Remove first above and watch the same six steps end with a web-console recovery instead. Confirm, then remove.

Onboarding a new device

A tablet, a new laptop, a replacement phone:

Install Tailscale, sign in, and approve the device in the admin console if approval is on.
Generate a key on the device. Never copy one in. Prefer hardware-backed if available.
Add the public key to each host's authorized_keys, with a comment naming the device and year.
Update the ACL policy if the device needs a group or tag it does not yet have.
Test plain SSH first and verify host key fingerprints.
Then add Mosh, then the saved host entry with tmux new -A -s phone.

Offboarding a device

Order matters — cut network access first, because it is instant and global:

Tailscale admin console → remove the device.
Remove its line from authorized_keys on every host.
If the key was file-based rather than hardware-backed, treat it as compromised and rotate anywhere else it was used.
Wipe remotely if the device is lost rather than retired.
Confirm: the device no longer appears in tailscale status anywhere.

# Find every host still trusting a key, by comment
grep -n "old-laptop" ~/.ssh/authorized_keys

# Remove it in place, keeping a backup
cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.bak
grep -v "old-laptop" ~/.ssh/authorized_keys.bak > ~/.ssh/authorized_keys

Rotating a key

The safe sequence adds before it removes, so you are never without a working key:

# 1. New key on the device
ssh-keygen -t ed25519 -C "macbook-2026-q3"

# 2. ADD it everywhere — do not remove anything yet
ssh-copy-id -i ~/.ssh/id_ed25519_new.pub yourname@host

# 3. Prove the new key works, on its own
ssh -i ~/.ssh/id_ed25519_new -o IdentitiesOnly=yes yourname@host 'echo ok'

# 4. Only now remove the old line, and delete the old private key

Periodic maintenance

Cadence	Task
Weekly	Check `/var/run/reboot-required` on Linux; reboot for kernel and OpenSSL updates. Confirm unattended-upgrades is still running.
Monthly	Review the tailnet device list. Skim auth logs for unfamiliar key fingerprints. Re-run the listening-port audit on each machine — new tools add new listeners.
Quarterly	Review every `authorized_keys` line. Confirm backups restore. Re-read your ACL policy against what you actually use now.
Yearly	Rotate file-based SSH keys. Verify recovery keys — FileVault, LUKS, tailnet lock — are still where you think and still correct.

# The monthly audit, on any machine
sudo ss -tulpn | grep LISTEN                 # Linux
netstat -an -p tcp | grep LISTEN             # macOS
ssh-keygen -lf ~/.ssh/authorized_keys        # who can log in
sudo grep 'Accepted publickey' /var/log/auth.log | tail -30

The daily workflow

# From the phone — one command, everything else is automatic
mosh macbook -- tmux new -A -s phone

# Inside, if you use Herdr for agents
herdr

# Detach and put the phone away; work continues
# tmux:  ctrl-b d

Long jobs belong in their own window

Give a long build or migration its own tmux window rather than running it in the pane you are also typing in. Then Ctrl-b d is always safe, and you can check on it from a different device entirely without disturbing anything.

Backups worth having

Everything above assumes you can rebuild a machine. That assumption is worth testing rather than holding.

Dotfiles in git — .tmux.conf, .ssh/config, herdr/config.toml. Never private keys.
Recovery keys in a password manager reachable from a device that is not the one they unlock.
The ACL policy file kept in version control; the admin console is not a backup.
A restore test, at least once. An untested backup is a hypothesis.

Never put private keys in a dotfiles repo

Even a private one. Repositories get cloned to machines you did not anticipate, shared with collaborators, and occasionally made public by accident. Keys are generated per device and stay there — that is the entire model, and a dotfiles repo is the most common way people break it without noticing.

Full deployment checklist

Everything, in one place. This state persists in your browser.

Tailnet: device approval on, key expiry considered per device, tailnet lock enabled.
Tailnet: ACL policy is default-deny and version controlled.
All four devices online in tailscale status; MagicDNS working.
MacBook: FileVault, SIP, Gatekeeper, firewall on.
MacBook: listeners audited; databases on loopback.
MacBook: one sshd only, bound to the tailnet IP, refused on LAN, runs = 1.
VPS: non-root sudo user, key-only auth, UFW default-deny.
VPS: public SSH closed after tailnet access confirmed; console recovery tested.
VPS: unattended-upgrades verified; Docker ports bound to loopback.
Ubuntu: LUKS confirmed, Secure Boot on, firmware password set.
Ubuntu: UFW default-deny, MAC randomisation, AppArmor enforcing.
iPhone: passcode, short auto-lock, encrypted backups.
iPhone: Blink with a Secure Enclave key; public key on every host.
Mosh installed everywhere; Wi-Fi→LTE roaming tested.
tmux everywhere with matching configs; Herdr prefix remapped if nesting.
Host key fingerprints verified from each client.
Recovery keys stored off-device and tested.

The commands in this guide change firewall and login settings, and can lock you out of a machine. Practise on something disposable first. Everything here is provided as is, with no warranty — you accept the risk of running it. Read the disclaimer.