Chapter 02 · The identity plane

SSH, keys & hardening

Tailscale decides who can reach the port. SSH decides who gets a shell. These are separate questions and it is worth keeping them separate in your head — a great many bad setups come from assuming the VPN makes the SSH configuration unimportant.

What actually happens when you log in

Public-key authentication is often described as "the server has your public key and lets you in." That skips the part that matters. Your private key is never sent, and the proof cannot be replayed against a different session.

Scenario
Client (your phone) Server (sshd) …or whoever answered TCP is open · nothing is authenticated yet 1 · version + algorithm exchange 2 · host key + Diffie-Hellman share client compares it against ~/.ssh/known_hosts known_hosts is empty — SSH asks, and you type yes shared secret established → session ID everything from here is encrypted; the ID is unique to this connection 3 · “I intend to use this public key” search ~/.ssh/authorized_keys match · iphone-blink no matching line no real authorized_keys accepts whatever you offer 4 · “prove you hold the private half” 4 · “that key is not authorized here” sign(session ID + context) private key never leaves replays a captured signature from session 4a91… — no key held nothing to sign no challenge ever came 5 · signature over this session’s ID verified → shell 5 · a real signature, over a session with the wrong machine shell opened — on the attacker’s box 5 · a signature over some other session’s ID session ID mismatch → rejected Permission denied (publickey)
  1. Connection openA TCP socket exists and nothing more. The server does not know who you are, and you do not yet know the server is the machine you meant to reach.
  2. Connection openSame start, but this device's public key was never added to authorized_keys on the server — a new phone, or one you already revoked.
  3. Connection openAn attacker who captured a signature from one of your earlier sessions is dialling. They hold the signature, not the private key.
  4. Connection openYour very first connection to the VPS, over the open internet. Someone sits on the path between you. Your known_hosts has no entry for this machine yet, so there is nothing to compare against.
  5. 1 · Agreeing on the rulesBoth sides announce their protocol version and the ciphers, key-exchange methods and MACs they support, then pick the strongest they share. Nobody is authenticated yet.
  6. 2 · Key exchange, and the session IDThe server sends its host key and a Diffie-Hellman share; the client checks that host key against known_hosts. The shared secret they derive produces a session ID unique to this connection — remember it, it is the hinge of the whole thing.
  7. 2 · You are asked, and you say yesThe attacker presents their host key. With nothing in known_hosts, SSH cannot tell you anything useful — it just prints a fingerprint and asks. You type yes, and the attacker is now pinned as the VPS. This is the only moment the attack can be caught, and it is caught by you reading that fingerprint, not by SSH.
  8. 3 · Offering a keyThe client names the public key it wants to use. The server looks for that exact key in ~/.ssh/authorized_keys and finds the iphone-blink line.
  9. 3 · Offering a keyThe client names its public key. The server reads ~/.ssh/authorized_keys line by line and finds nothing that matches.
  10. 3 · Offering a keyThe client names its public key. The attacker's sshd has no real authorized_keys to consult and no reason to be picky — it accepts anything you offer, because it wants you to carry on.
  11. 4 · The challengeThe server asks for proof. The client signs a blob that includes the session ID using the private key. The private key itself never crosses the wire, and never leaves the device.
  12. 4 · You sign, correctlyNothing has gone wrong at this layer. Your client signs a genuine challenge with your genuine private key, and the key still never leaves the device. That is worth sitting with: every cryptographic step here is behaving exactly as designed.
  13. 4 · No challenge is issuedAn unknown key gets no challenge at all — there is nothing to prove. The client offers its next key if it has one, otherwise it is finished.
  14. 4 · Nothing to sign withThe challenge arrives, but the attacker holds no private key. All they have is a signature captured from an earlier session, so that is what they send.
  15. 5 · VerifiedThe server checks the signature with the stored public key and confirms it covers this session ID. Proof accepted, shell granted.
  16. 5 · Permission deniedPermission denied (publickey). Note what did not happen: no password prompt, no partial access. With PasswordAuthentication no this is the only possible ending for an unknown key.
  17. 5 · Replay refusedThe signature is valid — for a different session. Its session ID does not match this one, so verification fails. This is why a signature is worth nothing outside the connection that produced it.
  18. 5 · A shell, on the wrong machineYou get a prompt, and it looks normal. Now the good news and the bad news. The attacker cannot reuse your signature to log into the real VPS — it is bound to the session ID they share with you, exactly as the replay case showed. What they get instead is this session: every command you type and every file you touch. Your key survived; your session did not. Give them agent forwarding as well and even that first sentence stops being true.
The signature covers the session ID. That is the whole trick. A signature captured from one connection is worthless on another, because the session ID differs. A malicious server you connect to cannot take your proof and replay it elsewhere — which is exactly what makes agent forwarding (below) so much more dangerous than it looks. Switch the scenario above to watch each failure mode play out — including the one where every cryptographic step succeeds and you still lose.

Step 2 is the one people skip

Notice that the client verifies the server's host key before authenticating. The first time you connect, SSH has nothing to compare against and asks you to accept it — trust on first use. Almost everyone types "yes" without looking.

If someone is positioned between you and the server on that first connection, they can present their own host key, and you will have pinned an attacker. Verify once, out of band: 

# On the server, print its fingerprints
for f in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf "$f"; done

# Compare against what the client stored
ssh-keygen -lf ~/.ssh/known_hosts

On a tailnet the exposure is small, since an attacker would already need to be inside your WireGuard mesh. Do it anyway for the VPS, which you will first reach over the open internet.

Choosing a key

TypeVerdictNotes
ed25519Use this Fast, 68-byte keys, no parameter choices to get wrong, immune to the bad-randomness failures that have bitten ECDSA. Supported everywhere that matters.
ecdsa-sk / ed25519-skExcellent if you have hardware Backed by a security key; requires a physical touch. The -sk suffix means the private key cannot be exfiltrated by malware.
rsa (4096)Acceptable, legacy Only for hosts too old for ed25519. Ensure rsa-sha2-512; SHA-1 signatures are disabled in modern OpenSSH.
ecdsaAvoid NIST curves, fiddly, no advantage over ed25519.
dsaNever Removed from OpenSSH. If you find one, it is a finding.
 
ssh-keygen -t ed25519 -C "iphone-blink-2026"

Always set a passphrase. A private key file without one is a plaintext credential — anything that can read the file owns your servers. The comment field is not decoration: it is how you identify a line in authorized_keys two years from now when deciding what to revoke.

One key per device, never per server

Generate a key on each device and copy the public half to every server. Never copy a private key between devices. The point is that losing a phone means revoking one line on each server, with no effect on your other machines — and it means a private key never travels.

authorized_keys can do more than list keys

Each line accepts options that constrain what the key may do. A key used only for a backup script has no business opening an interactive shell:

~/.ssh/authorized_keys
# Full interactive access from the tailnet only
from="100.64.0.0/10" ssh-ed25519 AAAAC3Nza...  iphone-blink

# A key that can ONLY run one command, with no PTY, no forwarding
restrict,command="/usr/local/bin/backup-report" ssh-ed25519 AAAAC3Nza...  backup-runner

restrict is the modern shorthand: it disables port forwarding, agent forwarding, X11, PTY allocation and ~/.ssh/rc in one word, and — importantly — automatically includes any future restrictions added by later OpenSSH versions.

Server configuration

A hardened sshd_config. Every line earns its place:

/etc/ssh/sshd_config.d/hardening.conf
# --- authentication ---
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitEmptyPasswords no
PermitRootLogin no
AllowUsers yourname
MaxAuthTries 4

# --- reduce what a session can do ---
AllowAgentForwarding no
X11Forwarding no
PermitTunnel no
GatewayPorts no

# --- keep sessions alive across bad networks ---
ClientAliveInterval 30
ClientAliveCountMax 6
TCPKeepAlive yes

# --- see what happened ---
LogLevel VERBOSE
Why LogLevel VERBOSE is not optional

At the default INFO, you learn that a session ended. At VERBOSE you learn which key fingerprint authenticated and how the session ended. The difference between Received disconnect ... disconnected by user and Read error ... Connection reset by peer is the difference between "the server closed it cleanly" and "the client was killed" — and that single distinction resolves most mystery disconnects.

Two configuration traps

First value wins. Unlike most config formats, sshd_config uses the first occurrence of a keyword, not the last. If Include sits at the top of the file — as it does on macOS — then settings in the included drop-in override the main file, which is usually what you want. If the Include is at the bottom, your drop-in is silently ignored. Check where it sits before debugging anything else.

Match blocks swallow what follows. A Match block extends until the next Match or the end of file. Put one at the end of an included drop-in and the directives after the Include in the parent file can be captured by it. Either place Match blocks last and terminate with Match all, or avoid them in drop-ins entirely. 

# Always validate before restarting. This catches both traps.
sudo sshd -t -f /etc/ssh/sshd_config

# Print the fully resolved configuration, after all includes and overrides
sudo sshd -T -f /etc/ssh/sshd_config | sort
Keep a second session open

Before restarting sshd on a remote machine, open a second SSH session and leave it connected. If the new configuration locks you out, that existing session is your way back in. On a VPS without console access, skipping this is how a machine becomes unreachable permanently.

The SSH agent, and why forwarding is off above

The agent holds decrypted keys in memory so you type a passphrase once per session. Agent forwarding (-A) extends that agent to the remote host — which means root on that host can use your agent socket to authenticate as you, to anywhere your key is trusted, for as long as you stay connected. You will not see it happen.

How you reach the private box
Your laptop the only place your private key exists VPS (the hop) shared, exposed, not fully trusted internal 10.0.0.5 the machine you actually want ssh-agent key unlocked in memory you typed your passphrase once · the agent holds the key 1 · ssh -A vps 1 · ssh internal $SSH_AUTH_SOCK a live door to your agent no socket, no keys just a forwarded TCP stream 2 · the VPS authenticates as you over the socket, not with a key it holds 2 · sealed laptop → internal · the VPS only moves bytes it cannot read root on the VPS or anyone who takes it 3 · signs as you — anywhere your key is trusted you see nothing 3 · nothing to borrow, nothing to read your key opened a session you never opened the hop carried bytes it could not read or use
  1. Your key, in memoryYou unlocked the key once and the agent is holding it. Everything that follows is about who else gets to ask that agent to sign things.
  2. Your key, in memorySame starting point: one agent, one unlocked key, on your laptop. The difference is entirely in how you make the hop.
  3. 1 · -A opens a doorYou log into the VPS with agent forwarding on. SSH creates a socket on the VPS and points $SSH_AUTH_SOCK at it. Anything that can reach that socket can ask your agent to sign — the agent does not know or care who is asking.
  4. 1 · No door is openedYou ask for internal, and SSH quietly opens a plain TCP connection through the VPS to get there. No socket is created on the VPS. There is no agent on it, and no key material of any kind.
  5. 2 · The hop logs in as youThe VPS reaches internal and, when challenged, hands the challenge back down the socket to your agent. Your agent signs it. This is agent forwarding working correctly — that is the uncomfortable part.
  6. 2 · You log in, through the hopYour laptop runs the whole handshake with internal itself. The VPS moves encrypted bytes between two sockets and never sees inside them. Same shell, same convenience, no trust extended.
  7. 3 · Now the hop turns on youRoot on the VPS reads $SSH_AUTH_SOCK out of your process environment and starts using it. Your agent signs for them exactly as it signs for you. They can log in as you to every machine that trusts your key, for as long as your session stays open. No prompt fires and nothing appears in your terminal.
  8. 3 · The hop turns on you anywaySame compromised host, same root, same intentions. There is no socket to find, so there is nothing to use. The traffic passing through is encrypted end to end between your laptop and internal, so there is nothing to read either.
  9. 4 · The costAgent forwarding converts "I logged into a machine" into "I lent that machine my identity". The blast radius is not the VPS — it is everywhere your key is trusted. If you ever must use -A, scope it to one host in ~/.ssh/config and prefer a hardware key that requires a physical touch per signature.
  10. 4 · The pointProxyJump gives you the same reachability with none of the delegation. This is why AllowAgentForwarding no is in the hardened config above: not because forwarding is broken, but because it does exactly what it says and that is more than you meant to offer.
Forwarding lends your identity; jumping only borrows a route. The danger in -A is not a bug — the agent is doing precisely what it was asked to do, for whoever asks. Notice that step 3 is identical in both scenarios: the hop is compromised either way. All that changes is whether there was anything on it worth taking.

Use ProxyJump instead. It authenticates each hop from your local machine, and the intermediate host never gets access to your keys:

~/.ssh/config
Host vps
  HostName vps.tailnet-name.ts.net
  User yourname
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes

# Reach a private box by hopping through the VPS — keys stay local
Host internal
  HostName 10.0.0.5
  User yourname
  ProxyJump vps

On macOS, UseKeychain yes plus AddKeysToAgent yes stores the passphrase in the login keychain so you unlock once per boot rather than once per session.

macOS has an extra gate

This surprises people, and it is specific to macOS. Apple's SSH runs its PAM stack, and that stack contains a service access control list: 

$ cat /etc/pam.d/sshd
account  required  pam_sacl.so  sacl_service=ssh
session  required  pam_launchd.so
...

pam_sacl enforces membership of the com.apple.access_ssh group — the list behind "allow access for these users" in the Remote Login settings. Setting UsePAM no skips the entire stack, which means that access control is not enforced at all. If you run a custom sshd on macOS, keep UsePAM yes and confirm your account is a member first: 

dseditgroup -o checkmember -m "$(whoami)" com.apple.access_ssh

Checklist

  • An ed25519 key exists on each client device, each with its own passphrase.
  • No private key has ever been copied between devices.
  • Every authorized_keys entry has a comment identifying the device.
  • PasswordAuthentication no and PermitRootLogin no on every server.
  • AllowUsers names only the accounts that should ever log in.
  • Agent forwarding is disabled; ProxyJump is used for multi-hop.
  • LogLevel VERBOSE is set and you know where the log lands.
  • sshd -t passes, and you verified with sshd -T that overrides landed.
  • Host key fingerprints verified out of band on first connection.

The commands in this guide change firewall and login settings, and can lock you out of a machine. Practise on something disposable first. Everything here is provided as is, with no warranty — you accept the risk of running it. Read the disclaimer.