Chapter 12 · Operate

Troubleshooting

Almost every failure here is one of five things, and they stack. Test them in order and you will find the real cause in about ninety seconds instead of changing sshd settings at random for an hour.

The ladder

Work upward. Each rung assumes the ones below it pass — there is no point debugging authentication if the packet never arrives.

Break one rung at a time, on purpose

The sandbox runs this exact ladder, and every probe it reports names the rung that decided the outcome. Break one thing there — drop a link, delete a grant, bind sshd to the tailnet address — and check that the rung it reports is the one you expected. Being wrong about that in a simulator is free; being wrong about it at 11pm is not.

Symptom
TEST WHAT COMES BACK 5 · Does the session hold? ssh host 'sleep 15; echo ok' read journalctl or /var/log for the reason 4 · Does authentication succeed? ssh -v host look for “Offering public key” / “Accepted” 3 · Is the port open? nc -z -w 3 <tailnet-ip> 22 test from the tailnet address, not localhost 2 · Is there a path? tailscale ping <device> note whether it says DERP or a direct IP 1 · Are both on the tailnet? tailscale status check both ends, not just yours start at the bottom and climb — each rung assumes the one below it passed the symptom you started from Permission denied (publickey). nothing above tells you which rung broke Connection refused something answered, and said no Operation timed out nothing answered at all client_loop: Broken pipe it worked, then stopped working pass · both ends up macbook 100.117.202.65 · active iphone 100.83.14.9 · active pass · direct path pong from 100.117.202.65 in 14ms no DERP relay in the middle stop · this is your answer no reply. Peer asleep, offline, or an ACL is dropping the packets. pass · port answers nc succeeded against the tailnet address — not localhost stop · this is your answer sshd down, bound to another address, wrong port, or a firewall rule. not run rung 2 already gave you the answer. pass · key accepted Offering public key … Accepted you land in a shell. stop · this is your answer wrong key, loose file permissions, AllowUsers, or macOS SACL missing. not run where most people start — and it would fail here for the wrong reason. stop · this is your answer client killed (iOS suspend), or a daemon restart taking its children. not run you already have your answer below.
  1. Permission denied (publickey)The daemon is healthy — it completed a handshake and made a decision. That feels like an authentication problem, which is exactly why people jump straight to rung 4. Climb from the bottom anyway; it costs about ninety seconds.
  2. Connection refusedSomething answered and said no. The host is reachable and nothing is listening on that port — but you do not know that yet, so start at rung 1.
  3. Connection timed outNothing answered at all. This is a network-layer symptom, not an SSH one: a dropped packet, an ACL, or a peer that is asleep.
  4. client_loop: send disconnect: Broken pipeIt worked, and then it stopped. The connection died underneath you and the client only noticed on its next write. Everything below is still fine, which is what makes this one confusing.
  5. 1 · Are both on the tailnet?tailscale status on both ends. Logged out, an expired key, an unapproved device or Tailscale simply not running all show up here, and nothing above this rung can work without it. In all four of these symptoms it passes — which is fine. It is the cheapest test on the ladder and it takes three seconds.
  6. 2 · Is there a path?tailscale ping answers directly and fast, so the two machines can genuinely reach each other. Note that it says direct rather than DERP — a relay would still work, just slower.
  7. 2 · There is no path — stop hereNo reply at all. The peer is asleep or offline, or an ACL is eating the packets. Persistent DERP instead of silence would be a different answer: slow, but not broken. Either way, you are done climbing. Nothing above this rung can pass while the packets never arrive.
  8. 3 · Is the port open?nc against the tailnet address succeeds, so sshd is up and bound where you expect. Testing localhost here would prove nothing — a tailnet-only daemon refuses that on purpose.
  9. 3 · The port is shut — stop hereNothing is listening on that port. sshd is down, bound to a different address, on a different port, or an ACL or firewall rule is dropping it. This is your answer. Run ss -tulpn or netstat -an -p tcp and compare what is actually bound against what you dialled.
  10. 3 · Not runRung 2 already told you the packets never arrive. Testing the port from here would time out too, and you would learn nothing new from it.
  11. 4 · Does authentication succeed?ssh -v shows the key being offered and accepted, and you land in a shell. So the credentials, the permissions and the AllowUsers list are all fine.
  12. 4 · Authentication fails — stop hereOnly now does the message mean what it says. Rerun with ssh -v and read which keys were offered; the causes are listed by likelihood further down this page.
  13. 4 · Not run — and this is the pointThis is the rung people reach for first, because "denied" and "refused" both sound like permission words. It would have failed here too, for a reason that has nothing to do with authentication, and you would have spent an hour editing sshd_config.
  14. 5 · The session does not hold — stop hereEverything below passed, so this is not a server problem to tune. The server log tells you which side caused it: a TCP reset means the client process was killed — iOS suspended the app — and no ClientAlive setting will help. The durable fix is Mosh plus tmux, so a dead client stops being an outage.
  15. 5 · Not runYou already have your answer further down the ladder. Fix that rung, then re-run from the bottom — a second cause hiding above the first is common, and the ladder is cheap to climb twice.
Resist starting at rung 4. "Permission denied" feels like an auth problem, so that is where people begin — but a wrong port or a stale ACL produces confusing symptoms further up that vanish the moment you check the rung below. Switch the symptom above and watch where the climb actually stops.

Symptoms and what they mean

Connection refused

Something answered and said no. The host is reachable; nothing is listening on that port. Check the port number first, then whether sshd is bound where you think: 

netstat -an -p tcp | grep LISTEN | grep 22     # macOS
sudo ss -tulpn | grep :22                      # Linux

If it shows 100.x.y.z.22, the daemon is tailnet-only and you must connect to that address — localhost will be refused, which is correct and by design.

Check the address family, not just the address

The nastiest version of this: sshd is bound to the tailnet IPv4 address only, MagicDNS also publishes an IPv6 record, and your client tries IPv6 first. Everything above passes — tailscale status is healthy, nc against the 100.x address succeeds — and ssh macbook still says refused. Compare the two directly: 

ssh -4 macbook 'echo v4 ok'
ssh -6 macbook 'echo v6 ok'

If v4 works and v6 does not, add the second ListenAddress line (chapter 05) rather than papering over it on the client.

Connection timed out

Nothing answered at all. This is a network-layer problem, not SSH: a dropped packet, an ACL, a firewall, or a peer that is asleep. Drop to rungs 1 and 2.

Permission denied (publickey)

The good news is the daemon is healthy — it completed a handshake and made a decision. Run with -v and read which keys were offered: 

ssh -v -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 yourname@host

Ordered by how often each is the actual cause:

  • The public key is not in authorized_keys, or landed in the wrong user's home directory.
  • Permissions. ~ must not be group/world-writable; ~/.ssh should be 700 and authorized_keys 600.
  • AllowUsers does not include your account.
  • The client offered a different key first and hit MaxAuthTries. IdentitiesOnly=yes settles it.
  • On macOS: UsePAM yes is set but you are not in com.apple.access_ssh.
chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys
dseditgroup -o checkmember -m "$(whoami)" com.apple.access_ssh   # macOS only

client_loop: send disconnect: Broken pipe

The connection died underneath you and the client noticed on its next write. The server log tells you which side caused it, and this single distinction saves hours:

Server log saysMeaningAction
Received disconnect ... disconnected by user Clean shutdown. The client asked to close. Normal. Look elsewhere.
Read error ... Connection reset by peer A TCP reset. The client process was killed — it never got to close the socket. Client-side. iOS suspended or terminated the app. No server change will help.
Timeout, client not responding Keepalives went unanswered for the full window. Network path died. Raise ClientAliveCountMax, and use Mosh.
Nothing at all The daemon itself was restarted, taking its children with it. Check the supervisor — see the respawn loop below.
 
# macOS, if the plist has -e
sudo tail -40 /var/log/com.local.sshd.err

# Linux
sudo journalctl -u ssh -n 40 --no-pager
The durable fix

Mosh removes the TCP connection that keeps breaking. tmux means that even when the client dies outright, nothing is lost. Together they turn this from an outage into a non-event — which is better than any amount of keepalive tuning.

Sessions appear to die instantly

Check your evidence before believing this

last prints durations as HH:MM, so every session shorter than one minute displays as (00:00). A perfectly healthy 30-second session looks identical to one that died on arrival. Confirm with a timed test before diagnosing anything: 

ssh host 'echo start=$(date +%T); sleep 15; echo end=$(date +%T)'

If that genuinely fails, the usual cause is the daemon being restarted underneath the session — a supervisor killing the process group takes every child session with it.

A respawn loop macOS

sudo launchctl print system/com.local.sshd | grep -E 'runs|last exit code'

runs = 1 is healthy. runs in the dozens or hundreds with last exit code = 255 means launchd is restarting the job continuously. Two causes:

  • Missing -D. sshd forks, the tracked parent exits, launchd restarts it, and the replacement finds the port already held by the orphan. Add -D.
  • The bind address does not exist yet. With ListenAddress set to a Tailscale IP, sshd cannot bind until Tailscale is up. Harmless with ThrottleInterval — it self-heals — but it is why SSH is briefly unavailable after a reboot.

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED

Take this seriously rather than reflexively deleting the line. Legitimate causes: the server was rebuilt, or its host keys were regenerated. Illegitimate cause: someone is between you and it. 

# Verify out of band — on the server itself, via a path you trust
for f in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf "$f"; done

# Only once the fingerprint matches
ssh-keygen -R '[hostname]:port'

Everything is slow

tailscale ping <device>

A direct path is typically 1–50 ms. A DERP relay adds real latency. Persistent relay means UDP hole punching is failing — common on restrictive corporate or hotel networks. It is not a security problem, and Mosh's local echo makes even a relayed session feel responsive because keystrokes no longer wait for a round trip.

tmux: command not found over SSH

A PATH problem, not a missing binary. ssh host 'cmd' runs a non-login, non-interactive shell that reads only ~/.zshenv — while Homebrew's PATH is set in ~/.zprofile, which only login shells read: 

ssh macbook -t '/opt/homebrew/bin/tmux new -A -s phone'   # full path
ssh macbook -t 'zsh -lc "tmux new -A -s phone"'          # or force a login shell

A config change had no effect

Remember that sshd_config takes the first value for each keyword, not the last. Where the Include line sits decides whether your drop-in wins. Ask the daemon what it actually resolved: 

sudo sshd -T -f /etc/ssh/sshd_config | grep -i <setting>

Also confirm you reloaded the right service, and that the running process is using the config file you edited: 

ps -eo pid,command | grep '[s]shd'

Emergency access

If you are locked out of the VPS
  1. Use the provider's web console — this is why you tested it in chapter 08.
  2. Failing that, boot into recovery/rescue mode and mount the disk.
  3. Restore the previous config: cp /etc/ssh/sshd_config.d/99-hardening.conf.bak ...
  4. Reopen SSH temporarily: ufw allow OpenSSH.
  5. Fix, verify from a second terminal, then close it again.

The habit that prevents all of this: keep a second session open while changing SSH, and always back up a config before replacing it.

Command reference

QuestionCommand
Who is on the tailnet?tailscale status
Direct or relayed?tailscale ping <device>
My tailnet address?tailscale ip -4
What is listening?ss -tulpn · netstat -an -p tcp
Is a port reachable?nc -z -w 3 <host> <port> — -w is portable; -G is macOS only
Reachable over IPv6 specifically?ssh -6 <host> true · compare with ssh -4
Why did auth fail?ssh -v (add more vs for detail)
Effective sshd config?sudo sshd -T -f /etc/ssh/sshd_config
Is the config valid?sudo sshd -t
Server-side session log?journalctl -u ssh -f · tail -f /var/log/*sshd.err
Which keys are trusted?ssh-keygen -lf ~/.ssh/authorized_keys
Host key fingerprints?ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
Is the daemon flapping?launchctl print system/<label> · systemctl status ssh
Any tmux sessions alive?tmux ls

The commands in this guide change firewall and login settings, and can lock you out of a machine. Practise on something disposable first. Everything here is provided as is, with no warranty — you accept the risk of running it. Read the disclaimer.