Chapter 13 · Prove it All machines

The lab

Every other chapter asks you to do something irreversible to a machine you rely on: close the only open port, disable the only working login, hand your one key to a device you have not tested yet. This chapter builds a throwaway copy of the whole topology so you can make those moves somewhere a mistake costs you a limactl delete instead of a support ticket.

The lab is optional, and it is not a prerequisite

Nothing in chapters 01–12 depends on this one. If you are building for yourself on machines you can walk over to and plug a keyboard into, skip it. The lab earns its keep when you are about to do this to a box you can only reach over the network — a VPS, a machine in another building, a laptop you posted to someone.

If six gigabytes of VMs is more than you want to spend today

Chapter 14 is the same topology as a working model in your browser — no VMs, no downloads, nothing to tear down. It runs the identical five-rung ladder, it has a hostile machine you can add to the network, and every switch you flip there prints the real command for the rig below. It cannot surprise you the way real kernels can, which is exactly why this chapter still exists. But it is the cheaper place to make your first mistakes, and it costs nothing to try before committing to the build.

And if you want the real kernels without the afternoon, there is a middle path: cd lab && docker compose up brings the same topology up as containers, with an attacker you can switch on.

A green lab is evidence, not a guarantee

The whole point of building this is to make mistakes cheaply, and it works — but a rig that passes is telling you about the rig. Your VPS has a different provider's networking, your laptop a different distribution, your phone a carrier you do not control. Carry the confidence across, not the assumption.

When you do move to the real machines: keep a proven second way in before you change anything, follow chapter 11's build order rather than your memory of it, and read each command instead of pasting a block. Everything in this guide is provided as is with no warranty and you accept the risk of using it — see the disclaimer and the CC BY-SA 4.0 licence.

The trap: a lab that proves the boring half

The obvious lab is three virtual machines on one laptop. It is also nearly useless for this particular guide, and the reason is worth understanding before you build anything.

Three VMs on one host share a network. They can already see each other. Every connection between them is direct, on the first try, forever. So a lab like that will happily confirm your sshd_config is valid and your firewall rules load — and it will tell you nothing about the two chapters that are hardest to trust on faith: chapter 01, which is entirely about what happens when two machines cannot see each other, and chapter 03, which is entirely about what happens when the link dies mid-sentence.

The lab is not a pile of machines. It is a network shape. Getting the shape wrong produces a rig that passes everything and proves half of it.

How the lab network is wired
THE RIG, AND HOW ITS NETWORK IS WIRED WHAT THIS RUN ACTUALLY PROVES lab-vps public box · ch 08 lab-ubuntu the laptop · ch 09 lab-roam the phone's stand-in THE TAILNET coordination server DERP relay THE LINK, DURING THIS RUN WHAT THE RIG COSTS YOU one L2 segment its own NAT its own NAT its own NAT direct, always direct, punched relayed via DERP a virtual switch that has never once dropped a packet there is nothing here to break clean, but the two ends are strangers to each other eth0 down for 20s — then up, on a new address 20% loss, 180ms jitter, UDP to :41641 dropped eth0 down for 20s, under loss, on a relayed path 3 VMs · ~6 GB disk · one flag: --network=vzNAT the cheapest rig, and the one that flatters you most 3 VMs · ~6 GB disk · Lima's default networking, no flags free, and already more honest than the shared segment the same 3 VMs · plus tc netem and two nft rules ten more minutes, and it is the only run that proves ch 03 01 · NAT traversal and DERP 02 · sshd config and keys 03 · Mosh across a link that dies 08 / 09 · hardening, UFW, Docker 11 · the build order, run for real 12 · the diagnostic ladder SO WHAT WAS THE RUN WORTH never exercised proven nothing to survive proven proven two rungs of five Everything passed, and the two chapters you most wanted confidence in were never put under any load at all. punching proven proven a clean handover only proven proven four rungs of five The default Lima rig, with no extra work — and it already makes two strangers find each other through two NATs. direct and relayed proven proven, the hard way proven proven all five rungs The only run where Mosh had something real to survive — and the only one that ever touched the relay path. the machines are identical in all three runs · everything that changes is the shape of the network between them
  1. 0 · Three machines on one wireOne --network=vzNAT flag puts all three VMs on a shared segment with real addresses, and they can ping each other before Tailscale is even installed. This is the rig most people build, because it is the one that looks most like a network diagram. Hold on to that: they can already reach each other.
  2. 0 · Three machines that cannot see each otherDrop the flag and Lima gives every VM its own user-mode NAT, with no route between them — which sounds like a defect and is actually the feature. Three machines behind three separate routers is precisely the situation chapter 01 exists to solve, and you got it by typing less.
  3. 0 · The same three machines againIdentical to the NAT run at this point. What makes this one different arrives at step 4, and it is two nft rules and a tc invocation rather than any change to the machines themselves.
  4. 1 · They join the tailnettailscale up on each, then approve them in the admin console if approval is on. Use a throwaway tailnet on a separate account, never the real one — the whole point is to write ACLs that are wrong on purpose, and you do not want a bad policy landing on the machines you actually use.
  5. 1 · They join the tailnetSame three commands. Note what has to happen now that did not have to happen on the shared segment: each node reports its NAT mapping to the coordination server, learns the others' candidate addresses, and starts probing. None of that machinery ran in the first scenario.
  6. 1 · They join the tailnetSame again. Join before you make the network hostile — a node that has never once talked to the coordination server has a different failure mode from one that has, and it is the second one you are trying to test.
  7. 2 · Ask how they actually connectedtailscale status is the honesty check, and here it reads direct 192.168.105.13:41641 — a private address on the shared wire. WireGuard came up on the first packet because the two ends were already neighbours. Nothing was traversed. The DERP relay in the diagram has not carried a single byte and never will.
  8. 2 · Ask how they actually connectedNow direct means something: the two nodes exchanged candidates, both NATs held a mapping open, and the path came up without a relay. Run tailscale netcheck alongside it — it prints whether UDP works, which DERP region is nearest, and whether your mapping varies by destination, which is the tell for the hard kind of NAT.
  9. 2 · It falls back, and that is the pointrelay "fra" means direct failed and the session is riding DERP over TCP 443. This is the path that carries you on a hotel network or a mobile carrier that blocks UDP, it is meaningfully slower, and until you have seen it in a lab you do not know whether your setup survives it. One nft rule buys you that knowledge.
  10. 3 · Run the chapters against itNow work through 08 and 09 exactly as written: patch, non-root user, key login, sshd -t, UFW default-deny, the Docker publish trap. Every one of these passes here, and every one of them would have passed on a single VM with no network at all. That is what makes this scenario comfortable and misleading.
  11. 3 · Do the irreversible thing on purposeThe same hardening, and then the move the whole guide builds towards: delete the public :22 rule while your only remaining way in is the tailnet. On a real VPS that is the step people postpone for weeks. Here it costs nothing to get wrong, so get it wrong once deliberately — close it before writing the ACL and watch chapter 11's ordering lesson happen to you.
  12. 3 · Do it over the relayed pathSame removal, except your remaining route is DERP under 20% packet loss. If closing port 22 is going to feel dangerous anywhere, it is here — which is exactly why this is the run worth doing before you touch anything real.
  13. 4 · There is nothing to breakYou can take an interface down, but both ends come back on the same address, on the same switch, in the same second. Mosh has nothing to recover from and tmux has nothing to reattach to. Chapters 03 and 04 look green and were never asked a question.
  14. 4 · Take the link awayip link set eth0 down, wait, bring it up. The VM returns on a different mapping, ssh reports a dead socket, and Mosh reconnects because the session is keyed to a secret rather than to an address. This is a clean handover — a train entering a tunnel, not a bad café.
  15. 4 · Take it away badlyThe same outage, but under loss and jitter and on a relayed path. Run ssh and mosh side by side into the same box and watch one die and the other resume. That single side-by-side is the most persuasive thing in this entire guide, and you cannot produce it on a network that behaves.
  16. 5 · A green run that proves the boring halfConfig validity, firewall rules, build order — all genuinely confirmed. NAT traversal, relay fallback and session survival — all untouched. If you stop here, be honest in your notes about which half you tested, because the untested half is the half that fails at 11pm on someone else's Wi-Fi.
  17. 5 · The default rig, and it is already goodNo flags, no extra tooling: three Lima VMs behind three NATs, finding each other the way your real machines will have to. This is the sensible place to stop for most people, and it is what the build below gives you.
  18. 5 · The only run that tests the interesting halfTen extra minutes of tc and nft moves you from "my configuration is valid" to "this holds when the network is actively against me". Those are very different claims, and only one of them is worth the effort of building a lab at all.
The machines never change. Only the network between them does — and that is the entire difference between a lab that flatters you and a lab that tells you something. Switch from One shared segment to NAT + a hostile link and watch the top and third rows on the right go from red to green: those are chapters 01 and 03, the two you would most like to be sure of, and they are the two a comfortable lab silently skips.

What a lab can, and cannot, prove

Before building anything, be clear about where the boundary is. Three buckets, and the third one matters as much as the first two.

BucketChaptersVerdict
Configuration & policy 02, 04, 08, 09, 11 Proven exactly. The lab runs the same daemons with the same config files.
Needs a shaped network 01, 03, 12 Proven only once you add NAT, loss and link failure. A flat lab silently skips these.
Real hardware only 06, 07, and the FileVault and Secure Enclave parts of 05 Not provable here at all. Do these on the device.
The third bucket is not a gap you can close

A VM has no Secure Enclave, so enclave-bound keys (chapter 06) and hardware-backed keystores (chapter 07) have nothing to bind to. Nor can a lab reproduce carrier-grade NAT, a vendor's battery killer, or iOS suspending an app in your pocket. Those chapters are verified on the phone in your hand, and no amount of virtualisation changes that. Better to write that down than to let a green lab imply coverage it does not have.

The fast path: docker compose up

Everything below this section builds the rig by hand, and that is worth doing once: you learn where each piece sits by putting it there. But it is ten commands, three VM downloads and an afternoon, and there is a version of the same topology that takes one command. 

cd lab && docker compose up -d
open http://localhost:8099

Docker is the only thing you need installed. The control server, the coordination server and the certificate authority are all services in the same stack, so there is no Go, no Python and no npm on your machine. What comes up is four containers on four isolated segments, each behind its own NAT router, running real tailscaled on real TUN devices and putting real WireGuard frames on real wires — plus a page where every switch is a command that runs and every probe reports the rung that decided it.

The Lima rig, belowlab/
SetupTen commands and three VM imagesOne command
CoordinationA real throwaway tailnetHeadscale, in the stack
Proves chapter 01 as writtenYesPartly — a compatible reimplementation
Offline & repeatableNoYes
An attacker on the wireYou build oneevil-box, behind a profile
Teaches you the moving partsYes, that is the pointNo — it hides them

The two are not rivals. Build the rig below once to understand the shape of the thing; use lab/ when you want to re-run the whole battery against a change, or point something hostile at it, or produce the same evidence twice on a Tuesday. Its README covers the services, the ports, what each attack proves, and — the part worth reading first — the three places where the containers and the chapter 14 sandbox disagree with each other.

It uses Headscale, and this chapter tells you not to

Two paragraphs below, this chapter argues for a real throwaway tailnet over Headscale, and it still means it: testing Tailscale's NAT traversal and ACL semantics against a different implementation is testing a different product. lab/ diverges deliberately, because a rig you are going to point an attacker at should not need an account, an internet connection or anybody's production infrastructure — and because make reset should be able to destroy the entire control plane, which is not a thing you can do to somebody else's.

So: use the throwaway tailnet when you are checking whether Tailscale behaves as chapter 01 describes. Use lab/ when you are checking whether your configuration holds. Those are different questions and it is worth knowing which one you are asking.

The attacker never starts on its own

evil-box sits behind a compose profile and does not exist until you run docker compose --profile attack up -d. When it does exist, it attacks containers you started, on hardware you own, on bridges that reach nothing else. The control server holds the Docker socket, has no authentication, is published to 127.0.0.1 only, and refuses to start if that is changed — which is a guardrail worth leaving in place rather than a formality.

Building the rig

Three Linux VMs, one per role in the guide. This uses Lima, which wraps Apple's Virtualization framework on Apple silicon and QEMU elsewhere. Nothing here is Mac-specific in spirit — the same three machines work under multipass, vagrant or plain virt-install. 

brew install lima            # macOS
# or: sudo apt install -y lima   on Ubuntu 24.04+

# Three machines, each behind its own user-mode NAT.
# The absence of a --network flag is the important part.
limactl start --name=lab-vps    --cpus=2 --memory=2 template://ubuntu-24.04
limactl start --name=lab-ubuntu --cpus=2 --memory=2 template://ubuntu-24.04
limactl start --name=lab-roam   --cpus=2 --memory=2 template://ubuntu-24.04

limactl list

Three roles, mapped to the chapters they stand in for:

MachinePlaysChapter
lab-vpsThe always-on box with a public address08
lab-ubuntuThe laptop that joins the same tailnet09
lab-roamThe roaming client — the phone's stand-in03, 04

Drop into any of them with limactl shell lab-vps. From there, every command in chapters 08 and 09 works verbatim.

Why no shared network

Lima's default user-mode networking gives each VM its own NAT and no route to its siblings. Reach for --network=vzNAT or --network=lima:shared and you undo that, putting everything on one segment — which is the flat lab from the simulation above. Leave the flag off. The isolation is the feature, and it is free.

The control plane

Tailscale needs a coordination server, and the lab cannot invent one. Two options, and they prove different things.

A throwaway tailnetHeadscale in the lab
What it isReal Tailscale, free plan, on a separate emailAn open-source reimplementation of the control server
Coordination & DERPTailscale's own, exactly as in productionYours, with a DERP you configure
Offline / repeatableNo — needs internetYes
Proves chapter 01Yes, as writtenPartly — it is a different implementation
Use a throwaway tailnet

Chapter 01 describes Tailscale's actual behaviour — its NAT traversal, its DERP mesh, its ACL semantics, tailnet lock. Testing that against Headscale is testing a different product that happens to speak a compatible protocol. Sign up with a second email, and keep the lab tailnet entirely separate from the real one so a deliberately broken ACL can never reach a machine you care about.

Making the network hostile

This is the part that turns a rig into a test. All three are run inside a lab VM, and all three are reversible.

1 · Loss and latency

# 20% loss, 180ms delay with 40ms of jitter — a bad café
sudo tc qdisc add dev eth0 root netem loss 20% delay 180ms 40ms

# put it back
sudo tc qdisc del dev eth0 root

2 · Force the relay path

Blocking WireGuard's UDP port makes direct paths fail, so Tailscale falls back to DERP over TCP 443 — the same thing that happens on a network that blocks UDP outbound. 

sudo nft add table inet lab
sudo nft add chain inet lab out { type filter hook output priority 0 \; }
sudo nft add rule inet lab out udp dport 41641 drop
sudo nft add rule inet lab out udp sport 41641 drop

# confirm it took effect — this should now say relay, not direct
tailscale status
tailscale netcheck

# put it back
sudo nft delete table inet lab

3 · Take the link away

# the train entering a tunnel
sudo ip link set eth0 down; sleep 20; sudo ip link set eth0 up
Run that one from the console, not over SSH

Taking eth0 down kills the session you typed it in. Use limactl shell from the host, or wrap it so it restores itself even if your session dies: sudo sh -c 'ip link set eth0 down; sleep 20; ip link set eth0 up' &. The lab is where you learn that reflex cheaply.

Prove it: the tests worth running

A lab is only worth building if it produces verdicts. Each row here is a command and the output that means pass — run them in order, because several depend on the one above.

ChapterRunPass looks like
01 tailscale netcheck UDP: true, a nearest DERP region, and MappingVariesByDestIP reported either way — you now know which NAT you are behind
01 tailscale status direct with the UDP block off; relay "…" with it on. Both paths seen at least once
01 Write a default-deny ACL with no SSH grant, then ssh Connection refused. The policy is doing something — which you cannot know from a permissive default
02 sudo sshd -t, then ssh -o PasswordAuthentication=yes Config valid; the password attempt is rejected without a prompt
03 ssh and mosh into lab-vps side by side, then drop eth0 for 20s ssh reports a broken pipe; mosh redraws and carries on. The whole chapter, in one screenshot
04 Start work in tmux, kill the client hard, reconnect with tmux new -A -s lab Attaches to the running session; the job never noticed
08 docker run -d -p 8080:80 nginx, then scan from lab-ubuntu Port 8080 is reachable despite UFW denying it — the Docker bypass, seen with your own eyes
08 Delete the public :22 rule while connected over the tailnet You are still logged in, and a fresh public connection times out
11 Deliberately run the build order wrong: close :22 at step 2, before the ACL You lock yourself out, and recover via limactl shell — the console stand-in
12 Break one rung at a time, then walk the ladder Each fault is found at the rung that owns it, not two rungs later
The two tests that justify the whole chapter

The Docker-bypass one and the locked-out build-order one. Both are things you have read and believed; neither becomes real until you have watched it happen. Everything else in this table you could reasonably take on trust.

Adding a Mac to the lab

The rig above has no macOS node, so chapter 05 sits outside it. If your host is an Apple silicon Mac you can add one, using Apple's own Virtualization framework via Tart: 

brew install openai/tools/tart
tart clone ghcr.io/cirruslabs/macos-tahoe-base:latest lab-mac
tart run lab-mac

That is roughly a 25 GB download for a VM that behaves like a real ARM Mac: launchd, pfctl, sshd, Tailscale and lsof -i all work as chapter 05 describes, and cloning the VM gives you an instant snapshot to reset to.

Three things it will not tell you
  • FileVault. A VM has no Secure Enclave, so the encryption keys are not protected the way they are on your actual MacBook. The commands work; the guarantee is weaker.
  • Secure Enclave keys. Nothing to bind to, so the key-custody argument in chapter 10 cannot be exercised here.
  • Two at a time. Apple's framework permits at most two macOS guests running at once on a host, so a Mac node is a singleton in this lab.
Not on Apple silicon?

Then skip this section rather than reaching for a container. The projects that run macOS inside Docker need an x86 Linux host with /dev/kvm and emulate an Intel Mac, so they cannot help an ARM host — and Docker Desktop does not expose KVM to containers on any platform. The three-VM rig above is unaffected and still proves eight of the twelve chapters.

Tearing it down

limactl stop lab-vps lab-ubuntu lab-roam
limactl delete lab-vps lab-ubuntu lab-roam

tart delete lab-mac          # if you built one

Then remove the lab machines from the tailnet admin console, and revoke the auth keys you used. A stale lab node in an ACL group is exactly the kind of thing chapter 10 warns about — the lab is throwaway, but its tailnet identity is not throwaway until you throw it away.

Checklist

  • Either three VMs running, each on its own NAT — or lab/ up with docker compose.
  • A throwaway tailnet on a separate account, never the real one.
  • All three nodes joined and visible in tailscale status.
  • tailscale netcheck run and its output read, not just executed.
  • A direct path observed at least once.
  • A relay path observed at least once, with the UDP block in place.
  • A default-deny ACL written and proven to actually deny.
  • Chapters 08 and 09 run end to end against lab-vps and lab-ubuntu.
  • Public :22 closed over the tailnet, and survived.
  • The Docker UFW bypass reproduced and seen from another machine.
  • ssh and mosh compared side by side across a 20-second outage.
  • tmux reattached after a hard client kill.
  • The build order deliberately run wrong once, and recovered from.
  • Lab nodes removed from the tailnet and auth keys revoked at teardown.
  • If you used lab/: the eleven-check audit run against hardened, and its one failure understood rather than ignored.

The commands in this guide change firewall and login settings, and can lock you out of a machine. Practise on something disposable first. Everything here is provided as is, with no warranty — you accept the risk of running it. Read the disclaimer.