Secure your servers by blocking off sshd

Wednesday, 30 November 2022

Creating a computer in the cloud is easy, securing it is not. The general approach is to block all inbound access to all ports except the ones that your webserver and sshd are running on. This approach can get you quite far, but you’re still going to have lots of requests from bad actors trying to find a way in. Tools exist to help you by monitoring auth logs and applying filters, but they’re reactive, and have limited utility as a preventative measure.

Instead, let’s just block off the sshd port too. But how do we log in if sshd isn’t visible? Enter Tailscale. Here’s the general idea: install the Tailscale daemon on the machine in question. This lets you access it directly (and securely) from all other machines that you’ve installed Tailscale on.

Tailscale Setup

Quick note: I don’t work for Tailscale and have no real affiliation. Just a happy user.

Start by installing and logging into Tailscale on your main dev machine. You should have a Tailscale icon in your menubar, and it should say you’re Connected.

Once that’s done, let’s do the same thing on the server:

# 1. Install Tailscale
#    Granular instructions here: https://tailscale.com/kb/1031/install-linux/
$ curl -fsSL https://tailscale.com/install.sh | sh

# 2. Start Tailscale
$ sudo tailscale up

# 3. Look for other machines in your Tailscale network
#    (You should see your main dev machine here)
$ tailscale status
100.100.100.100   formulate.dev        tim@        linux   -
100.100.100.101   nudges.fyi           tim@        linux   -
100.100.100.102   m1-mba               tim@        macOS   active; direct 82.82.82.82:41641, tx 11464 rx 12760
100.100.100.103   tims-iphone          tim@        iOS     offline

# 4. Find the server's Tailscale IP address
$ tailscale ip -4
100.100.100.100

Now from your dev machine, try SSH-ing to the server. Make sure Tailscale is running!

# Use the Tailscale IP
$ ssh 100.100.100.100 uptime
12:46:44 up 28 days, 15:57,  2 users,  load average: 0.15, 0.05, 0.01

# Or hostname (this requires MagicDNS: https://tailscale.com/kb/1081/magicdns)
$ ssh formulate.dev 'uname -a'
Linux nudges 5.10.0-11-amd64 #1 SMP Debian 5.10.92-1 (2022-01-18) x86_64 GNU/Linux

Block Access

Now for the fun part! sshd is accessible via the Tailscale IP, so you can disallow access via the regular public IP without losing access to the server. To start, let’s install a firewall:

$ sudo apt update
$ sudo apt install ufw

We can’t enable ufw yet because it defaults to disallowing all incoming connections, which would lock us out. Let’s first allow all connections via the Tailscale interface:

$ sudo ufw allow in on tailscale0

And then turn on the firewall:

$ sudo ufw enable

At this point you should be able to SSH in with your server’s Tailscale IP address but not its public IP address. Problem solved!

Key Expiry

There’s one other thing you’ll have to do to avoid being locked out of your server. Tailscale periodically expires client keys and requires manual reauthentication. If this happens on your server you’ll be locked out, because you can’t get in to reauth.

Make sure you disable key expiry for all servers you’ve set up this way.

Poking Holes

Finally, poke holes in your firewall for legitimate incoming traffic. For example, you may want to allow traffic on port 443 solely from Cloudflare IP addresses:

curl https://www.cloudflare.com/ips-v4 > /tmp/ipv4
curl https://www.cloudflare.com/ips-v6 > /tmp/ipv6

for cfip in `cat /tmp/ipv4`; do ufw allow proto tcp from "$cfip" to any port 443; done
for cfip in `cat /tmp/ipv6`; do ufw allow proto tcp from "$cfip" to any port 443; done

Finally, your firewall will look something like this:

$ ufw status
Status: active

To                         Action      From
--                         ------      ----
Anywhere on tailscale0      ALLOW       Anywhere
443/tcp                     ALLOW       173.245.48.0/20
443/tcp                     ALLOW       103.21.244.0/22
443/tcp                     ALLOW       103.22.200.0/22
443/tcp                     ALLOW       103.31.4.0/22
443/tcp                     ALLOW       141.101.64.0/18
443/tcp                     ALLOW       108.162.192.0/18
443/tcp                     ALLOW       190.93.240.0/20
443/tcp                     ALLOW       188.114.96.0/20
443/tcp                     ALLOW       197.234.240.0/22
443/tcp                     ALLOW       198.41.128.0/17
443/tcp                     ALLOW       162.158.0.0/15
443/tcp                     ALLOW       104.16.0.0/13
443/tcp                     ALLOW       104.24.0.0/14
443/tcp                     ALLOW       172.64.0.0/13
443/tcp                     ALLOW       131.0.72.0/22
Anywhere (v6) on tailscale0 ALLOW       Anywhere (v6)
443/tcp   (v6)              ALLOW       2400:cb00::/32
443/tcp   (v6)              ALLOW       2606:4700::/32
443/tcp   (v6)              ALLOW       2803:f800::/32
443/tcp   (v6)              ALLOW       2405:b500::/32
443/tcp   (v6)              ALLOW       2405:8100::/32
443/tcp   (v6)              ALLOW       2a06:98c0::/29
443/tcp   (v6)              ALLOW       2c0f:f248::/32

Summary

Finally, let’s make sure this works (I’ve hidden my server’s actual Tailscale and public IP addresses behind the tailscale.formulate.dev and ssh.formulate.dev hostnames respectively):

This is a great way to quickly and reliably secure personal servers without the hassle of maintaining something like fail2ban. You can even block off all non-Tailscale access if you want to host something for yourself – you can’t really beat that for security!

Edit