Self-Hostember #9: Securing and Hardening Your Server

ℹ️ This post is part of a series , and it assumes that you’ve followed the instructions from Day Zero first.

Tradeoffs of self-hosting

Self-hosting services gives you immense power and control, but with that also comes responsibility. When you are using third-party services, you are not only paying for storage and compute but also for security. The provider is responsible for ensuring that only authorized parties have access to your data and that if that assumption ever breaks, they know as soon as possible.

Of course, it is not to say that third-party providers are necessarily perfect at securing your data. We have many data breaches to disabuse ourselves of that notion.

However, at least in theory, Google and Facebook have competent people on staff who are paid to ensure that access to customer data is protected at all times. You as the self-hoster are trading off the convenience of having a highly paid team of experts protecting your data for control and privacy.

It must also be mentioned that bigger platforms are also huge centralized sources of data and are thus highly attractive breach targets. Self-hosted services on a VM does not provide similar incentives to hackers, unless it can be part of a wide net vulnerability like a popular web-scale DB being left exposed to the internet by default.

That all being said, it is 100% possible to self-host services securely without tearing all your hair out constantly looking out for CVEs and analyzing logs. Some basic steps go a very long way to ensuring that your system is not an easy target for attackers, and we will look at some of them here.

Tiers of security measures

We will look at a completely made up hierarchy of security measures, in increasing order of provided security guarantees. So Tier 1 provides stronger guarantees than Tier 2 and so on.

We will look at them in decreasing order of guarantees just to spice things up (and also surface the best mitigations early).

Tier 5: Point-to-Point VPNs

Tldr; If your services only need to be accessed by you and a small, trusted group of people, avoid exposing your services to the public internet and access them via a point-to-point VPN like Wireguard/Tailscale/Netbird instead.

VPNs (Virtual Private Networks) ensure that your connection to your interlocutor is end-to-end encrypted. The consumer “privacy” VPNs (e.g. Migadu, Surfshark etc.) are the first things people think of as VPNs, but we are talking about security VPNs today.

Security VPNs are useful to establish a “tunnelled” connection between two machines you have control over. This connection is encrypted using public key cryptography and so is resistant to snooping (as long as quantum computers are not practical, at least).

The state of the art for this is Wireguard . Almost all other more popular tools like Tailscale are built on top of Wireguard itself, because it performs so much better than other options like OpenVPN. Wireguard is built into the Linux kernel, which allows it to achieve its performance levels.

While configuring Wireguard manually is a bit of a chore, it is not impossible for one person to manage, especially if you only have a couple of servers. However, distributing credentials to your less-techy family members might be a bridge too far. This is where Tailscale and Netbird shine, because they handle the key distribution for you (amongst other things).

If the usecase suits your needs, this is the most secure option as it prevents random people on the internet from hitting your services directly and exploiting any vulnerabilities.

Tier 4: Firewall

Tldr; If you cannot depend on a point-to-point VPN because you need to publicly expose your services then it’s 100% worth spending time configuring your firewall.

The specific firewall you choose to configure depends on your operating system, but ufw is a good choice for Ubuntu and can also be easily installed on Debian.

Using ufw is really easy:

1. First we ensure we don’t get locked out.

sudo ufw allow OpenSSH

2. Then enable the firewall

sudo ufw enable

3. Check the status

sudo ufw status

You should see something like:

Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere

In most cases, you should disallow all traffic except TCP on 80 and 443 (and 22 for SSH). If using Wireguard directly, you will also need to allow UDP traffic on the Wireguard port (default 51820). This reduces the surface area of exploits to only those protocols.

You can peruse the ArchWiki entry for more details.

Internally on Linux, all firewalls work by manipulating the iptables or nftables rules. Learning some basics of manipulating them would also be helpful to add firewall rules that are too advanced for simpler tools like ufw.

Tier 3: Intrusion Prevention Tools

Tldr; Intrusion prevention tools like fail2ban allow you to automate banning of suspicious sources of activity from reaching your services.

If you hadn’t heard of the popular tool fail2ban , well, now you have. It is the most popular tool in the intrusion prevention space. It constantly looks at your access logs and identifies IPs that are pounding your server too often and can either put them in a cooldown jail or permanently ban them.

There are other tools in this class as well:

• sshguard : Like fail2ban but lighter-weight with fewer features.
• crowdsec : Uses a community-maintained DB of malicious IPs to trigger bans.

These tools also work by modifying iptables, so they can be thought of as dynamic firewalls but perform a complementary service to traditional firewalls so they should be used alongside - not instead of - them.

Tier 2: Linux Security Modules

Tldr; Ensure that a suitable Linux Security Module (LSM) like AppArmor or SELinux is enabled on your system.

Linux Security Modules are a framework to define security checks for various processes on the system. One class of implementation of this framework is in the “mandatory access control” space. Some examples include AppArmor and SELinux. The main benefit is that they prevent applications from accessing resources that they shouldn’t need to. In case a particular application is compromised, the surface area of the vulnerability is contained.

Fun fact: SELinux came out of the NSA (yes, that one).

If this sounds too complicated, don’t worry. Here’s a gist of things to do:

1. Ensure that the LSM favoured by your distro is enabled and enforced.

For AppArmor, you can use:

sudo apparmor_status

Which should have an output like:

apparmor module is loaded.
119 profiles are loaded.
24 profiles are in enforce mode.

If it’s not active or no policies are enforced, then you should probably go ahead and enable them!

You can follow this nice lab to learn more about AppArmor .

Tier 1: SSH

Tldr; Utilize SSH, disable password authentication, and direct root login.

If you are using a Linux or BSD server, you are probably accessing it via SSH already. To make your access even more secure, we can ensure that SSH is configured with certain defaults changed. These settings are modified in your sshd_config file (usually in /etc).

0. Setup SSH keys

Generate a SSH keypair for yourself on your local machine with ssh-keygen (use ed25519 as the algorithm instead of the older RSA), and use ssh-copy-id to copy over the public key to your remote server for public key access.

1. Disable Password Authentication

This ensures that you need to use your generated SSH key to authenticate.

PasswordAuthentication no

2. Prohibit Root Login (or via password)

You can prohibit root login directly altogether, which will require you to login as a non-root user and then promote yourself to root when required.

PermitRootLogin no

Or you can keep root login but you should disable password auth. Ensure that your public key is present in root’s authorized_keys file (use ssh-copy-id if not).

PermitRootLogin prohibit-password

With this, you should already be better placed than most VMs out in the world. Keep your SSH key secure, and default to creating new keypairs for each new host instead of reusing them across new computers.

Tier 0: Non-root account

Tldr; Create and use non-root accounts for running services/docker containers.

Hopefully you are already doing this! If not, please do not run your services as root. root has privileged access that can easily be inherited by your services even if they don’t require them and can later be used in an exploit.

On Debian/Ubuntu you can create new users with (assuming you are root):

adduser <username>

Which will guide you through the user creation flow.

You may also want to add the user into sudoers with:

usermod -aG sudo <username>

Which will allow you to elevate your privileges when necessary with sudo.

All your services and docker containers should be run with non-root users as much as possible. Some services will automatically create users for their own use (e.g. caddy), for others you may need to configure it yourself.

Bonus: Rootless Docker

Tldr; Run docker in rootless mode to minimize the attack surface

Here’s a bonus tip: you can run the docker daemon without root privileges. The default docker install needs root permissions, but it is also possible to configure it post-install to run in rootless mode and not need elevated privileges.

This is not as well-known as it ought to be! The instructions are available on the official Docker website , but if you installed Docker using your system’s package manager, it may be as easy as running:

dockerd-rootless-setuptool.sh install

The dockerd-rootless-setuptool.sh install script is installed as part of docker when installed via apt/yum etc.

Do note that there are some limitations with the rootless mode. One key one is that AppArmor profiles are not supported - but this is worth the tradeoff for me. (You can also use Podman instead, which theoretically has AppArmor support but I haven’t personally verified this).

Conclusion

And that’s it. With some fairly straightforward mitigations, we can make our servers harder to breach and our self-hosted services more secure. While it is difficult to see this as anything but a “cost” of self-hosting, it is a surmountable hill that becomes easier after the first time. With some practice you may also want to automate a lot of it with scripts 😉

Thanks for reading this far, and follow along for future posts in #Self-hostember . You can now follow us on Mastodon to keep up to date with us as well!