7.0 KiB
Server Setup
Effective 14 July 2024
This document outlines how we set up our remote servers for running projects.
1. Provision a Server
We use DigitalOcean as our provider. Regardless of your choice, provision a new VPS using the latest Ubunutu LTS version. Add your ssh key AND Naomi's ssh key in the setup process.
2. Set Up User
You should never run applications on root. SSH into the new VPS to prepare your user.
2.1. Creating the User
You'll need to set a password for the root account first.
passwd
Once you have set a password, ensure that you have provided it to Naomi to store in the vault.
Create an nhcarrigan user for our organisation.
adduser nhcarrigan
Set a different password, and provide that to Naomi as well. For all of the user information, use the default blank values.
Add the new user to the sudoers file.
usermod -aG sudo nhcarrigan
Then sync the SSH keys so we can authenticate as that user.
rsync --archive --chown=nhcarrigan:nhcarrigan ~/.ssh /home/nhcarrigan
While you are there, set the timezone for the server to our business' local timezone.
sudo timedatectl set-timezone America/Los_Angeles
3. Preparing For Web Requests
To prepare the server to receive web requests, you'll need to follow a few steps.
3.1. SSL Certificate
Note
If the Firewall has been set up, you'll need to temporarily allow port 80 for the certificate to generate.
We use LetsEncrypt to provision our SSL certificates. If it is not installed, install it with:
sudo snap install --classic certbot
Then link the snap to our usr directory.
sudo ln -s /snap/bin/certbot /usr/bin/certbot
Generate a certificate with:
sudo certbot certonly --standalone
And allow applications to read it:
sudo chmod -R a+rwx /etc/letsencrypt
When you need to renew the certificate:
sudo certbot renew
3.2. NGINX
All requests should be routed through NGINX. At no point should an application run directly on ports 80 or 443.
Install NGINX:
sudo apt-get install nginx
Edit the configuration file:
sudo emacs /etc/nginx/conf.d/server.conf
Use this template to set up a reverse proxy on the standard HTTPS port 443:
server {
listen 443 ssl;
server_name subdomain.domain.tld;
ssl_certificate /etc/letsencrypt/live/subdomain.domain.tld/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/subdomain.domain.tld/privkey.pem;
location / {
proxy_set_header Host $host;
proxy_pass https://127.0.0.1:port;
proxy_redirect off;
}
}
Validate that the config is correct with:
sudo nginx -t
If so, restart NGINX to apply the changes:
sudo systemctl restart nginx
4. Securing the Server
We have a minimum level of security that is required on ALL of our servers. This section should not be treated as the best effort, but as the minimal requirements to comply with our policies.
4.1. Firewall
We use ufw as our firewall. First, enable the SSH port.
sudo ufw allow "OpenSSH"
Then, allow the standard HTTPS port and deny the standard HTTP port.
sudo ufw deny http
sudo ufw allow https
Enable the firewall. You may get dropped from the SSH connection.
sudo ufw enable
4.2. Fail2Ban
We also use Fail2Ban to block IP addresses which fail to make requests too often.
Install the tool:
sudo apt-get install fail2ban
Configure the NGINX jail in /etc/fail2ban/jail.d/nginx-auth.conf:
[nginx-auth]
enabled = true
filter = nginx-auth
logpath = /var/log/nginx/access.log
maxretry = 3
findtime = 86400
bantime = 86400
Configure the NGINX filter in /etc/fail2ban/filter.d/nginx-auth.conf:
[Definition]
failregex = ^<HOST> - .* \[.*\] ".*" (4\d{2}) .*$
Because we use Cloudflare, you'll need to grab the original IP for all requests. Start by creating a file to store Cloudflare's IPs.
sudo touch /etc/nginx/cloudflare_ips.conf
Then create your script:
nano ~/update_cf_ips.sh
#!/bin/bash
# Create a temporary file
temp_file=$(mktemp)
# Download IPv4 ranges and format each line
curl -s https://www.cloudflare.com/ips-v4 | while read ip; do
echo "set_real_ip_from $ip;" >> "$temp_file"
done
# Download IPv6 ranges and format each line
curl -s https://www.cloudflare.com/ips-v6 | while read ip; do
echo "set_real_ip_from $ip;" >> "$temp_file"
done
# Add the real_ip_header directive
echo "real_ip_header CF-Connecting-IP;" >> "$temp_file"
# Replace the old file with the new one
sudo mv "$temp_file" /etc/nginx/cloudflare_ips.conf
# Test Nginx configuration
sudo nginx -t
# If the test is successful, reload Nginx
if [ $? -eq 0 ]; then
sudo systemctl reload nginx
echo "Nginx configuration updated and reloaded successfully."
else
echo "Nginx configuration test failed. Please check your configuration."
fi
Make it executable and run it:
sudo chmod +x update_cf_ips.sh
sudo ./update_cf_ips.sh
If it runs as expected, set it up to run on a CRON.
sudo crontab -e
0 3 * * 1 ~/update_cf_ips.sh
Then, update the /etc/nginx/nginx.conf to use all of this new logic. This goes at the end of your http directive block.
# Look at the real IP, not the cloudflare IP.
include /etc/nginx/cloudflare_ips.conf;
log_format custom_format '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'"$http_x_forwarded_for"';
access_log /var/log/nginx/access.log custom_format;
Confirm the NGINX configuration is correct:
sudo nginx -t
Then restart everything.
sudo systemctl restart nginx
sudo systemctl restart fail2ban
To view banned IPs:
sudo fail2ban-client status nginx-auth
And to unban them:
sudo fail2ban-client set nginx-auth unbanip <ip>
5. Running a Project
Now you are ready to start running the project.
5.1. Node.js
Most of our projects will run on Node. For a new machine, you'll need to set that up.
We use nvm to manage Node versions. Fetch and run the install script:
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash
The script will automatically update the .bashrc file to load nvm into the PATH. Reload that:
source ~/.bashrc
Install the long-term support Node version.
nvm install --lts
This should automatically set it as the default. When updating, be sure to remove any older versions!
Finally, install pnpm as the package manager.
npm i -g pnpm
5.2. PM2
All of our processes run with PM2 to allow for monitoring and auto-restarts. You'll need to install it.
pnpm i -g pm2
To start a project, use this template:
pm2 start '<script>' --name '<name>'
Then run pm2 save to save the application list.