# Server Setup **Effective 14 July 2024** This document outlines how we set up our remote servers for running projects. ## 1. Provision a Server We use DigitalOcean as our provider. Regardless of your choice, provision a new VPS using the **latest Ubunutu LTS version**. Add your `ssh` key AND Naomi's `ssh` key in the setup process. ## 2. Set Up User You should never run applications on root. SSH into the new VPS to prepare your user. ### 2.1. Creating the User You'll need to set a password for the `root` account first. ```bash passwd ``` Once you have set a password, ensure that you have provided it to Naomi to store in the vault. Create an `nhcarrigan` user for our organisation. ```bash adduser nhcarrigan ``` Set a **different** password, and provide that to Naomi as well. For all of the user information, use the default blank values. Add the new user to the sudoers file. ```bash usermod -aG sudo nhcarrigan ``` Then sync the SSH keys so we can authenticate as that user. ```bash rsync --archive --chown=nhcarrigan:nhcarrigan ~/.ssh /home/nhcarrigan ``` While you are there, set the timezone for the server to our business' local timezone. ```bash sudo timedatectl set-timezone America/Los_Angeles ``` ## 3. Preparing For Web Requests To prepare the server to receive web requests, you'll need to follow a few steps. ### 3.1. SSL Certificate > [!NOTE] > If the Firewall has been set up, you'll need to temporarily allow port 80 for the certificate to generate. We use LetsEncrypt to provision our SSL certificates. If it is not installed, install it with: ```bash sudo snap install --classic certbot ``` Then link the snap to our `usr` directory. ```bash sudo ln -s /snap/bin/certbot /usr/bin/certbot ``` Generate a certificate with: ```bash sudo certbot certonly --standalone ``` And allow applications to read it: ```bash sudo chmod -R a+rwx /etc/letsencrypt ``` When you need to renew the certificate: ```bash sudo certbot renew ``` ### 3.2. NGINX All requests should be routed through NGINX. At no point should an application run directly on ports 80 or 443. Install NGINX: ```bash sudo apt-get install nginx ``` Edit the configuration file: ```bash sudo emacs /etc/nginx/conf.d/server.conf ``` Use this template to set up a reverse proxy on the standard HTTPS port 443: ```nginx server { listen 443 ssl; server_name subdomain.domain.tld; ssl_certificate /etc/letsencrypt/live/subdomain.domain.tld/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/subdomain.domain.tld/privkey.pem; location / { proxy_set_header Host $host; proxy_pass https://127.0.0.1:port; proxy_redirect off; } } ``` Validate that the config is correct with: ```bash sudo nginx -t ``` If so, restart NGINX to apply the changes: ```bash sudo systemctl restart nginx ``` ## 4. Securing the Server We have a minimum level of security that is required on ALL of our servers. This section should not be treated as the best effort, but as the minimal requirements to comply with our policies. ### 4.1. Firewall We use `ufw` as our firewall. First, enable the SSH port. ```bash sudo ufw allow "OpenSSH" ``` Then, allow the standard HTTPS port and **deny** the standard HTTP port. ```bash sudo ufw deny http sudo ufw allow https ``` Enable the firewall. You may get dropped from the SSH connection. ```bash sudo ufw enable ``` ### 4.2. Fail2Ban We also use Fail2Ban to block IP addresses which fail to make requests too often. Install the tool: ```bash sudo apt-get install fail2ban ``` Configure the NGINX jail in `/etc/fail2ban/jail.d/nginx-auth.conf`: ```ini [nginx-auth] enabled = true filter = nginx-auth logpath = /var/log/nginx/access.log maxretry = 3 findtime = 86400 bantime = 86400 ``` Configure the NGINX filter in `/etc/fail2ban/filter.d/nginx-auth.conf`: ```ini [Definition] failregex = ^ - .* \[.*\] ".*" (4\d{2}) .*$ ``` Because we use Cloudflare, you'll need to grab the original IP for all requests. Start by creating a file to store Cloudflare's IPs. ```bash sudo touch /etc/nginx/cloudflare_ips.conf ``` Then create your script: ```bash nano ~/update_cf_ips.sh ``` ```bash #!/bin/bash # Create a temporary file temp_file=$(mktemp) # Download IPv4 ranges and format each line curl -s https://www.cloudflare.com/ips-v4 | while read ip; do echo "set_real_ip_from $ip;" >> "$temp_file" done # Download IPv6 ranges and format each line curl -s https://www.cloudflare.com/ips-v6 | while read ip; do echo "set_real_ip_from $ip;" >> "$temp_file" done # Add the real_ip_header directive echo "real_ip_header CF-Connecting-IP;" >> "$temp_file" # Replace the old file with the new one sudo mv "$temp_file" /etc/nginx/cloudflare_ips.conf # Test Nginx configuration sudo nginx -t # If the test is successful, reload Nginx if [ $? -eq 0 ]; then sudo systemctl reload nginx echo "Nginx configuration updated and reloaded successfully." else echo "Nginx configuration test failed. Please check your configuration." fi ``` Make it executable and run it: ```bash sudo chmod +x update_cf_ips.sh sudo ./update_cf_ips.sh ``` If it runs as expected, set it up to run on a CRON. ```bash sudo crontab -e ``` ```bash 0 3 * * 1 ~/update_cf_ips.sh ``` Then, update the `/etc/nginx/nginx.conf` to use all of this new logic. This goes at the end of your `http` directive block. ```conf # Look at the real IP, not the cloudflare IP. include /etc/nginx/cloudflare_ips.conf; log_format custom_format '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '"$http_x_forwarded_for"'; access_log /var/log/nginx/access.log custom_format; ``` Confirm the NGINX configuration is correct: ```bash sudo nginx -t ``` Then restart everything. ```bash sudo systemctl restart nginx sudo systemctl restart fail2ban ``` To view banned IPs: ```bash sudo fail2ban-client status nginx-auth ``` And to unban them: ```bash sudo fail2ban-client set nginx-auth unbanip ``` ## 5. Running a Project Now you are ready to start running the project. ### 5.1. Node.js Most of our projects will run on Node. For a new machine, you'll need to set that up. We use `nvm` to manage Node versions. Fetch and run the install script: ```bash curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash ``` The script will automatically update the `.bashrc` file to load `nvm` into the PATH. Reload that: ```bash source ~/.bashrc ``` Install the long-term support Node version. ```bash nvm install --lts ``` This should automatically set it as the default. When updating, be sure to remove any older versions! Finally, install `pnpm` as the package manager. ```bash npm i -g pnpm ``` ### 5.2. PM2 All of our processes run with PM2 to allow for monitoring and auto-restarts. You'll need to install it. ```bash pnpm i -g pm2 ``` To start a project, use this template: ```bash pm2 start '