---
title: Server Setup
---

This document outlines how we set up our remote servers for running projects.

## 1. Provision a Server

We use DigitalOcean as our provider. Regardless of your choice, provision a new VPS using the **latest Ubuntu LTS version**. Add your `ssh` key AND Naomi's `ssh` key in the setup process.

## 2. Set Up User

You should never run applications on root. SSH into the new VPS to prepare your user.

### 2.1. Creating the User

You'll need to set a password for the `root` account first.

```bash
passwd
```

Once you have set a password, ensure that you have provided it to Naomi to store in the vault.

Create an `nhcarrigan` user for our organisation.

```bash
adduser nhcarrigan
```

Set a **different** password, and provide that to Naomi as well. For all of the user information, use the default blank values.

Add the new user to the sudoers file.

```bash
usermod -aG sudo nhcarrigan
```

Then sync the SSH keys so we can authenticate as that user.

```bash
rsync --archive --chown=nhcarrigan:nhcarrigan ~/.ssh /home/nhcarrigan
```

While you are there, set the timezone for the server to our business' local timezone.

```bash
sudo timedatectl set-timezone America/Los_Angeles
```

## 3. Preparing For Web Requests

To prepare the server to receive web requests, you'll need to follow a few steps.

### 3.1. SSL Certificate

:::note
If the Firewall has been set up, you'll need to temporarily allow port 80 for the certificate to generate.
:::

We use LetsEncrypt to provision our SSL certificates. If it is not installed, install it with:

```bash
sudo snap install --classic certbot
```

Then link the snap to our `usr` directory.

```bash
sudo ln -s /snap/bin/certbot /usr/bin/certbot
```

Generate a certificate with:

```bash
sudo certbot certonly --standalone
```

And allow applications to read it:

```bash
sudo chmod -R a+rwx /etc/letsencrypt
```

When you need to renew the certificate:

```bash
sudo certbot renew
```

### 3.2. NGINX

All requests should be routed through NGINX. At no point should an application run directly on ports 80 or 443.

Install NGINX:

```bash
sudo apt-get install nginx
```

Edit the configuration file:

```bash
sudo emacs /etc/nginx/conf.d/server.conf
```

Use this template to set up a reverse proxy on the standard HTTPS port 443:

```nginx
server {
    listen 443 ssl;
    server_name subdomain.domain.tld;
    ssl_certificate /etc/letsencrypt/live/subdomain.domain.tld/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/subdomain.domain.tld/privkey.pem;

    location / {
        proxy_set_header Host $host;
        proxy_pass https://127.0.0.1:port;
        proxy_redirect off;
    }
}
```

Validate that the config is correct with:

```bash
sudo nginx -t
```

If so, restart NGINX to apply the changes:

```bash
sudo systemctl restart nginx
```

## 4. Securing the Server

We have a minimum level of security that is required on ALL of our servers. This section should not be treated as the best effort, but as the minimal requirements to comply with our policies.

### 4.1. Firewall

We use `ufw` as our firewall. First, enable the SSH port.

```bash
sudo ufw allow "OpenSSH"
```

Then, allow the standard HTTPS port and **deny** the standard HTTP port.

```bash
sudo ufw deny http
sudo ufw allow https
```

Enable the firewall. You may get dropped from the SSH connection.

```bash
sudo ufw enable
```

### 4.2. Fail2Ban

We also use Fail2Ban to block IP addresses which fail to make requests too often.

Install the tool:

```bash
sudo apt-get install fail2ban
```

Configure the NGINX jail in `/etc/fail2ban/jail.d/nginx-auth.conf`:

```ini
[nginx-auth]
enabled = true
filter = nginx-auth
logpath = /var/log/nginx/access.log
maxretry = 3
findtime = 86400
bantime = 86400
```

Configure the NGINX filter in `/etc/fail2ban/filter.d/nginx-auth.conf`:

```ini
[Definition]
failregex = ^<HOST> - .* \[.*\] ".*" (4\d{2}) .*$
```

Because we use Cloudflare, you'll need to grab the original IP for all requests. Start by creating a file to store Cloudflare's IPs.

```bash
sudo touch /etc/nginx/cloudflare_ips.conf
```

Then create your script:

```bash
nano ~/update_cf_ips.sh
```

```bash
#!/bin/bash

# Create a temporary file
temp_file=$(mktemp)

# Download IPv4 ranges and format each line
curl -s https://www.cloudflare.com/ips-v4 | while read ip; do
    echo "set_real_ip_from $ip;" >> "$temp_file"
done

# Download IPv6 ranges and format each line
curl -s https://www.cloudflare.com/ips-v6 | while read ip; do
    echo "set_real_ip_from $ip;" >> "$temp_file"
done

# Add the real_ip_header directive
echo "real_ip_header CF-Connecting-IP;" >> "$temp_file"

# Replace the old file with the new one
sudo mv "$temp_file" /etc/nginx/cloudflare_ips.conf

# Test Nginx configuration
sudo nginx -t

# If the test is successful, reload Nginx
if [ $? -eq 0 ]; then
    sudo systemctl reload nginx
    echo "Nginx configuration updated and reloaded successfully."
else
    echo "Nginx configuration test failed. Please check your configuration."
fi
```

Make it executable and run it:

```bash
sudo chmod +x update_cf_ips.sh
sudo ./update_cf_ips.sh
```

If it runs as expected, set it up to run on a CRON.

```bash
sudo crontab -e
```

```bash
0 3 * * 1 ~/update_cf_ips.sh
```

Then, update the `/etc/nginx/nginx.conf` to use all of this new logic. This goes at the end of your `http` directive block.

```nginx
# Look at the real IP, not the cloudflare IP.
include /etc/nginx/cloudflare_ips.conf;

log_format custom_format '$remote_addr - $remote_user [$time_local] '
                        '"$request" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent" '
                        '"$http_x_forwarded_for"';

access_log /var/log/nginx/access.log custom_format;
```

Confirm the NGINX configuration is correct:

```bash
sudo nginx -t
```

Then restart everything.

```bash
sudo systemctl restart nginx
sudo systemctl restart fail2ban
```

To view banned IPs:

```bash
sudo fail2ban-client status nginx-auth
```

And to unban them:

```bash
sudo fail2ban-client set nginx-auth unbanip <ip>
```

## 5. Uploading a Project

To upload a project, you should **not** use `git` to clone the project to the machine. Instead, start by cloning the project to your local environment and navigating to the directory:

```bash
git clone <url>
cd /path/to/project
```

Then sync the project up to the machine, ignoring any installed packages.

```bash
rsync -av --exclude='node_modules' ./ <server name>:/home/nhcarrigan/<project directory>
```

## 6. Running a Project

Now you are ready to start running the project.

### 6.1. Node.js

Most of our projects will run on Node. For a new machine, you'll need to set that up.

We use `nvm` to manage Node versions. Fetch and run the install script:

```bash
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash
```

The script will automatically update the `.bashrc` file to load `nvm` into the PATH. Reload that:

```bash
source ~/.bashrc
```

Install the long-term support Node version.

```bash
nvm install --lts
```

This should automatically set it as the default. When updating, be sure to remove any older versions!

Finally, install `pnpm` as the package manager.

```bash
npm i -g pnpm
```

### 6.2. PM2

All of our processes run with PM2 to allow for monitoring and auto-restarts. You'll need to install it.

```bash
pnpm i -g pm2
```

To start a project, use this template:

```bash
pm2 start '<script>' --name '<name>'
```

Then run `pm2 save` to save the application list.
