Configuring a Production Instance

Installing Andes on a new linux box

Carry out the following commands from the server command line:

Install Prerequisites / Basic Server Configuration

Set up and install all the dependencies:

sudo apt update
sudo apt upgrade
sudo apt install apache2 mysql-server arp-scan curl python3 python3-dev python3-venv python3-pip default-libmysqlclient-dev cups libcups2-dev unzip libapache2-mod-wsgi-py3 gettext cifs-utils samba redis-server links sqlite3 pkg-config default-mysql-client

Install command line utility for converting mysql sql dumps to sqlite3 databases:

cd /opt
sudo git clone https://github.com/dumblob/mysql2sqlite
sudo chmod -R 777 ./mysql2sqlite
sudo ln -s /opt/mysql2sqlite/mysql2sqlite /usr/local/bin/mysql2sqlite

Set up the NTP time-server:

open the timesyncd config file
```
sudo nano /etc/systemd/timesyncd.conf
```

add the following line:

# Common NTP servers that be used are `ntp.ubuntu.com`,`time.google.com`, `DFNSB2YwpDCP001.ENT.dfo-mpo.ca` (for DFO intranet), ...
NTP=ENTER.YOUR.NTP.SERVER.ADDRESS.HERE
FallbackNTP=ntp.ubuntu.com

ensure timesync is on:
```
sudo timedatectl set-ntp on
```
check status of timesyncd to make sure there is a no-zero packet count:
```
timedatectl timesync-status
```

Take care of user permissions. dialout group is for accessing COMs; adm is for accessing logs; www-data is the user/group associated with apache2
1. Add your user (assuming your username is admin_andes) into the following groups:
```
sudo usermod -aG www-data,dialout,adm admin_andes
```
2. Add the apache www-data user into the following groups:
```
sudo usermod -aG dialout,adm www-data
```
3. OPTIONAL: Add www-data user into the sudo group. Assign a password to the www-data user. This step is needed if you want the web users to be able to run administrative functions on the server. it is risky.
```
sudo adduser www-data sudo
sudo passwd www-data  # best to use a randomly-generated password.
```
4. Re-login as your user to realize the new permissions:
```
sudo su admin_andes
```

Installing Andes from Github and installing python dependencies

Create a github personal access token (github.com > settings > developer settings > personal access tokens > generate new token > check off repo)
Copy the github PAT on your server:
```
echo "MY_GITHUB_PAT" > ~/git_token
```

install andes project from git repo:

sudo mkdir /opt/andes_root
sudo chown -R admin_andes /opt/andes_root/
cd /opt/andes_root
python3 -m venv venv
git clone https://github.com/dfo-gulf-science/andes.git
source ./venv/bin/activate
cd andes
git config credential.helper store
git pull origin stage  # this will just store your security credentials
sudo git pull origin stage  # this will just store your security credentials (under sudo)
pip install -r ./requirements.txt
git config --global --add safe.directory /opt/andes_root/andes
sudo git config --global --add safe.directory /opt/andes_root/andes

check to see if everything installed correctly:
```
python manage.py check
```

Mission up the MySQL Database

set up the database and users:

 sudo mysql
 CREATE DATABASE andesdb CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
 CREATE USER 'andes_user'@'localhost' IDENTIFIED BY 'smellyfish123*';
 GRANT ALL PRIVILEGES ON andesdb.* TO 'andes_user'@'localhost';
 GRANT PROCESS ON *.* TO 'andes_user'@'localhost';
 FLUSH PRIVILEGES;
 exit

note, if the database is already created, simply alter the table with the character set and collation below:

 ALTER DATABASE andesdb CHARACTER SET utf8mb4;
 ALTER DATABASE andesdb COLLATE utf8mb4_0900_ai_ci;

This will pn;u be applied to new tables/data, to modify the collation of an existing table (for examplel shared_models_observationtype) use this command:

ALTER TABLE shared_models_observationtype CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;

Create a .env file with database credentials for andes in order to point the application to the newly created database. Clone the example file in this repo as a starting point:
```
cp ./docs/examples/example_env.txt ./.env
```
Make any adjustments, as required using
```
nano ./.env
```
Preform a check to make sure the database is properly connected:
```
python manage.py check
```
You should see a message along the lines of: *** Connecting to database 'ANDESDB' on host 'LOCALHOST' ***

Loading Initial Data into the Database

NOTE: This step assumes you are using a .SQL to load the data. For other options and scenarios, please see on upgrading and restoring backups.

Find a way to get the .SQL file onto the server. Using a tool like MobaXterm is super convenient but alternatives include using Google Drive which might look something like this:
```
 cd ~
 fileId=1JI1taRu-Tvw8Bj55J5TkcUg20hrJpvzM
 gdown -O db.zip https://drive.google.com/uc?id=${fileId}
 unzip db.zip
```
It is important that you take note of the versions / git hash of the instance which generated the data fixture. These information should be stamping into the fixture file’s filename, e.g. andes**v2.2fc80f3c9**_CAR2023300_2023_08_25_12_58_27_mysql.sql.zip. Again, see the section on upgrading and restoring backups for more details on this.

Unzip the .SQL file:

unzip ~/andes_v2.2_fc80f3c9_CAR2023300_2023_08_25_12_58_27_mysql.sql.zip

Load the data into the MySQL database:

sudo mysql andesdb < andes_v2.2_fc80f3c9_CAR2023300_2023_08_25_12_58_27_mysql.sql

That’s it!

Mission up Celery

(thanks in part to https://pythad.github.io/articles/2016-12/how-to-run-celery-as-a-daemon-in-production)

This is an important reference: https://docs.celeryproject.org/en/latest/userguide/daemonizing.html#init-script-celerybeat

Celery is used to run asynchronous tasks in Andes. This is useful when we want to have a function run but not hold up the web server (e.g., generating a large report, waiting for GPS signal, etc).
In order for celery tasks to be executed, you will need to make use of a message server. As a part of the basic setup instructions, you would have already installed Redis. Before continuing below, make sure the redis service is up and running:
```
sudo service redis-server status
```
By default, it listens on localhost:6379. This is also the default expectation of Andes, so if this is the configuration you are using, no further action is required.
Celery Beat is used for running periodic tasks asynchronously.

Create a celery configuration file based on the example provided in this repo:

sudo cp /opt/andes_root/andes/docs/examples/celeryd.conf /etc/default/celeryd

Create the following folder and provide the www-data user access it:

sudo mkdir /var/run/celery
sudo chown -R :www-data /var/run/celery
sudo chmod -R 771 /var/run/celery

Create a systemd for Celery based on the example in this repo:

sudo cp /opt/andes_root/andes/docs/examples/celeryd.service.txt /etc/systemd/system/celeryd.service

start the daemon and check the status:

sudo systemctl daemon-reload

sudo systemctl enable celeryd.service
sudo systemctl start celeryd.service
sudo systemctl status celeryd.service

NOTE: it is very important to make sure you restart celery services upon deployment. These commands are included in the example deployment script (see next step).

NOTE: In order to Andes to use Celery, it has to be enabled in the .env file: USE_CELERY = True. Celery is only available on Linux machines.

Deploy the application

NOTE: It is highly recommended to be deploying the application from the same version / git hash of andes as the instance used to general the SQL file imported in the previous step.

Clone the example deployment script into its own file:
```
cp ./docs/examples/deploy.sh.txt ./deploy.sh
```
Edit the script, making any necessary adjustments:
```
nano ./deploy.sh
```
Save the script (ctrl-s + ctrl-x). If you have been following this guide closely, The only thing you should need to update is the variable called target_branch.
Give the cloned script executable permissions:
```
sudo chmod 774 ./deploy.sh
```
Add a symlink to the usr/local/bin directory so that the script can be run from anywhere, anytime:
```
sudo ln -s /opt/andes_root/andes/deploy.sh /usr/local/bin/deploy.sh
```
Run the script:
```
sudo deploy.sh
```

Configure Apache

If using webcams, please enable https with a self-signed certificate

Clone the example apache config from this repo to the apache folder on the server:
```
sudo cp ./docs/examples/000-default.conf /etc/apache2/sites-available/000-default.conf
```
Make any changes as required.
Restart the Apache service:
```
sudo service apache2 restart
```
check that everything is running correctly in Apache:
```
sudo service apache2 status
```

Configure Systemd Services for Nav, Depth and Net Telemetry Listeners

Recording of NMEA data is essentially outsourced to a background process. This process is very stable and will be fired up upon boot. Here are the steps to create those systemd services

Clone the example nav-depth listener config and example scanmar listener config from this repo to the systemd/system on the server:
```
sudo cp ./docs/examples/nav_depth_listener.service.txt /etc/systemd/system/nav_depth_listener.service
sudo cp ./docs/examples/scanmar_listener.service.txt /etc/systemd/system/scanmar_listener.service
```
Make sure to make any necessary adjustments to the scripts (e.g., file paths, settings, etc.).
start and enable the daemon and check the status:

sudo systemctl daemon-reload

sudo systemctl enable nav_depth_listener.service scanmar_listener.service
sudo systemctl start nav_depth_listener.service scanmar_listener.service
sudo systemctl status nav_depth_listener.service scanmar_listener.service

Static Server IP

It is a good idea to set the local address of your Andes server to be a static IP address, as opposed to being dynamically assigned by a DHCP server.

Make sure the file in the following folder /etc/cloud/cloud.cfg.d/ is set to network: {config: disabled}. The name of the file will vary.
Check if there are any files in the following folder: /etc/netplan/. If there are, back them up somewhere before proceeding:
```
mkdir ~/netplan_backup
sudo cp /etc/netplan/00-installer-config* ~/netplan_backup/ 
```

Clone the example netplan config file as your starting point for the new config:

sudo cp ./docs/examples/00-installer-config.yaml /etc/netplan/00-installer-config.yaml

Modify the config file so that the name of your network interface card (NIC) is shown and make any necessary adjustments:
```
sudo nano /etc/netplan/00-installer-config.yaml
```
I like to configure the wifi card (if applicable) on dhcp is so that if you needed to connect to a hotspot, you can do so readily.

Apply the changes to the netplan:

sudo chmod 600 /etc/netplan/00-installer-config.yaml
sudo netplan apply

CUPS and Printer Setup

Printers can be a challenging part of the setup but these docs should help simplify it. The printing in Andes depends on CUPS operating on the Andes Server. For troubleshoot purposes, it is get if there is unrestricted access to the CUPS admin console, although this is a security risk. To remove all restrictions do the following steps:

First backup the config file: sudo cp /etc/cups/cupsd.conf /etc/cups/cupsd.conf.backup
Clone the example cups config file into the cups system folder:
```
sudo cp ./docs/examples/cupsd.conf /etc/cups/cupsd.conf
```

We have had a few nightmarish situations where communication between CUPS and the printer has not been good. In at least some cases it seems to have had to do with the way the cups-browsed service automatically adds the printer (sometimes it attaches to the wrong URI or driver). A clean and reliable way to set up printers is to add them manually to CUPS using lpadmin command line tool:

For example, here is how you would add a new cups printer:

But first, you will have to go into the printer firmware settings (via web browser, see below for finding printer ip on a network) and setting the printer to a static ip. If you are really smart, you will make corresponding entries into the router’s DHCP IP reservations (see Cradlepoint notes). This way, there will not be a chance of the DHCP server assigning that IP to another host, in which case you’ll be in trouble.

printer_name="my_special_label_printer"  # this will be the name of the printer in CUPS admin
printer_ip_or_node=192.168.x.x  # the ip or node name (e.g. BRN0080774FCA91) of the printer
decription="This is the description of my printer at station 1"  # description display in CUPS admin
location="Station 1"  # location display in cups admin 

lpadmin -p "$printer_name" -E -v ipp://$printer_ip_or_node/ipp/print -m everywhere -D "$decription" -L "$location"

If you are doing this, it is not a horrible idea to use static IPs for your printers OR you can also use the IPP network node name of the printer instead of the IP (e.g., ipp://BRN0080774FCA91/ipp/print).

Please refer to the CUPS docs for more details, or from the command line, lpadmin --help.

A quick way to check the IPs on the network assigned to a “Brother” host is:

sudo arp-scan -l | grep Brother

You can leave out the | grep Brother to see a list of all hosts on the network. Also, in our experience, if your printer is connected by wifi, it might not be listed as a Brother printer and you will have to do some detective work to figure out the IP.