HOW TO BECOME ~ SPO ~

PROCESS & CHECKLIST FOR STAKING POOL OPERATOR

[A] Pre-requirements

[B] BUILDING

0. Install GNU Screen

Screen or GNU Screen is a terminal multiplexer.

In other words, it means that you can start a screen session and then open any number of windows (virtual terminals) inside that session.

Processes running in Screen will continue to run when their window is not visible even if you get disconnected.

#1. Install GNU screen

sudo apt-get update -y

sudo apt-get upgrade -y

sudo apt-get install screen

screen --version

#2. Open up a new screen.

screen

#3. Run the process for Install CNODE

#4. Type: Ctrl + A, and then Ctrl + D. This will detach your screen session but leave your processes running!

#5. Resume your previously detached session. feel free to close the SSH terminal. whenever you feel like it, ssh back into your GCP VM, and type

screen -r

#6. To find the session ID list the current running screen sessions with

screen -ls

#7. To kill all detached screens, run:

screen -ls | grep pts | cut -d. -f1 | awk '{print $1}' | xargs kill

1. Install and Update packages Ubuntu dependencies

sudo usermod -aG sudo <User_name>

sudo apt-get update -y

sudo apt-get upgrade -y

sudo apt autoremove

sudo apt-get autoclean

sudo apt-get -y install automake build-essential pkg-config libffi-dev libgmp-dev libssl-dev libtinfo-dev libsystemd-dev zlib1g-dev

sudo apt-get -y install python3 systemd yarn libsodium-dev libncursesw5 libtool autoconf

sudo apt-get -y install make g++ tmux git jq wget htop nload curl zip unzip rsync wget cron

sudo apt-get -y install openssh-server net-tools gparted

2. Install Cabal : the Haskell build tool

#Install Cabal v3.4.0.0

cd

wget https://downloads.haskell.org/~cabal/cabal-install-3.4.0.0/cabal-install-3.4.0.0-x86_64-ubuntu-16.04.tar.xz

tar -xf cabal-install-3.4.0.0-x86_64-ubuntu-16.04.tar.xz

rm cabal-install-3.4.0.0-x86_64-ubuntu-16.04.tar.xz

mkdir -p ~/.cabal/bin

mv cabal ~/.cabal/bin/


#Install Cabal v3.6.2.0

cd

wget https://downloads.haskell.org/~cabal/cabal-install-3.6.2.0/cabal-install-3.6.2.0-x86_64-linux-alpine.tar.xz

tar -xf cabal-install-3.6.2.0-x86_64-linux-alpine.tar.xz

rm cabal-install-3.6.2.0-x86_64-linux-alpine.tar.xz

mkdir -p ~/.cabal/bin

mv cabal ~/.cabal/bin/


echo export PATH=~/.cabal/bin:$PATH >> ~/.bashrc

source ~/.bashrc

echo $PATH

cabal --version

3. Installing GHC - The Glorious Glasgow Haskell Compilation System

cd

# Download and install version 8.10.4 of GHC

wget https://downloads.haskell.org/ghc/8.10.4/ghc-8.10.4-x86_64-deb9-linux.tar.xz

tar -xf ghc-8.10.4-x86_64-deb9-linux.tar.xz

rm ghc-8.10.4-x86_64-deb9-linux.tar.xz

cd ghc-8.10.4

./configure

sudo make install


# Download and install version 8.10.7 of GHC. The easiest way to do this is to use ghcup.

curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | sh

ghcup install ghc 8.10.7

ghcup install cabal 3.4.0.0

ghcup set ghc 8.10.7

ghcup set cabal 3.4.0.0


cd

ghc --version

4. Install Libsodium

mkdir $HOME/git

cd $HOME/git

git clone https://github.com/input-output-hk/libsodium

cd libsodium

git checkout 66f017f1

./autogen.sh

./configure

make

sudo make install

5. Update CNODE PATH

export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"

export PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH"

echo export CNODE_HOME=/opt/cardano/cnode >> $HOME/.bashrc

echo export CARDANO_NODE_SOCKET_PATH="$CNODE_HOME/sockets/node0.socket" >> $HOME/.bashrc


source ~/.bashrc

6. Build cardano-node & cardano-cli binary

cd $HOME/git

git clone https://github.com/input-output-hk/cardano-node.git


cd cardano-node

git fetch --all --recurse-submodules --tags && git tag

git checkout tags/<TAGGED VERSION>


cabal configure --with-compiler=ghc-8.10.7

echo "package cardano-crypto-praos" >> cabal.project.local

echo " flags: -external-libsodium-vrf" >> cabal.project.local

# Build Core/Relay Node Online

cabal build all


cp -p dist-newstyle/build/x86_64-linux/ghc-8.10.2/cardano-node-1.24.2/x/cardano-node/build/cardano-node/cardano-node ~/.cabal/bin/

cp -p dist-newstyle/build/x86_64-linux/ghc-8.10.2/cardano-cli-1.24.2/x/cardano-cli/build/cardano-cli/cardano-cli ~/.cabal/bin/


which cardano-node && which cardano-cli

cardano-node --version

cardano-cli --version

# Build Air-gapped Node Offline

$CNODE_HOME/scripts/cabal-build-all.sh -o

7. Create CNTools cnode #files #db #priv #scripts #sockets #logs

mkdir "$HOME/tmp"

cd "$HOME/tmp"

curl -sS -o prereqs.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/prereqs.sh

chmod 700 prereqs.sh

./prereqs.sh

8. Create TMUX scripts #start_all #stop_all

cd git

git clone https://github.com/stakepool247/CardanoHaskellTestnetScripts.gitcd CardanoHaskellTestnetScripts/

git checkout master

cp *.sh $CNODE_HOME/scripts/


cd $CNODE_HOME/scripts/

chmod +x start_all.sh stop_all.sh node.sh

[C] OPERATING

Infogragh cnode base: #cardano-node #cardano-cli #cncli #cntools

Infogragh : Staking Pool's KEY

Workflow : Register Staking Pool Workflow use CNTools

Workflow : KES Rotate

[D] OPTIMIZING

Blockchain Topology Network

Each stake pool must run at least one block-producing node and one relay node.

The block-producing node holds the keys and certificates necessary to issue blocks, but it is not directly connected to the network.

For security reasons, it must be connected only to one or more relay nodes controlled by the stake pool operator.

Then, the relay nodes connect to other relays on the network.

This is configured in the topology.json file together with the server firewall.

Topology :: Pool Topology network

Pool Topology {2 Core ; 2 Relay}

Pool Topology {3 Core ; 3 Relay}

Pool Topology {2 Core ; 4 Relay}

System :: create a swap file

# Type the following command to create a 5GB swap file on Ubuntu:

sudo dd if=/dev/zero of=/swapfile bs=1G count=5

# Verify that file has been created on the server:

ls -lh /swapfile

# Secure and set correct file permission for security reasons:

sudo chown root:root /swapfile

sudo chmod 0600 /swapfile

# Enable the swap space on Ubuntu:

sudo mkswap /swapfile

# Activate the swap file

sudo swapon /swapfile

# Verify new swap file and settings on Ubuntu

swapon -s

# Make sure the swap file enabled when server comes on line after the reboot

sudo nano /etc/fstab

# Append the following line:

/swapfile none swap sw 0 0

#Save and close the file.

# Check total and used swap size:

grep -i --color swap /proc/meminfo

cat /proc/swaps

vmstat

vmstat 1 5

# Monitor swap space usage

free -g

free -k

free -m

# Verify swap file and its usage:

top

htop

atop

# Disable swapfile on Ubuntu

sudo swapoff /swapfile

Scripts :: wget/curl large file from google drive

cd $CNODE_HOME/scripts/

sudo nano downG.sh

#-----------

#!/bin/bash

if [ $# != 2 ]; then

echo "Usage: downG.sh ID save_name"

exit 0

fi

confirm=$(wget --quiet \

--save-cookies /tmp/cookies.txt \

--keep-session-cookies \

--no-check-certificate 'https://docs.google.com/uc?export=download&id='$1 -O- | \

sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')

echo $confirm

wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$confirm&id=$1" -O $2 && rm -rf /tmp/cookies.txt

#-----------

sudo chmod +x downG.sh


# Get db epoch 252

$CNODE_HOME/scripts/downG.sh 1Jyuh8tw7A4SVEP96TpdCV_HpR9ZUdZpC db_252.zip

# Get cardano-node & cardano-cli Tags-1.26.1

$CNODE_HOME/scripts/downG.sh 1gVN0ZFAvng0R-FsVCUlRjXA7XhecqcIP cabal_bin_1261.zip

# Get cardano-node & cardano-cli Tags-1.25.1

$CNODE_HOME/scripts/downG.sh 1tI1dmxU4AM8MaQBenkF2EJZYeATe5n37 cabal_bin_1251.zip

Scripts :: clearlogs

cd $CNODE_HOME/scripts/

sudo nano clearlogs.sh

----------------

#!/bin/bash

/bin/rm -rf /opt/cardano/cnode/logs/node0-20*

/bin/rm -rf /opt/cardano/cnode/logs/node0.json

----------------------

sudo chmod +x clearlogs.sh

Scripts :: cleanCaches

cd $CNODE_HOME/scripts/

sudo nano cleanCaches.sh

----------------

#!/bin/bash

# Clean caches and buffers

free -h && sudo sysctl -w vm.drop_caches=3 && sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches && free -h

----------------------

sudo chmod +x cleanCaches.sh

Scripts :: memory alert

cd $CNODE_HOME/scripts

sudo nano memory_alert.sh

-------------------

#!/bin/bash

ramusage=$(free | awk '/Mem/{printf("RAM Usage: %.2f\n"), $3/$2*100}'| awk '{print $3}')

num_ramusage=$(numfmt --from auto "$ramusage" 2>&-)

#Alert action :: Reboot

if [[ $num_ramusage > 75 ]]; then

sudo systemctl restart cnode.service

#echo $ramusage

fi

-------------------

sudo chmod +x memory_alert.sh

Scripts :: Drop IP RTT--- Peer IN

cd $CNODE_HOME/scripts

sudo nano peerIN_check.sh

-------------------

#!/bin/bash

#Show Peer IN Number

echo "======================"

echo "Total Peer IN"

echo $(ss -tnp state established 2>/dev/null | \

grep "$(pgrep -fn "[c]ardano-node*.*--port 6000")," | \

grep -v ":$(ss -tnp state established "( dport = :6000 )" 2>/dev/null | \

grep cncli | awk '{print $3}' | cut -d: -f2) " | \

awk -v port=":6000" '$3 ~ port {print}' | wc -l)

# Get Peer IN IP and Port

peers=$(ss -tnp state established 2>/dev/null | \

grep "$(pgrep -fn "[c]ardano-node*.*--port 6000")," | \

grep -v ":$(ss -tnp state established "( dport = :6000} )" 2>/dev/null | \

grep cncli | awk '{print $3}' | cut -d: -f2) " | \

awk -v port=":6000" '$3 ~ port {print $4}')

netstatSorted=$(printf '%s\n' "${peers[@]}" | sort)

echo "======================"

echo "Peer IN list"

echo $netstatSorted

echo "======================"

echo "Check RTT Ping ICMP for each IP"

lastpeerIP=""

for peer in ${netstatSorted}; do

peerIP=$(echo "${peer}" | cut -d: -f1)

peerIP2=$(printf '%s' "${peerIP}")

echo $peerIP2

#echo $(ping -c 2 -i 0.3 -w 1 "${peerIP}" 2>&1 | tail -n 1 | cut -d/ -f5 | cut -d. -f1)

if checkPEER=$(ping -c 2 -i 0.3 -w 1 "${peerIP}" 2>&1); then # Incoming connection, ping OK, show RTT.

echo "Incoming connection, ping OK, show RTT"

#peerRTT=$(echo "${checkPEER}" | tail -n 1 | cut -d/ -f5 | cut -d. -f1)

echo $(ping -c 2 -i 0.3 -w 1 "${peerIP}" 2>&1 | tail -n 1 | cut -d/ -f5 | cut -d. -f1)

echo "---------------------------------------"

else # Incoming connection, ping failed, set as unreachable

#peerRTT=99999

echo "Incoming connection, ping failed, Unreachable"

echo "DROP IP"

#sudo iptables -A INPUT -s "${peerIP}" -p icmp -j DROP

#sudo iptables -A INPUT -s "${peerIP}" -p tcp -j DROP

sudo iptables -A INPUT -s "${peerIP}" -j DROP

echo "---------------------------------------"

fi

lastpeerIP=${peerIP}

done

# Save Unique rule iptables and Restore

sudo iptables-save | awk '/^COMMIT$/ { delete x; }; !x[$0]++' > /tmp/iptables.conf

sudo iptables -F

sudo iptables-restore < /tmp/iptables.conf

sudo iptables -L

#if [ -f /tmp/iptables.conf ] ; then /bin/rm -f /tmp/iptables.conf ; fi

-------------------

sudo chmod +x peerIN_check.sh

Crontab :: create cron job schedule

crontab -e

------------------------------------------

# m h dom mon dow command

MAILTO=""


#=== WEKKLY ===

#--: At 00:10 on Monday

10 0 * * 1 sudo apt-get update -y

#--: At 00:20 on Monday

20 0 * * 1 sudo apt-get upgrade -y

#--: At 00:30 on Monday

30 0 * * 1 sudo apt autoremove

#--: At 00:40 on Monday

40 0 * * 1 sudo apt-get clean


#=== REBOOT ===

#--: Delete iptables on startup {RELAY ONLY}

@reboot sleep 15 && sudo /usr/sbin/iptables -F


#=== DAILY ===

#--:At 05:11 : clearlogs

11 5 * * * /opt/cardano/cnode/scripts/clearlogs.sh

#--:At 05:12 : clean caches

22 5 * * * /opt/cardano/cnode/scripts/cleanCaches.sh

#--:At 05:13 : Run poolstatsUpdater {HOST}

13 5 * * * /opt/cardano/cnode/scripts/poolstatsUpdater.sh


#=== HOURLY ===

#--:At minute 5: Run topologyUpdater {RELAY ONLY}

5 * * * * /opt/cardano/cnode/scripts/topologyUpdater.sh


#=== EVERY FEW MINUTES ===

#--:At every 15th minute :: check RAM High usage then reboot cnode

*/15 * * * * /opt/cardano/cnode/scripts/memory_alert.sh

#--:At every 20th minute :: DROP IP - PING ICMP RTT--- Peer IN CNODE {RELAY ONLY}

*/20 * * * * /opt/cardano/cnode/scripts/peerIN_check.sh


#=== EVERY FEW HOURS ===

#--:Every 24th hour : At minute 10 : Delete all iptable rule chain {RELAY ONLY}

10 */24 * * * sudo /usr/sbin/iptables -F

------------------------------------------

# Display cron logs

sudo grep -i cron /var/log/syslog

sudo journalctl -u cron

sudo journalctl -u cron -b | more

sudo journalctl -u cron -b | grep something

sudo journalctl -u cron -b | grep -i error

VirtualBox : SharedFolders

# Mount VMShared Folder

sudo mkdir ~/Desktop/VMShared

sudo chmod 777 ~/Desktop/VMShared

sudo mount -t vboxsf VMShared ~/Desktop/VMShared

# Un mount VMShared Folder

sudo umount ~/Desktop/VMShared

[E] MONITORING

Infogragh : Staking Pool Monitoring with Prometheus / Grafana

HOST :: Run setup_mon on Node Host for monitoring

cd $CNODE_HOME/scripts

curl -s -o setup_mon.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/setup_mon.sh

chmod 750 setup_mon.sh

#Install Prometheus, node_exporter, grafana-server

sudo ./setup_mon.sh

--Prometheus (default): http://127.0.0.1:9090/metrics

--Node exp metrics: http://127.0.0.1:9091

--Node metrics: http://127.0.0.1:12798

--Grafana (default): http://0.0.0.0:5000

sudo systemctl status prometheus

sudo systemctl status node_exporter

sudo systemctl status grafana-server

# Modify Prometheus config files

cd /opt/cardano/monitoring/prometheus

sudo nano prometheus.yml

# Install Grafana plugin

cd /opt/cardano/monitoring/grafana/bin

sudo ./grafana-cli --pluginsDir "/opt/cardano/monitoring/grafana/data/plugins" plugins install <grafana-plugin-name>

# Grafana plugin name example

-- grafana-clock-panel

-- grafana-googlesheets-datasource


sudo systemctl restart grafana-server

sudo systemctl status grafana-server


# Grafana dashboard ID

-- 10879 Node Exporter 0.16 + for Prometheus Monitoring display board

-- 11074 Node Exporter for Prometheus Dashboard EN v20201010

-- 11207 Node Exporter for Prometheus Dashboard English Version UPDATE 1102

-- 11952 Node Exporter for Prometheus Dashboard EN v20191102

HOST :: Create node_exp_poolstats get data from adapools

# Create pool stats folder

cd /opt/cardano/monitoring

mkdir -p pool.stats

# Create scripts poolstatsUpdater

cd $CNODE_HOME/scripts

sudo nano poolstatsUpdater.sh

---------------------------

#!/bin/bash

#Get TRADA Pool stats

curl https://js.adapools.org/pools/f25197cb96c05ec03b8fc5ef195ea87e76ba8fab3b0dcc0cd499e140/summary.json 2>/dev/null \

| jq '.data | del(.hist_bpe, .handles, .hist_roa, .pool_id_bech32, .db_ticker, .db_name, .db_url, .ticker_orig, .pool_id, .group_basic, .tax_ratio_old, .tax_fix_old, .direct, .db_description)' \

| tr -d \"{},: \

| awk NF \

| sed -e 's/^[ \t]*/tradapool_/' > /opt/cardano/monitoring/pool.stats/tradapools.prom

------------------------

#Save and set permission

sudo chmod +x poolstatsUpdater.sh

# Create node_exp_poolstats service

cd /etc/systemd/system/

sudo nano node_exp_poolstats.service

--------------------------------

[Unit]

Description=Node Exporter

Wants=network-online.target

After=network-online.target

[Service]

User=<$USER>

Restart=on-failure

ExecStart=/opt/cardano/monitoring/exporters/node_exporter --collector.textfile.directory="/opt/cardano/monitoring/pool.stats" --collector.textfile --web.listen-address=":9100"

WorkingDirectory=/opt/cardano/monitoring/exporters

LimitNOFILE=3500

[Install]

WantedBy=default.target

--------------------------------

sudo chmod +x node_exp_poolstats.service

# Set Prometheus config file

cd /opt/cardano/monitoring/prometheus

sudo nano prometheus.yml

--------------------------------

scrape_configs:

- job_name: 'poolstats' # To crape data form text collector source import adapools

scrape_interval: 15s

static_configs:

- targets: ['127.0.0.1:9100']

labels:

alias: 'adapools'

type: 'pool-stats'

--------------------------------

sudo systemctl daemon-reload

sudo systemctl restart node_exporter

sudo systemctl restart prometheus


sudo systemctl enable node_exp_poolstats

sudo systemctl start node_exp_poolstats


sudo systemctl status node_node_exporter

sudo systemctl status prometheus

sudo systemctl status node_exp_poolstats

# CronJob : Daily : Run poolstatsUpdater

crontab -e

------------------

25 5 * * * /opt/cardano/cnode/scripts/poolstatsUpdater.sh

------------------

HOST :: Reverse Proxy Grafana with Nginx

# Install Nginx

sudo apt install nginx

nginx -v

sudo service nginx status

# Create a new Nginx configuration for Grafana

cd /etc/nginx/sites-enabled

sudo nano YOUR-DOMAIN-NAME.conf

And copy/paste the example below ( note: Open Port 80 for HTTP)

--------------------

server {

listen 80;

listen [::]:80;

server_name YOUR-DOMAIN-NAME;


location / {

proxy_pass http://localhost:5000/;

}

}

--------------------

# Save and test the new configuration has no errors

sudo nginx -t

# (Option) Enable permission

sudo chown -R user_name:sudo /etc/nginx/

sudo chown -R user_name:sudo /var/log/

sudo chown -R user_name:sudo /run/nginx.pid

# Restart Nginx

sudo service nginx restart

sudo service nginx status

HOST :: Add SSL for Grafana-server web

sudo apt-get update

sudo apt-get install software-properties-common

sudo add-apt-repository universe

sudo apt-get update

# Install Certbot with the Nginx option (Note: Open Port 443 for HTTPS)

sudo apt-get install certbot python3-certbot-nginx

sudo certbot --nginx --register-unsafely-without-email

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Congratulations! You have successfully enabled https://YOUR-DOMAIN-NAME

You should test your configuration at: https://www.ssllabs.com/ssltest/analyze.html?d=YOUR-DOMAIN-NAME

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

IMPORTANT NOTES:

- Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/YOUR-DOMAIN-NAME/fullchain.pem

Your key file has been saved at: /etc/letsencrypt/live/YOUR-DOMAIN-NAME/privkey.pem

Your cert will expire on 2021-04-19. To obtain a new or tweaked version of this certificate in the future, simply run certbot again with the "certonly" option. To non-interactively renew *all* of your certificates, run "certbot renew"

- Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal.

- If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate

Donating to EFF: https://eff.org/donate-le

# Check changed the settings of your Nginx configuration

cat /etc/nginx/sites-enabled/YOUR-DOMAIN-NAME.conf

# Restart NGINX

sudo service nginx restart

sudo service nginx status

CNODE :: Install node_exporter

cd $CNODE_HOME/scripts

curl -s -o setup_node_exporter.sh https://raw.githubusercontent.com/DamjanOstrelic/Cardano-stuff/master/setup_node_exporter.sh

sudo chmod 700 setup_node_exporter.sh

sudo nano setup_node_exporter.sh

# Modify two command line

-- NEXP_DIR="/opt/cardano/monitoring/node_exporter-1.0.1.linux-amd64"

-- cd /opt/cardano

sudo ./setup_node_exporter.sh