Blunix Risk Management¶

This document describes how to use the components provided with Blunix to reduce the risk of downtime or outages on Debian Linux servers and cloud instances.

Risk management
Redundant server setups
Desaster recovery plan

Redundant setups¶

This chapter describes how to failover redundant setups

MariaDB Master-Slave replication¶

MariaDB can be setup accompanied by the role-mariadb-replication-scripts. The role installs BASH scripts below /root/mariadb-scripts/ on both the master and slave server.

Wording:
db1: normally the master
db2: normally the slave

The roles BASH scripts are used to:

initiate replication: from db1 to db2
failover: promote db2 to be a master
failback: make db1 the active master again and re-add db2 as a slave

Initializing replication¶

After installing MariaDB for the first time on both db1 and db2, use this script on db1 to initialize the replication:

# RUN THIS ON DB 1 !!! (master)
root@cus-www-prod-db-1 ~ # cd mariadb-scripts/
root@cus-www-prod-db-1 ~/mariadb-scripts # ./setup-mastar-slave-replication.sh

This script:

db1: enable read-only mode (SET GLOBAL read_only = 1;)
db2: removes and re-creates all of db1's databases
db1: mysqldump all databases to db2
db2: initiates replication (CHANGE MASTER TO AND START SLAVE)

This means that if replication fails for some reason it can be re-initiated using this script.

Failover - promote slave to be the master¶

This has to be done in two steps:

First, run the script on DB2 !!!

# RUN THIS ON DB 2 !!! (slave)
root@cus-www-prod-db-2 ~ # cd mariadb-scripts/
root@cus-www-prod-db-2 ~/mariadb-scripts # ./promote-slave.sh

This script will:

db1: try to run a SHUTDOWN (with a timeout of 3 seconds)
db2: promote to act as master and accept writes

As the second step, you have to update the /etc/hosts entries on the "webworkers" to point (commonly) the db entry to db2. If in doubt, maybe restart php-fpm or uwsgi or similar to make the application workers use the new connections and kill the old ones.

Failback - use db1 as master again and make db2 a slave again¶

To "fail back", run this script:

# RUN THIS ON DB 1 !!!
root@cus-www-prod-db-2 ~ # cd mariadb-scripts/
root@cus-www-prod-db-2 ~/mariadb-scripts # ./failback.sh

This script will:

db1: make db1 think its a master
db2: enable read only mode (SET GLOBAL read_only = 1;)
db1: removes and re-creates all of db2's databases
db2: mysqldump all databases to db1
db1: enale slave replication using db2 as a master
db1: promote to master
db2: run SHUTDOWN

After this script is finished, re-run setup-mastar-slave-replication.sh on db1 to re-attach db2 as a slave.

Debugging¶

On db1, to show relevant variables and settings in mysql:

root@cus-www-prod-db-2 ~ # cd mariadb-scripts/
root@cus-www-prod-db-2 ~/mariadb-scripts # ./check-settings.sh

Checking if repliction is working:¶

On db1:

root@cus-www-prod-db-2 ~ # cd mariadb-scripts/
root@cus-www-prod-db-2 ~/mariadb-scripts # ./create-test-time.sh

This script will:

db1: create a database called "test"
db1: create a table in the "test" database called "test"
db1: write the current date to that table
db1: SELECT * FROM test
sleep 0.5
db2: SELECT * FROM test

Example output:

[...]
+ mysql --raw --skip-column-names --silent -e 'SELECT * FROM test;' test
2022-01-01 00:00:51
+ sleep 0.5
+ mysql -u root -psecret -h 1.3.5.7 --raw --skip-column-names --silent -e 'SELECT * FROM test;' test
2022-01-01 00:00:51

Make sure that the test date string is replicated to db2.