Configure Highly Available HAProxy with Keepalived on Ubuntu 20.04

George · Сен. 14, 2022, 01:16

In this guide, we are going to learn how to configure highly available HAProxy with Keepalived on Ubuntu 20.04. Keepalived can work with HAProxy to provide the failover services to a backup route in the event that a specific route becomes unavailable. This ensures a more robust and scalable high availability environment.

While Keepalived uses Linux virtual server (LVS) to perform load balancing and failover tasks on the active and passive routers, HAProxy performs load balancing and high-availability services to TCP and HTTP applications.

Keepalived utilizes Virtual Router Redundancy Protocol to send periodic advertisements between the master (active) and backup (passive) LVS routers (which in our case is the HAProxy servers since are load balancing web apps) to determine the state of each other. If a master server fails to advertise itself within a predefined period of time, Keepalived initiates failover and the backup server becomes the master.

All virtual servers are assigned a Virtual IP, also known as floating IP. This is a publicly routable IP/address. It is assigned dynamically to an active server at any given time.

Configure Highly Available HAProxy with Keepalived on Ubuntu 20.04
This setup requires that you already have an HAProxy server setup and running. We have covered the installation and setup of HAProxy load balancer on various systems in our previous guides;

Install and Setup HAProxy on Ubuntu 20.04

Install and Setup HAProxy on CentOS 8

Setup HAProxy Load Balancer on Fedora 30/Fedora 29
Configure HAProxy Load Balancer with SSL on Ubuntu 18.04/Debian 10/9

In this tutorial, we will be using two HAProxy servers with Keepalived for high availability. Below is our deployment architecture.

haproxy keepalived
Assuming that you already have HAProxy setup, proceed with installation and configuration of Keepalived on Ubuntu 20.04.

Install and Configure Keepalived on Ubuntu 20.04
Install Keepalived on HAProxy Servers
In our demo environment, we are running HAProxy servers on Ubuntu 20.04. Therefore, assuming your system package cache is up-to-date, run the command below install Keepalived on Ubuntu 20.04

root@lb01:~# apt install keepalived
Similarly, install Keepalived on second HAProxy server.

root@lb02:~# apt install keepalived
Configure IP forwarding and non-local binding
To enable Keepalived service to forward network packets to the backend servers, you need to enable IP forwarding. Run this command on both HAProxy servers;
sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.conf
Similarly, you need to enable HAProxy and Keepalived to bind to non-local IP address, that is to bind to the failover IP address (Floating IP).

echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf
Reload sysctl settings;

sysctl -p
Configure Keepalived
The default configuration file for Keepalived should be /etc/keepalived/keepalived.conf. However, this configuration is not created by default. Create the configuration with the content below;

vim /etc/keepalived/keepalived.conf
Keepalived Configuration for Master Node (lb01)
# Global Settings for notifications
global_defs {
notification_email {
id@domain.com # Email address for notifications
}
notification_email_from id@domain.com # The from address for the notifications
smtp_server 127.0.0.1 # SMTP server address
smtp_connect_timeout 15
}

# Define the script used to check if haproxy is still working
vrrp_script chk_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
weight 2
}

# Configuration for Virtual Interface
vrrp_instance LB_VIP {
interface enp0s8
state MASTER # set to BACKUP on the peer machine
priority 101 # set to 99 on the peer machine
virtual_router_id 51

smtp_alert # Enable Notifications Via Email

authentication {
auth_type AH
auth_pass myP@ssword # Password for accessing vrrpd. Same on all devices
}

# The virtual ip address shared between the two loadbalancers
virtual_ipaddress {
192.168.100.200
}

# Use the Defined Script to Check whether to initiate a fail over
track_script {
chk_haproxy
}
}
Keepalived Configuration for Backup Node (lb02)

global_defs {
notification_email {
id@domain.com
}
notification_email_from id@domain.com
smtp_server 127.0.0.1
smtp_connect_timeout 15
}

vrrp_script chk_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
weight 2
}

vrrp_instance LB_VIP {
interface enp0s8
state BACKUP
priority 100
virtual_router_id 51

smtp_alert

authentication {
auth_type AH
auth_pass myP@ssword
}

virtual_ipaddress {
192.168.100.200
}

track_script {
chk_haproxy
}
}
Note that the notification section is optional. You can however follow the link below to install and configure Postfix to use Gmail SMTP as relay;
Configure Postfix to Use Gmail SMTP on Ubuntu 20.04

Read more about the configuration parameters used above on Keepalived man pages and Keepalived Configuration synopsis.

Running Keepalived on Ubuntu 20.04
You can now start and enable Keepalived to run on system boot on all nodes;
systemctl enable --now keepalived
Check the status on Master Node;
systemctl status keepalived
● keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2020-06-06 18:24:20 UTC; 6s ago
Main PID: 7097 (keepalived)
Tasks: 2 (limit: 2282)
Memory: 2.1M
CGroup: /system.slice/keepalived.service
├─7097 /usr/sbin/keepalived --dont-fork
└─7107 /usr/sbin/keepalived --dont-fork

Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Initial state master is incompatible with AH authentication - clearing
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: Registering gratuitous ARP shared channel
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Entering BACKUP STATE (init)
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: Remote SMTP server [127.0.0.1]:25 connected.
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: VRRP_Script(chk_haproxy) succeeded
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Changing effective priority from 100 to 102
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: SMTP alert successfully sent.
Jun 06 18:24:23 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Entering MASTER STATE
Jun 06 18:24:23 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: Remote SMTP server [127.0.0.1]:25 connected.
Jun 06 18:24:23 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: SMTP alert successfully sent
You can as well check the status on the slave node.

Check the IP address assigment;

On the master node;

ip --brief add
lo UNKNOWN 127.0.0.1/8 ::1/128
enp0s3 UP 10.0.2.15/24 fe80::a00:27ff:fe9d:888e/64
enp0s8 UP 192.168.100.81/24 192.168.100.200/32 fe80::a00:27ff:feba:9e8c/64
enp0s9 UP 192.168.57.7/24 fe80::a00:27ff:fe31:b7db/64
See the highlighted line, it is assigned a floating IP, 192.168.100.200/32.
On the slave;

lo UNKNOWN 127.0.0.1/8 ::1/128
enp0s3 UP 10.0.2.15/24 fe80::a00:27ff:fefe:fc06/64
enp0s8 UP 192.168.100.80/24 fe80::a00:27ff:fe75:9eae/64
enp0s9 UP 192.168.58.4/24 fe80::a00:27ff:fef4:41/64
Now, let us take down interface enp0s8, on the master node. Be sure to login from the console before you can take the interface down.

ip link set enp0s8 down
Check Keepalived status on the backup node;

systemctl status keepalived
● keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2020-06-06 19:24:27 UTC; 26min ago
Main PID: 9948 (keepalived)
Tasks: 2 (limit: 2282)
Memory: 2.3M
CGroup: /system.slice/keepalived.service
├─9948 /usr/sbin/keepalived --dont-fork
└─9949 /usr/sbin/keepalived --dont-fork

Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: Registering gratuitous ARP shared channel
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: (LB_VIP) Entering BACKUP STATE (init)
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: Remote SMTP server [127.0.0.1]:25 connected.
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: VRRP_Script(chk_haproxy) succeeded
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: (LB_VIP) Changing effective priority from 99 to 101
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: SMTP alert successfully sent.
Jun 06 19:50:39 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: (LB_VIP) Entering MASTER STATE
Jun 06 19:50:39 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: Remote SMTP server [127.0.0.1]:25 connected.
Jun 06 19:50:40 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: SMTP alert successfully sent.
Configure HAProxy with Keepalived Virtual IP
The only change that we are going to make on our HAProxy configuration file is to adjust the frontend bind IP and the stats IP address to the VIP or the floating IP, which in our case is 192.168.100.200

vim /etc/haproxy/haproxy.cfg
...
frontend kifarunixlb
bind 192.168.100.200:443 ssl crt /etc/ssl/certs/haproxy.pem
default_backend webapps
option forwardfor
...
listen stats
bind 192.168.100.200:8443 ssl crt /etc/ssl/certs/haproxy.pem
stats enable
...
Save and exit the config file.

Restart HAProxy.

systemctl restart haproxy
You should now be able to access your web apps even when one of the HAProxy servers goes down via the floating IP. Sample screenshot for our setup is in below;

vip app01
That marks the end of our guide on how to configure highly available HAProxy with Keepalived on Ubuntu 20.04

George · Сен. 14, 2022, 01:35

Итак, виртуальный ip настроен, отказоустойчивость получена. Настраиваем синхронизацию нужных файлов (конфиг и сертификаты)

Для синхронизации будем использовать unison
устанавливаем на оба сервера

Код: bash

apt install unison

Далее настраиваем (если не настроена ранее) ssh связь между серверами
vi /etc/ssh/sshd_config
and change

PermitRootLogin without-password
to

PermitRootLogin yes
afterwards, restart the ssh-daemon:

service ssh restart
On server1:

Create a private/public key pair:

ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): <-- ENTER
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): <-- ENTER
Enter same passphrase again: <-- ENTER
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
f3:d0:62:a7:24:6f:f0:1e:d1:64:a9:9f:12:6c:98:5a root@server1
The key's randomart image is:
+---[RSA 2048]----+
| |
| . |
| + |
| + * |
| E S + |
| o O @ . |
| . B + |
| o o |
| . |
+-----------------+

It is important that you do not enter a passphrase otherwise the mirroring will not work without human interaction so simply hit ENTER!

Next, we copy our public key to server2.example.tld:

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.0.106
The authenticity of host '192.168.0.106 (192.168.0.106)' can't be established.
ECDSA key fingerprint is 25:d8:7a:ee:c2:4b:1d:92:a7:3d:16:26:95:56:62:4e.
Are you sure you want to continue connecting (yes/no)? <-- yes (you will see this only if this is the first time you connect to server2)
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.0.106's password: <- enter root password from server2
Now try logging into the machine:

ssh root@192.168.0.106
And check /root/.ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting.

cat /root/.ssh/authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAACBAPhiAexgEBexnw0rFG8lXwAuIsca/V+lhmv5lhF3BqUfAbL7e2sWlQlGhxZ8I2UnzZK8Ypffq6Ks+lp46yOs7MMXLqb7JBP9gkgqxyEWqOoUSt5hTE9ghupcCvE7rRMhefY5shLUnRkVH6hnCWe6yXSnH+Z8lHbcfp864GHkLDK1AAAAFQDddQckbfRG4C6LOQXTzRBpIiXzoQAAAIEAleevPHwi+a3fTDM2+Vm6EVqR5DkSLwDM7KVVNtFSkAY4GVCfhLFREsfuMkcBD9Bv2DrKF2Ay3OOh39269Z1rgYVk+/MFC6sYgB6apirMlHj3l4RR1g09LaM1OpRz7pc/GqIGsDt74D1ES2j0zrq5kslnX8wEWSHapPR0tziin6UAAACBAJHxgr+GKxAdWpxV5MkF+FTaKcxA2tWHJegjGFrYGU8BpzZ4VDFMiObuzBjZ+LrUs57BiwTGB/MQl9FKQEyEV4J+AgZCBxvg6n57YlVn6OEA0ukeJa29aFOcc0inEFfNhw2jAXt5LRyvuHD/C2gG78lwb6CxV02Z3sbTBdc43J6y root@server1.example.tld
Disallow root-login with a password. Open /etc/sshd_config:

vi /etc/ssh/sshd_config
and change

PermitRootLogin yes
to

PermitRootLogin without-password
afterwards, restart the ssh-daemon:

service ssh restart
Logout from server2:

exit
logout
Connection to 192.168.0.106 closed.

на сервере1:

Код: bash

mkdir /root/.unison

Код: bash

nano /root/.unison/haproxy.prf

... and add the following content:

# Roots of the synchronization
root = /etc/haproxy
root = ssh://192.168.0.106//etc/haproxy

# Paths to synchronize
#path = www
#path = vmail

# Some regexps specifying names and paths to ignore
#ignore = Path stats ## ignores /var/www/stats
#ignore = Path stats/* ## ignores /var/www/stats/*
#ignore = Path */stats ## ignores /var/www/somedir/stats, but not /var/www/a/b/c/stats
#ignore = Name *stats ## ignores all files/directories that end with "stats"
#ignore = Name stats* ## ignores all files/directories that begin with "stats"
#ignore = Name *.tmp ## ignores all files with the extension .tmp
#ignore = Name sess_*
#ignore = Name *access.log*
ignore = Name errors
#ignore = Name webalizer.conf

# When set to true, this flag causes the user interface to skip
# asking for confirmations on non-conflicting changes. (More
# precisely, when the user interface is done setting the
# propagation direction for one entry and is about to move to the
# next, it will skip over all non-conflicting entries and go
# directly to the next conflict.)
auto=true

# When this is set to true, the user interface will ask no
# questions at all. Non-conflicting changes will be propagated;
# conflicts will be skipped.
batch=true

# !When this is set to true, Unison will request an extra
# confirmation if it appears that the entire replica has been
# deleted, before propagating the change. If the batch flag is
# also set, synchronization will be aborted. When the path
# preference is used, the same confirmation will be requested for
# top-level paths. (At the moment, this flag only affects the
# text user interface.) See also the mountpoint preference.
confirmbigdel=true

# When this preference is set to true, Unison will use the
# modification time and length of a file as a `pseudo inode
# number' when scanning replicas for updates, instead of reading
# the full contents of every file. Under Windows, this may cause
# Unison to miss propagating an update if the modification time
# and length of the file are both unchanged by the update.
# However, Unison will never overwrite such an update with a
# change from the other replica, since it always does a safe
# check for updates just before propagating a change. Thus, it is
# reasonable to use this switch under Windows most of the time
# and occasionally run Unison once with fastcheck set to false,
# if you are worried that Unison may have overlooked an update.
# The default value of the preference is auto, which causes
# Unison to use fast checking on Unix replicas (where it is safe)
# and slow checking on Windows replicas. For backward
# compatibility, yes, no, and default can be used in place of
# true, false, and auto. See the section "Fast Checking" for more
# information.
fastcheck=true

# When this flag is set to true, the group attributes of the
# files are synchronized. Whether the group names or the group
# identifiers are synchronizeddepends on the preference numerids.
group=true

# When this flag is set to true, the owner attributes of the
# files are synchronized. Whether the owner names or the owner
# identifiers are synchronizeddepends on the preference
# extttnumerids.
owner=true

# Including the preference -prefer root causes Unison always to
# resolve conflicts in favor of root, rather than asking for
# guidance from the user. (The syntax of root is the same as for
# the root preference, plus the special values newer and older.)
# This preference is overridden by the preferpartial preference.
# This preference should be used only if you are sure you know
# what you are doing!
prefer=newer

# When this preference is set to true, the textual user interface
# will print nothing at all, except in the case of errors.
# Setting silent to true automatically sets the batch preference
# to true.
silent=true

# When this flag is set to true, file modification times (but not
# directory modtimes) are propagated.
times=false

# When this flag is set, Unison will log all changes to the filesystems on a file.
log=false

# When this flag is set to true, groups and users are synchronized numerically, rather than by name.
# The special uid 0 and the special group 0 are never mapped via user/group names even if this
# preference is not set.
numericids=true

второй файл -

Код: bash

nano /etc/root/.unison/ssl.prf

Код: bash

mkdir /root/.unison
vi /root/.unison/web.prf
... and add the following content:


# Roots of the synchronization
root = /etc/ssl
root = ssh://192.168.0.106//etc/ssl
 
# Paths to synchronize
#path = www
#path = vmail
 
# Some regexps specifying names and paths to ignore
#ignore = Path stats    ## ignores /var/www/stats
#ignore = Path stats/*  ## ignores /var/www/stats/*
#ignore = Path */stats  ## ignores /var/www/somedir/stats, but not /var/www/a/b/c/stats
#ignore = Name *stats   ## ignores all files/directories that end with "stats"
#ignore = Name stats*   ## ignores all files/directories that begin with "stats"
#ignore = Name *.tmp    ## ignores all files with the extension .tmp
ignore = Name sess_*
ignore = Name *access.log*
ignore = Name error.log
ignore = Name webalizer.conf
   
#          When set to true, this flag causes the user interface to skip
#          asking for confirmations on non-conflicting changes. (More
#          precisely, when the user interface is done setting the
#          propagation direction for one entry and is about to move to the
#          next, it will skip over all non-conflicting entries and go
#          directly to the next conflict.)
auto=true
   
#          When this is set to true, the user interface will ask no
#          questions at all. Non-conflicting changes will be propagated;
#          conflicts will be skipped.
batch=true
   
#          !When this is set to true, Unison will request an extra
#          confirmation if it appears that the entire replica has been
#          deleted, before propagating the change. If the batch flag is
#          also set, synchronization will be aborted. When the path
#          preference is used, the same confirmation will be requested for
#          top-level paths. (At the moment, this flag only affects the
#          text user interface.) See also the mountpoint preference.
confirmbigdel=true
   
#          When this preference is set to true, Unison will use the
#          modification time and length of a file as a `pseudo inode
#          number' when scanning replicas for updates, instead of reading
#          the full contents of every file. Under Windows, this may cause
#          Unison to miss propagating an update if the modification time
#          and length of the file are both unchanged by the update.
#          However, Unison will never overwrite such an update with a
#          change from the other replica, since it always does a safe
#          check for updates just before propagating a change. Thus, it is
#          reasonable to use this switch under Windows most of the time
#          and occasionally run Unison once with fastcheck set to false,
#          if you are worried that Unison may have overlooked an update.
#          The default value of the preference is auto, which causes
#          Unison to use fast checking on Unix replicas (where it is safe)
#          and slow checking on Windows replicas. For backward
#          compatibility, yes, no, and default can be used in place of
#          true, false, and auto. See the section "Fast Checking" for more
#          information.
fastcheck=true

#          When this flag is set to true, the group attributes of the
#          files are synchronized. Whether the group names or the group
#          identifiers are synchronizeddepends on the preference numerids.
group=true
   
#          When this flag is set to true, the owner attributes of the
#          files are synchronized. Whether the owner names or the owner
#          identifiers are synchronizeddepends on the preference
#          extttnumerids.
owner=true
   
#          Including the preference -prefer root causes Unison always to
#          resolve conflicts in favor of root, rather than asking for
#          guidance from the user. (The syntax of root is the same as for
#          the root preference, plus the special values newer and older.)
#          This preference is overridden by the preferpartial preference.
#          This preference should be used only if you are sure you know
#          what you are doing!
prefer=newer
   
#          When this preference is set to true, the textual user interface
#          will print nothing at all, except in the case of errors.
#          Setting silent to true automatically sets the batch preference
#          to true.
silent=true
   
#          When this flag is set to true, file modification times (but not
#          directory modtimes) are propagated.
times=false

#          When this flag is set, Unison will log all changes to the filesystems on a file.
log=false

#          When this flag is set to true, groups and users are synchronized numerically, rather than by name.
#          The special uid 0 and the special group 0 are never mapped via user/group names even if this
#          preference is not set.
numericids=true

И создаем скрипты запуска
mkdir /root/scripts
vi /root/scripts/unison.sh
and add the following content:

#!/bin/sh
lockdir=/tmp
UNISON=/usr/bin/unison
LOGGER=/usr/bin/logger

if [ ! -f /$lockdir/unison ]; then
touch $lockdir/unison

$UNISON -testserver haproxy
rc=$?
if [[ $rc != 0 ]] ; then
echo "error"
$LOGGER -d -t unison "web - error connecting remote"
else
$UNISON haproxy
fi

rm $lockdir/unison
else
$LOGGER -d -t unison "unison already running"
fi

vi /root/scripts/unison-ssl.sh
and add the following content:

#!/bin/sh
lockdir=/tmp
UNISON=/usr/bin/unison
LOGGER=/usr/bin/logger

if [ ! -f /$lockdir/unison ]; then
touch $lockdir/unison

$UNISON -testserver ssl
rc=$?
if [[ $rc != 0 ]] ; then
echo "error"
$LOGGER -d -t unison "web - error connecting remote"
else
$UNISON ssl
fi

rm $lockdir/unison
else
$LOGGER -d -t unison "unison already running"
fi

READY! Осталось только запихнуть в crontab (crontab -e)

Моя шпаргалка :)

Новости:

Configure Highly Available HAProxy with Keepalived on Ubuntu 20.04

George

George