Replicated Local Storage (RLS) with DRBD9+Xen on Ubuntu 16.04LTS Xenial

Hardware Setup

Server 1 and 2 have 4 1GB nic:
eno1 connected lan
eno2,eno3 and eno4 directly connected their respective ports on the other server for bonding with cat 6 or 7 rj45 cables.

2 raid 1 HD (raid 5 isn’t recommended for DRBD)
sda1 system
sdb1 lvm type for DRBD

Setup network+bonding

on both servers

apt-get install ifenslave bridge-utils 
nano /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual

auto peth0
iface peth0 inet static
	address  # on server2
        bridge_ports eno1
        bridge_stp off    

auto eno2
iface eno2 inet manual
bond-master bond0

auto eno3
iface eno3 inet manual
bond-master bond0

auto eno4
iface eno4 inet manual
bond-master bond0

auto bond0
iface bond0 inet static
      address # on server2
      mtu 9000
      bond-slaves eno2 eno3 eno4
      bond_mode balance-rr
      bond_miimon 100
      bond_downdelay 200
      bond_updelay 200

check my article about bonding here

Install xen

on both servers
check my article about XEN install here

Configure LVM

on both servers

nano /etc/lvm/lvm.conf
write_cache_state = 0

restart computer, bonding and xen should ok.

Add DRBD sources

on both servers

nano /etc/apt/sources.list.d/linbit-ubuntu-linbit-drbd9-stack-xenial.list
deb-src xenial main
apt-get install wget
wget -O- | apt-key add -
apt-key add gpg-pubkey-53B3B037-282B6E23.asc
apt-get update

Install DRBD9

on both servers

apt-get install drbd-utils python-drbdmanage drbd-dkms

Configure the DRBD9 pool

on both servers

vgcreate drbdpool /dev/sdb1
apt-get install thin-provisioning-tools
lvcreate -L 499G --thinpool drbdthinpool drbdpool

Leave a little free space on drbdpool or else it will fail to create the thinpool

Configure DRBD9 cluster

on server 1:
copy key on server2 and test ssh [email protected]

ssh-copy-id [email protected]

on server 2:
copy key on server1 and test ssh [email protected]

ssh-copy-id [email protected]

on server 1:

drbdmanage init
drbdmanage add-node server2

Configuring storage plugins:

drbdmanage modify-config
loglevel = INFO
storage-plugin =
drbdmanage modify-config --node server1
pool-name = drbdthinpool
drbdmanage modify-config --node server2
pool-name = drbdthinpool

Restart both server.

Create disk for our first Xen Virtual machine

on server 1:

drbdmanage add-volume VM-disk 10G --deploy 2
drbdmanage list-resources 
drbdadm status
drbdmanage list-volumes --groupby Size

Check if VM-disk is present on both server

drbdsetup status --verbose --statistics
lvdisplay /dev/drbdpool | grep "LV Path"
  LV Path                /dev/drbdpool/.drbdctrl_0
  LV Path                /dev/drbdpool/.drbdctrl_1
  LV Path                /dev/drbdpool/VM-disk_00

If VM-disk_00 is not on server2, stop right here something wrong with the cluster!
Check drbdmanage config on both node: especially storage-plugin and pool-name
delete VM-disk and restart to apply changes.

Install VM with xen-tools

First i create a base vm on system disk:

xen-create-image --hostname=base --dhcp --vcpus=2 --pygrub --dist=xenial

Then i copy it on the DRBD ressource:

apt-get install dcfldd
dcfldd if=/dev/rootvg/base-disk of=/dev/drbd/by-res/VM-disk/0 bs=1M

Create a xen config:

nano /etc/xen/VM.cfg
bootloader = '/usr/lib/xen-4.6/bin/pygrub'
vcpus       = '20'
memory      = '16384'
root        = '/dev/xvda2 ro'
disk        = [
name        = 'VM'
dhcp        = 'dhcp'
vif         = [ 'mac=00:cc:3E:86:3C:c1,bridge=peth0' ]

on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

Check drbd status:

drbdsetup status
.drbdctrl role:Primary
  volume:0 disk:UpToDate
  volume:1 disk:UpToDate
  server2 role:Secondary
    volume:0 peer-disk:UpToDate
    volume:1 peer-disk:UpToDate

VM role:Secondary
  server2 role:Secondary

As soon as we launch the VM on server 1 the VM DRBD ressource will be automatically promoted to primary
and will not be mountable/usable on server2.

Start vm on server1:

xl create /etc/xen/VM.cfg

Check drbd status:

.drbdctrl role:Primary
  volume:0 disk:UpToDate
  volume:1 disk:UpToDate
  server2 role:Secondary
    volume:0 peer-disk:UpToDate
    volume:1 peer-disk:UpToDate

VM role:Primary
  server2 role:Secondary

drbdmanage common operations

snapshot & duplicate
drbdmanage create-snapshot VM-disk-snap VM-disk server1
drbdmanage restore-snapshot VM-disk2  VM-disk VM-disk-snap
drbdmanage remove-snapshot VM-disk VM-disk-snap

drbdmanage list-snapshots
drbdmanage list-resources
drbdmanage list-volumes

Add 10G to a 50G vm:

drbdmanage resize-volume VM-disk2 0 60GiB

Cannot reduce,rename with drbdmanage yet

drbdmanage list-volumes
drbdmanage list-resources
drbdmanage remove-volume VM-disk 0
drbdmanage remove-resource VM-disk
Show DRBD9 ressources disks
apt-get install tree
tree /dev/drbd/by-res
Backup VM
lvcreate -L 50,05G -n VM-backup /dev/datavg
drbdmanage create-snapshot VM-snap VM server1
ionice -c2 -n7 dcfldd if=/dev/drbdpool/VM.VM-snap_00 of=/dev/datavg/VM-backup bs=1M
drbdmanage remove-snapshot VM VM-snap
Split Brain/Standalone mode Recover

server1 got our latest datas, we need to resync server2:

on server2:

drbdadm disconnect .drbdctrl
drbdadm secondary .drbdctrl
drbdadm connect --discard-my-data .drbdctrl

on server1:

drbdadm disconnect .drbdctrl
drbdadm connect .drbdctrl

on server2:

drbdadm disconnect VM-disk
drbdadm secondary VM-disk
drbdadm connect --discard-my-data VM-disk

on server1:

drbdadm disconnect VM-disk
drbdadm connect VM-disk

Next step configure HA Cluster with PCS

auto start drbd if you won’t use an HA cluster manager

nano /etc/systemd/system/drbdmanaged.service
Description=DRBDManage Service
Requires=dbus.service sshd.service sshd.service

ExecStart=//usr/bin/python /usr/bin/dbus-drbdmanaged-service

systemctl enable drbdmanaged

reboot and check:

systemctl status drbdmanaged -l
drbdsetup status --verbose --statistics


Leave a Reply

Your email address will not be published. Required fields are marked *