How to install GlusterFS with a replicated high availability storage volume on Ubuntu Linux 16.04 LTS


want to share my web server DocumentRoot /var/www/html/ across 2 Apache web server. Both of my web servers are behind load balanced reverse proxy powered by Nginx. How do I setup and install GlusterFS (“distributed storage”) which is a scalable network filesystem on Ubuntu Linux 16.04 LTS server?


GlusterFS is a free and open source network distributed storage file system. Network distributed storage file systems are very popular amoung high trafficked websites, cloud computing servers, streaming media services, CDN (content delivery networks) and more. This tutorial shows you how to install GlusterFS on Ubuntu Linux 16.04 LTS server and configure 2 nodes high availability storage for your web server.

Our sample setup

Fig.01: Our scale-out network-attached storage file system for /var/www/html is made of two nodes


A single cloud or bare metal server is going to be a single point of failure. For example /var/www/html/ can be a single point of failure. So you deployed two Apache web-servers. However, how do you make sure /var/www/html/ synced with both Apache server? You do not want to server different images or data to clients. To keep your /var/www/html/ in sync you need a clustered storage. Even if one node goes down the other keeps working. Moreover, when failed node comes online, it should sync missing file from another server in /var/www/html/.

What you need to setup GlusterFS with a replicated high availability storage?

Minimum two separate storage (can be cloud or bare metal boxes)

Each server needs a separate partition or disk for GlusterFS. For demo purpose I am using /dev/vdb1 as a brick volume.

A private network (LAN/VLAN) between servers

Ubuntu Linux 16.04 LTS on both servers

Okay enough talk, lets get started with GlusterFS installation and configuration on a Ubuntu.

For demo purpose I used two nodes. But you must use 4 or more node in production to avoid split brain issues.

Step 1 – Setup /etc/hosts

First setup /etc/hosts files:

$ sudo vi /etc/hosts


Set correct private IP address as per fig.01 or as per your setup: gfs01    gfs02

Close and save the file. Test it:

$ ping -c2 gfs01

$ ping -c2 gfs02


Sample outputs:

PING gfs01 ( 56(84) bytes of data.

64 bytes from gfs01 ( icmp_seq=1 ttl=64 time=0.033 ms

64 bytes from gfs01 ( icmp_seq=2 ttl=64 time=0.036 ms


— gfs01 ping statistics —

2 packets transmitted, 2 received, 0% packet loss, time 999ms

rtt min/avg/max/mdev = 0.033/0.034/0.036/0.006 ms

Step 2 – Configure iptables to accept all traffic between gfs01 and gfs02

Type the following on gfs01 to allow all traffic from lan node gfs02:

$ sudo ufw allow proto any from to


Type the following on gfs02 to allow all traffic from lan node gfs01:

$ sudo ufw allow proto any from to

Step 3 – Configure glusterfs stable repository

Type the following command on both gfs01 and gfs02 server to install latest stable glusterfs server:

$ sudo add-apt-repository ppa:gluster/glusterfs-3.11

$ sudo apt-get update


Sample outputs:

Fig.02: Use the following above code to install latest stable package

Step 4 – Install glusterfs stable package

Type the following apt-get command/apt command on both gfs01 and gfs02 servers:

$ sudo apt-get install glusterfs-server


Sample outputs:

Fig.03: Install server and client components

Step 4 – Hold glusterfs package upgrade for safety reasons

Your shared storage brick might corrup if glusterfs upgraded while running it. It is best to filter out of automatic updates for safety. Later you learn how to upgrade glusterfs safety. Type the following command on both gfs01 and gfs02 server:

$ sudo apt-mark hold glusterfs*


Sample outputs:

glusterfs-client set on hold.

glusterfs-common set on hold.

glusterfs-server set on hold.

glusterfs-dbg set on hold.

Step 5 – Setup and format disk in each server

You need to type the following commands on gfs01 server (be careful with device names while partition block devices):

$ sudo fdisk /dev/vdb

$ sudo mkfs.ext4 /dev/vdb1

$ sudo mkdir -p /nodirectwritedata/brick1

$ sudo echo  /dev/vdb1 /nodirectwritedata/brick1 ext4 defaults 1 2  >> /etc/fstab

$ mount -a

$ sudo mkdir /nodirectwritedata/brick1/gvol0

$ df -H


You need to type the following commands on gfs02 server:

$ sudo fdisk /dev/vdb

$ sudo mkfs.ext4 /dev/vdb1

$ sudo mkdir -p /nodirectwritedata/brick2

$ sudo echo  /dev/vdb1 /nodirectwritedata/brick2 ext4 defaults 1 2  >> /etc/fstab

$ mount -a

$ df -H

$ sudo mkdir /nodirectwritedata/brick2/gvol0


Sample outputs:

Fig.04: Prepare the bricks by setting up partition block devices and formatting with ext4 file system

Warning: Do not edit or write files directly to a /nodirectwritedata/brick1/ or /nodirectwritedata/brick2/ brick on each server. A direct write will corrupt your volume.

Step 6 – Verify that glusterfs service started

Type the following commands:

$ sudo systemctl status glusterfs-server.service


Sample outputs:

? glusterfs-server.service – LSB: GlusterFS server

Loaded: loaded (/etc/init.d/glusterfs-server; bad; vendor preset: enabled)

Active: active (running) since Tue 2017-02-28 00:07:22 IST; 26min ago

Docs: man:systemd-sysv-generator(8)

Tasks: 7

Memory: 16.3M

CPU: 1.412s

CGroup: /system.slice/glusterfs-server.service

??3429 /usr/sbin/glusterd -p /var/run/


Feb 28 00:07:20 ubuntu-box2 systemd[1]: Starting LSB: GlusterFS server…

Feb 28 00:07:20 ubuntu-box2 glusterfs-server[3420]:  * Starting glusterd service glusterd

Feb 28 00:07:22 ubuntu-box2 glusterfs-server[3420]:    …done.

Feb 28 00:07:22 ubuntu-box2 systemd[1]: Started LSB: GlusterFS server.


If not running start it:

$ sudo systemctl start glusterfs-server.service


Enable glusterfs at boot time:

$ sudo systemctl enable glusterfs-server.service


Sample outputs:

glusterfs-server.service is not a native service, redirecting to systemd-sysv-install

Executing /lib/systemd/systemd-sysv-install enable glusterfs-server

Step 7 – Configure the trusted pool

From gfs01 server type:

$ sudo gluster peer probe gfs02

peer probe: success. Host gfs02 port 24007 already in peer list


From gfs02 server type:

$ sudo gluster peer probe gfs01

peer probe: success. Host gfs01 port 24007 already in peer list

Step 8 – Set up a GlusterFS volume

From gfs01 (or gfs02) sever type:

# gluster volume create gvol0 replica 2 gfs01:/nodirectwritedata/brick1/gvol0 gfs02:/nodirectwritedata/brick2/gvol0


Sample outputs:

volume create: gvol0: success: please start the volume to access data

# gluster volume start gvol0


Sample outputs:

volume start: gvol0: success

To see status, enter:

# gluster volume status

Status of volume: gvol0

Gluster process                             TCP Port  RDMA Port  Online  Pid

Brick gfs01:/nodirectwritedata/brick1/gvol0 49152     0          Y       2621

Brick gfs02:/nodirectwritedata/brick2/gvol0 49152     0          Y       3674

Self-heal Daemon on localhost               N/A       N/A        Y       3694

Self-heal Daemon on gfs01                   N/A       N/A        Y       2641


Task Status of Volume gvol0

There are no active volume tasks

To see info about your volume, enter:

# gluster volume info


Sample outputs:


Volume Name: gvol0

Type: Replicate

Volume ID: 5d871eed-182e-461f-9eb6-06798eeb2c04

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 2 = 2

Transport-type: tcp


Brick1: gfs01:/nodirectwritedata/brick1/gvol0

Brick2: gfs02:/nodirectwritedata/brick2/gvol0

Options Reconfigured:

transport.address-family: inet

performance.readdir-ahead: on

nfs.disable: on

Step 9 – Client mounts

Now our cluster is up and running. It is the time our clients can mount shared stoage. You can have 1,2,3,4 or as many client you want as per your network resources. You can configure gfs01 and gfs02 as client itself too.

Client mount on gfs01/gfs02 node itself

Type the following command:

$ sudo mkdir /mnt/www/

$ sudo mount -t glusterfs gfs01:/gvol0 /mnt/www/


Update the /etc/fstab file:

echo  gfs01:/gvol0 /mnt/www glusterfs defaults,_netdev 0 0  >> /etc/fstab

Save and close the file. Mount it:

$ sudo mount -a

$ df -H


Sample outputs:

Filesystem                           Size  Used Avail Use% Mounted on

udev                                 981M     0  981M   0% /dev

tmpfs                                138M  2.5M  136M   2% /run

/dev/mapper/ubuntu–box–1–vg-root   37G  2.0G   33G   6% /

tmpfs                                489M     0  489M   0% /dev/shm

tmpfs                                5.0M     0  5.0M   0% /run/lock

tmpfs                                489M     0  489M   0% /sys/fs/cgroup

/dev/vda1                            472M  105M  343M  24% /boot

/dev/vdb1                            4.8G   11M  4.6G   1% /nodirectwritedata/brick1

tmpfs                                 98M     0   98M   0% /run/user/0

gfs02:/gvol0                         4.8G   11M  4.6G   1% /mnt/www

Client mount on a different server named www1

If www1 is using a Debian/Ubuntu Linux OS, type:

$ sudo add-apt-repository ppa:gluster/glusterfs-3.11

$ sudo apt-get update

$ sudo apt-get install glusterfs-client

$ sudo mkdir -p /mnt/www/


Edit /etc/hosts file and update it as follows: gfs01    gfs02

Save and close the file. Update /etc/fstab file to mount a shared storage:

Update the /etc/fstab file:

echo  gfs01:/gvol0 /mnt/www glusterfs defaults,_netdev 0 0  >> /etc/fstab

Save and close the file. Mount it:

$ sudo mount -a

$ df -H

Step 10 – Test it

Cd /mnt/www/ and create some files on www1 or gfs02:

# cd /mnt/www

# mkdir test

# cd test

# cp /etc/*.conf .


You should see files on both gfs02, gfs01 and www1.

A note about setting

Finally run the following command on both gfs02 and gfs01 to set network ping timeout to 5 seconds from default 42:

$ sudo gluster volume set gvol0  5


Verify it

$ sudo gluster volume get gvol0

Option                                  Value

——                                  —–                    5

Otherwise glusterfs client will hang out when other node goes down for 42 seconds. You just reduced this to 5 seconds.

A note about Apache DocumentRoot

You need to edit httpd.conf and set DocumentRoot to point to /mnt/www/ directory.


This entry is 1 of 4 in the GlusterFS Tutorial series. Keep reading the rest of the series:

How to install GlusterFS on a Ubuntu Linux

How to mount Glusterfs volumes inside LXC/LXD

How to enable TLS/SSL encryption with Glusterfs storage

How to add a new brick to an existing replicated GlusterFS volume on Linux



Leave a Reply

Your email address will not be published. Required fields are marked *