How to enable TLS/SSL encryption with Glusterfs storage cluster on Linux

I

setup GlusterFS cluster on Ubuntu Linux. How do I enable network encryption using TLS/SSL for my Glusterfs storage cluster on Linux for security reasons when using over insecure network such as WAN (wide area network) or the Internet? How do I setup I/O encryption and management encryption with Glusterfs?

 

GlusterFS allows you to setup TLS (Transport Layer Security) using the OpenSSL. GlusterFS has two option for TLS/SSL. First, it can do encryption of the I/O communication between servers and clients. Second encryption of the glusterd management server i.e. whenever you issue command line status or create new volumes and so on.

Fig.01: Using SSL with Glusterfs

Files needed for setting up network encryption in GlusterFS

You need to create the following files for TLS/SSL encryption for each node:

/etc/ssl/glusterfs.pem – Your certificate. This file must be created on each node, and you must not share with any other node or any other server in cluster.

/etc/ssl/glusterfs.key – Your private key. This file must be created on each node, and you must not share with any other node or any other server in cluster.

/etc/ssl/glusterfs.ca – Concatenation of all nodes certificate. This file is not unique and must be the identical on all server nodes and clients.

/var/lib/glusterd/secure-access – Enabling TLS encryption on the management path for each node. You need to create this file on each node with the touch command.

Let us say you have a four-node GlusterFS cluster as follows:

IP address Node hostname used for TLS identities for authorization

192.168.2.45 gfs01

192.168.2.46 gfs01

192.168.2.47 gfs01

192.168.2.48 gfs01

The common name (CN), must match the hostname of your node and it must be set using either DNS or /etc/hosts file on each node:

$ cat /etc/hosts

127.0.0.1 localhost

127.0.1.1 ubuntu-box-1

192.168.2.45 gfs01

192.168.2.46 gfs02

192.168.2.47 gfs03

192.168.2.48 gfs04

Using SSL with Glusterfs

Type the following command on each node (gfs01, gfs02, gfs03, and gfs04).

How to generate a private key for each node and client

$ cd /etc/ssl/

$ sudo openssl genrsa -out glusterfs.key 2048

 

Sample outputs:

Generating RSA private key, 2048 bit long modulus

……………………………………………………………..+++

…………………………..+++

e is 65537 (0x10001)

How to generate a a signed certificate for each node

From gfs01 node (note the common name (CN) must match your hostname as set using DNS or /etc/hosts):

$ cd /etc/ssl/

$ sudo openssl req -new -x509 -key glusterfs.key -subj  /CN=gfs01  -out glusterfs.pem

 

From gfs02 node (note the common name (CN) must match your hostname as set using DNS or /etc/hosts):

$ cd /etc/ssl/

$ sudo openssl req -new -x509 -key glusterfs.key -subj  /CN=gfs02  -out glusterfs.pem

 

From gfs03 node (note the common name (CN) must match your hostname as set using DNS or /etc/hosts):

$ cd /etc/ssl/

$ sudo openssl req -new -x509 -key glusterfs.key -subj  /CN=gfs03  -out glusterfs.pem

 

From gfs04 node (note the common name (CN) must match your hostname as set using DNS or /etc/hosts):

$ cd /etc/ssl/

$ sudo openssl req -new -x509 -key glusterfs.key -subj  /CN=gfs04  -out glusterfs.pem

 

For client01 and so on:

$ cd /etc/ssl/

$ sudo openssl req -new -x509 -key glusterfs.key -subj  /CN=client01  -out glusterfs.pem

How create the certificates of the Certificate Authorities (CA) file

From gfs01 (type command on gfs01 node) copy all the .pem files from all other nodes including clients as follows:

$ mkdir /tmp/ca/

$ cd /tmp/ca/

$ scp root@gfs02:/etc/ssl/glusterfs.pem gfs02.pem

$ scp root@gfs03:/etc/ssl/glusterfs.pem gfs03.pem

$ scp root@gfs04:/etc/ssl/glusterfs.pem gfs04.pem

copy file from client01 too ##

$ scp root@client01:/etc/ssl/glusterfs.pem client01.pem

 

Type the following command to concatenate the collected files into a single file:

this file is for all server nodes only ###

$ cat /etc/ssl/glusterfs.pem gfs02.pem gfs03.pem gfs04.pem client01.pem > glusterfs.ca

this file is for all clients only ###

$ cat /etc/ssl/glusterfs.pem gfs02.pem gfs03.pem gfs04.pem > glusterfs-client.ca

 

Now put glusterfs.ca file on all the server nodes:

$ sudo cp glusterfs.ca /etc/ssl/

$ scp glusterfs.ca root@gfs02:/etc/ssl/

$ scp glusterfs.ca root@gfs03:/etc/ssl/

$ scp glusterfs.ca root@gfs04:/etc/ssl/

now put file on clients only ###

$ scp glusterfs-client.ca root@client01:/etc/ssl/glusterfs.ca

$ rm -rf /tmp/ca/

How to enable management encryption on server nodes

Type the following command on each node (gfs01,gfs02,gfs03,gfs04):

$ sudo touch /var/lib/glusterd/secure-access

 

Restart the glusterd on all server:

$ systemctl restart glusterfs-server.service

How to enable management encryption on client nodes

Assuming that your volume name is gvol0. Type the following command on all the clients:

$ sudo touch /var/lib/glusterd/secure-access

 

Unmount the volume on all the clients:

$ sudo umount /var/www

 

Mount the volume on all the clients:

$ sudo mount -t glusterfs gfs01:/gvol0 /mnt/www/

How do I enable TLS/SSL I/O encryption for a volume named gvol0?

First stop the volume gvol0 to enable SSL on, run (type all commands on gfs01 node):

$ sudo gluster volume stop gvol0

You need to nmount the volume on all the clients/servers (if any) ##

restart glusterfs-server.service on nodes such as gfs01,gfs02,gfs03,gfs04,client01 ##

$ sudo sudo umount /var/www

$ sudo systemctl restart glusterfs-server.service

 

Next, type the following command that includes names of all the servers and client you want to give access to the volume. The syntax is:

$ sudo gluster volume set {VOLNAME} auth.ssl-allow  {server1,server2,client1,client2,….}

 

For example:

$ sudo gluster volume set gvol0 auth.ssl-allow  gfs01,gfs02,gfs03,gfs04,client01

 

In this example, all TLS authenticated clients mount and access your volume:

$ sudo gluster volume set gvol0 auth.ssl-allow  *

 

Turn on the client side ssl for the volume:

$ gluster volume set gvol0 client.ssl on

 

Turn on the server side ssl for the volume:

$ gluster volume set gvol0 server.ssl on

 

Start the volume, run:

$ sudo gluster vol start gvol0

 

You need mount the volume on all the clients/servers (if any):

$ sudo mount -t glusterfs gfs01:/gvol0 /mnt/www/

 

Verify that SSL is enabled on the I/O path for the volume named gvol0:

$ sudo grep -i ssl /var/log/glusterfs/bricks/nodirectwritedata-brick1-gvol0.log

150:     option auth.login./nodirectwritedata/brick1/gvol0.ssl-allow gfs01,gfs02

152:     option transport.socket.ssl-enabled on

[2017-03-07 08:24:38.893211] I [socket.c:343:ssl_setup_connection] 0-tcp.gvol0-server: peer CN = gfs01

[2017-03-07 08:24:38.893248] I [socket.c:346:ssl_setup_connection] 0-tcp.gvol0-server: SSL verification succeeded (client: 192.168.2.45:49149)

[2017-03-07 08:24:39.146162] I [socket.c:343:ssl_setup_connection] 0-tcp.gvol0-server: peer CN = gfs02

[2017-03-07 08:24:39.146197] I [socket.c:346:ssl_setup_connection] 0-tcp.gvol0-server: SSL verification succeeded (client: 192.168.2.46:49148)

References

Setting up GlusterFS with SSL/TLS from the GlusterFS administrator guide.

Red Hat GlusterFS Guide

This entry is 3 of 4 in the GlusterFS Tutorial series. Keep reading the rest of the series:

How to install GlusterFS on a Ubuntu Linux

How to mount Glusterfs volumes inside LXC/LXD

How to enable TLS/SSL encryption with Glusterfs storage

How to add a new brick to an existing replicated GlusterFS volume on Linux

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *