Docker Swarm and Shared Storage Volumes

As we know, volumes provide a flexible and powerful way to add persistent storage to managed dockers, but what should we do if we want to share storage volumes across several Docker hosts, for instance, a Swarm cluster? In this topic, we will consider a simple method of creating shared volumes usable across several swarm nodes using the sshfs volume driver. This is a great alternative to cloud based volumes using AWS EBS

How it works?

Considering the case when the majority of applications are data oriented there is a requirement to share files/directories amongst each other. In fact, it also requires files and directories to be saved and distributed.
We will consider vieux plugin which allows us to mount remote folder using sshFS in our containers easily. Please note that this plugin supports volume drivers only. Now, we will consider a simple exmplae of how to operate with sshFS plugin.

• First step, you need to install the plugin:

docker plugin install vieux/sshfs

Accept the request of the plugin privileges by typing y

• Now, check whether plugin was successfully installed:

docker plugin ls

You are good to go if you see true under ENABLED column

• Next, try to create a volume using installed plugin:

docker volume create -d vieux/sshfs --name sshfsvolume -o sshcmd=username@hostname:/some/path mysshfsvolume

• Verify that plugin created the volume:

docker volume ls

• Great, now you can use created volume:

docker run -it --rm -v sshfsvolume:/path busybox ls /path

You will get the content of remote machine hostname:/some/path
Note: You can stop and/or remove plugin with the following commands:

docker plugin disable
docker plugin remove

Objective

Let’s say you want to run an Apache service in a Docker Swarm cluster with several replicas, but you want to run these containers with a shared customized Apache configuration file. This file can be used for all mentioned replicas and you want to store this file in a certain location so that the Apache configuration can be changed without the need to re-create running containers.  

You have to create a shared storage volume housed on a storage server. This volume should be accessible to all containers in the cluster, doesn’t matter of which node they are running on. And it should contain the Apache configuration file, and should be mounted to each of the containers replica.

Prerequisites

• Swarm Manager (SM) with /etc/httpd/conf directory
• Swarm Worker (SW) with /etc/httpd/conf directory
• Storage Server (SS) with /etc/docker/shared directory
• Volume Storage (VS) will be created using vieux/sshfs driver
• swarmuser on SM, SW and SS

Setup

1. In order to set up the external storage location you have to create shared directory at /etc/docker/shared on Storage Server. Please make sure swarmuser can read and write to the created directory:

sudo mkdir -p /etc/docker/shared
sudo chown swarmuser:swarmuser /etc/docker/shared

2. Next step would be to copy Apache configuration file to create directory on Storage Server:

cp /etc/httpd/conf/httpd.conf /etc/docker/shared

3. Now, we have to install the vieux/sshfs docker plugin on Swarm Manager cluster and Swarm Worker nodes:

docker plugin install --grant-all-permissions vieux/sshfs

4. And last step would be to create an Apache service/container called apache-web using the apache:latest image with N number of replicas. Mount shared apache-vol volume to the containers at /etc/httpd/conf/. Publish port 9773 on the containers to port 8080 on the nodes. Please run the following command on Swarm Manager:

docker service create -d \
  --replicas=<N> \
  --name apache-web \
  -p 8080:9773 \
  --mount volume-driver=vieux/sshfs,source=apache-vol,target=/etc/httpd/conf/,volume-opt=sshcmd=swarmuser@<Storage Server IP>:/etc/docker/shared,volume-opt=password=<swarmuser password> apache:latest

Please note, that docker volume apache-vol was created using the vieux/sshfs driver that stores data in /etc/docker/shared/ on the Storage Server. You should use the swarmuser to do this.
Worth to mention that you should create the volume using docker service create command so that the volume will be configured automatically on all Swarm Workers that execute the service’s tasks.

Now you can verify whether service is working properly:

curl localhost:8080

Or by checking Apache default web page on localhost.

Potential problems and risks

There are several problems you may face during the setup or mount activities such as:

• Error while mounting the volume:

docker: Error response from daemon: error while mounting volume '/var/lib/docker/plugins/<ID>/mnt/volumes/<VolumeID>': VolumeDriver.Mount: exit status 2.
ERRO[0000] error getting events from daemon: request canceled

In this case, you should try to create using the volume using vieux/sshfs:next instead of vieux/sshfs

• Error while trying to run:

docker: Error response from daemon: error while mounting volume '/var/lib/docker/plugins/<ID>/mnt/volumes/<VolumeID>': VolumeDriver.Mount: sshfs command execute failed: exit status 1 (read: Connection reset by peer

That might be a permission issue. Please try to run it as root user or check your key pair:

docker plugin install vieux/sshfs DEBUG=1 sshkey.source=/<username>/.ssh/

or

ssh-copy-id -i /<username>/.ssh/ <username>@<hostname>:/<username/.ssh/>

Note:
Worth to mention that there are several potential risks of using sshFS:

• sshFS will wait forever on a dead connection once server get unreachable
• Creating volumes with vieux/sshfs in the compose file may lead to zombie process accumulation
• sshFS is limited to remote mounting and anvailable on system with FUSE availability only
• sshFS is more of relevance when limited few users/services are performing the access on the filesystem at the same time