Docker swarm cluster at DCCN

The first swarm cluster at DCCN was developed in order to deploy and manage service components (e.g. DICOM services, data streamer, data stager) realising the automatic lab-data flow. The inital setup consists of 8 nodes repurposed from the HPC and the EXSi clusters.

System architecture

All docker nodes are bare-matel machines running CentOS operating system. The nodes are provisioned using the DCCN linux-server kickstart. They all NFS-mount the /home and /project directories, and use the active directory service for user authentication and authorisation. Only the TG members are allowed to SSH login to the docker nodes.

All docker nodes also NFS-mount the /mnt/docker directory for sharing container data. The figure below shows the architecture of the DCCN swarm cluster.

_images/dccn_swarm_architecture.png

Fig. 1 The DCCN swarm cluster - a simplified illustration of the architecture.

Image registry

Within the swarm cluster, a private image registry is provided to as a central repository of all container images. The data store of the registry is located in /mnt/docker/registry which is a shared NFS volume on the central storage.

The registry endpoint is docker-registry.dccn.nl:5000. It requires user authentication for uploading (push) and downloading (pull) container images. New user can be added by using the script /mnt/docker/scripts/microservices/registry/add-user.sh.

An overview of image repositories can be browsed here.

Note

For the sake of simplicity, the internal private registry is using a self-signed X.509 certificate. In order to trust it, one needs to copy the certificate of the docker registry server to the docker host, under the directory, e.g. /etc/docker/certs.d/docker-registry.dccn.nl:5000/ca.crt.

Service orchestration

For deploying multiple service components as a single application stack, the docker compose specification v3 is used together with the docker stack management interface (i.e. the docker stack command).

An example docker-compose file for orchestrating three services for the data-stager application is shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
version: "3"

services:

    db:
        image: docker-registry.dccn.nl:5000/redis
        volumes:
            - /mnt/docker/data/stager/ui/db:/data
        networks:
            default:
                aliases:
                    - stagerdb4ui
        deploy:
            placement:
                constraints: [node.labels.function == production]

    service:
        image: docker-registry.dccn.nl:5000/stager:1.7.0
        ports:
            - 3100:3000
        volumes:
            - /mnt/docker/data/stager/config:/opt/stager/config
            - /mnt/docker/data/stager/cron:/cron
            - /mnt/docker/data/stager/ui/log:/opt/stager/log
            - /project:/project
            - /var/lib/sss/pipes:/var/lib/sss/pipes
            - /var/lib/sss/mc:/var/lib/sss/mc:ro
        networks:
            default:
                aliases:
                    - stager4ui
        environment:
            - REDIS_HOST=stagerdb4ui
            - REDIS_PORT=6379
        depends_on:
            - db
        deploy:
            placement:
                constraints: [node.labels.function == production]

    ui:
        image: docker-registry.dccn.nl:5000/stager-ui:1.1.0
        ports:
            - 3080:3080
        volumes:
            - /mnt/docker/data/stager/ui/config:/opt/stager-ui/config
        networks:
            default:
                aliases:
                    - stager-ui
        depends_on:
            - service
        deploy:
            placement:
                constraints: [node.labels.function == production]

networks:
    default:

Whenever the docker compose specification is not applicable, a script to start a docker service is provided. It is a bash script wrapping around the docker service create command.

All the scripts are located in the /mnt/docker/scripts/microservices directory.