▍Docker Swarm¶

Настройка Docker Swarm кластера¶

Вводная часть¶

ip	hostname	role
192.168.0.5	srv-mainframe	worker
192.168.0.10	srv-nas	worker
192.168.0.20	srv-k8s	manager

В качестве распределенного хранилища будет использоваться Ceph.

Инициализируем кластер¶

Подключаемся к серверу 192.168.0.20, он у нас будет выступать как менеджер кластера и выполняем:

docker swarm init
Swarm initialized: current node (0fj9o5pujcpr7f9peqz3226ma) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-67aol7t1pk4tlf14j1tizzfu6ueqmlvtoeo33qppzoynr9x41w-12s2726265k3idz508i40rqxp 192.168.0.20:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

Посмотрим корректно ли виден узел, выполнив:

docker node ls
ID                            HOSTNAME        STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
5qq5trg5ob8099fp1bfsvsyxa *   srv-k8s         Ready     Active         Leader           24.0.2

Теперь подключим наши оставшиеся два сервера как воркеры:

daffin@srv-nas:~$ docker swarm join --token SWMTKN-1-67aol7t1pk4tlf14j1tizzfu6ueqmlvtoeo33qppzoynr9x41w-12s2726265k3idz508i40rqxp 192.168.0.20:2377
This node joined a swarm as a worker.

daffin@srv-mainframe:~$ docker swarm join --token SWMTKN-1-67aol7t1pk4tlf14j1tizzfu6ueqmlvtoeo33qppzoynr9x41w-12s2726265k3idz508i40rqxp 192.168.0.20:2377
This node joined a swarm as a worker.

Проверяем:

docker node ls
ID                            HOSTNAME        STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
5qq5trg5ob8099fp1bfsvsyxa *   srv-k8s         Ready     Active         Leader           24.0.2
53vf5hrv2axhha67mn7oyjely     srv-mainframe   Ready     Active                          24.0.2
319m269268dotfxgocudztaz2     srv-nas         Ready     Active                          24.0.2

Отлично, теперь у нас есть узел-менеджер и два рабочих узла и все они активны.

Запуск сервисов¶

Для примера запустим Traefik (Обратный прокис сервер с SSL Letsencrypt) + Authelia (Единая точка авторизации) + Portainer (Панель управления swarm кластера) + Nextcloud и другие сервисы для работы всего этого стэка.

nano /srv/docker/nextcloud-stack.yml

version: '3.9'

services:
################################################# MANAGER
  traefik:
    image: traefik
    ports:
      - 80:80
      - 443:443/tcp
      - 443:443/udp
    depends_on:
      - authelia
    environment:
      - SELECTEL_API_TOKEN=dfghdghkdsjfgd986h8ghgfsgh
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /docker/conf/traefik:/cert
    networks:
      - traefik-public
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      update_config:
        parallelism: 1
        monitor: 60s
        failure_action: rollback
        order: start-first
      placement:
        constraints: [node.role == manager]
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.traefik.entrypoints=http"
        - "traefik.http.routers.traefik.rule=Host(`traefik.example.ru`)"
        - "traefik.http.middlewares.traefik-auth.basicauth.users=traefik:$$apr1$$o9hx23SG$$NXWqxvi12VGsqn8yL9tR8/" # traefik passwd123; echo $(htpasswd -nb user password) | sed -e s/\\$/\\$\\$/g
        - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
        - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
        - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
        - "traefik.http.routers.traefik-secure.entrypoints=https"
        - "traefik.http.routers.traefik-secure.rule=Host(`traefik.example.ru`)"
        - "traefik.http.services.traefik-secure.loadbalancer.server.port=8080"
        - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
        - "traefik.http.routers.traefik-secure.tls=true"
        - "traefik.http.routers.traefik-secure.tls.certresolver=selectel"
        - "traefik.http.routers.traefik-secure.tls.domains[0].main=example.ru"
        - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.example.ru"
        - "traefik.http.middlewares.test-ipwhitelist.ipwhitelist.ipstrategy.excludedips=172.18.0.0/24"
        - "traefik.http.middlewares.WhitelistHome.ipwhitelist.sourcerange=172.0.0.0/8, 192.168.0.0/16, 10.0.0.0/8"
        - "traefik.http.routers.traefik-secure.service=api@internal"


  authelia:
    image: authelia/authelia
    volumes:
      - /docker/conf/authelia:/config
    environment:
      - TZ=Europe/Moscow
    networks:
      - traefik-public
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      update_config:
        parallelism: 1
        monitor: 60s
        failure_action: rollback
        order: start-first
      placement:
        constraints: [node.role == manager]
      labels:
        - 'traefik.enable=true'
        - 'traefik.http.routers.authelia.rule=Host(`login.example.ru`)'
        - 'traefik.http.routers.authelia.entrypoints=https'
        - 'traefik.http.routers.authelia.tls=true'
        - 'traefik.http.middlewares.authelia.forwardauth.address=http://authelia:9091/api/verify?rd=https://login.example.ru/'
        - 'traefik.http.middlewares.authelia.forwardauth.trustForwardHeader=true'
        - 'traefik.http.middlewares.authelia.forwardauth.authResponseHeaders=Remote-User, Remote-Groups, Remote-Name, Remote-Email'
        - 'traefik.http.middlewares.authelia-basic.forwardauth.address=http://authelia:9091/api/verify?auth=basic'
        - 'traefik.http.middlewares.authelia-basic.forwardauth.trustForwardHeader=true'
        - 'traefik.http.middlewares.authelia-basic.forwardauth.authResponseHeaders=Remote-User, Remote-Groups, Remote-Name, Remote-Email'
        - "traefik.http.services.authelia.loadbalancer.server.port=9091"
        - "traefik.docker.network=traefik-public"


  portainer:
    image: portainer/portainer-ce
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    environment:
      - TZ=Europe/Moscow
    volumes:
      - /docker/conf/portainer:/data
    networks:
      - traefik-public
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      update_config:
        parallelism: 1
        monitor: 60s
        failure_action: rollback
        order: start-first
      placement:
        constraints: [node.role == manager]
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.portainer.rule=Host(`portainer.example.ru`)"
        - "traefik.http.routers.portainer.entrypoints=https"
        - "traefik.http.routers.portainer.tls=true"
        - "traefik.http.routers.portainer.middlewares=WhitelistHome,authelia@docker"
        - "traefik.http.services.portainer.loadbalancer.server.port=9000"

################################################# GLOBAL

  agent:
    image: portainer/agent
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    environment:
      AGENT_CLUSTER_ADDR: tasks.agent
    networks:
      - traefik-public
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
      update_config:
        parallelism: 1
        monitor: 60s
        failure_action: rollback
        order: start-first

################################################# WORKER

  mysql:
    image: mariadb:10.8
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-S", "/var/lib/mysql/mysqld.sock", "--silent"]
      interval: 5s
      timeout: 5s
      retries: 10
    ports:
      - "3306:3306"
    volumes:
      - /docker/data/mysql:/var/lib/mysql
      - /etc/localtime:/etc/localtime:ro
    networks:
      - traefik-public
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      update_config:
        parallelism: 1
        monitor: 60s
        failure_action: rollback
        order: start-first
      placement:
        constraints: [node.role == worker]

  mysql-backup:
    image: mariadb:10.8
    environment:
      - BACKUP_NUM_KEEP=30
      - BACKUP_FREQUENCY=1d
      - ROOT=password12345
    volumes:
      - /backup/mysql_dump:/dump
      - /etc/localtime:/etc/localtime:ro
    entrypoint: |
      bash -c 'bash -s <<EOF
      trap "break;exit" SIGHUP SIGINT SIGTERM
      sleep 2m
      while /bin/true; do
        mysqldump -h mysql -p$$ROOT --events --routines --triggers -A --opt | gzip -c > /dump/dump_\`date +%d-%m-%Y"_"%H_%M_%S\`.sql.gz
        ls -tr /dump/dump_*.sql.gz | head -n -"$$BACKUP_NUM_KEEP" | xargs -r rm
        sleep $$BACKUP_FREQUENCY
      done
      EOF'
    networks:
      - traefik-public
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      update_config:
        parallelism: 1
        monitor: 60s
        failure_action: rollback
        order: start-first
      placement:
        constraints: [node.role == worker]


  phpmyadmin:
    image: phpmyadmin/phpmyadmin
    environment:
      - PMA_ARBITRARY=0
      - PMA_HOSTS=mysql
      - UPLOAD_LIMIT=100M
      - HIDE_PHP_VERSION=true
    volumes:
      - /docker/conf/phpmyadmin/session:/sessions
      - /etc/localtime:/etc/localtime:ro
    networks:
      - traefik-public
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      update_config:
        parallelism: 1
        monitor: 60s
        failure_action: rollback
        order: start-first
      placement:
        constraints: [node.role == worker]
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.pma.tls=true"
        - "traefik.http.routers.pma.rule=Host(`phpmyadmin.example.ru`)"
        - "traefik.http.routers.pma.middlewares=WhitelistHome"
        - "traefik.http.services.pma.loadbalancer.server.port=80"


  redis:
    image: redis:7.0-alpine
    command: redis-server --appendonly yes
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
    volumes:
      - /docker/data/redis:/data
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "6379:6379"
    networks:
      - traefik-public
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      update_config:
        parallelism: 1
        monitor: 60s
        failure_action: rollback
        order: start-first
      placement:
        constraints: [node.role == worker]


  nextcloud:
    image: nextcloud:latest
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /docker/data/www/cloud.example.ru:/var/www/html
      - /docker/data/nextcloud:/data
      - type: tmpfs
        target: /tmp
    depends_on:
      - mysql
      - redis
    environment:
      REDIS_HOST: redis
      MYSQL_HOST: mysql:3306
      MYSQL_DATABASE: nextcloud
      MYSQL_USER: nextcloud
      MYSQL_PASSWORD: password12345
      TRUSTED_PROXIES: traefik
      NEXTCLOUD_TRUSTED_DOMAINS: cloud.example.ru
      NEXTCLOUD_DATA_DIR: /data
      SMTP_HOST: smtp.mail.ru
      SMTP_SECURE: tls
      SMTP_PORT: 587
      SMTP_NAME: [email protected]
      SMTP_PASSWORD: password12345
      MAIL_FROM_ADDRESS: [email protected]
    networks:
      - traefik-public
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      update_config:
        parallelism: 1
        monitor: 60s
        failure_action: rollback
        order: start-first
      placement:
        constraints: [node.role == worker]
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.nextcloud.middlewares=nextcloud,nextcloud_redirect"
        - "traefik.http.routers.nextcloud.tls=true"
        - "traefik.http.routers.nextcloud.rule=Host(`cloud.example.ru`)"
        - "traefik.http.routers.nextcloud.middlewares=default-headers@file,nextcloud_redirect"
        - "traefik.http.middlewares.nextcloud.headers.customFrameOptionsValue=ALLOW-FROM https://cloud.example.ru"
        - "traefik.http.middlewares.nextcloud.headers.contentSecurityPolicy=frame-ancestors 'self' cloud.example.ru *.example.ru"
        - "traefik.http.middlewares.nextcloud_redirect.redirectregex.permanent=true"
        - "traefik.http.middlewares.nextcloud_redirect.redirectregex.regex=https://(.*)/.well-known/(card|cal)dav"
        - "traefik.http.middlewares.nextcloud_redirect.redirectregex.replacement=https://$${1}/remote.php/dav/"
        - "traefik.http.services.nextcloud.loadbalancer.server.port=80"
        - "traefik.docker.network=traefik-public"


  croncloud:
    image: nextcloud:latest
    user: www-data
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /docker/data/www/cloud.example.ru:/var/www/html
      - /docker/data/nextcloud:/data
    depends_on:
      - mysql
      - redis
    entrypoint: |
      bash -c 'bash -s <<EOF
        trap "break;exit" SIGHUP SIGINT SIGTERM
        while [ ! -f /var/www/html/config/config.php ]; do
          sleep 1
        done
        while true; do
          php -f /var/www/html/cron.php;
          sleep 5m
        done
      EOF'
    networks:
      - traefik-public
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      update_config:
        parallelism: 1
        monitor: 60s
        failure_action: rollback
        order: start-first
      placement:
        constraints: [node.role == worker]


networks:
  traefik-public:
    external: true

Запускаем

docker stack deploy --with-registry-auth -c /srv/docker/nextcloud-stack.yml stage

Ждём и смотрим статус сервисов:

docker service ls
ID             NAME                          MODE         REPLICAS   IMAGE                                           PORTS
lvrln6c7qees   enlr_agent                    global       3/3        portainer/agent:latest                                  
sei6rvr2cfn5   enlr_authelia                 replicated   1/1        authelia/authelia:latest                        
qmqmrzvnfbir   enlr_croncloud                replicated   1/1        nextcloud:latest                                
vv1vtjktjebn   enlr_mysql                    replicated   1/1        mariadb:10.8                                    *:3306->3306/tcp          
89mgqn411vbl   enlr_mysql-backup             replicated   1/1        mariadb:10.8                                               
prvgl8dmjhur   enlr_nextcloud                replicated   1/1        nextcloud:latest                                
2dbjaaar6wcj   enlr_phpmyadmin               replicated   1/1        phpmyadmin/phpmyadmin:latest                        
x30v9rzouj51   enlr_redis                    replicated   1/1        redis:7.0-alpine                                *:6379->6379/tcp           
ju4c6ch2h4j4   enlr_traefik                  replicated   1/1        traefik                                         *:80->80/tcp, *:443->443/tcp, *:443->443/udp

Сервисы стартанули, можно пробовать проверять

Принудительно обновить сервис

└─> docker service update --image=registry.example.ru/traefik:latest --force test_traefik 

test_traefik
overall progress: 1 out of 1 tasks 
1/1: running   [==================================================>] 
verify: Service converged

Ограничения по ресурсам

    deploy:
      mode: replicated
      replicas: 2
      # service resource management
      resources:
        # Hard limit - Docker does not allow to allocate more
        limits:
          cpus: '0.25'
          memory: 512M
        # Soft limit - Docker makes best effort to return to it
        reservations:
          cpus: '0.25'
          memory: 256M

Неподдерживаемые настройки в docker-compose v2

"build",
"cap_add",
"cap_drop",
"cgroup_parent",
"devices",
"dns",
"dns_search",
"domainname",
"external_links",
"ipc",
"links",
"mac_address",
"network_mode",
"privileged",
"read_only",
"restart",
"security_opt",
"shm_size",
"stop_signal",
"tmpfs"