Docker.io

Reference Architectures - Docker, Inc. ❗!important
Docker Reference Architecture: Designing Scalable, Portable Docker Container Networks - Docker, Inc.
Deploy and scale containers with Docker native, open source orchestration PyCon 2017 - YouTube

Dockercon 2021 | DockerCon 2021
Docker Production Best Practices from Bret Fisher at DockerCon - YouTube DockerCon 2017

Docker Compose

TODO: remove pre-v3 Compose file

Compose is a tool for defining and running multi-containers Docker applications on a single Docker host or cluster using standalone Docker Swarm.
For multi-hosts deployment on Swarm mode use docker stack on Docker 1.13+.

Best Practices Around Production Ready Web Apps with Docker Compose - YouTube
Best Practices Around Production Ready Web Apps with Docker Compose — Nick Janetakis
Docker Tip #94: Docker Compose v2 and Profiles Are the Best Thing Ever — Nick Janetakis

docker/compose-switch

Compose file can be stacked for environment specific settings.

Docker acquired Orchard and used its Fig as the basis of Docker Compose (released early-2015). See also decking.io.

Compose V2 GA in 2022-04
Announcing Compose V2 General Availability - Docker

Docker Compose | Docker
Overview of Docker Compose | Docker Documentation
Docker Compose Files Version 2 - YouTube
Compose file | Docker Documentation
Compose file version 2 reference | Docker Documentation
Compose file version 3 reference | Docker Documentation
Controlling startup order in Compose | Docker Documentation ❗!important, wait for service
The definitive Guide to Docker compose
Docker Compose tips for any developers - DEV Community 👩‍💻👨‍💻 create a second instance with shared volume to execute commands
How to deploy on remote Docker hosts with docker-compose - Docker Blog

docker-compose | Baeldung

The right way to Docker Compose 🐳 🐙 - YouTube

Compose Version 3 added deploy key, which is ignored in docker-compose up for single node deployment. In swarm mode, docker stack (supported in 1.13+) will read the settings in deploy key and apply them with docker service calls internally.
docker.github.io/compose-file.md at 8524552f99e5b58452fcb1403e1c273385988b71 · aanand/docker.github.io
docker service create | Docker Documentation Template variables for swarm services

docker/awesome-compose: Awesome Docker Compose samples

Docker Compose will BLOW your MIND!! (a tutorial) - YouTube

Docker Tip #60: What Really Happens When You Run docker-compose up? — Nick Janetakis

A beginner’s guide to Docker — how to create a client/server side with docker-compose
How we happily dockerized our development environment (part 1/2) | Hacker Noon
Dockerized development environment on steroids (part 2/2) | Hacker Noon

From Docker to Docker compose way! - DEV Community
Introduction to Docker Compose Tool for Multi-Container Applications | Linux.com | The source for Linux information
Docker Volumes and Networks with Compose | Linux.com | The source for Linux information
Docker tutorial: Get started with Docker Compose | InfoWorld
How to Run Multiple Containers with Docker Compose
You’re using docker-compose wrong - Earthly Blog

使用 docker-compose 进行 python 开发 | YL Notes
Dockerizing Flask with Compose and Machine - from localhost to the cloud - Real Python realpython/orchestrating-docker

Docker Compose Tutorial: Orchestrate Containers for Development | Codeship | via @codeship
Orchestrate Containers for Development with Docker Compose | via @codeship
A Docker Container Pattern - Compose Configuration - Levvel
Beyond Docker Compose
Easy deploy a nodejs app to prod and staging with docker cascading Compose files

Docker Compose UI by Francesco Uliana

Examples

bitnami-docker/subrion.md at master · bitnami/bitnami-docker
bitnami-docker/redmine.md at master · bitnami/bitnami-docker
https://raw.githubusercontent.com/sameersbn/docker-redmine/master/docker-compose.yml
https://raw.githubusercontent.com/sameersbn/docker-gitlab/master/docker-compose.yml

Dockage

devops.md#Uptime Kuma

Dockge
louislam/dockge: A fancy, easy-to-use and reactive self-hosted docker compose.yaml stack-oriented manager
Dockge: A New Way To Manage Your Docker Containers - YouTube

Docker Swarm (standalone)

Docker Swarm is native clustering for Docker Engines. Exposes Docker Remote API to be utilized by standard Docker tools and other tools. Swarm comes with a default scheduler, but it can be replaced by others (e.g. Aurora, Kubernetes).

Swarm, unlike Kubernetes, is not declarative.

Docker acquired Conductant (founded by the creators of Aurora) in March 2016. We're probably going to see Aurora features in Swarm.

#Docker Swarm mode

Docker Swarm | Docker
Swarm Overview

Docker Tutorial 11 - Docker Swarm - YouTube with hosted discovery
Libswarm (in a nutshell) · Container42
Have Docker, Will Swarm — On Docker — Medium setup Swarm in a SINGLE machine

SwarmWeek: It’s a Wrap – Thanks Y’all! | Docker Blog
SwarmWeek – Video Tutorials to Join Your First Docker Swarm Cluster | Docker Blog
Docker Swarm - YouTube
ScaleSwarm: Auto-scaling a swarm cluster - YouTube
Sentient Clusters: Micro-services, Atlassian products, and the Docker Swarm - Nicola Paolucci - YouTube 30 minutes walk-through

Docker Machine, Compose & Swarm Tutorials | Codeship | via @codeship
Running Services within a Docker Swarm - via @codeship | via @codeship ❗!important
linux - how to create docker overlay network between multi hosts? - Stack Overflow
Levvel Blog - Running a Distributed Docker Swarm on AWS
NGINX as a Reverse Proxy for Docker Swarm Clusters | via @codeship
Review: Docker Swarm soars, and the sky's the limit | InfoWorld
Deploying Containers with Docker Swarm and Docker Networking | Technology Conversations code
Docker Austin: How Do I Even Swarm? | Carina by Rackspace
Distributed Data Analysis with Docker Swarm | via @codeship naive map-reduce with Docker Swarm
Evaluating Container Platforms at Scale — On Docker — Medium
Docker Swarm: Flat File Engine Discovery — On Docker — Medium
Rescheduling containers on node failures with Docker Swarm 1.1 - Container Solutions
Couchbase Cluster on Docker Swarm using Docker Compose and Docker Machine
Deploy Docker Swarm cluster on one host | Nan Xiao's Blog

Docker Swarm Exceeds Kubernetes Performance at Scale | Docker Blog
Docker Swarm beats Kubernetes? Not so fast | InfoWorld

Scaling To Infinity with Docker Swarm, Docker Compose and Consul

Scaling To Infinity: The Quest For Fully Automated, Scalable, Self-Healing System With Zero-Downtime

Scaling To Infinity with Docker Swarm, Docker Compose and Consul (Part 1/4) – A Taste of What Is To Come | Technology Conversations
Scaling To Infinity with Docker Swarm, Docker Compose and Consul (Part 2/4) – Manually Deploying Services | Technology Conversations
Scaling To Infinity with Docker Swarm, Docker Compose and Consul (Part 3/4) – Blue-Green Deployment, Automation and Self-Healing Procedure | Technology Conversations
Scaling To Infinity with Docker Swarm, Docker Compose and Consul (Part 4/4) – Scaling Individual Services | Technology Conversations

vfarcic/docker-swarm

Docker Swarm mode

Docker 1.12 (June 2016) included Swarm mode (via SwarmKit) for Swarm capability without Docker Swarm standalone.
Docker 1.12: Now with Built-in Orchestration! | Docker Blog
DockerCon 2016: What's New in Docker - Docker 1.12 | via @codeship
Comparing Swarm, Swarmkit and Swarm Mode | Sreenivas Makam's Blog
swarm/README.md at master · docker/swarm
Docker tutorial: Get started with Docker swarm mode | InfoWorld
Container Orchestration with Docker and Swarm
What is Docker Swarm Mode and When Should You Use It? – CloudSavvy IT

Overview - User Guide| Alibaba Cloud Documentation Center Docker Swarm standalone vs Swarm mode
Docker Swarm vs Swarm Mode | Neil Cresswell | Pulse | LinkedIn

Docker Swarm In A Nutshell
4 Important Services Everyone Should Deploy In A Docker Swarm

It's 2018, Is Swarm Dead? Answered by a Docker Captain. TL;DR: NO
Future of Docker Swarm in 2018 on Built-in Container Orchestration
Docker Swarm Firewall Ports

Swarm mode overview
Getting started with swarm mode | Docker Documentation
Setting up a Docker Swarm cluster using Docker in Docker | Callista Enterprise
Docker Swarm » ADMIN Magazine
Docker Swarm Mode - YouTube

Swarm mode key concepts | Docker Documentation
Task: A Docker container and the commands to run inside the container. It is the atomic scheduling unit of swarm.
Service: definition of the tasks to execute on the worker nodes.
Stack: a collection of services that make up an application in a specific environment.

How nodes work | Docker Documentation
How services work | Docker Documentation

moby/swarmkit: A toolkit for orchestrating distributed systems at any scale. It includes primitives for node discovery, raft-based consensus, task scheduling and more.

swarmzilla/swarm3k: SwarmZilla 3000 Collaborative Project
Docker Swarm Lessons from Swarm3K
Taming SwarmZilla: 150k Containers in 3K+ Docker Swarm Nodes

swarm - alex ellis' blog
Top 5 of Docker Swarm

Running 1,000 Containers in Docker Swarm - via @codeship | via @codeship ❗!important, kernel tuning
Introduction to Docker Swarm Orchestration - UpCloud
Massively Scalable with Docker SwarmKit — Medium
Trying the new docker 1.12 swarm mode locally using machine – Mantika – Bringing latest research in Machine Learning to industry
All Hail the New Docker Swarm - Container Solutions
Docker Swarm Is Dead. Long Live Docker Swarm.
Running Services within a Docker Swarm - via @codeship | via @codeship
Docker Tips: Rollout and Rollback In Docker Swarm - Better Programming - Medium

One-shot containers with Docker Swarm

docker service create --restart-policy=none --name crawler1 -e url=http://blog.alexellis.io -d crawl_site alexellis2/href-counter

Limitation

The number of connections to ingress loadbalancer (using kernel's IPVS) is bound by the number of available ports per source IP. So high concurrency access will result in connections waiting.

IPVS Software - Advanced Layer-4 Switching
IPVS - LVSKB

[SWARM] Very poor performance for ingress network with lots of parallel requests · Issue #35082 · moby/moby
Pauses/delays with overlay network on swarm · Issue #31746 · moby/moby

attachable network

Docker Stacks and Attachable networks

docker network create --driver=overlay --attachable attachable-overlay
docker service create --publish 80:80 --network=attachable-overlay --name nginx nginx
docker run --network=attachable-overlay -ti alpine:latest sh

When using with stack deploy, define the network as external.

networks:
  attachable-overlay:
    external: true

swarm network

How Docker Swarm Container Networking Works - Under the Hood

Service Discovery and Load balancing Internals in Docker 1.12 – Sreenivas Makam's Blog DNS strategy explained with Docker Swarm
Use swarm mode routing mesh | Docker Documentation
Solving the routing mess for services using Docker – Luis Herrera Benítez – Medium
Load Balancing Docker Swarm Mode - UpCloud
Docker 1.12 swarm mode - round robin inside and out

Ingress for Swarm
Home - Docker Flow Proxy

`docker service`

docker service | Docker Documentation
docker service create | Docker Documentation
docker service update | Docker Documentation

A service is defined by an image plus its scheduling options. It is only applicable in Swarm mode.
The containers will be automatically prefixed with the service name.
service ps list the containers and nodes they are running on.
Deploy services to a swarm | Docker Documentation
docker service create | Docker Documentation

--mount option for docker service
Give a service access to volumes or bind mounts | Docker Documentation
Add bind-mounts or volumes | Docker Documentation

Key	Value
`type`	`mount` (default), `bind`, `tmpfs`
`src`	host path if `type`== `mount`; volume name if `type`== `volume`
`dst`/`target`	container path
`volume-driver`	`local` (default), `flocker`, `glusterfs`, `ceph`
`volume-label`	volume label
`volume-opt`	volume option

--mount can specify driver and create volume, --volume (of docker run) can only bind existing volume

Create services using templates | Docker Documentation some values can be set with variables templates

Default host name is container id.
Container name is {{.Service.Name}}.{{.Task.Slot}}.{{.Task.ID}}

`docker stack`

docker stack | Docker Documentation
Get Started, Part 5: Stacks | Docker Documentation
Deploy a stack to a swarm | Docker Documentation

A stack is a collection of services defined in Compose Version 3. It is only applicable in Swarm mode. A stack gets its own overlay network (as in docker-compose).
The services, containers and network will be automatically prefixed with the stack name.

The docker stack deploy command is effectively the equivalent of Docker Compose (a Python app) re-written in Golang.

play-with-docker/stacks: Docker stacks samples

Docker and Swarm Mode – Part 1 | Gabriel Schenker's Blog
Docker and Swarm Mode – Part 2 | Gabriel Schenker's Blog

Docker Compose from development to production – Basilio Vera – Medium
Deploy Docker Compose (v3) to Swarm (mode) Cluster - Codefresh
Deploy Swarm Services with the new docker stack and a compose file!

secrets/config

Manage sensitive data with Docker secrets | Docker Documentation
Use Secrets in Compose
Compose file version 3 reference - secrets | Docker Documentation

Store configuration data using Docker Configs | Docker Documentation
Compose file version 3 reference - configs | Docker Documentation

The Definitive Cheatsheet for Docker Secrets
How to create a Docker secret and use it to deploy a service | TechRepublic

vs Kubernetes

Docker Inevitably Embraces Kubernetes Container Orchestration
Docker Clustering Tools Compared: Kubernetes vs Docker Swarm | Technology Conversations

Docker Swarm vs. Kubernetes: What You Really Need To Know - DZone Cloud
Kubernetes vs. Docker: Comparing The Two Container Orchestration Giants! - DZone Cloud

CLI

Logging into container of a stack

# get unique container token
CONTAINER_TOKEN=$(docker service ps  -f 'name=STACK_SERVICE.1' STACK_SERVICE -f desired-state=running -q --no-trunc )
docker exec -ti STACK_SERVICE.1.${CONTAINER_TOKEN} bash

Script for distributed use case with docker-machine (from this post):

#! /bin/bash

set -e

exec_task=$1
exec_instance=$2

strindex() {
  x="${1%%$2*}"
  [[ "$x" = "$1" ]] && echo -1 || echo "${#x}"
}

parse_node() {
  read title
  id_start=0
  name_start=`strindex "$title" NAME`
  image_start=`strindex "$title" IMAGE`
  node_start=`strindex "$title" NODE`
  dstate_start=`strindex "$title" DESIRED`
  id_length=name_start
  name_length=`expr $image_start - $name_start`
  node_length=`expr $dstate_start - $node_start`

  read line
  id=${line:$id_start:$id_length}
  name=${line:$name_start:$name_length}
  name=$(echo $name)
  node=${line:$node_start:$node_length}
  echo $name.$id
  echo $node
}

if true; then
   read fn
   docker_fullname=$fn
   read nn
   docker_node=$nn
fi < <( docker service ps -f name=$exec_task.$exec_instance --no-trunc -f desired-state=running $exec_task | parse_node )

echo "Executing in $docker_node $docker_fullname"

eval `docker-machine env $docker_node`

docker exec -ti $docker_fullname /bin/bash

Docker Machine

You can use Docker Machine to:

Provision and manage supported infrastructures
Provision Swarm clusters

Machine Overview
Machine concepts and help

Configs (usually certs) are stored in .docker/machine.
Do note that generic driver will change the hostname and (re)install docker on the host to Docker service will be restarted/

Java Magazine, January/February 2016, (72)

Docker Swarm and Docker Compose with Ben Firshman, Product Manager at Docker, Inc. - YouTube (little bit out-dated)

Provision

docker-machine create | Docker Documentation
Machine drivers | Docker Documentation

Use --engine-install-url with Rancher's install script to install a particular version of Docker on the nodes.

--engine-install-url=https://get.docker.com/
--engine-install-url=https://test.docker.com/
--engine-install-url=https://releases.rancher.com/install-docker/1.13.sh
--engine-install-url=https://releases.rancher.com/install-docker/17.05.sh

Operation

# assuming a node `node1` is provisioned
docker-machine ls

# set and under Docker environment
eval $(docker-machine env node1)
eval $(docker-machine env -u)

docker-machine ssh node1
docker-machine inspect node1 -f {{ .Driver.IPAddress }} # `node_ip`
docker run -d -p 8000:80 --name webserver nginx:alpine
# visit http://<node_ip>:8080
docker rm -f $(docker ps -ql)

docker-machine stop node1
docker-machine rm node1

Experimenting with Native Docker tooling · Container42 deploy on Digital Ocean

AWS

Amazon Web Services (AWS) EC2 example | Docker Documentation

Create IAM account and generate access key pair.
Save to ~/.aws/credentials:

[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY

docker-machine create --driver amazonec2 --amazonec2-region us-west-2 node1
# us-west-1a is sometime full, cannot allocate subnet

Amazon Web Services | Docker Documentation

Digital Ocean

Digital Ocean example | Docker Documentation

Create API token

docker-machine create --driver digitalocean --digitalocean-access-token xxxxx node1

Digital Ocean | Docker Documentation