How to run a command on every node in a Docker Swarm cluster

Iain Collins

Nov 15, 2017·3 min read

Docker Swarm is a great way to deploy Docker Containers on AWS or Azure, both of which are supported cloud providers for Docker Community Edition (CE) and Enterprise Edition (EE).

A common admin task can be needing to run a command on all nodes on a cluster, either for monitoring or when debugging problems.

Executing Containers on all nodes in a cluster is easy, and the docker stats command lets you easily view the memory, CPU, network and IO usage of all Containers on an individual node.

However, unless you are running the Enterprise Edition or something like the Portainer management UI, there is easy way to see the status of all containers at once — and there is no built-in docker command for running commands directly on nodes.

Combine docker, cut and xargs

If you have a Docker Swarm cluster you will already be familiar with using the docker node command to list nodes in the cluster:

> docker node ls
ID                          HOSTNAME             STATUS 
2v77j8x4ljejeeku9tao00tcw * swarm-manager000000  Ready ox633ovtcp5ux986fb30x6d6x   swarm-manager000001  Ready 
qfig0i98hhxdekf7wfld8jo7q   swarm-worker000000   Ready 

If you want to run commands on all nodes in a cluster, it’s pretty easy to combine this output with xargs using cut and ssh.

Examples

These examples assume you have SSH access to nodes already setup — this is the default behaviour after a Docker Swarm cluster has been setup with Docker Community Edition or Docker Enterprise Edition — if you can’t access all the nodes via SSH when logged in to the cluster you should resolve that first.

Checking uptime and load on all nodes

docker node ls | cut -c 31-49 | grep -v HOSTNAME | xargs -I"SERVER" sh -c "echo SERVER; ssh SERVER uptime"swarm-manager000000
 03:21:23 up 42 days,  6:26,  load average: 0.22, 0.14, 0.10
swarm-manager000001
 03:33:42 up 41 days, 14:27,  load average: 0.00, 0.02, 0.00
swarm-worker000000
 03:33:42 up 40 days,  9:33,  load average: 0.05, 0.03, 0.00

Checking free memory (in MB) on all nodes

docker node ls | cut -c 31-49 | grep -v HOSTNAME | xargs -I"SERVER" sh -c "echo SERVER; ssh SERVER free -m"swarm-manager000000
             total  used  free  shared  buffers  cached
Mem:          6956  5068  1888     162      182    1329
-/+ buffers/cache:  3555  3400
Swap: 0 0 0
swarm-manager000001
             total  used  free  shared  buffers  cached
Mem:           6956 6158   797     162     2320    2301
-/+ buffers/cache:  1537  5419
Swap: 0 0 0
swarm-worker000000
             total  used  free  shared  buffers  cached
Mem:          6956  6075   880     162     1956    2636
-/+ buffers/cache:  1482  5473
Swap: 0 0 0

Get detailed info about all containers

Being able to view detailed info for all containers on the cluster can make identifying problematic services and resolving performance problems much easier.

You can easily get a list of all running containers using docker node:

docker node ps $(docker node ls -q) | grep Running

However, if you want detailed information about CPU, memory, network and IO usage you can get that by combining docker node, cut, xargs and ssh then invoking docker stats on each node:

docker node ls | cut -c 31-49 | grep -v HOSTNAME | xargs -I"SERVER" sh -c "echo SERVER; ssh SERVER /usr/local/bin/docker stats --no-stream"

Hopefully this has been helpful. If you have any tips for good Docker management tools would be great if you could share them below!