Tired of having virtualbox and a thousand docker containers dragging your macbook air down in the mud? Put your docker host in the clouds!
In this post we will create a docker host running in the cloud and hook that up to our local development environment using a SSH based "VPN".
Her is a diagram.
+---------------+ SSH +----------------+ | development | tun0 tunnel tun0 | docker host | | | <------------------------------->| | | (client) | 10.0.0.1 10.0.0.2 | (cloud_server) | +-------+-------+ point to point connection +-------+--------+ eth0 | | eth0 192.168.0.2 | | 10.0.0.10 | _ _ | docker0 | ( ` )_ | 172.17.42.1 ----------------- ( ) `) ------------------- (_ (_ . _) _) INTERNET
The Cloud Server
Pick a cloud, any cloud.
$ ssh <cloud_server_ip>
Follow the appropriate instructions to install docker. By default the docker daemon binds to a unix socket only. To be able to communicate with docker over the network, we need to bind to a network interface.
$ sudo vi /etc/init/docker.conf "$DOCKER" -d $DOCKER_OPTS -H unix:///var/run/docker.sock -H tcp://0.0.0.0:4243
WARNING! Binding to
0.0.0.0 will expose the docker host on all network interfaces on the server!! That might not be what you want. If you only want to talk to the docker host over the tunnel, you can bind to
10.0.0.2. Let's keep it exposed while we are testing, but remember to go back and fix that later.
To make this work, we need to permit root login over ssh. Now, that might seem like a security hazard. But, we are only going to allow login using keys (not passwords), and later on we are going to limit the access of root over ssh to only run a specific command.
$ sudo vi /etc/ssh/sshd_config PermitRootLogin yes PermitTunnel yes PasswordAuthentication no # <- optional, but recommended! $ sudo service ssh restart
Friendly tip: When modifying sshd_config and restarting the ssh service; test your new configuration on a different session/terminal, keeping the one you already have open, just in case you messed up something.
We also need to remove any initial scripting under root's autorized_keys added to prevent login as root.
$ sudo vi /root/.ssh/authorized_keys
Remove anything resembling the following:
no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"ubuntu\" rather than the user \"root\".';echo;sleep 10"
...so that your file starts with
$ sudo su -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
If your on OSX (like me); install tuntap.
Grab the docker client!
$ curl https://get.docker.io/builds/Darwin/x86_64/docker-latest -o docker $ chmod +x docker $ sudo cp docker /usr/local/bin/
Connection to your docker host is as simple as:
(client) $ sudo ssh -w 0:0 -i key.pem root@<cloud_server_ip>
-w flag to
ssh will have ssh create a tunnel device on each end of the connection. The
0:0 indicates which tunnel device at each end. Since we have specified 0 at each end, both the device on the server and the client will be
tun0. Now we need to route some traffic so we can talk to docker on the server.
Tunnels & Routes & Docker
Setup the tunnel devices:
(server) $ sudo ifconfig tun0 10.0.0.2 pointopoint 10.0.0.1 (client) $ sudo ifconfig tun0 10.0.0.1 10.0.0.2 (client) $ ping 10.0.0.2 PING 10.0.0.2 (10.0.0.2): 56 data bytes 64 bytes from 10.0.0.2: icmp_seq=0 ttl=64 time=58.643 ms 64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=58.410 ms 64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=58.586 ms ^ woohoo!
(client) $ sudo route -n add -net 10.0.0.0 10.0.0.2 (client) $ sudo route -n add -net 126.96.36.199 10.0.0.2
Set the docker host:
(client) $ export DOCKER_HOST=tcp://172.17.42.1:4243
(client) $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
Addtional setup & security
Now, all this is quite a bit to remember, so I definately recommend scripting parts of the setup. One neat thing you can do that will both simplify the process and strengthen security is to only allow root to run a specific command over ssh, and have that command be opening the tunnel.
First, add the
tun0 interface to
$ sudo vi /etc/network/interfaces iface tun0 inet static address 10.0.0.2 pointopoint 10.0.0.1 netmask 255.255.255.0 # Forward traffic into server side network iptables -t nat -A POSTROUTING --source 10.0.0.2 -j SNAT --to-source 10.0.0.10
Add a command to root's
authorized_keys (first thing in the file) to bring up the interface on connection.
$ sudo vi /root/.ssh/authorized_keys tunnel="0",command="/sbin/ifdown tun0;/sbin/ifup tun0" ssh-rsa ....
Only allow commands for root over ssh.
$ sudo vi /etc/ssh/sshd_config PermitRootLogin forced-commands-only $ sudo service ssh restart
Now, whenever you ssh in as root the server will try to bring up the tunnel interface.
If you want simple dns based service discovery for you containers over you new cloud bridge, apply the same tools and techniques discussed in my previous article Vagrant Skydocking to this cloud bridge. You will be pinging
redis.staging.yourapp in no time.
I have been applying this technique for connecting to different VPCs I manage on amazon. Once scripted it's really nice being able to connect to a specific environment with one command and be hands on the docker host(s) and the containers in that environment.
Originally I had been hoping to also use a VPC for development purposes. Unfortunately the latency from my location to the datacenter gets annoying when trying to get efficient feedback loops. This is however not a fault of this approach, but the distance from me to the datacenter. Your milage may vary. Setting up a development host closer to home did the trick.
Time to dance!