Vagrant skydocking

by Asbjorn Enge

I've been working quite a bit with docker lately. If you haven't yet checked it out, it's about time. Docker is already popping paradigms.

A bridge over vagrant water

Since I'm on OSX I'm running my docker host on Virtualbox via Vagrant.

Instead of having to forward ports and using lots of -p args when spawning containers, I wanted to bridge my host and the vm's docker interface, so that I could ping my containers from my OSX terminal.

Create a private_network in your Vagrantfile. I'm picking an ip on a different subnet than the docker0 interface to avoid any potential conflicts.

Vagrant::VERSION >= "1.1.0" and Vagrant.configure("2") do |config|
	config.vm.network "private_network", ip: "10.2.0.10", netmask: "255.255.0.0"
	config.vm.provider :virtualbox do |vb|
		vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
	end
end

The vb.customize is to allow forwarding packets for the bridge interface. The --nicpromisc2 translates to Promiscuous mode for nic2, where nic2 -> eth1. So --nocpromisc3 would change that setting for eth2, etc.

After reloading vagrant we need create a route on the host. Basically, any traffic trying to reach the docker subnet (172.17.0.0) should be routed to our new interface inside the vm (10.2.0.10).

# OSX
__aSyNcId_<_pSdeYvbj__gt; sudo route -n add -net 172.17.0.0 10.2.0.10
# Linux (untested)
__aSyNcId_<_pSdeYvbj__gt; sudo route -net 172.17.0.0 netmask 255.255.0.0 gw 10.2.0.10

You now have a bridge from your host to your docker network!!

__aSyNcId_<_pSdeYvbj__gt; IP=`docker inspect -format='{{.NetworkSettings.IPAddress}}' skydns`
__aSyNcId_<_pSdeYvbj__gt; ping $IP
PING 172.17.0.3 (172.17.0.3) 56(84) bytes of data.
64 bytes from 172.17.0.3: icmp_req=1 ttl=64 time=0.232 ms
64 bytes from 172.17.0.3: icmp_req=2 ttl=64 time=0.103 ms
^C
--- 172.17.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1009ms
rtt min/avg/max/mdev = 0.103/0.167/0.232/0.065 ms

Skydock

Docker is all about distributed systems; packing single components inside containers and have them talk to eachother. One of the painpoints when shattering your monolith is linking all those loose components together.

(Docker provides a -link parameter for linking containers. But this quickly falls short in complex scenarios.)

I was just about to dig into service discrovery solutions like etcd or similar, when Michael Crosby posted his skydock (video). It's brilliant! It let's you discover your services via DNS. I won't go into setting up skydock, just check out the awesome tutorial by Michael.

So, with skydock my containers can discover eachother via DNS names like myservice.env.domain.com. Awesome! But, with my network bridge set up, so can my host!! No? That would be really nice for development...

__aSyNcId_<_pSdeYvbj__gt; curl elasticsearch.dev.domain.com:9200
curl: (6) Could not resolve host: elasticsearch.dev.domain.com

﴾͡๏̯͡๏﴿ ... Ah, we need to hook up skydns as a nameserver. This is where I stray a little from Michael's skydock tutorial. I had some issues binding to the docker0 interface (docker v0.7.6), so instead I'm using the skydns container as the nameserver directly (PS! this requires passing a -dns <skydns_ip> arg to each new container). Either way, we have to edit resolv.conf.

__aSyNcId_<_pSdeYvbj__gt; sudo vi /etc/resolv.conf
   # nameserver 172.17.42.1 <- skydock tutorial
   nameserver 172.17.0.3 # <- skydns container ip
__aSyNcId_<_pSdeYvbj__gt; dig elasticsearch.dev.domain.com
;; ANSWER SECTION:
elasticsearch.dev.domain.com.	20	IN	A	172.17.0.7

✌(-‿-)✌ ... Hoplah! Now, hopefully that will be it for you and you're all set to curl containers from the comforts of your host terminal! I however, had one more issue to solve...

__aSyNcId_<_pSdeYvbj__gt; curl elasticsearch.dev.domain.com:9200
curl: (6) Could not resolve host: elasticsearch.dev.domain.com # w00000000t???

OSX weirdness

Apparently OSX is rather weird in how it handles DNS. dig, host, etc. can resolve the host just fine, but other tools like curl and even ping does not obey resolv.conf. I eventually stumbled across the issue and found this script that apparently solves it for most people. It didn't help. Eventually I added the DNS server via OSX network preferences, and that did the trick.

__aSyNcId_<_pSdeYvbj__gt; curl elasticsearch.dev.domain.com:9200
{
	"ok" : true,
	"status" : 200,
	"name" : "Damian, Margo",
	"version" : {
		"number" : "1.0.0.Beta2",
		"build_hash" : "296cfbe390dc51bb00c00ba48ad0c8a9efabcfe9",
		"build_timestamp" : "2013-12-02T15:46:27Z",
		"build_snapshot" : false,
		"lucene_version" : "4.6"
	},
	"tagline" : "You Know, for Search"
}

I'm now a ᕙ༼ຈل͜ຈ༽ᕗ curl’er of containers!!

Credits

Docker, Skydock and Skydns all deserve a big fat ♥.
I followed this guide by Lukas Pustina to set up my vagrant networking.
Gifs from here and faces from there.
Thanks!