I've been using SSH tunnels to reach my cluster from remote, but I needed a more proper solution for this. Using OpenVPN could also give me DNS configuration for free and I can be able to use *.mesos domains from my laptop.

Initially, I set up OpenVPN manually on a host but it didn't get too long to lose that node and hence lose remote access to my cluster and I found myself fighting with SSH tunnels again in no time. So I decided to run OpenVPN with marathon, and it will always be running without manually taking care of possible problems. If service or the underlying node dies or lost network connection, it will just start on another node.

Planning

These are needed in my case:
* OpenVPN configuration
* docker image for OpenVPN for ease of deployment
* a marathon config to run it as a service in cluster
* a public IP address, preferably with a memorable DNS record
* port-mapping for public IP, may not be needed if your nodes have public IP addresses

I started searching for a proper docker image for running OpenVPN and found kylemanna/openvpn. It is well documented and running pretty well. But I had to run it with my own configuration, so I prepared my own OpenVPN docker image with my configuration in it and pushed it to my private docker-registry.

Preparing my own OpenVPN image for my setup

I'll do the preparations in one of my hosts (ubuntu@ubuntu1:~/openvpn$):

These environment variables are needed before start:
* PUBLIC_IP (to reach our cluster from outside)
* MESOS_NETWORK and MESOS_NETMASK (local network)
* DNS_SERVER1 and DNS_SERVER2 (mesos-dns server ip addresses)

I can just fill in those from the local host I'm on (some dirty shell-fu here):

sudo apt-get install ipcalc
PUBLIC_IP=$(curl -s icanhazip.com)
MESOS_NETWORK=$(ipcalc $(ip addr | awk "/$(hostname -i)/"'{print $2}') | tr '/' ' ' | awk '/Network:/ {print $2}')
MESOS_NETMASK=$(ifconfig | tr : ' ' | awk "/$(hostname -i)/"'{print $7}')
read DNS_SERVER1 DNS_SERVER2 rest < <(echo $(dig +short mesos-dns.marathon.mesos))

Preparing the OpenVPN configuration for my cluster:

mkdir openvpn
cd openvpn
docker run --rm -it \
  -v $(pwd)/etc-openvpn:/etc/openvpn \
  kylemanna/openvpn \
  ovpn_genconfig -d -D -N \
    -u tcp://$PUBLIC_IP:1194 \
    -p "dhcp-option DNS $DNS_SERVER1" \
    -p "dhcp-option DNS $DNS_SERVER2" \
    -p "route $MESOS_NETWORK $MESOS_NETMASK"

# Initialize the PKI, this will take some time to generate
echo -e "yes\n\n" | docker run -e DEBUG=1 --rm -i -v $(pwd)/etc-openvpn:/etc/openvpn kylemanna/openvpn ovpn_initpki nopass

Since I have openvpn config in./etc-openvpn directory, I can generate a OpenVPN client and get the .ovpn client config file (mesos.ovpn) for it.

docker run --rm -it -v $(pwd)/etc-openvpn:/etc/openvpn kylemanna/openvpn easyrsa build-client-full mesos nopass
docker run --rm -it -v $(pwd)/etc-openvpn:/etc/openvpn kylemanna/openvpn ovpn_getclient mesos > mesos.ovpn

These extra parameter are needed to be added to the .ovpn file when connecting from a linux client:

cat <<EOF >> mesos.ovpn
script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf
EOF

It's time to build and push my own OpenVPN image to my docker-repository:

cat <<EOF > Dockerfile
FROM kylemanna/openvpn
COPY etc-openvpn /etc/openvpn
EOF

# restore file permissions for etc-openvpn
sudo chown -R $(whoami). etc-openvpn

docker build -t docker-registry.marathon.mesos:5000/openvpn .
docker push docker-registry.marathon.mesos:5000/openvpn

Marathon config for OpenVPN

Deploying my OpenVPN to the cluster using my own image:

cat <<EOF > openvpn-marathon.json
{
  "cpus": 0.1,
  "mem": 50,
  "id": "/openvpn",
  "instances": 1,
  "container": {
    "docker": {
      "image": "docker-registry.marathon.mesos:5000/openvpn",
      "network": "BRIDGE",
      "forcePullImage": true,
      "parameters": [{"key":"cap-add", "value":"NET_ADMIN"}],
      "portMappings": [{"containerPort": 1194, "servicePort": 1194}]
    }
  },
  "dependencies": ["/mesos-dns", "/docker-registry"],
  "healthChecks": [{"protocol": "TCP"}]
}
EOF

Starting the OpenVPN service:

curl -sL -X POST \
  -H 'content-type: application/json' \
  leader.mesos:8080/v2/apps \
 -d@openvpn-marathon.json | jq .

And this is it, I have OpenVPN is running via marathon, if it will be killed or something happens to the node it's running on, marathon will just start a new OpenVPN and it will be reachable via haproxy on any node.

And pushing the source directory into a repo, so I can later update your config, add new keys and re-build a new image:

git init
git add .
git commit -m 'initial commit'
# I also pushed this to a private remote repo

Here is what I have now, I can use any node in the cluster for proxying to OpenVPN:

Public IP port-mapping

Now it's time to add port-mapping to my ADSL router at home. But if I just add a port-mapping to point a host I have that node will be a single point of failure. Since I have haproxy-marathon-bridge running on every node I can add 3 port mappings to 3 different hosts:

Now even if I lose the first 2 nodes, I can still be able to connect to the OpenVPN. But hey I have now 3 servers for OpenVPN but I have only remote definition in mesos.ovpn file:

ubuntu@ubuntu1:~/openvpn$ grep 'remote ' mesos.ovpn
remote 89.101.5.12 1194 tcp
ubuntu@ubuntu1:~/openvpn$ 

To fully automate the OpenVPN high-availability, I also need to update the remote definition in mesos.ovpn file like this:

remote 89.101.5.12 1194 tcp
remote 89.101.5.12 1195 tcp
remote 89.101.5.12 1196 tcp

Now my system looks like this, every piece (of course except modem) is high-available:

And all I need to do for using my OpenVPN is copying the mesos.ovpn to my laptop and running this command:

laptop:~$ sudo openvpn mesos.ovpn

And now, after connecting with OpenVPN, I can use these URLs also from my laptop (I have chronos and marathon running on all mesos-master nodes):
* Marathon: http://leader.mesos:8080/
* Mesos: http://leader.mesos:5050/
* Chronos: http://leader.mesos:4400/

Here is the marathon web interface screenshot with OpenVPN service:

Bonus for the no-static IP case and route53

And here is a bonus, if you don't have static IP address and also using route53 service like me, here is a home made no-ip solution. The IP address assigned to my router does occasionally change. I wrote the following script to update home.bdgn.net DNS record periodically to point up-to-date IP of home router:

ubuntu@ubuntu1:~$ sudo pip install awscli
ubuntu@ubuntu1:~$ cat ~/bin/update-home-dns 
#!/bin/bash
export AWS_ACCESS_KEY_ID=AKIA***
export AWS_SECRET_ACCESS_KEY=***
ZONE_ID=Z3***
DOMAIN_NAME=home.bdgn.net.

CURRENT_IP=$(curl -s icanhazip.com)
RECORDED_IP=$(aws route53 list-resource-record-sets  --hosted-zone-id $ZONE_ID | jq -c '.ResourceRecordSets[] | select(.Name | contains("'$DOMAIN_NAME'")) | .ResourceRecords[0].Value' | tr -d '"')

echo would replace $RECORDED_IP with $CURRENT_IP
if [ "$CURRENT_IP" != "$RECORDED_IP" ]; then
    aws route53 change-resource-record-sets --hosted-zone-id $ZONE_ID --change-batch '{"Comment": "auto-update from '`hostname`' ('`hostname -i`')", "Changes": [{"Action": "UPSERT", "ResourceRecordSet": {"Name": "'$DOMAIN_NAME'", "Type": "A", "TTL": 60, "ResourceRecords": [{"Value": "'$CURRENT_IP'"}]}}]}'
fi
ubuntu@ubuntu1:~$ crontab -l
*/10 * * * * ~/bin/update-home-dns  &>/dev/null
ubuntu@ubuntu1:~$ 

And with having update-home-dns script regularly updating home.bdgn.net DNS record, I can now use domain name instead of static IP address for remote definitions in mesos.ovpn client config file, just like this:

remote home.bdgn.net 1194 tcp
remote home.bdgn.net 1195 tcp
remote home.bdgn.net 1196 tcp

But if I put this onto any host like this, this will likely be broken in time because none of my hosts are persistent and any node can just disappear anytime, I'd better run this script via chronos, but I'm leaving this to another blog post.