Docker and HAProxy a match made in heaven

Once upon a time I setup my home network, I could access stuff from outside and all was well in the world. Over time this home network grew and I exposed more and I forwarded more ports from the router to the backend server apps. A few of which were as follows, each had their own port and own domain.

NameDomainInternal PortExternal Port
Gogs (git)g.example.com6000444
Gogs (git) (ssh)g.example.com60012222
homeassistanth.example.com8123445
owncloudo.example.com443443
jenkinsj.example.com8085446
letsencryptg.example.com
h.example.com
o.example.com
j.example.com
8080

There was more than just that but this will do for the purpose of this post, this meant anytime I did any work on my router it was a pain as I had to re-forward all the ports.

Like any enterprise I didnt design in this tech debt nor was it planned, it evolved one app at time and then all of a sudden it was a mess, and I never cleaned up because “its working”, “its too much hassle”,”i’ll do that when I am doing x, actually thats extra work i’ll leave it”, etc etc. Most of you reading this blog will understand, and will have likely used or heard one of those excuses at least once in your lifetime.

So now onto the interesting part, enter docker and haproxy. I couldn’t take the above any more and I was deploying other stuff in docker elsewhere so I began with taking each of the apps and getting them into docker on the host. This was easier than expected with docker hub providing all the images I could want for. I did rebuild the jenkins one though as I did want a few tweaks to how I was using it, but that was just an extra layer on top of the base jenkins image.

Once all the apps were dockerized and exposed on the same internal ports as before, with the exception of owncloud as it was blocking port 443. It got moved to an http port (8080) instead of being directly exposed as ssl.

The next step was haproxy, thanks to this post I was able to get the basics of haproxy working with letsencypt and multiple domains.

My docker startup for haproxy started to look like this:

docker run -d --name haproxy --restart=always \
-p 2222:2222 \
-p 80:80 \
-p 443:443 \
-p 8998:8998 \
-v /srv/haproxy:/usr/local/etc/haproxy:ro \
-v /etc/letsencrypt/live:/certs/live:ro \
-v /etc/letsencrypt/archive:/certs/archive:ro \
haproxy:1.8-alpine

Port 2222: Will be the ssh port for git
Port 443: SSL/TLS Port for all webapps
Port 8998: Internal network only, stats for haproxy
port 80: Used for letsencrypt

 

All these bind on addr 0.0.0.0 which is fine as this is the interface for the docker container not the docker host. Remembering this is important as although the apps on 10.0.0.1 are on the docker host too, we cannot use 127.0.0.1 as that would localhost for the *container* not the docker host.

My haproxy config became something like this:
global
tune.ssl.default-dh-param 2048
maxconn 2048

defaults
log 127.0.0.1 local0
option tcplog
timeout connect 5000ms
timeout check 5000ms
timeout client 30000ms
timeout server 30000ms

listen stats
bind 0.0.0.0:8998
mode http
stats enable
stats realm Haproxy\ Statistics
stats refresh 20s
stats uri /

frontend letsencypt
bind *:80

default_backend letsencrypt-backend

frontend main
mode http

bind 0.0.0.0:443 ssl crt /certs/live/h.example.com/combined.pem crt /certs/live/o.example.com/combined.pem crt /certs/live/j.example.com/combined.pem crt /certs/live/g.example.com/combined.pem

reqadd X-Forwarded-Proto:\ https

acl letsencrypt-acl path_beg /.well-known/acme-challenge/
use_backend letsencrypt-backend if letsencrypt-acl

use_backend bk_o if { ssl_fc_sni o.example.com }
use_backend bk_h if { ssl_fc_sni h.example.com }
use_backend bk_j if { ssl_fc_sni j.example.com }
use_backend bk_g if { ssl_fc_sni g.example.com }

default_backend bk_c

frontend g_ssh
mode tcp
bind 0.0.0.0:2222

default_backend bk_g_ssh
timeout client 1h

backend bk_g_ssh
mode tcp
server g 10.0.0.1:6001

backend bk_h
mode http
option forwardfor
server h 10.0.0.2:8123

backend bk_o
mode http
option forwardfor
server o 10.0.0.1:8080

backend bk_j
mode http
option forwardfor
server j 10.0.0.1:8085

http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }

reqrep ^([^\ :]*)\ /(.*) \1\ /\2
acl response-is-redirect res.hdr(Location) -m found
rspirep ^Location:\ (http|https)://10.0.0.1:8085/jenkins/(.*) Location:\ \1://j.example.com/jenkins/\2 if response-is-redirect

backend bk_g
mode http
option forwardfor
server git 10.0.0.1:6000

backend letsencrypt-backend
mode http
option forwardfor
server letsencrypt 10.0.0.1:8888

 

The sharp eyed will notice the ssl certs mentions combined.pem, yet letsencypt does produce combined certs well to achieve that a quick bash script creates those.

combine_pem.sh
#!/bin/bash -xe

cd /etc/letsencrypt/live
for i in `ls`;
do
for x in `ls $i/fullchain*`;
do
cat $x $(echo $x | sed 's/fullchain/privkey/') > $(echo $x | sed 's/fullchain/combined/')
done
done

 

I then added a crontab entry to renew all my certs, the post-hook here is important as it first of calls the combine script to generate our combined ssl certs then it tells docker to restart haproxy so that it picks up the new certs.

certbot renew --tls-sni-01-port=8888 --noninteractive --force-renewal --post-hook "sh /data/scripts/util/combine_pem.sh; docker restart haproxy"

 

My new port forwarding became:

NameDomainInternal PortExternal Port
Gogs (git) (web)g.example.com6000
Gogs (git) (ssh)g.example.com6001
homeassistanth.example.com8123
owncloudo.example.com8080
jenkinsj.example.com8085
letsencryptg.example.com8888
haproxy (web)g.example.com
j.example.com
h.example.com
o.example.com
443443
haproxy (git ssh)g.example.com22222222
haproxy (letsencrypt)g.example.com
j.example.com
h.example.com
o.example.com
8080

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This blog is kept spam free by WP-SpamFree.