Once upon a time I setup my home network, I could access stuff from outside and all was well in the world. Over time this home network grew and I exposed more and I forwarded more ports from the router to the backend server apps. A few of which were as follows, each had their own port and own domain.
| Name | Domain | Internal Port | External Port |
|---|---|---|---|
| Gogs (git) | g.example.com | 6000 | 444 |
| Gogs (git) (ssh) | g.example.com | 6001 | 2222 |
| homeassistant | h.example.com | 8123 | 445 |
| owncloud | o.example.com | 443 | 443 |
| jenkins | j.example.com | 8085 | 446 |
| letsencrypt | g.example.com h.example.com o.example.com j.example.com | 80 | 80 |
There was more than just that but this will do for the purpose of this post, this meant anytime I did any work on my router it was a pain as I had to re-forward all the ports.
Like any enterprise I didnt design in this tech debt nor was it planned, it evolved one app at time and then all of a sudden it was a mess, and I never cleaned up because “its working”, “its too much hassle”,”i’ll do that when I am doing x, actually thats extra work i’ll leave it”, etc etc. Most of you reading this blog will understand, and will have likely used or heard one of those excuses at least once in your lifetime.
So now onto the interesting part, enter docker and haproxy. I couldn’t take the above any more and I was deploying other stuff in docker elsewhere so I began with taking each of the apps and getting them into docker on the host. This was easier than expected with docker hub providing all the images I could want for. I did rebuild the jenkins one though as I did want a few tweaks to how I was using it, but that was just an extra layer on top of the base jenkins image.
Once all the apps were dockerized and exposed on the same internal ports as before, with the exception of owncloud as it was blocking port 443. It got moved to an http port (8080) instead of being directly exposed as ssl.
The next step was haproxy, thanks to this post I was able to get the basics of haproxy working with letsencypt and multiple domains.
My docker startup for haproxy started to look like this:
docker run -d --name haproxy --restart=always \
-p 2222:2222 \
-p 80:80 \
-p 443:443 \
-p 8998:8998 \
-v /srv/haproxy:/usr/local/etc/haproxy:ro \
-v /etc/letsencrypt/live:/certs/live:ro \
-v /etc/letsencrypt/archive:/certs/archive:ro \
haproxy:1.8-alpine
Port 2222: Will be the ssh port for git
Port 443: SSL/TLS Port for all webapps
Port 8998: Internal network only, stats for haproxy
port 80: Used for letsencrypt
All these bind on addr 0.0.0.0 which is fine as this is the interface for the docker container not the docker host. Remembering this is important as although the apps on 10.0.0.1 are on the docker host too, we cannot use 127.0.0.1 as that would localhost for the *container* not the docker host.
My haproxy config became something like this:
global
tune.ssl.default-dh-param 2048
maxconn 2048
defaults
log 127.0.0.1 local0
option tcplog
timeout connect 5000ms
timeout check 5000ms
timeout client 30000ms
timeout server 30000ms
listen stats
bind 0.0.0.0:8998
mode http
stats enable
stats realm Haproxy\ Statistics
stats refresh 20s
stats uri /
frontend letsencypt
bind *:80
default_backend letsencrypt-backend
frontend main
mode http
bind 0.0.0.0:443 ssl crt /certs/live/h.example.com/combined.pem crt /certs/live/o.example.com/combined.pem crt /certs/live/j.example.com/combined.pem crt /certs/live/g.example.com/combined.pem
reqadd X-Forwarded-Proto:\ https
acl letsencrypt-acl path_beg /.well-known/acme-challenge/
use_backend letsencrypt-backend if letsencrypt-acl
use_backend bk_o if { ssl_fc_sni o.example.com }
use_backend bk_h if { ssl_fc_sni h.example.com }
use_backend bk_j if { ssl_fc_sni j.example.com }
use_backend bk_g if { ssl_fc_sni g.example.com }
default_backend bk_c
frontend g_ssh
mode tcp
bind 0.0.0.0:2222
default_backend bk_g_ssh
timeout client 1h
backend bk_g_ssh
mode tcp
server g 10.0.0.1:6001
backend bk_h
mode http
option forwardfor
server h 10.0.0.2:8123
backend bk_o
mode http
option forwardfor
server o 10.0.0.1:8080
backend bk_j
mode http
option forwardfor
server j 10.0.0.1:8085
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
reqrep ^([^\ :]*)\ /(.*) \1\ /\2
acl response-is-redirect res.hdr(Location) -m found
rspirep ^Location:\ (http|https)://10.0.0.1:8085/jenkins/(.*) Location:\ \1://j.example.com/jenkins/\2 if response-is-redirect
backend bk_g
mode http
option forwardfor
server git 10.0.0.1:6000
backend letsencrypt-backend
mode http
option forwardfor
server letsencrypt 10.0.0.1:8888
The sharp eyed will notice the ssl certs mentions combined.pem, yet letsencypt does produce combined certs well to achieve that a quick bash script creates those.
combine_pem.sh
#!/bin/bash -xe
cd /etc/letsencrypt/live
for i in `ls`;
do
for x in `ls $i/fullchain*`;
do
cat $x $(echo $x | sed 's/fullchain/privkey/') > $(echo $x | sed 's/fullchain/combined/')
done
done
I then added a crontab entry to renew all my certs, the post-hook here is important as it first of calls the combine script to generate our combined ssl certs then it tells docker to restart haproxy so that it picks up the new certs.
certbot renew --tls-sni-01-port=8888 --noninteractive --force-renewal --post-hook "sh /data/scripts/util/combine_pem.sh; docker restart haproxy"
My new port forwarding became:
| Name | Domain | Internal Port | External Port |
|---|---|---|---|
| Gogs (git) (web) | g.example.com | 6000 | |
| Gogs (git) (ssh) | g.example.com | 6001 | |
| homeassistant | h.example.com | 8123 | |
| owncloud | o.example.com | 8080 | |
| jenkins | j.example.com | 8085 | |
| letsencrypt | g.example.com | 8888 | |
| haproxy (web) | g.example.com j.example.com h.example.com o.example.com | 443 | 443 |
| haproxy (git ssh) | g.example.com | 2222 | 2222 |
| haproxy (letsencrypt) | g.example.com j.example.com h.example.com o.example.com | 80 | 80 |
