High intensity port multiplexing using haproxy

Tuesday, February 27, 2018
As I am sure you already know, IPv4 addresses are in limited supply right now.  The solution to this is IPv6 which greatly enlarges the available address space.  The problem is that IPv6 is not yet deployed everywhere, so there is still a need to figure out how to maximize the usage of your existing IPv4 addresses.

I have a VPS on the Internet which only provides 1 IPv4 address.  Of course, I want to run multiple services on this VPS.  I also want to use well-known ports to decrease the chance of being blocked from accessing my VPS.

There are several tools that can handle port multi-plexing.  Probably among the most widely used are haproxy and sslh.  Both of these tools are probably available in your Linux package manager.

SSLH is very easy to use but it only multiplexes SSL and SSH sessions.  If you want more than 2 services on the same port then this tool is not for you.

HAPROXY is a bit more complicated to set up but it is also a lot more configurable.  This post will describe the way that I have haproxy configured to host multiple services.  I will post the full configuration file at the bottom of this post for easy copying and pasting.

NOTE: When you are reading the code below, any text that is underlined needs to be replaced with values that are appropriate to your installation.

The first step in configuring haproxy is to set up the "frontend"  This is the portion of haproxy that listens for incoming connections.  Your "frontend" might look like this:
frontend ssl
        mode tcp
        bind <ipaddress>:<port>
        tcp-request inspect-delay 3s
        tcp-request content accept if { req.ssl_hello_type 1 }
This basically tells haproxy which IP address and port to listen on for incoming connections.  You can also use the IP address 0.0.0.0 for every available IP address, if you have multiple.

The "inspect-delay" tells haproxy how long it should wait to receive data from the client before making a decision about what to do with the incoming connection.  This is required due to the difference in the way that HTTPS and SSH sessions are negotiated.  This is also the way that we distinguish the traffic type.

Once you have this front-end configured, you next need to configure your access control lists which connect your front-end to your backend(s).

The ACL for an SSH session looks like this:
        acl     <ssh label>             payload(0,7)    -m bin 5353482d322e30
This will detect SSH sessions and mark them with <ssh_label>  This is an arbitrary label and you can pick any name you want.  The only requirement is that it matches the rules that connect to the SSH backend.

Your "use_backend" statement for SSH would then look like:
        use_backend <ssh backend name>                     if <ssh label>
As before, the <ssh backend name> is an arbitrary label you can pick.  The only requirement again is that the backend name must match the backend definition.

Since we are now talking about the backend, here is what an SSH backend would look like:

backend openssh
        mode tcp
        timeout server 3h
        server openssh <ip address>:<port>
Typically you would use an IP address of 127.0.0.1 to mean localhost or the local machine.  The default port for SSH is 22.  It is possible to use any IP address and port you want in this definition.  That would be useful if the SSH server is on a different machine on a network behind your haproxy system.

Now we can add additional services.  It is common for a single web-server to host multiple web-sites.  These web-sites are identified by their DNS name.  On the server side this is called SNI or Server Name Indication.

Let's start by setting up an ACL for server1.visideas.com
        acl     <server one>               req.ssl_sni             -i server1.visideas.com
Then the matching use_backend rule would look like:
        use_backend <server 1 backend> if <server one acl> { req.ssl_hello_type 1 }
 Finally, your matching backend might look like:
backend <server 1 backend>
        mode tcp
        server webserver <server 1 IP>:<server 1 port>
 There are also some powerful matching criteria that you can use in your ACL's.  For example, both of these are valid:
        acl     <some acl>            req.ssl_sni             -m end .visideas.com
        acl     <different acl>            req.ssl_sni             -m found
The first line matches any domain name that ends in .visideas.com and marks it with <some acl>.  The second line matches any name and tags it with <different acl>.  These lines will not mark any requests received that were directed directly to an IP address.

Another use_backend that is useful is:
        use_backend <another backend>                      if { req.ssl_hello_type 1 }
The ssl_hello_type of 1 indicates the presence of an HTTPS request.  Since there is no tag name after the "if" this ACL would catch requests which were sent to this haproxy server by IP address.  This means that you can route traffic which came in by specifying IP address to an alternate service.

The final ACL that I will discuss is:
        use_backend <shadowsocks>                 if !{ req.ssl_hello_type 1 } !{ req.len 0 }
This ACL can detect traffic that is meant to be sent to a Shadowsocks server.  This traffic is identified because it does not contain an ssl_hello_type of 1 and it sends traffic immediately without waiting - i.e. the request length is not 0.

There are probably other protocols that this statement would match as well but I am using it for Shadowsocks.

Now, as promised, here is my complete haproxy.conf.  Again, please remember to change everything that is underlined to match your specific settings.

This configuration allows me to access the following services on port 443:
  1. An nginx server when accessed as https://s.visideas.com/
  2. An Apache2 server when access as https://k.visideas.com/ or https://*.visideas.com/ or https://<any DNS name>
  3. A Monit server when accessed as https://monit.visideas.com/
  4. An OpenConnect SSL VPN server when accessed as https://<ip address>/
  5. A Shadowsocks server when accessed using a Shadowsocks client
  6. An SSH server
global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon

defaults
    log    global
    mode    tcp
    option  tcplog   
    option    dontlognull
    maxconn  2000
    timeout connect  5000
    timeout client 500000
    timeout server 500000

frontend ssl
    mode tcp
    bind <host IP>:443
    tcp-request inspect-delay 3s
    tcp-request content accept if { req.ssl_hello_type 1 }

    acl    ssh_payload        payload(0,7)    -m bin 5353482d322e30

        acl     www-monit        req.ssl_sni        -i monit.visideas.com
        acl     www-s        req.ssl_sni        -i s.visideas.com
        acl     www-r        req.ssl_sni        -i r.visideas.com
        acl     www-k        req.ssl_sni        -m end .visideas.com
        acl     www-k        req.ssl_sni        -m found

    use_backend www-monit            if www-monit { req.ssl_hello_type 1 }
    use_backend nginx-s        if www-s { req.ssl_hello_type 1 }
    use_backend apache2-k        if www-k { req.ssl_hello_type 1 }
    use_backend ocserv            if { req.ssl_hello_type 1 }
    use_backend openssh            if ssh_payload
    use_backend openssh            if !{ req.ssl_hello_type 1 } { req.len 0 }
    use_backend shadowsocks            if !{ req.ssl_hello_type 1 } !{ req.len 0 }

backend openssh
    mode tcp
    timeout server 3h
    server openssh 127.0.0.1:22

backend ocserv
    mode tcp
    timeout server 24h
    server sslvpn 127.0.0.1:4443

backend nginx-s
    mode tcp
    server webserver 127.0.0.1:8443

backend apache2-k
    mode tcp
    server webserver 127.0.0.1:10443

backend www-monit
    mode tcp
    server webserver 127.0.0.1:2812

backend shadowsocks
    mode tcp
    server socks 127.0.0.1:8530
I hope this helps you with maximizing the value of your shared IPv4 addresses with haproxy.