NAT, pf & jails

Sometimes you want to use jails on the same hosts that do the NAT. That of course isn’t by any means a complicated task and pf can do that very easily. The problems begin when you want to connect from one jail to a NATed IP (in the example 192.168.122.251) to a port that is redirected to another jail on the same system. Here is one of the solutions.

Let’s assume this configuration: external interface is em0, the public IP assigned to that interface will be used to NAT all connections from small network used by jails. Jails IPs are aliased on lo1 interface.

1
2
3
4
5
6
7
# ifconfig lo1
lo1: flags=8049<up,loopback,running,multicast> metric 0 mtu 16384
        options=600003<rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>
        inet 10.10.10.10 netmask 0xffffffff
        inet 10.10.10.11 netmask 0xffffffff
        nd6 options=29<performnud,ifdisabled,auto_linklocal>
        groups: lo

The pf configuration is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# cat /etc/pf.conf
ext_if=“em0”
jail_net=“10.10.10.0/24”

set skip on lo0

nat pass on $ext_if from $jail_net to any -> $ext_if

rdr pass on $ext_if proto tcp to port 80 -> 10.10.10.11/24 port 8080

pass in on $ext_if proto tcp from any to $ext_if port 22 modulate state
pass out all

We have two jails:

1
2
3
4
# jls
   JID  IP Address      Hostname                      Path
     1  10.10.10.10     j1.local                      /jails/j1
     2  10.10.10.11     j2.local                      /jails/j2

On j2 we started an http server that listens on port 8080, connections to our NATed IP are redirected to that port and jail. Yet when you want to connect from j1 to NATIP:80, the connection will fail!

All the communication between the IP address assigned to a single host use loopback interface lo0, regardless of the interface they are bind to. This is a very important clue. This is what we see after initialising connection from j1 to j2 (that are aliased on lo1!) using internal addresses:

1
2
3
4
# tcpdump -ni lo0
16:13:02.606606 IP 10.10.10.10.48619 > 10.10.10.11.8080: Flags [S], seq 3273691539, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 15981952 ecr 0], length 0
16:13:02.606690 IP 10.10.10.11.8080 > 10.10.10.10.48619: Flags [S.], seq 2726574164, ack 3273691540, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 3545711907 ecr 15981952], length 0
16:13:02.606699 IP 10.10.10.10.48619 > 10.10.10.11.8080: Flags [.], ack 1, win 1276, options [nop,nop,TS val 15981952 ecr 3545711907], length 0

Now, since we know that, we can add a line like

rdr pass on lo0 proto tcp from $jail_net to ($ext_if) port 80 -> 10.10.10.11 port 8080

Yet it’ll still not work as expected:

1
2
16:30:35.037426 IP 10.10.10.10.44554 > 192.168.122.251.80: Flags [S], seq 4166748979, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 17034382 ecr 0], length 0
16:30:35.037484 IP 192.168.122.251.80 > 10.10.10.10.44554: Flags [R.], seq 0, ack 4166748980, win 0, length 0

We need to remove set skip on lo0 to allow pf to filter on this interface. After that, we can add explicit pass quick on lo0 all to simulate set skip rule.

Finally, we should be able to connect from j1 to j2 using our public/NATed IP!

NAT, pf & jails

Mysterious Code - Your AWS & DevOps consultants