Sometimes you want to use jails on the same hosts that do the NAT. That of course isn’t by any means a complicated task and
pf can do that very easily. The problems begin when you want to connect from one jail to a NATed IP (in the example 192.168.122.251) to a port that is redirected to another jail on the same system. Here is one of the solutions.
Let’s assume this configuration: external interface is
em0, the public IP assigned to that interface will be used to NAT all connections from small network used by jails. Jails IPs are aliased on
# ifconfig lo1 lo1: flags=8049
metric 0 mtu 16384 options=600003 inet 10.10.10.10 netmask 0xffffffff inet 10.10.10.11 netmask 0xffffffff nd6 options=29 groups: lo
pf configuration is as follows:
# cat /etc/pf.conf ext_if=“em0” jail_net=“10.10.10.0/24” set skip on lo0 nat pass on $ext_if from $jail_net to any -> $ext_if rdr pass on $ext_if proto tcp to port 80 -> 10.10.10.11/24 port 8080 pass in on $ext_if proto tcp from any to $ext_if port 22 modulate state pass out all
We have two jails:
# jls JID IP Address Hostname Path 1 10.10.10.10 j1.local /jails/j1 2 10.10.10.11 j2.local /jails/j2
On j2 we started an http server that listens on port 8080, connections to our NATed IP are redirected to that port and jail. Yet when you want to connect from j1 to NATIP:80, the connection will fail!
All the communication between the IP address assigned to a single host use loopback interface
lo0, regardless of the interface they are bind to. This is a very important clue. This is what we see after initialising connection from j1 to j2 (that are aliased on lo1!) using internal addresses:
# tcpdump -ni lo0 16:13:02.606606 IP 10.10.10.10.48619 > 10.10.10.11.8080: Flags [S], seq 3273691539, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 15981952 ecr 0], length 0 16:13:02.606690 IP 10.10.10.11.8080 > 10.10.10.10.48619: Flags [S.], seq 2726574164, ack 3273691540, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 3545711907 ecr 15981952], length 0 16:13:02.606699 IP 10.10.10.10.48619 > 10.10.10.11.8080: Flags [.], ack 1, win 1276, options [nop,nop,TS val 15981952 ecr 3545711907], length 0
Now, since we know that, we can add a line like
rdr pass on lo0 proto tcp from $jail_net to ($ext_if) port 80 -> 10.10.10.11 port 8080
Yet it’ll still not work as expected:
16:30:35.037426 IP 10.10.10.10.44554 > 192.168.122.251.80: Flags [S], seq 4166748979, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 17034382 ecr 0], length 0 16:30:35.037484 IP 192.168.122.251.80 > 10.10.10.10.44554: Flags [R.], seq 0, ack 4166748980, win 0, length 0
We need to remove
set skip on lo0 to allow pf to filter on this interface. After that, we can add explicit
pass quick on lo0 all to simulate set skip rule.
Finally, we should be able to connect from j1 to j2 using our public/NATed IP!