August 6, 2004


Playing with binpatch. It's a nice idea, but the packages store the compilation date & time in some of the files (like the kernel), so there's no way to get reliable MD5's between the master site & the patches I build. Oh well, it's better than nothing.

One bugger with the system as it comes from is that the patches don't build without fixing the makefile. The that comes with (for 3.5) doesn't unpack sys.tar.gz in the correct location. Patch below.

kagome# rcsdiff -u
RCS file:,v
retrieving revision 1.1
diff -u -r1.1
--- /tmp/T0r32753 Fri Aug 6 22:45:00 2004
+++ Fri Aug 6 22:34:30 2004
@@ -191,7 +191,7 @@
@echo "===> Extracting sources"
@mkdir -p ${WRKSRC}/sys && \
tar xzpf ${DISTDIR}/src.tar.gz -C ${WRKSRC}&& \
- tar xzpf ${DISTDIR}/sys.tar.gz -C ${WRKSRC}/sys && \
+ tar xzpf ${DISTDIR}/sys.tar.gz -C ${WRKSRC}&& \
touch -f ${.TARGET}

# Extracts the OpenBSD installation files

| The binary patch(es) has/have been created in
| /usr/binpatch/packages
| To install a binpatch just unpack it under /, e.g.:
| # cd /usr/binpatch/packages
| # tar xzpf binpatch-3.5-i386-001.tgz -C /

Posted by benjamin at 10:46 PM

PF Tricks

Running a firewall as a bridge can have great advantages, but proxying is difficult. Here's the info I wish I had when I started trying to figure this out.

Network Architecture

               ISP1                 ISP2               ISP3
                 |                    |                  |
                              sk1    sk2     sk3
                               |      |       |
               +---------------+      |       +------------+
               |                      |                    |
            Traffic                   |                 Traffic
            Shaper                    |                  Shaper
               |                      | B                  |
               |                      | y                  | D
               |                      | p                  | e
 +----------------+                   | a                  | f
 | bridge     em2 |                   | s                  | a
 | firewall   ] [ |                   | s                  | u
 |  em0       em1 |                   |                    | l
 +----------------+                   |                    | t
    |          |                      |                    |
    |          |                      |                    |
    |          |                      |                    |
    |          +---------------+      |        +-----------+
    |                          |      |        |
    |                         xl1    xl2      xl3
    |                         Int-Sorting-Router
    |                                xl0
    |                                 |
    |                                 |
    |                                gx0
    |                          AggregationRouter
    |                            gx4 gx1 gx3  gx2
    |                             /   |   |    \
    |                            /    |   |     \
    |                           /     |   |      +--------Research Network
    |                          /      |   |                 (
    |                         /       |   |
    |                        /        |   +------Main Network
    |                       /         |           (
    |                      /          |
    |                     /           +-------Limited Network
    +---Management Net---+                     (

Infrastructure Network ( - primarily /30's for router links

The network this firewall was designed for looked like the diagram above. The
firewall is built using OpenBSD and pf running as a bridge, with transparent
proxying of web traffic. The Internal Sorting Router is used to direct traffic
from the Research Network across the Bypass link; traffic from the Main &
Management Networks thru the "Default" link (and thus a traffic shaper); and
traffic from the "Limited Network" thru the firewall & traffic shaper. The
ISR uses SOURCE ROUTING to perform these tasks.

The External Sorting Router routes traffic back across the proper link based on
the destination. One thing to note in this setup is that traffic between the
4 networks does not reach the ISR, because the Aggregation Router routes it.

Note: the interface names are arbitrary, but valid, and chosen to be unique
across all devices in the diagram. Also, the device numbers are chosen to give
an indication of the normal path for a packet, and device number 0 is used to
indicate the default route.

On the firewall, em1 & em2 form a bridge, and em0 is the management/proxy
interface. All proxied web requests reach the outside world via em0.

By design, OpenBSD's bridge implementation does not give direct access to the
traffic for user-space processes. Thus, to pluck packets off of the bridge, we
have to jump into some of the less well documented features of pf.

With pf disabled, and the bridge enabled, a host in the limited network will
send traffic along the path gx1:gx0:xl0:xl1:em1:em2:sk1:sk0, and traffic will
return along the path sk0:sk1:em2:em1:xl1:xl0:gx0:gx1.

Transparent Proxy

To enable transparent proxying, we build the transparent flavor of squid, and
configure it as described in the widely available online documentaion. In our
case, we set is up listening on . I'll skip the details of
our squid config, and let you use your imagination to determine what you want
to do.

Interface Configuration

I'll go into more depth about why the interfaces need to be configured this
way, but let's get it working now, and explain why & how later.

On the Firewall

# ifconfig em0 netmask 0xffff0000 up
# ifconfig em1 netmask 0xfffffffc up


# ifconfig xl1 netmask 0xfffffffc up
# ifconfig xl1 aliax netmask 0xfffffffc up


# ifconfig sk1 netmask 0xfffffffc up

We now have a point-to-point between ISR and ESR, and a separate
point-to-point between ISR and Firewall(em1).

Static Routing on Firewall

On the firewall, we need the IP stack to know how to send packets to the
hosts we're proxying for.

# route add

Configuring pf

I'll show the critical parts of the pf configuration here, and explain them more

    table <limited> const { }
    # i
    rdr on em1 proto tcp from <limited> to any port 80 -> port 3128
    # ii
    pass  in quick log on em1 route-to (lo0 proto tcp from <limited> \
                to any port 80 keep state
    # iii
    pass out quick log on lo0 route-to (em1 proto tcp from \
                port 3128 to <limited> keep state


When a host in ( sends a tcp packet to
(, here's what happens:

Notation: Original packet O: sIP(sMAC):sport -> dIP(dMAC):dport
Modified packet M: sIP(sMAC):sport -> dIP(dMAC):dport

1. The rdr rule(i) rewrites the IP and TCP headers, changing the destination IP and port
of the packet. The packet still has a destination MAC address of sk1 because that's what
ISR determined to be the MAC address of the next-hop router.
O: ->
M: ->

2. The route-to rule (ii) rewrites the destination MAC address by looking up the MAC of, and adjusting the packet. The packet is then sent out on lo0.
O: ->
M: ->

3. The proxy process listening on get's the packet (SYN), and sends a reply
(SYN/ACK) to the requesting host ( The IP stack knows that these packets are to
be sent out on lo0 because of the static route we created earlier.
O: ->

4. Before sending the packet out, the kernel sends it to pf for processing. Here rule (iii)
is triggered, and it rewrites the destination MAC with the MAC address of
The packet is then sent to em1 to be output.
O: ->
M: ->

5. The packet from -> gets processed by the rdr rule (i) and
the source address and TCP port are rewritten to the ones the client expects
(, and the packet is sent out the interface em1.
O: ->
M: ->

6. The internal routing from ISR to the requesting host processes the packet, and the
requesting host gets a response it was expecting. The MAC addresses will change along
the way as the packet traverses the routers in the path.

The remainder of the TCP conversation will follow this loop for packets being sent to/from
the proxy interface.

Can't I Skip ... ?

For all of this to work, the IP stack & kernel need enough information to rewrite all parts
of the packet headers. Each of the parts above plays a part in this.

Without the static route to the network, the stack will follow the default route
and send the packet out em0 with a source address of This will not work.

Without the point-to-point link from em1 to xl1, the stack does not know which MAC address
to use when it puts the packet back on the bridge. In my testing, I saw packets on em1 with
the source MAC and destination MAC both having the MAC address of em1. This will not
I found this to be the most odd, considering 'ifconfig bridge0' showed that the bridge knew
xl1's MAC address, but the bridge is will insulated from the rest of the stack, and will not
share this information. This is far better than having the bridge build in assumptions which
will be wrong for many cases.

Hope that helps.

Posted by benjamin at 10:56 PM