... when it's a ct mark!
All jokes aside, I wanted to talk about a frustrating situation I have been banging my head against in the last few months. Those who are keeping up with news in the Gargoyle community would already be aware that I have recently rewritten Gargoyle to switch from fw3 (iptables) to fw4 (nftables).
Let's dive right in with some background info...
Connection marks are used for implementing firewall logic that is too complex to represent with a single rule or a set of rules that need to build on each other or be logically grouped. They are also used to implement what I'll refer to as stateful rules (not to be confused with stateful firewalling), allowing us to make decisions based on previous packets in the connection.
iptables CONNMARK extension
The CONNMARK extension is used to interact with the netfilter mark for the connection related to a packet. It can be used to match a mark on the connection, set or clear the connection mark, and to save or restore the packet mark to/from the connection mark. See iptables-extensions man page.
nftables ct mark expression
The ct mark expression is the nftables equivalent of the iptables CONNMARK extension. See nftables man page.
On paper, a worthy replacement... or is it..?
Compare the pair
Let's look at two "equivalent" commands. I'll pull some examples from Gargoyle just to make it relevant.
But... the results are not what you might initially expect.ct mark set mark & 0x7F literally means:
Take the Packet Mark and mask it by 0x0000007F, then copy this to the Connection Mark.
So if we had an initial connection mark of 0x12340000 and we had a packet mark 0xA, if you were hoping for a result of 0x1234000A you will be sorely disappointed. Instead you get 0x0000000A.
Some of you would see that as an obvious outcome. I tip my hat to you my intellectual superior. Let's keep moving.
No worries! You shrug this setback off because you also are an intellectual, and you try this masterpiece:
ct mark set ct mark & 0xFFFFFF80 | mark & 0x7F this time meaning:
Take my existing Connection Mark and mask it by 0xFFFFFF80 (i.e. clear the 0x7F bits). Take the Packet Mark and mask it by 0x0000007F (as we did before). Now just OR those together...
ERR: Right hand side of binary operation | must be constant
It's at this point that I went full Michael Scott when Toby returned to the office.
And for Linux Kernel 6.6 (which is what OpenWrt 24.10 runs), that's the end of our journey. You can't natively solve this problem. ct mark is not a full replacement of CONNMARK. How this was allowed to be a thing for a significant period of time baffles me. Using masked marks in this fashion is not uncommon.
And for this author, the next course of action was to continue rewriting the Gargoyle firewall inclusive of this limitation and just see how it goes. There were signs it wasn't working great... but they were easy enough to look past in most situations. It wasn't until a few forum posts came up reporting issues that I was able to get back into it and try to unstick the mess I had made. This was a gross underestimation on my part, sorry guys!
Deus ex machina
I had a little vent about this situation on the OpenWrt forums. Another helpful user highlighted that as of Linux Kernel 6.13 support for non-constant right hand sides of binary operations (what a mouthful) was here!
Sweet, what kernel does OpenWrt snapshot currently run? 6.12? Shit.
So if I want to just wait for a solution I'm at least 2 years away by the time the next stable OpenWrt releases, snapshot moves on to a newer kernel, AND THEN I have to do the leg work of getting Gargoyle up to date. In the meantime I've shipped garbage to my beta testers. No beuno.
Alright so let's look at these commits and hope they aren't too nasty. You don't have to click these unless you really want to see how the sausage is made...
libnftnl
- include: add new bitwise boolean attributes to nf_tables.h
- expr: bitwise: rename some boolean operation functions
- expr: bitwise: add support for kernel space AND, OR and XOR
nftables
- src: allow binop expressions with variable right-hand operands
Linux Kernel
- netfilter: bitwise: rename some boolean operation functions
- netfilter: bitwise: add support for doing AND, OR and XOR
Alright, these don't look too invasive, and luckily they apply cleanly to the our OpenWrt 24.10 (Kernel 6.6) base, with one minor modification due to a function prototype change in nft_parse_register_load. While this is trivial to undo, we would be reopening a can of worms known as CVE-2022-1015... so we need to grab a few more patches to fix that too. On OpenWrt which is a single user system, I would argue that this CVE is meaningless. If you've allowed someone to get close enough to your infrastructure that they can craft a malicious command to exploit an out of bounds read, they already have the keys to the kingdom... but if you're reading this in the context of another system, caveat lector. You might like to read Yordan Stoychev's write up on this CVE if you're interested in how this vulnerability works.
One quick backport later and a few tests and we are cooking with gas now. We can write our wonderful ct mark set ct mark & 0xFFFFFF80 | mark & 0x7F and the appropriate bitwise math gets gobbled up happily. No more clobbering of Connection Marks, and another @Lantis bug paved over like a council worker with a pot hole. We all know it's not the nicest fix but it will do for now.
So when is a CONNMARK not a CONNMARK?
... when it's a ct mark... on kernels older than 6.13
With these fixes on board, guess what works much better? Quotas and QoS on Gargoyle. I'll be releasing these fixes into the wild in the next week once I've finished testing in my own network.
