This is actually a well-trodden area of datacenter interconnect (DCI) devices that do line-rate encryption (to crazy levels like 400G+) to protect those links that may have easily accessible fibers strung along poles, for instance, to prevent just the vampirism described in the post. Packetlight, Ciena, Infinera and others.
Just spit-balling a little, but I wonder if Wireguard is the best tool here given that the author is only using it for a single point-to-point link and they control the devices on both ends. That CPU supports AES-NI and probably does it a lot faster than Wireguard's ChaCha20 (hard to get numbers for their server CPU, but the tiny little x86 mini PC I use as my router does AES XTS at 43Gbps according to `cryptsetup benchmark`).
You might see better performance by tunneling the vxlan connection using a different technology which can use AES-NI? Then again, Wireguard is definitely still a good tool for stuff like this, and maybe the performance penalty isn't a big deal here.
My solution ended up using tc's mirred[0] action for implementing a fully L2-transparent frame relay. I wonder if their setup achieves the same degree of transparency, because afaiui, that's just not possible involving a 802.1Q-compliant (Linux) bridge.
I spent close to a week optimizing my setup looking at kernel flame graphs and perf results, reading adapter-specific tuning guides and driver source, and can say that the only really meaningful performance optimizations (in both the Broadwell- and Zen3/Vermeer-based implementations I tried) were disabling mitigations in the kernel (esp. on Zen3, that boosted performance by more than 20%), and getting CPU frequency scaling/idle states sorted out correctly (which yielded much higher wins on the older Broadwell uarch, because power state transition appears to happen much quicker on Zen3).
As for the solution presented in the (on the whole really great; I love it!) article, I have my doubts about the effectiveness of the cargo-culted "sysctl tuning" mentioned - TCP, for example, is simply not involved at all in the described setup, so "tuning" its buffer allocations cannot have any effect on the workload.
Kudos to the writers for solving their problem in a creative, cost-effective and maintainable way! :)
[0]: https://www.man7.org/linux/man-pages/man8/tc-mirred.8.html
vm.dirty_ratio = 40
vm.dirty_background_ratio = 10
vm.swappiness=10
governor=performance
energy_perf_bias=performance
min_perf_pct=100
I can't believe they were under any memory pressure, so the first three presumably made no difference, but it's also quite surprising to me that the default ondemand cpu governor was responsible for such a dramatic performance hit. Not throttling up quickly enough leading to higher latency maybe? Very interesting anyway.
But getting back to the main topic of the paper: why not just S2S IPSec the link?
"Oh, no, a truck run into the pole carrying the copper/fiber, it must be an accident and no intervention is going on undetected because of the outage."
What we really need is promiscuous connectivity , but fully untrusted connections. It's maddening why it's hard to communicate 2 wireless devices while they are literally sharing the same radio spectrum and multiple radios could be used to talk to each other.
# setup a 8020 MTU on wg0 interface to account for the 80 bytes wireguard headers overhead
# 20-byte IPv4 header or 40 byte IPv6 header, 8-byte UDP header 4-byte type, 4-byte key index, 8-byte nonce, 16-byte authentication tag)
/sbin/ip li set dev wg0 mtu 8020
Got to use it again to set up a remote telescope for the eclipse.
I have a computer setup with a one-way gige connection for reviewing potentially malicious content in an air-gapped manner. The transmit side transceiver needs to see an incoming signal, so I just use one of these to feed its own output back into it:
9.71 Gbps with wg on a 10GBps link with sysctl tunings, custom MTUs,.
I had heard of token ring, but not 10BASE5: https://en.wikipedia.org/wiki/10BASE5
I'd love to see your IT manager's face when you propose it :D