I’ve been playing around with making VyOS an application aware firewall solution. This was pretty easy to setup using VyOS and nprobe.
Here’s the article:
8 Likes
Very good read! Thanks for pushing the envelope on the filtering side of things. I use my instance of VyOS as a firewall after retiring hardware that previous ran Opnsense, so appreciate this!
1 Like
No problem! One interesting thing with using this on VyOS over OpnSense is the advertised performance difference of Linux vs. FreeBSD. Here is a graphic from the nprobe product page:
Considering my system was barely utilized when maxing out my 1Gbps internet, I believe the Linux performance numbers given. I don’t care to setup OpnSense, so I can’t compare the 2, but I’m very happy with the performance on VyOS.
I was honestly surprised at how easy this was to set up. Almost a direct bolt-on to VyOS and performed very well. It’s to the point that I would suggest to anyone using VyOS as their internet facing device to try it out (will likely require 1.4 or greater). Even if you don’t use it to filter traffic, it can be incredibly useful to see a general list of services you’re device is talking to.
04/May/2024 00:26:02 [nprobe.c:4117] Average traffic: [717.00 pps][All Traffic 5.62 Mb/sec][IP Traffic 5.47 Mb/sec][ratio 0.98]
04/May/2024 00:26:02 [nprobe.c:4125] Current traffic: [125.00 pps][255.88 Kb/sec]
04/May/2024 00:26:02 [nprobe.c:4133] L7 Proto Diff Total
04/May/2024 00:26:02 [nprobe.c:4147] Unknown/0 1.07 MB 7.80 GB
04/May/2024 00:26:02 [nprobe.c:4147] DNS/5 6.40 KB 4.44 MB
04/May/2024 00:26:02 [nprobe.c:4147] HTTP/7 0 B 22.89 KB
04/May/2024 00:26:02 [nprobe.c:4147] NTP/9 76 B 46.31 KB
04/May/2024 00:26:02 [nprobe.c:4147] Outlook/21 16.07 KB 13.72 MB
04/May/2024 00:26:02 [nprobe.c:4147] ntop/26 0 B 745.02 KB
04/May/2024 00:26:02 [nprobe.c:4147] Skype_TeamsCall/38 0 B 74.58 KB
04/May/2024 00:26:02 [nprobe.c:4147] IMAPS/51 0 B 158.33 KB
04/May/2024 00:26:02 [nprobe.c:4147] Discord/58 0 B 26.07 KB
04/May/2024 00:26:02 [nprobe.c:4147] OCSP/63 0 B 57.82 KB
04/May/2024 00:26:02 [nprobe.c:4147] Yahoo/70 75.42 KB 11.97 MB
04/May/2024 00:26:02 [nprobe.c:4147] DisneyPlus/71 0 B 793.49 KB
04/May/2024 00:26:02 [nprobe.c:4147] Steam/74 308 B 2.27 MB
04/May/2024 00:26:02 [nprobe.c:4147] STUN/78 1.99 KB 859.23 KB
04/May/2024 00:26:02 [nprobe.c:4147] IPSec/79 3.15 KB 1.52 MB
04/May/2024 00:26:02 [nprobe.c:4147] ICMP/81 14.11 KB 6.73 MB
04/May/2024 00:26:02 [nprobe.c:4147] RTP/87 14.90 KB 2.34 MB
04/May/2024 00:26:02 [nprobe.c:4147] TLS/91 12.72 KB 168.12 MB
.......many more services omitted for brevity
One thing I didn’t mention in the article is you can also use custom protocol lists, malware lists, and GeoIP data that you create or download.
{ "custom_protocols": "/etc/nprobe/protos.txt" }
{ "category_file": "/etc/nprobe/lists/nfw_malware_list.txt" }
{ "category_file": "/etc/nprobe/lists/nfw_mining_list.txt" }
{ "geoip": { "asn": "/data/dbip-asn-lite-2021-04.mmdb", "city": "/data/dbip-city-lite-2021-04.mmdb" }}
1 Like
This! I saw that when followed the link on your page and was intrigued by that.
Had the APU deployed for years but fiber came and it didn’t have the “oomph” for anything past 1G so that opened the door to trying something new in VyOS.
Have learned a lot since then as a lot of knowledge gets obfuscated behind the GUI.
Regarding the performance difference I think its better to look at x86 and test the very same hardware for both cases.
When going to PC Engines APU and other which isnt x86 based there are plenty of misconfiguration in the kernel complies thats going on and can heavily affect the resulting performance such as pagesizes being used etc. Same with the software being compiled.
Something that gives away that there is something odd going on with their performance numbers is that vanilla Linux who just software routes packets does that at 550Mbps with the PC Engine APU but if you also slap on nprobe that would suddently go at 600Mbps where it should be the other way around.
That is a router doing A should be one speed and the same router doing A+B should be slower.
Looking performance wise Netflix are using FreeBSD servers that dumps 800Gbps of encrypted TLS traffic per server according to: https://papers.freebsd.org/2022/EuroBSDCon/gallatin-The_Other_FreeBSD_Optimizations-Netflix.files/euro2022.pdf
I doubt they would use FreeBSD if the performance difference would be 5x to Linux.
Hi!
I need some assistance regarding this setup.
nprobe seems to be working somewhat but only, that I can see it is logging anything - but the traffic query is irritating me.
04/Dec/2024 16:28:24 [nprobe.c:4117] Average traffic: [2.00 pps][All Traffic 2.35 Kb/sec][IP Traffic 1.84 Kb/sec][ratio 0.80]
04/Dec/2024 16:28:24 [nprobe.c:4125] Current traffic: [1.00 pps][1.92 Kb/sec]
04/Dec/2024 16:28:24 [nprobe.c:4133] L7 Proto Diff Total
04/Dec/2024 16:28:24 [nprobe.c:4147] Unknown/0 7.90 KB 689.77 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] HTTP/7 0 B 156 B
04/Dec/2024 16:28:24 [nprobe.c:4147] NTP/9 76 B 4.53 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] ICMP/81 840 B 70.90 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] TLS/91 756 B 48.21 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] Google/126 0 B 38.78 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] Apple/140 0 B 30.46 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] AppleiCloud/143 0 B 14.16 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] AppleiTunes/145 1.68 KB 25.02 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] Amazon/178 0 B 208 B
04/Dec/2024 16:28:24 [nprobe.c:4147] Twitch/195 0 B 14.00 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] Github/203 0 B 224 B
04/Dec/2024 16:28:24 [nprobe.c:4147] Reddit/205 0 B 143 B
04/Dec/2024 16:28:24 [nprobe.c:4147] Cloudflare/220 0 B 8.17 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] ApplePush/238 0 B 3.19 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] GoogleServices/239 0 B 9.08 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] AppleSiri/254 0 B 1.65 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] AmazonAWS/265 207 B 38.93 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] Azure/276 0 B 2.17 KB
04/Dec/2024 16:28:24 [nprobe.c:4147] iCloudPrivateRelay/277 0 B 755 B
04/Dec/2024 16:28:24 [nprobe.c:4147] GoogleCloud/284 79 B 9.64 KB
It is confusing low throughput. When i check monitor interface for traffic, its on average 1 MB and more - i tried streaming netflix meanwhile.
So where is my traffic going? Do i not understand something important?
There must be something misconfigured because even the ips.conf (testing a ‘twitch’ drop) is not working at all.
Here are my rules (i disabled global-state policies since when they were up, there was no traffic at all logged by nprobe)
Rulesets Information
---------------------------------
ipv4 State Policy
State Packets Bytes Conditions
------- --------- ------- -----------------------------------------------
invalid 24 1387 ct state invalid prefix "[STATE-POLICY-INV-D]"
---------------------------------
ipv4 Firewall "forward filter"
Rule Action Protocol Packets Bytes Conditions
------- -------- ---------- --------- ------- --------------------------------------------------------------------------------------------
2 drop all 0 0 ct state invalid
5 offload all 23308 2286634 ct state { established, related } flow add @VYOS_FLOWTABLE_OT1
15 drop tcp_udp 6613 447516 meta l4proto { tcp, udp } ip daddr 49.51.0.0/16 th dport 1-65535 ip saddr 192.168.11.22
20 queue all 23308 2286634 ct state { established, related } prefix "[ipv4-FWD-filter-20-Q]" queue flags bypass to 25
25 accept all 0 0 ct state { established, related } iifname @I_WAN accept
1000 accept all 10048 1365889 iifname @I_LAN accept
default drop all 0 0
---------------------------------
ipv4 Firewall "input filter"
Rule Action Protocol Packets Bytes Conditions
------- -------- ---------- --------- ------- -----------------------------------------------------------------------
1 accept all 60038 9095899 ct state { established, related } accept
2 drop all 0 0 ct state invalid
10 accept all 117 4356 ct state { related, new } iifname "eth1" accept
40 accept tcp_udp 4980 347529 meta l4proto { tcp, udp } th dport 53 ip saddr @N_NET-INSIDE-v4 accept
50 accept all 0 0 ip saddr 127.0.0.0/8 accept
1000 accept all 4621 1317474 iifname "eth0" accept
1010 accept all 0 0 iifname "lo" accept
default drop all 0 0
---------------------------------
ipv4 Firewall "name OUTSIDE-IN"
Rule Action Protocol Packets Bytes
------- -------- ---------- --------- -------
default drop all 0 0
---------------------------------
Any help appreciated!
ETH0 = LAN
ETH1 = WAN
Topology is something like this: Local Network (192.168.11.0/24) → Eth0 → FW → ETH1 → 192.168.10.0/24 → Fritzbox → WWW