As far as I know, We can quickly load lots of network records with ipset command on vyos 1.2/1.3. But on vyos 1.4, the iptables was switched to nftables. The ipsetcommand has gone.
# set a firewall group
set firewall group network-group goeip_us
# Shell script to loop add the network ranges with ipset
for networks in `cat /config/usipranges.txt`; do sudo ipset add geoip_us $networks;done
the ipset to load the ip networks very fast.
The question is how to loop add network groups with ipset like command on vyos 1.4 when vyos boot.
the file contain lots of ips，if we set it in vyos configure will make vyos booting very slow. I need to quickly load a file contain more that 8000 networks records.
#!/usr/bin/env python3
import subprocess
import sys
def add_networks_to_set(set_name, filename, batch_size=1000):
table_name = "vyos_filter" # Fixed table name
try:
with open(filename, 'r') as file:
networks = [line.strip() for line in file if line.strip()]
for i in range(0, len(networks), batch_size):
batch_networks = networks[i:i+batch_size]
networks_string = ', '.join(batch_networks)
command = f"nft add element ip {table_name} {set_name} {{ {networks_string} }}"
subprocess.run(command, check=True, shell=True)
print(f"Successfully added batch of networks to {set_name}")
except subprocess.CalledProcessError as e:
print(f"Error adding networks to {set_name}: {e}")
except FileNotFoundError:
print(f"File {filename} not found")
except Exception as e:
print(f"An error occurred: {e}")
def main():
if len(sys.argv) != 3:
print("Usage: python3 test.py <set name> <filename>")
sys.exit(1)
set_name = sys.argv[1]
filename = sys.argv[2]
add_networks_to_set(set_name, filename)
if __name__ == "__main__":
main()
Here’s how to use it:
Create a group:
vyos@Hub# set firewall group network-group TEST
vyos@Hub# commit
Verify group name as a nftables set:
root@Hub:/home/vyos# sudo nft list table ip vyos_filter
table ip vyos_filter {
set N_TEST {
type ipv4_addr
flags interval
auto-merge
}
chain VYOS_FORWARD_filter {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 accept comment "FWD-filter default-action accept"
}
chain VYOS_INPUT_filter {
type filter hook input priority filter; policy accept;
counter packets 36 bytes 3641 accept comment "INP-filter default-action accept"
}
chain VYOS_OUTPUT_filter {
type filter hook output priority filter; policy accept;
counter packets 29 bytes 2932 accept comment "OUT-filter default-action accept"
}
chain VYOS_FRAG_MARK {
type filter hook prerouting priority -450; policy accept;
ip frag-off & 16383 != 0 meta mark set 0x000ffff1 return
}
}
You can see our set is called N_TEST
Run the script in this format: python3 <name of script> <name of set> <name of file>
root@Hub:/home/vyos# python3 test.py N_TEST iplist.txt
Successfully added 192.168.1.0/24 to N_TEST
Successfully added 10.0.0.0/8 to N_TEST
Successfully added 172.16.0.0/12 to N_TEST
Verify:
root@Hub:/home/vyos# sudo nft list set ip vyos_filter N_TEST
table ip vyos_filter {
set N_TEST {
type ipv4_addr
flags interval
auto-merge
elements = { 10.0.0.0/8, 172.16.0.0/12,
192.168.1.0/24 }
}
}
Try something like this. The script can be saved in /config and set up as a cron job.
set firewall group network-group goeip_us
#!/bin/vbash
source /opt/vyatta/etc/functions/script-template
configure
for i in `cat /root/list.txt`;
do
set firewall group network-group goeip_us network $i
done;
commit
exit
Thanks @L0crian, I was just about to comment on yours. It’s much better at adding it directly to the table. I think I’ll try it
@echowings my suggestion for about 1000 IPs, it takes just under two minutes to complete on my machine with an I5 Haswell per its log entry. Machine startup time isn’t long IMO as the group is defined and theres only one drop traffic firewall rule to deny from that table.
The bigger drawback to me is more the fact that it adds these entries to the configuration as a table so running a show conf prints (in your case 8000) extra entries to screen but the tradeoff is a rather simple script using VyOS commands.
I should add that if you make a change to the firewall config section, it will zero out the list populated with the python script. Any section, not necessarily to that specific network group. You’d need to account for that.
This could likely be added fairly easily to VyOS. That would prevent that behavior. Usage could be something like:
set firewall group network-group TEST from-file iplist.txt
I see that there’s a Feature Request that would already cover this, but I’m not sure what the status of it is: https://vyos.dev/T4797
I’d also like to see the lists be able to be populated through API calls to sites like ipinfo.io. That could easily allow you to populate group objects for entire entities like Microsoft or Amazon.
After testing the script with 20000 IPs, it was populating very slow with the iteration. I batched each iteration into batches of 1000 to speed the process up.
It took .59 seconds to load the 20000 IPs.
Updated Script (updated in my previous post as well):
#!/usr/bin/env python3
import subprocess
import sys
import time
def add_networks_in_batches(set_name, filename, batch_size=1000):
table_name = "vyos_filter" # Fixed table name
total_added = 0 # Initialize counter for total added IPs
start_time = time.time() # Record the start time
try:
with open(filename, 'r') as file:
networks = [line.strip() for line in file if line.strip()]
for i in range(0, len(networks), batch_size):
batch_networks = networks[i:i+batch_size]
networks_string = ', '.join(batch_networks)
command = f"nft add element ip {table_name} {set_name} {{ {networks_string} }}"
subprocess.run(command, check=True, shell=True)
print(f"Successfully added batch of networks to {set_name}")
total_added += len(batch_networks) # Update count of added IPs
except subprocess.CalledProcessError as e:
print(f"Error adding networks to {set_name}: {e}")
except FileNotFoundError:
print(f"File {filename} not found")
except Exception as e:
print(f"An error occurred: {e}")
end_time = time.time() # Record the end time
elapsed_time = end_time - start_time # Calculate elapsed time
# Print summary
print(f"\nTotal of {total_added} networks were added to {set_name}.")
print(f"Process took {elapsed_time:.2f} seconds.")
def main():
if len(sys.argv) != 3:
print("Usage: python3 test.py <set name> <filename>")
sys.exit(1)
set_name = sys.argv[1]
filename = sys.argv[2]
add_networks_in_batches(set_name, filename)
if __name__ == "__main__":
main()
@L0crian Thanks for you sharing.
it is really fast
vyos@vyos# bash 1.sh
Successfully added batch of networks to N_TEST
Successfully added batch of networks to N_TEST
Successfully added batch of networks to N_TEST
Successfully added batch of networks to N_TEST
Successfully added batch of networks to N_TEST
Successfully added batch of networks to N_TEST
Successfully added batch of networks to N_TEST
Total of 6143 networks were added to N_TEST.
Process took 0.34 seconds.
[edit]
vyos@vyos# run show firewall group
Firewall Groups
Name Type References Members
------ ------------- ------------ --------------
HELLO network_group N/D 192.168.0.0/24
192.168.1.0/24
TEST network_group N/D N/D
[edit]
vyos@vyos#
Can you run command like this
$show firewall group network-group TEST
Can you get the network lists after the script running? I can’t load it in firewall group. vyos_filter has the network ranges. Do you know how to let vyos got the network lists?
nftables will try to optimize the list. For instance, if you add 10.0.0.0/24 and 10.0.1.0/24, it will merge those into 10.0.0.0/23. So you won’t be able to see your original list item-for-item, since they have been optimized. You’d have to just look at your original list unfortunately.
Example:
vyos@Hub# sudo nft add element ip vyos_filter N_TEST {10.0.0.0/24}
vyos@Hub# sudo nft add element ip vyos_filter N_TEST {10.0.1.0/24}
root@Hub:/home/vyos# sudo nft list set ip vyos_filter N_TEST
table ip vyos_filter {
set N_TEST {
type ipv4_addr
flags interval
auto-merge
elements = { 10.0.0.0/23 }
}
}
VyOS won’t show you the IPs inside of the firewall group, but they will be in use.
vyos@Hub# run show firewall group TEST
Firewall Groups
Name Type References Members
------ ------------- ---------------------- ---------
TEST network_group ipv4-forward-filter-10 N/D
ipv4-output-filter-10
Here’s the order of operations to make this work:
Create the firewall group in VyOS
Apply the group to a rule and commit
Populate the list with the python script
Sadly, any future changes to the firewall section will zero out the values created with the script which will be a bit of a pain. You just have to remember to rerun the script when you make a change.
@L0crian ,
I just tested the script. Upgraded vyos 1.3 to vyos 1.4, with the script to set the PBR. Unfortunately, It’s not working. If the show firewall group network-group TEST is empty. the PBR is not working. VyOS does not accept the NFT filter group.
Nice! Glad you got it working. I’ll add an argument to the script once I get back to my computer. That way anybody that tries it later will be able to use it for filtering and PBR.
Yeah, I mentioned that above. Obviously the best way would be for this functionality to be added to VyOS directly. Something like:
set firewall group network-group TEST from-file iplist.txt
Short of that, you’d have to make it part of your workflow when making firewall changes.
I wouldn’t recommend this for firewall rules since it would leave little periods where traffic isn’t blocked (unless the group is called in an allow rule), but you can create something to track if VyOS changed the ruleset. You can do something like this:
Create a network-group called SET_TRACK
Run the script to add IPs to your set
Add an IP to the SET_TRACK group
Schedule a task to run a script that looks if that IP disappeared from SET_TRACK. If the IP is no longer present, the original script is called.
Here’s the 2 scripts.
Original script
This was modified to require the table name as an argument.
#!/usr/bin/env python3
import subprocess
import sys
import time
def add_networks_in_batches(table_name, set_name, filename, batch_size=1000):
total_added = 0 # Initialize counter for total added IPs
start_time = time.time() # Record the start time
try:
with open(filename, 'r') as file:
networks = [line.strip() for line in file if line.strip()]
for i in range(0, len(networks), batch_size):
batch_networks = networks[i:i+batch_size]
networks_string = ', '.join(batch_networks)
command = f"nft add element ip {table_name} {set_name} {{ {networks_string} }}"
subprocess.run(command, check=True, shell=True)
print(f"Successfully added batch of networks to {set_name}")
total_added += len(batch_networks) # Update count of added IPs
except subprocess.CalledProcessError as e:
print(f"Error adding networks to {set_name}: {e}")
except FileNotFoundError:
print(f"File {filename} not found")
except Exception as e:
print(f"An error occurred: {e}")
end_time = time.time() # Record the end time
elapsed_time = end_time - start_time # Calculate elapsed time
# Print summary
print(f"\nTotal of {total_added} networks were added to {set_name}.")
print(f"Process took {elapsed_time:.2f} seconds.")
def main():
if len(sys.argv) != 4:
print("Usage: python3 test.py <table name> <set name> <filename>")
sys.exit(1)
table_name = sys.argv[1]
set_name = sys.argv[2]
filename = sys.argv[3]
add_networks_in_batches(table_name, set_name, filename)
if __name__ == "__main__":
main()
Tracking script
You will need to modify the subprocces.run in the run_original_script function to call whatever you named your script.
#!/usr/bin/env python3
import subprocess
set_name = "N_SET_TRACK"
table_name = "vyos_filter"
dummy_ip = "127.0.0.152"
def check_set_track(set_name, table_name, dummy_ip):
command = f"nft list set ip {table_name} {set_name}"
result = subprocess.run(command, shell=True, capture_output=True, text=True)
return dummy_ip in result.stdout
def add_dummy_to_set_track(set_name, table_name, dummy_ip):
command = f"nft add element ip {table_name} {set_name} {{ {dummy_ip} }}"
subprocess.run(command, check=True, shell=True)
def run_original_script(table_name, set_name, filename):
subprocess.run(["python3", "test.py", table_name, set_name, filename], check=True)
if __name__ == "__main__":
if not check_set_track(set_name, table_name, dummy_ip):
print("Ruleset has been reset. Repopulating IP sets...")
run_original_script("vyos_filter", "N_TEST", "iplist.txt")
# If you need to populate another set, just replicate the line above with the appropriate arguments
# run_original_script("table name", "set2", "file2.txt")
add_dummy_to_set_track(set_name, table_name, dummy_ip)
else:
print("N_SET_TRACK check passed. No need to repopulate IP sets.")
Also from-url or similar would be nice (with http, https, ftp, sftp, ftps, scp support).
Or just “from-source” and have the backend autodetect if its a local path, http, https etc.
Perhaps add a refresh setting aswell like this so it automatically get refreshed every 60min:
set firewall group network-group TEST from-file iplist.txt refresh 60m
Also with some opmode command to force a manual refresh and some show command to show when it was last synced and how many rows were detected at that time (and size of the file) etc.
