Solution to ban large numbers of IP addresses published in blacklists

We like to block the torgateway IPs (7000 entries, https://www.dan.me.uk/torlist/) directly on our vyos router.

Now, with vyos we would need for every entry this line:

set firewall group address-group blacklist address x.x.x.x

This will creates as well ip Sets tables.
See:
{code}
vyos@router1:~$ sudo su -
root@router2:~# ipset list
Name: GROUP-ADMINS
Type: hash:ip
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 16560
References: 5
Members:
192.168.20.1
192.168.20.2
192.168.20.3
192.168.20.4
192.168.20.5
{code}

Actually we using already the python library vymgmt ( Welcome to VyMGMT’s documentation! — VyMGMT 0.1 documentation) for blocking 1200 SPAM networks. The python script (automated by a cronjon) is doing a remote login and adds these entries. Up to now it needs around 20 minutes to execute.

{code}
import vymgmt

for item in networks:
vyos.set(“firewall group network-group {0} network {1}”.format(ipListName, item))

{code}

Like torgateway block list contains even more IPs, we have to find an other solution than with the python library, to solve this.

What is the recommended way on vyos to block a large number of IPs, without the vyos get exhausted by this job (reboot time?)

There is a git project existing (GitHub - trick77/ipset-blacklist: A bash script to ban large numbers of IP addresses published in blacklists.) which updates the ipset list by a script. As default configuration it will block around 123’000 IP Adresses in less than 5 seconds on my own linux machine.
May i can integrate this script and bypass the “set firewall group address-group blacklist address x.x.x.x” way without breaking the system ?

Thanks for your help!

Regards,
Philipp

Don’t put any entries in the group configuration. Just leave the group unpopulated.
Run ipset commands to directly populate the group, on my edgerouter-x it will only take a few seconds, to download/combine several blacklists and populate an ipset with 7000+ entries, … and I bet your CPU is way more powerfull than the ER-X

Fantastic tutorial. Helped me get on the right path.