Latest rolling image many open ports by default

h00k · September 9, 2023, 3:00pm

i recently upgraded from an old 2022 image to the latest 1.5-rolling-202309080021 and kicked off a port scan to see if anything naughty was going on. Found many open ports exposed:

2609/tcp open system-monitor
2612/tcp open qpasa-agent
2613/tcp open smntubootstrap
2616/tcp open appswitch-emp
2617/tcp open cmadmin
2622/tcp open metricadbc

connecting to these ports returns this

Escape character is ‘^]’.
Vty password is not set.
Connection closed by foreign host.

seems like a bad thing to have a bunch of ports exposed just from an image update? i cant seem to find any docs on this.

side note: T5160 seems insane - my simple config just became 3x larger and more complex. i ended up landing on an early august build to not rope dealing with a weekend image upgrade.

Apachez · September 9, 2023, 3:21pm

There is a subthread about this in T5160 which is about the refactoring of the firewall code in VyOS:

https://vyos.dev/T5160#156423

In short, current take seems to be that you are on your own to properly setup the INPUT/OUTPUT filter (in the firewall settings) you wish for your installation.

I personally would prefer to have some kind of basic CoPP ACL similar to how Arista does things (mentioned at the link above).

Also note that the result from your portscanner isnt necessary correct.

Here is the output of my VyOS 1.5-rolling-202309080021 in the lab:

vyos@vyos:~$ sudo netstat -atunp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 192.168.56.2:22         0.0.0.0:*               LISTEN      3196/sshd: /usr/sbi 
tcp        0      0 127.0.0.1:199           0.0.0.0:*               LISTEN      3053/snmpd          
tcp        0      0 127.0.0.1:2601          0.0.0.0:*               LISTEN      909/zebra           
tcp        0      0 127.0.0.1:2602          0.0.0.0:*               LISTEN      923/ripd            
tcp        0      0 127.0.0.1:2605          0.0.0.0:*               LISTEN      916/bgpd            
tcp        0      0 127.0.0.1:2604          0.0.0.0:*               LISTEN      929/ospfd           
tcp        0      0 127.0.0.1:2609          0.0.0.0:*               LISTEN      942/babeld          
tcp        0      0 127.0.0.1:2608          0.0.0.0:*               LISTEN      939/isisd           
tcp        0      0 127.0.0.1:2613          0.0.0.0:*               LISTEN      958/eigrpd          
tcp        0      0 127.0.0.1:2612          0.0.0.0:*               LISTEN      948/ldpd            
tcp        0      0 127.0.0.1:2617          0.0.0.0:*               LISTEN      966/bfdd            
tcp        0      0 127.0.0.1:2616          0.0.0.0:*               LISTEN      961/staticd         
tcp        0      0 127.0.0.1:2622          0.0.0.0:*               LISTEN      914/mgmtd           
tcp        0      0 192.168.56.2:22         192.168.56.1:49638      ESTABLISHED 3403/sshd: vyos [pr 
tcp6       0      0 ::1:2606                :::*                    LISTEN      934/ospf6d          
tcp6       0      0 ::1:2603                :::*                    LISTEN      926/ripngd          
udp        0      0 192.168.56.2:47615      0.0.0.0:*                           3053/snmpd          
udp        0      0 0.0.0.0:3784            0.0.0.0:*                           966/bfdd            
udp        0      0 0.0.0.0:3784            0.0.0.0:*                           966/bfdd            
udp        0      0 0.0.0.0:3784            0.0.0.0:*                           966/bfdd            
udp        0      0 127.0.0.1:123           0.0.0.0:*                           2964/chronyd        
udp        0      0 127.0.0.1:161           0.0.0.0:*                           3053/snmpd          
udp        0      0 192.168.56.2:161        0.0.0.0:*                           3053/snmpd          
udp        0      0 127.0.0.1:323           0.0.0.0:*                           2964/chronyd        
udp        0      0 0.0.0.0:4784            0.0.0.0:*                           966/bfdd            
udp        0      0 0.0.0.0:4784            0.0.0.0:*                           966/bfdd            
udp        0      0 0.0.0.0:4784            0.0.0.0:*                           966/bfdd            
udp6       0      0 :::3784                 :::*                                966/bfdd            
udp6       0      0 :::3784                 :::*                                966/bfdd            
udp6       0      0 :::3784                 :::*                                966/bfdd            
udp6       0      0 :::3785                 :::*                                966/bfdd            
udp6       0      0 :::3785                 :::*                                966/bfdd            
udp6       0      0 :::3785                 :::*                                966/bfdd            
udp6       0      0 ::1:161                 :::*                                3053/snmpd          
udp6       0      0 ::1:323                 :::*                                2964/chronyd        
udp6       0      0 :::4784                 :::*                                966/bfdd            
udp6       0      0 :::4784                 :::*                                966/bfdd            
udp6       0      0 :::4784                 :::*                                966/bfdd

In my case in the above whats listening to 127.0.0.1 or ::1: (loopback for IPv6) wont be accessible from the outside (you could still protect that in the firewall if not to get a log of abusive srcip’s).

In my case only bfdd listens for any IP-address for obvious reasons (could be limited down to only allow from the srcip’s I do have a bfd peer with).

Would be nice if you could provide the output of sudo netstat -atunp after a freshly rebooted VyOS box and compare to your portscan?

Apachez · September 9, 2023, 3:31pm

My results (TCP) regarding MGMT-interface:

#nmap -T5 -A -P0 -n -p1-65535 192.168.56.2
Starting Nmap 7.92 ( https://nmap.org ) at 2023-09-09 17:30 CEST
Nmap scan report for 192.168.56.2
Host is up (0.0046s latency).
Not shown: 65534 closed tcp ports (conn-refused)
PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 9.2p1 (protocol 2.0)

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 3.45 seconds

and towards the WAN-interface:

#nmap -T5 -A -P0 -n -p1-65535 192.168.1.2
Starting Nmap 7.92 ( https://nmap.org ) at 2023-09-09 17:29 CEST
Nmap scan report for 192.168.1.2
Host is up (0.0097s latency).
All 65535 scanned ports on 192.168.1.2 are in ignored states.
Not shown: 65535 closed tcp ports (conn-refused)

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 2.45 seconds

h00k · September 9, 2023, 4:04pm

hey dude thanks for the weekend reply i didnt even think to fire off a netstat and seeing these ports are the routing protocol daemons makes perfect sense.

i tried to revert down to 1.4-202308111749 from 1.5-latest but couldnt login to the box. so i booted from the 2022 image and then upgraded to 1.4-202308111749 and its working as expected.

The routing protocol daemons are all still running but no longer exposed on any of my interfaces.

I noticed when i upgraded my config was “automagically” upgrade to the “new” format. maybe an issue with the config transpiler? I am happy to spin up a vm on a fresh 1.5-latest if you;d like me to confirm but i bet this is an issue parsing the config

h00k · September 9, 2023, 4:10pm

also if you are taking community feedback - really hate the overly verbose new syntax. was really disappointed when i did my upgrade and my config felt overwhelming. vyos was a favorite of mine due to its relatively simple config syntax =[

Apachez · September 9, 2023, 4:29pm

Im just a happy community member myself and have nothing to do with whatever decisions the VyOS maintainers make.

For me I prefer the new syntax but thats probably since I have been using iptables since around year 2000. Nowadays nft (nftables) is the userland tool to be used to program the firewall in the Linux kernel.

With that being said I wouldnt be surprised if the zone-based approach would return (but based on the new syntax) in future.

Im currently doing my own template for that where my approach compared to the original zone-based syntax is to only care for traffic towards a zone (fewer zones to keep track of).

Hopefully it will become good enough to become a template for future zone-based syntax if that will return.

Wall of text (note that the below have default drop for FORWARD filter and default allow for INPUT/OUTPUT filter so you will probably need to adjust things if you want to use my template below):

set firewall global-options all-ping 'enable'
set firewall global-options broadcast-ping 'disable'
set firewall global-options ip-src-route 'disable'
set firewall global-options ipv6-receive-redirects 'disable'
set firewall global-options ipv6-source-validation 'strict'
set firewall global-options ipv6-src-route 'disable'
set firewall global-options log-martians 'enable'
set firewall global-options receive-redirects 'disable'
set firewall global-options resolver-cache
set firewall global-options resolver-interval '60'
set firewall global-options send-redirects 'disable'
set firewall global-options source-validation 'strict'
set firewall global-options syn-cookies 'enable'
set firewall global-options twa-hazards-protection 'disable'
set firewall group interface-group DMZ interface 'eth2'
set firewall group interface-group LAN interface 'eth3'
set firewall group interface-group MGMT interface 'eth0'
set firewall group interface-group WAN interface 'eth1'
set firewall group ipv6-network-group V6_DMZ
set firewall group ipv6-network-group V6_LAN
set firewall group ipv6-network-group V6_MGMT
set firewall group ipv6-network-group V6_WAN network '::/0'
set firewall group network-group V4_BOGONS network '0.0.0.0/8'
set firewall group network-group V4_BOGONS network '10.0.0.0/8'
set firewall group network-group V4_BOGONS network '100.64.0.0/10'
set firewall group network-group V4_BOGONS network '127.0.0.0/8'
set firewall group network-group V4_BOGONS network '169.254.0.0/16'
set firewall group network-group V4_BOGONS network '172.16.0.0/12'
set firewall group network-group V4_BOGONS network '192.0.0.0/24'
set firewall group network-group V4_BOGONS network '192.0.2.0/24'
set firewall group network-group V4_BOGONS network '192.168.0.0/16'
set firewall group network-group V4_BOGONS network '198.18.0.0/15'
set firewall group network-group V4_BOGONS network '198.51.100.0/24'
set firewall group network-group V4_BOGONS network '203.0.113.0/24'
set firewall group network-group V4_BOGONS network '224.0.0.0/4'
set firewall group network-group V4_BOGONS network '240.0.0.0/4'
set firewall group network-group V4_DMZ network '192.168.2.0/24'
set firewall group network-group V4_LAN network '192.168.3.0/24'
set firewall group network-group V4_MGMT network '192.168.56.0/24'
set firewall group network-group V4_RFC1918 network '10.0.0.0/8'
set firewall group network-group V4_RFC1918 network '172.16.0.0/12'
set firewall group network-group V4_RFC1918 network '192.168.0.0/16'
set firewall group network-group V4_WAN network '192.168.1.0/24'
set firewall group network-group V4_WAN network '0.0.0.0/0'
set firewall ipv4 forward filter default-action 'drop'
set firewall ipv4 forward filter rule 10 action 'accept'
set firewall ipv4 forward filter rule 10 state established 'enable'
set firewall ipv4 forward filter rule 10 state related 'enable'
set firewall ipv4 forward filter rule 20 action 'drop'
set firewall ipv4 forward filter rule 20 state invalid 'enable'
set firewall ipv4 forward filter rule 30 action 'jump'
set firewall ipv4 forward filter rule 30 jump-target 'V4_TO_WAN'
set firewall ipv4 forward filter rule 30 outbound-interface interface-group 'WAN'
set firewall ipv4 forward filter rule 40 action 'jump'
set firewall ipv4 forward filter rule 40 jump-target 'V4_TO_DMZ'
set firewall ipv4 forward filter rule 40 outbound-interface interface-group 'DMZ'
set firewall ipv4 forward filter rule 50 action 'jump'
set firewall ipv4 forward filter rule 50 jump-target 'V4_TO_LAN'
set firewall ipv4 forward filter rule 50 outbound-interface interface-group 'LAN'
set firewall ipv4 input filter default-action 'accept'
set firewall ipv4 input filter rule 10 action 'accept'
set firewall ipv4 input filter rule 10 state established 'enable'
set firewall ipv4 input filter rule 10 state related 'enable'
set firewall ipv4 input filter rule 20 action 'drop'
set firewall ipv4 input filter rule 20 state invalid 'enable'
set firewall ipv4 input filter rule 999999 action 'accept'
set firewall ipv4 input filter rule 999999 inbound-interface interface-name 'lo'
set firewall ipv4 input filter rule 999999 source address '127.0.0.0/8'
set firewall ipv4 name V4_TO_DMZ default-action 'drop'
set firewall ipv4 name V4_TO_LAN default-action 'drop'
set firewall ipv4 name V4_TO_MGMT default-action 'drop'
set firewall ipv4 name V4_TO_WAN default-action 'drop'
set firewall ipv4 output filter default-action 'accept'
set firewall ipv4 output filter rule 10 action 'accept'
set firewall ipv4 output filter rule 10 state established 'enable'
set firewall ipv4 output filter rule 10 state related 'enable'
set firewall ipv4 output filter rule 20 action 'drop'
set firewall ipv4 output filter rule 20 state invalid 'enable'
set firewall ipv4 output filter rule 999999 action 'accept'
set firewall ipv4 output filter rule 999999 destination address '127.0.0.0/8'
set firewall ipv4 output filter rule 999999 outbound-interface interface-name 'lo'
set firewall ipv6 forward filter default-action 'drop'
set firewall ipv6 forward filter rule 10 action 'accept'
set firewall ipv6 forward filter rule 10 state established 'enable'
set firewall ipv6 forward filter rule 10 state related 'enable'
set firewall ipv6 forward filter rule 20 action 'drop'
set firewall ipv6 forward filter rule 20 state invalid 'enable'
set firewall ipv6 forward filter rule 30 action 'jump'
set firewall ipv6 forward filter rule 30 jump-target 'V6_TO_WAN'
set firewall ipv6 forward filter rule 30 outbound-interface interface-group 'WAN'
set firewall ipv6 forward filter rule 40 action 'jump'
set firewall ipv6 forward filter rule 40 jump-target 'V6_TO_DMZ'
set firewall ipv6 forward filter rule 40 outbound-interface interface-group 'DMZ'
set firewall ipv6 forward filter rule 50 action 'jump'
set firewall ipv6 forward filter rule 50 jump-target 'V6_TO_LAN'
set firewall ipv6 forward filter rule 50 outbound-interface interface-group 'LAN'
set firewall ipv6 input filter default-action 'accept'
set firewall ipv6 input filter rule 10 action 'accept'
set firewall ipv6 input filter rule 10 state established 'enable'
set firewall ipv6 input filter rule 10 state related 'enable'
set firewall ipv6 input filter rule 20 action 'drop'
set firewall ipv6 input filter rule 20 state invalid 'enable'
set firewall ipv6 input filter rule 999999 action 'accept'
set firewall ipv6 input filter rule 999999 inbound-interface interface-name 'lo'
set firewall ipv6 input filter rule 999999 source address '::1/128'
set firewall ipv6 name V6_TO_DMZ default-action 'drop'
set firewall ipv6 name V6_TO_LAN default-action 'drop'
set firewall ipv6 name V6_TO_MGMT default-action 'drop'
set firewall ipv6 name V6_TO_WAN default-action 'drop'
set firewall ipv6 output filter default-action 'accept'
set firewall ipv6 output filter rule 10 action 'accept'
set firewall ipv6 output filter rule 10 state established 'enable'
set firewall ipv6 output filter rule 10 state related 'enable'
set firewall ipv6 output filter rule 20 action 'drop'
set firewall ipv6 output filter rule 20 state invalid 'enable'
set firewall ipv6 output filter rule 999999 action 'accept'
set firewall ipv6 output filter rule 999999 destination address '::1/128'
set firewall ipv6 output filter rule 999999 outbound-interface interface-name 'lo'

So with the above template stuff that the VyOS runs itself like SSH, VRRP, DHCP-server etc must be allowed for in the INPUT/OUTPUT filters (preferably change the default-action to drop).

But all the regular firewalling (between interfaces/zones) goes into the V4_TO_ / V6_TO_.

Technically zones are just interface-groups.

Edit 230930: Turns out that state established and related can be defined in the same rule nowadays so the above example is now updated for that.

Edit 231007: Added ipv6-source-validation strict so uRPF strict (BCP38) is used for both IPv4 and IPv6. Your usecase might need to change “strict” to “loose”.

h00k · September 9, 2023, 4:38pm

lol we’re probably the same age and i’ve been avoiding iptables since around the same time (miss you Shorewall).

thanks for the template. honestly i plan on just sticking with the old build for the zone configs because zones just click in my little brain. ill save all this new mumbo-jumbo for a rainy weekend and rewrite all my configs from scratch.

thanks for your help @Apachez appreciate you

Apachez · September 11, 2023, 8:37am

There is a new nightly out now, could you please (if possible) try that and then verify with netstat what is actually listening on interfaces other than 127.0.0.1 and ::1: ?

https://vyos.net/get/nightly-builds/

Apachez · September 11, 2023, 10:43am

Note that there have been identified some kind of race-condition between frr and vyos-router service which might explain the reason for why frr sometimes starts with “default” settings and exposing interfaces vs when its correctly started (as defined by /etc/frr/daemons file) and everything is locked down to localhost.

This race-condition could also explain some other anomalies who already have tasks created over at https://vyos.dev

h00k · September 11, 2023, 4:49pm

how can i say no to you, you’re the hero of these forums I will need to do an ova export of my current vm, spin it up in a lab, perform the upgrade, run the port scan.

give me a few hours to bang out some regular work and i can work on this in the bg.

JoeN · September 11, 2023, 5:14pm

Hi @h00k - Noted! Thank you for sharing your feedback with us. Your feedback is essential to us as we continue to refine VyOS products.

Best regards,
Joe

h00k · September 12, 2023, 4:51pm

i fired up a fresh vm on the latest rolling 1.5-rolling-202309120021 and just did a netstat and looks like these services are listening on all interfaces and are open.

Do you want me to test a config upgrade to see if the results are the same? cant imagine it will be diff

Apachez · September 12, 2023, 6:01pm

Can you verify that /usr/lib/systemd/system/vyos-router.service is missing:

Requires=frr.service

and that /etc/frr/daemons looks like this?

zebra=yes
bgpd=yes
ospfd=yes
ospf6d=yes
ripd=yes
ripngd=yes
isisd=yes
pimd=no
pim6d=yes
ldpd=yes
nhrpd=no
eigrpd=yes
babeld=yes
sharpd=no
pbrd=no
bfdd=yes
staticd=yes

vtysh_enable=yes
zebra_options="  -s 90000000 --daemon -A 127.0.0.1 -M snmp"
bgpd_options="   --daemon -A 127.0.0.1 -M snmp"
ospfd_options="  --daemon -A 127.0.0.1 -M snmp"
ospf6d_options=" --daemon -A ::1 -M snmp"
ripd_options="   --daemon -A 127.0.0.1 -M snmp"
ripngd_options=" --daemon -A ::1"
isisd_options="  --daemon -A 127.0.0.1 -M snmp"
pimd_options="  --daemon -A 127.0.0.1"
pim6d_options=" --daemon -A ::1"
ldpd_options="  --daemon -A 127.0.0.1 -M snmp"
mgmtd_options=" --daemon -A 127.0.0.1"
nhrpd_options="  --daemon -A 127.0.0.1"
eigrpd_options="  --daemon -A 127.0.0.1"
babeld_options="  --daemon -A 127.0.0.1"
sharpd_options="  --daemon -A 127.0.0.1"
pbrd_options="  --daemon -A 127.0.0.1"
staticd_options="  --daemon -A 127.0.0.1"
bfdd_options="  --daemon -A 127.0.0.1"

watchfrr_enable=no
valgrind_enable=no

Also do you have any config configured?

And is this a live-cd boot or installed on a harddrive (well virtual disk since you run it as a VM)?

Apachez · September 13, 2023, 1:39am

Hopefully this commit once merged will fix this issue:

github.com/vyos/vyos-1x

frr: T5239: fix process startup order

vyos:current ← c-po:t5239-frr

opened 08:02PM - 11 Sep 23 UTC

c-po

+16 -21

## Change Summary - Reuse existing utility functions to check if a boot is …ongoing (boot_configuration_complete()) - Run system_frr.py script to configure FRR daemon before initial launch - Add safety net to always have FRR running on the system This does yet not solve the error in T5239 but it's a small step towards the solution. ## Types of changes  - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Code style update (formatting, renaming) - [x] Refactoring (no functional changes) - [ ] Migration from an old Vyatta component to vyos-1x, please link to related PR inside obsoleted component - [ ] Other (please describe): ## Related Task(s) * https://vyos.dev/T5239 ## Related PRs * https://github.com/vyos/vyos-build/pull/401 ## Component(s) name FRR and vyos-router startup ## Proposed changes ## How to test  Smoketests ## Checklist: - [x] I have read the [**CONTRIBUTING**](https://github.com/vyos/vyos-1x/blob/current/CONTRIBUTING.md) document - [x] I have linked this PR to one or more Phabricator Task(s) - [x] I have run the components [**SMOKETESTS**](https://github.com/vyos/vyos-1x/tree/current/smoketest/scripts/cli) if applicable - [ ] My commit headlines contain a valid Task id - [ ] My change requires a change to the documentation - [ ] I have updated the documentation accordingly

Apachez · September 13, 2023, 6:55pm

Now the above PR have been merged so expect it to be in the next nightly (230914 or later).

Please get back with the results after tomorrow (that is when the merge exists in the nightly).

h00k · September 21, 2023, 1:28pm

hey sorry for the delay - fiscal EOY partying over here.

live cd
vyos-router.service is identical
frr/daemons is identical

grabbing the latest iso to check it, standby

Apachez · September 24, 2023, 3:05pm

How did it go?

Except for the fixes of how FRR is started during boot there now also only exists a single daemons.tmpl to be used (and that have also been cleaned up to better match the original daemons file provided by FRR stable/9.0).