VyOS Zabbix Agent Configuration: No Response on TCP Port 10050

Bugs
1

I attempted to configure the Zabbix Agent functionality on VyOS, but it doesn’t seem to be working as expected. Below are the details of the current situation:

Environment:

  • VyOS Version: 1.5-rolling-202501100007
  • Zabbix Server Version: 7.0
  • Zabbix Server Address: 192.168.11.2

VyOS Configuration:

set interfaces ethernet eth0 address '192.168.11.101/24'
set service monitoring zabbix-agent listen-address '192.168.11.101'
set service monitoring zabbix-agent port '10050'
set service monitoring zabbix-agent server '192.168.11.2'
set service monitoring zabbix-agent server-active '192.168.11.2'

Connectivity Between Zabbix Server and VyOS:
The connection between the Zabbix Server and VyOS is confirmed via ICMP.

shakapon@zabbix-server:~$ ping 192.168.11.101
PING 192.168.11.101 (192.168.11.101) 56(84) bytes of data.
64 bytes from 192.168.11.101: icmp_seq=1 ttl=64 time=6.79 ms
64 bytes from 192.168.11.101: icmp_seq=2 ttl=64 time=0.645 ms

Zabbix Agent Binding Status:

  • Confirmed that the agent is bound to 192.168.11.101:10050 and is in the LISTEN state.
sudo netstat -tulnp | grep 10050
tcp        0      0 192.168.11.101:10050    0.0.0.0:*               LISTEN      4012/zabbix_agent2

Zabbix Agent Status:
The Zabbix Agent 2 service appears to be running correctly.

sudo systemctl enable zabbix-agent2
(snip)
     Active: active (running) since Sat 2025-01-11 20:15:53 JST; 20s ago
(snip)

Packet Capture:
Capturing packets on eth0 shows that the Zabbix Server sends traffic to 192.168.11.101:10050, but VyOS doesn’t respond.

Firewall Settings:
No firewall rules are configured.

vyos@vyos:~$ show firewall
Rulesets Information

TCPDUMP Output:
Here is the captured packet when traffic is sent to the Zabbix Agent:

vyos@vyos:~$ sudo tcpdump -i eth0 port 10050 -vv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:00:48.035513 IP (tos 0x0, ttl 64, id 46039, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.11.2.35726 > 192.168.11.101.zabbix-agent: Flags [S], cksum 0x97e6 (incorrect -> 0xe74c), seq 1034714365, win 64240, options [mss 1460,sackOK,TS val 4238238705 ecr 0,nop,wscale 7], length 0

Zabbix Agent Logs:
The logs also indicate that the Zabbix Agent is starting without any issues.

vyos@vyos:~$ sudo journalctl -u zabbix-agent2
Jan 11 20:23:47 vyos systemd[1]: Started zabbix-agent2.service - Zabbix Agent 2.
Jan 11 20:23:47 vyos zabbix_agent2[6173]: Starting Zabbix Agent 2 (6.0.37)
Jan 11 20:23:48 vyos zabbix_agent2[6173]: Zabbix Agent2 hostname: [vyos]
Jan 11 20:23:48 vyos zabbix_agent2[6173]: Press Ctrl+C to exit.

Reboot:
Rebooting VyOS did not change the situation.

Assumption:
It seems that the Zabbix Agent 2 is running on VyOS, and traffic from the Zabbix Server is reaching VyOS. However, for some reason, the agent does not respond to the requests.

Are there any additional points I should check?
I would appreciate any advice or guidance.

Thank you!

2

Try to check generated configuration:

cat /run/zabbix/zabbix-agent2.conf

Probably will find some bugs here. Or zabbix-agent version is not compatible with the zabbix-server version. It is still not clear.

Check the logs:

/var/log/zabbix/zabbix_agent2.log
3 
vyos@vyos:~$ cat /run/zabbix/zabbix-agent2.conf
# Generated by ${vyos_conf_scripts_dir}/service_monitoring_zabbix-agent.py

PidFile=/run/zabbix/zabbix_agent2.pid
LogFile=/var/log/zabbix/zabbix_agent2.log
ControlSocket=/run/zabbix/agent.sock

### Option: LogFileSize
#       Maximum size of log file in MB.
#       0 - disable automatic log rotation.
#
# Range: 0-1024
LogFileSize=0
DebugLevel=3

Server=192.168.11.2
ServerActive=192.168.11.2


ListenPort=10050
ListenIP=192.168.11.101

BufferSend=5
BufferSize=100


Timeout=3

vyos@vyos:~$ sudo cat /var/log/zabbix/zabbix_agent2.log
(snip)
2025/01/12 01:49:55.992689 Starting Zabbix Agent 2 (6.0.37)
2025/01/12 01:49:55.995984 OpenSSL library (OpenSSL 3.0.15 3 Sep 2024) initialized
2025/01/12 01:49:55.996009 default context ciphersuites: TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-PSK-AES128-CBC-SHA PSK-AES128-GCM-SHA256 PSK-AES128-CCM8 PSK-AES128-CCM PSK-AES128-CBC-SHA256 PSK-AES128-CBC-SHA
2025/01/12 01:49:55.996054 psk context ciphersuites: TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-PSK-AES128-CBC-SHA PSK-AES128-GCM-SHA256 PSK-AES128-CCM8 PSK-AES128-CCM PSK-AES128-CBC-SHA256 PSK-AES128-CBC-SHA
2025/01/12 01:49:55.996075 using configuration file: /etc/zabbix/zabbix_agent2.conf
2025/01/12 01:49:55.996127 using plugin 'Agent' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996132 using plugin 'Ceph' (built-in) providing following interfaces: exporter, runner, configurator
2025/01/12 01:49:55.996137 using plugin 'Cpu' (built-in) providing following interfaces: exporter, collector, runner
2025/01/12 01:49:55.996142 using plugin 'DNS' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996146 using plugin 'Docker' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996150 using plugin 'File' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996156 using plugin 'Hw' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996159 using plugin 'Kernel' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996163 using plugin 'Log' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996166 using plugin 'MQTT' (built-in) providing following interfaces: watcher, configurator
2025/01/12 01:49:55.996169 using plugin 'Memcached' (built-in) providing following interfaces: exporter, runner, configurator
2025/01/12 01:49:55.996172 using plugin 'Memory' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996175 using plugin 'Modbus' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996182 using plugin 'MongoDB' (/usr/sbin/zabbix-agent2-plugin/zabbix-agent2-plugin-mongodb) providing following interfaces: exporter, runner, configurator
2025/01/12 01:49:55.996189 using plugin 'Mysql' (built-in) providing following interfaces: exporter, runner, configurator
2025/01/12 01:49:55.996192 using plugin 'NetIf' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996196 using plugin 'Oracle' (built-in) providing following interfaces: exporter, runner, configurator
2025/01/12 01:49:55.996201 using plugin 'PostgreSQL' (/usr/sbin/zabbix-agent2-plugin/zabbix-agent2-plugin-postgresql) providing following interfaces: exporter, runner, configurator
2025/01/12 01:49:55.996208 using plugin 'Proc' (built-in) providing following interfaces: exporter, collector
2025/01/12 01:49:55.996213 using plugin 'ProcExporter' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996217 using plugin 'Redis' (built-in) providing following interfaces: exporter, runner, configurator
2025/01/12 01:49:55.996220 using plugin 'Smart' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996223 using plugin 'Sw' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996226 using plugin 'Swap' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996229 using plugin 'SystemRun' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996232 using plugin 'Systemd' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996240 using plugin 'TCP' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996242 using plugin 'UDP' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996245 using plugin 'Uname' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996247 using plugin 'Uptime' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996250 using plugin 'Users' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996252 using plugin 'VFSDev' (built-in) providing following interfaces: exporter, collector
2025/01/12 01:49:55.996254 using plugin 'VFSDir' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996257 using plugin 'VfsFs' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996260 using plugin 'WebCertificate' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996262 using plugin 'WebPage' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996264 using plugin 'ZabbixAsync' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996267 using plugin 'ZabbixStats' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:49:55.996269 lowering the plugin ZabbixSync capacity to 1 as the configured capacity 100 exceeds limits
2025/01/12 01:49:55.996272 using plugin 'ZabbixSync' (built-in) providing following interfaces: exporter
2025/01/12 01:49:55.996286 [Modbus] Config is valid
2025/01/12 01:49:55.996323 Plugin communication protocol version is 6.0.13
2025/01/12 01:49:55.996336 Zabbix Agent2 hostname: [Zabbix server]
2025/01/12 01:49:55.996352 starting manager
2025/01/12 01:49:55.996477 [101] starting server connector for [192.168.11.2:10051]
2025/01/12 01:49:55.996499 [101] starting memory cache
2025/01/12 01:49:55.996505 [0] starting listener for '0.0.0.0:10050'
2025/01/12 01:49:55.996542 listening for control connections on /run/zabbix/agent.sock
2025/01/12 01:49:56.997645 [101] In refreshActiveChecks() from [192.168.11.2:10051]
2025/01/12 01:49:56.997839 [101] cannot connect to [192.168.11.2:10051]: dial tcp :0->192.168.11.2:10051: connect: network is unreachable
2025/01/12 01:49:56.997845 [101] active check configuration update from host [Zabbix server] started to fail
2025/01/12 01:49:56.997848 [101] End of refreshActiveChecks() from [192.168.11.2:10051]
2025/01/12 01:50:03.085371 listener has been stopped
2025/01/12 01:50:03.085416 [101] server connector has been stopped
2025/01/12 01:50:03.085427 manager has been stopped
2025/01/12 01:50:03.085443 [101] memory cache has been stopped
2025/01/12 01:50:03.085459 Zabbix Agent 2 stopped. (6.0.37)
2025/01/12 01:50:03.092494 Starting Zabbix Agent 2 (6.0.37)
2025/01/12 01:50:03.092978 OpenSSL library (OpenSSL 3.0.15 3 Sep 2024) initialized
2025/01/12 01:50:03.092999 using configuration file: /run/zabbix/zabbix-agent2.conf
2025/01/12 01:50:03.093035 using plugin 'Agent' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093039 using plugin 'Ceph' (built-in) providing following interfaces: exporter, runner, configurator
2025/01/12 01:50:03.093043 using plugin 'Cpu' (built-in) providing following interfaces: exporter, collector, runner
2025/01/12 01:50:03.093046 using plugin 'DNS' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093048 using plugin 'Docker' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093051 using plugin 'File' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093055 using plugin 'Hw' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093058 using plugin 'Kernel' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093065 using plugin 'Log' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093070 using plugin 'MQTT' (built-in) providing following interfaces: watcher, configurator
2025/01/12 01:50:03.093072 using plugin 'Memcached' (built-in) providing following interfaces: exporter, runner, configurator
2025/01/12 01:50:03.093075 using plugin 'Memory' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093077 using plugin 'Modbus' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093079 using plugin 'Mysql' (built-in) providing following interfaces: exporter, runner, configurator
2025/01/12 01:50:03.093083 using plugin 'NetIf' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093086 using plugin 'Oracle' (built-in) providing following interfaces: exporter, runner, configurator
2025/01/12 01:50:03.093089 using plugin 'Proc' (built-in) providing following interfaces: exporter, collector
2025/01/12 01:50:03.093091 using plugin 'ProcExporter' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093093 using plugin 'Redis' (built-in) providing following interfaces: exporter, runner, configurator
2025/01/12 01:50:03.093096 using plugin 'Smart' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093098 using plugin 'Sw' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093100 using plugin 'Swap' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093103 using plugin 'SystemRun' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093105 using plugin 'Systemd' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093110 using plugin 'TCP' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093114 using plugin 'UDP' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093116 using plugin 'Uname' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093119 using plugin 'Uptime' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093121 using plugin 'Users' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093123 using plugin 'VFSDev' (built-in) providing following interfaces: exporter, collector
2025/01/12 01:50:03.093126 using plugin 'VFSDir' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093128 using plugin 'VfsFs' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093131 using plugin 'WebCertificate' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093133 using plugin 'WebPage' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093135 using plugin 'ZabbixAsync' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093137 using plugin 'ZabbixStats' (built-in) providing following interfaces: exporter, configurator
2025/01/12 01:50:03.093139 lowering the plugin ZabbixSync capacity to 1 as the configured capacity 100 exceeds limits
2025/01/12 01:50:03.093141 using plugin 'ZabbixSync' (built-in) providing following interfaces: exporter
2025/01/12 01:50:03.093349 Plugin communication protocol version is 6.0.13
2025/01/12 01:50:04.000818 Zabbix Agent2 hostname: [vyos]
2025/01/12 01:50:08.002238 [101] cannot connect to [192.168.11.2:10051]: dial tcp :0->192.168.11.2:10051: i/o timeout
2025/01/12 01:50:08.002273 [101] active check configuration update from host [vyos] started to fail

When checking the result of sudo cat /var/log/zabbix/zabbix_agent2.log, it shows that Zabbix-Agent2 starts once when VyOS boots, then immediately stops, and starts again. Additionally, two versions of the Zabbix Agent, 6.0.13 and 6.0.37, are visible.

4

Check connectivity from the zabbix agent

nc -zv 192.168.11.2 10051

Good question if the zabbix-agent2 is compatible with zabbix-server 7.0
Did you check zabbix-forums?

5

I’m using Zabbix 7.2 and my VyOS is being monitored by Zabbix just fine.

You haven’t set a Hostname. That’s why it’s not working.

Hostname=<something> should be in your /run/zabbix/zabbix-agent2.conf somewhere.

You need set service monitoring zabbix-agent host-name <something> and the something should align with whatever name you’ve set in your Zabbix Server.

6

I tried setting up a Zabbix Server of Version 6.0.37 at the same address, but the situation remains unchanged.

XXXXXXXX@zabbix60:~$ dpkg -l zabbix-server-pgsql
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version                Architecture Description
+++-===================-======================-============-========================================================
ii  zabbix-server-pgsql 1:6.0.37-1+ubuntu24.04 amd64        Zabbix network monitoring solution - server (PostgreSQL)
XXXXXXXX@zabbix60:~$
XXXXXXXX@zabbix60:~$ sudo zabbix_get -s localhost -p 10050 -k system.hostname
zabbix60
XXXXXXXX@zabbix60:~$ sudo zabbix_get -s 192.168.11.101 -p 10050 -k system.hostname
zabbix_get [27994]: Timeout while executing operation
XXXXXXXX@zabbix60:~$ ping 192.168.11.101
PING 192.168.11.101 (192.168.11.101) 56(84) bytes of data.
64 bytes from 192.168.11.101: icmp_seq=1 ttl=64 time=6.78 ms
64 bytes from 192.168.11.101: icmp_seq=2 ttl=64 time=1.30 ms
^C
--- 192.168.11.101 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 1.297/4.037/6.778/2.740 ms

vyos@vyos:~$ nc -zv 192.168.11.2 10051 -w 10
nc: connect to 192.168.11.2 port 10051 (tcp) timed out: Operation now in progress

When I captured traffic on TCP/10051 using Wireshark, I observed the following behavior:

  • VyOS: Sends TCP/SYN.
  • Zabbix Server: Responds with TCP/SYN+ACK.
  • VyOS: Does not return a TCP/ACK and stops there.

WireSharkClip

7

I tried adding the following configuration, but the situation did not change:

set service monitoring zabbix-agent host-name 'vyos'
set service monitoring zabbix-agent listen-address '192.168.11.101'
set service monitoring zabbix-agent port '10050'
set service monitoring zabbix-agent server '192.168.11.2'
set service monitoring zabbix-agent server-active 192.168.11.2

I also confirmed that the settings are reflected in zabbix-agent2.conf:

vyos@vyos:~$ cat /run/zabbix/zabbix-agent2.conf
# Generated by ${vyos_conf_scripts_dir}/service_monitoring_zabbix-agent.py

PidFile=/run/zabbix/zabbix_agent2.pid
LogFile=/var/log/zabbix/zabbix_agent2.log
ControlSocket=/run/zabbix/agent.sock

### Option: LogFileSize
#       Maximum size of log file in MB.
#       0 - disable automatic log rotation.
#
# Range: 0-1024
LogFileSize=0
DebugLevel=3

Server=192.168.11.2
ServerActive=192.168.11.2

Hostname=vyos

ListenPort=10050
ListenIP=192.168.11.101

BufferSend=5
BufferSize=100

Timeout=3
8

OK from your Zabbix server, if you telnet/netcat to 192.168.11.101 port 10050 what happens?

9 
XXXXXXXX@zabbix60:~$ ping 192.168.11.101
PING 192.168.11.101 (192.168.11.101) 56(84) bytes of data.
64 bytes from 192.168.11.101: icmp_seq=1 ttl=64 time=1.26 ms
^C
--- 192.168.11.101 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.262/1.262/1.262/0.000 ms
XXXXXXXX@zabbix60:~$ telnet 192.168.11.101 10050
Trying 192.168.11.101...
telnet: Unable to connect to remote host: Connection timed out

WireSharkClip2
WireSharkClip22116×162 58.3 KB

10

What about SSH port? Is the situation the same?

11

The issue has been resolved.
It turned out to be a problem with my environment.
Both the Zabbix Server and VyOS were running on the same KVM instance for testing purposes. The root cause was that TCP checksum offloading was enabled on the Zabbix Server.
Since the environment was entirely virtualized, the physical NIC did not calculate the checksums, and packets with incorrect checksums were sent to VyOS.
VyOS discarded these packets as they were considered to have invalid checksums.
By executing sudo ethtool --offload NICXX rx off tx off on the Zabbix Server, the issue was resolved.

The hint about checking whether SSH connections were working was very helpful.
Thank you very much!

12

Thank you very much for coming back and explaining what the issue was, and how you fixed it. These sorts of posts are always helpful for someone else in the future.

Glad you got it resolved!