Multiple ports for High Availability virtual-server

Feature requests
1

Virtual Server is very useful, but currently only supports a single port (in CLI). The virtual server seems to be a huge improvement on NAT, and being able to add multiple ports would be incredibly helpful.

Something along the lines of:

set high-availability virtual-server 203.0.113.1 algorithm 'source-hashing'
set high-availability virtual-server 203.0.113.1 delay-loop '10'
set high-availability virtual-server 203.0.113.1 forward-method 'nat'
set high-availability virtual-server 203.0.113.1 persistence-timeout '180'
set high-availability virtual-server 203.0.113.1 rule 10 port '8280'
set high-availability virtual-server 203.0.113.1 rule 10 protocol 'tcp'
set high-availability virtual-server 203.0.113.1 rule 10 real-server 192.0.2.11 port '80'
set high-availability virtual-server 203.0.113.1 rule 10 real-server 192.0.2.12 port '80'
set high-availability virtual-server 203.0.113.1 rule 20 port '443'
set high-availability virtual-server 203.0.113.1 rule 20 protocol 'tcp'
set high-availability virtual-server 203.0.113.1 rule 20 real-server 192.0.2.11 port '443'
set high-availability virtual-server 203.0.113.1 rule 20 real-server 192.0.2.12 port '443'

Perhaps even support for port groups i.e.

set high-availability virtual-server 203.0.113.1 rule 10 group port-group 'http_https'
set high-availability virtual-server 203.0.113.1 rule 10 protocol 'tcp'
set high-availability virtual-server 203.0.113.1 rule 10 real-server 192.0.2.11 group port-group 'http_https'
2

It is a good idea if it supported by keepalived

3

As far as I can understand, keepalived does support multiple ports, and the configuration is exactly the same as for a single port - just another block with the new port and config i.e.:

virtual_server 1.2.3.11 80 {
    delay_loop 6
    lb_algo wlc
    lb_kind NAT
    nat_mask 255.255.255.0
    persistence_timeout 0
    protocol TCP
    real_server 1.2.3.5 80 {
        weight 1
        inhibit_on_failure
        MISC_CHECK {
            misc_path /usr/local/bin/check_http.sh 1.2.3.5
        }
    }
}

virtual_server 1.2.3.11 443 {
    delay_loop 6
    lb_algo wlc
    lb_kind NAT
    nat_mask 255.255.255.0
    persistence_timeout 0
    protocol TCP
    real_server 1.2.3.5 443 {
        weight 1
        inhibit_on_failure
        MISC_CHECK {
            misc_path /usr/local/bin/check_https.sh 1.2.3.5
        }
    }
}

(config stolen directly from keepalived’s sourceforge: Re: [Keepalived-devel] how to do it with more than one port ? | keepalived)

Port groups might be slightly more difficult, and I would assume looping through the ports in the group and adding groups for each of them - which may for all I know be exactly how groups are treated in other parts of the system.

4

I created the task T4904

1 Like
5

Fwmark will be available in the next rolling release
Example of marking and config in the task

6

Amazing! Thank you so much for very quickly implementing this! :slight_smile:

Will try the next build when it’s out.

7

Hi, thank you so much for making this! The fwmark works very well!

There is still an issue if different ports should be routed to different servers.

Say I wanted to port 80/443 to a webserver/reverse proxy, and 22 to a bastion server, I cannot see that would be possible with the current setup.

With the fwmark everything will go to the same server(s).

Either way, excellent upgrade for sure! Thank you very much for making it!

Adding rules to the fwmark would allow me to do:

set high-availability virtual-server 1.2.3.4 rule 10 fwmark '111'
set high-availability virtual-server 1.2.3.4 rule 10 protocol 'tcp'
set high-availability virtual-server 1.2.3.4 rule 10 real-server 10.0.0.4 health-check script '/bin/true'
set high-availability virtual-server 1.2.3.4 rule 10 real-server 10.0.0.4 port '0'

set high-availability virtual-server 1.2.3.4 rule 20 fwmark '112'
set high-availability virtual-server 1.2.3.4 rule 20 protocol 'tcp'
set high-availability virtual-server 1.2.3.4 rule 20 real-server 10.0.0.5 health-check script '/bin/true'
set high-availability virtual-server 1.2.3.4 rule 20 real-server 10.0.0.5 port '22'

set policy route PR interface 'eth0'

set policy route PR rule 10 destination port '80,443'
set policy route PR rule 10 protocol 'tcp'
set policy route PR rule 10 set mark '111'

set policy route PR rule 20 destination port '22'
set policy route PR rule 20 protocol 'tcp'
set policy route PR rule 20 set mark '112'
8

Does it not solve with fwmark for different ports?
Virtual server it just any name in this case and all decision will be match mark and real server configs

set high-availability virtual-server foo rule 20 fwmark ‘111’

set high-availability virtual-server bar rule 20 fwmark ‘112’

9

I can set up different rules and make differences for the PR part, the problem is the set high-availability side of the equation.

I guess this would work if I had multiple IPs, but I’m trying to get 80/443 to one server, and 22 to another server on the same IP address.

So with the current setup that I have running:

set high-availability virtual-server 1.2.3.4 fwmark '111'
set high-availability virtual-server 1.2.3.4 protocol 'tcp'
set high-availability virtual-server 1.2.3.4 real-server 10.0.0.4 health-check script '/bin/true'
set high-availability virtual-server 1.2.3.4 real-server 10.0.0.4 port '0'

There doesn’t seem to be a way for me now to say “and for fwmark ‘112’ send it to that other server instead”

As far as I can understand that would look like this:

set high-availability virtual-server 1.2.3.4 fwmark '111'
set high-availability virtual-server 1.2.3.4 fwmark '112'
set high-availability virtual-server 1.2.3.4 protocol 'tcp'
set high-availability virtual-server 1.2.3.4 real-server 10.0.0.4 health-check script '/bin/true'
set high-availability virtual-server 1.2.3.4 real-server 10.0.0.4 port '0'
set high-availability virtual-server 1.2.3.4 real-server 10.0.0.5 health-check script '/bin/true'
set high-availability virtual-server 1.2.3.4 real-server 10.0.0.5 port '22'

Which I don’t think would work.

10

Try Marks above and this rules
foo and bar as it is in the config

11

I’m an idiot! Thank you :smiley:

12

Let us know if it works as expected
Also replace health-check script to own script check
As it was just an example that returns always true state

:slight_smile:

13

Hi. I just tested this and think it would be good to allow multiple virtual hosts on the same IP but on different ports, translated to other real server ports, without having to resort to fwmark (not even sure fwmark can do that). Something like this:

set high-availability virtual-server 1.2.3.4:80 real-server 5.6.7.8 port 8080
set high-availability virtual-server 1.2.3.4:443 real-server 5.6.7.8 port 8443

Perhaps add a prefix node to specify the type of virtual server (host:port, fwmark or virtual server group). .

set high-availability virtual-server address_port 1.2.3.4:80 real-server 5.6.7.8 port 8080
set high-availability virtual-server fwmark 111 real-server 5.6.7.8
set high-availability virtual-server group my_group ...

The type could technically be decoded from the node name itself even without the prefix

This matches more closely the options from keepalived.conf(5) — keepalived — Debian unstable — Debian Manpages

A virtual_server can be a declaration of one of <IPADDR> [<PORT>] , fwmark <INTEGER> or group <STRING>

The syntax for virtual_server is :
virtual_server <IPADDR> [<PORT>]  |
virtual_server fwmark <INTEGER> |
virtual_server group <STRING> {
...
}
14

I agree with @ahovda.

Having an easy way to do loadbalancing is a somewhat unique feature these days in both firewalls and routers.

That is be able to do this through the “set high-availability virtual-server” without having to involve a 3rd party solution such as F5, A10 or Kemp among others or internally without having to do some ninjawork with fwmarks etc.

On the other hand while looking at this in version VyOS 1.4-rolling-202306290317 wouldnt the below do just whats being asked or am I missing something here?

 high-availability {
     virtual-server 1.2.3.4:80 {
         algorithm round-robin
         delay-loop 10
         forward-method direct
         persistence-timeout 180
         port 80
         protocol tcp
         real-server 2.2.2.2 {
             port 8080
         }
         real-server 3.3.3.3 {
             port 8080
         }
     }
     virtual-server 1.2.3.4:443 {
         algorithm round-robin
         delay-loop 10
         forward-method direct
         persistence-timeout 180
         port 443
         protocol tcp
         real-server 2.2.2.2 {
             port 8443
         }
         real-server 3.3.3.3 {
             port 8443
         }
     }
 }
15

Based on your ports you need reverse-proxy :wink:
https://docs.vyos.io/en/latest/configuration/loadbalancing/reverse-proxy.html

16

If reverse-proxy in VyOS also can do non-http requests (DNS, SMTP etc) then whats the difference between virtual-server and reverse-proxy?

In my world a reverse-proxy have always been limited to http/https requests (and because of that it can also operate on the http(s) request itself including ssl/tls stuff including acting as ssl-termination that is client connects to the reverse-proxy using https but once the request is forwarded to the backend it can be decrypted as plain http).

17

Reverse proxy doesn’t support UDP packets.
Different backends. keepalived-ipvsadm vs haproxy. If you need udp balancing or several marked ports you should use of course ha virtual-server

18

From looking at the j2 template that creates the keepalived.conf file if you use the addr:port syntax as a virtual server conf node, you will afaict end up with the wrong syntax in the .conf file.

virtual_server 1.2.3.4:80 80 {}
instead of
virtual_server 1.2.3.4 80 {}

19

So we have stumbled upon a configuration bug?

That is the config accepts a syntax which then results in bad configuration used by the service?

20

As I remember Virtual server could any name, like server01, foo, bar