No SSH Needed: Automate VyOS Configuration on Proxmox VE and KubeVirt with a Kubernetes Operator

[tjjh89017]

Hey everyone!

I just released v1.0.0 of vRouter-Operator, a Kubernetes operator that pushes VyOS configuration automatically via QEMU Guest Agent. No SSH, no network access to the router needed.

It now supports two providers:

• KubeVirt — for VyOS VMs running inside Kubernetes (tested on Harvester HCI v1.7.1)
• Proxmox VE — for VyOS VMs running on an external Proxmox cluster (tested on Proxmox VE v9.1.6)

You define your config as Kubernetes resources (VRouterTemplate, VRouterBinding, VRouterTarget), and the operator renders and applies it to your VyOS VMs automatically. It also detects reboots and re-applies config after recovery.

For Proxmox users, the experience feels like writing your VyOS set commands once, and letting the operator handle the rest. No more logging into each VM manually. If a router reboots, the config gets re-applied automatically. And if your VM moves between PVE nodes, the operator just follows it.

GitHub: GitHub - tjjh89017/vrouter-operator: Deploy your vRouter(vyos) in your Kubernetes with KubeVirt · GitHub

Would love to hear if anyone else is managing VyOS this way, or if you have ideas for improvement!

[echowings]

How can we manage VyOS configurations for different environments or multiple VyOS instances in such a scenario?

[tjjh89017]

Not really understand your question.

If I understand correctly, you can have multiple different VRouterTemplate to present different environment. and you can bind those template to the device separately.

So let’s consider you have two environment, two VyOS router for each env.

and the ENV_A is BGP cluster, ENV_B is just VRRP or something. (Just think that are two different sets with quite different usage/purpose)

So here you can define VRouterTemplate named bgp-template

apiVersion: vrouter.kojuro.date/v1
kind: VRouterTemplate
metadata:
  name: bgp-template
  namespace: default
spec:
  commands: |
    set protocols bgp system-as {{ .ASN }}
    ........(all others you need for the router)

apiVersion: vrouter.kojuro.date/v1
kind: VRouterTarget
metadata:
  name: vyos-bgp-1
  namespace: default
spec:
  provider:
    type: kubevirt
    kubevirt:
      name: vyos-bgp-1   # VirtualMachine name
      namespace: default
  params:
    ASN: "65000"

apiVersion: vrouter.kojuro.date/v1
kind: VRouterTarget
metadata:
  name: vyos-bgp-2
  namespace: default
spec:
  provider:
    type: kubevirt
    kubevirt:
      name: vyos-bgp-2   # VirtualMachine name
      namespace: default
  params:
    ASN: "65001"

apiVersion: vrouter.kojuro.date/v1
kind: VRouterBinding
metadata:
  name: bgp-router
  namespace: default
spec:
  templateRef:
    name: bgp-template
  save: true
  targetRefs:
    - name: vyos-bgp-1
    - name: vyos-bgp-2

And after this, you will get two routers with ASN65000 and ASN65001.

We still have two routers without any config, you can define another Template and Binding to bind the different config to those two different router, and you will have VRRP role router (for example)

---
kind: VRouterTarget
metadata:
  name: vyos-vrrp-1
spec:
  provider:
    <omitted>
  params:
    vrrp_priority: 255
---
kind: VRouterTemplate
metadata:
  name: vrrp-template
spec:
  commands: |
    set high-availability vrrp group XXX priority {{ .vrrp_priority }}
    .......<all others for vrrp>
---
kind: VRouterBinding
metadata:
  name: vrrp-binding
spec:
  templateRef:
    name: vrrp-template
  targetRefs:
    - name: vyos-vrrp-1
    - name: vyos-vrrp-2

---

If you are asking, why would we need this?
It can be a vyos cloud controller or controller just like Juniper Apstra to help you deploy BGP-EVPN or something with many devices without manual config.
If we add a new VXLAN or VLAN or VRF, we can simply edit the template to have another new config, and it will use template and the params to generate the final config and push them to the router.

Let’s say a reall deployment I had before. I used Harvester HCI (they don’t have tenant network at that time), I deploy multiple VyOS router in the Harvester Nodes to be the BGP-EVPN endpoint. and if I add a new Harvester Node, I will need to install another VyOS in that specific node for the EVPN. If I add another Customer Tenant, I have to modify vyos config to support new VXLAN and VRF. So I can use this to gerenate all configs for all my routers.

This is similar to, if you used VyOS to be the customer router (managed by you, your customer is a IaaS user), you can use KubeVirt with this template engine, to deploy VyOS on demand. When an user needs a new Tenant, and you can deploy a vyos router with pre-defined config set to deploy the service rapidly.

Hope this can answer you.
If not, please give me more context.
thank you!

[tjjh89017]

Or you can check the demo video.
Even I didn’t use two or more deployment to demo.
But you can imagine if I have more VyOS router and more K8s resource(YAML).
We can deploy different VyOS with different config with each resource.

demo youtube link