Deploying VyOS using Terraform on ESXi

General questions
1

I have a requirement to deploy small networks of roughly 30 VMs that are used for training, and then destroy them after about a week. I use Terraform to automate the deployment of these networks. This deployment is happening on-prem on ESXi.

I use Vyos for all the routers. However, I’m having an issue deploying Vyos using Terraform. I can get the Vyos VM to deploy, but I cannot get the network adapters to accept any configurations from Terraform. When I add any scripts to configure the interfaces, Terraform fails. What ends up happening is the VM deploys, it contains the correct number of network interfaces (set using Terraform), but none of the interfaces have an IP address. This is the point where I’m stuck.

To get past the Terraform issue I use Ansible to configure the Vyos routers, but in order for Ansible to connect to the router it needs an IP address. So to get Ansible to work I need to console into each router in order to set the IP address manually. This works, but these extra steps really hinders the automation process. Sometimes I’m deploying four or more routers, and having to console into each router to configure it is taking a lot of time, and introduces errors.

I’ve tried using the ‘extra_config’ module of Terraform in order to try and pass configurations to Vyos during the deployment phase, but I cannot get it to work.

Here is the minimal Terraform code I’m using to deploy the VM:

# DEPLOY Core-Router Router
resource "vsphere_virtual_machine" "RTR-Core-01" {
  name                          = "${var.RGE_PREFIX}-RTR-Core-01"        
  folder                        = "${vsphere_folder.BlueSpace.path}"
  resource_pool_id              = "${data.vsphere_resource_pool.pool.id}"
  datastore_id                  = "${data.vsphere_datastore.datastore.id}"
  num_cpus                      = 2
  memory                        = 4096
  guest_id                      = "${data.vsphere_virtual_machine.Core-Router-01.guest_id}"
  scsi_type                     = "${data.vsphere_virtual_machine.Core-Router-01.scsi_type}"
  firmware                      = "efi"
  wait_for_guest_net_timeout    = -1
 
  network_interface {
    network_id                  = "${data.vsphere_network.VLAN-0.id}"
    adapter_type                = "vmxnet3"
  }

  network_interface {
    network_id                  = "${data.vsphere_network.VLAN-0.id}"
    adapter_type                = "vmxnet3"
  }

  disk {
    label                       = "disk0"
    size                        = "${data.vsphere_virtual_machine.Core-Router-01.disks.0.size}"
    eagerly_scrub               = "${data.vsphere_virtual_machine.Core-Router-01.disks.0.eagerly_scrub}"
    thin_provisioned            = "${data.vsphere_virtual_machine.Core-Router-01.disks.0.thin_provisioned}"
  }

  extra_config = {
    "guestinfo.userdata" = base64gzip(file("vyos.commands.yaml"))
    "guestinfo.userdata.encoding" = "gzip+base64"
  }
 
  clone {
    template_uuid               = "${data.vsphere_virtual_machine.Core-Router-01.id}"
  }
}

and here is the vyos.commands.yaml file:

write_files:
  - path: /opt/vyatta/etc/config/scripts/vyos-postconfig-bootup.script
    owner: vyos:vyos
    permissions: '0775'
    content: |
      #!/bin/bash
      set interfaces eth0 address '1.1.1.1/24'

Once the Vyos VM boots, I look in/opt/vyatta/etc/config/scripts/vyos-postconfig-bootup.script but the file is untouched.

If anyone could show me the correct way to implement the ‘write_files’ feature of Vyos using Terraform I’m sure I’d be able to completely automate the deployment process.

Kelly

2

Hello kelly ,

I think it could help you, there are some syntax errors . Could you try it ? :

write_files:
  - path: /opt/vyatta/etc/config/scripts/vyos-postconfig-bootup.script
    owner: root:vyattacfg
    permissions: '0775'
    content: |
      #!/bin/vbash
      source /opt/vyatta/etc/functions/script-template
       
      configure
      set interfaces ethernet eth0 address '1.1.1.1/24'
      commit
      exit

it should work

3

Hi Fernando,

Thanks! I’ll give it a try right now.

Kelly

4

Unfortunately your suggestion did not work.

I made the changes, and when I ran Terraform, the VM deployed, but both NICs are still unconfigured at the end of the deployment.

I also inspected /opt/vyatta/etc/config/scripts/vyos-postconfig-bootup.script, and it was also unmodified, leading me to believe that the ‘write_files’ module did not work.

Note that I had to revert back to using the creds vyos:vvyos, because that’s what I used when I created the VM template.

I hope that my logic is correct:
In order to automate the deployment of Vyos, I first started by manually deploying an instance of Vyos. I executed the ‘install image’ command, and then created a template of the VM. I’ve really done nothing else to the VM other than install the OS.

When I run Terraform, I’m referencing the Vyos VM Template I created above. This is the same process I use for Windows and other Linux VMs and so far this is working fine.

When I first started trying to deploy the Vyos VM Template with Terraform, I observed that the standard Terraform commands to configure network interfaces for Linux failed to work with Vyos - and when I removed the offending commands the Vyos VM deploys, but without the NICs being configured… which is why I’m now seeking assistance on how to move forward.

If this logic is flawed I’m open to suggestions.

I’m currently investigating using Hashicorp Packer as an alternative to create the Vyos Template, but I’m not confident that the issue is with the template itself.

5

UPDATE:

I made some pretty good progress.

My requirement is that I need to be able to assign an IP to each NIC on deployment. I don’t really care how this happens, as long as it gets done with as little fuss as possible.

I would like to use Terraform to assign the IPs, but in the meantime I decided to try a different approach and manually edited the /opt/vyatta/etc/config/scripts/vyos-postconfig-bootup.script file directly on the Template (thanks to @Fernando for the correct syntax) and sure enough, when I now deploy the Vyos template via Terraform the VM correctly deploys and it has a static IP that now allows me to use Ansible to continue to configuration process. This alone is going to save me a lot of time and frustration.

The only drawback is that if the network I’m building has five routers, then I need to have five templates - with each template being assigned a unique static IP. I would then use Ansible to access each of the five routers to finish the configuration.

The ‘best-case’ scenario is to get the Terraform ‘write_files’ module to work, which I’ll continue to work on.

Kelly

1 Like
6

great news , I couldn’t find so much information about write_files module on terraform (how it works), as I understand it should match with the cloud-init files. it could be useful for your deployment :

https://docs.vyos.io/en/latest/automation/cloud-init.html

thanks for the update.

7

If I understand correctly you have to use cloud-init-compatible images
I may be wrong

8

Hi Viacheslav,

You are probably correct. I am currently using vyos-1.4-rolling-202210090955-amd64.iso.

I’ve been doing research, and users have been making references to specific logs and folders that do not exist in the version of Vyos I’m working with.

I’ll go to the downloads web page and see if there are ‘cloud-init’ specific versions for download.

I think I mentioned this, but I’m working on-prem and not on AWS, GC, etc. I find that there is a lot of information and help for the big three cloud providers (not just Vyos, for everything), and not quite as much for on-prem.

Kelly

9

Predefined Cloud init images are allowed for users with paid subscription
You can deploy your own image with cloud-init ref GitHub - vyos/vyos-vm-images

10

Thank you very much! You just clarified why I’m not having success with Terraform, Vyos and the ‘write_files’ module!

I made a lot of progress today, and will investigate the option of building my own cloud-init enabled version of Vyos.

Kelly