HowTo build and operate an agnostic router inside a VMware datacenter environment


I want to give back some work that I have done with vyos in the hope
someone else can be enlighted as I was during the discovery of this
exciting universe.

First of all what is my goal: I want to start a tiny router as a sandboxed
virtualized environment using only the own compiled ISO image, the footprint
of approx. 220MB is adequate. Boot time is a couple of seconds, configuration
time is quicker than boot time. It means using the live CD give me enough degree
of flexibility to make everything what i want to do (NATting & Firewalling)

A reader could ask, why do you not install the router image on a partition
and use the much simpler read-write environment?

It is a matter of security, the system lives in memory. It has a defined
set of functionality no more else.

The prerequisite to use such a system are the following:

  • Knowledge how to build the iso image WITH open-vm-tools (See my other howto)
  • Knowledge of VMware and how to build an OVF image with OVF environment vars
  • Knowledge of XML and XSLT handling how to extract key value pairs from the
    XML document
  • Knowledge of linux boot/init procedures and how to hook into it
  • Knowledge of bash/vbash how to execute and configure the router from a script
  • Knowledge of SQL if you are using sqlite3 as a file oriented RDBMS

It is a lot and frankly speeking you need a little more of other tips and tricks.
I cannot explain everything in it’s entirety, it’s only a recipe just to give you and idea how I have solved this.

  1. Include sqlite3 from debian squeezy if you’re going to use sqlite3 for
    keystore management, it is not required, you can use the normal configuration
    file. It is up to you.
  2. Open (/build-iso/livecd/templates/community/config.boot.default) and
    preconfigure the router with some static variables like time server, DNS
    server, users & password, change the hostname, set ssh service & port for
    dynamic reconfiguration after instanciation
  3. Go to (/build-iso/livecd/config.vyatta/chroot_local-includes) and create
    the etc directory
  4. Copy the /etc/rc.local file into the newly created etc directory
  5. Open the rc.local file and Insert after
# By default this script does nothing.

# set keyboad to german
sudo loadkeys /usr/share/keymaps/i386/qwertz/de.kmap.gz

# initialize router
  1. Build the IS0 WITH open-vm-tools (we need a working vmtoolsd environment)
  2. Define an empty VMware image with 512 vRAM
  3. Mount the ISO and dont forget to enable connection at startup
  4. Go into the option tab and check the next BOOT into BIOS to change the
    boot order after the next reboot (CD-ROM 1st, HDD after)
  5. Define the network in which this router will work (e.g. “VM_Network”)
  6. Select the VMware image and click on “Edit settings”
  7. Now it’s time to go into the VMware vSphere Console and insert some
    OVF key value pairs
  8. Go to the “Options” tab and enable the “vAPP options”
  9. Now go to the “OVF settings” and check “Vmware Tools” under “OVF Environment Transport” (important!)
  10. Go to Advanced tab and click on “Properties” on the right
  11. Create at least three keys value pairs: “gateway, router ipaddress, network mask”
Key     Label Type Value
EGW0  EGW0  ip
EIP0    EIP0    ip
ENM0   ENM0  ip
  1. Now it’s time to build the glue code inside /build-iso/livecd/config.vyatta/chroot_local-includes/usr/local/bin/usr/local/bin/initRouter.
    Sorry I can only outline the steps. It’s to much code to publish here.
  • you need some tools from /usr/bin directory like vmtoolsd, ipcalc, xsltproc
  • with /usr/bin/vmtoolsd --cmd ‘info-get guestinfo.ovfEnv’ > /config/config.xml
    you generate the XML file at boot time like
<?xml version="1.0" encoding="UTF-8"?>
      <Kind>VMware ESXi</Kind>
      <Vendor>VMware, Inc.</Vendor>
         <Property oe:key="EGW0" oe:value=""/>
         <Property oe:key="EIP0" oe:value=""/>
         <Property oe:key="ENM0" oe:value=""/>
      <ve:Adapter ve:mac="00:50:56:93:37:44" ve:network="VM_Network" ve:unitNumber="7"/>
  • with an inline code you can extract later the key value pairs with xsltproc.
    Store this code under /config/config.xsl:
	cat > ${_OVF_XSL_FILE} << 'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
<xsl:output method="text" indent="no"/>
<xsl:template match="/">
<xsl:for-each select="oe:Environment/oe:PropertySection/oe:Property">
<xsl:value-of select="@oe:key"/>
<xsl:value-of select="@oe:value"/>
  1. Execute /usr/bin/xsltproc /config/config.xsl /config/config.xml > /config/config.cfg
    and the key value pairs looks like:
  1. Now you are able to use some bash code to get the key and the appropriate value or put it into an sqlite3 file database
  2. create a database config.db in /config with:
sqlite3 /config/config.db "create table OVF_KEY_TABLE (KEY TEXT PRIMARY KEY, VALUE TEXT);"
  1. import /config/config.cfg into /config/config.db with:
sqlite3 /config/config.db ".import /config/config.cfg OVF_KEY_TABLE";
  1. Now you’re able to go with SQL or plain-file. SQL can be useful if combined with a RESTful/JSON and
    sqlite3 PHP module inside lighttpd for lifecycle management. This is another story, may be on another HowTo
  2. It is time now to inject this into the router /usr/local/bin/initRouter script using the vyatta command wrapper opt/vyatta/sbin/vyatta-cfg-cmd-wrapper
  3. here roughly a template, feel free to add every thing you like and give
    me feedback if you find any trick or a better solution for this.
    NOTE: I use /usr/bin/ipcalc to generate the prefix from the network mask inside my function getOVF_CFGvalPF
opt/vyatta/sbin/vyatta-cfg-cmd-wrapper begin
opt/vyatta/sbin/vyatta-cfg-cmd-wrapper set interfaces ethernet eth0 address $(getOVF_CFGvalue ${OVF_CFG_FILE} EIP0)/$(getOVF_CFGvalPF ${OVF_CFG_FILE} EPF0)
opt/vyatta/sbin/vyatta-cfg-cmd-wrapper set system gateway-address $(getOVF_CFGvalue ${OVF_CFG_FILE} EGW0)

opt/vyatta/sbin/vyatta-cfg-cmd-wrapper commit
opt/vyatta/sbin/vyatta-cfg-cmd-wrapper save
opt/vyatta/sbin/vyatta-cfg-cmd-wrapper end
  1. Add more code for stability and security like CHECK/BUILD sum
  2. Add more code for router live cycle management (Search, Create, Read, Update, Delete)
  3. Put the initRouter script under /build-iso/livecd/config.vyatta/chroot_local-includes/usr/local/bin
  4. Rebuild ISO
  5. Test it

I know it is only a recipe, of course we can exchange any suggestion or
thoughts you might have on this topic.

NOTE: At this point you can ask why is there no persistence layer? There is a bug deep inside the debian live environment,
I am currently trying to find the bug. It is hard, I want to format a tiny ext2 /dev/sda1 partition (10MB)
as /config and unionfs/aufs mount it rw to the ISO.
I tried to boot isolinux with persistent flag set,
currently this is not working as expected.
Stay tuned I hope to fix this or you can help me on this :slight_smile: