VyOS as docker container journey

Hello,
I’m new to this forum but I would like to share my experience with VyOS (1.3) running as docker container for lab purpose, as I not find any relevant topic here. I’ve done everything on docker in Debian WSL 2 mode on Windows 10 Enterprise 1909, but I’ve seen same behavior on bare metal Ubuntu 18 LTS.

I’ve begun with building a container from “official” instructions found on Vyos-build’s Github. So I ended with this build command:

docker build --no-cache --pull --compress -f Dockerfile -t vyos:1.3-`date -u +%Y%m%d%H%M%S` --build-arg BUILD_DATE="`date -u --rfc-3339=seconds`" --build-arg VYOS_VERSION=1.3 --build-arg DEBIAN_VERSION=jessie --progress plain .. 

Everything run smoothly, and it ended with a fresh docker image of VyOS 1.3 :slight_smile:

Next I succesfully start a container r1 with this image:

docker run -v /lib/modules:/lib/modules --privileged --name r1 -d vyos:1.3-20201030211657

Then I connect to it via:

docker exec -it r1 su vyos

…and get a beautiful VyOS prompt :love_you_gesture:t2::

vyos@vyos:/tmp$

Then things went bad with first try to assign address on interface:

vyos@vyos# run show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             172.17.0.3/16                     u/u
lo               127.0.0.1/8                       u/u
vyos@vyos#
vyos@vyos#
vyos@vyos# set interfaces ethernet eth0 address 10.1.1.1/24
[edit]
vyos@vyos# commit
[ interfaces ethernet eth0 ]
VyOS had an issue completing a command.

We are sorry that you encountered a problem while using VyOS.
There are a few things you can do to help us (and yourself):
- Make sure you are running the latest stable version of VyOS
  the code is available at https://downloads.vyos.io/?dir=release/current
- Contact us using the online help desk
  https://support.vyos.io/
- Join our community on slack where our users exchange help and advice
  https://vyos.slack.com

When reporting problems, please include as much information as possible:
- do not obfuscate any data (feel free to contact us privately if your
  business policy requires it)
- and include all the information presented below

Report Time:      2020-10-30 21:29:14
Image Version:    VyOS 1.3dev0-2371-g700d03d2
Release Train:    equuleus

Built by:         Sentrium S.L.
Built on:         2020-10-30 21:22:20 UTC
Build UUID:       eef4f706-7c74-4a4c-a15e-bc2e3ae7d035
Build Commit ID:  unknown

Architecture:     x86_64
Boot via:         installed image
System type:      Microsoft Hyper-V guest

Hardware vendor:  Unknown
Hardware model:   Unknown
Hardware S/N:     Unknown
Hardware UUID:    Unknown

Traceback (most recent call last):
  File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 94, in <module>
    apply(c)
  File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 86, in apply
    e.update(ethernet)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/ethernet.py", line 278, in update
    super().update(config)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1077, in update
    self.add_ipv6_eui64_address('fe80::/64')
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 522, in add_ipv6_eui64_address
    self.add_addr(f'{eui64}/{prefixlen}')
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 782, in add_addr
    self._cmd(f'ip addr add "{addr}" '
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 51, in _cmd
    return cmd(command, self.debug)
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 179, in cmd
    raise OSError(code, feedback)
FileNotFoundError: [Errno 2] failed to run command: ip addr add "fe80::42:acff:fe11:3/64" dev "eth0"
returned:
exit code: 2

noteworthy:
cmd 'ip addr add "fe80::42:acff:fe11:3/64" dev "eth0"'
returned (out):

returned (err):
RTNETLINK answers: Permission denied

[[interfaces ethernet eth0]] failed
Commit failed

:open_mouth: :open_mouth: :cry:

I get the point that it try to assign IPv6 address in the process of assigning an IPv4 address… Weird :expressionless:

So I try to disable IPv6 on VyOS with:

vyos@vyos# set system ipv6 disable

…and got the same issue after restarting the container :confused: (in the mean time I’d put instruction for docker run to bind a persistent volume for /config).

So I came back to initial github’s instructions and find a note about initializing docker in IPv6 mode. I followed instructions, apply and restart docker but this time docker don’t let me start the container:

docker: Error response from daemon: Address already in use.

:sob:

I googled it for a while and don’t find any useful tips to correct this issue… It seems dockerd never deallocate an IPv6 (from space defined in daemon.json) an then cannot start new containers on default docker network bridge.

So I try another approach: creating a dedicated network, IPv6 enabled with different subnet than the one defined in daemon.json, and run VyOS container on it:

docker network create --internal --ipv6 --attachable --subnet fe81::/64 net6

docker run -v /lib/modules:/lib/modules --privileged --name r1 --network net6 -d vyos:1.3-20201030211657

Container successfully start :smiley: , then I can plug in and try to assign IPs:

C:\>docker exec -it r1 su vyos
vyos@vyos:/tmp$ sh interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             172.20.0.2/16                     u/u
                 fe81::2/64
lo               127.0.0.1/8                       u/u
                 ::1/128

vyos@vyos# set interfaces ethernet eth0 address 1.1.1.1/24
[edit]
vyos@vyos# commit
[edit]
vyos@vyos# run show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             1.1.1.1/24                        u/u
lo               127.0.0.1/8                       u/u
                 ::1/128
[edit]
vyos@vyos#

ET VOILA ! :tada: :confetti_ball:

I hope you enjoyed it and also that this topic may help ones that may fall into this issue :slight_smile: !
Cheers :beers: !

PS: I’ve found a Vyos bug that may be the one I encounter here: T2516, may @Dmitry be interrested :wink:

OK guys, I was wrong, partially due to my lake of knowledge of how networking work in Docker.

First of all, how not to Docker for Vyos:

  • you don’t need to use IPv6 networks (docker network create --ipv6), it just workaround part of the main problem, cause you can then re-address existing interfaces, but adding interface (ie: dummys) just not work because Docker also try to bind IPv6 on those interface and you fall in the issue explained in my previous post:
FileNotFoundError: [Errno 2] failed to run command: ip addr add "fe80::42:acff:fe11:3/64" dev "eth0"
returned:
  • you must not use the bridge driver, this driver is made only for network abstraction between containers and is not intended to be use when a container need kind of “real” interfaces (L2 networking functionalities). For example, one big issue is that bridge driver do not support multicast…
  • don’t try to address ethernet interfaces from inside the container, it just don’t work, let this part to Docker driver/ipam.

It took me looong time, many fails and many unsuccessful googling to understand this :expressionless:

So the good news now : you definitively can use Vyos as container in Docker (for lab purpose) :smiley:

Now how to Docker for Vyos (the right way):

  • use macvlan driver, but keep in mind that it will bind your docker network to a “physical” interface of the Docker host, so you may disrupt the local network behind this interface.
  • force IPv6 inside the container, with --sysctl net.ipv6.conf.all.disable_ipv6=0, no need to enable it on Docker side (via daemon.json parameters).
  • let Docker manage IPs : define subnets at networks creation time, then addresses at containers creation time. Except for dummy interfaces.

Finally docker commands used to run my Vyos containers:

  1. network creation:
docker network create -d macvlan -o parent=eth0 --subnet 172.30.0.0/24 pub
  1. container creation:
docker run --rm --network pub --sysctl net.ipv6.conf.all.disable_ipv6=0 -v /lib/modules:/lib/modules --privileged --name vyos_test -d vyos:1.3-20201030211657

--rm is just here for testing purpose, as it automatically remove the container when in stop state

  1. enjoy your container:
vyos@vyos:/tmp$ conf
[edit]
vyos@vyos# set interfaces dummy dum0 address 1.1.1.1/32
[edit]
vyos@vyos# commit
[edit]
vyos@vyos# set interfaces dummy dum0 address fe88::1/64
[edit]
vyos@vyos# commit
[edit]
vyos@vyos# run sh int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
dum0             1.1.1.1/32                        u/u
                 fe88::1/64
eth0             172.30.0.2/24                     u/u
lo               127.0.0.1/8                       u/u
                 ::1/128
[edit]
vyos@vyos#

multicast support testing (via OSPF) with 3 Vyos containers in my lab:

vyos@R0:/tmp$ sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route

K>* 0.0.0.0/0 [0/0] via 10.10.10.1, eth0, 00:01:26
C>* 1.0.0.0/32 is directly connected, lo, 00:01:24
O>* 1.0.0.1/32 [110/20] via 10.10.10.10, eth0, 00:00:01
O>* 1.0.0.2/32 [110/20] via 10.10.10.20, eth0, 00:00:01
O   10.10.10.0/24 [110/10] is directly connected, eth0, 00:00:02
C>* 10.10.10.0/24 is directly connected, eth0, 00:01:26

As a side note, I forgot to mention that I’m running Docker version 19.03.13, but it may be important because some things have moved particularly on IPv6 support between 17.x and 18.x.

Hope it may help :slight_smile:

Hello @netjourn8, I prefer to use MACVlan interfaces for containerized VyOS, because does not possible to disable IPAM on docker. I hope some issues should disappear on enable ipv6 in docker daemon globally.

Hello @Dmitry, totally agree with you on macvlan driver, I just cannot manage to find a note about it, and it’s why I’ve tried to put this knowledge here :wink:

For the IPAM I’ve tried many scenarii but all ended as epic fail :confused:. It’s why I let Docker manage IPs on “connected” networks between VyOS containers.

Finally about IPv6 settings: actually VyOS run smoothly on my host without it enabled globally on Docker’s daemon side. I’ll keep this tweak in mind and let you know if I needed it in my oncoming tests.

Some additional information:
If you need run multiple VyOS daemon on non systemd based distributive, required create directory and mount cgroup

mkdir /sys/fs/cgroup/system
mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/system/

Preparing networking for VyOS (veth)

  1. Create veth for R1
ip link add vyos-r1-tmp type veth peer name vyos-r1-eth0
  1. Run VyOS without network
docker run -d --name R1 --net=none --privileged -v /lib/modules:/lib/modules vyos/vyos:1,3-rolling /sbin/init

3.Attach network to VyOS router

ip link set netns `docker inspect --format '{{.State.Pid}}' R1` dev vyos-r1-tmp name eth0 up
ip link set vyos-r1-eth0 up
  1. Create a bridge and attach to WAN interface on host machine
brctl addbr br0
ip link set br0 up
brctl addif br0 eth0
brctl addif br0 vyos-r1-eth0

I believe for macvlan we can do a similar way

If you need to run VyOS with a predefined configuration file, that’s also possible.

  1. Create the directory and add the file with minimal configuration
mkdir config
nano config/config.boot
  1. Add configuration. As an example:
system {
    host-name "vyos"
    login {
        user vyos {
            authentication {
                encrypted-password "$6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/"
                plaintext-password ""
            }
        }
    }
  }
  1. Run container
docker run -it --name R1 --net=none --privileged -v /lib/modules:/lib/modules -v $(pwd)/config:/opt/vyatta/etc/config vyos/vyos:1.3-rolling /sbin/init

Where is $(pwd)/config - path to directory with config.boot and /opt/vyatta/etc/config - path in container

Also possible to use:
config/auth directory for cryptographic materials, usually used for openvpn, ipsec, sstp, etc.
config/scripts - directory can store some scripts. As an example this directory can contain script with name vyos-postconfig-bootup.script

Note: Container should build with symlink
ln -s unsquashfs/opt/vyatta/etc/config unsquashfs/config

Thank you @Dmitry !

I’m already mounting specific /config volume for each of my containers :wink:

About networking, as I said previously I’m running my lab on a Windows Host, so I’m not sure I can use cgroups nor brctl from the WSL environment…

I’ll give it a try, then let you know.