Container example shown on Vyos May update not working on 1.4 rolling

phillipmcmahon · June 10, 2021, 11:43pm

Tried running the Zabbix container example mentioned on the Vyos May project update - VyOS Project May/June 2021 Update

set container network zabbix-net prefix 172.20.0.0/16
set container network zabbix-net description 'Network for Zabbix component containers'

set container name mysql-server image mysql:8.0
set container name mysql-server network zabbix-net
set container name mysql-server environment 'MYSQL_DATABASE' value 'zabbix'
set container name mysql-server environment 'MYSQL_USER' value 'zabbix'
set container name mysql-server environment 'MYSQL_PASSWORD' value 'zabbix_pwd'
set container name mysql-server environment 'MYSQL_ROOT_PASSWORD' value 'root_pwd'

set container name zabbix-java-gateway image zabbix/zabbix-java-gateway:alpine-5.2-latest
set container name zabbix-java-gateway network zabbix-net

set container name zabbix-server-mysql image zabbix/zabbix-server-mysql:alpine-5.2-latest
set container name zabbix-server-mysql network zabbix-net
set container name zabbix-server-mysql environment 'DB_SERVER_HOST' value 'mysql-server'
set container name zabbix-server-mysql environment 'MYSQL_DATABASE' value 'zabbix'
set container name zabbix-server-mysql environment 'MYSQL_USER' value 'zabbix'
set container name zabbix-server-mysql environment 'MYSQL_PASSWORD' value 'zabbix_pwd'
set container name zabbix-server-mysql environment 'MYSQL_ROOT_PASSWORD' value 'root_pwd'
set container name zabbix-server-mysql environment 'ZBX_JAVAGATEWAY' value 'zabbix-java-gateway'
set container name zabbix-server-mysql port zabbix source 10051
set container name zabbix-server-mysql port zabbix destination 10051

set container name zabbix-web-nginx-mysql image zabbix/zabbix-web-nginx-mysql:alpine-5.2-latest
set container name zabbix-web-nginx-mysql network zabbix-net
set container name zabbix-web-nginx-mysql environment 'MYSQL_DATABASE' value 'zabbix'
set container name zabbix-web-nginx-mysql environment 'ZBX_SERVER_HOST' value 'zabbix-server-mysql'
set container name zabbix-web-nginx-mysql environment 'DB_SERVER_HOST' value 'mysql-server'
set container name zabbix-web-nginx-mysql environment 'MYSQL_USER' value 'zabbix'
set container name zabbix-web-nginx-mysql environment 'MYSQL_PASSWORD' value 'zabbix_pwd'
set container name zabbix-web-nginx-mysql environment 'MYSQL_ROOT_PASSWORD' value 'root_pwd'
set container name zabbix-web-nginx-mysql port http source 80
set container name zabbix-web-nginx-mysql port http destination 8080

However on commit I get the following error(s)

[ container ]
Resolving “mysql” using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull Docker…
Getting image source signatures
Copying blob sha256:37ba2d8bd4fe60e7cd8ba1fe43fec4fde06acd0f4407a3db4b4dea1450f19f99
Copying blob sha256:1651b0be3df3b7c2b57c16e7932836c239fe5f5f1625e390147b5ea0dac3b3e0
Copying blob sha256:69692152171afee1fd341febc390747cfca2ff302f2881d8b394e786af605696
Copying blob sha256:6d278bb05e94d4d1aa8bf5b6db09d8b5d8421b61026ba29c78405521e3aa49eb
Copying blob sha256:0f86c95aa2427b656dbc0bf01413bb8af354ea168e660887eb93ee816ad360a1
Copying blob sha256:951da7386bc8b010c95056608b66b1e7431ef0e895a1717528d3a16cda93f341
Copying blob sha256:f7fddf10c2c2d7643ac60e6e748d418847c6c5b8d2d772d6d777e5bc4cfed299
Copying blob sha256:497efbd93a3eb09094c94c3a06b0a2fc6b8cc224812b464a8dd4889d5c2af266
Copying blob sha256:16415d159dfb70afbd4d25c43bd44d506c6d3b15d7f7518494331f7001c7d024
Copying blob sha256:0e530ffc6b73401d9bb9ecff4a3ec072184ae79f4d858fa37f6373a7b731f483
Copying blob sha256:b0a4a1a771782ef8e5e7d4ef8ace68fd47ba9b9cacff585f7b6b619b0eb60ea2
Copying blob sha256:cd90f92aa9ef2e393330b60b73d8e18f473a780a7caaa2e97378379d08da920e
Copying config sha256:c0cdc95609f1fc1daf2c7cae05ebd6adcf7b5c614b4f424949554a24012e3c09
Writing manifest to image destination
Storing signatures
c0cdc95609f1fc1daf2c7cae05ebd6adcf7b5c614b4f424949554a24012e3c09
0
VyOS had an issue completing a command.

We are sorry that you encountered a problem while using VyOS.
There are a few things you can do to help us (and yourself):

Make sure you are running the latest version of the code available at
VyOS Community
Consult the forum to see how to handle this issue
https://forum.vyos.io
Join our community on slack where our users exchange help and advice
https://vyos.slack.com

When reporting problems, please include as much information as possible:

do not obfuscate any data (feel free to contact us privately if your
business policy requires it)
and include all the information presented below

Report Time: 2021-06-11 01:37:23
Image Version: VyOS 1.4-rolling-202106100417
Release Train: sagitta

Built by: autobuild@vyos.net
Built on: Thu 10 Jun 2021 06:42 UTC
Build UUID: 14d353f8-0079-4295-81b3-2a156c9c68bc
Build Commit ID: 65b596062f015c

Architecture: x86_64
Boot via: installed image
System type: VMware guest

Hardware vendor: VMware, Inc.
Hardware model: VMware Virtual Platform
Hardware S/N: VMware-56 4d 7f e2 82 d3 d9 ce-88 cb ca de 92 1a 21 84
Hardware UUID: e27f4d56-d382-ced9-88cb-cade921a2184

Traceback (most recent call last):
File “/usr/libexec/vyos/conf_mode/containers.py”, line 267, in
apply(c)
File “/usr/libexec/vyos/conf_mode/containers.py”, line 253, in apply
_cmd(f’podman run --name {name} -dit --net {network} {ipparam} {port} {volume} {env_opt} {image}')
File “/usr/libexec/vyos/conf_mode/containers.py”, line 42, in _cmd
return cmd(command)
File “/usr/lib/python3/dist-packages/vyos/util.py”, line 161, in cmd
raise OSError(code, feedback)
OSError: [Errno 126] failed to run command: podman run --name mysql_server -dit --net zabbix_net -e MYSQL_DATABASE=zabbix -e MYSQL_PASSWORD=zabbix_pwd -e MYSQL_ROOT_PASSWORD=root_pwd -e MYSQL_USER=zabbix mysql:8.0
returned:
exit code: 126

noteworthy:
cmd ‘curl --unix-socket /run/podman/podman.sock ‘http://d/v3.0.0/libpod/containers/mysql_server/exists’’
returned (out):
{“cause”:“no such container”,“message”:“no such container”,“response”:404}
returned (err):
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 75 100 75 0 0 297 0 --:–:-- --:–:-- --:–:-- 297
cmd ‘podman run --name mysql_server -dit --net zabbix_net -e MYSQL_DATABASE=zabbix -e MYSQL_PASSWORD=zabbix_pwd -e MYSQL_ROOT_PASSWORD=root_pwd -e MYSQL_USER=zabbix mysql:8.0’
returned (out):

returned (err):
time=“2021-06-11T01:37:22+02:00” level=error msg=“Error adding network: failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table nat' is incompatible, use 'nft' tool.\n\n" time="2021-06-11T01:37:22+02:00" level=error msg="Error while adding pod to CNI network \"zabbix_net\": failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table nat’ is incompatible, use ‘nft’ tool.\n\n”
Error: error configuring network namespace for container 7368d5bb332c96c1b7131f4d9ccb153a6744471f9be8ba3d88c318133e3248d3: failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table `nat’ is incompatible, use ‘nft’ tool.

[[]] failed
Commit failed
[edit]

Any tips to get this working?

Viacheslav · June 11, 2021, 6:07am

Try to change network name without “-” for example network zabbixnet
It seems it replace zabbix-net to zabbix_net

Viacheslav · June 11, 2021, 12:23pm

I create a task T3614

phillipmcmahon · June 11, 2021, 12:38pm

Cool, thanks a lot.

Might be good to update the news article VyOS Project May/June 2021 Update in the meantime and remove the hypens in the example.

Viacheslav · June 11, 2021, 12:43pm

The names will be fixed soon.

phillipmcmahon · June 11, 2021, 12:46pm

still get the following error. I replaced zabbix-net with zabbixnet.

VyOS had an issue completing a command.

We are sorry that you encountered a problem while using VyOS.
There are a few things you can do to help us (and yourself):

Make sure you are running the latest version of the code available at
VyOS Community
Consult the forum to see how to handle this issue
https://forum.vyos.io
Join our community on slack where our users exchange help and advice
https://vyos.slack.com

When reporting problems, please include as much information as possible:

do not obfuscate any data (feel free to contact us privately if your
business policy requires it)
and include all the information presented below

Report Time: 2021-06-11 14:45:13
Image Version: VyOS 1.4-rolling-202106102016
Release Train: sagitta

Built by: autobuild@vyos.net
Built on: Fri 11 Jun 2021 01:17 UTC
Build UUID: 2394fc33-a732-4bae-bfa2-2df3d9b26fe0
Build Commit ID: 931b023809e5b1

Architecture: x86_64
Boot via: installed image
System type: VMware guest

Hardware vendor: VMware, Inc.
Hardware model: VMware Virtual Platform
Hardware S/N: VMware-56 4d 7f e2 82 d3 d9 ce-88 cb ca de 92 1a 21 84
Hardware UUID: e27f4d56-d382-ced9-88cb-cade921a2184

Traceback (most recent call last):
File “/usr/libexec/vyos/conf_mode/containers.py”, line 267, in
apply(c)
File “/usr/libexec/vyos/conf_mode/containers.py”, line 253, in apply
_cmd(f’podman run --name {name} -dit --net {network} {ipparam} {port} {volume} {env_opt} {image}')
File “/usr/libexec/vyos/conf_mode/containers.py”, line 42, in _cmd
return cmd(command)
File “/usr/lib/python3/dist-packages/vyos/util.py”, line 161, in cmd
raise OSError(code, feedback)
OSError: [Errno 126] failed to run command: podman run --name mysql_server -dit --net zabbixnet -e MYSQL_DATABASE=zabbix -e MYSQL_PASSWORD=zabbix_pwd -e MYSQL_ROOT_PASSWORD=root_pwd -e MYSQL_USER=zabbix mysql:8.0
returned:
exit code: 126

noteworthy:
cmd ‘curl --unix-socket /run/podman/podman.sock ‘http://d/v3.0.0/libpod/containers/mysql_server/exists’’
returned (out):
{“cause”:“no such container”,“message”:“no such container”,“response”:404}
returned (err):
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 75 100 75 0 0 297 0 --:–:-- --:–:-- --:–:-- 297
cmd ‘podman run --name mysql_server -dit --net zabbixnet -e MYSQL_DATABASE=zabbix -e MYSQL_PASSWORD=zabbix_pwd -e MYSQL_ROOT_PASSWORD=root_pwd -e MYSQL_USER=zabbix mysql:8.0’
returned (out):

returned (err):
time=“2021-06-11T14:45:12+02:00” level=error msg=“Error adding network: failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table nat' is incompatible, use 'nft' tool.\n\n" time="2021-06-11T14:45:12+02:00" level=error msg="Error while adding pod to CNI network \"zabbixnet\": failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table nat’ is incompatible, use ‘nft’ tool.\n\n”
Error: error configuring network namespace for container 67d0d4c9867600e3f8d998f94714d6d93012d57c465e59243e57785bca36421b: failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table `nat’ is incompatible, use ‘nft’ tool.

[[]] failed
Commit failed
[edit]

Viacheslav · June 11, 2021, 1:03pm

Maybe have the same bug with a hyphen. Before fix it replaced all tagNodes from “x-x” to “x_x”
By the way, with the FIX it seems to work fine

vyos@r1-roll# commit
[ container ]
Resolving "mysql" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/mysql:8.0...
Getting image source signatures
Copying blob sha256:37ba2d8bd4fe60e7cd8ba1fe43fec4fde06acd0f4407a3db4b4dea1450f19f99
Copying blob sha256:1651b0be3df3b7c2b57c16e7932836c239fe5f5f1625e390147b5ea0dac3b3e0
Copying blob sha256:951da7386bc8b010c95056608b66b1e7431ef0e895a1717528d3a16cda93f341
Copying blob sha256:0f86c95aa2427b656dbc0bf01413bb8af354ea168e660887eb93ee816ad360a1
Copying blob sha256:6d278bb05e94d4d1aa8bf5b6db09d8b5d8421b61026ba29c78405521e3aa49eb
Copying blob sha256:69692152171afee1fd341febc390747cfca2ff302f2881d8b394e786af605696
Copying blob sha256:f7fddf10c2c2d7643ac60e6e748d418847c6c5b8d2d772d6d777e5bc4cfed299
Copying blob sha256:497efbd93a3eb09094c94c3a06b0a2fc6b8cc224812b464a8dd4889d5c2af266
Copying blob sha256:16415d159dfb70afbd4d25c43bd44d506c6d3b15d7f7518494331f7001c7d024
Copying blob sha256:0e530ffc6b73401d9bb9ecff4a3ec072184ae79f4d858fa37f6373a7b731f483
Copying blob sha256:b0a4a1a771782ef8e5e7d4ef8ace68fd47ba9b9cacff585f7b6b619b0eb60ea2
Copying blob sha256:cd90f92aa9ef2e393330b60b73d8e18f473a780a7caaa2e97378379d08da920e
Copying config sha256:c0cdc95609f1fc1daf2c7cae05ebd6adcf7b5c614b4f424949554a24012e3c09
Writing manifest to image destination
Storing signatures
c0cdc95609f1fc1daf2c7cae05ebd6adcf7b5c614b4f424949554a24012e3c09
Resolving "zabbix/zabbix-java-gateway" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/zabbix/zabbix-java-gateway:alpine-5.2-latest...
Getting image source signatures
Copying blob sha256:339de151aab4bc06eed8409daae147c408478cb538dacb90cc63f19ad4eba80b
Copying blob sha256:eb863fed5a2db3e089c92f69d9029188959337537d0dcdc82b67afaff76c386d
Copying blob sha256:acc2a621c1cd8a458fb18102487fcf56f1ffcf56684ed1af1e259d6f34712025
Copying blob sha256:56207528e7e1147d195404c9201a53e0fbd520504b75acac4669ca193c1d44db
Copying blob sha256:405d297f04340062210eee752505d6ed7ad842c4fcaca6d10470be1690e6d326
Copying blob sha256:5b6c27b6f81d50896bc4afa9ed9e4beb0532d0a44f83d599c84cf63b05291b64
Copying config sha256:7926f30300187b577c39334658ef6f83c9b5b4037b6b4cf1d2cd01034fbc0a05
Writing manifest to image destination
Storing signatures
7926f30300187b577c39334658ef6f83c9b5b4037b6b4cf1d2cd01034fbc0a05
Resolving "zabbix/zabbix-server-mysql" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/zabbix/zabbix-server-mysql:alpine-5.2-latest...
Getting image source signatures
Copying blob sha256:339de151aab4bc06eed8409daae147c408478cb538dacb90cc63f19ad4eba80b
Copying blob sha256:3594cf9b32e0d42bcbb8dd59fc5efc86c4b03124e7db4778bf398ff0569fde5a
Copying blob sha256:7fbdd17c527b6461d3ba5e6e044a45b1ebcde7139a3212af54153fec847c3bc3
Copying blob sha256:076501ae414bf5e0061ff65f22083f8bffbf54ace9f470049414aedbdaa120fd
Copying config sha256:be3fbad628e08fa0a6b8746911b48e2f9548298f56a60946246f9152b0a85aeb
Writing manifest to image destination
Storing signatures
be3fbad628e08fa0a6b8746911b48e2f9548298f56a60946246f9152b0a85aeb
0
0
0

[edit]
vyos@r1-roll# run show container 
CONTAINER ID  IMAGE                                                   COMMAND               CREATED             STATUS                     PORTS                     NAMES
22cfc911b0f4  docker.io/library/mysql:8.0                             mysqld                2 minutes ago       Exited (1) 30 seconds ago                            mysql-server
feea7f453e1d  docker.io/zabbix/zabbix-java-gateway:alpine-5.2-latest  /usr/sbin/zabbix_...  About a minute ago  Up About a minute ago                                zabbix-java-gateway
395447518bfd  docker.io/zabbix/zabbix-server-mysql:alpine-5.2-latest  /usr/sbin/zabbix_...  23 seconds ago      Up 22 seconds ago          0.0.0.0:10051->10051/tcp  zabbix-server-mysql
[edit]
vyos@r1-roll#

phillipmcmahon · June 11, 2021, 2:59pm

Still getting the nft error.

Removed all hyphens where applicable.

set container network zabbix_net prefix 172.20.0.0/16
set container network zabbix_net description ‘Network for Zabbix component containers’

set container name mysql_server image mysql:8.0
set container name mysql_server network zabbix_net
set container name mysql_server environment ‘MYSQL_DATABASE’ value ‘zabbix’
set container name mysql_server environment ‘MYSQL_USER’ value ‘zabbix’
set container name mysql_server environment ‘MYSQL_PASSWORD’ value ‘zabbix_pwd’
set container name mysql_server environment ‘MYSQL_ROOT_PASSWORD’ value ‘root_pwd’

set container name zabbix_java_gateway image zabbix/zabbix-java-gateway:alpine-5.2-latest
set container name zabbix_java_gateway network zabbix_net

set container name zabbix_server_mysql image zabbix/zabbix-server-mysql:alpine-5.2-latest
set container name zabbix_server_mysql network zabbix_net
set container name zabbix_server_mysql environment ‘DB_SERVER_HOST’ value ‘mysql_server’
set container name zabbix_server_mysql environment ‘MYSQL_DATABASE’ value ‘zabbix’
set container name zabbix_server_mysql environment ‘MYSQL_USER’ value ‘zabbix’
set container name zabbix_server_mysql environment ‘MYSQL_PASSWORD’ value ‘zabbix_pwd’
set container name zabbix_server_mysql environment ‘MYSQL_ROOT_PASSWORD’ value ‘root_pwd’
set container name zabbix_server_mysql environment ‘ZBX_JAVAGATEWAY’ value ‘zabbix_java_gateway’
set container name zabbix_server_mysql port zabbix source 10051
set container name zabbix_server_mysql port zabbix destination 10051

set container name zabbix_web_nginx_mysql image zabbix/zabbix-web-nginx-mysql:alpine-5.2-latest
set container name zabbix_web_nginx_mysql network zabbix_net
set container name zabbix_web_nginx_mysql environment ‘MYSQL_DATABASE’ value ‘zabbix’
set container name zabbix_web_nginx_mysql environment ‘ZBX_SERVER_HOST’ value ‘zabbix_server_mysql’
set container name zabbix_web_nginx_mysql environment ‘DB_SERVER_HOST’ value ‘mysql_server’
set container name zabbix_web_nginx_mysql environment ‘MYSQL_USER’ value ‘zabbix’
set container name zabbix_web_nginx_mysql environment ‘MYSQL_PASSWORD’ value ‘zabbix_pwd’
set container name zabbix_web_nginx_mysql environment ‘MYSQL_ROOT_PASSWORD’ value ‘root_pwd’
set container name zabbix_web_nginx_mysql port http source 80
set container name zabbix_web_nginx_mysql port http destination 8080

returned (err):
time=“2021-06-11T16:57:39+02:00” level=error msg=“Error adding network: failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table nat' is incompatible, use 'nft' tool.\n\n" time="2021-06-11T16:57:39+02:00" level=error msg="Error while adding pod to CNI network \"zabbix_net\": failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table nat’ is incompatible, use ‘nft’ tool.\n\n”
Error: error configuring network namespace for container 8d9048a254dc197fb961d300a49e7547c64f9fb8b69004e7c0d1dfc11698cefb: failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table `nat’ is incompatible, use ‘nft’ tool.

[[]] failed
Commit failed
[edit]

Viacheslav · June 11, 2021, 3:06pm

Try to create containers one by one. If it possible reboot the node before it.
With which container is this problem?

phillipmcmahon · June 11, 2021, 3:09pm

I can do all of that for sure - definitely in pre-alpha phase

Will play around and report back.

Viacheslav · June 11, 2021, 4:34pm

Bug with container/network names will be fixed in the next rolling release.

phillipmcmahon · June 12, 2021, 9:05am

I played around. Still getting those “nat not good, use nft” errors and concluded it’s too rough around the edges for me to spend time on right now.

Viacheslav · June 12, 2021, 9:17am

https://phabricator.vyos.net/T3499

phillipmcmahon · June 12, 2021, 9:45am

thanks a lot. seems like a showstopper for containers until that is fixed.

zach · July 17, 2021, 2:20am

Hello. Have you considered using macvlan?

With macvlan, the container is given access to a physical network interface on the host. This interface can configure multiple subinterfaces. And each subinterface is capable of having its own MAC and IP address. In the case of Podman containers, the container will present itself as if it is on the same network as the host.

It would seem this should would be seen by VyOS as just another external host on that interface. If so, firewall rules/NAT can be configured as usual.

I have not gotten a chance to test yet, but I like the idea of using container macvlan networking as it would be handled uniformly by VyOS, including DHCP/IPAM.

Example macvlan network creation:

sudo podman network create -d macvlan -o parent=eth0 webnetwork

https://fossies.org/linux/podman/docs/tutorials/basic_networking.md#macvlan

EDIT
Did some limited testing using a VM on Vultr. Seems like the primary challenge for me at the moment is figuring out how best to use a single, public interface (eth0). DHCP is not working as expected with pseudo-ethernet (peth0) used to create a podman macvlan network.

ERRO[0144] Error while adding pod to CNI network "podmannet": error calling DHCP.Allocate: no more tries

Take 2. I think my first mistake was bypassing VyOS config, instead using podman CLI directly. On that note, if we are to use a VyOS-centric configuration, how would we even go about configuring a macvlan podman network? It does not seem possible currently, although it seems like it would make a lot of sense in the context of VyOS to use macvlan podman networks to better align container networking with VyOS as it stands. Any thoughts?

Viacheslav · July 17, 2021, 4:37am

Not sure if it possible
http://docs.podman.io/en/latest/markdown/podman-network-create.1.html#driver-d

From their documentation:

Driver to manage the network (default “bridge”). Currently only bridge is supported.

zach · July 17, 2021, 5:17am

Judging by GitHub issues, it’s supported, so docs are likely out-of-date. [3.0]add macvlan as a supported network driver by baude · Pull Request #9198 · containers/podman · GitHub

This issue in particular shines some light on the nature of podman macvlan network config: Cannot start podman container with static IP address on macvlan network · Issue #10283 · containers/podman · GitHub

My issue at the moment is not having a local interface to bind the podman network to. Is this a good use case for dummy interfaces in VyOS?

It could make container networking configuration in VyOS feel more native, what do you think?

zach · July 17, 2021, 10:58pm

Hi again,

I slept on this and think I can better describe the issue now. What this appears to boil down to is Podman managing either L2 or L3.

Current implementation of Podman support in VyOS assumes we must use Podman’s bridge network driver. By using the bridge network driver, NAT configuration is delegated to Podman which creates an awkward layer of complexity between Podman and VyOS. Since Podman does not support nftables, this does not work anyway.

The alternative is to use Podman’s macvlan network driver bridged to a VyOS-managed interface. By doing so, each container is assigned a MAC address like any other host in an IP network. IPAM can be handled in two ways: either a static IP address (e.g. podman run --ip 192.0.1.2) or the standard DHCP Server service in VyOS.

When each container gets a MAC, we can use VyOS to manage networking in the same, standard way. This is a big win in my opinion.

What I’m not clear on is what kind of interface would make most sense here. From what I can tell, a dummy interface might work. Pseudo-ethernet does not seem to fit as it requires an existing source interface.

I look forward to hearing from others on this topic so we can start using containers in a clear, effective way and make the most of VyOS.

Thank you!

Viacheslav · July 18, 2021, 7:54am

There are 2 options that you can use right now. It is a “bridge” and “host” networks.
For bridge you can set nat rules by VyOS CLI.
The source code for containers you can find here containers.py

Do you know all full commands which should add MACVLAN in podman?

zach · July 18, 2021, 2:01pm

Checked the containers script, and it seems to align with my understanding.

Podman host network breaks container isolation and allows complete access to host networking stack. It is my understanding that host network should only be used in development.

Podman bridge network uses portmap to expose container ports on the host. This is where Podman tries to manage NAT and breaks on VyOS.

Podman macvlan network requires a host interface and exposes the container as a typical host on that interface. This decouples NAT from Podman and allows the container networking to be managed in VyOS in a consistent, more explicit way.

To create a macvlan network:
sudo podman network create -d macvlan -o parent=ethX webnetwork
(What interface do we use here? To me, it would make sense to use some kind of virtual interface dedicated to this container macvlan network. I am not clear on what kind of virtual interface would be most appropriate for this use case.)

If using DHCP, we need to use CNI DHCP daemon which acts as a DHCP proxy for containers (most containers do not, and should not, include dhclient). To enable CNI DHCP daemon for containers:
/usr/lib/cni/dhcp daemon &

One caveat I’ve seen reported is the CNI DHCP daemon uses inconsistent MAC in DHCP requests, so static mappings may not work as expected. Simple workaround (maybe even best practice) is assigning a static IP to the container directly, e.g. podman run --ip 192.0.1.2 my-image