Bind permissions for podman container

General questions
1

Hello!

After an update from version 1.4-rolling-202106271939, my podman container for CoreDNS doesn’t start up.

Logs show that there are bind mount errors:

...
[WARNING] An external plugin (/go/pkg/mod/github.com/ori-edge/k8s_gateway@v0.1.6/setup.go line 56) is using the deprecated function Normalize. This will be removed in a future versions of CoreDNS. The plugin should be updated to use OriginsFromArgsOrServerBlock or NormalizeExact instead.
[INFO] plugin/k8s_gateway: Starting k8s_gateway controller
Listen: listen tcp 10.10.0.1:53: bind: permission denied
[WARNING] An external plugin (/go/pkg/mod/github.com/ori-edge/k8s_gateway@v0.1.6/setup.go line 56) is using the deprecated function Normalize. This will be removed in a future versions of CoreDNS. The plugin should be updated to use OriginsFromArgsOrServerBlock or NormalizeExact instead.
[INFO] plugin/k8s_gateway: Starting k8s_gateway controller
Listen: listen tcp 10.11.0.1:53: bind: permission denied
[WARNING] An external plugin (/go/pkg/mod/github.com/ori-edge/k8s_gateway@v0.1.6/setup.go line 56) is using the deprecated function Normalize. This will be removed in a future versions of CoreDNS. The plugin should be updated to use OriginsFromArgsOrServerBlock or NormalizeExact instead.
[INFO] plugin/k8s_gateway: Starting k8s_gateway controller
Listen: listen tcp 10.10.0.1:53: bind: permission denied
[WARNING] An external plugin (/go/pkg/mod/github.com/ori-edge/k8s_gateway@v0.1.6/setup.go line 56) is using the deprecated function Normalize. This will be removed in a future versions of CoreDNS. The plugin should be updated to use OriginsFromArgsOrServerBlock or NormalizeExact instead.
[INFO] plugin/k8s_gateway: Starting k8s_gateway controller
Listen: listen tcp 10.20.0.1:53: bind: permission denied
[WARNING] An external plugin (/go/pkg/mod/github.com/ori-edge/k8s_gateway@v0.1.6/setup.go line 56) is using the deprecated function Normalize. This will be removed in a future versions of CoreDNS. The plugin should be updated to use OriginsFromArgsOrServerBlock or NormalizeExact instead.
[INFO] plugin/k8s_gateway: Starting k8s_gateway controller
Listen: listen tcp 10.40.0.1:53: bind: permission denied
...

CoreFile here:

vyos@fw# cat /config/coredns/Corefile
.:53 {
  bind 127.0.0.1 10.9.18.1 10.10.0.1 10.20.0.1 10.40.0.1 10.30.0.1 10.50.0.1 10.5.0.1 10.11.0.1
  errors
  log
  prometheus :9153

  hosts /etc/hosts {
    reload "1m"
    fallthrough
  }

  k8s_gateway holthome.net {
    ttl 30
    kubeconfig /config/kubeconfig
    fallthrough
  }

  forward . tls://1.1.1.1 tls://1.0.0.1 {
    tls_servername cloudflare-dns.com
  }
}

And vyos config for container here:

vyos@fw# show container
 name coredns {
     allow-host-networks
     image ghcr.io/k8s-at-home/coredns:v1.8.4
     volume config_folder {
         destination /config
         source /config/coredns
     }
     volume hosts {
         destination /etc/hosts
         source /etc/hosts
     }
 }

Here’s the podman container info:

vyos@fw# sudo podman inspect coredns
[
    {
        "Id": "f222199478d18dc3589279e317f5794295409ba3483c7a2319321a7d11be330f",
        "Created": "2021-09-15T10:39:09.297543349-04:00",
        "Path": "/coredns",
        "Args": [
            "-conf",
            "/config/Corefile"
        ],
        "State": {
            "OciVersion": "1.0.2-dev",
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 69625,
            "ConmonPid": 69616,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-09-15T10:48:06.686754668-04:00",
            "FinishedAt": "2021-09-15T10:48:06.402237508-04:00",
            "Healthcheck": {
                "Status": "",
                "FailingStreak": 0,
                "Log": null
            }
        },
        "Image": "a03250528a6331b147bb5ed5bb5620234769f197d08a9fb0203e503b49447e12",
        "ImageName": "ghcr.io/k8s-at-home/coredns:v1.8.4",
        "Rootfs": "",
        "Pod": "",
        "ResolvConfPath": "/run/containers/storage/vfs-containers/f222199478d18dc3589279e317f5794295409ba3483c7a2319321a7d11be330f/userdata/resolv.conf",
        "HostnamePath": "/run/containers/storage/vfs-containers/f222199478d18dc3589279e317f5794295409ba3483c7a2319321a7d11be330f/userdata/hostname",
        "HostsPath": "/run/containers/storage/vfs-containers/f222199478d18dc3589279e317f5794295409ba3483c7a2319321a7d11be330f/userdata/hosts",
        "StaticDir": "/config/containers/storage/vfs-containers/f222199478d18dc3589279e317f5794295409ba3483c7a2319321a7d11be330f/userdata",
        "OCIConfigPath": "/config/containers/storage/vfs-containers/f222199478d18dc3589279e317f5794295409ba3483c7a2319321a7d11be330f/userdata/config.json",
        "OCIRuntime": "crun",
        "ConmonPidFile": "/run/containers/storage/vfs-containers/f222199478d18dc3589279e317f5794295409ba3483c7a2319321a7d11be330f/userdata/conmon.pid",
        "Name": "coredns",
        "RestartCount": 1593,
        "Driver": "vfs",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "EffectiveCaps": null,
        "BoundingCaps": [
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_NET_BIND_SERVICE",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYS_CHROOT"
        ],
        "ExecIDs": [],
        "GraphDriver": {
            "Name": "vfs",
            "Data": null
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/config/coredns",
                "Destination": "/config",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/hosts",
                "Destination": "/etc/hosts",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Dependencies": [],
        "NetworkSettings": {
            "EndpointID": "",
            "Gateway": "",
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "",
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "",
            "Networks": {
                "podman": {
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "NetworkID": "podman",
                    "DriverOpts": null,
                    "IPAMConfig": null,
                    "Links": null
                }
            }
        },
        "ExitCommand": [
            "/usr/bin/podman",
            "--root",
            "/config/containers/storage",
            "--runroot",
            "/run/containers/storage",
            "--log-level",
            "warning",
            "--cgroup-manager",
            "systemd",
            "--tmpdir",
            "/run/libpod",
            "--runtime",
            "crun",
            "--storage-driver",
            "vfs",
            "--events-backend",
            "journald",
            "container",
            "cleanup",
            "f222199478d18dc3589279e317f5794295409ba3483c7a2319321a7d11be330f"
        ],
        "Namespace": "",
        "IsInfra": false,
        "Config": {
            "Hostname": "fw",
            "Domainname": "",
            "User": "nonroot:nonroot",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "container=podman",
                "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt",
                "HOME=/home/nonroot",
                "HOSTNAME=fw"
            ],
            "Cmd": [
                "/coredns",
                "-conf",
                "/config/Corefile"
            ],
            "Image": "ghcr.io/k8s-at-home/coredns:v1.8.4",
            "Volumes": null,
            "WorkingDir": "/home/nonroot",
            "Entrypoint": "",
            "OnBuild": null,
            "Labels": {
                "coredns-external-plugins": "alternate, mdns, wgsd, coredns-git, k8s_gateway",
                "maintainer": "Devin Buhl \u003cdevin.kray@gmail.com\u003e, Bernd Schorgers \u003cme@bjw-s.dev\u003e",
                "org.opencontainers.image.authors": "Anthony Rabbito \u003chello@anthonyrabbito.com\u003e",
                "org.opencontainers.image.source": "https://github.com/k8s-at-home/container-images",
                "org.opencontainers.image.title": "CoreDNS",
                "org.opencontainers.image.version": "1.8.4"
            },
            "Annotations": {
                "io.container.manager": "libpod",
                "io.kubernetes.cri-o.Created": "2021-09-15T10:39:09.297543349-04:00",
                "io.kubernetes.cri-o.TTY": "true",
                "io.podman.annotations.autoremove": "FALSE",
                "io.podman.annotations.init": "FALSE",
                "io.podman.annotations.privileged": "FALSE",
                "io.podman.annotations.publish-all": "FALSE",
                "org.opencontainers.image.stopSignal": "15"
            },
            "StopSignal": 15,
            "CreateCommand": [
                "podman",
                "run",
                "--detach",
                "--interactive",
                "--tty",
                "--replace",
                "--memory",
                "512m",
                "--memory-swap",
                "0",
                "--restart",
                "on-failure",
                "--name",
                "coredns",
                "-v",
                "/config/coredns:/config",
                "-v",
                "/etc/hosts:/etc/hosts",
                "--net",
                "host",
                "ghcr.io/k8s-at-home/coredns:v1.8.4"
            ],
            "Umask": "0022"
        },
        "HostConfig": {
            "Binds": [
                "/config/coredns:/config:rw,rprivate,rbind",
                "/etc/hosts:/etc/hosts:rw,rprivate,rbind"
            ],
            "CgroupManager": "systemd",
            "CgroupMode": "private",
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "k8s-file",
                "Config": null,
                "Path": "/config/containers/storage/vfs-containers/f222199478d18dc3589279e317f5794295409ba3483c7a2319321a7d11be330f/userdata/ctr.log",
                "Tag": "",
                "Size": "0B"
            },
            "NetworkMode": "host",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "on-failure",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [],
            "CapDrop": [
                "CAP_AUDIT_WRITE",
                "CAP_NET_RAW",
                "CAP_MKNOD"
            ],
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": [],
            "IpcMode": "private",
            "Cgroup": "",
            "Cgroups": "default",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "private",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [],
            "Tmpfs": {},
            "UTSMode": "private",
            "UsernsMode": "",
            "ShmSize": 65536000,
            "Runtime": "oci",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 536870912,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 2048,
            "Ulimits": [
                {
                    "Name": "RLIMIT_NOFILE",
                    "Soft": 1048576,
                    "Hard": 1048576
                },
                {
                    "Name": "RLIMIT_NPROC",
                    "Soft": 4194304,
                    "Hard": 4194304
                }
            ],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "CgroupConf": null
        }
    }
]

Did something change in the underlying permissions for Podman that would cause this error?

2

Try reboot the router after update 2 times.
After the first reboot in some cases, containers don’t start.

3

Thanks!

I deleted the container config altogether, commit / save, rebooted two times. Re-deployed the container config, rebooted again, error still persists.

Also, should note, running podman via a systemd config file on this vyos box instead of the vyos container config block works fine.

A la:

[Unit]
Description=CoreDNS container
Wants=network.target
After=network-online.target

[Service]
Restart=on-failure
TimeoutStopSec=60
ExecStartPre=/usr/bin/podman rm --force --volumes --ignore coredns
ExecStart=/usr/bin/podman run -d --rm --net host --name coredns -v /config/coredns/:/config/ -v /etc/hosts:/etc/hosts:ro {{ coredns['container']['repository'] }}:{{ coredns['container']['tag'] }}
ExecStop=/usr/bin/podman stop -t 10 coredns
ExecStopPost=/usr/bin/podman stop -t 10 coredns
Type=forking

[Install]
WantedBy=multi-user.target default.target
4

@Viacheslav – did something change recently in the default caps that are added to podman containers?

I’m able to reproduce my error when running podman directly with this command:

sudo /usr/bin/podman run --net host --name coredns -v /config/coredns/:/config/ -v /etc/hosts:/etc/hosts:ro ghcr.io/k8s-at-home/coredns:v1.8.4

However, I’m able to get it to work by adding the --cap-add command to the run command:

sudo /usr/bin/podman run --net host --name coredns --cap-add CAP_NET_BIND_SERVICE -v /config/coredns/:/config/ -v /etc/hosts:/etc/hosts:ro ghcr.io/k8s-at-home/coredns:v1.8.4

Did the configurator recently switch to a rootless config by default?

5

@carpenike Thanks for writing this up! I ran into the same issue and was lucky to find that @anthr76 submitted a PR for this in containers: T4006: Add capabilities net-bind-service by anthr76 · Pull Request #1073 · vyos/vyos-1x · GitHub.
The change was released on 2021-11-25 1.4 Sagitta — VyOS 1.4.x (sagitta) documentation

The following fixed the issue for me:

set container name coredns cap-add 'net-bind-service'