Build for arm64 failed

Hi,

I am trying to build ver1.4 for arm64 to run VyOS on a Raspberry Pi. I am using GitHub - runborg/vyos-pi-builder for convenience.

I have run the following command:

sudo docker run -it --privileged -v "$(pwd)":/vyos -v /dev:/dev -w /vyos --sysctl net.ipv6.conf.lo.disable_ipv6=0 vyos/vyos-build:current-arm64 sudo bash -x build-image.sh

The result is as follows:

Current UID/GID: 1000/1000
+ set -x
+ set -e
++ pwd
+ ROOTDIR=/vyos
+ rm -rf vyos-build
+ git clone http://github.com/vyos/vyos-build vyos-build
Cloning into 'vyos-build'...
warning: redirecting to https://github.com/vyos/vyos-build/
remote: Enumerating objects: 11133, done.
remote: Counting objects: 100% (589/589), done.
remote: Compressing objects: 100% (280/280), done.
remote: Total 11133 (delta 309), reused 471 (delta 275), pack-reused 10544
Receiving objects: 100% (11133/11133), 7.40 MiB | 7.60 MiB/s, done.
Resolving deltas: 100% (6600/6600), done.
++ find build -type f -name '*.deb'
++ grep -v -e -dbgsym_ -e libnetfilter-conntrack3-dbg
find: ‘build’: No such file or directory
+ cd vyos-build
+ echo 'Copy new default configuration to the vyos image'
Copy new default configuration to the vyos image
+ cp /vyos/config.boot.default data/live-build-config/includes.chroot/opt/vyatta/etc/config.boot.default
+ VYOS_BUILD_FLAVOR='data/generic-arm64.json ./configure'
+ make iso
[2022-12-17 03:23:05] lb clean 
P: Cleaning chroot
set -o pipefail
./build-vyos-image iso
exit 0
I: Checking if packages required for VyOS image build are installed
build/config
I: Cleaning the build workspace
[2022-12-17 03:23:06] lb clean 
P: Cleaning chroot
I: Setting up additional APT entries
I: Configuring live-build
[2022-12-17 03:23:06] lb config noauto --architectures amd64 --bootappend-live boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=ttyS0,115200 console=tty0 net.ifnames=0 biosdevname=0 --bootappend-live-failsafe live components memtest noapic noapm nodma nomce nolapic nomodeset nosmp nosplash vga=normal console=ttyS0,115200 console=tty0 net.ifnames=0 biosdevname=0 --linux-flavours amd64-vyos --linux-packages linux-image-5.15.82 --bootloader syslinux,grub-efi --binary-images iso-hybrid --checksums sha256 md5 --debian-installer none --distribution bullseye --iso-application VyOS --iso-publisher root@82255319fe13 --iso-volume VyOS --debootstrap-options --variant=minbase --exclude=isc-dhcp-client,isc-dhcp-common,ifupdown --include=apt-utils,ca-certificates,gnupg2 --mirror-bootstrap http://deb.debian.org/debian --mirror-chroot http://deb.debian.org/debian --mirror-chroot-security http://deb.debian.org/debian-security --mirror-binary http://deb.debian.org/debian --mirror-binary-security http://deb.debian.org/debian-security --archive-areas main contrib non-free --firmware-chroot false --firmware-binary false --updates true --security true --backports true --apt-recommends false --apt-options --yes -oAPT::Get::allow-downgrades=true --apt-indices false
P: Updating config tree for a debian/bullseye/amd64 system
P: Symlinking hooks...
I: Starting image build
[2022-12-17 03:23:07] lb build 
P: live-build 20210407
P: Building for a debian/bullseye/amd64 system
[2022-12-17 03:23:07] lb bootstrap 
P: Setting up clean exit handler
[2022-12-17 03:23:08] lb bootstrap_cache restore
W: skipping /usr/lib/live/build/bootstrap_cache, foreign architecture(s).
[2022-12-17 03:23:08] lb bootstrap_debootstrap 
W: skipping /usr/lib/live/build/bootstrap_debootstrap, foreign architecture(s).
[2022-12-17 03:23:08] lb bootstrap_cache save
W: skipping /usr/lib/live/build/bootstrap_cache, foreign architecture(s).
[2022-12-17 03:23:08] lb chroot_prep install all-except-archives
E: the following stage is required to be done first: bootstrap
E: An unexpected failure occurred, exiting...
+ cd /vyos
+ bash build-u-boot.sh
Cloning into 'u-boot'...
remote: Enumerating objects: 895136, done.
remote: Counting objects: 100% (895136/895136), done.
remote: Compressing objects: 100% (132519/132519), done.
remote: Total 895136 (delta 752336), reused 894630 (delta 751902)
Receiving objects: 100% (895136/895136), 175.41 MiB | 4.22 MiB/s, done.
Resolving deltas: 100% (752336/752336), done.
Updating files: 100% (19209/19209), done.
Configuring u-boot for PI4
Building u-boot for PI4
Cleaning up
+ DEVTREE=bcm2711-rpi-cm4
+ PIVERSION=4
+ bash build-pi-image.sh vyos-build/build/live-image-arm64.hybrid.iso
Using uboot from ./u-boot.bin
VYOS Raspberry Pi3/4 image builder
Using input file:  vyos-build/build/live-image-arm64.hybrid.iso
Using output file: vyos-bcm2711-rpi-cm4.img
losetup: vyos-build/build/live-image-arm64.hybrid.iso: failed to set up loop device: No such file or directory
OOOPS!!! we crashed.. :/ starting a crude cleanup.

I think maybe the live-build failed and did not create the live-image-arm64.hybrid.iso file, but I don’t know why.

Does anyone have a solution?

The data/generic-arm64.json file seems to be missing.
This is probably due to a redesign of the flavor system.

I made the following changes and the build now proceeds:

#VYOS_BUILD_FLAVOR='data/generic-arm64.json ./configure'
#make iso
/vyos/vyos-build/build-vyos-image iso --architecture arm64

However, it does not complete and gives the following error:

[2022-12-18 13:49:04] lb chroot_install-packages install
P: Begin installing packages (install pass)...
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package grub-efi-arm is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  grub2-common

E: Unable to locate package linux-image---linux-packages
E: Package 'grub-efi-arm' has no installation candidate
E: An unexpected failure occurred, exiting...
P: Begin unmounting filesystems...
P: Saving caches...

Apparently, I need to prepare and load the correct flavor file.

I found the following README:

This README describes the format of the flavor file, but not how to use it.

In this case, should we prepare the generic-arm64.toml flavor file and do the following?

VYOS_BUILD_FLAVOR=data/generic-arm64.toml
make iso

I am not sure if this is the correct solution, but I changed build-image.sh of vyos-pi-builder as follows, and the build was completed tentatively. I have not yet confirmed that the VyOS image built with this script works properly.

build-image.sh is:

- # Clean out the build-repo and copy all custom packages
- rm -rf vyos-build
- git clone http://github.com/vyos/vyos-build vyos-build
  for a in $(find build -type f -name "*.deb" | grep -v -e "-dbgsym_" -e "libnetfilter-conntrack3-dbg"); do
  	echo "Copying package: $a"
  	cp $a vyos-build/packages/
  done
  
  cd vyos-build
  
  echo "Copy new default configuration to the vyos image"
  cp ${ROOTDIR}/config.boot.default data/live-build-config/includes.chroot/opt/vyatta/etc/config.boot.default
  
  # Build the image
- VYOS_BUILD_FLAVOR=data/generic-arm64.json
- ./configure
- make iso
+ /vyos/vyos-build/build-vyos-image iso --architecture arm64
  
  cd $ROOTDIR
  
  # Build u-boot
  bash build-u-boot.sh

Makefile is:

  container:
- 	sudo rm -rf vyos-build
- 	git clone -b current --single-branch https://github.com/vyos/vyos-build
- 	sudo docker build --arch arm64 vyos-build/docker -v /usr/bin/qemu-aarch64-static:/usr/bin/qemu-aarch64-static -t vyos/vyos-build:current-arm64
+ 	sudo docker build --platform arm64 vyos-build/docker -t vyos/vyos-build:current-arm64

- iso-local:
- 	sudo docker run --rm -it --arch arm64 --privileged -v /usr/bin/qemu-aarch64-static:/usr/bin/qemu-aarch64-static -v "$(shell pwd)":/vyos -v /dev:/dev --sysctl net.ipv6.conf.lo.disable_ipv6=0 localhost/vyos/vyos-build:current-arm64 /bin/bash -c 'cd /vyos; /bin/bash -x build-image.sh'
  
  iso-registry:
- 	sudo docker run --rm -it --arch arm64 --privileged -v /usr/bin/qemu-aarch64-static:/usr/bin/qemu-aarch64-static -v "$(shell pwd)":/vyos -v /dev:/dev --sysctl net.ipv6.conf.lo.disable_ipv6=0 vyos/vyos-build:current-arm64 /bin/bash -c 'cd /vyos; /bin/bash -x build-image.sh'
+ 	sudo docker run --rm -it --platform arm64 --privileged -v "$(shell pwd)":/vyos -v /dev:/dev --sysctl net.ipv6.conf.lo.disable_ipv6=0 vyos/vyos-build:current-arm64 /bin/bash -c 'cd /vyos; /bin/bash -x build-image.sh'

In addition, the following preparations were required.

$ git clone https://github.com/runborg/vyos-pi-builder
$ cd vyos-pi-builder
$ git clone -b current --single-branch https://github.com/vyos/vyos-build

$ wget -qO- https://repos.influxdata.com/influxdb.key > sudo tee vyos-build/data/live-build-config/archives/influxdb.key.chroot

$ vi vyos-build/data/architectures/arm64.toml
+ additional_repositories = [
+   "deb [arch=arm64] https://repos.influxdata.com/debian bullseye stable"
+ ]
  
+ kernel_flavor = "arm64-vyos"
  
  # Packages added to images for x86 by default
  packages = [
-   "grub-efi-arm"
+   "grub-efi-arm64",
+   "telegraf"
  ]
  bootloaders = "grub-efi"
$ vi build-pi-kernel.sh
set -x
set -e
ROOTDIR=$(pwd)
cd vyos-build/packages/linux-kernel/
git clone https://github.com/raspberrypi/linux
./build-kernel.sh
git clone https://github.com/accel-ppp/accel-ppp.git
./build-accel-ppp.sh
$sudo docker run --rm -it --platform arm64 --privileged -v "$(pwd)":/vyos -v /dev:/dev --sysctl net.ipv6.conf.lo.disable_ipv6=0 vyos/vyos-build:current-arm64 /bin/bash -c 'cd /vyos; /bin/bash -x build-pi-kernel.sh'

but I am concerned about the following warning.

W: Possible missing firmware /lib/firmware/e100/d102e_ucode.bin for built-in driver e100
W: Possible missing firmware /lib/firmware/e100/d101s_ucode.bin for built-in driver e100
W: Possible missing firmware /lib/firmware/e100/d101m_ucode.bin for built-in driver e100
W: Possible missing firmware /lib/firmware/intel/ice/ddp/ice.pkg for built-in driver ice
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125b-2.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125a-3.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8107e-2.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8107e-1.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168fp-3.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168h-2.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168h-1.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168g-3.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168g-2.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8106e-2.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8106e-1.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8411-2.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8411-1.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8402-1.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168f-2.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168f-1.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8105e-1.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168e-3.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168e-2.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168e-1.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-2.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-1.fw for built-in driver r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8156b-2.fw for built-in driver r8152
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8156a-2.fw for built-in driver r8152
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8153c-1.fw for built-in driver r8152
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8153b-2.fw for built-in driver r8152
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8153a-4.fw for built-in driver r8152
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8153a-3.fw for built-in driver r8152
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8153a-2.fw for built-in driver r8152

I forgot to build the firmware, so I did the following additionally.

build-pi-kernel.sh:

  set -x
  set -e
  ROOTDIR=$(pwd)
  cd vyos-build/packages/linux-kernel/
  git clone https://github.com/raspberrypi/linux
  ./build-kernel.sh
+ ln -s linux/drivers/firmware linux-firmware
+ ./build-linux-firmware.sh
  git clone https://github.com/accel-ppp/accel-ppp.git
  ./build-accel-ppp.sh

vyos-build/data/architectures/arm64.toml:

  additional_repositories = [
    "deb [arch=arm64] https://repos.influxdata.com/debian bullseye stable"
  ]
  
  kernel_flavor = "arm64-vyos"
  
  # Packages added to images for x86 by default
  packages = [
    "grub-efi-arm64",
+   "vyos-linux-firmware",
    "telegraf"
  ]
  bootloaders = "grub-efi"

The wi-fi device did not work even after running build-linux-firmware.sh. However, I did the following on VyOS and it is now recognized.

$ sudo -i
# wget -qO- https://github.com/openwrt/cypress-nvram/blob/master/brcmfmac43456-sdio.txt > /lib/firmware/brcm/brcmfmac43455-sdio.txt

In addition, because the kernel version included the unfamiliar “v8”, the following workaround was implemented.

vyos-build/build-vyos-image:

- --linux-packages linux-image-{{kernel_version}} \
+ --linux-packages linux-image-{{kernel_version}}-v8 \

vyos-build/packages/linux-kernel/build-linux-firmware.sh:

- FW_FILES=$(find ${LINUX_SRC}/debian/linux-image/lib/modules/${KERNEL_VERSION}${KERNEL_SUFFIX}/kernel/drivers/net -name *.ko | xargs modinfo | grep "^firmware:" | awk '{print $2}')
+ FW_FILES=$(find ${LINUX_SRC}/debian/linux-image/lib/modules/${KERNEL_VERSION}-v8${KERNEL_SUFFIX}/kernel/drivers/net -name *.ko | xargs modinfo | grep "^firmware:" | awk '{print $2}')

It appears to be working at least for the moment, so I will use this image for a while.