Wrong permissions on /dev/fuse on VyOS as a Xen PV guest

[aurinoco]

Dear All,

We have installed VyOS version 1.1.6 as a Xen Para-Virtualized guest. When we enter configure mode, we get the following error message:

vyos@vyos:~$ configure 
fuse: failed to open /dev/fuse: Permission denied
[edit]
vyos@vyos# 

We can make changes to the config, but, when trying to commit the changes, the VyOS virtual machine freezes with the following output (not reachable via SSH anymore, and Xen console freezes).

vyos@vyos:~$ configure 
fuse: failed to open /dev/fuse: Permission denied
[edit]
vyos@vyos# set system host-name edge-acn2.iciti.av
[edit]
vyos@vyos# commit
[ service ssh ]
Stopping OpenBSD Secure Shell server: sshd.

[ system package repository community ]
Removing entry from /etc/apt/sources.list...

[ system syslog ]
Can't use an undefined value as an ARRAY reference at /opt/vyatta/sbin/vyatta_update_syslog.pl line 96.

[ system login ]
All login methods can not be deleted

delete [ system login ] failed
[ system ntp ]
Stopping NTP server: ntpd.

[ system console device hvc0 ]
Warning: Access to system console is unconfigured

[ system console device ttyS0 ]
Warning: Access to system console is unconfigured

The vyos user is part of the fuse group, but when looking at the permissions of /dev/fuse we can see that, even though the group is set to fuse, there are no group read/write permissions on the file.

vyos@vyos:~$ groups
users adm disk sudo dip quaggavty vyattacfg fuse
vyos@vyos:~$ ls -l /dev/fuse
crw------- 1 root fuse 10, 229 Nov  4 06:32 /dev/fuse
vyos@vyos:~$

If we manually change the permissions of /dev/fuse to group read/write, then we are able to enter configure mode and commit. But on reboot, the permissions of /dev/fuse revert, and the problems with configure and commit re-present themselves.

The procedure we followed to create the Xen VM with VyOS is the following:

In order to install VyOS from its ISO, we need to first create the DomU as an HVM, mount the VyOS ISO, boot from it, install as normal (with the exception of generating Xen PV config files, which VyOS will prompt for when detecting a Xen DomU), then install the PV drivers, and finally convert the DomU to PV.

# Create a logical volume for the virtual machine's hard drive (20 GB in this case)
lvcreate -nedge-ern2.iciti.av -L20G xen2.nc

Create the configuration file at /etc/xen/edge-ern2.iciti.av and paste the following:

   builder='hvm'
   memory = 512
   vcpus=1
   name = "edge-ern2.iciti.av"
   vif = ['bridge=xenbr0','bridge=xenbr1']
   disk = ['phy:/dev/xen2.nc/edge-ern2.iciti.av,hda,w','file:/root/vyos-1.1.6-amd64.iso,hdc:cdrom,r']
   acpi = 1
   device_model_version = 'qemu-xen'
   boot="d"
   sdl=0
   serial='pty'

Then:

# Make the DomU start automatically on boot of Dom0
ln -s /etc/xen/{,auto/}edge-ern2.iciti.av

Obtain vyos-1.1.6-amd64.iso and place it in /root of xen2

# Start the virtual machine
cd /etc/xen/
xl create edge-ern2.iciti.av
# Find it's ID
xl list
# Connect to the virtual machine's console (replace 6 with the actual ID reported by xl list)
xl console 6

Login to VyOS, and run ‘install image’. Proceed with normal VyOS installation. When asked whether to generate Xen config files for PV conversion, answer Yes. When installation is complete, shutdown the DomU and proceed.

Edit /etc/xen/edge-ern2.iciti.av, and change the following line:

   disk = ['phy:/dev/xen2.nc/edge-ern2.iciti.av,hda,w','file:/root/vyos-1.1.6-amd64.iso,hdc:cdrom,r']

to

   disk = ['phy:/dev/xen2.nc/edge-ern2.iciti.av,hda,w','file:/root/xs-tools-6.5.0.iso,hdc:cdrom,r']

Obtain xs-tools-6.5.0.iso (Downloaded the XenServer ISO from http://downloadns.citrix.com.edgesuite.net/10175/XenServer-6.5.0-xenserver.org-install-cd.iso, and within it, inside packages.main/tools-iso.tar.bz2, inside which /opt/xensource/packages/iso/ contained the xs-tools-6.5.0.iso) and place it in /root of xen2.

Proceed:

# Start the virtual machine again
xl create edge-ern2.iciti.av
# Find its ID
xl list
# Connect to the console (replace 9 with the actual ID reported by xl list)
xl console 9

In the virtual machine, mount the CD-ROM drive and install XenTools:

sudo su
mount /dev/cdrom /mnt
/mnt/Linux/install.sh

Shutdown the machine instead of rebooting as suggested by the XenTools installer.

Now, we convert the virtual machine to a PV machine, and unmount the ISO image. Edit /etc/xen/edge-ern2.iciti.av once again, and modify it to contain the following:

memory = 512
maxmem = 1024
vcpus=1
name = "edge-ern2.iciti.av"
vif = ['bridge=xenbr0','bridge=xenbr1']
disk = ['phy:/dev/xen2.nc/edge-ern2.iciti.av,hda,w']
bootloader="pygrub"

Now start the virtual machine again, and let it be up and running:

xl create edge-ern2.iciti.av
xl list
xl console 11 (replace 11 with the actual ID reported by xl list)

We have tried variations of the above procedure where we omitted installation of the Xen Tools package, this did not help. Another variation was to omit the last step of conversion from Xen HVM (fully virtualized) to Xen PV (para-virtualized) virtual machine, this did help. The problem seems to present itself only when VyOS is running as a Xen PV guest, but not when running as a Xen HVM guest, or on physical hardware (same VyOS version 1.1.6).

Does anyone know of any solutions or workarounds?

Thank you

[SoreGums]

I’ve setup a Ubuntu 16.10 host with xen 4.7
Installed vyos from vyos-1.1.7-amd64.iso
and have run into the same problem as OP.
However when I setup vyos under citrix xenserver 6.5 I was able to run as PV guest with no problems following: http://forum.vyos.net/showthread.php?tid=6473
Running into a few different problems using xen on ubuntu vs xenserver on metal… might need to to go back to xenserver.