IPv6 traceback on 1.4?

Hi all,
i’m trying to migrate a working 1.3 config to 1.4 but i get the following error when some IPv6 are added to firewall, interfaces or protocol (example bgp and static) sections:

Traceback (most recent call last):
File “/usr/bin/config-mgmt”, line 33, in
sys.exit(load_entry_point(‘vyos==1.3.0’, ‘console_scripts’, ‘config-mgmt’)())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/vyos/config_mgmt.py”, line 608, in run
config_mgmt = ConfigMgmt()
^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/vyos/config_mgmt.py”, line 70, in init
config = Config()
^^^^^^^^
File “/usr/lib/python3/dist-packages/vyos/config.py”, line 85, in init
self._config_source = ConfigSourceSession(session_env)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/vyos/configsource.py”, line 145, in init
self._session_config = ConfigTree(session_config_text)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/vyos/configtree.py”, line 143, in init
raise ValueError(“Failed to parse config: {0}”.format(msg))
ValueError: Failed to parse config: internal indexing error

I have no coding abilities for debugging python, so i generated a custom ISO with python3 packages from bullseye (3.9.2) instead of bookworm (3.11.2) but the problem remains.

Sometimes the configuration is working removing some sections, sometimes i add a new acl rule with a IPv6 address or a new IPv6 BGP peer or a blackhole/static route6 and i get the traceback.

Any clues?
oTo

If you can provide the full config you are trying to migrate along with the source and destination version numbers we may be able to give you more info. My assumption would be either some weird syntax in the existing config that was not expected for the config migration.

Dear Kyle,
thank you for your answer.

It’s not possible to migrate from 1.3 directly, migration script fails. It works only by removing some random IPv6 sections (sometimes even removing just one random static route6 is enough, other times you need to remove IPv6 address on interfaces or some firewall ACL or some BGP peers).
The same problem happens if i copy an already 1.4 configuration (rolling from 11 feb 23) from a production router, migration fails.
I just rebuilt a new ISO with the latest commits on vyos-1x (192c8ed), same thing. Tried a new ISO with a backported previous version of vyatta-cfg (before config-mgmt changes), same thing.
I double (30 times really) checked the syntax and even started config from scratch one command at a time (so syntax is validated), at some point something cracks and you can’t add or change anything IPv6 related.

I think this is something related to a code change in the last 2 months but i don’t have the expertise to find what and where, i just blindly rebuild packages trying to backtrack commits.
oTo

Im currently running about 16 VyOS boxes on 1.4 rolling built within the last 2 weeks and haven’t run into any issues. My networks are primarily IPv6 and are upgraded usually every month. Without being able to see the config so we can replicate and troubleshoot the issue for you there’s not much anyone will be able to tell you.

Dear Kyle,
probably i’ve been able to pinpoint the issue.

Can you please try to add this command to your test config and let me know if you get any errors?

set protocol static route6 2a00:1000:1::1/128 next-hop 2a00:1000:2000::11
set protocol static route6 2a0b:1000::/32 blackhole

oTo

I was able to run those commands without issues. I rebooted the router aswell so it would load them from boot to see if there were any issues too and there was not.

interfaces {
    ethernet eth0 {
        address 172.16.4.155/24
    }
    ethernet eth1 {
        address 2602:fb68:c1:1::15c/64
        mtu 1400
    }
    ethernet eth2 {
        address 23.136.233.58/27
        mtu 1400
    }
    loopback lo {
    }
}
nat {
    source {
        rule 100 {
            outbound-interface eth2
            source {
                address 172.16.0.0/16
            }
            translation {
                address masquerade
            }
        }
    }
}
protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 23.136.233.33 {
            }
        }
        route6 ::/0 {
            next-hop 2602:fb68:c1:1::1 {
            }
        }
        route6 2a00:1000:1::1/128 {
            next-hop 2a00:1000:2000::11 {
            }
        }
        route6 2a00:1000::/32 {
            blackhole {
            }
        }
    }
}
service {
    ntp {
        allow-client {
            address 0.0.0.0/0
            address ::/0
        }
        server 0.pool.ntp.org {
        }
        server 1.pool.ntp.org {
        }
        server 2.pool.ntp.org {
        }
    }
    ssh {
    }
}
system {
    config-management {
        commit-revisions 100
    }
    conntrack {
        modules {
            ftp
            h323
            nfs
            pptp
            sip
            sqlnet
            tftp
        }
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
    host-name vyos
    login {
        user vyos {
            authentication {
                encrypted-password ****************
                plaintext-password ****************
                public-keys idenifier {
                    key ****************
                    type ssh-rsa
                }
            }
        }
    }
    name-server 8.8.8.8
    syslog {
        global {
            facility all {
                level info
            }
            facility protocols {
                level debug
            }
        }
    }
}
vyos@vyos:~$ show ipv6 route 
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

S>* ::/0 [1/0] via 2602:fb68:c1:1::1, eth1, weight 1, 00:03:18
C>* 2602:fb68:c1:1::/64 is directly connected, eth1, 00:03:23
S>* 2a00:1000::/32 [1/0] unreachable (blackhole), weight 1, 00:03:18
S>  2a00:1000:1::1/128 [1/0] via 2a00:1000:2000::11 (recursive), weight 1, 00:03:18
  *                            unreachable (blackhole), weight 1, 00:03:18
C * fe80::/64 is directly connected, eth2, 00:03:23
C * fe80::/64 is directly connected, eth0, 00:03:23
C * fe80::/64 is directly connected, eth1, 00:03:23
C>* fe80::/64 is directly connected, lo, 00:03:26
vyos@vyos:~$ show version 
Version:          VyOS 1.4-rolling-202304160706
Release train:    current

Built by:         kylem@serverforge.org
Built on:         Sun 16 Apr 2023 07:06 UTC
Build UUID:       d616ea6f-7763-49de-8a48-97f548eda4b8
Build commit ID:  ea4aede9c4daa8

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  QEMU
Hardware model:   Standard PC (i440FX + PIIX, 1996)
Hardware S/N:     
Hardware UUID:    a1e08cf3-5e26-4a89-aae9-ab82d729eaf6

Copyright:        VyOS maintainers and contributors
vyos@vyos:~$ cat /etc/frr/frr.conf
frr version 8.5
frr defaults traditional
hostname vyos
log syslog
log facility local7
service integrated-vtysh-config
!
ip route 0.0.0.0/0 23.136.233.33
ipv6 route ::/0 2602:fb68:c1:1::1
ipv6 route 2a00:1000::/32 blackhole
ipv6 route 2a00:1000:1::1/128 2a00:1000:2000::11
!
rpki
exit
!

Dear Kyle,
after countless tests i tried a new vanilla config on the latest rolling:

vyos@vyos:~$ sh conf
interfaces {
    ethernet eth0 {
        address 192.168.1.100/24
        hw-id ac:1f:6b
    }
    ethernet eth1 {
        hw-id ac:1f:6b
    }
    ethernet eth2 {
        hw-id ac:1f:6b
    }
    ethernet eth3 {
        hw-id ac:1f:6b
    }
    ethernet eth4 {
        hw-id ac:1f:6b
    }
    ethernet eth5 {
        hw-id ac:1f:6b
    }
    ethernet eth6 {
        hw-id ac:1f:6b
    }
    ethernet eth7 {
        hw-id 00:07:43
    }
    ethernet eth8 {
        hw-id 00:07:43
    }
    ethernet eth10 {
        hw-id 00:07:43
    }
    ethernet eth11 {
        hw-id 00:07:43
    }
    loopback lo {
    }
}
protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 192.168.1.1 {
            }
        }
    }
}
service {
    ntp {
        allow-client {
            address 0.0.0.0/0
            address ::/0
        }
        server time1.vyos.net {
        }
        server time2.vyos.net {
        }
        server time3.vyos.net {
        }
    }
    ssh {
    }
}
system {
    config-management {
        commit-revisions 100
    }
    conntrack {
        modules {
            ftp
            h323
            nfs
            pptp
            sip
            sqlnet
            tftp
        }
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
    host-name vyos
    login {
        user vyos {
            authentication {
                encrypted-password ****************
                plaintext-password ****************
            }
        }
    }
    syslog {
        global {
            facility all {
                level info
            }
            facility protocols {
                level debug
            }
        }
    }
}
vyos@vyos:~$ sh ver
Version:          VyOS 1.4-rolling-202304130846
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Thu 13 Apr 2023 08:46 UTC
Build UUID:       1d949ed4-c78a-4abb-a849-8f0d7617acd3
Build commit ID:  e52a5136ef375e

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  Supermicro
Hardware model:   Super Server
Hardware S/N:     0123456789
Hardware UUID:    00000000-0000-0000-0000-ac1f6b64a4f0

Copyright:        VyOS maintainers and contributors
vyos@vyos# set protocol static route6 2a00:1000:1::1/128 next-hop 2a00:1000:2000::11
[edit]
vyos@vyos# set protocol static route6 2a0b:1000::/32 blackhole
[edit]
vyos@vyos# comp
Traceback (most recent call last):
  File "/usr/bin/config-mgmt", line 33, in <module>
    sys.exit(load_entry_point('vyos==1.3.0', 'console_scripts', 'config-mgmt')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/config_mgmt.py", line 608, in run
    config_mgmt = ConfigMgmt()
                  ^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/config_mgmt.py", line 70, in __init__
    config = Config()
             ^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/config.py", line 85, in __init__
    self._config_source = ConfigSourceSession(session_env)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/configsource.py", line 145, in __init__
    self._session_config = ConfigTree(session_config_text)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 143, in __init__
    raise ValueError("Failed to parse config: {0}".format(msg))
ValueError: Failed to parse config: internal indexing error

Boom, not working.

Good morning, I was able to replicate the issue with your config on VyOS 1.4-rolling-202304130846. Il let you know what i find.

Seems to be something that image doesnt like with the blackhole command, I cant seem to pin down what exactly is causing it, however my image i initially tested it on is a more recent build and does not seem to have the issue, and the previous rolling image vyos-1.4-rolling-202304120317-amd64.iso does not seem to have the issue either so it seems the issue was already corrected. Could you see if you still have the issue with vyos-1.4-rolling-202304120317-amd64.iso? You are also welcome to use my image if you want to try the one i originally tested with.

https://s3-us.vyos.io/rolling/current/vyos-1.4-rolling-202304120317-amd64.iso
https://repo.serverforge.org/vyos/sagitta/vyos-1.4-rolling-202304160706-amd64.iso

Dear Kyle,
i tried with both images. Here the results:

vyos@vyos:~$ sh ver
Version:          VyOS 1.4-rolling-202304120317
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Wed 12 Apr 2023 03:17 UTC
Build UUID:       8b62fd39-3a24-4239-8a49-fe145e9f00f0
Build commit ID:  e52a5136ef375e

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  Supermicro
Hardware model:   Super Server
Hardware S/N:     0123456789
Hardware UUID:    00000000-0000-0000-0000-ac1f6b64a4f0

Copyright:        VyOS maintainers and contributors
vyos@vyos:~$ conf
[edit]
vyos@vyos# set protocol static route6 2a00:1000:1::1/128 next-hop 2a00:1000:2000::11
[edit]
vyos@vyos# set protocol static route6 2a0b:1000::/32 blackhole
[edit]
vyos@vyos# comp
Traceback (most recent call last):
  File "/usr/bin/config-mgmt", line 33, in <module>
    sys.exit(load_entry_point('vyos==1.3.0', 'console_scripts', 'config-mgmt')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/config_mgmt.py", line 608, in run
    config_mgmt = ConfigMgmt()
                  ^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/config_mgmt.py", line 70, in __init__
    config = Config()
             ^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/config.py", line 85, in __init__
    self._config_source = ConfigSourceSession(session_env)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/configsource.py", line 145, in __init__
    self._session_config = ConfigTree(session_config_text)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 143, in __init__
    raise ValueError("Failed to parse config: {0}".format(msg))
ValueError: Failed to parse config: internal indexing error
vyos@vyos:~$ sh ver
Version:          VyOS 1.4-rolling-202304160706
Release train:    current

Built by:         kylem@serverforge.org
Built on:         Sun 16 Apr 2023 07:06 UTC
Build UUID:       d616ea6f-7763-49de-8a48-97f548eda4b8
Build commit ID:  ea4aede9c4daa8

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  Supermicro
Hardware model:   Super Server
Hardware S/N:     0123456789
Hardware UUID:    00000000-0000-0000-0000-ac1f6b64a4f0

Copyright:        VyOS maintainers and contributors

vyos@vyos# set protocol static route6 2a00:1000:1::1/128 next-hop 2a00:1000:2000::11
[edit]
vyos@vyos# set protocol static route6 2a0b:1000::/32 blackhole
[edit]
vyos@vyos# comp
Traceback (most recent call last):
  File "/usr/bin/config-mgmt", line 33, in <module>
    sys.exit(load_entry_point('vyos==1.3.0', 'console_scripts', 'config-mgmt')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/config_mgmt.py", line 608, in run
    config_mgmt = ConfigMgmt()
                  ^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/config_mgmt.py", line 70, in __init__
    config = Config()
             ^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/config.py", line 85, in __init__
    self._config_source = ConfigSourceSession(session_env)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/configsource.py", line 145, in __init__
    self._session_config = ConfigTree(session_config_text)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 143, in __init__
    raise ValueError("Failed to parse config: {0}".format(msg))
ValueError: Failed to parse config: internal indexing error

From what i see the issue is only present when both blackhole AND next-hop rules are used at the same time. I can put as many blackholes as i want without problem, as soon as i set a static route, booom, it crashes.

oTo

Thats strange, i was only able to replicate it on 1.4-rolling-202304130846, maybe someone else will have some better luck trying to replicate it.

The only difference i see is i’m using bare metal and you’re using virtualized, i don’t know how can change something in this regard.
I tried to blacklist some nic drivers to reduce the number of interfaces but the issue remained.

I’m losing some life expectancy behind this problem :sweat_smile:

I created the task ⚓ T5185 Static IPv6 route with blackhole fails

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.