Documentation Proposal: Update Getting Started VyOS with PPPoE and IPv6

Development
1

Hello,

I suspect I am one of many new users of VyOS who were inspired to try it out after having seen the blog post, on Hackernews VyOS from Scratch about a month ago. I like VyOS a lot, because it reminds me of configuring Cisco equipment I did 20 years ago. Kudos to Kroy for creating this post. I also have a Dell Wyse 5070 and am using a GPON SFP module to replace my ISP’s equipment. I ended up abandoning both OPNSense and OpenWRT because of Realtek Ethernet PHY driver issues with PPPoE connections. On OpenWRT, the PPPoE connection would drop in just a few minutes and never reconnect. On OPNSense, the drivers could not cope with my ISPs/upgrade to 500/500 Mbps service and would hang during high throughput. This is a known problem with Realtek drivers and has something to do with a lack of queues, surprisingly the cheap CPE equipment provided by ISPs do not have these issues. Ultimately, it is my intention to virtualize VyOS on Proxmox with a passed-through HPE 530 SFP+ Broadcom-based PCIe NIC and upgrade to 2.5 Gbps service, but this will bring with it its own bnx2x driver/firmware issues which are documented on the DSLReports forums.

Unfortunately, as Kroy noted in an update, the commands in the blog post are a bit out of date. Plus, the guide doesn’t cover setting up IPv6 or PPPoE. Initially, I was able to get IPv4 and PPPoE running, but I had to revert back to my ISP’s router shortly thereafter because too many websites don’t resolve without IPv6. In particular, my search engine of choice duckduckgo didn’t work.

This weekend, I spent another few hours researching and reading forum posts on how to solve the IPv6 issue. I just got it all working a few minutes ago and I started writing this post. I am thinking of opening a PR to improve the documentation, but I am unsure if this would be appreciated. I feel that Kroy’s guide should become part of the official documentation; the explanations of what the commands do are some of the best I’ve ever read. However, there were a few steps missing for my particular situation.

If I were to open a PR, I’d like to address in current issues I see in VyOS documentation:

  • There are multiple documentation pages which have inconsistent names for interfaces, firewall rules, and firewall zones. Particularly, they use different underscores and dashes. Also, numbering may be a problem. I read the contributing guidelines, but I am not a RestructuredText expert. In other documentation tools, this would be solved with judicious use of variables. Or perhaps a recommended set of basic rules, names, and zones would help get users a minimal configuration, and then they could begin their customization from there. I’m of course targeting novice/new users.

  • I am not running VRF. I will look into this in the future, but it seems new users should be able to get by without using it.

  • I also used a post on the Level One Tech’s forum to get some information and read a half-dozen posts on this forum and reddit, but they did not yield a solution, except that it was likely that my firewall was dropping the router advertisements. There was some good advice about using tcpdump,

  • It’s still not clear to me what 'set service router-advert <interface> name-server <address>' should be, I tried finding this information in my ISP’s router, but cannot be sure I input the correct addresses. This seems like it should be automatic/dynamically configured as OpenWRT/OPNsense are able to handle it. I also found posts suggesting an alternative to use ip adjust-mss 'clamp-mss-to-pmtu' and set service router-advert interface <interface> managed-flag. I took a long break from networking/tinkering when IPv6 was being rolled out.

  • I found a post, from a Taiwanese user Ramax, 2021-03-18 VyOS Dual Stack, which led to them creating a PR and the generation of the PPPoE IPv6 Basic Setup for Home Network , this was the key!

My problem came down to a missing step in Ramax’s advice for anyone who had set up a zone-based firewall as instructed by Kroy: the user needs to add the firewall rules to the zone.


set firewall zone WAN from LOCAL firewall ipv6-name WAN_LOCAL

set firewall zone LOCAL from WAN firewall ipv6-name WAN_LOCAL

Once I added those settings, I was able to ping6 address from the VyOS shell, and access IPv6-based sites from my client devices.

I suppose the question is how should I proceed? I’d like to offer my help improving the documentation.

  • I want to give full credit to the individuals who wrote the original content and not rip the off.

  • I would like to re-structure the documentation to make it more modular, so that it targets users who have ISPs with a variety of configuration types (DHCP/PPPoE).

I’m looking forward to the discussion and perhaps becoming a long-term contributor to VyOS documentation.

3 Likes
2

Unfortunately, I spoke too soon. My PPPoE interface was able to pick up an IPv6 address, but the other Ethernet address never did. I thought it could have something to do with my heavily edited configuration and poorly name firewall rules (due to following multiple guides). I tried updating to the latest nightly/rolling release to start over, but it seems the configuration syntax has heavily changed, including zone-policy (and the firewall rule hierarchy completely changed). It’s discouraging that these changes are not documented or that that the commands have not been frozen… it makes the using the documentation very difficult and discouraging for newcomers. Plus, it’s very disconcerting to know the basic syntax of commands will continually change.

1 Like
3

I made one more attempt to get things running, but the configuration steps are not clear :frowning:

  • My first attempt was using vyos-1.4-rolling-202308050917-amd64.iso
  • My second attempt (update) was with vyos-1.4-rolling-202308230020-amd64.iso
  • Yesterday, I tried vyos-1.2.9-S1-amd64.iso, but encountered a problem setting up the firewall on the PPPoE interface, using the instructions on the wiki for the Crux build:
    Note: my Eth1 is a NIC with a SFP module and Eth0 is 1000BASE-T Ethernet.
set interface ethernet eth1 pppoe 0 firewall in name NET-IN
set interface ethernet eth1 pppoe 0 firewall local name NET-LOCAL
set interface ethernet eth1 pppoe 0 firewall out name NET-OUT

[ interfaces ethernet eth1 pppoe 0 firewall in name NET-IN ]
Configuration error: Rule set "NET-IN" is not configured

The pppoe interface did connect, but the echo requests were likely being rejected by the lack of a firewall rule allowing them to pass through.

show interfaces pppoe pppoe0 log

Sun Aug 27 13:35:40 UTC 2023: PPP interface pppoe0 created
Sun Aug 27 13:35:41 UTC 2023: Stopping PPP daemon for pppoe0
Sun Aug 27 13:35:41 UTC 2023: Starting PPP daemon for pppoe0
Send PPPOE Discovery V1T1 PADI session 0x0 length 12
 dst ff:ff:ff:ff:ff:ff  src <source MAC>
 [service-name] [host-uniq  9d 0e 00 00]
Recv PPPOE Discovery V1T1 PADO session 0x0 length 30
 dst <DEST MAC>  src <SOURCE MAC>
 [host-uniq  9d 0e 00 00] [AC-name BRAS-NAME [service-name]
Send PPPOE Discovery V1T1 PADR session 0x0 length 12
 dst <DEST MAC>  src <SOURCE MAC>
 [service-name] [host-uniq  9d 0e 00 00]
Recv PPPOE Discovery V1T1 PADS session 0x1 length 30
 dst <DEST MAC>  src <SOURCE MAC>
 [service-name] [host-uniq  9d 0e 00 00] [AC-name BRAS-NAME]
PADS: Service-Name: ''
PPP session is 1
Connected to <A_MAC> via interface eth1
using channel 1
Using interface ppp0
Connect: ppp0 <--> eth1
sent [LCP ConfReq id=0x1 <mru 1492> <magic 0x46ce0440>]
rcvd [LCP ConfReq id=0xb6 <mru 1492> <auth pap> <magic 0x5bb11889>]
sent [LCP ConfAck id=0xb6 <mru 1492> <auth pap> <magic 0x5bb11889>]
rcvd [LCP ConfAck id=0x1 <mru 1492> <magic 0x46ce0440>]
sent [LCP EchoReq id=0x0 magic=0x46ce0440]
sent [PAP AuthReq id=0x1 user="<USER>" password=<hidden>]
rcvd [LCP EchoRep id=0x0 magic=0x5bb11889]
rcvd [PAP AuthAck id=0x1 ""] 00 00 00
PAP authentication succeeded
peer from calling number <A_MAC> authorized
sent [IPCP ConfReq id=0x1 <addr 0.0.0.0> <ms-dns1 0.0.0.0> <ms-dns2 0.0.0.0>]
rcvd [IPCP ConfReq id=0xbd <AN IP>]
sent [IPCP ConfAck id=0xbd <AN IP>]
rcvd [IPCP ConfNak id=0x1 <AN IP + DNS1 DNS2>]
sent [IPCP ConfReq id=0x2 <AN IP + DNS1 DNS2>]
rcvd [IPCP ConfAck id=0x2 <AN IP + DNS1 DNS2>]
Script /etc/ppp/ip-pre-up started (pid 3773)
Script /etc/ppp/ip-pre-up finished (pid 3773), status = 0x0
local  IP address <IP>
remote IP address <REMOTE IP>
primary   DNS address <ISP_DNS>
secondary DNS address  <ISP_DNS2>
Script /etc/ppp/ip-up started (pid 3813)
Script /etc/ppp/ip-up finished (pid 3813), status = 0x0
No response to 3 echo-requests
Serial link appears to be disconnected.
Connect time 4.0 minutes.
Sent 12129 bytes, received 6127 bytes.
Script /etc/ppp/ip-down started (pid 4029)
sent [LCP TermReq id=0x2 "Peer not responding"]
Script /etc/ppp/ip-down finished (pid 4029), status = 0x0
sent [LCP TermReq id=0x3 "Peer not responding"]
Connection terminated.
Modem hangup
Send PPPOE Discovery V1T1 PADI session 0x0 length 12

I’m in a chicken-or-the-egg situation. To make a contribution to the documentation (and get access to a better build), I need a working system which requires working documentation.

4

The documentation in the PPPoE IPv6 Basic Setup guide is mostly alright, except that advertising an MTU of 1492 for the internal/LAN interface is non-sense (as that would limit LAN traffic too).

For getting things started you don’t need any firewall rules (as soon as everything works as intended you should of course set some up). I recommend that you post your full config (show | strip-private) and possibly what your network is supposed to look like in the end (simple WAN-LAN setup, or multiple VLANs, …), as it’s hard to say what’s missing otherwise.

This is simply the address of the DNS server you want your router to advertise to clients, similar to a DNS server advertised via DHCP. You can use the server of your ISP if you feel like it (if they even have IPv6 ones), or even better one of the many others, like those of Cloudflare, Quad9, or Google.

Not sure what you mean with “alternative”, but clamping the MSS for both IPv4 and IPv6 might be necessary, as some service providers (like Fastly) are incapable and or unwilling to configure their network properly, breaking Path MTU Discovery. When using PPPoE, your outgoing MTU is limited to 1492, so you might want set the following MSS values on the PPPoE interface (not on any LAN interfaces): IPv4 = 1452, IPv6 = 1432.

You most certainly don’t want this, as this would require you to also set up a DHCPv6 server, which is something you don’t really need or want in a home network.

3 Likes
5

look at these links should help you

(PPPoE Sub-interfaces : VyOS Support Portal)version 1.3 or older

latest version 1.4

2 Likes
6

@DerEnderKeks Thank you for your feedback. I had time today to try this again on vyos-1.4-rolling-202308050917-amd64.iso. I followed your advice to not set up a firewall, but this time it seems that I was unable to get an IPv6 address on my WAN interface from VyOS.

I tried two cabling configurations. As noted, eth0 is the built-in Ethernet adapter on the Dell 5070. eth1 is an SFP module. I can plug in my GPON SFP module here, but I also I have a set of SFP transceivers I plug into a switch, where I usually have my GPON SFP module; this is so I can check the OpenWRT GUI of the module to make sure the status is ‘5’. However, today I was surprised to see that when I forgot to disconnect my ISP-provided router, the PPPoE client was able to connect to my ISP; I effectively have two sessions and two ip ipv4 addresses from my ISP - whoops, this reminds me of the time in the 90s my dial-up ISP charged me an extra fee for connecting two modem sessions from different phone lines - This is only possible because the ISP does not provide the PPPoE username/password tied to their gateway. Regardless, moving the fiber connection from the ISP’s equipment and connecting it to my GPON SFP had no effect.

I tried two different pppoe configurations. I have pasted them both here.

In addition, I did perform a TCPdump and I do see some IPv6-related traffic, but I don’t know what to do with this.

 interfaces {
     ethernet eth0 {
         address xxx.xxx.40.1/24
         description LAN
         hw-id xx:xx:xx:xx:xx:68
     }
     ethernet eth1 {
         description WAN_PPPoE
         hw-id xx:xx:xx:xx:xx:18
     }
     loopback lo {
     }
     pppoe pppoe0 {
         authentication {
             password xxxxxx
             username xxxxxx
         }
         default-route-distance 10
         description PPPoE
         dhcpv6-options {
             pd 0 {
                 interface eth0 {
                     address 65534
                     sla-id 0
                 }
                 length 64
             }
         }
         ipv6 {
             address {
                 autoconf
             }
         }
         no-peer-dns
         source-interface eth1
     }
 }
 nat {
     source {
         rule 100 {
             outbound-interface pppoe0
             source {
                 address xxx.xxx.40.0/24
             }
             translation {
                 address masquerade
             }
         }
     }
 }
 service {
     dhcp-server {
         shared-network-name xxxxxx {
             subnet xxx.xxx.40.0/24 {
                 default-router xxx.xxx.40.1
                 name-server xxx.xxx.40.1
                 range 0 {
                     start xxx.xxx.40.50
                     stop xxx.xxx.40.125
                 }
                 range 1 {
                     start xxx.xxx.40.200
                     stop xxx.xxx.40.250
                 }
             }
         }
     }
     dns {
         forwarding {
             allow-from xxx.xxx.40.0/24
             cache-size 0
             listen-address xxx.xxx.40.1
             name-server xxx.xxx.0.1 {
             }
             name-server xxx.xxx.1.1 {
             }
             name-server xxx.xxx.4.4 {
             }
             name-server xxx.xxx.8.8 {
             }
         }
     }
     ntp {
         allow-client xxxxxx
             address xxx.xxx.0.0/0
             address ::/0
         }
         server xxxxx.tld {
         }
         server xxxxx.tld {
         }
         server xxxxx.tld {
         }
     }
     router-advert {
         interface eth0 {
             name-server xxxx:xxxx:4860::8888
             name-server xxxx:xxxx:4860::8844
             prefix ::/64 {
                 valid-lifetime 172800
             }
         }
     }
     ssh {
         port 22
     }
 }
 system {
     config-management {
         commit-revisions 100
     }
     conntrack {
         modules {
             ftp
             h323
             nfs
             pptp
             sip
             sqlnet
             tftp
         }
     }
     console {
         device ttyS0 {
             speed 115200
         }
     }
     host-name xxxxxx
     login {
         user xxxxxx {
             authentication {
                 encrypted-password xxxxxx
                 plaintext-password xxxxxx
             }
         }
     }
     name-server xxx.xxx.40.1
     syslog {
         global {
             facility all {
                 level info
             }
             facility local7 {
                 level debug
             }
         }
     }
 }

I also tried an alternative PPPoE configuration:

pppoe pppoe0 {
         authentication {
             password xxxxxx
             username xxxxxx
         }
         default-route-distance 10
         description PPPoE
         dhcpv6-options {
             pd 0 {
                 interface eth0 {
                     address 100
                 }
             }
         }
         ipv6 {
             address {
                 autoconf
             }
         }
         no-peer-dns
         source-interface eth1
     }

tcpdump -i pppoe0 -vv ip6
tcpdump: listening on pppoe0, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
14:18:20.727166 IP6 (flowlabel 0x404bb, hlim 1, next-header UDP (17) payload length: 85) <<AN_IPV6_ADDRESSS>>.dhcpv6-client > <<AN_IPV6_ADDRESSS>>.dhcpv6-server: [udp sum ok] dhcp6 solicit (xid=f3ec0c (client-ID type 4) (elapsed-time 1644) (IA_PD IAID:0 T1:0 T2:0 (IA_PD-prefix ::/64 pltime:4294967295 vltime:4294967295)))
14:18:37.962638 IP6 (flowlabel 0x404bb, hlim 1, next-header UDP (17) payload length: 85) <<AN_IPV6_ADDRESSS>>.dhcpv6-client > <<AN_IPV6_ADDRESSS>>.dhcpv6-server: [udp sum ok] dhcp6 solicit (xid=f3ec0c (client-ID type 4) (elapsed-time 3368) (IA_PD IAID:0 T1:0 T2:0 (IA_PD-prefix ::/64 pltime:4294967295 vltime:4294967295)))
14:19:13.097026 IP6 (flowlabel 0x404bb, hlim 1, next-header UDP (17) payload length: 85) <<AN_IPV6_ADDRESSS>>.dhcpv6-client > <<AN_IPV6_ADDRESSS>>.dhcpv6-server: [udp sum ok] dhcp6 solicit (xid=f3ec0c (client-ID type 4) (elapsed-time 6881) (IA_PD IAID:0 T1:0 T2:0 (IA_PD-prefix ::/64 pltime:4294967295 vltime:4294967295)))

^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

Let me know if there is something relavant in the redacted information, I might be able to send it in a private message.

7

Your PPPoE config looks almost alright, some notes:

  • default-route-distance - is there any reason you set this? (probably not an issue though)
  • dhcpv6-options/pd/0/interface/eth0/address - I suggest using 1 for the router, but that’s personal preference (easier to type)
  • dhcpv6-options/pd/0/interface/eth0/sla-id - not sure what the default is, I would always set it, 0 is fine (for the first delegated subnet)
  • dhcpv6-options/pd/0/length - most likely a problem when set to 64 (also the default when unset). This value must match what the ISP actually offers. Best case 48, more likely 56, but other values are possible. If your ISP actually only provides a single /64 (or really anything less than /56), they suck. It might help if you tell us who your ISP is.

It sounds like PPPoE itself and IPv4 are working already, but just to be sure:

  • Can you ping external IPs on v4?
  • What does the PPPoE log look like now?
  • Output of ip a s pppoe0 (redacted, but please leave the first segment of IPv6 addresses visible)
  • Output of show log dhcpv6 client interface pppoe0 (the lines related to the last connection attempt are enough)
  • Does your ISP require a specific VLAN for the PPPoE connection? Unless your GPON module removes the tag already, this maybe needs to be configured too. (Here in Germany, on DSL connections, you usually have to use VLAN 7)

The packets in the dump are just from the DHCPv6 client, which is asking for a prefix, but apparently never gets any reply.

1 Like
9

Thank you @DerEnderKeks. Thank you @himurae.
IPv6 is finally working from my LAN interface. Previously, IPv4 would come up on all interfaces, but IPv6 would only sometimes appear on the pppoe0 interface, which allowed ping6 from the vyos CLI.

Now everything appears to be working.
The next steps are to begin hardening this installation.

Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             <ipv4>.40.1/24                     u/u  LAN
                 <4>:<4>:<4>:<4>::1/64              
eth1             -                                 u/u  WAN_PPPoE
eth1.xxxx          -                                 u/u  
eth1.xxxx         -                                 u/u  
eth1.xxxx         -                                 u/u  
eth1.xxxx        -                                 u/u  
lo               127.0.0.1/8                       u/u  
                 ::1/128                                
pppoe0           <IPv4>/32                  u/u  PPPoE
                 <4>:<4>:<4>:<4>:<4>:<4>:<4>:<4>/64

Here are the changes I made:

  • I modified the pppoe0 interface’s dhcpv6-options.
  • I removed default-route-distance 210.
  • I added four VLANs to the configuration, which I collected from the DIY GPON guides I found. One of these I set as dhcpv6.
  • I added adjust-mss clamp-mss-to-pmtu to both pppoe0’s ip and ipv6 settigns.
  • I added the ipv6 dns information (google + cloudflare).
 interfaces {
     ethernet eth0 {
         address xxx.xxx.40.1/24
         description LAN
         hw-id xx:xx:xx:xx:xx:68
     }
     ethernet eth1 {
         description WAN_PPPoE
         hw-id xx:xx:xx:xx:xx:18
         vif xxxx {
             address dhcp
         }
         vif xxxx {
             address dhcp
         }
         vif xxxx {
             address dhcp
         }
         vif xxxx {
             address dhcpv6
         }
     }
     loopback lo {
     }
     pppoe pppoe0 {
         authentication {
             password xxxxxx
             username xxxxxx
         }
         description PPPoE
         dhcpv6-options {
             pd 0 {
                 interface eth0 {
                     address 1
                     sla-id 0
                 }
             }
         }
         ip {
             adjust-mss clamp-mss-to-pmtu
         }
         ipv6 {
             address {
                 autoconf
             }
             adjust-mss clamp-mss-to-pmtu
         }
         no-peer-dns
         source-interface eth1
     }
 }
 nat {
     source {
         rule 100 {
             outbound-interface pppoe0
             source {
                 address xxx.xxx.40.0/24
             }
             translation {
                 address masquerade
             }
         }
     }
 }
 service {
     dhcp-server {
         shared-network-name xxxxxx {
             subnet xxx.xxx.40.0/24 {
                 default-router xxx.xxx.40.1
                 name-server xxx.xxx.40.1
                 range 0 {
                     start xxx.xxx.40.50
                     stop xxx.xxx.40.125
                 }
                 range 1 {
                     start xxx.xxx.40.200
                     stop xxx.xxx.40.250
                 }
             }
         }
     }
     dns {
         forwarding {
             allow-from xxx.xxx.40.0/24
             cache-size 0
             listen-address xxx.xxx.40.1
             name-server xxx.xxx.0.1 {
             }
             name-server xxx.xxx.1.1 {
             }
             name-server xxx.xxx.4.4 {
             }
             name-server xxx.xxx.8.8 {
             }
             name-server xxxx:xxxx:4860::8844 {
             }
             name-server xxxx:xxxx:4860::8888 {
             }
             name-server xxxx:xxxx:4700::1001 {
             }
             name-server xxxx:xxxx:4700::1111 {
             }
         }
     }
     ntp {
         allow-client xxxxxx
             address xxx.xxx.0.0/0
             address ::/0
         }
         server xxxxx.tld {
         }
         server xxxxx.tld {
         }
         server xxxxx.tld {
         }
     }
     router-advert {
         interface eth0 {
             name-server xxxx:xxxx:4860::8888
             name-server xxxx:xxxx:4860::8844
             prefix ::/64 {
                 valid-lifetime 172800
             }
         }
     }
     ssh {
         port 22
     }
 }
 system {
     config-management {
         commit-revisions 100
     }
     conntrack {
         modules {
             ftp
             h323
             nfs
             pptp
             sip
             sqlnet
             tftp
         }
     }
     console {
         device ttyS0 {
             speed 115200
         }
     }
     host-name xxxxxx
     login {
         user xxxxxx {
             authentication {
                 encrypted-password xxxxxx
                 plaintext-password xxxxxx
             }
         }
     }
     name-server xxx.xxx.40.1
     syslog {
         global {
             facility all {
                 level info
             }
             facility local7 {
                 level debug
             }
         }
     }
 }
10

“It’s still not clear to me what 'set service router-advert <interface> name-server <address>' should be”
is it necessary to specify this name server if yes then what happens when i use adguard home for dns ,since it has upstream dns configured on it does it change anything???

this post has pushed me to also try ipv6 on my pppoe connection @byb also please explain your lan setup for ipv6 since it seems no ipv6 on ur lan i have basic connection eth0 wan and eth1 lan

11

This is the address of the name server that is advertised using router advertisements, which causes systems to use it as their default name server and you should probably configure it. If you have some kind of local DNS resolver, then you can use its address instead of a public resolver. The upstream DNS servers of such a local server are usually unaffected by this configuration, unless it also uses the DNS servers of the host it’s running on, which wouldn’t make much sense.

In their setup, IPv6 is delegated from the PPPoE interface to the LAN (eth0) interface using DHCP-PD. This causes the interface to get an address, and combined with the router-advert config, the prefix of that address is advertised to the network, which in turn causes other devices on that network to be able to get an IPv6 address.

PS: Please try to phrase and punctuate your sentences at least a little better, it’s really hard to read like this. With tools like ChatGPT and DeepL Write freely available, there is no excuse to post a chain of sentences with no punctuation and no capitalization :wink:

2 Likes
13

This post helped me configure ipv6 now only firewall rules i am not familiar