Hello Guys,
recently i had a look at the following Project https://github.com/wangyu-/tinyfecVPN
The tinyfecvpn aims to eliminate packet loss by introducing Foward Error Correction.
I want tinyfecvpn to be in vyos as a new tunnel type but i do not know where to start developement.
What i do know is that there need to be interface definitions for the cli, a service template, log-integration and possibly more.
Can you point me where to get started and what i should keep in mind ?
Note: I didn`t put this in Feature Requests because i first want to try develop this feature. If this is wrong or unwise please tell me:)
Jokes aside… what Im currently missing (please inform me if this already exists) with VyOS (and similar) is something that:
Can enable/disable FEC (forward error correction - good to avoid resending packets for bad links) along with tweakable setting for how much should be FEC and how much should be payload.
Can enable/disable efficient encryption to the packets between the routers (such as wireguard who can utilize multicore and AES-NI (AES128 and AES256 among others) - using both passphrase and x509 for authentication). This encryption must be NAT-traversal compatible (UDP?).
Multipath for redundancy (same packet is duplicated over multiple physical paths - on receiving router the duplicates are thrown away so the hostB gets a clean non-duplicated stream).
Multipath for linkaggregation (a single TCP/UDP stream is divided over multiple physical paths - on receiving router these paths are deaggregated and whats forwarded to the host is a clean stream where the packets are in order (just as the host at sending site sent these). Example 10Gbps hostA have a single TCP/UDP stream which is shared over 10x1Gbps physical links and on receiving end deaggregated again into a single 10Gbps stream towards hostB. The loadsharing algo must be able to do round-robin but also various weighting. Like if you got 1Gbps + 500Mbps links then the 1Gbps link should get 2/3 of the total load.
Virtual reassemble, sending hostA is sending lets say MTU 2000 bytes. This is split into 1500 + 500 reminder (or whatever MTU the physical paths between these sites have) but on receiving router reassembled so hostA sent a 2000 byte packet and hostB received a 2000 byte packet.
Network drawing:
HostA ↔ VyoSA ↔ various physical paths and networks ↔ VyoSB ↔ HostB
Hi @Apachez ,
I`ll try to answer your questions.
First of all i am not aware of any current implementations in Vyos, but similar solutions exist with SD-WAN Providers like Huawei, Fortinet oder Peplink and possibly more
What Tinyfecvpn can/cannot do:
Can enable/disable FEC (forward error correction - good to avoid resending packets for bad links) along with tweakable setting for how much should be FEC and how much should be payload.
Sure but not directly a percentage, as Fec is applied by numbers of packets e.g. 20 Packets payload an 10 packets fec for around 30% Loss tolerance but that is variable
Can enable/disable efficient encryption to the packets between the routers (such as wireguard who can utilize multicore and AES-NI (AES128 and AES256 among others) - using both passphrase and x509 for authentication). This encryption must be NAT-traversal compatible (UDP?).
It has XOR encryption which i wouldn’t consider encryption, but it wraps all traffic in UDP
There is no multipathing integrated though after working on a tinyfecvpn implementation i will work on something involving glorytun, which will do multipath aggregation.
Virtual reassemble, sending hostA is sending lets say MTU 2000 bytes. This is split into 1500 + 500 reminder (or whatever MTU the physical paths between these sites have) but on receiving router reassembled so hostA sent a 2000 byte packet and hostB received a 2000 byte packet.
Since the Fec Engine has to chop and buffer packets anyway this is very much possible and one of my favorite features.
The one problem with FEC is that it is very compute intensive, so you are not pushing 1G through a single tunnel. Max i got was 400 Mbit payload on a E5 2686v4 with nearly no fec and some optimisations mind you this application is single threaded. Around 200-300 are realistic with with an older-ish CPU Core(Intel 6th gen)
BR
Niklas
When it comes to encryption perhaps it can be wrapped around wireguard (that is either first encrypt and then push through the FEC-engine or do FEC first and then encrypt the result) which overall seems promising today (compared to ipsec and others)?
The primary usecase for the features I listed is to deal with devices out in the field (think using various cellular providers as uplinks incl satlinks). Which gives that there is currently no need of +10Gbps throughput for such tunneling (hostA ↔ hostB) but of course it should be as efficient as possible (for example be able to utilize multicore etc) so if it can get close to 10G or more I wouldnt say no
I do know there exist various commercial vendors who does this but they are just that commercial AND expensive. I would very much like to see the listed features show up in VyOS (and by that also be able to select the hardware I want for various reasons example due to COO (Country of Origin))
I do know that multipath for example do exist natively in the Linux kernel nowadays. And we have wireguard for the encryption part. So its about the FEC stuff that perhaps is missing and then something that can combine them all according to my wishlist.
These(tinyfecvpn/udp2raw/udp-speed-up) programming or technologies send more packets to jam the traffic, The performance or bandwidth is very poor. The best practice is to find a good network connection without any packet loss. Then build vpn connection with Wireguard.
I fully agree. When possible skip FEC and such, but there are valid usecases like the ones @Apachez mentioned. For me the usecase is something similar. I need to tunnel voice traffic through an unreliable ADSL line and i found fec to be an effective solution to this.
But lets get back to the topic on how to implement this
I know how to parameter tinyfecvpn to do as i like, but i need some guidance on how to integrate into vyos.
Thats not how real life works, specially when you are out in the field and not sitting in a nice datacenter.
Shaky connections are not uncommon and for those FEC will sacrifice some bandwidth and cpu processing with the return that even a broken or missing packet can be reconstructed of what actually got through this particular link. Without FEC (forward error correction) the packet is either entirely lost or if arrived but broken it must be resent which only works for TCP traffic (unless the application will resolve this itself).
I am fully aware of the manual option, but this is not what i intend to do. I want to develop it into vyos, so no messing around with the linux CLI. I want it to be integrated into vyos so you could possibly do something like
set interface tunnel tunX encapsulation tinyfec
set interface tunnel tunX parameters fec xxxx
just using the Linux below does not satisfy me or my collegues who are network engineers but not linux engineers. A consistent cli for creating tunnels including tinyfec is what i am seeking.
The the purpose of this thread is not to ask if tinyfec should be integrated or what fec aims to improve, as this is no feature request. I aim to gather information regarding the development proceedings on how i should go about integrating the feature like this for my use.
Which is like one of the main purposes by using VyOS.
You get a consistent frontend (the vyos config) which does stuff for you in the background without you having to learn the syntax of each and every software and their config files.
Regarding the actual naming of the commands I would probably vote for something like:
set interface tunnel tunX type tinyfec
set interface tunnel tunX parameters tinyfec xxxx
With the reasons that “encapsulation” already have wellknown usage through for example “udp encapsulation” - there is no such thing as a “tinyfec encapsulation”. Using “type” (or anything other than “encapsulation”) would be prefered.
And when it comes to parameters it would be nice if that would match the type name.
Okay,
i understand your concerns with encapsulation not being the right term used, but tinyfecvpn is essentially gre with fec(very oversimpified).
Another option would be to make it an entirely new interface type, as it is indeed different from existing things. Tinyfec also has different operating modes which would be represented better with it being a new type of interface.
Nerver mind. If really in a datacenter, There isn’t have a pppoe with packet loss wan connection.
I never use this like programme to build vpn connection. I perfer to wireguard.
Why not call you boss to buy a stable WAN connection? If your collegues are netwrok engineers, Dose they agree with you about the solution? Trust me, Stable wan connection is the foundation of networking. It just like you want to build a castle on sand with this solution, you never know What time it will falling. Plus if you build the solution, may somedays your ISP will banned you vpn connection since you are try to jam the traffic. It is not a good solution.
Of course when possible i’d buy the reliable wan connection, but in some cases you don’t have that option.
I’ll give you an example. We recently had a customer with a remote branch. The connectivity options were unreliable DSL, poor coverage LTE or sattelite internet. If economically possible we would of course have laied down fibre. What are our options then to connect our customer to a reliable WAN?
One economically friendly option is to use FEC which does not eliminate the problem, but makes the available connections usable for voip and multicast.
It is not in question if fec is able to help with some of these problems at a cost of bandwidth or compute. I’d much rather have 70 Mbit reliable connectivity instead of 150Mbit hit and miss.
My collegues are fully aware of these restrictions and would welcome a solution like this.
Last but not least why would an ISP block my VPN? The bandwidth with fec does not go over my subscribed plan. I can’t image why an ISP would care if i use fec or not. For them it would be exactly the same or maybe even more predictable because of no retransmissions. “Jamming the traffic” is clearly beside the point. It does not give me more bandwidth than is already payed for.
Blocking Fec traffic would also go against local jurisdiction which clearly mandates net neutrality, at least here in the EU.
Cheers
