Edit: This post has been a learning experience for me, the original question here is not possible as I understand it now. So for anyone reading this, don’t let my ignorance confuse you.
version 1.3.2
Router is doing eBGP only, no OSPF. 3 BGP neighbors, 2 upstream, 1 downstream. There is a single /28 subnet from downstream that I would like to make use upstream 2 and all other traffic use upstream 1.
I can’t seem to make anything happen here. It’s like the router just ignores the setup, so I’m wondering if I have something wrong. I will post the sanitized config.
In your configuration, you import the prefix 12.42.xx.96/28 from 10.10.6.9 with a weight of 500.
It is not clear if it is upstream/downstream.
If you want that source of some /28 prefix will use ISP 2 and not use ISP1 for outbound direction, you have to use “PBR/policy route/load-balancing”. From your question, it is not clear if is it only for outbound or inbound (or both) directions.
yes sorry, 10.10.6.9 is the upstream I want the /28 to find it’s default route, whereas the other upstream 10.10.12.17 is where I want all the other traffic to go. 10.10.21.2 is the downstream tower where the /28 is originating from
This may be showing my ignorance or just my lack of clarity of mind, but is there not a way to use prefix-list and route-maps to effectively weight the upstream for a particular subnet?
Ultimately in the long run, I would like to use bgp communities to traffic engineer the routes on our network. I haven’t got the network as a whole to that point yet.
Our network is a full eBGP mesh where each tower router has it’s own ASN, there is no IGP like OSPF. So this case is just a random tower in the middle of our network where I want to split traffic from the downstream router and make it use 2 different default routes from upstream depending on the source subnet. The default routes are multiple hops away from this router.
From what I am understanding from your description, you are indeed looking for Policy Based Routing. Weight in BGP is only locally significant to the router it’s set on, that property is not sent to other bgp routers, and is only used to pick a route when you have 2 or more equivalent routes. It’s also going to affect all traffic to/from that prefix on that router without using PBR and is really only going to affect inbound and internal traffic routing since.you can’t announce a /28 to the Internet.
Or set the customers default route statically to the desired edge router and give the customer router a static route to the /32 gateway of the edge router of the router directly connected to the customer. So if your edge router is 10.0.0.1 and the p-node is 10.1.0.1 you’d have:
set protocols static route 0.0.0.0/0 next-hop 10.0.0.1
set protocols static route 10.0.0.1/32 next-hop 10.1.0.1
Then the customer traffic would take the default route on your edge router to the desired upstream, but so will all other traffic exiting that edge node. So this will only work if you have an edge router dedicated to that desired upstream with no other default routes.
@zbiles@Viacheslav Thanks for jumping on here. I wasn’t clear enough in my explanation. It was kindly pointed out to me in other communication that I have a fundamental misunderstanding of what I was trying to do here.
@zbiles yes, I think PBR could fix this here, but I have found PBR not dynamic enough. I detest static routing of just about any sort for the same reason.
What I believe the correct solution is, would be to tag the subnet with a bgp community at the downstream tower where it is originated and then on the edge router where I am injecting the default routes I can use as prepend to favor one edge over another for that specific subnet.
In the provided image, the T6 tower has 2 different subnets. I want one to use Edge1 and the other to use Edge2. As I mentioned this is strictly eBGP, full mesh, no IGP or RR’s
I don’t see any way to do what you are trying to do with BGP unless your routes are in different VRFs. If you prepend the default route from the edge router that’s going to influence every single downstream BGP router and all the traffic on them. Essentially this would shift all network traffic out the edge route with the shorter AS path. You could potentially shift all traffic from T6 using this method, but it sounds like you just want a single subnet, which isn’t going to work unless your upstream will accept the /28 network? If that’s connecting to somethind like DIA, /24 is usually the minimum size prefix a provider will allow to be announced. Not knowing how your peering is arranged I’m making some assumptions here.
If you can indeed anounce the /28 to your upstream you can prepend the /28 or completely withdraw that from the non-preferred path, then set a community on the preferred default route anouncement on edge 1, match that in a route-map on t6 to set the weight to prefer that default route, but again it’s going to affect all traffic on T6, not just the single /28 subnet. So doesn’t really accomplish your goal but I think that’s as close as you can get just using routing to direct traffic.
Your solution of using communities will only influence inbound traffic from the internet, not outbound traffic, that will still flow to whichever edge router (default route) has the lower BGP metric. Unless that’s all you need to accomplish?
@zbiles Sorry if I am not making this clear, but this is all on-net. Yes, in public space, a /24 is minimum but this is on-net inside our Wisp, we run eBGP for our internal routing protocol. Each router at each tower has it’s own private ASN.
I get that, but your edge 1 and edge 2 are connected to an upstream provider correct?
Those are the only 2 places you can influence inbound routing. I’m assuming you don’t want assymetric routing?
This is actually fixing an asymmetric routing situation right now, as a side note to the real problem.
Yes edge 1 and 2 are peered with an upstream provider.
Well asymmetric routing only a “problem” if you have stateful devices in the path. So it’s not necessarily bad, but I still can’t see a way to get what you are looking to do with with BGP only when it’s a single /28 prefix. Assuming you’re not doing MPLS either?
I think the closest you could get would be to prepend or withdraw the /24 that containns the /28 from the non-preferred path, and then throw weight on the default route at T6, but then that would likely cause asymmetric traffic on the other traffic on T6 which probably isn’t desirable. That’s where PBR is really the solution unless I’m still missing something?
With BGP you can’t just set the default route for the /28 prefix and nothing else on T6 unless you put the /28 subnet on an interface in a separate VRF, then set the default route for that VRF to go out edge 1 by peering a separate BGP instance in the VRF to edge 1, but then you’d still have the inbound issue where your announcement to the ISP is going to be /24 or larger and so you’d be influencing inbound routing for that whole /24, not just the /28.
It also looks like edge1 and edge2 aren’t directly connected? That could possibly open up some options if they were and you could direct routing between those after it enters inbound from the ISP before traversing your internal network.
Am I understanding you to say that on an edge router, Edge 1 lets say, where I am originating a default route that I cannot advertise that default route to downstream peers with AS prepending for some source IP traffic and no AS pending for other different source IP traffic?
@zbiles In this example online, it’s using mikrotik routers but vyos should be just as capable. It appears they are doing very similar to what I am trying to do. Forcing traffic on different paths on-net using ebgp communities. Unfortunately I don’t see in the article where the actual weight is applied
The example I linked here, is very similar to what I was hoping to achieve albeit in reverse. I’m trying to traffic engineer upload traffic instead of download traffic. And the subtle thing that I was missing is that the listed example is treating originated subnets as a whole, 100% down one path or another, whereas what I was trying to do, again in a reverse scenario, is treat the default route differently based on live traffic source. Which is not possible without PBR I believe, as you mentioned earlier today.
I think what I tend to lose focus on is that BGP doesn’t directly influence live traffic. It only sets up the routes in the routing table, than then influence live traffic to go where it does.
Obivously making a few assumptions because it’s a powerpoint, but a couple of key differences here; in this example I believe it’s assumed that 10.10.0.0/23 is the entire supernet for the twr-3 site. A more equivalent example if they were trying to just send 10.10.0.0/28 over the southern path, but the rest of the /23 supernet over the northern path.
They also only have a single egress so you don’t have inbound influencing to worry about like you do in your case since you can’t prepend just the /28. In order for that design to do what you want you have to control the connection on both ends, which you don’t since it’s going to the internet. They are also using OSPF to influence the load balancing on top of BGP as well which is something you don’t have either.
