LACP with Cisco

Has anyone had experience bonding with a Cisco switch? I’m using a pair of stacked 3750G-48 switches and am having some difficulty. The bonds come up but don’t pass certain traffic. From inside the network I can get out to any one of a number of websites, etc. From outside the network I can SSH into various devices as well as connect to internal websites. I also have several IPsec tunnels on Vyos that are up. However, IP phones cannot register to a softswitch that is inside. Or if they register, I RTP packets won’t traverse the network.

If I connect just a single interface to the switch with vif’s on that interface, everything works perfectly.

Here’s my configuration:

Vyos:

interfaces {
bonding bond0 {
hash-policy layer2
mode 802.3ad
mtu 9000
vif 10 {
address x.x.x.x/27
description VOIP-PUBLIC
vrrp {
vrrp-group 1 {
advertise-interval 1
preempt true
virtual-address x.x.x.x
}
}
}
vif 20 {
address 100.64.20.62/26
description san
vrrp {
vrrp-group 1 {
advertise-interval 1
preempt true
virtual-address 100.64.20.1
}
}
}

{{snip}}

ethernet eth3 {
    bond-group bond0
    description "Left Ethernet - switch port g1/0/1"
    duplex auto
    hw-id a0:36:9f:02:63:32
    mtu 9000
    smp_affinity auto
    speed auto
}
ethernet eth4 {
    bond-group bond0
    description "Center Left Ethernet - switch port g2/0/1"
    duplex auto
    hw-id a0:36:9f:02:63:33
    mtu 9000
    smp_affinity auto
    speed auto
}

On the Cisco:

interface Port-channel10
switchport trunk encapsulation dot1q
switchport mode trunk

interface GigabitEthernet1/0/1
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 10 mode active
.
.
.
interface GigabitEthernet2/0/1
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 10 mode active
!

Any insights would be appreciated.

I have almost an identical setup, with pretty much the exact configuration (except I don’t have mtu 9000 set). Unfortunately, I can’t help troubleshoot very much as this setup is still using Vyatta VC6.2-2011.02.09, so it’s not quite the same. I actually don’t think it’s a problem with the LACP links. I would expect there to be widespread failure if they weren’t working properly, this feels like a layer 3 problem but I have no proof. I would try shut on one of the the cisco ports, and see if the behavior persists. If it goes away I think we can say that it’s something wrong with the agg ports.

I have the same result whether using Vyatta 6.5 or Vyos 1.1.5. As to it being a layer 3 issue, from inside the network I can visit websites, get my mail, ssh to various devices and that always works. From outside the network, everything seems to work except RTP and (sometimes) SIP registrations.

I’ll try disconnecting one of the ports and see what that does.

Shutting down a port didn’t help. I’m really at a loss here as it works fine for everything (as far as I’ve seen) except RTP. Any other suggestions?

What version of IOS are you running on the 3750Gs?

Sorry I don’t have any sort of RTP or SIP experience and not in use in this environment.

Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(35)SE5, RELEASE SOFTWARE (fc1)
System image file is “flash:c3750-ipbase-mz.122-35.SE5/c3750-ipbase-mz.122-35.SE5.bin”

Switch   Ports  Model              SW Version              SW Image            
------   -----  -----              ----------              ----------          
*    1   52     WS-C3750G-48TS     12.2(35)SE5             C3750-IPBASE-M      
     2   52     WS-C3750G-48TS     12.2(35)SE5             C3750-IPBASE-M   

interface Port-channel3
 description rtr01-bond0
 switchport trunk encapsulation dot1q
 switchport mode trunk

interface GigabitEthernet1/0/47
 description rtr01-eth0
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-protocol lacp
 channel-group 3 mode active

interface GigabitEthernet2/0/47
 description rtr01-eth1
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-protocol lacp
 channel-group 3 mode active

I’m willing to try anything at this point… This is truly baffling.

Are you using firewalls in any relevant places?

What is the softswitch? If you are sure that layer3 is working (firewall,routing,nat, etc.) then I might look at layer2. Pull out wireshark/tshark/tcpdump and try to see the layer 2 traffic flowing in and out of the switch, the vyos router, and the softswitch. I think LACP uses virtual mac addresses to present channel-groups to switch and host. Maybe there is something there that the softswitch isn’t sending/receiving the frames because of mac address/ARP strangeness. What’s the topology? are both the softswitch and the vyos router plugged into the cisco switchstack? You could try taking VRRP out of the equation to eliminate that. You may need a transparent bridge and some frame/packet traces to figure this one out.

No firewalls at all. Traffic takes the same path if I eliminate LACP and just use VIFs and VRRP on an ethernet interface. In that instance, everything works fine. Only when I go to a bond on Vyos and a port channel on the Cisco do I not pass RTP.


The softswitch lives on a Xenserver and as stated above, all of this works fine when using a straight ethernet interface with VIFs. Only when I go to a bonded interface with a port channel on the switch do I run into trouble.

Everything (Vyos, softswitch, etc.) is connected to the 3750s.


As an aside, I have 2 SANs running Ubuntu 14.04.2 and they connect to the switches with LACP with no problems, although they aren’t running RTP :slight_smile:

let’s take a look at some packet captures between the soft switch and the cisco switch and between the cisco switch and the vyos router