Hello,
I am experiencing a seemingly random failure of port forwarding back to a Windows Server 2016 Read Only Domain Controller for external LDAP connections. External connections to port 389 and 636 will simply stop responding for anywhere from 5 - 45 minutes. Sometimes just port 389 starts responding again, sometimes both 389 and 636. They will respond to queries for a seemingly random amount of time before becoming unreachable again. This appears to affect both TCP and UDP traffic. Sometimes, I am unable to get HTTP or HTTPS responses from the other servers we expose to the outside world when this is happening as well.
To monitor the services, I have a VM external to our network running Icinga 2 and using check_tcp to check the connectivity of 389/tcp and 636/tcp. I am also using curl and ldapsearch on another VM with a different provider to eliminate the possibility of this being a carrier issue.
Things I’ve tried:
- Increasing the arp table size to 32768
- Setting up a 1-to-1 NAT with 172.16.0.21
- Break down a single DNAT rule with “protocol tcp_udp” in to 4 separate DNAT rules declaring the port and tcp or udp
- Lowered MTU on the OUTSIDE interface down to 1472, then back to 1500
- Enabled “generic-segmentation” and “scatter-gather” offloading options on both INSIDE and OUTSIDE interfaces
- Created a firewall rule to allow fragments but I did not see any hits so I deleted the rule
- Disabled flow control on OUTSIDE and INSIDE interfaces
- Created pseudo-ethernet interfaces for each external address instead of having them all on eth3
I am coming from a Cisco ASA 5525-X and did not have this issue on that platform. However, I understand that the ASA sets up a direct 1-to-1 NAT/BiNAT which is different to how VyOS handles it.
Our hardware is a Dell R630 with 64gb of ram. Dual Xeon E5-2670 v3, hyperthreading is disabled. We are currently using the two copper ethernet ports for inside and outside.
vyos@vyos:~$ sh ver
Version: VyOS 1.2.6
Release Train: crux
Built by: Sentrium S.L.
Built on: Fri 11 Sep 2020 11:54 UTC
Build UUID: bb54bb8d-8c25-4395-bbe4-625798facebd
Build Commit ID: 706d01f247bb83
Architecture: x86_64
Boot via: installed image
System type: bare metal
Hardware vendor: Dell Inc.
Hardware model: PowerEdge R630
Hardware S/N: 25P3GB2
Hardware UUID: 4c4c4544-0035-5010-8033-b2c04f474232
Copyright: VyOS maintainers and contributors
DNAT counters after about 15 hours-
The 400 range is the LDAP. The counters on 401 seem very high to me.
vyos@vyos:~$ show nat destination stat
rule pkts bytes interface
---- ---- ----- ---------
100 92224 8180 eth3
200 92242 8173 eth3
300 225 13216 eth3
400 1520 91116 eth3
401 3948K 265M eth3
402 0 0 eth3
403 1734 102K eth3
500 885 44504 eth3
600 7168 391K eth3
700 434 20753 eth3
800 192 8416 eth3
850 16 728 eth3
851 16 720 eth3
852 10 400 eth3
900 1451 79004 eth3
Config-
vyos@vyos:~$ show conf
firewall {
all-ping enable
broadcast-ping disable
config-trap disable
group {
address-group ACCESSCONTROL-INTERNAL {
...
}
address-group FORMS-INTERNAL {
address 172.16.1.230
}
address-group HELPDESK-INTERNAL {
address 10.80.0.40
}
address-group HVAC-INTERNAL {
address 10.90.0.10
}
address-group LDAP-INTERNAL {
address 172.16.0.21
}
address-group MOODLE-INTERNAL {
address 10.80.0.30
}
address-group NS02-INTERNAL {
address 10.80.2.11
}
address-group NS03-INTERNAL {
address 10.80.2.12
}
address-group SECURITYDVR-INTERNAL {
...
}
address-group VPN-INTERNAL {
address 10.80.3.20
description "OpenVPN Proxy VRRP"
}
network-group HVAC-VENDOR {
...
}
network-group SECURITY-VENDOR {
...
}
network-group SHUNNED-RANGES {
description "Bad actors"
network 85.93.20.0/24
...
}
port-group ACCESSCONTROL-PORTS-TCP {
port https
}
port-group FORMS-PORTS-TCP {
port http
port https
}
port-group HELPDESK-PORTS-TCP {
port http
port https
}
port-group HVAC-PORTS-TCPUDP {
...
}
port-group LDAP-PORTS-TCPUDP {
port 389
port 636
}
port-group MOODLE-PORTS-TCP {
port http
port https
}
port-group NS02-PORTS-TCPUDP {
port domain
}
port-group NS03-PORTS-TCPUDP {
port domain
}
port-group SECURITYDVR-PORTS-TCPUDP {
...
}
port-group VPN-PORTS-TCP {
...
}
}
ipv6-receive-redirects disable
ipv6-src-route disable
ip-src-route disable
log-martians disable
name OUTSIDE-IN {
default-action drop
rule 1 {
action accept
description "Allow established/related"
state {
established enable
related enable
}
}
rule 2 {
action drop
description "Drop invalid"
state {
invalid enable
}
}
rule 10 {
action drop
description "Drop bad actors"
source {
group {
network-group SHUNNED-RANGES
}
}
}
rule 100 {
action accept
description "NS02"
destination {
group {
address-group NS02-INTERNAL
port-group NS02-PORTS-TCPUDP
}
}
protocol tcp_udp
state {
new enable
}
}
rule 200 {
action accept
description "NS03"
destination {
group {
address-group NS03-INTERNAL
port-group NS03-PORTS-TCPUDP
}
}
protocol tcp_udp
state {
new enable
}
}
rule 300 {
action accept
description "VPN Proxy VRRP"
destination {
group {
address-group VPN-INTERNAL
port-group VPN-PORTS-TCP
}
}
protocol tcp
state {
new enable
}
}
rule 400 {
action accept
description "LDAP - 76.191.126.251"
destination {
group {
address-group LDAP-INTERNAL
port-group LDAP-PORTS-TCPUDP
}
}
protocol tcp_udp
state {
new enable
}
}
rule 500 {
action accept
description "Forms"
destination {
group {
address-group FORMS-INTERNAL
port-group FORMS-PORTS-TCP
}
}
protocol tcp
state {
new enable
}
}
rule 600 {
action accept
description "Moodle"
destination {
group {
address-group MOODLE-INTERNAL
port-group MOODLE-PORTS-TCP
}
}
protocol tcp
state {
new enable
}
}
rule 700 {
action accept
description "HVAC"
...
}
rule 800 {
action accept
description "Access Control appliance"
...
}
rule 850 {
action accept
description "Security DVRs"
...
}
rule 900 {
action accept
description "HelpDesk"
destination {
group {
address-group HELPDESK-INTERNAL
port-group HELPDESK-PORTS-TCP
}
}
protocol tcp
state {
new enable
}
}
}
name OUTSIDE-LOCAL {
default-action drop
rule 1 {
action accept
description "Allow established/related"
state {
established enable
related enable
}
}
rule 2 {
action drop
description "Drop invalid"
state {
invalid enable
}
}
rule 10 {
action accept
description "Respond to ICMP echo request"
icmp {
type-name echo-request
}
protocol icmp
state {
new enable
}
}
}
receive-redirects disable
send-redirects enable
source-validation disable
syn-cookies enable
twa-hazards-protection disable
}
interfaces {
ethernet eth0 {
duplex auto
hw-id 24:6e:96:00:f3:c0
smp-affinity auto
speed auto
}
ethernet eth1 {
duplex auto
hw-id 24:6e:96:00:f3:c2
smp-affinity auto
speed auto
}
ethernet eth2 {
address 10.99.0.1/29
description INSIDE-COPPER
disable-flow-control
duplex auto
hw-id 24:6e:96:00:f3:e0
offload-options {
generic-segmentation on
scatter-gather on
}
smp-affinity auto
speed auto
}
ethernet eth3 {
address 76.191.126.242/28
address 76.191.126.243/28
address 76.191.126.244/28
address 76.191.126.245/28
address 76.191.126.246/28
address 76.191.126.247/28
address 76.191.126.248/28
address 76.191.126.249/28
address 76.191.126.250/28
address 76.191.126.251/28
address 76.191.126.252/28
address 76.191.126.253/28
address 76.191.126.254/28
address x.x.142.162/28
address x.x.142.163/28
address x.x.142.164/28
address x.x.142.165/28
address x.x.142.166/28
address x.x.142.167/28
address x.x.142.168/28
address x.x.142.169/28
address x.x.142.170/28
address x.x.142.171/28
address x.x.142.172/28
address x.x.142.173/28
address x.x.142.174/28
description OUTSIDE-COPPER
disable-flow-control
duplex full
firewall {
in {
name OUTSIDE-IN
}
local {
name OUTSIDE-LOCAL
}
}
hw-id 24:6e:96:00:f3:e1
mtu 1500
offload-options {
generic-segmentation on
scatter-gather on
}
smp-affinity auto
speed 1000
}
loopback lo {
}
}
nat {
destination {
rule 100 {
description "NS02 - TCP/UDP - domain"
destination {
address 76.191.126.243
port 53
}
inbound-interface eth3
protocol tcp_udp
translation {
address 10.80.2.11
}
}
rule 200 {
description "NS03 - TCP/UDP - domain"
destination {
address 76.191.126.244
port 53
}
inbound-interface eth3
protocol tcp_udp
translation {
address 10.80.2.12
}
}
rule 300 {
description "VPN - TCP - OpenVPN ports"
...
}
rule 400 {
description "LDAP - TCP/UDP - ldap/ldaps"
destination {
address 76.191.126.251
port 389
}
inbound-interface eth3
protocol tcp
translation {
address 172.16.0.21
}
}
rule 401 {
destination {
address 76.191.126.251
port 389
}
inbound-interface eth3
protocol udp
translation {
address 172.16.0.21
}
}
rule 402 {
destination {
address 76.191.126.251
port 636
}
inbound-interface eth3
protocol udp
translation {
address 172.16.0.21
}
}
rule 403 {
destination {
address 76.191.126.251
port 636
}
inbound-interface eth3
protocol tcp
translation {
address 172.16.0.21
}
}
rule 500 {
description "Forms - TCP - http/https"
destination {
address 76.191.126.252
port 80,443
}
inbound-interface eth3
protocol tcp
translation {
address 172.16.1.230
}
}
rule 600 {
description "Moodle - TCP - http/https"
destination {
address 76.191.126.248
port 80,443
}
inbound-interface eth3
protocol tcp
translation {
address 10.80.0.30
}
}
rule 700 {
description "CH HVAC - TCP/UDP - http/https/custom"
...
}
rule 800 {
description "Access Control - TCP - https"
...
}
rule 850 {
description "Security DVR - CH - TCP"
...
}
rule 851 {
description "Security DVR - TP - TCP "
...
}
rule 852 {
description "Security DVR - MK - TCP"
...
}
rule 900 {
description "HelpDesk - TCP - http/https"
destination {
address 76.191.126.246
port 80,443
}
inbound-interface eth3
protocol tcp
translation {
address 10.80.0.40
}
}
}
source {
rule 5 {
outbound-interface eth3
source {
address 10.80.2.11
}
translation {
address 76.191.126.243
}
}
rule 6 {
outbound-interface eth3
source {
address 10.80.2.12
}
translation {
address 76.191.126.244
}
}
rule 7 {
outbound-interface eth3
source {
address 172.16.0.51
}
translation {
address 76.191.126.253
}
}
rule 8 {
outbound-interface eth3
source {
address 172.17.0.50
}
translation {
address 76.191.126.253
}
}
rule 9 {
outbound-interface eth3
source {
address 172.18.0.50
}
translation {
address 76.191.126.253
}
}
rule 11 {
outbound-interface eth3
source {
address 10.80.3.0/24
}
translation {
address 76.191.126.247
}
}
rule 13 {
outbound-interface eth3
source {
address 172.16.0.28
}
translation {
address 76.191.126.251
}
}
rule 14 {
outbound-interface eth3
source {
address 172.16.0.21
}
translation {
address 76.191.126.251
}
}
rule 15 {
outbound-interface eth3
source {
address 10.80.0.30
}
translation {
address 76.191.126.248
}
}
rule 100 {
description "Masquerade outgoing"
outbound-interface eth2
translation {
address 76.191.126.242
}
}
rule 200 {
description "Masquerade incoming"
outbound-interface eth3
translation {
address masquerade
}
}
}
}
protocols {
static {
route 0.0.0.0/0 {
next-hop 76.191.126.241 {
}
}
route 10.0.0.0/10 {
next-hop 10.99.0.3 {
}
}
route 10.65.0.0/24 {
next-hop 10.99.0.3 {
}
}
route 10.80.0.0/16 {
next-hop 10.99.0.3 {
}
}
route 10.90.0.0/24 {
next-hop 10.99.0.3 {
}
}
route 10.91.0.0/24 {
next-hop 10.99.0.3 {
}
}
route 10.92.0.0/24 {
next-hop 10.99.0.3 {
}
}
route 10.97.0.0/29 {
next-hop 10.99.0.3 {
}
}
route 10.100.0.0/16 {
next-hop 10.99.0.3 {
}
}
route 10.101.0.0/16 {
next-hop 10.99.0.3 {
}
}
route 10.102.0.0/16 {
next-hop 10.99.0.3 {
}
}
route 172.16.0.0/12 {
next-hop 10.99.0.3 {
}
}
route 172.16.0.0/16 {
next-hop 10.99.0.3 {
}
}
}
}
service {
lldp {
interface eth2 {
}
}
snmp {
community public {
authorization ro
network 172.16.0.0/16
network 172.17.0.0/16
network 172.18.0.0/16
}
}
ssh {
port 22
}
}
system {
config-management {
commit-revisions 100
}
console {
device ttyS0 {
speed 115200
}
}
domain-name bellevuechristian.org
domain-search {
domain bellevuechristian.org
}
host-name vyos
ip {
arp {
table-size 32768
}
}
login {
user vyos {
authentication {
encrypted-password ****************
plaintext-password ****************
}
level admin
}
}
name-server 172.16.0.21
name-server 172.16.0.20
name-server 172.16.0.28
ntp {
server 209.51.161.238 {
}
server 216.218.192.202 {
}
server 216.218.254.202 {
}
}
syslog {
global {
facility all {
level info
}
facility protocols {
level debug
}
}
host 172.16.0.109 {
facility all {
level err
}
}
}
time-zone America/Los_Angeles
}
Here is a very basic diagram of my critical path-
