Troubleshooting Random Timeouts

General questions
1

Would greatly appreciate help from this community on this strange issue I’m having with, specifically one client who has quite a few VPS (KVM) servers and bare metal servers. He’s the only one reporting this issue out of all other VLANs, but I’ve had NOC at my colo check and the troubleshooting results proved that something was halting at the VyOS level and I don’t know why. Everything looks normal (BGP entries, static routes).

NOC

Type escape sequence to abort.
Tracing the route to xxx.xxx.116.5

1 xxx.xxx.55.210 1 msec 0 msec 0 msec
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *

My trace from our edge router shows traffic destined for xxx.xxx.116.5 is reaching your BGP neighbor interface xxx.xxx.55.210, once it enters your environment it is not reaching its destination.

Problem Summary
Some of the following assigned IPs do not communicate with the “outside world”, according to customer. Please investigate. See attached. Routing is fine on our end. Gateway pings fine.

  • Happens on multiple random IPv4 subnets (random A/B/C classes too)
  • Only one client reporting this
  • We’re using 2 x VyOS routers (2nd = failover)
  • Generally, issue happens on random individual IPs assigned within that reported subnet

From client:

Below is the firewall configuration. All servers have the same settings.

firewall
firewall918×384 101 KB

Additional troubleshooting results provided by client

traceroute
traceroute1640×485 243 KB

above shows it halting once it hits our VyOS router.

curl

wget
wget1020×148 45.5 KB

Ping
Ping712×489 35 KB

2

Are you doing NAT or something else involving session/state? You might be exhausting your state table.

3

I’m not sure how the last network tech configured it, but he’s gone and I have to take over. What’s the best way to determine this?

4

show conntrack table ipv4

sudo conntrack -L [Shows the same as above just faster and unformatted]

sudo conntrack -S [Shows you some conntrack stats]

sudo cat /proc/sys/net/netfilter/nf_conntrack_max will show you the max number of conntrack entires configured.

5

Thank you.

sudo conntrack -L
https://pastebin.com/exkDZjH3

user@vy-pe1.io7.xxxxxxx.com# sudo conntrack -S
cpu=0           found=0 invalid=17498477 ignore=1034066070 insert=0 insert_failed=0 drop=0 early_drop=0 error=17493460 search_restart=19182 
cpu=1           found=0 invalid=29764 ignore=138356 insert=0 insert_failed=0 drop=0 early_drop=0 error=29764 search_restart=3 
cpu=2           found=0 invalid=17448761 ignore=729372084 insert=0 insert_failed=0 drop=0 early_drop=0 error=17444161 search_restart=19235 
cpu=3           found=0 invalid=29942 ignore=118189 insert=0 insert_failed=0 drop=0 early_drop=0 error=29942 search_restart=15 
cpu=4           found=0 invalid=17531213 ignore=663331847 insert=0 insert_failed=0 drop=0 early_drop=0 error=17526731 search_restart=19170 
cpu=5           found=0 invalid=38368 ignore=141304 insert=0 insert_failed=0 drop=0 early_drop=0 error=38368 search_restart=7 
cpu=6           found=0 invalid=17428880 ignore=823055153 insert=0 insert_failed=0 drop=0 early_drop=0 error=17424671 search_restart=19075 
cpu=7           found=0 invalid=48839 ignore=148694 insert=0 insert_failed=0 drop=0 early_drop=0 error=48839 search_restart=6 
cpu=8           found=0 invalid=17296701 ignore=4147443507 insert=0 insert_failed=0 drop=0 early_drop=0 error=17292814 search_restart=18887 
cpu=9           found=0 invalid=39608 ignore=181809 insert=0 insert_failed=0 drop=0 early_drop=0 error=39608 search_restart=9 
cpu=10          found=0 invalid=17554660 ignore=712055142 insert=0 insert_failed=0 drop=0 early_drop=0 error=17550559 search_restart=18855 
cpu=11          found=0 invalid=48101 ignore=180192 insert=0 insert_failed=0 drop=0 early_drop=0 error=48101 search_restart=18 
cpu=12          found=0 invalid=17374745 ignore=572014250 insert=0 insert_failed=0 drop=0 early_drop=0 error=17371247 search_restart=19091 
cpu=13          found=0 invalid=83776 ignore=196545 insert=0 insert_failed=0 drop=0 early_drop=0 error=83776 search_restart=13 
cpu=14          found=0 invalid=34363345 ignore=3102387812 insert=0 insert_failed=0 drop=0 early_drop=0 error=34357880 search_restart=31457 
cpu=15          found=0 invalid=53205 ignore=147055 insert=0 insert_failed=0 drop=0 early_drop=0 error=53205 search_restart=7 



user@vy-pe1.io7.xxxxxxx.com# sudo cat /proc/sys/net/netfilter/nf_conntrack_max
262144
6

Doesn’t look like a conntrack problem sorry, you’re using less than 300 conntrack entires of your 262,144 available.

7

Does anyone else possibly know?

8

Your traceroute shows only single hop responding. Looks like the outside world has no route back to you
So do a traceroute from the outside world, towards your public IPs . Maybe not all of your addresses are advertised.

9

A little more info. I have a /22 routed to a VPS (KVM).


■ Normal operating IP
xxx.xxx.116.2
xxx.xxx.116.3
xxx.xxx.116.4
xxx.xxx.116.7
xxx.xxx.116.9
xxx.xxx.116.12
xxx.xxx.116.13
xxx.xxx.116.14
xxx.xxx.116.18
xxx.xxx.116.19
xxx.xxx.116.20
xxx.xxx.116.21
xxx.xxx.116.23
xxx.xxx.116.25
xxx.xxx.116.26
xxx.xxx.116.28
xxx.xxx.116.29
xxx.xxx.116.30
xxx.xxx.116.31
xxx.xxx.116.33
xxx.xxx.116.34
xxx.xxx.116.37
xxx.xxx.116.38
xxx.xxx.116.42
xxx.xxx.116.43
xxx.xxx.116.45
xxx.xxx.116.49
xxx.xxx.116.51
xxx.xxx.116.52
xxx.xxx.116.53
xxx.xxx.116.54
xxx.xxx.116.55
xxx.xxx.116.57
xxx.xxx.116.59
xxx.xxx.116.60
xxx.xxx.116.62
xxx.xxx.116.64
xxx.xxx.116.66
xxx.xxx.116.67
xxx.xxx.116.69
xxx.xxx.116.70
xxx.xxx.116.71
xxx.xxx.116.77
xxx.xxx.116.80
xxx.xxx.116.81
xxx.xxx.116.82
xxx.xxx.116.83
xxx.xxx.116.84
xxx.xxx.116.85
xxx.xxx.116.87
xxx.xxx.116.91
xxx.xxx.116.93
xxx.xxx.116.94
xxx.xxx.116.98
xxx.xxx.116.100
xxx.xxx.116.102
xxx.xxx.116.106
xxx.xxx.116.108
xxx.xxx.116.110
xxx.xxx.116.113
xxx.xxx.116.114
xxx.xxx.116.115
xxx.xxx.116.117
xxx.xxx.116.118
xxx.xxx.116.121
xxx.xxx.116.124
xxx.xxx.116.125
xxx.xxx.116.127
xxx.xxx.116.129
xxx.xxx.116.135
xxx.xxx.116.140
xxx.xxx.116.142
xxx.xxx.116.143
xxx.xxx.116.144
xxx.xxx.116.151
xxx.xxx.116.157
xxx.xxx.116.161
xxx.xxx.116.163
xxx.xxx.116.165
xxx.xxx.116.166
xxx.xxx.116.173
xxx.xxx.116.177
xxx.xxx.116.179
xxx.xxx.116.180
xxx.xxx.116.181
xxx.xxx.116.184
xxx.xxx.116.186
xxx.xxx.116.188
xxx.xxx.116.190
xxx.xxx.116.191
xxx.xxx.116.195
xxx.xxx.116.198
xxx.xxx.116.201
xxx.xxx.116.206
xxx.xxx.116.207
xxx.xxx.116.211
xxx.xxx.116.216
xxx.xxx.116.218
xxx.xxx.116.219
xxx.xxx.116.221
xxx.xxx.116.222
xxx.xxx.116.223
xxx.xxx.116.226
xxx.xxx.116.227
xxx.xxx.116.230
xxx.xxx.116.231
xxx.xxx.116.234
xxx.xxx.116.235
xxx.xxx.116.236
xxx.xxx.116.237
xxx.xxx.116.243
xxx.xxx.116.244
xxx.xxx.116.248
xxx.xxx.116.250
xxx.xxx.116.251
xxx.xxx.116.253
xxx.xxx.117.32
xxx.xxx.117.94
xxx.xxx.117.95
xxx.xxx.117.121
xxx.xxx.117.215
xxx.xxx.118.4
xxx.xxx.118.8
xxx.xxx.118.10
xxx.xxx.118.23
xxx.xxx.118.25
xxx.xxx.118.31
xxx.xxx.118.39
xxx.xxx.118.63
xxx.xxx.118.68
xxx.xxx.118.74
xxx.xxx.118.75
xxx.xxx.118.82
xxx.xxx.118.101
xxx.xxx.118.105
xxx.xxx.118.112
xxx.xxx.118.115
xxx.xxx.118.121
xxx.xxx.118.123
xxx.xxx.118.124
xxx.xxx.118.128
xxx.xxx.118.129
xxx.xxx.118.132
xxx.xxx.118.136
xxx.xxx.118.138
xxx.xxx.118.140
xxx.xxx.118.141
xxx.xxx.118.144
xxx.xxx.118.145
xxx.xxx.118.146
xxx.xxx.118.148
xxx.xxx.118.150
xxx.xxx.118.151
xxx.xxx.118.152
xxx.xxx.118.154
xxx.xxx.118.156
xxx.xxx.118.160
xxx.xxx.118.161
xxx.xxx.118.168
xxx.xxx.118.169
xxx.xxx.118.170
xxx.xxx.118.174
xxx.xxx.118.177
xxx.xxx.118.179
xxx.xxx.118.181
xxx.xxx.118.183
xxx.xxx.118.184
xxx.xxx.118.185
xxx.xxx.118.195
xxx.xxx.118.203
xxx.xxx.118.204
xxx.xxx.118.208
xxx.xxx.118.210
xxx.xxx.118.211
xxx.xxx.118.212
xxx.xxx.118.215
xxx.xxx.118.218
xxx.xxx.118.222
xxx.xxx.118.224
xxx.xxx.118.226
xxx.xxx.118.228
xxx.xxx.118.229
xxx.xxx.118.232
xxx.xxx.118.237
xxx.xxx.118.241
xxx.xxx.118.242
xxx.xxx.118.243
xxx.xxx.118.246
xxx.xxx.118.247
xxx.xxx.118.248
xxx.xxx.118.249
xxx.xxx.118.250
xxx.xxx.118.253
xxx.xxx.119.2
xxx.xxx.119.49
xxx.xxx.119.56
xxx.xxx.119.108
xxx.xxx.119.144
xxx.xxx.119.145
xxx.xxx.119.150
xxx.xxx.119.221


■ IP whose communication cannot be verified
xxx.xxx.116.5	
xxx.xxx.116.6	
xxx.xxx.116.8	
xxx.xxx.116.10	
xxx.xxx.116.11	
xxx.xxx.116.15	
xxx.xxx.116.16	
xxx.xxx.116.17	
xxx.xxx.116.22	
xxx.xxx.116.24	
xxx.xxx.116.27	
xxx.xxx.116.32	
xxx.xxx.116.35	
xxx.xxx.116.36	
xxx.xxx.116.39	
xxx.xxx.116.40	
xxx.xxx.116.41	
xxx.xxx.116.44	
xxx.xxx.116.46	
xxx.xxx.116.47	
xxx.xxx.116.48	
xxx.xxx.116.50	
xxx.xxx.116.56	
xxx.xxx.116.58	
xxx.xxx.116.61	
xxx.xxx.116.63	
xxx.xxx.116.65	
xxx.xxx.116.68	
xxx.xxx.116.72	
xxx.xxx.116.73	
xxx.xxx.116.74	
xxx.xxx.116.75	
xxx.xxx.116.76	
xxx.xxx.116.78	
xxx.xxx.116.79	
xxx.xxx.116.86	
xxx.xxx.116.88	
xxx.xxx.116.89	
xxx.xxx.116.90	
xxx.xxx.116.92	
xxx.xxx.116.95	
xxx.xxx.116.96	
xxx.xxx.116.97	
xxx.xxx.116.99	
xxx.xxx.116.101	
xxx.xxx.116.103	
xxx.xxx.116.104	
xxx.xxx.116.105	
xxx.xxx.116.107	
xxx.xxx.116.109	
xxx.xxx.116.111	
xxx.xxx.116.112	
xxx.xxx.116.116	
xxx.xxx.116.119	
xxx.xxx.116.120	
xxx.xxx.116.122	
xxx.xxx.116.123	
xxx.xxx.116.126	
xxx.xxx.116.128	
xxx.xxx.116.130	
xxx.xxx.116.131	
xxx.xxx.116.133	
xxx.xxx.116.134	
xxx.xxx.116.136	
xxx.xxx.116.137	
xxx.xxx.116.138	
xxx.xxx.116.139	
xxx.xxx.116.141	
xxx.xxx.116.145	
xxx.xxx.116.146	
xxx.xxx.116.147	
xxx.xxx.116.148	
xxx.xxx.116.149	
xxx.xxx.116.150	
xxx.xxx.116.152	
xxx.xxx.116.153	
xxx.xxx.116.154	
xxx.xxx.116.155	
xxx.xxx.116.156	
xxx.xxx.116.158	
xxx.xxx.116.159	
xxx.xxx.116.160	
xxx.xxx.116.162	
xxx.xxx.116.164	
xxx.xxx.116.167	
xxx.xxx.116.168	
xxx.xxx.116.169	
xxx.xxx.116.170	
xxx.xxx.116.171	
xxx.xxx.116.172	
xxx.xxx.116.174	
xxx.xxx.116.175	
xxx.xxx.116.176	
xxx.xxx.116.178	
xxx.xxx.116.182	
xxx.xxx.116.183	
xxx.xxx.116.185	
xxx.xxx.116.187	
xxx.xxx.116.189	
xxx.xxx.116.192	
xxx.xxx.116.193	
xxx.xxx.116.194	
xxx.xxx.116.196	
xxx.xxx.116.197	
xxx.xxx.116.199	
xxx.xxx.116.200	
xxx.xxx.116.202	
xxx.xxx.116.203	
xxx.xxx.116.204	
xxx.xxx.116.205	
xxx.xxx.116.208	
xxx.xxx.116.209	
xxx.xxx.116.210	
xxx.xxx.116.212	
xxx.xxx.116.213	
xxx.xxx.116.214	
xxx.xxx.116.215	
xxx.xxx.116.217	
xxx.xxx.116.220	
xxx.xxx.116.224	
xxx.xxx.116.225	
xxx.xxx.116.228	
xxx.xxx.116.229	
xxx.xxx.116.232	
xxx.xxx.116.233	
xxx.xxx.116.238	
xxx.xxx.116.239	
xxx.xxx.116.240	
xxx.xxx.116.241	
xxx.xxx.116.242	
xxx.xxx.116.245	
xxx.xxx.116.246	
xxx.xxx.116.247	
xxx.xxx.116.249	
xxx.xxx.116.252	
xxx.xxx.116.254	
xxx.xxx.117.2	
xxx.xxx.117.3	
xxx.xxx.117.4	
xxx.xxx.117.5	
xxx.xxx.117.6	
xxx.xxx.117.7	
xxx.xxx.117.8	
xxx.xxx.117.9	
xxx.xxx.117.10	
xxx.xxx.117.11	
xxx.xxx.117.12	
xxx.xxx.117.13	
xxx.xxx.117.14	
xxx.xxx.117.15	
xxx.xxx.117.16	
xxx.xxx.117.17	
xxx.xxx.117.18	
xxx.xxx.117.19	
xxx.xxx.117.20	
xxx.xxx.117.21	
xxx.xxx.117.22	
xxx.xxx.117.23	
xxx.xxx.117.24	
xxx.xxx.117.25	
xxx.xxx.117.26	
xxx.xxx.117.27	
xxx.xxx.117.28	
xxx.xxx.117.29	
xxx.xxx.117.30	
xxx.xxx.117.31	
xxx.xxx.117.33	
xxx.xxx.117.34	
xxx.xxx.117.35	
xxx.xxx.117.36	
xxx.xxx.117.37	
xxx.xxx.117.38	
xxx.xxx.117.39	
xxx.xxx.117.40	
xxx.xxx.117.41	
xxx.xxx.117.42	
xxx.xxx.117.43	
xxx.xxx.117.44	
xxx.xxx.117.45	
xxx.xxx.117.46	
xxx.xxx.117.47	
xxx.xxx.117.48	
xxx.xxx.117.49	
xxx.xxx.117.50	
xxx.xxx.117.51	
xxx.xxx.117.52	
xxx.xxx.117.53	
xxx.xxx.117.54	
xxx.xxx.117.55	
xxx.xxx.117.56	
xxx.xxx.117.57	
xxx.xxx.117.58	
xxx.xxx.117.59	
xxx.xxx.117.60	
xxx.xxx.117.61	
xxx.xxx.117.62	
xxx.xxx.117.63	
xxx.xxx.117.64	
xxx.xxx.117.65	
xxx.xxx.117.66	
xxx.xxx.117.67	
xxx.xxx.117.68	
xxx.xxx.117.69	
xxx.xxx.117.70	
xxx.xxx.117.71	
xxx.xxx.117.72	
xxx.xxx.117.73	
xxx.xxx.117.74	
xxx.xxx.117.75	
xxx.xxx.117.76	
xxx.xxx.117.77	
xxx.xxx.117.78	
xxx.xxx.117.79	
xxx.xxx.117.80	
xxx.xxx.117.81	
xxx.xxx.117.82	
xxx.xxx.117.83	
xxx.xxx.117.84	
xxx.xxx.117.85	
xxx.xxx.117.86	
xxx.xxx.117.87	
xxx.xxx.117.88	
xxx.xxx.117.89	
xxx.xxx.117.90	
xxx.xxx.117.91	
xxx.xxx.117.92	
xxx.xxx.117.93	
xxx.xxx.117.96	
xxx.xxx.117.97	
xxx.xxx.117.98	
xxx.xxx.117.99	
xxx.xxx.117.100	
xxx.xxx.117.101	
xxx.xxx.117.102	
xxx.xxx.117.103	
xxx.xxx.117.104	
xxx.xxx.117.105	
xxx.xxx.117.106	
xxx.xxx.117.107	
xxx.xxx.117.108	
xxx.xxx.117.109	
xxx.xxx.117.110	
xxx.xxx.117.111	
xxx.xxx.117.112	
xxx.xxx.117.113	
xxx.xxx.117.114	
xxx.xxx.117.115	
xxx.xxx.117.116	
xxx.xxx.117.117	
xxx.xxx.117.118	
xxx.xxx.117.119	
xxx.xxx.117.120	
xxx.xxx.117.122	
xxx.xxx.117.123	
xxx.xxx.117.124	
xxx.xxx.117.125	
xxx.xxx.117.126	
xxx.xxx.117.127	
xxx.xxx.117.128	
xxx.xxx.117.129	
xxx.xxx.117.130	
xxx.xxx.117.131	
xxx.xxx.117.132	
xxx.xxx.117.133	
xxx.xxx.117.134	
xxx.xxx.117.135	
xxx.xxx.117.136	
xxx.xxx.117.137	
xxx.xxx.117.138	
xxx.xxx.117.139	
xxx.xxx.117.140	
xxx.xxx.117.141	
xxx.xxx.117.142	
xxx.xxx.117.143	
xxx.xxx.117.144	
xxx.xxx.117.145	
xxx.xxx.117.146	
xxx.xxx.117.147	
xxx.xxx.117.148	
xxx.xxx.117.149	
xxx.xxx.117.150	
xxx.xxx.117.151	
xxx.xxx.117.152	
xxx.xxx.117.153	
xxx.xxx.117.154	
xxx.xxx.117.155	
xxx.xxx.117.156	
xxx.xxx.117.157	
xxx.xxx.117.158	
xxx.xxx.117.159	
xxx.xxx.117.160	
xxx.xxx.117.161	
xxx.xxx.117.162	
xxx.xxx.117.163	
xxx.xxx.117.164	
xxx.xxx.117.165	
xxx.xxx.117.166	
xxx.xxx.117.167	
xxx.xxx.117.168	
xxx.xxx.117.169	
xxx.xxx.117.170	
xxx.xxx.117.171	
xxx.xxx.117.172	
xxx.xxx.117.173	
xxx.xxx.117.174	
xxx.xxx.117.175	
xxx.xxx.117.176	
xxx.xxx.117.177	
xxx.xxx.117.178	
xxx.xxx.117.179	
xxx.xxx.117.180	
xxx.xxx.117.181	
xxx.xxx.117.182	
xxx.xxx.117.183	
xxx.xxx.117.184	
xxx.xxx.117.185	
xxx.xxx.117.186	
xxx.xxx.117.187	
xxx.xxx.117.188	
xxx.xxx.117.189	
xxx.xxx.117.190	
xxx.xxx.117.191	
xxx.xxx.117.192	
xxx.xxx.117.193	
xxx.xxx.117.194	
xxx.xxx.117.195	
xxx.xxx.117.196	
xxx.xxx.117.197	
xxx.xxx.117.198	
xxx.xxx.117.199	
xxx.xxx.117.200	
xxx.xxx.117.201	
xxx.xxx.117.202	
xxx.xxx.117.203	
xxx.xxx.117.204	
xxx.xxx.117.205	
xxx.xxx.117.206	
xxx.xxx.117.207	
xxx.xxx.117.208	
xxx.xxx.117.209	
xxx.xxx.117.210	
xxx.xxx.117.211	
xxx.xxx.117.212	
xxx.xxx.117.213	
xxx.xxx.117.214	
xxx.xxx.117.216	
xxx.xxx.117.217	
xxx.xxx.117.218	
xxx.xxx.117.219	
xxx.xxx.117.220	
xxx.xxx.117.221	
xxx.xxx.117.222	
xxx.xxx.117.223	
xxx.xxx.117.224	
xxx.xxx.117.225	
xxx.xxx.117.226	
xxx.xxx.117.227	
xxx.xxx.117.228	
xxx.xxx.117.229	
xxx.xxx.117.230	
xxx.xxx.117.231	
xxx.xxx.117.232	
xxx.xxx.117.233	
xxx.xxx.117.234	
xxx.xxx.117.235	
xxx.xxx.117.236	
xxx.xxx.117.237	
xxx.xxx.117.238	
xxx.xxx.117.239	
xxx.xxx.117.240	
xxx.xxx.117.241	
xxx.xxx.117.242	
xxx.xxx.117.243	
xxx.xxx.117.244	
xxx.xxx.117.245	
xxx.xxx.117.246	
xxx.xxx.117.247	
xxx.xxx.117.248	
xxx.xxx.117.249	
xxx.xxx.117.250	
xxx.xxx.117.251	
xxx.xxx.117.252	
xxx.xxx.117.253	
xxx.xxx.117.254	
xxx.xxx.118.2	
xxx.xxx.118.3	
xxx.xxx.118.5	
xxx.xxx.118.6	
xxx.xxx.118.7	
xxx.xxx.118.9	
xxx.xxx.118.11	
xxx.xxx.118.12	
xxx.xxx.118.13	
xxx.xxx.118.14	
xxx.xxx.118.15	
xxx.xxx.118.16	
xxx.xxx.118.17	
xxx.xxx.118.18	
xxx.xxx.118.19	
xxx.xxx.118.20	
xxx.xxx.118.21	
xxx.xxx.118.22	
xxx.xxx.118.24	
xxx.xxx.118.26	
xxx.xxx.118.27	
xxx.xxx.118.28	
xxx.xxx.118.29	
xxx.xxx.118.30	
xxx.xxx.118.32	
xxx.xxx.118.33	
xxx.xxx.118.34	
xxx.xxx.118.35	
xxx.xxx.118.36	
xxx.xxx.118.37	
xxx.xxx.118.38	
xxx.xxx.118.40	
xxx.xxx.118.41	
xxx.xxx.118.42	
xxx.xxx.118.43	
xxx.xxx.118.44	
xxx.xxx.118.45	
xxx.xxx.118.46	
xxx.xxx.118.47	
xxx.xxx.118.48	
xxx.xxx.118.49	
xxx.xxx.118.50	
xxx.xxx.118.51	
xxx.xxx.118.52	
xxx.xxx.118.53	
xxx.xxx.118.54	
xxx.xxx.118.55	
xxx.xxx.118.56	
xxx.xxx.118.57	
xxx.xxx.118.58	
xxx.xxx.118.59	
xxx.xxx.118.60	
xxx.xxx.118.61	
xxx.xxx.118.62	
xxx.xxx.118.64	
xxx.xxx.118.65	
xxx.xxx.118.66	
xxx.xxx.118.67	
xxx.xxx.118.69	
xxx.xxx.118.70	
xxx.xxx.118.71	
xxx.xxx.118.72	
xxx.xxx.118.73	
xxx.xxx.118.76	
xxx.xxx.118.77	
xxx.xxx.118.78	
xxx.xxx.118.79	
xxx.xxx.118.80	
xxx.xxx.118.81	
xxx.xxx.118.83	
xxx.xxx.118.84	
xxx.xxx.118.85	
xxx.xxx.118.86	
xxx.xxx.118.87	
xxx.xxx.118.88	
xxx.xxx.118.89	
xxx.xxx.118.90	
xxx.xxx.118.91	
xxx.xxx.118.92	
xxx.xxx.118.93	
xxx.xxx.118.94	
xxx.xxx.118.95	
xxx.xxx.118.96	
xxx.xxx.118.97	
xxx.xxx.118.98	
xxx.xxx.118.99	
xxx.xxx.118.100	
xxx.xxx.118.102	
xxx.xxx.118.103	
xxx.xxx.118.104	
xxx.xxx.118.106	
xxx.xxx.118.107	
xxx.xxx.118.108	
xxx.xxx.118.109	
xxx.xxx.118.110	
xxx.xxx.118.111	
xxx.xxx.118.113	
xxx.xxx.118.114	
xxx.xxx.118.116	
xxx.xxx.118.117	
xxx.xxx.118.118	
xxx.xxx.118.119	
xxx.xxx.118.120	
xxx.xxx.118.122	
xxx.xxx.118.125	
xxx.xxx.118.126	
xxx.xxx.118.127	
xxx.xxx.118.130	
xxx.xxx.118.131	
xxx.xxx.118.133	
xxx.xxx.118.134	
xxx.xxx.118.135	
xxx.xxx.118.137	
xxx.xxx.118.139	
xxx.xxx.118.142	
xxx.xxx.118.143	
xxx.xxx.118.147	
xxx.xxx.118.149	
xxx.xxx.118.153	
xxx.xxx.118.155	
xxx.xxx.118.157	
xxx.xxx.118.158	
xxx.xxx.118.159	
xxx.xxx.118.162	
xxx.xxx.118.163	
xxx.xxx.118.164	
xxx.xxx.118.165	
xxx.xxx.118.166	
xxx.xxx.118.167	
xxx.xxx.118.171	
xxx.xxx.118.172	
xxx.xxx.118.173	
xxx.xxx.118.175	
xxx.xxx.118.176	
xxx.xxx.118.178	
xxx.xxx.118.180	
xxx.xxx.118.182	
xxx.xxx.118.186	
xxx.xxx.118.187	
xxx.xxx.118.188	
xxx.xxx.118.189	
xxx.xxx.118.190	
xxx.xxx.118.191	
xxx.xxx.118.192	
xxx.xxx.118.193	
xxx.xxx.118.194	
xxx.xxx.118.196	
xxx.xxx.118.197	
xxx.xxx.118.198	
xxx.xxx.118.199	
xxx.xxx.118.200	
xxx.xxx.118.201	
xxx.xxx.118.202	
xxx.xxx.118.205	
xxx.xxx.118.206	
xxx.xxx.118.207	
xxx.xxx.118.209	
xxx.xxx.118.213	
xxx.xxx.118.214	
xxx.xxx.118.216	
xxx.xxx.118.217	
xxx.xxx.118.219	
xxx.xxx.118.220	
xxx.xxx.118.221	
xxx.xxx.118.223	
xxx.xxx.118.225	
xxx.xxx.118.227	
xxx.xxx.118.230	
xxx.xxx.118.231	
xxx.xxx.118.233	
xxx.xxx.118.234	
xxx.xxx.118.235	
xxx.xxx.118.236	
xxx.xxx.118.238	
xxx.xxx.118.239	
xxx.xxx.118.240	
xxx.xxx.118.244	
xxx.xxx.118.245	
xxx.xxx.118.251	
xxx.xxx.118.252	
xxx.xxx.118.254	
xxx.xxx.119.3	
xxx.xxx.119.4	
xxx.xxx.119.5	
xxx.xxx.119.6	
xxx.xxx.119.7	
xxx.xxx.119.8	
xxx.xxx.119.9	
xxx.xxx.119.10	
xxx.xxx.119.11	
xxx.xxx.119.12	
xxx.xxx.119.13	
xxx.xxx.119.14	
xxx.xxx.119.15	
xxx.xxx.119.16	
xxx.xxx.119.17	
xxx.xxx.119.18	
xxx.xxx.119.19	
xxx.xxx.119.20	
xxx.xxx.119.21	
xxx.xxx.119.22	
xxx.xxx.119.23	
xxx.xxx.119.24	
xxx.xxx.119.25	
xxx.xxx.119.26	
xxx.xxx.119.27	
xxx.xxx.119.28	
xxx.xxx.119.29	
xxx.xxx.119.30	
xxx.xxx.119.31	
xxx.xxx.119.32	
xxx.xxx.119.33	
xxx.xxx.119.34	
xxx.xxx.119.35	
xxx.xxx.119.36	
xxx.xxx.119.37	
xxx.xxx.119.38	
xxx.xxx.119.39	
xxx.xxx.119.40	
xxx.xxx.119.41	
xxx.xxx.119.42	
xxx.xxx.119.43	
xxx.xxx.119.44	
xxx.xxx.119.45	
xxx.xxx.119.46	
xxx.xxx.119.47	
xxx.xxx.119.48	
xxx.xxx.119.50	
xxx.xxx.119.51	
xxx.xxx.119.52	
xxx.xxx.119.53	
xxx.xxx.119.54	
xxx.xxx.119.55	
xxx.xxx.119.57	
xxx.xxx.119.58	
xxx.xxx.119.59	
xxx.xxx.119.60	
xxx.xxx.119.61	
xxx.xxx.119.62	
xxx.xxx.119.63	
xxx.xxx.119.64	
xxx.xxx.119.65	
xxx.xxx.119.66	
xxx.xxx.119.67	
xxx.xxx.119.68	
xxx.xxx.119.69	
xxx.xxx.119.70	
xxx.xxx.119.71	
xxx.xxx.119.72	
xxx.xxx.119.73	
xxx.xxx.119.74	
xxx.xxx.119.75	
xxx.xxx.119.76	
xxx.xxx.119.77	
xxx.xxx.119.78	
xxx.xxx.119.79	
xxx.xxx.119.80	
xxx.xxx.119.81	
xxx.xxx.119.82	
xxx.xxx.119.83	
xxx.xxx.119.84	
xxx.xxx.119.85	
xxx.xxx.119.86	
xxx.xxx.119.87	
xxx.xxx.119.88	
xxx.xxx.119.89	
xxx.xxx.119.90	
xxx.xxx.119.91	
xxx.xxx.119.92	
xxx.xxx.119.93	
xxx.xxx.119.94	
xxx.xxx.119.95	
xxx.xxx.119.96	
xxx.xxx.119.97	
xxx.xxx.119.98	
xxx.xxx.119.99	
xxx.xxx.119.100	
xxx.xxx.119.101	
xxx.xxx.119.102	
xxx.xxx.119.103	
xxx.xxx.119.104	
xxx.xxx.119.105	
xxx.xxx.119.106	
xxx.xxx.119.107	
xxx.xxx.119.109	
xxx.xxx.119.110	
xxx.xxx.119.111	
xxx.xxx.119.112	
xxx.xxx.119.113	
xxx.xxx.119.114	
xxx.xxx.119.115	
xxx.xxx.119.116	
xxx.xxx.119.117	
xxx.xxx.119.118	
xxx.xxx.119.119	
xxx.xxx.119.120	
xxx.xxx.119.121	
xxx.xxx.119.122	
xxx.xxx.119.123	
xxx.xxx.119.124	
xxx.xxx.119.125	
xxx.xxx.119.126	
xxx.xxx.119.127	
xxx.xxx.119.128	
xxx.xxx.119.129	
xxx.xxx.119.130	
xxx.xxx.119.131	
xxx.xxx.119.132	
xxx.xxx.119.133	
xxx.xxx.119.134	
xxx.xxx.119.135	
xxx.xxx.119.136	
xxx.xxx.119.137	
xxx.xxx.119.138	
xxx.xxx.119.139	
xxx.xxx.119.140	
xxx.xxx.119.141	
xxx.xxx.119.142	
xxx.xxx.119.143	
xxx.xxx.119.146	
xxx.xxx.119.147	
xxx.xxx.119.148	
xxx.xxx.119.149	
xxx.xxx.119.151	
xxx.xxx.119.152	
xxx.xxx.119.153	
xxx.xxx.119.154	
xxx.xxx.119.155	
xxx.xxx.119.156	
xxx.xxx.119.157	
xxx.xxx.119.158	
xxx.xxx.119.159	
xxx.xxx.119.160	
xxx.xxx.119.161	
xxx.xxx.119.162	
xxx.xxx.119.163	
xxx.xxx.119.164	
xxx.xxx.119.165	
xxx.xxx.119.166	
xxx.xxx.119.167	
xxx.xxx.119.168	
xxx.xxx.119.169	
xxx.xxx.119.170	
xxx.xxx.119.171	
xxx.xxx.119.172	
xxx.xxx.119.173	
xxx.xxx.119.174	
xxx.xxx.119.175	
xxx.xxx.119.176	
xxx.xxx.119.177	
xxx.xxx.119.178	
xxx.xxx.119.179	
xxx.xxx.119.180	
xxx.xxx.119.181	
xxx.xxx.119.182	
xxx.xxx.119.183	
xxx.xxx.119.184	
xxx.xxx.119.185	
xxx.xxx.119.186	
xxx.xxx.119.187	
xxx.xxx.119.188	
xxx.xxx.119.189	
xxx.xxx.119.190	
xxx.xxx.119.191	
xxx.xxx.119.192	
xxx.xxx.119.193	
xxx.xxx.119.194	
xxx.xxx.119.195	
xxx.xxx.119.196	
xxx.xxx.119.197	
xxx.xxx.119.198	
xxx.xxx.119.199	
xxx.xxx.119.200	
xxx.xxx.119.201	
xxx.xxx.119.202	
xxx.xxx.119.203	
xxx.xxx.119.204	
xxx.xxx.119.205	
xxx.xxx.119.206	
xxx.xxx.119.207	
xxx.xxx.119.208	
xxx.xxx.119.209	
xxx.xxx.119.210	
xxx.xxx.119.211	
xxx.xxx.119.212	
xxx.xxx.119.213	
xxx.xxx.119.214	
xxx.xxx.119.215	
xxx.xxx.119.216	
xxx.xxx.119.217	
xxx.xxx.119.218	
xxx.xxx.119.219	
xxx.xxx.119.220	
xxx.xxx.119.222	
xxx.xxx.119.223	
xxx.xxx.119.224	
xxx.xxx.119.225	
xxx.xxx.119.226	
xxx.xxx.119.227	
xxx.xxx.119.228	
xxx.xxx.119.229	
xxx.xxx.119.230	
xxx.xxx.119.231	
xxx.xxx.119.232	
xxx.xxx.119.233	
xxx.xxx.119.234	
xxx.xxx.119.235	
xxx.xxx.119.236	
xxx.xxx.119.237	
xxx.xxx.119.238	
xxx.xxx.119.239	
xxx.xxx.119.240	
xxx.xxx.119.241	
xxx.xxx.119.242	
xxx.xxx.119.243	
xxx.xxx.119.244	
xxx.xxx.119.245	
xxx.xxx.119.246	
xxx.xxx.119.247	
xxx.xxx.119.248	
xxx.xxx.119.249	
xxx.xxx.119.250	
xxx.xxx.119.251	
xxx.xxx.119.252	
xxx.xxx.119.253	
xxx.xxx.119.254

Would appreciate help troubleshooting from here.