Tracing a TCP Connection Failure
25 Aug 2019 · Comments: · Tags: traceroute, tracert, tracetcpSummary
A sysadmin often has to facilitate requests to provide access from the network they manage to resources hosted by an external party. If the target resource cannot be accessed, they have to establish whether this is due to the traffic being blocked by a device on their network or by that of the other party.
One way to determine this is for the sysadmin to examine the logs of any relevant software firewalls and routing devices (routers, layer three switches, firewalls, etc) on their network. The purpose of this post is to demonstrate some alternative techniques which are particularly useful to someone without access to such logs. It will focus specifically on diagnosing TCP connection failures to IPv4 endpoints using route tracing tools.
Scenario
For the purpose of this post I’ll be using a fictitious scenario. A computer within a corporate network with private IP 192.168.0.5 requires SSH (TCP port 22) access to an external endpoint with public IP 203.0.113.1.
I’ll use two simple network topologies:
The source computer’s default gateway is 192.168.0.254, which is a layer three switch.
+------------------------------------------------+
| CORPORATE NETWORK |
| |
| +----------+ +---------+ +----------+ | __ _ +-------------+
| | SOURCE | | LAYER 3 | | FIREWALL | | _( )_( )_ | DESTINATION |
| | COMPUTER |---->| SWITCH |---->| |--|-->(_ _ _)---->| ENDPOINT |
| | | | | | | | (_) (__) | |
| +----------+ +---------+ +----------+ | +-------------+
| 192.168.0.5 192.168.0.254 192.168.0.253 | 203.0.113.1
| |
+------------------------------------------------+ The source computer’s default gateway is 192.168.0.253, which is a firewall.
+--------------------------------+
| CORPORATE NETWORK |
| |
| +----------+ +----------+ | __ _ +-------------+
| | SOURCE | | FIREWALL | | _( )_( )_ | DESTINATION |
| | COMPUTER |---->| |--|-->(_ _ _)---->| ENDPOINT |
| | | | | | (_) (__) | |
| +----------+ +----------+ | +-------------+
| 192.168.0.5 192.168.0.253 | 203.0.113.1
| |
+--------------------------------+ For each topology, I’ll demonstrate how route tracing tools behave under three different conditions:
- The source computer successfully connects to the destination endpoint.
- The source computer fails to connect to the destination endpoint due to the firewall within the corporate network blocking the traffic.
- The source computer fails to connect to the destination endpoint due to a device outside the corporate network blocking the traffic.
Route Tracing Tools
Route tracing tools provide details of the path across an IP network to a specified endpoint. Each routing device that’s traversed is listed along with associated statistics. This post is only concerned with the path taken, therefore I won’t be elaborating on the significance of the statistics.
traceroute (Unix-like OSs)
On Unix-like systems, route tracing can be achieved with the traceroute command.
Its default behaviour is to send a series of UDP packets with destination port
numbers ranging from 33434 to 33534 which is of no use in this scenario. What is
of use is an alternative mode of operation which is to send TCP SYN packets to a
TCP port of your choice.
sudo traceroute -T -p 22 203.0.113.1Example Output
Topology A. Condition 1 (successful connection)
The output shows that the destination endpoint was reached.
traceroute to 203.0.113.1 (203.0.113.1), 30 hops max, 60 byte packets
1 192.168.0.254 (192.168.0.254) 0.099 ms 0.103 ms 0.142 ms
2 192.168.0.253 (192.168.0.253) 0.246 ms 0.283 ms 0.331 ms
3 * 198.51.100.5 (198.51.100.5) 0.485 ms *
4 * * *
5 198.51.100.21 (198.51.100.21) 5.830 ms 5.821 ms 2.629 ms
6 203.0.113.1 (203.0.113.1) 0.441 ms 0.557 ms 0.437 msTopology A. Condition 2 (blocked by internal firewall)
The output shows that the traffic only got as far as the layer three switch.
traceroute to 203.0.113.1 (203.0.113.1), 30 hops max, 60 byte packets
1 192.168.0.254 (192.168.0.254) 1.67 ms 1.663 ms 2.085 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *Topology A. Condition 3 (blocked externally)
The output shows that the traffic successfully traversed the two routing devices within the corporate network.
traceroute to 203.0.113.1 (203.0.113.1), 30 hops max, 60 byte packets
1 192.168.0.254 (192.168.0.254) 0.099 ms 0.103 ms 0.142 ms
2 192.168.0.253 (192.168.0.253) 0.246 ms 0.283 ms 0.331 ms
3 * 198.51.100.5 (198.51.100.5) 0.485 ms *
4 * * *
5 198.51.100.21 (198.51.100.21) 5.830 ms 5.821 ms 2.629 ms
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *Topology B. Condition 1 (successful connection)
The output shows that the destination endpoint was reached.
traceroute to 203.0.113.1 (203.0.113.1), 30 hops max, 60 byte packets
1 192.168.0.253 (192.168.0.253) 0.291 ms 0.211 ms 0.413 ms
2 * 198.51.100.5 (198.51.100.5) 0.315 ms 0.339 ms
3 * * *
4 198.51.100.21 (198.51.100.21) 6.012 ms 5.121 ms 2.711 ms
5 203.0.113.1 (203.0.113.1) 0.551 ms 0.471 ms 0.481 msTopology B. Condition 2 (blocked by internal firewall)
The output shows that the traffic failed to traverse a single routing device.
traceroute to 203.0.113.1 (203.0.113.1), 30 hops max, 60 byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *Topology B. Condition 3 (blocked externally)
The output shows that the traffic successfully traversed the only routing device within the corporate network.
traceroute to 203.0.113.1 (203.0.113.1), 30 hops max, 60 byte packets
1 192.168.0.253 (192.168.0.253) 0.281 ms 0.223 ms 0.298 ms
2 * 198.51.100.5 (198.51.100.5) 0.413 ms *
3 * * *
4 198.51.100.21 (198.51.100.21) 5.710 ms 5.929 ms 2.599 ms
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *tracetcp (Windows)
Windows ships with tracert, a route tracing tool with only one mode of
operation which is to send ICMP echo requests. This is of no use since the
scenario in this post is to determine the point at which traffic is being
blocked for a specific TCP port. Fortunately there is a suitable third party
tool for Windows, tracetcp.
tracetcp operates in the same way as traceroute by sending TCP SYN packets
on a TCP port of your choice.
tracetcp 203.0.113.1:22Example Output
When testing against Topology A I kept getting strange output like this:
1 1 ms 2 ms 14 ms 192.168.0.254
2 * * * Request timed out.
3 192.168.0.254 reports: ICMP REDIRECT recievedThis comment on the r/sysadmin subreddit seems to confirm that such behaviour occurs when a routing device routes to another routing device within the same network segment (which is exactly how Topology A is designed). This Github issue seems to imply that the behaviour is a bug, which at the time of writing has not been fixed.
To work around this I had to use the -g parameter to specify that the firewall
should be used as the gateway, thus bypassing the layer three switch:
tracetcp 203.0.113.1:22 -g 192.168.0.253Invoking the -g parameter to use the firewall as the gateway means that
topologies A and B are rendered equivalent. Therefore what follows applies to
both Topology A and Topology B.
Condition 1 (successful connection)
The output shows that the destination endpoint was reached.
Tracing route to 203.0.113.1 on port 22
Over a maximum of 30 hops.
1 6 ms 3 ms 8 ms 192.168.0.253
2 1 ms 1 ms 1 ms 198.51.100.5
3 5 ms 5 ms 6 ms 198.51.100.21
4 * * * Request timed out.
5 1 ms 1 ms 1 ms 203.0.113.1Condition 2 (blocked by internal firewall)
The output shows that the traffic fails to traverse a single routing device.
Tracing route to 203.0.113.1 on port 22
Over a maximum of 30 hops.
1 * * * Request timed out.
2 * * * Request timed out.
3 * * * Request timed out.
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
12 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.Condition 3 (blocked externally)
The output shows that the traffic successfully traversed the only routing device within the corporate network.
Tracing route to 203.0.113.1 on port 22
Over a maximum of 30 hops.
1 6 ms 3 ms 8 ms 192.168.0.253
2 1 ms 1 ms 1 ms 198.51.100.5
3 * * * Request timed out.
4 5 ms 3 ms 6 ms 198.51.100.21
5 * * * Request timed out.
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
12 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.Default Gateway
Unless a static route has been defined on a computer, traffic bound for an external party should traverse the default gateway as the first hop. Something to be aware of is that if the gateway is a virtual/logical address then the IP that’s recorded as the first hop by a route tracing tool may differ.
A virtual/logical address may be used for load balancing purposes, such as with VRRP (Virtual Router Redundancy Protocol) or HSRP (Hot Standby Router Protocol). In such cases the route tracing tool will show the first physical address of the path taken.
See also:
Comments