Basics of Linux - Network Troubleshooting

Checking for Physical Connections

Although it is highly unlikely that in a Data Center environment to find that a network cable has been physically unplugged from a Server but sometimes this can really happen and all of your troubleshooting will be for naught.

Linux provides a simple command to check for Physical Connectivity.

ethtool <interface name>







-




Check IP Setttings

In linux the command to check ip settings assigned to an interface is 

ifconfig <interface name>

interface name is optional in above command, if specified like eth0 or em1  the command will than display information regarding that interface only.

other related commands are:

ifup <interface name>

ifdown <interface name>

here interface name is mandatory so kernel can change the state of that interface

Finding out Network Gateways

The Route command manipulates the kernel's IP routing tables. route command belongs to the net-tools suite, it will be replaced by iproute2 suite in the future, we will get into more details about iproute2 in another article.  

So route commands primary use is to set up static routes to specific hosts or networks via an interface after it has been configured with the ifconfig program.

When the add or del options are used, route modifies the routing tables. Without these options, route displays the current contents of the routing tables.
route -n




The highlighted part is the default gateway.

Checking if Network Gateway is reachable

Whenever troubleshooting a network related problem, after you have established that the physically connectivity is present, this should be your first step.

If default gateway is reachable then proceed to further troubleshooting. 

 Adding/Removing a Default Gateway

 route add default gw [IP] [dev]
 route del default
 

Performing Network Analysis with Network Specific Tools

a.       PING

Ping is a computer network administration software utility used to test the reachability of a host on an Internet Protocol (IP) network and to measure the round-trip time for messages sent from the originating host to a destination computer.

The name comes from active sonar terminology which sends a pulse of sound and listens for the echo to detect objects underwater.

Ping uses the ICMP protocol’s mandatory ECHO_REQUEST datagram to elicit an ICMP ECHO_RESPONSE from a host or gateway.  ECHO_REQUEST datagrams (‘‘pings’’) have an IP and ICMP header, followed by a struct timeval and then an arbitrary number of ‘‘pad’’ bytes used to fill out the packet.

ICMP Errors

The most useful ICMP packet is the Destination Unreachable message. Error messages are typically generated by routers, and sent to the original source of the packet.  Destination Unreachable type messages can have a Code value of 0 to 15, most commonly seen errors are:

Value	Description	Meaning
0	Network Unreachable	generated by a router if a route to the destination network is not available
1	Host Unreachable	generated by a router if a route to the destination host on a directly connected network is not available (does not respond to ARP)
7	Destination Host unknown	generated only when a router can determine (from link layer advice) that the destination host does not exist

     

What if you receive a Request Timed Out message, does it mean that the destination host is down?

It is not always possible to know the exact reason behind a "no-answer". As a matter of fact, there may be a lot of different reasons. We will consider some major reasons below.

Case 1

There is an external firewall installed at the host end, in which case it will drop all ICMP connections.

Case 2

There is an internal firewall at the destination server end

Case 3

There is some routing/connection issue at destination end.

The basic information provided by ICMP Ping is of connectivity between hosts and even that information may sometimes not be available because of Security Policies implemented on destination routers/servers. Also PING does not tell what services are running on the remote server.

b.      TELNET

Telnet is a text based communication program that allows you to connect to a remote server/router over a network. Its normal use is to login to a server that has shell access to allow you to run commands on the server. However we can also use telnet to check if a service is running on remote server. All traffic sent through telnet protocol is unencrypted.




Above we have just verified that a service is up and running at 172.20.w0.82 on port 7334.

Apache httpd and tomcat servers typically present non-interactive shells on telnet. But you can find that an SMTP server will always present an interactive shell.

Below we will send an email through Nayatel’s SMTP server by using telnet. This is useful when you are trying to troubleshoot email problems.

telnet mail.xyz.com 25




The highlighted parts are the commands entered by user.

c.       NETSTAT

When you type netstat alone(without any options)  at the prompt, the result is a list of all the TCP connections between your computer and others.

The IP addresses are not shown in this mode. You see the more human-friendly hostnames.

  Proto: Protocol being used by this connection

  Recv: Received Packets

  Send: Sent Packets

  Local Address: The local IP/host that has initiated the connection

  Foreign Address: Address of the Machine to which the connection has been made

  State: The state of Network Connection [ESTABLISHED/TIME_WAIT/CLOSING etc]

d.      TRACEROUTE

An important part of administering servers is monitoring network connectivity.

There are a few tools that are as simple to use, but valuable to know. In this guide, we will discuss how to use a tool called traceroute to diagnose where a network issue may be happening.

We will also look at a utility called mtr which combines much of the functionality of ping and traceroute into one interface.

How to Use Traceroute

Traceroute is a simple tool to show the pathway to a remote server. This can be anything from a website that you are attempting to visit, to a printer on your local network.

The traceroute program is installed by default on just about every Linux distribution, so you shouldn't need to install it.

To call it, we simply need to provide a website or IP address that we would like to explore.

How to Read Traceroute's Output

The first line tells us the conditions that traceroute is operating under:

traceroute to google.com (173.194.38.137), 30 hops max, 60 byte packets

It gives the specified host, the IP address that DNS returns for that domain, the maximum number of hops to check, and the size of the packet that will be used.

The maximum number of hops can be adjusted with the -m flag. If the host you are trying to route to is over 30 hops away, you may need to specify a larger value here. The maximum value you can set is 255.

traceroute -m 255 propakistani.com

You can adjust the size of the packet that is sent to each hop by giving the integer after the hostname:

traceroute google.com 70

traceroute to google.com (173.194.38.128), 30 hops max, 70 byte packets

 1  192.241.160.254 (192.241.160.254)  0.364 ms  0.330 ms  0.319 ms

 2  192.241.164.237 (192.241.164.237)  0.284 ms  0.343 ms  0.321 ms

After the first line, each subsequent line represents a "hop", or intermediate hosts that your traffic must pass through to reach the computer represented by the host you specified.

Each line has the following format:

hop_number   host_name   (IP_address)  packet_round_trip_times

Here is an example of a hop you might see:

3  nyk-b6-link.telia.net (62.115.35.101)  0.311 ms  0.302 ms  0.293 ms

Here is what each field means:

·         hop_number: A sequential count of the number of degrees of separation the host is from your computer. Traffic from hosts with higher numbers have to go through more computers to get routed.

·         host_name: This field contains the result of a reverse DNS lookup on the host's IP address, if available. If no information is returned from the reverse DNS query, the IP address itself is given.

·         IP_address: This field contains the IP address for this network hop.

·         packetroundtrip_times: The remainder of the line gives the round-trip times for a packet to the host and back again. By default, three packets are sent to each host and each attempt is appended to the end of the line.

o    If you would like to change the number of packets that are tested against each host, you can specify a number with the -q option, like this:

o    traceroute -q1 google.com

If you would like to forgo the reverse DNS lookup to speed up the trace, you can pass the -n flag:

traceroute -n google.com

traceroute to google.com (74.125.235.7), 30 hops max, 60 byte packets

 1  192.241.160.253  0.626 ms  0.598 ms  0.588 ms

 2  192.241.164.241  2.821 ms  2.743 ms  2.819 ms

 3  69.31.95.133  1.470 ms  1.473 ms  1.525 ms

If your traceroute dissolves into some asterisks (*), there may be a problem with the route to the host, in any case this demands further investigation

15  209.85.248.220 (209.85.248.220)  121.809 ms 72.14.239.12 (72.14.239.12)              

16  72.14.239.247 (72.14.239.247)  101.001 ms  92.478 ms  92.448 ms

17  * * 209.85.250.124 (209.85.250.124)  175.083 ms

18  * * *

19  * * *

What Does a Route Issue Mean?

If your traceroute attempt stops at a particular hop or node and cannot find a route to the host, you have a problem.

While the hop where the route fails to return may be the location of the networking issue, it isn't always that easy to diagnose.

Due to the fact that each ping represents a round-trip packet, and the situation where packets often use different pathways in either direction, it may indicate a problem in a completely different, possibly closer route.

It also may be the case that the problem is with the hop directly after the last hop you see. It is difficult to diagnose the exact location of the problem unless you can get a return traceroute from that specific hop. This is usually not possible outside of your own network.

How To Use MTR

A dynamic alternative to the traceroute program is mtr. Combining the functionality of ping and traceroute, mtr allows you to constantly poll a remote server and see how the latency and performance changes over time.

mtr 172.20.43.198

While the output may look similar, the big advantage over traceroute is that the output is constantly updated. This allows you to accumulate trends and averages, and also allows you to see how the network performance varies over time.