5 Linux network troubleshooting commands

Networking configuration and troubleshooting are crucial tasks that sysadmins need to perform regularly. Some of these tasks can be challenging. However, when dealing with connectivity issues, using the right tools will assist you in achieving the results in a faster and consistent way.

The ip command

The ip command is an all-around utility to show and manipulate network objects on your Linux system, including IP addresses, routes, and ARP tables. It's a useful tool to configure the network, as well as to troubleshoot network connectivity issues.

The ip command replaces the functionality of many commands provided with the old net-tools package such as ifconfig, route, and arp, but it adds many other features.

In its most basic form, you can just run ip and provide a network object to manipulate, such as an address, link, or route, and a subcommand to perform an action. If you do not give a subcommand, many objects default to the show subcommand to display information related to that object.

Here is the basic syntax:

ip <OBJECT> [COMMAND]

For example, to see the link status on all network devices, run ip link show:

$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:b5:c7:2b brd ff:ff:ff:ff:ff:ff

Because show is the default subcommand, you can also obtain the same result by running ip link or even ip l (many objects recognize an abbreviation).

The ip command manages many objects. The main ones are:

• link or l - controls the status of network devices
• address or a - manipulates IP addresses on devices
• route or r - handles routing table entries
• neighbor or n - controls ARP table entries

You can see a full list of objects and commands by running ip help.

Some useful examples of the ip command

Show network statistics -s in human readable format -h for a specific network interface:

$ ip -s -h l show dev enp1s0
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:b5:c7:2b brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    820M       303k     0       182k    0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    19.9M      60.9k    0       0       0       0    

Show the IP addresses of all interfaces:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:b5:c7:2b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.169/24 brd 192.168.122.255 scope global noprefixroute enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::7ecf:5cc8:5c1f:1009/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

Show the IP address of a single interface e.g., enp1s0:

$ ip a show dev enp1s0

Add another IP address to an interface (requires sudo or root user):

$ sudo ip a change 192.168.122.170 dev enp1s0

Display the routing table:

$ ip route
default via 192.168.122.1 dev enp1s0 proto static metric 100 
192.168.122.0/24 dev enp1s0 proto kernel scope link src 192.168.122.169 metric 100

Add a route (default gateway):

$ sudo ip route add default via 192.168.122.1 dev enp1s0

The ip command is a lower-level interface to configure network options on your Linux system. While it's useful as a troubleshooting tool, it may be harder to set the network with it. For this reason, many distributions provide a higher-level interface to accomplish the same task. Next, let's take a look at the Network Manager configuration tool nmcli, which provides an integrated way to configure the network.

The nmcli tool

Network Manager is a network configuration application available by default with many Linux distributions, including RHEL and Fedora. Network Manager runs as a daemon, and its goal is to provide a higher-level interface to make network configuration easier and more automated.

It's common for users of desktop Linux or servers with a graphical interface to use Network Manager GUI clients to configure the network. For cases where you work with headless servers or if you want to automate configuration via shell scripts, the nmcli tool comes in handy.

Network Manager and nmcli are a versatile combination that allow you to verify and configure many network options. The basic syntax for nmcli is:

nmcli <OBJECT> [COMMAND] [ARGUMENTS]

The most common objects are:

• general - shows Network Manager status and permissions
• networking - shows, enables, and disables networking
• radio - shows, enables, and disables WiFi and WWAN
• device - shows and manipulates the status of network devices
• connection - manages Network Manager connection profiles

It's impossible to cover all the options on a single blog post. Let's check some examples of how to use nmcli for common network configurations tasks. For more examples, consult the official nmcli-examples page or the man pages with man nmcli-examples.

Check the status of network devices:

$ nmcli device status
DEVICE  TYPE      STATE      CONNECTION 
enp1s0  ethernet  connected  enp1s0
enp7s0  ethernet  disconnected  --
lo      loopback  unmanaged  -- 

Show all connection profiles:

$ nmcli con show 
NAME    UUID                                  TYPE      DEVICE 
enp1s0  1bb35a4a-ad02-4cad-978a-4a97ea9527cb  ethernet  enp1s0

Show details about a specific connection (e.g., enp1s0):

$ nmcli connection show enp1s0 

Retrieve specific parameters from a connection (e.g., IP address and DNS):

$ nmcli -g ip4.address,ip4.dns connection show enp1s0
192.168.122.169/24
192.168.122.1

Modify connection parameters, e.g., switch from DHCP to manual connection:

$ sudo nmcli con mod enp1s0 ipv4.method manual ipv4.addresses 192.168.122.169/24 ipv4.dns 192.168.122.1 ipv4.gateway 192.168.122.1

Restart the connection to enable new options:

$ sudo nmcli con down enp1s0
$ sudo nmcli con up enp1s0

Add a new connection profile:

$ sudo nmcli connection add con-name enp7s0 ifname enp7s0 type ethernet ip4 192.168.64.88/24 

nmcli also allows you to edit a connection profile interactively. This interface provides help and auto-completion via the Tab key, which guides you through the many options available. Activate the editor using nmcli connection edit CONNECTION-NAME:

$ sudo nmcli connection edit enp7s0 

===| nmcli interactive connection editor |===

Editing existing '802-3-ethernet' connection: 'enp7s0'

Type 'help' or '?' for available commands.
Type 'print' to show all the connection properties.
Type 'describe [<setting>.<prop>]' for detailed property description.

You may edit the following settings: connection, 802-3-ethernet (ethernet), 802-1x, dcb, sriov, ethtool, match, ipv4, ipv6, tc, proxy.

In the editor, you can define specific parameters with the set command. Use the Tab key auto-completion to see all available options:

nmcli> set ipv4.<PRESS TAB>
addresses            dhcp-hostname        dhcp-timeout         dns-search           may-fail             routes
dad-timeout          dhcp-hostname-flags  dns                  gateway              method               route-table
dhcp-client-id       dhcp-iaid            dns-options          ignore-auto-dns      never-default        routing-rules
dhcp-fqdn            dhcp-send-hostname   dns-priority         ignore-auto-routes   route-metric         
nmcli> set ipv4.addresses 192.168.64.90/24

Then you can print the current values:

nmcli> print ipv4.addresses 
ipv4.addresses: 192.168.64.88/24, 192.168.64.90/24

When you finish your changes, save them to the connection and quit to complete the operation:

nmcli> save
Connection 'enp7s0' (94170029-5620-4f90-ad78-704b21480b1a) successfully updated.
nmcli> quit

Finally, restart the connection to make the changes effective.

Network Manager and nmcli are a complete solution for network configuration. If you have never used it before, start with some of the basic commands and gradually move to more complex settings. Consult the man pages and examples to execute specific tasks.

Next, let's review some commands that help you with network troubleshooting, starting with name resolution.

The nslookup command

When managing connectivity issues, DNS name resolution is often a source of headaches. The nslookup utility helps you check and troubleshoot DNS name resolution.

This command is available with the bind-utils package on RHEL8 and Fedora systems. Install it with dnf:

$ sudo dnf install -y bind-utils

To quickly check the name resolution for a particular host, use nslookup with the hostname as an argument. The command uses the default DNS configuration for the name resolution:

$ nslookup redhat.com
Server:        192.168.122.1
Address:    192.168.122.1#53

Non-authoritative answer:
Name:    redhat.com
Address: 209.132.183.105

If the name resolution fails, you can use an alternative name server for the resolution by providing its address as the third argument:

$ nslookup redhat.com 192.168.0.9
Server:        192.168.0.9
Address:    192.168.0.9#53

Non-authoritative answer:
Name:    redhat.com
Address: 209.132.183.105

This information is useful for troubleshooting as it helps to identify whether the issue is local due to a specific server, or something more extensive.

You can also use it to run reverse DNS queries by providing the IP address instead of a hostname:

$ nslookup 209.132.183.105
105.183.132.209.in-addr.arpa    name = redirect.redhat.com.

The nslookup utility is a useful tool to help troubleshoot network issues related to DNS name resolution. Next, let's take a look at troubleshooting local network sockets.

The ss utility

Another common task when troubleshooting network connectivity issues is determining whether a connection is established or a particular service is available on a server.

The ss command, short for socket statistics, is a convenient tool that displays network socket information. It's the modern replacement for netstat that provides similar functionality, but includes a few extra features.

Use ss with no options to see the complete list of all established network sockets for socket types TCP, UDP, and UNIX.

$ ss
Netid    State     Recv-Q    Send-Q                                               Local Address:Port              Peer Address:Port
u_str    ESTAB     0         0                                                                * 25812                        * 25811
u_str    ESTAB     0         0                                      /run/systemd/journal/stdout 23604                        * 23603
... TRUNCATED OUTPUT
u_str    ESTAB     0         0                                                                * 22566                        * 22171
icmp6    UNCONN    0         0                                                                *:ipv6-icmp                    *:*   
icmp6    UNCONN    0         0                                                                *:ipv6-icmp                    *:*   
tcp      ESTAB     0         0                                                  192.168.122.169:ssh              192.168.122.1:45626

Since the default command displays all established network connections, this list can be extensive on a busy machine, with over a thousand entries. To help you with troubleshooting, ss provides several filtering options.

You can display socket information for specific socket types with the following command-line parameters: -t for TCP, -u for UDP, and -x for UNIX.

For example, display TCP socket information with ss -t:

$ ss -t
State           Recv-Q            Send-Q                         Local Address:Port                        Peer Address:Port
ESTAB           0                 0                            192.168.122.169:ssh                        192.168.122.1:45626

You can also filter the list by source or destination hostname or IP address. For example, here is a list of established connections for destination IP address 192.168.122.1:

$ ss dst 192.168.122.1
Netid         State         Recv-Q          Send-Q                     Local Address:Port                    Peer Address:Port
tcp           ESTAB         0               0                        192.168.122.169:ssh                    192.168.122.1:45626

These commands are useful to troubleshoot network connectivity in general or between specific hosts in the network. Another practical application for ss is to verify whether network services are listening on the local machine with the correct address and port. To verify listening sockets, use the option -l:

$ ss -l
Netid   State    Recv-Q   Send-Q                           Local Address:Port                     Peer Address:Port
nl      UNCONN   0        0                                         rtnl:NetworkManager/909                   *
nl      UNCONN   0        0                                         rtnl:systemd-resolve/1122                 *
... TRUNCATED OUTPUT
udp     UNCONN   0        0                                      0.0.0.0:hostmon                       0.0.0.0:*
udp     UNCONN   0        0                                127.0.0.53%lo:domain                        0.0.0.0:
udp     UNCONN   0        0                                    127.0.0.1:323                           0.0.0.0:*
udp     UNCONN   0        0                                         [::]:hostmon                          [::]:*
udp     UNCONN   0        0                                        [::1]:323                              [::]:*
tcp     LISTEN   0        128                                    0.0.0.0:ssh                           0.0.0.0:*
tcp     LISTEN   0        128                                    0.0.0.0:hostmon                       0.0.0.0:*
tcp     LISTEN   0        128                                       [::]:ssh                              [::]:
tcp     LISTEN   0        128                                       [::]:hostmon                          [::]:*

You can use the same filtering options described above to filter specific socket types. For example, to list all listening TCP sockets enter:

$ ss -lt
State            Recv-Q            Send-Q                       Local Address:Port                          Peer Address:Port
LISTEN           0                 128                                0.0.0.0:ssh                                0.0.0.0:*
LISTEN           0                 128                                0.0.0.0:hostmon                            0.0.0.0:
LISTEN           0                 128                                   [::]:ssh                                   [::]:
LISTEN           0                 128                                   [::]:hostmon                               [::]:

Instead of displaying the service name, like ssh or hostmon, ss can display port numbers by using the -n option:

$ ss -ltn
State             Recv-Q            Send-Q                       Local Address:Port                        Peer Address:Port
LISTEN            0                 128                                0.0.0.0:22                               0.0.0.0:
LISTEN            0                 128                                0.0.0.0:5355                             0.0.0.0:
LISTEN            0                 128                                   [::]:22                                  [::]:
LISTEN            0                 128                                   [::]:5355                                [::]:

Finally, another useful option is -p to display process related information such as user name and process ID (PID). Some services may require elevated privileges via sudo or as root to list process information:

$ sudo ss -tnlp
State      Recv-Q     Send-Q         Local Address:Port           Peer Address:Port
LISTEN     0          128                  0.0.0.0:22                  0.0.0.0:*         users:(("sshd",pid=919,fd=5))
LISTEN     0          128                  0.0.0.0:5355                0.0.0.0:*         users:(("systemd-resolve",pid=1122,fd=13))
LISTEN     0          128                     [::]:22                     [::]:*         users:(("sshd",pid=919,fd=7))
LISTEN     0          128                     [::]:5355                   [::]:*         users:(("systemd-resolve",pid=1122,fd=15))

The ss tool is an essential command in the sysadmin toolbox for network troubleshooting tasks. Next, let's take a look at tracepath to trace network connectivity between hosts.

The tracepath command

The tracepath command is a network troubleshooting tool that displays the network connectivity path between the local host and a remote host, identifying all routers used to route the traffic between them.

In case you're unable to connect to a network service in a remote host, tracepath helps you determine where the issue is.

tracepath is a replacement for traceroute, offering similar functionality. The main difference is that tracepath uses random UDP ports instead of the ICMP protocol for the trace, thus not requiring elevated privileges to run.

For the basic usage, provide the hostname or IP address of the destination host. You can also provide the option -n to display IP addresses for the routers instead of their names:

$ tracepath -n sat65server
 1?: [LOCALHOST]                      pmtu 1500
 1:  192.168.122.1                                         0.415ms 
 1:  192.168.122.1                                         0.299ms 
 2:  192.168.10.10                                         0.904ms 
 3:  192.168.88.1                                          1.127ms 
 4:  192.168.0.95                                          2.020ms
     Resume: pmtu 1500 

If tracepath cannot connect to a network hop, it displays no reply. By default, the maximum number of hops it tries is 30, which is usually enough. You can change that with the option -m.

Suppose it receives no replies after a particular hop. That is a good indicator where to go next for your troubleshooting task. tracepath information is not definitive, as the traffic could be blocked for several reasons. However, it helps you narrow down the issue and focus on the resolution efforts.

For security reasons, many routers on the Internet block traffic, so tracepath may not be as useful for tracing connectivity with Internet services. It's still useful for local network troubleshooting.

What's next?

In this article, we explored five essential network configuration and troubleshooting tools for Linux systems. These tools are powerful and provide many options that are hard to cover in a single post. We encourage you to check them out and look at documentation and man pages to see how you can incorporate them into your workflow.

There are other useful Linux network tools, such as tcpdump, nmap, and firewall-cmd. These tools are covered in other Enable Sysadmin articles.

[ Network hard to manage? Check out Network Automation for Everyone, a free book from Red Hat. ]