Previous Section  < Day Day Up >  Next Section

7.2. Network Performance Tools

This section describes the Linux network performance tools available to diagnose performance problems. We start with the tools to determine the lowest level of network performance (physical statistics) and add tools that can investigate the layers above that.

7.2.1. mii-tool (Media-Independent Interface Tool)

mii-tool is an Ethernet-specific hardware tool primarily used to configure an Ethernet device, but it can also provide information about the current configuration. This information, such as the link speed and duplex setting, can be useful when tracking down the cause of an under-performing network device.

7.2.1.1 Network I/O Performance-Related Options

mii-tool requires root access to be used. It is invoked with the following command line:


mii-tool [-v] [device]


mii-tool prints the Ethernet settings for the given device. If no devices are specified, mii-tool displays information about all the available Ethernet devices. If the -v option is used, mii-tool displays verbose statistics about the offered and negotiated network capabilities.

7.2.1.2 Example Usage

Listing 7.1 shows the configuration of eth0 on the system. The first line tells us that the Ethernet device is currently using a 100BASE-T full-duplex connection. The next few lines describe the capabilities of the network card in the machine and the capabilities that the card has detected of the network device on the other end of the wire.

Listing 7.1.

[root@nohs linux-2.6.8-1.521]# /sbin/mii-tool -v eth0

eth0: negotiated 100baseTx-FD, link ok

  product info: vendor 00:00:00, model 0 rev 0

  basic mode:   autonegotiation enabled

  basic status: autonegotiation complete, link ok

  capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD

  advertising:  100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control

  link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD


mii-tool provides low-level information about how the physical level of the etheRnet device is configured.

7.2.2. ethtool

ethtool provides similar capabilities to mii-tool for configuration and display of statistics for Ethernet devices. However, ethtool is the more powerful tool and contains more configuration options and device statistics.

7.2.2.1 Network I/O Performance-Related Options

ethtool requires root access to be used. It is invoked with the following command line:


ethtool [device]


ethtool prints out configuration information about the given Ethernet device. If no devices are provided, ethtool prints statistics for all the Ethernet devices in the system. The options to change the current Ethernet settings are described in detail in the ethtool main page.

7.2.2.2 Example Usage

Listing 7.2 shows the configuration of eth0 on the system. Although the device supports many different speed and link settings, it is currently connected to a full-duplex 1,000Mbps link.

Listing 7.2.

[root@scrffy tmp]# /sbin/ethtool eth0

Settings for eth0:

        Supported ports: [ TP ]

        Supported link modes:   10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Half 1000baseT/Full

        Supports auto-negotiation: Yes

        Advertised link modes:  10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Half 1000baseT/Full

        Advertised auto-negotiation: Yes

        Speed: 1000Mb/s

        Duplex: Full

        Port: Twisted Pair

        PHYAD: 0

        Transceiver: internal

        Auto-negotiation: on

        Supports Wake-on: g

        Wake-on: d

        Link detected: yes


ethtool is simple to run, and it can quickly provide information about an improperly configured network device.

7.2.3. ifconfig (Interface Configure)

The primary job of ifconfig is to set up and configure the network interfaces in a Linux box. It also provides rudimentary performance statistics about all the network devices in the system. ifconfig is available on almost every Linux machine that uses networking.

7.2.3.1 Network I/O Performance-Related Options

ifconfig is invoked with the following command line:


ifconfig [device]


If no device is specified, ifconfig shows statistics about all the active network devices. Table 7-1 describes the performance statistics that ifconfig provides.

Table 7-1. Performance-Specific ifconfig Statistics

Column

Explanation

RX packets

The number of packets that this device has received.

TX packets

The number of packets that this device has transmitted.

errors

The number of errors when transmitting or receiving.

dropped

The number of dropped packets when transmitting or receiving.

overruns

The number of times the network device did not have enough buffer space to send or receive a packet.

frame

The number of low-level Ethernet frame errors.

carrier

The number of packets discarded because of link media failure (such as a faulty cable).


Although primarily for network configuration, ifconfig provides a reasonable number of statistics that you can use to determine the health and performance of each of the network devices in the system.

7.2.3.2 Example Usage

Listing 7.3 shows the network performance statistics from all the devices in the system. In this case, we have an Ethernet card (eth0) and the loopback (lo) device. In this example, the Ethernet card has received ~790Mb of data and has transmitted ~319Mb.

Listing 7.3.

[ezolt@wintermute tmp]$ /sbin/ifconfig

eth0      Link encap:Ethernet  HWaddr 00:02:E3:15:A5:03

          inet addr:192.168.0.4  Bcast:192.168.0.255  Mask:255.255.255.0

          UP BROADCAST NOTRAILERS RUNNING  MTU:1500  Metric:1

          RX packets:1047040 errors:0 dropped:0 overruns:0 frame:0

          TX packets:796733 errors:12 dropped:0 overruns:12 carrier:12

          collisions:0 txqueuelen:1000

          RX bytes:829403956 (790.9 Mb)  TX bytes:334962327 (319.4 Mb)

          Interrupt:19 Base address:0x3000



lo        Link encap:Local Loopback

          inet addr:127.0.0.1  Mask:255.0.0.0

          UP LOOPBACK RUNNING  MTU:16436 Metric:1

          RX packets:102 errors:0 dropped:0 overruns:0 frame:0

          TX packets:102 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:6492 (6.3 Kb)  TX bytes:6492 (6.3 Kb)


The statistics provided by ifconfig represent the cumulative amount since system boot. If you bring down a network device and then bring it back up, the statistics do not reset. If you run ifconfig at regular intervals, you can eyeball the rate of change in the various statistics. You can automate this by using the watch command or a shell script, both of which are described in the next chapter.

7.2.4. ip

Some of the network tools, such as ifconfig, are being phased out in favor of the new command: ip. ip enables you to configure many different aspect of Linux networking, but it can also display performance statistics about each network device.

7.2.4.1 Network I/O Performance-Related Options

When extracting performance statistics, you invoke ip with the following command line:


ip -s [-s] link


If you call ip with these options, it prints statistics about all the network devices in the system, including the loopback (lo) and simple Internet transition (sit0) device. The sit0 device allows IPv6 packets to be encapsulated in IPv4 packets and exists to ease the transition between IPv4 and IPv6. If the extra -s is provided to ip, it provides a more detailed list of low-level Ethernet statistics. Table 7-2 describes some of the performance statistics provided by ip.

Table 7-2. Network Performance ip Output Statistics

Column

Explanation

bytes

The total number of bytes sent or received.

packets

The total number of packets sent or received.

errors

The number of errors that occurred when transmitting or receiving.

dropped

The number of packets that were not sent or received as a result of a lack of resources on the network card.

overruns

The number of times the network did not have enough buffer space to send or receive more packets.

mcast

The number of multicast packets that have been received.

carrier

The number of packets discarded because of link media failure (such as a faulty cable).

collsns

This is the number of collisions that the device experienced when transmitting. These occur when two devices are trying to use the network at the exact same time.


ip is a very versatile tool for Linux network configuration, and although its main function is the configuration of the network, you can use it to extract low-level device statistics as well.

7.2.4.2 Example Usage

Listing 7.4 shows the network performance statistics from all the devices in the system. In this case, we have an Ethernet card, the loopback device, and the sit0 tunnel device. In this example, the Ethernet card has received ~820Mb of data and has transmitted ~799Mb.

Listing 7.4.

[ezolt@nohs ezolt]$ /sbin/ip -s link



1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue



    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00



    RX: bytes  packets  errors  dropped overrun mcast



    4460       67       0       0       0       0



    TX: bytes  packets  errors  dropped carrier collsns



    4460       67       0       0       0       0



2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000



    link/ether 00:10:b5:59:2c:82 brd ff:ff:ff:ff:ff:ff



    RX: bytes  packets  errors  dropped overrun mcast



    799273378  920999   0       0       0       0



    TX: bytes  packets  errors  dropped carrier collsns



    820603574  930929   0       0       0       0



3: sit0: <NOARP> mtu 1480 qdisc noop



    link/sit 0.0.0.0 brd 0.0.0.0



    RX: bytes  packets  errors  dropped overrun mcast



    0          0        0       0       0       0



    TX: bytes  packets  errors  dropped carrier collsns



    0          0        0       0       0       0


Much like ifconfig, ip provides system totals for statistics since the system has booted. If you use watch (described in the next chapter), you can monitor how these values change over time.

7.2.5. sar

As discussed in previous chapters, sar is one of the most versatile Linux performance tools. It can monitor many different things, archive statistics, and even display information in a format that is usable by other tools. sar does not always provide as much detail as the area-specific performance tools, but it provides a good overview.

Network performance statistics are no different. sar provides information about the link-level performance of the network, as do ip and ifconfig; however, it also provides some rudimentary statistics about the number of sockets opened by the transport layer.

7.2.5.1 Network I/O Performance-Related Options

sar collects network statistics using the following command:


sar [-n DEV | EDEV | SOCK | FULL ] [DEVICE] [interval] [count]


sar collects many different types of performance statistics. Table 7-3 describes the command-line options used by sar to display network performance statistics.

Table 7-3. sar Command-Line Options

Option

Explanation

-n DEV

Shows statistics about the number of packets and bytes sent and received by each device.

-n EDEV

Shows information about the transmit and receive errors for each device.

-n SOCK

Shows information about the total number of sockets (TCP, UDP, and RAW) in use.

-n FULL

Shows all the network statistics.

interval

The length of time between samples.

count

The total number of samples to take.


The network performance options that sar provides are described in Table 7-4.

Table 7-4. sar Network Performance Statistics

Option

Explanation

rxpck/s

The rate of packets received.

txpck/s

The rate of packets sent.

rxbyt/s

The rate of bytes received.

txbyt/s

The rate of bytes sent.

rxcmp/s

The rate of compressed packets received.

txcmp/s

The rate of compressed packets sent.

rxmcst/s

The rate of multicast packets received.

rxerr/s

The rate of receive errors.

txerr/s

The rate of transmit errors.

coll/s

The rate of Ethernet collisions when transmitting.

rxdrop/s

The rate of received frames dropped due to Linux kernel buffer shortages.

txdrop/s

The rate of transmitted frames dropped due to Linux kernel buffer shortages.

txcarr/s

The rate of transmitted frames dropped due to carrier errors.

rxfram/s

The rate of received frames dropped due to frame-alignment errors.

rxfifo/s

The rate of received frames dropped due to FIFO errors.

txfifo/s

The rate of transmitted frames dropped due to FIFO errors.

totsck

The total number of sockets in use.

tcpsck

The total number of TCP sockets in use.

udpsck

The total number of UDP sockets in use.

rawsck

The total number of RAW sockets in use.

ip-frag

The total number of IP fragments.


Considering all the statistics that sar can gather, it really does provide the most system-level performance statistics in a single location.

7.2.5.2 Example Usage

In Listing 7.5, we examine the transmit and receive statistics of all the network devices in the system. As you can see, the eth0 device is the most active. In the first sample, eth0 is receiving ~63,000 bytes per second (rxbyt/s) and transmitting ~45,000 bytes per second (txbyt/s). No compressed packets are sent (txcmp) or received (rxcmp). (Compressed packets are usually present during SLIP or PPP connections.)

Listing 7.5.

[ezolt@wintermute sysstat-5.0.2]$ sar -n DEV 1 2

Linux 2.4.22-1.2174.nptlsmp (wintermute.phil.org)      06/07/04



21:22:29  IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s  rxcmp/s  txcmp/s rxmcst/s

21:22:30     lo      0.00      0.00      0.00      0.00     0.00     0.00     0.00

21:22:30   eth0     68.00     65.00  63144.00  45731.00     0.00     0.00     0.00



21:22:30  IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s  rxcmp/s  txcmp/s  rxmcst/s

21:22:31     lo      0.00      0.00      0.00      0.00     0.00     0.00      0.00

21:22:31   eth0     80.39     47.06  45430.39  30546.08     0.00     0.00      0.00



Average:  IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s  rxcmp/s  txcmp/s  rxmcst/s

Average:     lo      0.00      0.00      0.00      0.00     0.00     0.00      0.00

Average:   eth0     74.26     55.94  54199.50  38063.37     0.00     0.00      0.00


In Listing 7.6, we examine the number of open sockets in the system. We can see the total number of open sockets and the TCP, RAW, and UDP sockets. sar also displays the number of fragmented IP packets.

Listing 7.6.

[ezolt@wintermute sysstat-5.0.2]$ sar -n SOCK 1 2

Linux 2.4.22-1.2174.nptlsmp (wintermute.phil.org)    06/07/04

21:32:26    totsck   tcpsck   udpsck   rawsck   ip-frag

21:32:27       373      118        8        0         0

21:32:28       373      118        8        0         0

Average:       373      118        8        0         0


sar provides a good overview of the system's performance. However, when we are investigating a performance problem, we really want to understand what processes or services are consuming a particular resource. sar does not provide this level of detail, but it does enable us to observe the overall system network I/O statistics.

7.2.6. gkrellm

gkrellm is a graphical monitor that enables you to keep an eye on many different system performance statistics. It draws charts of different performance statistics, including CPU usage, disk I/O, and network usage. It can be "themed" to change its appearance, and even accepts plug-ins to monitor events not included in the default release.

gkrellm provides similar information to sar, ip, and ipconfig, but unlike the other tools, it provides a graphical view of the data. In addition, it can provide information about the traffic flowing through particular UDP and TCP ports. This is the first tool that we have seen that can show which services are consuming different amounts of network bandwidth.

7.2.6.1 Network I/O Performance-Related Options

gkrellm is invoked using the following command line:


gkrellm


None of gkrellm's command-line options configure the statistics that it monitors. You do all configurations graphically after gkrellm is started. To bring up the configuration screen, you can either right-click the gkrellm's title bar and select Configuration, or just press F1 when your cursor is in any area of the window. This brings up a configuration window (see Figure 7-1).

Figure 7-1.


Figure 7-2 shows the network configuration window. It is used to configure which statistics and which devices are shown in the final gkrellm output window.

Figure 7-2.


You can configure gkrellm to monitor the activity on a particular range of TCP ports. Doing so enables you to monitor the exact ports used by services such as HTTP or FTP and to measure the amount of bandwidth that they are using. In Figure 7-2, we have configured gkrellm to monitor the ports used by the bittorrent (BT) P2P application and the Web server (HTTP).

gkrellm is a flexible and powerful graphical performance-monitoring tool. It enables you to see how the system is currently performing and how its performance changes over time. The most difficult aspect of using gkrellm is reading the small default text. However, the appearance of gkrellm can be easily themed, so presumably, this could be easily fixed.

7.2.6.2 Example Usage

As stated previously, gkrellm can monitor many different types of events. In Figure 7-3, we pruned the output so that only statistics relevant to network traffic and use is displayed.

Figure 7-3.


As you can see in Figure 7-3, the top two graphs are the bandwidth used for the ports (BT and HTTP) that we set up in the configuration section, and the bottom two graphics are the statistics for each of the network devices (eth0 and lo). There is a small amount of bittorrent (BT) traffic, but no Web server traffic (HTTP). The Ethernet device eth0 had some large activity in the past, but is settling down now. The lighter shade in the eth0 indicates the number of bytes received, and the darker shade indicates the number of bytes transmitted.

gkrellm is a powerful graphical tool that makes it easy to diagnose the status of the system at a glance.

7.2.7. iptraf

iptraf is a real-time network monitoring tool. It provides a large number of modes to monitor network interfaces and traffic. iptraf is a console application, but its user interface is a cursor-based series of menus and windows.

Like the other tools mentioned previously in this chapter, it can provide information about the rate at which each network device is sending frames. However, it can also display information about the type and size of the TCP/IP packet and about which ports are being used for network traffic.

7.2.7.1 Network I/O Performance-Related Options

iptraf is invoked with the following command line:


iptraf [-d interface] [-s interface] [-t <minutes>]


If iptraf is called with no parameters, it brings up a menu that enables you to select the interface to monitor and type of information that you want to monitor. Table 7-5 describes the command-line parameters that enable you to see the amount of network traffic on a particular interface or network service.

Table 7-5. iptraf Command-Line Options

Option

Explanation

-d interface

Detailed statistics for an interface including receive, transmit, and error rates

-s interface

Statistics about which IP ports are being used on an interface and how many bytes are flowing through them

-t <minutes>

Number of minutes that iptraf runs before exiting


iptraf has many more modes and configuration options. Read its included documentation for more information.

7.2.7.2 Example Usage

iptraf creates a display similar to Figure 7-4 when it is invoked with the following command:


[root@wintermute tmp]# iptraf -d eth0 -t 1


Figure 7-4.


This command specifies that iptraf should display detailed statistics about Ethernet device eth0 and exit after it has run for 1 minute. In this case, we can see that 186.8kbps are received and 175.5kbps are transmitted by the eth0 network device.

The next command, whose results are shown in Figure 7-5, asks iptraf to show information about the amount of network traffic from each UDP or TCP port. iptraf was invoked with the following command:

Figure 7-5.



[root@wintermute etherape-0.9.0]# iptraf -s eth0 -t 10


Because the TCP or UDP ports of well-known services are fixed, you can use this to determine how much traffic each service is handling. Figure 7-5 shows that 29kb of HTTP data has been sent from eth0 and 25kb has been received.

Because iptraf is a console-based application, it doesn't require an X server or X server libraries. Even though iptraf cannot be controlled with a mouse, it is easy to use and configurable.

7.2.8. netstat

netstat is a basic network-performance tool that is present on nearly every Linux machine with networking. You can use it to extract information about the number and types of network sockets currently being used and interface-specific statistics regarding the number of UDP or TCP packets flowing to and from the current system. It also enables you to trace the owner of a socket back to a particular process or PID, which can prove useful when trying to determine the application responsible for network traffic.

7.2.8.1 Network I/O Performance-Related Options

netstat is invoked with the following command line:


netstat [-p] [-c] [–interfaces=<name>] [-s] [-t] [-u] [-w]


If netstat is called without any parameters, it shows information about system-wide socket usage and displays information about both Internet and UNIX domain sockets. (UNIX domain sockets are used for interprocess communication on the local machine, but do not indicate network traffic.) To retrieve all the statistics that netstat is capable of displaying, you must run it as root. Table 7-6 describes the command-line options of netstat that modify the types of network statistics that netstat displays.

Table 7-6. netstat Command-Line Options

Option

Explanation

-p

Displays the PID/program name responsible for opening each of the displayed sockets

-c

Continually updates the display of information every second

--interfaces=<name>

Displays network statistics for the given interface

--statistics|-s

IP/UDP/ICMP/TCP statistics

--tcp|-t

Shows only information about TCP sockets

--udp|-u

Shows only information about UDP sockets.

--raw|-w

Shows only information about RAW sockets (IP and ICMP)


netstanetstat also accepts some command-line options not described here. See the netstat man page for more details.

7.2.8.2 Example Usage

Listing 7.7 asks netstat to show the active TCP connections and to continually update this information. Every second, netstat displays new TCP network statistics. netstat does not enable you to set the length of time that it will monitor, so it will only stop if it is killed or interrupted (Ctrl-C).

Listing 7.7.

[root@wintermute ezolt]# netstat -t -c

Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address          Foreign Address         State

tcp        0      0 192.168.0.4:1023       fas.harvard.edu:ssh     ESTABLISHED

tcp        0      0 192.168.0.4:32844      216.239.39.147:http     TIME_WAIT

tcp        0      0 192.168.0.4:32843      216.239.39.147:http     TIME_WAIT

tcp        0      0 192.168.0.4:32853      skaiste.elekta.lt:http  ESTABLISHED

Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address          Foreign Address         State

tcp        0      0 192.168.0.4:1023       fas.harvard.edu:ssh     ESTABLISHED

tcp        0      0 192.168.0.4:32844      216.239.39.147:http     TIME_WAIT

tcp        0      0 192.168.0.4:32843      216.239.39.147:http     TIME_WAIT

tcp        0      0 192.168.0.4:32853      skaiste.elekta.lt:http  ESTABLISHED


Listing 7.8 asks netstat to once again print the TCP socket information, but this time, we also ask it to display the program that is responsible for this socket. In this case, we can see that SSH and mozilla-bin are the applications that are initiating the TCP connections.

Listing 7.8.

[root@wintermute ezolt]# netstat -t -p

Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address       Foreign Address        State       PID/Program name

tcp        0      0 192.168.0.4:1023    fas.harvard.edu:ssh    ESTABLISHED 1463/ssh

tcp        0      0 192.168.0.4:32844   216.239.39.147:http    TIME_WAIT -

tcp        0      0 192.168.0.4:32843   216.239.39.147:http    TIME_WAIT -

tcp        0      0 192.168.0.4:32853   skaiste.elekta.lt:http ESTABLISHED 1291/mozilla-bin


Listing 7.9 asks netstat to provide statistics about the UDP traffic that the system has received since boot.

Listing 7.9.

[root@wintermute ezolt]# netstat -s -u

Udp:

    125 packets received

    0 packets to unknown port received.

    0 packet receive errors

    152 packets sent


Listing 7.10 asks netstat to provide information about the amount of network traffic flowing through the eth0 interface.

Listing 7.10.

[root@wintermute ezolt]# netstat –interfaces=eth0

Kernel Interface table

Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP TX-OVR Flg

eth0       1500   0    52713      0      0      0   13711      1      0      1 BNRU


netstat provides a great number of network performance statistics about sockets and interfaces in a running Linux system. It is the only network-performance tool that maps the sockets used back to the PID of the process that is using it, and is therefore very useful.

7.2.9. etherape

etherape (a pun on the Windows-based network tool etherman) provides a visualization of the current network traffic. By default, it observes all the network traffic flowing on the network, not just those packets that the current machine is sending or receiving. However, it can be configured to only display network information for the current machine.

etherape is a little rough around the edges (in interface and documentation), but it provides a unique visual insight into how the network is connected, what types of services are being requested, and which nodes are requesting them. It creates a graph whose nodes represent the systems on the network. The nodes that are communicating have lines connecting them that increase in size as more network traffic flows between them. As a particular system's network usage increases, the size of the circle representing that system also increases. The lines connecting the different systems are colored differently depending on the protocols they are using to communicate with each other.

7.2.9.1 Network I/O Performance-Related Options

etherape uses the libpcap library to capture the network packets and, as a result, it must be run as root. etherape is invoked using the following command line:


etherape [-n] [-i <interface name>]


Table 7-7 describes some of the command-line options that change the interface that etherape monitors and whether resolved host names are printed on each node.

Table 7-7. etherape Command-Line Options

Option

Explanation

-n, --numeric

Shows only the IP number of the hosts rather than the resolved names

-i, --interface=<interface name>

Specifies the interface that will be monitored


All in all, etherape's documentation is rather sparse. The etherape man page describes a few more command lines that change its appearance and behavior, but the best way to learn it is to try it. In general, etherape is a great way to visualize the network.

7.2.9.2 Example Usage

Figure 7-6 shows etherape monitoring a relatively simple network. If we match up the color of the protocol to the color of the biggest circle, we see that this node is generating a high amount of SSH traffic. From the figure, it can be difficult to determine which node is causing this SSH traffic. Although not pictured, if we double-click the big circle, etherape creates a window with statistics pertaining to the node responsible for the traffic. We can use this to investigate each of the generators of network traffic and investigate their node names.

Figure 7-6.


etherape's output is periodically updated. As network traffic changes, its graph is updated. It can be fascinating just to watch the network traffic flow and see how it is used and changes over time.

    Previous Section  < Day Day Up >  Next Section