Network Analysis Using Wireshark Cookbook: Chapter No. 13 "Troubleshooting Bandwidth and Delay Problems"

Network Analysis Using Wireshark Cookbook
Yoram Orzach
Chapter No. 13 "Troubleshooting Bandwidth and Delay Problems"
In this package, you will find:

A Biography of the author of the book A preview chapter from the book, Chapter NO.13 "Troubleshooting Bandwidth and Delay Problems" A synopsis of the books content Information on where to buy this book
About the Author

Yoram Orzach gained his Bachelor's degree in Science from the Technion in Haifa, Israel, and worked in Bezeq as a systems engineer in the fields of transmission and access networks from 1991 to 1995. In 1995, he joined Netplus from the Leadcom group as technical manager, and since 1999 he has worked as the CTO of NDI Communications (www.ndi-com.com), involved in the design, implementation, and troubleshooting of data communication networks worldwide. Yoram's experience is both with corporate networks, service providers, and Internet service provider's networks, and among his customers are companies such as Comverse, Motorola, Intel, Ceragon networks, Marvel, HP, and others. Yoram's experience is in design, implementation, and troubleshooting, along with training for R&D, engineering, and IT groups.
For More Information: www.packtpub.com/network-analysis-using-wireshark-cookbook/book
Network Analysis Using Wireshark Cookbook

Wireshark has long become the market standard for network analysis, and with the growth of the Internet and TCP/IP-based networks, it became very popular for network analysis, troubleshooting, as well as for R&D engineers to understand what is actually running over the network and what are the problems that we face. This book is written from a practical point of view. The first part of it, from Chapter 1, Introducing Wireshark, to Chapter 6, Using the Expert Infos Window, describes the Wireshark software and how to work with it. This includes how to start it, where to locate it in the network, how to work with statistical tools, and how to use the Expert system. The second part, from Chapter 7, Ethernet, LAN Switching, and Wireless LAN, to Chapter 14, Understanding Network Security, describes how to use it for the analysis and troubleshooting of common networking protocols; among them, the TCP/IP protocol stack with emphasis on TCP performance issues, common Internet protocols such as HTTP, SMTP, POP and DNS, databases, Citrix and Microsoft Terminal Server, IP telephony, and multimedia applications. The last chapter is about network security. It describes how to locate security breaches and other problems in your network. As the name of the book implies, this is a Cookbook. It is a list of effective, targeted recipes of how to analyze networks. Every recipe comes with a specific issue, how to use Wireshark for it, where to look and what to look for, and what is the reason for what you see. To complete the picture, every recipe provides the theoretical foundations of the subject, in order to give the reader the required theoretical background. You will see many examples in the book, and all of them are real cases. Some of them took me minutes to solve, some hours, and some of them took many days. There is one thing common to all of them: work systematically, use the proper tools, try to get inside the head of the application writer, and like someone told me once, "Try to think like the network". Do this, use Wireshark, and you will get results. The purpose of this book is to try and get you there. Have fun!
What This Book Covers

Chapter 1, Introducing Wireshark, starts with introducing Wireshark, explaining where to locate it for effective network analysis. We will learn how to configure the basic parameters, the start window, the time values, and the coloring rules; and most importantly, we will learn how to use the Preferences window. Chapter 2, Using Capture Filters, explains how to use capture filters which are used in order to define what data will be captured. This chapter explains how to configure these filters and how to use them in order to capture only the desired data. Chapter 3, Using Display Filters, explains how to configure display filters which are used in order to display only the desired data, after the data is captured. This chapter explains how to configure these filters and how they can assist us in network troubleshooting. Chapter 4, Using Basic Statistics Tools, explains how to work with the basic Wireshark statistical features, starting from the simple tables that provides us with "who is talking" information, conversations and HTTP statistics, and others. Chapter 5, Using Advanced Statistics Tools, explains how to work with the advanced Wireshark statistical features, including the IO graphs and TCP stream graphs that provides us with powerful capabilities for network and application performance analysis. Chapter 6, Using the Expert Infos Window, explains how to work with the Expert system, which is a powerful tool that pinpoints various types of events, such as TCP retransmissions, zero-window, low TTL and routing loops, out-of-order segments, and other events that might influence the behavior of our network. Chapter 7, Ethernet, LAN Switching, and Wireless LAN, explains the Ethernet protocol and LAN switching, along with problems that might occur in this layer. It also focuses on Wireless LAN (WiFi), how to test it, and how to resolve problems in these networks. Chapter 8, ARP and IP Analysis, explains about ARP, IP, and how to analyze IP connectivity and routing problems. This chapter also explains how to find duplicate IP addresses, DHCP problems, and other related issues. Chapter 9, UDP/TCP Analysis, focuses on layer 4 protocols, TCP, and UDP, with emphasis on TCP performance issues. It provides recipes for allocation of TCP performance problems, such as retransmission, duplicate ACKs, sliding-window problems such as window-full and zero-window, resets, and other related issues. Chapter 10, HTTP and DNS, focuses on DNS, HTTP, and HTTPs. In this chapter, we will see how they work and what can go wrong in these protocols.
Chapter 11, Analyzing Enterprise Applications', Behavior, talks about other applications such as FTP, mail protocols, terminal services, and databases. We will see how they are affected by network problems and how we can solve network-related problems in these applications. Chapter 12, SIP, Multimedia, and IP Telephony, is about voice and video over IP, including recipes for finding VoIP SIP connectivity problems, RTP/RTCP performance problems, and video problems such as picture freezing and bad picture quality. Chapter 13, Troubleshooting Bandwidth and Delay Problems, provides recipes for finding problems caused by low-bandwidth, high-delay, and high-jitter networks. The chapter explains the behavior of TCP over high-delay, high-jitter networks, and what we can do in order to improve this behavior. Chapter 14, Understanding Network Security, focuses on TCP/IP-based network security, and it includes recipes for finding network scanning, SYN attacks, DOS/DDOS, and other attacks that can harm the network. This chapter provides recipes for finding various attack patterns and what causes them. Appendix, Links, Tools, and Reading, provides references to some useful links from which you can get further information about Wireshark: learning sources, additional software, and so on.
Troubleshooting Bandwidth and Delay Problems

In this chapter we have the following recipes:

13
Measuring total bandwidth on a communication link Measuring bandwidth and throughput per user and per application over a network connection Monitoring jitter and delay using Wireshark Discovering delay/jitter-related application problems
Introduction
When measuring communication lines, there are four major parameters that we should be aware of: bandwidth, delay, jitter, and packet loss. While there are applications that require high bandwidth, there are other applications that are more sensitive to delay and jitter. Packet loss can inuence all types of applications, but there are applications that are more sensitive to it and some that are less. In this chapter we will learn how to measure these parameters, how to check for network problems caused by it, and how to solve them when possible.
Measuring total bandwidth on a communication link

In this recipe, we will see how to measure the total bandwidth over a communication line. The rst thing of course is to verify the communication line with the service provider. Check whether it is a symmetric or an asymmetric line, and if it is asymmetric, check what the bandwidth is in both directions.
Getting ready
There are two cases that you might need to test:
When you measure a communication line between two ofces: in this case connect your laptop (or any PC on the network) to the LAN, and verify whether you have a server or another PC on the other side of the line When you measure a communication line to the Internet, make sure you have a testing server on the Service Provider (SP) side or on the Internet Service Provider (ISP) side
How to do it...
To check the bandwidth on a communication line, follow these steps: 1. Ask for the following details: 1. Ask the SP what the line bandwidth is. 2. If it is a line to the Internet, in addition to the preceding step ask the ISP what is the bandwidth to the Internet. 2. Locate a server, a PC, or a laptop on the remote location.
When using a PC or laptop for the test, don't forget that the PC itself should be strong enough to generate the traffic. A standard Windows 7 is able to generate around 200 Mbps per TCP connection, and when opening several connections, you can get into other limitations such as disk performance and so on. Therefore, it is recommended to try the transfer first on a LAN, where there are no bandwidth limits (practically), and only then to test the SP or the ISP lines. If you are using FTP, use an efficient one (FileZilla, for example). The best way of course is to use test equipment, if it's available. Dedicated test equipments are available from many vendors such as VeEX, Fluke Networks, and IXIA.
368
Chapter 13
In case you want to test the bandwidth between two sites, download and then upload a big file between nodes numbered as 1 and 2 or between nodes numbered as 1 and 3. A file big enough should load the line for a significant amount of time, that is, a minute or more. For example, if you want to test a 10 Mbps (Megabits per second) line, use a file of at least 10/8 = 1.25 MB (Megabytes). In case you want to test your connection to the Internet, usually you can perform the test on your service provider (numbered as 1 to 4 in the following diagram), and then to your Internet service provider (numbered as 1 to 5 in the following diagram).
If possible, it is better to use the IP or UDP test, since when you copy a file, it is done over TCP, so you can get into TCP issues that influence the test. For this purpose, use Iperf or another testing tool that can generate IP or UDP traffic.
In the following illustration, you can see two local networks connected via a Service Provider (SP) line. The site on the left is connected to the Internet through a rewall. The connection to the Internet goes through the Service Provider (SP, Server 4) to the Internet Service Provider (ISP, Server 5).
ISP Test Server
ISP Network SP Test Server

4
SP Network
FW
SP Network
Server
1 2 3
Server
369
Troubleshooting Bandwidth and Delay Problems Follow these steps to measure the bandwidth over the communication lines: 1. Use Wireshark Statistics | IO Graphs for the test.
Don't forget that Wireshark has its own limitations when working with high bandwidth lines. In this case, you can configure it to use multiples files. Personally, I prefer to use other tools (Omnipeek, for example) when monitoring lines of 200-300 Mbps and higher.
2. When testing your enterprise network, you can use software tools such as Iperf (http://sourceforge.net/projects/iperf/). Following are the steps to measure network bandwidth with IPerf: 1. Install Iperf on both ends of the connection. 2. Congure one side as a client, and the other side as a server. 3. Start the test and use I/O Graphs to verify that you have a stable bandwidth.
When downloading or uploading a file, do it with a single large file and not a directory of multiple files. When transferring many small-sized files, it will take time to open and transfer each one of them, so the test will not give good results.
When getting less bandwidth than expected, perform the following steps: 1. When getting a value up to around 5% more or less than expected, it can be due to the reasons mentioned in the There's more... section in this recipe. Check the congurations and the technology that the line is running on (SDH/SONet, Carrier Ethernet, and so on) 2. If you test the line with le copy, and in the IO graphs see sawtooth, there might be errors on the line. Check TCP retransmissions, and then check for errors in the switch/router port connected to the service provider.
To check switch or router port statistics, you can use console or telnet to connect to it and use the switch or router commands (for example, show interface commands in Cisco). You can also use SNMP management software or any MIB browser and browse the IfInErrors and InOutErrors objects.
370
Chapter 13 3. If you see a degradation of 80 to 90 percent of what you had expected (for example, you test a line of 100 Mbps and get 10 to 20 Mbps); in most of the cases, it is a duplex-mismatch problem. As shown in the How it works... section of this recipe. It isn't common, but it can also be that your service provider has a conguration problem. Check it with them. If none of the preceding cases are true, it can be that this is the reason.
How it works...
First, there are two different denitions; it is important to distinguish between:
Bandwidth: This is the total bits per second that can be transferred over a communications line Throughput: This is the effective application bytes per second that is transferred between the two ends of a connection
To check the bandwidth of a communication line, you can ask the service provider for the line details, or you can simply transfer some trafc over it, use Wireshark or SNMP tool, and see what you get. Most of the cases in which a duplex mismatch problem occurs is when you connect using Ethernet on one side with 100 Mbps full duplex, and the other side congured to auto-negotiate.
No. Setup What You Get Status Port A Port B Port A Port B V 100FD 100FD 100FD 100FD X 100FD AUTO 100FD 100HD X AUTO 100FD 100HD 100FD V AUTO AUTO 1000FD 1000FD V: OK, X: Mismatch
Port A Port B
1 2 3 4
As you see in the diagram, when you connect a device (a router in this example) to a switch, when both sides are manually congured, for example, to 100 Mbps Full Duplex (FDX), the intended conguration will take place (numbered 1 in the preceding diagram). When you congure both sides to auto-negotiation (numbered 4 in the preceding diagram), it will also be ne, and will be automatically set to 1 Gbps (in the case of gigabit adapters).
371
Troubleshooting Bandwidth and Delay Problems In the case when one side is congured to 100 FDX and the other side to auto negotiate, the auto negotiate will be automatically set to 100 Mbps Half-Duplex (HDX). In this case, when one side is set to HD and the other to FD, many packets will be lost, and you will experience signicant degradation in performance (numbered 2 and 3 in the preceding diagram).
There's more...
When we buy a line at a certain bandwidth, it can be that we'll get a little bit more or less of what we've bought. For example, when we buy 10 Mbps line, and the line runs over the Synchronous Digital Hierarchy (SDH) or Synchronous Optical Network (SONet) line; the 10 Mbps is made of 5 VC-12s, which is 5*2.176 Mbps, so the total bandwidth will be 10.88 Mbps. On the other hand if, for example, we use site-to-site VPN over the Internet, and the line is 10 Mbps, even if we have a very good Internet connection (for example, when the two ends are connected to the same ISP), the encryption mechanisms of the VPN itself can take 5 to 10 percent of the line, and when measuring it, you will get somewhere between 9.0 to 9.5 Mbps. In this case, for example, when you transfer a le over the line, you will see that the line is loaded with 10 Mbps (that is, the bandwidth), while what is left for the le copy is usually between 9.0 to 9.5 Mbps (that is, the throughput).
Measuring bandwidth and throughput per user and per application over a network connection
In many cases, we need to know not only the total bandwidth of a connection, (communication line or on a server port), but also who exactly are the consumers, that is from which IP addresses and port numbers the trafc is coming. In this recipe, we will see how to measure it. In order to see this, you can use proprietary tools that collect the data from the switch (RMON1, RMON2, sFlow) or router (Cisco Netow or Juniper Jow), or to use Wireshark with port mirror to the communication link, and this is what we'll learn in this recipe.
Getting ready
For using Wireshark to get trafc distribution, connect a laptop with a port mirror to the link you wish to monitor and start packet capture. You can also use the Tshark command from the CLI.
372
Chapter 13
How to do it...
For basic statistics on users and applications that are using the communications link, perform the following steps:
For general statistics: 1. From the Statistics menu, choose Conversations. 2. In the Conversations window, you see the statistics on the total number of packets captured until now. 3. You can also use graphical tools such as Compass (Chapter 11, Analyzing Enterprise Applications, Behavior).
For ow analysis, use IO graphs with lters on IP addresses and/or port numbers: 1. From the Statistics menu, select IO Graphs. 2. In the IO graphs window (Chapter 5, Using Advanced Statistics Tools), configure IP and port numbers and display filters for the applications that you wish to monitor.
For continuous monitoring, use Wireshark with multiple les with ring buffer, or use tools such as Netow or Jow for router monitoring.
How it works...
With Wireshark, like we learned in Chapter 1, Introducing Wireshark, we capture data and analyze it. In Netow, Jow, and applications that collect data from the router, the router periodically sends the collected data to the management console that analyzes it. In Remote Monitoring 1 (RMON1) and Remote Monitoring 2 (RMON2), when the end switch supports it, you access the data with the SNMP software that reads from the RMON1/RMON2 MIB. While RMON1 provides you layer 1 to 2 statistics, RMON2, when implemented provides you layer 3 to 4 statistics. The main standards of RMON were published in RFCs 2613, 2819, 3577, and 4502. In various applications and devices such as rewalls, Intrusion Detection Systems (IDS), Deep Packet Inspection (DPI) devices, and WAN Accelerators, you will get the data from the monitored device.
373
See also
Additional data on these applications can be found at: Cisco Netow: http://www.cisco.com/en/US/products/ps6601/products_ios_ protocol_group_home.html
http://www.ietf.org/rfc/rfc3954.txt
For Juniper Jow:

http://www.juniper.net/techpubs/software/erx/junose82/swconfig-ipservices/html/ip-jflow-stats-config2.html
sFlow:
http://www.ietf.org/rfc/rfc3176.txt
Various applications can be located in: For switch monitoring:

http://www.sflow.org/index.php http://tools.ietf.org/html/rfc3176
Monitoring jitter and delay using Wireshark

Jitter and delay are characteristics that can signicantly inuence various network applications. For monitoring jitter and delay on a communication line, you can use simple or graphical Ping tools that will show you the line characteristics. Wireshark on the other hand does not measure the end-to-end delay but the inuence that it has on the network trafc, that is inter-frame delay and how it inuences applications. In this recipe, we will see how to use Wireshark tools for monitoring these parameters, and in the next recipe we will see how to discover problems caused by them.
Getting ready
For monitoring delay on a communication line, rst use the ping command to get the feeling of the line, and then congure port mirror to the port you want to monitor.
374
Chapter 13
How to do it...
To monitor inter-frame delay: 1. From Statistics, select IO Graph. 2. For monitoring time between frames in a specic stream of data: 1. Click on a packet in the TCP or UDP stream. 2. Click on Follow TCP Stream or Follow UDP stream. 3. Copy the displayed filter string that showed up (numbered 1 in the next screenshot). 3. From statistics open IO Graph. 4. In IO Graph, in the Y Axis part (bottom-right side of the window), select Advanced... (numbered 2 in the following diagram). 5. Copy the TCP stream number (numbered 1 in the following diagram) to the Filter eld in the IO Graph (numbered 3 in the following diagram).
6. Select AVG(*) (numbered 4 in the preceding diagram). 7. Congure the lter frame.time_delta_displayed (numbered 5 in the preceding diagram).
8. In the graph (numbered 6 in the preceding diagram), you see the time between frames in milliseconds.
375
Troubleshooting Bandwidth and Delay Problems 9. By navigating to Statistics | TCP Stream Graph | Round Trip Time Graph, you will get the same results as shown in the following diagram:
10. In the diagram, we see that the Round Trip Time (RTT) varies between values that are lower than 10 ms and up to 200-300 ms. 11. To measure delays in layer 4, use the TCP lter tcp.analysis.ack_rtt that will give you the time that it takes to acknowledge every received packet.
How it works...
The software simply captures packets over the line, and shows you the time difference between them. It is important to notice that there is a delay or jitter, but we will not see where it is coming from. Delay is the time that it takes a packet to get from one end of the network to the other. It is usually referred to as RTT. Delay can be measured with simple Ping or graphical Ping tools. Delay is measured in seconds milliseconds (ms), microseconds (s), and so on.
376
Chapter 13 Jitter in IP networks measure the variations in delay. For example, if we have an average delay of 100 ms, and it varies between 80 ms and 120 ms, the jitter is 20 percent.
There's more...
Graphical Ping tools are available for free on many websites. You can use, for example,
http://www.colasoft.com/download/products/download_ping_tool.php.
Discovering delay/jitter-related application problems

Jitter and delay can inuence various types of applications. In applications that run over TCP, high delay reduces the effective throughput that can be sent and high jitter can cause packet losses and retransmissions. In multimedia applications that run over RTP, which runs over UDP, high jitter and delay can cause severe disturbances in the voice and video quality. In this recipe, we will get into the details: the inuence of behavior on TCP, and how it can inuence the application behavior. RTP over UDP behavior was discussed in Chapter 12, SIP, Multimedia, and IP Telephony.
Getting ready
When you ping a remote site and get high delays, and in the Wireshark you see many retransmissions, it can be because of high network or applications delay. Connect the Wireshark to the network and congure port mirror to the link that you test. The purpose of this recipe is to check whether the TCP retransmissions and duplicate ACKs are due to delay and jitter or other problems.
How to do it...
When experiencing many TCP retransmissions, perform the following tests: 1. Check whether retransmissions are coming from the same application or from the same IP address. In this case, it is a slow application or a slow device and probably not a network delay issue. If retransmissions are distributed between various applications and devices, it can be because of unstable line that causes network delays.
377
Troubleshooting Bandwidth and Delay Problems 2. Congure a display lter tcp.analysis.retransmissions (numbered 1 in the following diagram). A list of all retransmissions in the packet list will appear.
3. Down the packet details pane, expand the TCP Analysis Flags, and you will get:
The time since the original packet is retransmitted (numbered 2 in the preceding diagram). In this case, 0.225003000 seconds. The packet that is retransmitted (numbered 3 in the preceding diagram). In this case, packet number 1779.
4. Usually the Retransmission Time Out (RTO) timer will be around 0.2 seconds for local connections, and up to 0.3 to 0.4 seconds for international connections. Start with assuming 0.2 seconds. Refer to the How it works... section in this recipe for explanation about the RTO mechanism. 5. To check TCP delay over a connection, use IO Graphs with the following lters, as presented in the next diagram:
tcp.stream eq <the stream number> to get to the stream number right-click on a packet and select Follow TCP stream. frame.time_delta to see the time difference between frames in the TCP stream. This parameter actually shows inter-frame delta in layer 2, but since it is shown only for the stream, it will show us inter-frame deltas in a specific TCP stream.
378
Chapter 13
You get a graph that shows a very stable connection, with delays that are lower than 20 ms, except for the two increases in delay in time 250s (250 seconds since the beginning of the capture), that causes retransmissions. 6. When we add the tcp.analysis.ack_rtt lter on the same connection, we see the delays between TCP packets and ACKs.
379
Troubleshooting Bandwidth and Delay Problems In the following diagram, you see a graph which is an example for delays not due to line delay issues:
7.
You can see here that I've congured:

The Advance option in the Y Axis The Filter field: tcp.stream eq 0 (numbered 1 in the preceding diagram) to present a single stream The calculation AVG(*) to see the average (2) You can also configure MAX(*) and the filter tcp.analysis.ack_rtt to see the time to acknowledge every TCP sequence
What we've got is the time that it took to acknowledge every TCP packet. 8. Now, let's congure IO Graphs to see if there are TCP retransmissions, and why they happen:
Use the same IO Graph with Advance in Y Axis, and configure the second line. The Filter field: tcp.stream eq 0 (numbered 1 in the preceding diagram) to present a single stream. The calculation COUNT FRAMES(*) to see the average. The filter tcp.analysis.retransmissions to see the time to acknowledge every TCP packet.
380
Chapter 13
9. We can see from here that all retransmissions happened when there was a signicant increase in the delay, so it is a delay problem. We cannot say from here if the delay is from the network, from the end device or from the application, so for isolating the problem check how retransmissions are distributed (see Chapter 9, UDP/TCP Analysis). 10. Retransmissions that are not due to increase in RTT are probably due to packet losses.
How it works...
TCP uses the retransmission mechanism to ensure data delivery in the absence of any feedback from the remote data receiver. The duration of this timer is referred to as Retransmission Time Out (RTO). This mechanism was rst standardized in RFC1122 that specied that the RTO should be calculated as outlined in the Jacobson V. and M. Karels, Congestion Avoidance and Control, article from 1988. An update to this RFC came out in RFC 2988 in November 2000, and later in RFC 6298 Computing TCP's Retransmission Timer, June 2011.
381
There's more...
For delay variations, you can also navigate to Statistics | TCP Stream Graph | Round Trip Time Graph. When experiencing high delays, it also inuences the throughput you can get from the network. This is what is called as the bandwidth delay product as shown in the following gure:
Window Size [Bytes] RTT [Sec]
Throughput [Bytes/Sec] =
From here we can see that the higher the delay is, the lower the throughput becomes. In networks with high delays, for example, old cellular networks, satellite lines, and long distance international lines, we have several methods to improve the application's throughput. Among these methods are applications that use multiple connections per application, usage of the TCP increases the window size (comes as default in Window Vista and the higher versions, along with various Linux versions).
382
Where to buy this book

You can buy Network Analysis Using Wireshark Cookbook from the Packt Publishing website: http://www.packtpub.com/network-analysis-using-wiresharkcookbook/book.
Free shipping to the US, UK, Europe and selected Asian countries. For more information, please read our shipping policy.
Alternatively, you can buy the book from Amazon, BN.com, Computer Manuals and most internet book retailers.
www.PacktPub.com

Network Analysis Using Wireshark Cookbook: Chapter No. 13 "Troubleshooting Bandwidth and Delay Problems"

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Network Analysis Using Wireshark Cookbook: Chapter No. 13 "Troubleshooting Bandwidth and Delay Problems"

Transféré par

Droits d'auteur :

Formats disponibles

Network Analysis Using Wireshark Cookbook

Chapter No. 13 "Troubleshooting Bandwidth and Delay Problems"

In this package, you will find:

About the Author

For More Information: www.packtpub.com/network-analysis-using-wireshark-cookbook/book