Wednesday, December 30, 2015

CCIE 400-101: Network Principles - Latency, Windowing, BDP, Global Synchronization

Latency

  • Latency is most often used to describe the delay between the time that data is requested and the time when it arrives (also known as round-trip time or RTT for short).
  • However, there are essentially four components that contribute to total latency:
    1. Transmission delay: the time to transmit data on the link.
    2. Propagation delay: the time for a packet to travel accross the link.
    3. Processing delay:  the time for a device to process the data.
    4. Queuing delay: the time the data spent waiting in queue (memory) inside a device.
  • The combination of transmission, propagation, processing, and queuing delays often produces a complex and variable network latency profile.


  • Different delays can be calculated as follows:
    • Transmission delay (t) = number of bits to send (b) / link data rate (r). (t = b/r)
    • Propagation delay (p) = link distance (d) / speed of signal (s). (p = d/s)
      • Unless otherwise stated, s = c = 3 x 10^8 (the speed of light)
    • Processing delay and queuing delay are variable and depend on many factors.

Example: The latency is indicated in the 'time' column of the ping output. The ping statistics also list the minimum, maximum, and average latencies.

Pinging www.google.com [74.125.224.82] with 1500 bytes of data:
Reply from 74.125.224.82: bytes=1500 time=68ms TTL=52
Reply from 74.125.224.82: bytes=1500 time=68ms TTL=52
Reply from 74.125.224.82: bytes=1500 time=65ms TTL=52
Reply from 74.125.224.82: bytes=1500 time=66ms TTL=52
Reply from 74.125.224.82: bytes=1500 time=70ms TTL=52

Ping statistics for 74.125.224.82:
    Packets: Sent = 5, Received = 5, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 65ms, Maximum = 70ms, Average = 67ms


  • A networking technology with very high throughput and bad (high) latency can be worse for some applications than one with relatively low throughput but good (low) latency. 
  • See: Ping vs. Latency: What's the Difference?

Windowing

  • Essentially, the TCP window size is the amount of ("in flight") data that a sender can send before it needs to wait for an acknowledgement from the recipient. 
  • If the sender has not received acknowledgement, it will stop and wait. If this wait time exceeds a certain limit, a retransmission will occur. This is how TCP achieves reliable data transmission. 
  • Even if there is no packet loss in the network, windowing can limit throughput. 
  • Because TCP transmits data up to the window size before waiting for the acknowledgement, the full bandwidth of the network may not always get used. 
  • The window size is communicated via a 16-bit field in the TCP header, giving it a maximum size of 64 KB. 
  • Window scale provides a scale factor which is required for TCP to support window sizes that are larger than 64 KB. 
  • Essentially, window scaling simply extends the 16-bit window field to 32 bits. 
  • This is done by specifying a 1-byte shift count in the header options field. 
  • The true receive window size is left shifted by the value in the shift count. 
  • The maximum value of the shift count is 14, resulting in the maximum window size of 1 GB!


  • The window scaling option may be sent only once during a connection by each host, in its SYN packet.
  • The window size can be dynamically adjusted by modifying the value of the window field in the TCP header, but the scale factor remains static for the duration of the TCP connection.
  • Scaling is only in effect if both ends include the option; if only one end of the connection supports window scaling, it will not be enabled in either direction.

 

Bandwidth Delay Product (BDP)

  • In short, bandwidth delay product (BDP) is the optimum window size to fully utilize link bandwidth. 
  • The best possible network performance is achieved when the link between the sender and the receiver forwards a constant stream of data. 
  • Analogy: a water hose. To achieve maximum water flow, the hose should be full. As the hose increases in diameter and length, the volume of water to keep it full also increases. 
  • In networks, diameter equates to bandwidth, length is measured as round-trip time, and the TCP window size is analogous to the volume of water necessary to keep the hose full. 
  • On fast networks with large round-trip times, the TCP window size must be increased to achieve maximum TCP bandwidth. 
  • Theoretically, the TCP window size should be set to the BDP, which is simply computed as bandwidth * delay. 
  • Example: For 1 Gbps link with 70 ms delay, the BDP (optimum TCP window size) is 1,000,000,000 * 0.07 = 70,000,000 (bits). Convert that to bytes, you get 70,000,000 / 8 = 8,750,000 bytes = 8,75 MB.  
  • In order to fully utilize 1 Gbps link with 70 ms, the TCP window size must be 8,75 MB.  
  • Based on these calculations, it is easy to see why the typical default buffer size of 64 KBytes would be completely inadequate for this connection. With 64 KBytes you would get only 0.1% of the available bandwidth.

Global Synchronization

  • During times of congestion, routers can use a technique called tail drop to simply discard packets when the interface queue is filled to its maximum.
  • When tail drop happens, TCP has an automatic recovery from the dropped packets, which causes the sender to reduce its sending rate.
  • Bandwidth utilization drops significantly and interface queue becomes less congested.
  • After a certain amount of time, the sender starts increasing the rate again.
  • The problem is that almost all the senders will use the same time delay. When these delays expire, at the same time, all senders will send additional packets, the interface queue will overflow again, more packets will be dropped, the senders will all back off for a fixed delay.. and the cycle repeats.
  • This pattern of each sender decreasing and increasing transmission rates at the same time as other senders is referred to as "global synchronization".
  • The result is a sawtooth-shaped bandwidth utilization, causing inefficient use of available bandwidth and inconsistent application throughput.


References

Transmission and Propagation Delay
Congestion Avoidance
Tuning TCP for High Bandwidth-Delay Networks
A User's Guide to TCP Windows
TCP Windows and Window Scaling

Wednesday, December 23, 2015

CCIE 400-101: Network Principles - IPv4 and IPv6 PMTU, MSS

MSS

  • The TCP Maximum Segment Size (MSS) defines the maximum amount of data that a host is willing to accept in a single TCP/IP datagram.
  • The MSS value is sent as a TCP header option only in TCP SYN segments.
  • Each side of a TCP connection reports its MSS value to the other side.
  • Contrary to popular belief, the MSS value is not negotiated between hosts. The sending host is required to limit the size of data in a single TCP segment to a value less than or equal to the MSS reported by the receiving host.
  • The way MSS now works is that each host will first compare its outgoing interface MTU with its own buffer and choose the lowest value as the MSS to send.
  • The hosts will then compare the MSS size received against their own interface MTU and again choose the lower of the two values.
  • Often the send MSS value will be the same on each end of a TCP connection.
  • By default, the MSS = 1460 bytes in an Ethernet network (1500 bytes - 20-byte IP header - 20-byte TCP header = 1460 bytes).
  • TCP MSS takes care of fragmentation at the two endpoints of a TCP connection, but it doesn't handle the case where there is a smaller MTU link in the middle between these two endpoints.

PMTU

  • Path MTU (PMTU) is the smallest MTU in the path from source to destination.
  • Path MTU Discovery (PMTUD) was developed to avoid fragmentation in the path between the endpoints. It is used to dynamically determine the lowest MTU along the path from a packet's source to its destination.  
  • PMTUD is only supported by TCP. UDP and other protocols do not support it. 
  • If PMTUD is enabled on a host, and it almost always is, all TCP/IP packets from the host will have the DF bit set. 
  • If a router tries to forward an IP datagram, with the DF bit set, onto a link that has a lower MTU than the size of the packet, the router will drop the packet and return an Internet Control Message Protocol (ICMP) "Destination Unreachable" message to the source of this IP datagram, with the code indicating "fragmentation needed and DF set" (type 3, code 4).
  • A router returning an ICMP message indicating "fragmentation needed and DF set" should include the MTU of that next-hop network in the low-order 16 bits of the ICMP additional header field that is labeled "unused" in the ICMP specification. The high-order 16 bits remain unused, and MUST be set to zero.  Thus, the message has the following format:

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   Type = 3    |   Code = 4    |           Checksum            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           unused = 0          |         Next-Hop MTU          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |      Internet Header + 64 bits of Original Datagram Data      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+



  • When the source station receives the ICMP message, it will lower the packet size to reflect the smaller MTU. 
  • There are three things that can break PMTUD, two of which are uncommon and one of which is common.
    • A router can drop a packet and not send an ICMP message. (Uncommon)
    • A router can generate and send an ICMP message, but the sender ignores the message. (Uncommon)
    • A router can generate and send an ICMP message but the ICMP message gets blocked by a router or firewall between this router and the sender. (Common) 
  • The success or failure of PMTUD hinges upon ICMP unreachable messages getting through to the sender of a TCP/IP packet. 
  • The following commands will enable the necessary ICMP messages to pass through a firewall:

access-list 101 permit icmp any any unreachable
access-list 101 permit icmp any any time-exceeded
access-list 101 deny icmp any any
access-list 101 permit ip any any


 

References

Resolve IP Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPSEC

Tuesday, December 22, 2015

CCIE 400-101: Network Principles - IPv4 and IPv6 Fragmentation, TTL, IP MTU

IPv4 and IPv6 fragmentation

  • Although the maximum length of an IP datagram is 64K, most transmission links enforce a smaller maximum packet length limit, called an MTU (maximum transmission unit). 
  • The design of IP accommodates MTU differences by allowing routers to fragment IP datagrams as necessary. 
  • IP fragmentation involves breaking a datagram into a number of pieces that can be reassembled later. 
  • Fragmentation causes more overhead for the receiver when reassembling the fragments because the receiver must allocate memory for the arriving fragments and coalesce them back into one datagram after all of the fragments are received. 
  • Reassembly is very inefficient on a router whose primary job is to forward packets as quickly as possible. 
  • Another fragmentation issue involves handling dropped fragments. If one fragment of an IP datagram is dropped, then the entire original IP datagram must be resent, and it will also be fragmented.
  • Example: The original IP packet is 12,000 bytes. To transmit this data over the first local link, Device A splits it into four fragments. The first router must fragment them further into smaller packets to send them over the 1,300-byte MTU link. Note that the packets are reassembled only at the final destination.



  • Unlike in IPv4, IPv6 routers never fragment IPv6 packets.
  • Packets exceeding the MTU are dropped and this condition is signaled by a "Packet Too Big" ICMPv6 message (similar to the IPv4 method when the DF bit is set).
  • IPv6 hosts are expected to perform path MTU discovery to determine the maximum size of packets to send, and the upper-layer protocol is expected to limit the payload size.

 

TTL 

  • TTL, which stands for Time To Live, is an 8-bit field in the IP header.
  • TTL determines the lifetime of an IP packet.
  • The purpose of TTL is to prevent a data packet from circulating in the network indefinitely.
  • The initial TTL value is set by the sender, and is reduced by every router on the route to the packet's destination. Hence, the TTL can also be considered as a hop count.
  • Because of the 8-bit field size, TTL can support a maximum of 255 hops.
  • If the TTL reaches 0 before the packet reaches its final destination, the packet is discarded and an ICMP error message is sent back to the sender.
  • Different operating systems can use different default TTL values. For example, most Windows systems use a default TTL of 128.


IP MTU

  • Maximum Transmission Unit (MTU) is the size of the largest packet that the device can forward. 
  • If the packet is bigger than the MTU, the packet will be fragmented (unless the DF bit is set, in which case the packet is dropped). If a packet is fragmented, it is reassembled at its final destination. 
  • There are basically two MTU values: system or interface MTU and IP MTU. 
  • The interface MTU specifies the maximum size of the datagram that the interface can send (including the Layer 2 header). 
  • IP MTU defines the maximum size of the IP packet. 


  • The value of the MTU depends on the type of the transmission link. For example, the default MTU for Ethernet is 1500 bytes. In a normal transmission case (no additional encapsulation, such as GRE or IPsec), the sender may use the maximum payload length of 1460 bytes (1500 bytes - 20-byte IP header - 20-byte TCP header = 1460 bytes) without any potential risk of packet fragmentation/drop.
  • Sometimes the IP MTU must be lowered from its default value to support extra headers like GRE or IPsec. 
  • A larger MTU brings certain benefits like processing of fewer packets for the same amount of user data.  
  • Example: NFS, by default, has a read and write block size of 8192, so a NFS IP/UDP datagram will be approximately 8500 bytes (including NFS, UDP, and IP headers). A sending station connected to an Ethernet (MTU 1500) will have to fragment the 8500 byte datagram into six pieces; five 1500 byte fragments and one 1100 byte fragment. If any of the six fragments is dropped, the complete original datagram will have to be retransmitted. This creates more overhead and requires more processing resources from the router, as compared to sending one big packet of 8500 bytes (MTU 9000). 
  • On the contrary, larger packets can occupy the link for too long, thereby increasing delay.

 

References

The TCP/IP Guide: IP Datagram Size, MTU, and Fragmentation Overview