Fairly straight forward this time, you’ve configured your MTU/jumbo frames to be 9000 on your client and destination devices (say a laptop/desktop/server/san/nas) and on ALL your switching devices in between – you’ve done that right? ;)
So the next step is, we want to test if our new 9000 byte MTU is actually working and we can reap the benefits of a larger packet size (whether it’s on iSCSI, LAN, whatever) being of course a higher latency but also higher throughput. This depends on the OS you are running – on Mac OSX (that I run) it’s:
ping -D -s 8184 [destinationIP]
On Linux it’s:
ping -M do -s 8972 [destinationIP]
On Windows it’s:
ping -f -l 9000 [destinationIP]
The reason for the 8972 on *nix devices is that the ICMP/ping implementation doesn’t encapsulate the 28 byte ICMP (8) + IP (20) (ping + standard internet protocol packet) header – thus we must take the 9000 and subtract 28 = 8972.
On mac’s even though they are *nix kernels, the ping implementation only supports packets 8192 in size so we must remove the ICMP (8 byte) header as the ping implementation has already included the 20 byte IP header, 8192 - 8 = 8184.
(Apple macs DO support packets up to 9000 bytes, just the ICMP implementation they sport doesn’t…)
EDIT 31/10/13: According to BernieC in a comment here OSX does support 9000+ byte packets if you run the following command to increase its maximum datagram size:
sudo sysctl -w net.inet.raw.maxdgram=16384
It is also important to understand where I got my values from this is an IP packet’s layout, you can see the IP info is 20 bytes:
If you’ve forgotten to enable jumbo frames/9k MTU on your client device you’re sending the ping from you’ll see:
PING xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx): 8184 data bytes
ping: sendto: Message too long
If you have enabled jumbo frames on your client but not the destination (or a switch in between) you’ll see:
PING xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx): 8184 data bytes
Request timeout for icmp_seq 0
If you’ve done everything right and you’re set up ready to go you’ll get this:
PING xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx): 8184 data bytes
8192 bytes from xxx.xxx.xxx.xxx: icmp_seq=0 ttl=128 time=0.714 ms
Now rejoice in your lovely jumbo-framey goodness and a good 20-30% in sustained data throughput.
Great article… however (and you may have already mentioned and I missed it) but isn’t the “L” switch on the windows side lowercase? (or is this just to make sure it’s not confused with the “i”?
thanks again.
Rhonin, you can use either `-l` or `-L` – I left it upper case for clarity’s sake,
Myles
Not true. I just tried on my windows 8.1 box. If I use an uppercase -L I get a usage message. Incidently, I’m not getting any of the listed results either. What I see is
$ ping -f -l 8000 172.31.252.119
Pinging 172.31.252.119 with 8000 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Ping statistics for 172.31.253.119:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss)
I actually didn’t do anything to enable jumbo frames. I thought I would just test first and see what is already configured for jumbo frames and what isn’t. I take it this message means windows is not configure for jumbo frames.
I hadn’t tested for Windows 8/8.1 good to note, have added to guide.
Yes “Packet needs to be fragmented but DF set.” means you need to enable jumbo frames on your NIC: http://www.maximumpc.com/article/howtos/how_enable_jumbo_frames
This is from a long time ago but in case anybody else has the same question:
Win 10 [and probably 8] does the same encapsulation thing that Linux does.
So ‘Packet needs to be fragmented but DF set.’ means try it with 8972 instead of 9000 and it should work. 8973 will fail.
I just tested this.
“(Apple macs DO support packets up to 9000 bytes, just the ICMP implementation they sport doesn’t…)”
Not entirely true. The Mac has a maximum datagram size for raw packets that can be sent. By default, it’s 8192 bytes. And it’s controlled by one sysctl:
sudo sysctl -w net.inet.raw.maxdgram=16384
There’s another that can be set as well.
sudo sysctl -w net.inet.raw.recvspace=16384
As it turns out, more than just ICMP will use these settings. OSPF routing is another that will. But, basically, to get ping -s 9000 to work, you have to increase at least net.inet.raw.maxdgram.
Excellent info Bernie thanks, will add that to the guide!
Interesting commands and values but there are some mistakes about protocols: ICMP doesn’t use TCP (nor UDP). This is the IP header that is causing all these size troubles… Its header is also 20 bytes long without options.
ICMP itself it’s a protocol at the same level than TCP. When you ping you’re sending “echo request” message. The article makes a wrong reference to TCP, instead would be focused on Ethernet Frame Size that undelayed both TCP/ICMP protocols. However, article is very useful to easily explaining how to “test jumbo frames working” without falling too depth technnical.
Have updated the post to reflect this.
Also it appears that windows 2012 r2 requires that the ICMP and Ethernet frame bytes subtracted from the jumbo ping packets(9000 – 28).
ping -f -l 8972 1.1.1.1
Thanks for the article. Very Helpful.
You should also show how to verifiy on the receiving end that the jumbo frames arrive in the right size and have not been fragmented on the way – which would lead to the same ping response on the senders end.
This can be done using tcpdump and/or wireshark.
On a Mac, you need the -D switch, not -d (case matters). “D” is “don’t fragment,” while “d” sets the “SO_DEBUG” option.
Is there a way to check if you are setting the DF bit?
At what level @matejajovanovic ?
You can either set it or not set it on the ping?
I´m referring to the capture, is there a way to analyze the capture and use a filter that isolates the packets where DF is set?
Thank you. What I was looking for.
I have problems, with linux, when I try to send the ping +1500 packet size, it is not returning neither timeout nor success. I have modifed both MTU hosts to 9000. These are my commands:
mininet> h1 ifconfig h1-eth0
h1-eth0 Link encap:Ethernet direcciónHW 06:49:45:e2:7e:18
Direc. inet:10.0.0.1 Difus.:10.255.255.255 Másc:255.0.0.0
Dirección inet6: fe80::449:45ff:fee2:7e18/64 Alcance:Enlace
ACTIVO DIFUSIÓN FUNCIONANDO MULTICAST MTU:9000 Métrica:1
Paquetes RX:295 errores:0 perdidos:27 overruns:0 frame:0
Paquetes TX:36 errores:0 perdidos:20 overruns:0 carrier:0
colisiones:0 long.colaTX:1000
Bytes RX:67960 (67.9 KB) TX bytes:11499 (11.4 KB)
mininet> h2 ifconfig h2-eth0
h2-eth0 Link encap:Ethernet direcciónHW 2e:96:2a:84:5f:88
Direc. inet:10.0.0.2 Difus.:10.255.255.255 Másc:255.0.0.0
Dirección inet6: fe80::2c96:2aff:fe84:5f88/64 Alcance:Enlace
ACTIVO DIFUSIÓN FUNCIONANDO MULTICAST MTU:9000 Métrica:1
Paquetes RX:304 errores:0 perdidos:28 overruns:0 frame:0
Paquetes TX:33 errores:0 perdidos:0 overruns:0 carrier:0
colisiones:0 long.colaTX:1000
Bytes RX:70325 (70.3 KB) TX bytes:6954 (6.9 KB)
mininet> h1 ping h2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=2.41 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.386 ms
^C
— 10.0.0.2 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.386/1.399/2.412/1.013 ms
mininet> h1 ping -d -v -M “do” -s 1473 -c 3 h2
PING 10.0.0.2 (10.0.0.2) 1473(1501) bytes of data.
— 10.0.0.2 ping statistics —
3 packets transmitted, 0 received, 100% packet loss, time 2017ms
mininet> h1 ping -d -v -M “do” -s 1472 -c 3 h2
PING 10.0.0.2 (10.0.0.2) 1472(1500) bytes of data.
1480 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=2.65 ms
1480 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.419 ms
1480 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=0.089 ms
— 10.0.0.2 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.089/1.055/2.657/1.140 ms
mininet> h1 ping -M “do” -s 8972 -c 3 h2
PING 10.0.0.2 (10.0.0.2) 8972(9000) bytes of data.
— 10.0.0.2 ping statistics —
3 packets transmitted, 0 received, 100% packet loss, time 1999ms
I would say the 28 bytes consist of 8 byte ICMP header and 20 byte IP header, NOT TCP header. ICMP uses IP directly and NOT TCP neither UDP
Post updated.