Blah, Cloud.

Adventures in architectures

  • Twitter
  • GitHub
  • Home
  • Blog
  • Kubernetes on vSphere
  • Multi-tenant IaaS Networking
  • Me
    • About
    • CV
    • Contact
Home » Blog » Hardware » How to test if 9000 MTU/Jumbo Frames are working

How to test if 9000 MTU/Jumbo Frames are working

09/09/2013 by Myles Gray 23 Comments

Fairly straight forward this time, you’ve configured your MTU/jumbo frames to be 9000 on your client and destination devices (say a laptop/desktop/server/san/nas) and on ALL your switching devices in between – you’ve done that right? ;)

So the next step is, we want to test if our new 9000 byte MTU is actually working and we can reap the benefits of a larger packet size (whether it’s on iSCSI, LAN, whatever) being of course a higher latency but also higher throughput. This depends on the OS you are running – on Mac OSX (that I run) it’s:

ping -D -s 8184 [destinationIP]

On Linux it’s:

ping -M do -s 8972 [destinationIP]

On Windows it’s:

ping -f -l 9000 [destinationIP]

The reason for the 8972 on *nix devices is that the ICMP/ping implementation doesn’t encapsulate the 28 byte ICMP (8) + IP (20) (ping + standard internet protocol packet) header – thus we must take the 9000 and subtract 28 = 8972.

On mac’s even though they are *nix kernels, the ping implementation only supports packets 8192 in size so we must remove the ICMP (8 byte) header as the ping implementation has already included the 20 byte IP header, 8192 - 8 = 8184.

(Apple macs DO support packets up to 9000 bytes, just the ICMP implementation they sport doesn’t…)

EDIT 31/10/13: According to BernieC in a comment here OSX does support 9000+ byte packets if you run the following command to increase its maximum datagram size:

sudo sysctl -w net.inet.raw.maxdgram=16384

It is also important to understand where I got my values from this is an IP packet’s layout, you can see the IP info is 20 bytes:

IP Packet

If you’ve forgotten to enable jumbo frames/9k MTU on your client device you’re sending the ping from you’ll see:

PING xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx): 8184 data bytes
ping: sendto: Message too long

If you have enabled jumbo frames on your client but not the destination (or a switch in between) you’ll see:

PING xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx): 8184 data bytes
Request timeout for icmp_seq 0

If you’ve done everything right and you’re set up ready to go you’ll get this:

PING xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx): 8184 data bytes
8192 bytes from xxx.xxx.xxx.xxx: icmp_seq=0 ttl=128 time=0.714 ms

Now rejoice in your lovely jumbo-framey goodness and a good 20-30% in sustained data throughput.

Why not follow @mylesagray on Twitter for more like this!

Show some love:

  • Reddit
  • Twitter
  • Pocket
  • LinkedIn
  • Email
  • Telegram

Similar things I've written

Filed Under: Hardware, Infrastructure Tagged With: jumbo frame, linux, mtu, networking, osx, windows

About Myles Gray

Hi! I'm Myles, and I'm a Dev Advocate at VMware. Focused primarily on content generation, product enablement and feedback from customers and field to engineering.

Comments

  1. rhonin powers says

    23/10/2013 at 10:34

    Great article… however (and you may have already mentioned and I missed it) but isn’t the “L” switch on the windows side lowercase? (or is this just to make sure it’s not confused with the “i”?

    thanks again.

    Reply
  2. Myles Gray says

    23/10/2013 at 13:45

    Rhonin, you can use either `-l` or `-L` – I left it upper case for clarity’s sake,

    Myles

    Reply
    • docbill says

      02/03/2014 at 13:01

      Not true. I just tried on my windows 8.1 box. If I use an uppercase -L I get a usage message. Incidently, I’m not getting any of the listed results either. What I see is

      $ ping -f -l 8000 172.31.252.119

      Pinging 172.31.252.119 with 8000 bytes of data:
      Packet needs to be fragmented but DF set.
      Packet needs to be fragmented but DF set.
      Packet needs to be fragmented but DF set.
      Packet needs to be fragmented but DF set.

      Ping statistics for 172.31.253.119:
      Packets: Sent = 4, Received = 0, Lost = 4 (100% loss)

      I actually didn’t do anything to enable jumbo frames. I thought I would just test first and see what is already configured for jumbo frames and what isn’t. I take it this message means windows is not configure for jumbo frames.

      Reply
      • Myles Gray says

        03/03/2014 at 18:05

        I hadn’t tested for Windows 8/8.1 good to note, have added to guide.

        Yes “Packet needs to be fragmented but DF set.” means you need to enable jumbo frames on your NIC: http://www.maximumpc.com/article/howtos/how_enable_jumbo_frames

        Reply
      • EvilCreamsicle says

        12/01/2017 at 22:15

        This is from a long time ago but in case anybody else has the same question:
        Win 10 [and probably 8] does the same encapsulation thing that Linux does.
        So ‘Packet needs to be fragmented but DF set.’ means try it with 8972 instead of 9000 and it should work. 8973 will fail.

        I just tested this.

        Reply
  3. BernieC says

    31/10/2013 at 18:22

    “(Apple macs DO support packets up to 9000 bytes, just the ICMP implementation they sport doesn’t…)”

    Not entirely true. The Mac has a maximum datagram size for raw packets that can be sent. By default, it’s 8192 bytes. And it’s controlled by one sysctl:

    sudo sysctl -w net.inet.raw.maxdgram=16384

    There’s another that can be set as well.

    sudo sysctl -w net.inet.raw.recvspace=16384

    As it turns out, more than just ICMP will use these settings. OSPF routing is another that will. But, basically, to get ping -s 9000 to work, you have to increase at least net.inet.raw.maxdgram.

    Reply
  4. Myles Gray says

    31/10/2013 at 21:53

    Excellent info Bernie thanks, will add that to the guide!

    Reply
  5. SLiX says

    27/11/2013 at 04:46

    Interesting commands and values but there are some mistakes about protocols: ICMP doesn’t use TCP (nor UDP). This is the IP header that is causing all these size troubles… Its header is also 20 bytes long without options.

    Reply
    • Juan Camilo (@JuAnKmil0) says

      13/05/2016 at 14:41

      ICMP itself it’s a protocol at the same level than TCP. When you ping you’re sending “echo request” message. The article makes a wrong reference to TCP, instead would be focused on Ethernet Frame Size that undelayed both TCP/ICMP protocols. However, article is very useful to easily explaining how to “test jumbo frames working” without falling too depth technnical.

      Reply
      • Myles Gray says

        15/05/2016 at 15:28

        Have updated the post to reflect this.

        Reply
        • Ariel Sandberg-Maitland says

          07/12/2016 at 20:51

          Also it appears that windows 2012 r2 requires that the ICMP and Ethernet frame bytes subtracted from the jumbo ping packets(9000 – 28).
          ping -f -l 8972 1.1.1.1

          Thanks for the article. Very Helpful.

          Reply
  6. Peter Markom says

    23/01/2014 at 17:05

    You should also show how to verifiy on the receiving end that the jumbo frames arrive in the right size and have not been fragmented on the way – which would lead to the same ping response on the senders end.
    This can be done using tcpdump and/or wireshark.

    Reply
  7. DQ says

    10/03/2014 at 02:49

    On a Mac, you need the -D switch, not -d (case matters). “D” is “don’t fragment,” while “d” sets the “SO_DEBUG” option.

    Reply
  8. matejajovanovic says

    26/02/2015 at 09:03

    Is there a way to check if you are setting the DF bit?

    Reply
    • Myles Gray says

      05/03/2015 at 18:52

      At what level @matejajovanovic ?

      You can either set it or not set it on the ping?

      Reply
  9. matejajovanovic says

    13/03/2015 at 12:42

    I´m referring to the capture, is there a way to analyze the capture and use a filter that isolates the packets where DF is set?

    Reply
  10. Dave Jones says

    22/09/2015 at 15:23

    Thank you. What I was looking for.

    Reply
  11. JeanCarlos Chavarría Hughes says

    17/02/2016 at 05:32

    I have problems, with linux, when I try to send the ping +1500 packet size, it is not returning neither timeout nor success. I have modifed both MTU hosts to 9000. These are my commands:

    mininet> h1 ifconfig h1-eth0
    h1-eth0 Link encap:Ethernet direcciónHW 06:49:45:e2:7e:18
    Direc. inet:10.0.0.1 Difus.:10.255.255.255 Másc:255.0.0.0
    Dirección inet6: fe80::449:45ff:fee2:7e18/64 Alcance:Enlace
    ACTIVO DIFUSIÓN FUNCIONANDO MULTICAST MTU:9000 Métrica:1
    Paquetes RX:295 errores:0 perdidos:27 overruns:0 frame:0
    Paquetes TX:36 errores:0 perdidos:20 overruns:0 carrier:0
    colisiones:0 long.colaTX:1000
    Bytes RX:67960 (67.9 KB) TX bytes:11499 (11.4 KB)

    mininet> h2 ifconfig h2-eth0
    h2-eth0 Link encap:Ethernet direcciónHW 2e:96:2a:84:5f:88
    Direc. inet:10.0.0.2 Difus.:10.255.255.255 Másc:255.0.0.0
    Dirección inet6: fe80::2c96:2aff:fe84:5f88/64 Alcance:Enlace
    ACTIVO DIFUSIÓN FUNCIONANDO MULTICAST MTU:9000 Métrica:1
    Paquetes RX:304 errores:0 perdidos:28 overruns:0 frame:0
    Paquetes TX:33 errores:0 perdidos:0 overruns:0 carrier:0
    colisiones:0 long.colaTX:1000
    Bytes RX:70325 (70.3 KB) TX bytes:6954 (6.9 KB)

    mininet> h1 ping h2
    PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
    64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=2.41 ms
    64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.386 ms
    ^C
    — 10.0.0.2 ping statistics —
    2 packets transmitted, 2 received, 0% packet loss, time 1001ms
    rtt min/avg/max/mdev = 0.386/1.399/2.412/1.013 ms

    mininet> h1 ping -d -v -M “do” -s 1473 -c 3 h2
    PING 10.0.0.2 (10.0.0.2) 1473(1501) bytes of data.

    — 10.0.0.2 ping statistics —
    3 packets transmitted, 0 received, 100% packet loss, time 2017ms

    mininet> h1 ping -d -v -M “do” -s 1472 -c 3 h2
    PING 10.0.0.2 (10.0.0.2) 1472(1500) bytes of data.
    1480 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=2.65 ms
    1480 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.419 ms
    1480 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=0.089 ms

    — 10.0.0.2 ping statistics —
    3 packets transmitted, 3 received, 0% packet loss, time 2000ms
    rtt min/avg/max/mdev = 0.089/1.055/2.657/1.140 ms

    mininet> h1 ping -M “do” -s 8972 -c 3 h2
    PING 10.0.0.2 (10.0.0.2) 8972(9000) bytes of data.

    — 10.0.0.2 ping statistics —
    3 packets transmitted, 0 received, 100% packet loss, time 1999ms

    Reply
  12. dsev77 says

    27/04/2016 at 13:06

    I would say the 28 bytes consist of 8 byte ICMP header and 20 byte IP header, NOT TCP header. ICMP uses IP directly and NOT TCP neither UDP

    Reply
    • Myles Gray says

      15/05/2016 at 15:28

      Post updated.

      Reply
  13. Rudolph says

    18/11/2017 at 15:35

    Same with Windows 7, 8972 max size.

    Reply
  14. Zlatko Mitev says

    18/02/2018 at 20:21

    You might want to revise the part of the article where you do specify how to test Windows Jumbo Frame.

    Have a look here: https://kb.vmware.com/s/article/1003712

    Reply
  15. GUruraj Deshpande says

    06/05/2020 at 23:27

    Is MTU change for jumbo frame results in reboot of firewall ? or it is directly applied on interfaces?

    Reply

Leave a Reply Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Myles Gray

Hi! I'm Myles, and I'm a Dev Advocate at VMware. Focused primarily on content generation, product enablement and feedback from customers and field to engineering. Read More…

Categories

Tags

active directory authentication CBT cisco datastore dell design esxi fortigate iscsi jumbo frame kubernetes lab linux load-balancing lun md3000i mtu networking NginX nic nsx openSUSE osx pxe readynas san sdelete serial teaming ubuntu vcenter vcloud director vcsa vexpert video VIRL vmdk vmfs vmware vsan vsphere vsphere 6 vsphere beta windows

Subscribe to Blog via Email

Copyright © 2021 · News Pro Theme on Genesis Framework · WordPress · Log in

loading Cancel
Post was not sent - check your email addresses!
Email check failed, please try again
Sorry, your blog cannot share posts by email.