As you may know from a previous post, I am often on the road traveling to various user’s groups and conferences to talk about Total Network Visibility (two weeks ago we were at InteropNet and last week we were at Interactions 2013). I am sure that all of you have experienced slow networks where web pages take forever to download, applications time out, and VoIP calls become garbled (the “can you hear me now” syndrome). Our mission at PathSolutions is to take the slow out of the network by pinpointing potential network issues before they become network problems. That way your network is always stable and you (I am talking to the IT network guys here) spend far less time troubleshooting because your network is always performing optimally.

One of the things I am most often asked about is what causes a network to slow down.  There are many elements that can contribute to this and most are easily fixable—you just have to know what to look for. Unfortunately, there’s a lot of stress we put on our networks as our infrastructures are expected to support more devices than ever before. So while most problems are easily addressed, the issue comes down to being able to identify the root cause of a problem quickly.  Sounds easy but most network engineers will tell you that they spend the majority of their time searching for the problem, and only a few minutes remedying the problem. That’s why we are sponsoring a number of webinars over the coming months to talk about total network visibility and the challenges different industries face as they try to maintain stable networks.

On April 24 we took a look at municipal network infrastructures. For them the challenge is twofold: ensuring that day-to-day operations are running smoothly while emergencies are responded to swiftly. The City of Coquitlam is a Canadian municipality that recently installed and deployed our solution. In this webinar, Rafael Swieczyk, a senior network analyst at Coquitlam, talked about some of the challenges Coquitlam faced supporting their network infrastructure while meeting strict environmental mandates.

One of Coquitlam’s primary reasons for looking at network performance management solutions was the significant amount of time their network analysts spent just troubleshooting network problems. During the webinar, Raf talked about the amount of time the network team would spend looking at switches, manually looking for errors. As a result, the team did not spend much time dealing directly with their customers. Once Coquitlam installed our solution, they resolved network problems—some they were aware of and some not—much faster. As a result, their customer service levels increased dramatically.

It’s clear that no matter the industry, the “lost” time spent troubleshooting is a significant issue. Our goal is to help you buy back that time with continuous, automated performance monitoring (along with a number of other features that have been mentioned in previous posts and we will continue to talk about on this blog). So, like Coquitlam, you have more time to be proactive and support your customers when they encounter network problems.

Whether you are working for a municipality or in another industry, I believe that you will find this webinar to be informative. Best of all, it’s free. To replay the webinar, go here. And if you have any questions please contact us—we’ll be happy to help.

This year, PathSolutions was selected to help power InteropNet. The InteropNet is the cornerstone of Interop Las Vegas, supplying more than 10,000 attendees and 300 exhibitors with the most reliable, high-speed network throughout the entire conference center. It is also the premier example of how to design, deploy, and manage a multi-vendor, converged network.

The InteropNet network is large and complex, a multi-vendor network that has to be assembled, configured, and deployed onsite in less than four days.  It not only tests the interoperability between the different vendor products but is designed to showcase the latest technologies in a secure environment as the show also attracts hackers.

Of course, work begins on the InteropNet long before the start of Interop. The InteropNet NOC team—made up of the Sponsor Vendors and Volunteer Team Members (ITMs)—begins work via conference calls and online collaboration. Once the planning stage is completed, the entire infrastructure (hardware and software) is then built out on the Hot Stage, a warehouse located in Brisbane, CA. While there, it goes through a rigorous series of performance testing to ensure that it can operate successfully within the confines of the Interop site from both an infrastructure and performance perspective. Finally, it is broken down and transported to Interop’s Las Vegas site where it is re-assembled and tested—all within four days. From May 7 through May 9, the InteropNet is Interop’s technological backbone and is a testament to the skills of the team that helped design and build it as well as the power and interoperability of all the products that support it.

PathSolutions Role: Monitoring the Performance of the InteropNet

We were invited to be part of the NOC to assist with network troubleshooting and performance monitoring. By any definition, setting up the InteropNet is a daunting task but for us, the network monitoring piece was easy to set up, deploy, and manage.

Fast Installation and Deployment Helps to Quickly Resolve Problems

In keeping with our philosophy of delivering Total Network Visibility^®, our software was installed and configured to monitor the Hot Stage’s InteropNet network in twelve minutes. At that point, PathSolutions identified packet losses on some interfaces and the NOC team was able to quickly address and resolve those issues.

Easy Reconfiguration Aids the Hot Stage InteropNet Build Out

During Hot Stage, the network design is highly dynamic and changes on an almost daily basis. This can be equally exhausting and exhilarating. From a monitoring perspective, keeping up with those changes can be very difficult.  However, PathSolutions automated reconfiguration feature enabled us to rapidly re-discover how the network was connected as well as discover new devices.  As a result, as the InteropNet infrastructure changed, PathSolutions was able to quickly recognize those changes—saving all of the NOC team (myself included) valuable troubleshooting time.

Intuitive Interface Translates into Low Training Overhead

The NOC is made up of a large support staff of senior-level network folks that support the InteropNet network both at Hot Stage and during the show.  Since our software was designed for network engineers by network engineers, its intuitive interface made it easy for NOC members to use.  As a result, it became the go-to solution for identifying all layer-1, layer-2, or layer-3 problems in the InteropNet network.

Breadth & Depth of Coverage Aids Error Discovery and Resolution

Our software is configured to monitor the health, performance, and status of every link, switch, and router in the InteropNet.  It queries 18 error counters (more than any other monitoring package) from every link so packet loss can be identified no matter what the source or cause. As a result, the Interop NOC team is able to get more root-cause problems identified (such as bad cabling, duplex mismatches, VLAN tagging problems, etc.) and resolved due to the intelligent analysis of information coming from the network equipment. This significantly reduces the time spent troubleshooting.

Low Overhead Saves Money and Resources

The entire InteropNet network is being monitored with a single low-end server, a single-core processor, and 4 GB of RAM.  As a result, expensive server hardware or a bunch of remote agents are not necessary—saving money and resources.

See It In Action in the NOC or at Booth #743

To see PathSolutions software in action, take a look at it running in the NOC.  It is also running on a low-end netbook in our booth (located to the side of the display computer) and is configured to monitor the entire show network.  Both demos illustrate how, with very little hardware, PathSolutions software can disclose more information on the network than any other offering.

If you’re at Interop 2013 in Las Vegas this year, please stop by our booth. We’d be happy to show you how we’re monitoring the InteropNet network as well as talk about your specific network troubleshooting issues—the most common question I get is how to determine the root-cause of a network problem. This can be very difficult to do but our solutions make it easy!

As you probably already know, I was at the ACUTA Conference and Exhibition last week in San Diego. While there, I was honored to find out that Network Performance Manager (our network performance management solution) was selected as a finalist for the Best of Interop 2013 awards.

Readers of this blog know that we write about “all things related to network performance management,” particularly what you (network engineers) can do to achieve stable, high performing VoIP, video, and data networks. This was never an easy job and today, due to the proliferation of digital devices and the fact that we all work in a digital world, it could be said that the network is the most critical component of your business infrastructure.

I use the word “business” because if the network is not performing, neither is your business. This is why Network Performance Manager provides:

• Continuous Monitoring: Health collected every 5 minutes on every interface.
• Broad Coverage: Monitors every interface on every device in the network for performance.
• Deep Analysis: 18 error counters collected and analyzed on every interface.
• Intelligent Reporting: Identifies & diagnoses the root cause of network problems in plain-English.

It also happens to be what we like to call IT Friendly—packed with features like a fully integrated port mapper, PoE monitoring, VoIP troubleshooting, and more.

I like to say that Network Performance Manager is designed to do the heavy performance lifting because it acts like an extra senior-level network engineer on your IT staff.  It is intelligent: collecting all the relevant performance information, analyzing it, diagnosing the problem, and then, recommending a fix. But unlike the engineer, Network Performance Manager is fully automated—no personal interaction is required to perform these steps so the amount of time spent troubleshooting (time-to-answer) is significantly reduced.

As a networking geek familiar with the ever growing complexity of the network infrastructure, I wanted to develop a network troubleshooting solution that DID NOT need a lot of care, feeding, and support. That way, getting to and maintaining a healthy network is always achievable. Ask any network engineer what their biggest challenge is and they will tell you that there’s never enough time. Ask any network manager what their biggest challenge is and they will tell you that their budget constrains the number of resources (and time) they can apply to address a given problem. In contrast, our solution is built to automatically extract the information necessary to troubleshoot and maintain a healthy network.

Earlier this year, we were honored to be selected as a provider for Interop’s 2013 InteropNet. We are equally honored to be a finalist for the Best of Interop 2013 and extend our congratulations to all the finalists. If you are going to Interop Las Vegas, please stop by and visit us at booth #743. We’d like to know what performance management issues your struggling with and show you how we can help.

Later this month I will be blogging about what it’s like, as a selected provider, to get InteropNet up and running so stay tuned!

The 42^nd Annual ACUTA Conference and Exhibition is from April 14 through 17 and I will be heading to the conference venue in San Diego this weekend. ACUTA is the acronym for The Association for Information Communications Technology Professionals in Higher Education. It represents over 1900 individuals at about 700 institutions of higher education. Its members range from smaller schools and community colleges to the 50 largest U.S. institutions. Needless to say, that’s quite a mix!

The challenges that ACUTA attendees face are quite similar to those that attended BrainStorm 14.0: ensuring a stable network within strict budgetary and resource guidelines. Of course, educational institutions are not immune to the added stress that the proliferation of digital devices and academic and administrative applications can have on a network infrastructure. On campuses today a network slowdown, or outage, can disrupt student information systems, email, class schedules, campus news, and many other administrative processes. More than ever, a reliable network is the key to preventing this from happening.

Over the years, we’ve helped many educational institutions keep their communications and processes running smoothly. One thing most of those customers have in common is too few resources and, as a result, not enough time to dedicate to troubleshooting network issues. After our solution is installed and deployed (in less than 12 minutes on any size network and I tend to say this a lot because there’s no other performance management solution on the market that can do this), the most consistent feedback we get is that the network team spends far less time troubleshooting. And of course, that saved time can be used to support an always growing infrastructure and all the processes that run on top of it.

But don’t just take my word for it. Here’s what Chad Evans, a Systems Architect at Wenatchee Valley College, had to say:

“We recently went from 12 in my department down to 7. In order to give the same level of service, we depend on PathSolutions because it does most of the network performance management heavy lifting. For example, one report I use automatically identifies duplex mismatches which is a very common and very time consuming problem to troubleshoot. These issues can be addressed in minutes. Before, it might be days or weeks before those issues came to my attention. I often find that we will fix network issues before we even get a help desk notification. PathSolutions enables me to discover anything out of the ordinary that is happening on our network and address it before it causes a bigger problem.”

If you’re going to ACUTA, please stop by our booth, 103, and say hello. We’d be happy to show you how (as Chad says) to identify anything out of the ordinary that is happening on your network. If you’re not attending but want to learn more about our solutions, a great place to start would be with this blog. You might also be interested in watching a number of presentations and videos located on our Resource Center page.

Until next time!

I was recently asked by one of the managers at an organization to explain what a MAC Address was and what role it plays in the network.  Trying to find out what he already knew, I asked, “What do you think a MAC address might be?”  The answer I got was: “The mailing address of the warehouse where they make Mac & Cheese?”  (sigh)  I see my work is cut out for me.  (To be fair, it was cold out and nearly lunchtime!)  “Go ahead and slip some lunch into the microwave oven and while we’re waiting for it to heat up, I’ll try to explain things,” I said with as much patience as I could muster.

Addresses: Where Do You Want To Go?

All information sent on an Ethernet LAN network is broken up into “Frames” for easy transport.  Like packages or envelopes passing through post offices, these frames need to be labeled with an address, which enables anything handling that frame to know where to send the LAN frame and to whom the LAN frame is being sent.  The same type of LAN address scheme is used to identify the senders of these Frames so that replies can be sent.The addresses used in an Ethernet network are known as MAC (Media Access Control) addresses.  MAC Addresses are used in Ethernet, WiFi, Bluetooth, and a number of other network types.  Let’s take a look at a typical Ethernet Frame:

You can see the MAC Destination and MAC Source addresses in the typical Ethernet Frame shown in Figure 1.  (See the 3^rd and 4^th fields.)  In order to speed up LAN equipment, the Destination address and Source (Sender) address are sent in the very first parts of a frame.  This enables LAN equipment to more quickly sort network traffic and determine whether a network frame needs attention from them.

Writing MAC Addresses

MAC Addresses are 6 bytes (or 48 bits long), providing 281,474,976,710,656 addresses in theory.  In order to make these addresses easier for humans to remember, these addresses are usually written as six two-digit hexadecimal numbers, such as 01:23:45:67:89:AB.  Some other, less common notations may write the same MAC address as: 01:23:45:67:89:ab, 01-23-45-67-89-AB, or 0123.4567.89ab.

Like phone numbers, there are supposed to be no duplicate addresses within the same network.  In order to keep track of addresses, and establish some order, addresses are broken up into “chunks,” somewhat like area-codes.  Dividing up the MAC addresses, has made it easier for creators of networking equipment to each own a “block” of addresses that they can pick address numbers from.

Dissecting a MAC Address

The first three bytes of a MAC address were originally known as OUI’s, or Organizational Unique Identifiers.  Each manufacturer of networking equipment was assigned an OUI, and was free to assign their own numbers in that block.  If you were assigned an OUI of 01:23:45, the network equipment you manufactured would typically be numbered from 01:23:45:00:00:01 through 01:23:45:FF:FF:FE and you could be reasonably certain that other manufacturers would not use your numbers.  So, to summarize, the first three bytes (depicted in orange) are assigned to a manufacturer of networking equipment and the manufacturer assigns the last three bytes of an address (depicted in blue).

“Special” Address Numbers

There are some addresses that are used for special purposes.  A frame addressed to FF:FF:FF:FF:FF:FF is known as a “broadcast” and is received by every device on a network segment.  (You can think of this as the equivalent of walking into a room, and yelling, “Hey, all of you listen up!”)  Similarly, packets could be broadcast to all equipment of a particular brand by using the manufacturer’s 3-byte OID paired with a value of “FF:FF:FF” for the 3-byte NIC (blue) portion of the address.  (For example, packets addressed to 01:23:45:FF:FF:FF will be received by every device with an address starting with 01:23:45.)

In more current versions of Ethernet, the first byte of the MAC address has some special properties.  If the lowest bit in the first byte of an address is a ‘1’ (making it an odd number), the Ethernet frame that follows may be a multi-cast packet, which is like a limited broadcast that is sent on the network once but may be received by multiple endpoints at the same time.  (This multitasking technique is typically used for applications such as video, which can be sent on the network once, and viewed by many devices at the same time.)

So What?

What good is it that we know the address format of a LAN frame?  This can be a key way to find the source of a network problem!

If your network equipment is complaining that it is seeing strange network packets coming from a particular MAC Address, you may be able to tell the manufacturer of that equipment by looking at the first three bytes of the MAC Address.  Why not use the Layer 3 IP/Internet address to identify a bad network device?  For any devices such as IP Phones that use DHCP to assign IP addresses, a network device may change IP Addresses every time that it is reset, making it difficult to positively identify a network device.

Looking up OID’s: Manual or Automatic?

While some network management systems may show you an ethernet address, they will generally not decode them for you.

You can manually look up an OID address prefix with the IEEE organization that assigns them, but PathSolutions provides a much more convenient way of accessing the manufacturer information.

The latest version of PathSolutions software includes a new feature that integrates MAC Address decoding with the User Interface.  If you are looking at an Ethernet MAC Address within the software, you need only hover your mouse over the address in order to find the manufacturer information from the OID portion of the address.   Frequently this information is just enough to allow a network administrator to tell if that MAC address belongs to an IP Phone or a user laptop, saving valuable time in tracking down and fixing network problems.

Until Next Time!

So, know you know what a MAC address is, and that we can frequently use the first three bytes of a MAC address to find out useful information about the manufacturer of that network device.

If your network management system isn’t pinpointing your network performance issues down to the actual device, and you are still looking up Ethernet addresses manually, you owe it to yourself to take the latest version of our software out for a test drive!

(DING!)   And now I’m off to eat my cheese and pasta treat – hopefully someplace without getting any additional interruptions.  I hope YOU have a good lunch too!

In case you wanted some reading material to learn more about Ethernet Addresses, or you are already an expert looking for the latest detailed technical updates, here are some helpful links from the IEEE Registration Authority:

• http://standards.ieee.org/develop/regauth/tut/index.html
• http://standards.ieee.org/develop/regauth/tut/macgrp.pdf

Want to see more articles like this?  Comment below!

Follow us on Twitter, LinkedIn, Facebook, & YouTube.

Image Credit – Macaroni and Cheese http://upload.wikimedia.org/wikipedia/commons/7/74/Macaroni_and_cheese.jpg

Next to reading (or writing or editing) posts about runts, giant Ethernet frames, and lots of other things that can impact network performance, I enjoy sharing what I’ve learned in more than 20 years of troubleshooting large, medium, and small networks. This week I was in Wisconsin speaking at BrainStorm 14.0, the K20 Technology Conference for IT Personnel, and on my way back to the Bay Area stopped in at Boise to participate in a local Avaya User’s Group meeting (we will be exhibiting at the IAUG’s conference, IAUG CONVERGE2013, in June).

The BrainStorm Conference is sponsored by META (Midwest Educational Technology Association), a group of IT professionals in public and private schools located around the La Crosse, Wisconsin area. This is a great organization and conference that focuses on the network infrastructure and other related technical areas. This year more than 350 schools, ranging from elementary to colleges and universities, attended and more than 75 sessions were offered. My session was on fixing a slow network and it was packed with IT professionals, most of whom were running VoIP and video on their networks. Certainly, supporting VoIP and video can put a strain on your network which is why performance management is so critical.

There are many interesting elements, some much easier to detect than others, that can cause a network to be “slow” and those who attended my session all wanted to learn how to prevent this from happening. This is why our mission is to deliver Total Network Visibility—with this you can quickly and easily identify any network anomaly and as any network professional can tell you, the longer it takes to fix your network problem the more of an impact it can have on critical services. We now live in a digital world and the network that supports it, in my opinion, will either help this world run smoothly or cause it to implode.

Every company and institution also struggles with budgetary constraints—how do you stay within tight budget guidelines without impacting the performance of your network? Again, this is why Total Network Visibility should be a baseline requirement for any network as it can pinpoint weaknesses as well as highlight underperforming devices that can be utilized.

Certainly, my session with the Idaho Avaya User’s Group in Boise focused on how to reduce the time spent on network troubleshooting while, at the same time, increasing the overall performance of their network and productivity of their network staff. The key, again, is complete visibility into the entire network which is what our VoIP and network performance management solutions are designed to deliver. They also significantly reduce the amount of time the network team spends troubleshooting because our solutions collect and analyze the information provided by 18 error counters on every device and then identify and diagnose the root-cause of the network problem in plain-English.

Well, I am now back from taking network visibility on the road. Whenever I travel on business and talk with conference attendees, customers, and partners, I am reminded that our network infrastructures are the cornerstones of businesses, organizations, and educational institutions.  This is why our blog covers network issues in so much depth and why all of us at PathSolutions are focused on delivering Total Network Visibility which, in turn, ensures stable, high performing networks. Next up for the PathSolutions team: Enterprise Connect. We hope to see you there!

What network problem is keeping you up at night? Let us know in the comments and you might see a drill-down on the topic in one of our upcoming posts!

“Always in the middle of the night and usually during a rainstorm…just part of the job,” mused the kindly old English veterinarian.  He was just cleaning up after delivering the prize sow’s piglets when farmer Wallace, a border collie, and his young son Timmy arrived with more buckets of hot water.  “How many?” queried the Yorkshire farmer.  “Ten, plus a runt,” answered the vet.  “What’s a runt?” asked the inquisitive young lad.  “A piglet that may be too small or weak to survive. Let’s go back to the house for some tea by the fire, and I’ll tell you all about it!”

Today’s Episode

There are several ways that networks can resemble barnyards, but we promise today’s informative post will be 100% manure-free—as long as we stay away from certain departments’ file servers!

One of the errors that can be sometimes seen on an Ethernet network is the occurrence of a “runt frame.” Like runt piglets, runt frames are smaller and not as healthy as they should be.  Unlike piglets, (which can be rescued with special care), runt frames must be thrown away by properly working Ethernet equipment.

Healthy, Normal-Sized Ethernet Frames

Before we talk about smaller-than-normal Ethernet Frames, it helps to review what a normal Ethernet frame looks like.  As we see in the diagram below, an Ethernet frame must be AT LEAST 64 bytes long (excluding PreAmble, Frame Delimiter, and Inter-Frame Gap).

When Smaller Isn’t Better

When a program only needs to send a couple of bytes of information in an Ethernet frame, it is forced to add ‘padding’ of extra ‘empty’ bytes/octets in order to make the Ethernet Frame large enough to be sent on the network.  This is akin to using massive quantities of bubble-wrap for a small item being shipped in a large box.  It may seem like wasted space, but the shipping company won’t ship boxes smaller than a certain size!

Why Have Minimum Frame Sizes?

So the most obvious question that technical people ask at this point is: “Why have a minimum Ethernet frame size?”  This is an excellent question.  To answer, we need to travel back in time to look at how the original Ethernet networks operated.  If you were around in those exciting days of yesteryear, Ethernet was not cabled with telephone-like connectors—it was cabled with a single, thick cable that snaked through a computer room, with individual computers “tapped” into the same conductor, which was used for both sending and receiving frames.

In order to allow multiple computers to share a single wire for both transmitting and receiving, a number of rules needed to be set up.  These rules were designed to keep transmissions safe from interference by other computers and to establish some rules about sharing the network cable fairly.

When two computers attempted to send frames on the Ethernet cable at the same time, the two “conversations” would interfere with each other as the signals “collided” with each other on the cable instead of being delivered to the intended receiver.  In order to avoid these collisions, engineers designed a set of rules known as “CSMA/CD: Carrier Sense Multiple Access, Collision Detection” which we’ll cover in detail in a later episode.  For our discussion here, we just need to know that collisions happen when more than one computer transmits on a single line at the same time and the CSMA/CD rules help to avoid this.

One way to help avoid collisions is to have a minimum frame size.  Longer frames make it easier for the other computers to tell when somebody else is using the network cable and helps them avoid talking at the same time.  Think of the original Ethernet as one giant conference call on a speakerphone:  If people always recite a full paragraph of text before leaving the line open for others there will be far fewer instances of two people talking at once than if each caller was only able to randomly shout only a single word, and then wait a random number of seconds.  The longer chunks of information being sent are more noticeable and easier to avoid “talking over.”

Manufacturers of networking equipment have deployed two major revisions of counters designed to track LAN interface errors.  The original set of counters is known as RFC-1213 and a later more extensive set of counters was introduced with the RMON standard. When looking at the original standard (RFC-1213), many people are surprised to see that it does not include counters for “Runts.”  Instead, runt packets typically show up in other measurements. Usually, runt packets will show up in the counters as a high number of collisions (Single Collision Frames and Multiple Collision Frames) as this is a prime cause of runt frames.  You may also see high levels of Alignment Errors on your network, depending on how the runt packets are interpreted by network devices.

Despite the lack of a dedicated error-counter in the RFC-1213 standard, many types of network capture equipment, protocol analyzers, and troubleshooting equipment track and count runt frames on a network link.  Unfortunately, these analyzers typically only diagnose one network link at a time so the network admin is still left with the problem of finding the offending network connection.

 The RMON network monitoring standard introduced two new counters, (etherStatsUndersizePkts & etherStatsFragments), which were designed to track and count runt frames.  In our lab and field testing, we’ve found some major setbacks to using these RMON counters to uncover runt packets.

The first setback is that many device manufacturers do not support RMON, or if they do, they support a very limited number of RMON groups.  We’ve also found that even if a device supports a particular RMON group, they may not support the counters that you want and may not actually count and track runts since this requires support from the underlying network hardware and chipsets.

Another drawback to using RMON counters is that some manufacturers appear to be pulling away from RMON support since it can use a great deal of memory and CPU on their networking devices which may decrease their performance.

In the standards bodies, where engineers are working on managing 40 Gigabit Ethernet and other new technology, there have been some recent debates about possibly adding a better set of counters to the next generation of networking gear and tracking runts better.  If and when the standards bodies do add any useful counters to the standard, PathSolutions will be furiously testing  network manufacturers’ implementations of the new standards and seeing how they help diagnose network problems so that we can continue to make our software (and our customers) smarter!

What Causes Runts

There are a number of possible causes of runts, none which should occur on a normal, healthy network!  The most likely causes are excessive collisions, which may distort Ethernet frames, causing only the first half of a frame to be seen before it is cut off by a collision.  (Imagine a limousine crossing the train tracks and only the first part of the car makes it across before the train collides with it and knocks off the rear wheels and trunk). 

Other, less common causes may include: LAN interfaces or transceivers that have been damaged and are breaking the rules, sending out malformed/shortened frames, taking completely normal frames and shortening them as they are repeated down another network segment, or falsely sensing collisions that aren’t there.  We have seen network hardware drivers that don’t pad the packets correctly, ignore inter-packet gap timing, or don’t police packet length properly.  Fortunately, malfunctioning LAN interface chips are rare; when they are going to fail, they typically stop transmitting altogether.  I say fortunate, since it can take a huge amount of time and troubleshooting expertise to find a malfunctioning LAN interface chip, especially if the problem is intermittent.  (Dead interfaces are easy to diagnose, crazy interfaces are hard, and only slightly crazy interfaces can be next to impossible to find!)

Tip: If you are seeing only ONE interface reporting high collisions and alignment errors remove that one machine from your network and test it separately.  If you are seeing ALL of your network interfaces report an error EXCEPT one happy interface that reports no problems at all remove the only “happy” machine from your network and see if the rest of your network suddenly gets well.

Until Next Time!

So now you know what a runt frame is, its symptoms and causes, and you should have an idea of what to keep an eye on and how to fix runt problems.  Heavily congested network bottlenecks may need more bandwidth (a faster connection).  High collisions and alignment errors on a lightly used interface may indicate cabling issues.  If runts appear to be coming from a specific LAN interface, it may be time to check the health of that interface card, swapping the hardware out in order to see if the issue goes away.

Here’s hoping that your network is healthy and free from these “small” problems!!!

Want to see more articles like this?  Comment below!

Follow us on Twitter, LinkedIn, Facebook, & YouTube.

Image Credits: Microsoft Office Clip Art, unless otherwise specified.

Image Credit – Bubble Wrap   http://en.wikipedia.org/wiki/File:Bubble_Wrap.jpg

We’ve all heard the mantra: “Bigger is Better.”  Sometimes that’s true (as in paychecks) and sometimes this is less than true (as in Attack of the Giant _fill-in-the-blank_ B-Movie monster).  Just as in many other things in life, there have been debates over size in the networking world as well.  How big is too big?  Will monster-sized Ethernet Frames run through my downtown network, flattening everything in their path, and causing crowds of users to go running through the streets in terror?

Find out more, in this week’s exciting post!  While we promise not to knock holes in your buildings, or crush any cars while we tackle this monster topic, we still advise all of our customers not to trade any of their cows for magic beanstalk beans.

It All Started with an Ordinary Frame…

In order to understand strange extra-large variations of Ethernet LAN Frames, we should first make sure everyone is generally familiar with some of the standard features of an Ethernet Frame.  An Ethernet Frame consists of a number of information fields used for signaling, traffic routing, classification, and error handling.  These fields are shown in blue in Figure 1 below.

The area shown in green is the portion of an Ethernet frame that carries actual data.  In a standard Ethernet Frame, this field can be as large as 1500 bytes or as small as 42 bytes (when using 802.1Q Tagging).  Some protocols may use very small packets to send a single piece of data (for example, the temperature of a thermostat), while other protocols may use maximum-sized Ethernet frames to move large amounts of data (such as performing a file-transfer, or disk-backup). By offering programmers a choice of Ethernet frame sizes, it was hoped that the standards bodies could avoid a standards-war between advocates of very small frames, and those who favored larger frames.

One primary difference between large and small frames is the ratio of ‘overhead’ bytes to payload bytes.  Figure 2 shows a quick comparison of the amount of bandwidth available to send data on a line versus the amount of data used for ‘overhead’ signaling.

In addition to signaling overhead using up bandwidth on a network connection, the more signaling that exists on a line, the more work that networking equipment has to perform to process that traffic, driving up network equipment processor usage.

Why Limit Yourself?

The original Ethernet Standard Frame was built to run on a 10Mbps wire that was shared between all attached computer nodes.  At the time, Full Duplex connections with dedicated bandwidth, High Speed Ethernet Switches, and 10 Gigabit Ethernet connections were the stuff of science fiction.  A payload limit of 1500 bytes was a reasonable maximum frame size for the equipment imagined at the time the standard was written.  The 1500 byte payload kept collisions down to a manageable level, allowed more traffic to share the line, and minimized the amount of memory that network devices required for buffers.

Jumbo Frames

In the early days of Gigabit Ethernet, some specialized server-to-server Ethernet applications began to experiment with “Jumbo Ethernet Frames.”  Jumbo Frames allowed these specialized servers to stretch their payload from the standard 1500 bytes to as large as 9000 bytes.  With the new frame sizes, server backups and large data transfers between servers was dramatically sped up, while minimizing the impact to server processor and I/O performance.  Eventually, LAN switch vendors began to support Jumbo Frames but not all of them supported payload frame sizes as high as 9000.

Baby Giant Frames

Today, some Carrier Ethernet Equipment (used for high-speed Metropolitan Area Ethernet networks) use a payload size of 1600.  This allows network operators to take a customer’s standard “1500” sized Ethernet Payload, wrap some additional signaling around it (such as MPLS), and fit the standard frame inside their “Baby Giant” Ethernet Frame.

Super Jumbo Frames

There are a small number of specialized research networks that are currently experimenting with Ethernet Frames using a payload larger than 9000.  These are becoming known as “Super Jumbo” frames.

Oh-Oh!  When Giant Frames Go Bad

There are a number of special challenges with using larger-than-normal frames.  If two computers decide to use a 9000 byte Jumbo Frame, EVERY piece of LAN equipment between these two computers MUST support Jumbo Frames, and be configured to handle Jumbo Frames at least that large.  If a single piece of LAN equipment cannot handle a frame that large, the result is like a 14 foot (4.27 meter) tall truck attempting to drive through a 10 foot (3.05 meter) tall tunnel!  The crunching noise you will hear will be the Ethernet frame crashing to the floor as the standard Ethernet equipment decides that the frame is too large, must be faulty, and should not be forwarded on the network.  As a result, no jumbo frames will make it through that part of the network, and the resulting re-transmissions will likely clog network links at both ends.

What does a Giant Sound Like?

Jumbo Frames sent over slow speed connections have also been known to reduce the ability of multiple computers or applications to “play nicely” over a single network connection.  You might think of a section of road where three lanes merge into one.  A large number of tiny cars can flow together without too much disruption.  However, if a 1/2 mile (0.8 kilometer) long railway train were to merge into the same road section, the other traffic lanes would need to wait long enough that many cars would arrive at their traffic destinations very late.  Jumbo Frames, like long railway trains, typically do better traveling on dedicated, specialized paths.  When short VoIP voice packets encounter long delays waiting behind Jumbo Frames, the result is typically long periods of dropout silences as the flow of audio packets comes to a stop, then resumes.

Finding Giants, or Low Bridges, & Tunnels

If you are using Jumbo Frames, you will want to keep an eye out for the “Low Bridges and Tunnels” on your network.  If you are using only standard Ethernet frames, you might want to keep an eye out for any stray Giants, Jumbos, or other XXL creatures appearing on your network.  While you may not be planning to use Jumbo Frames, you wouldn’t be the first network administrator to find an ambitious user re-configuring their machine to use larger frames (remember Bigger is Better?).

PathSolutions software can help to detect Giant Frames.  Giant Frames are frames which are too large to be handled by a given piece of network equipment.  PathSolutions network management systems can track whenever Ethernet network equipment is being sent packets larger than they can handle and what network interface is having problems with these frames.  These errors show up as “Frame Too Long” errors, as shown in Figure 3 below.

MTU = Maximum Payload Size

How do you know if your equipment can handle Jumbo Frames, or other non-standard XXL Frames?  The answer is to look at your equipment’s “MTU” settings.  MTU stands for “Maximum Transfer Unit”, and is the “techy” name that computer scientists use to describe the maximum payload size of an Ethernet Frame.  If your Ethernet equipment is set for an MTU of “1500”, it is using a standard Ethernet Frame.  If you see a larger number in your equipment configuration, it can handle (and may be sending) Jumbo Frames.

Fee Fi Fo Fum!

So, now that you know how to keep your network safe from giants, defeating extra large problems with ease, I hope that this knowledge proves useful in helping your network to run at maximum performance, and operate happily ever after!

Want to see more articles like this?  Comment below!

Follow us on Twitter, LinkedIn, Facebook, & YouTube.

Image Credit 1 –Office Clipart, Jumbo Jet

Image Credit 2 – Jack & The Giant Killer, WikiMedia Commons.  http://commons.wikimedia.org/wiki/File:Rackham_giant2.jpg

Since few vendors take the time to explain what these terms mean, and many sales people have only memorized them as industry “buzzwords,” we’ll take today’s post to provide a practical explanation for our readers.

Too Many Triple Espressos

While jitter may sound like a great description of the symptoms of that one guy in the office who has downed a few too many triple espressos, the term has a very specific definition for network engineers.

Out of Order

When your voice, or any audio stream, gets sent using VoIP technology, it generally gets recorded and sent across the network in small ‘snippets’ of between 10 and 30 milliseconds (.01 to .03 of a second).  In order to reproduce the sound at the other end, these recordings should arrive in order, so that all of the noises can be reproduced faithfully at the far end.  You might think of it a bit like chopping this paragraph into individual words and sending them in separate envelopes.  If there was no way to re-assemble the words into the correct order, it would be impossible for an average person receiving them to make any sense of what the writer was attempting to say.  For this reason, Real-Time-Protocol numbers these mini-recordings to assist the other end’s efforts to play them back in the correct order.  (As an aside, in addition to acting like page-numbers in a book, these number sequences also help us to detect when there are missing bits of audio in the stream of snippets to be re-assembled and played at the far end of a call).

Jitter = Delay Variation

At this point, you may be wondering what causes the packets containing these snippets to arrive out of order.  The one word answer is jitter:

• When network and other delays are constant, every packet sent takes exactly the same amount of time and packets arrive in the order that they were sent. 
• When the delay across a network varies widely, some packets may take 150 milliseconds to arrive, while others, taking only 60 milliseconds, may arrive ahead of the slower packets, causing the snippets to arrive at their destination in a different order from which they were sent.

Why Am I Stuck In Traffic or Causes of Jitter

In the course of an hour-long VoIP conversation, the delay across the network may swing wildly, from near-instantaneous to over ¼ of a second.  The more changes that occur in delay, the more packets arrive out of order.

These rapid swings in network delay can be caused by a number of factors.  Just like many morning commuters, the packet may get stuck waiting in a router buffer when a surge of other network traffic is attempting to cross a slower-speed Wide-Area-Network link.  Whether there is too much traffic at times or the link is too slow, network bottlenecks are frequent sources of unpredictable delays.   Bursts of traffic may also cause random delays for a handful of packets, increasing jitter.

If different packets take different routes, they may arrive at different times.  This can be caused by parallel routes to the same destination, load-sharing, or route tables changing the path that a packet takes when getting to its destination.  (We’ve all seen this when we wait in line at the grocery store – whichever line we get in always seems to come to a screeching halt!).  A ‘flapping’ route, where a low-delay link is available intermittently every few seconds, can wreak havoc on the best efforts of equipment to maintain a predictable delay and to keep audio packets arriving in order.

Finding Jitter

One common mistake network managers make when looking for jitter problems is to not measure delay frequently enough.  If you look at network delay once per hour, you will be unlikely to notice changes of delay that occur three times per second – despite hundreds of swings in delay times, the network delay at 2 p.m. may be very close to the network delay at 3 p.m.  If delay is not measured frequently enough, jitter will only likely be detected by the network manager when frustrated users complain of random echoing and ‘drop-outs’ of silence that appear in their calls.

Please wait… Buffering…

Once we identify and correct any temporary network impairments, removed traffic bottlenecks, and architected our network properly (to the best of our budget) we will almost inevitably wind up with some residual (left-over) jitter.  How do VoIP devices deal with out-of-order packets?  One method is known as the “jitter buffer.”  A jitter buffer acts as a “waiting room,” allowing packets to be swapped back into their original order.  In figure 2, the receiving VoIP device has a two-packet jitter buffer.  The good news?  This allows the device to swap two out-of-order packets back in to their original sequence. The bad news?  This waiting room enforces a mandatory wait based on the number of chairs — the receiving device must wait until the buffer is filled with two packets before it begins playing any audio.  (Many of you have seen a similar effect waiting for web-based videos to load.)  A two-packet jitter buffer forces a two-packet long delay on the line.  If you are using 10ms packets, this will insert an additional 20ms delay.  For 30ms packets, this inserts an additional 60ms delay.  As discussed in previous articles, too much delay results in bad voice quality.

Figure 3 shows a larger three packet jitter buffer handling a VoIP connection with even greater jitter, causing packets to arrive even more out-of-order than figure 2.  In this case, the larger jitter buffer adds more delay but is able to compensate for packets further out of order.  As we know from previous discussions ((hyperlink:echo blog entry)), if this longer buffer causes excessive delay, voice quality will degrade.

No Buffer:  Let It Be

What if packets arrive out-of-order but creating a jitter buffer adds too much delay?  There is one other common option:  In some cases, calls will sound comparatively better if out-of-order packets are merely discarded (or dropped).  This technique may cause a one or two packet long ‘dropout’ silence (cellphone users may describe this as “cutting out,” but has the benefit of not adding any additional delay to a call that might be at near maximum delay already.

Tall, Grande, Venti…

So, now that you know what Jitter is and what a Jitter Buffer does you can make smarter decisions and size your Jitter Buffer (0-3) to handle out-of-order packets.  Do you know how much jitter is on your important network links and how this affects your VoIP system’s voice quality and call reliability?  Try a 30-day demonstration of Path Solutions network management solutions to find out!

Now that I’ve stopped shaking, I think I’m off to find a refill with several in-sequence sugar packets!

Want to see more (or less) articles like this? Comment below!

Follow us on Twitter, LinkedIn, Facebook, & YouTube.

Photo Credits

Wikipedia – Espresso Machine. http://en.wikipedia.org/wiki/File:Modern_espresso_machine.jpg