How do you manage performance when crossing other people’s networks?
(Cue Film Noir Music, Radio Announcer Voice...)
Ladies and Gentlemen, today we’re bringing you, “Tales from the Field–Stories we’ve picked up from our agents in the field. These stories are true, but the names have been changed to protect the innocent."
Clouds So Thick, Nobody Can See What Lurks In Those Dark Alleyways of the Network…
Recently, I’ve returned from a large cloud technology conference. In a few weeks, there is another one scheduled. While everyone is touting the advantages of cloud computing, I thought it would be timely to provide a tale illustrating some of the performance challenges facing this new technology.
Managing Other People’s Stuff
In the “old” days, in order to guarantee the performance of an application companies bought their own servers, networks, and PC Clients. If everything was in the same building, (or at least on the same network), it didn’t’ take an angry mob of users long to find out if their performance was due to a slow server, network, or PC. Likewise, company management could quickly point to a single scapegoat, making everyone slightly happier, (except perhaps the goat).
With cloud technology, a set of servers can be located in one or more datacenters, served by one or more ISP’s. The corporate network may be built using a variety of ISP/Carriers and vendors. Users may be virtually connected to these servers via a third set of Carriers (sometimes wirelessly), using a collection of client devices, some of which aren’t owned by the company.
We recently encountered a particularly frustrated IT manager. For purposes of this story, we’ll call him Frustrated Fred. Fred had a virtual server hosted at a remote datacenter. Nearly all of the users at headquarters, branch offices, and teleworkers at home were happy with their application performance using a variety of ISP’s and wireless providers. Everyone was happy except for a VIP we’ll call VIPeter. Screens that loaded instantly seemed to take minutes–everything appeared to be ten times slower when VIPeter was working from home.
Fred sighed. As is often the case, the one user with a problem happens to be the one person that can’t be ignored or bypassed–Peter was a Vice-President.
Fred tested the datacenter network and servers–all were good. Fred tested the network performance on both the corporate network, and the datacenter network. Again, everything passed with flying colors.
Fred managed to reach the ISP who provided broadband access at VIPeter’s house. After speaking with their customer service (who tested VIPeter’s line with a passing grade), Fred decided to drive out and test VIPeter’s Internet connection personally, convinced that the helpful customer service person just refilled their coffee cup and took a break instead of testing the proper line.
When Fred tested VIPeter’s line, he was shocked to discover that the line was working properly, speeding through all other applications except for the one hosted at Fred’s datacenter.
Now What? The Mystery Deepens…
This is where PathSolutions picked up the mystery trail. All of the network providers had tested their individual links with passing grades. Something was missing from the equation.
A very short while after installing software from PathSolutions, the graphs began to tell the tale. The chain of networks from the datacenter to VIPeter’s house crossed several ISPs. Between two of these ISP’s a significant number of packets were being dropped (although each individual ISP network was fine).
ISPs exchange data with each other at facilities called Internet Exchange Points, or IXPs. These backbone sites of the internet provide connection points for traffic flowing from one Internet provider to another. How traffic is exchanged between ISPs is governed by something called a “peering agreement."
If It Wasn’t For You Meddling Kids…
Unlike some “owner only” network monitoring products, PathSolutions was able to track performance not only of individual networks, but also the performance of the links between them. When Fred called the ISPs to tell them that the problem was in the link between the ISPs, they were skeptical. Their disbelief soon vanished, as their tests confirmed that a router had been mis-configured at the IXP, causing packets to be dropped between them.
Each ISP had a small handful of complaints from customers that certain traffic was slow, but since their own individual networks had seemed healthy, they had dismissed the sporadic complaints, chalking them up to grumpy users who stayed up too late playing Internet games. They couldn’t believe that this problem had existed for quite some time, and that it was able to be diagnosed by a smart customer like Fred.
At the end, Fred solved a problem affecting some customers of both ISPs and VIPeter was convinced that Fred was an Internet Genius who not only fixed the company networks, but also fixed the public Internet. Fred made a quick mental note to remind VIPeter of this the next time that bonuses and reviews rolled around.
(Cue movie music.)
“All in a Net Admin Day’s Work,” Fred mumbled as he drove off into the sunset, headed to his local watering-hole for a well-deserved reward.
Stay tuned for our next cliffhanger episode, same place, same channel!
Related blog post: Monitoring and Troubleshooting Cloud Services: A Higher Standard
Want to read more about IXPs? Visit Wikipedia: Internet Exchange Point
 Cloud: Microsoft Office Clip Art (Online).
 River Rothay Stepping Stones, Wikimedia Commons, User Strobilomyces