VoIP Monitor
Poll frequency: 00:05:00
Last poll: 1/19/2012 8:36:59 PM
Network health: DEGRADED (1.0%)
Network Assessment Report
Report generated on 1/19/2012 at 8:36:59 PM
Network Monitor Release v4.2 (5265)

Contents

  1. Executive Summary
    1. Explanation of objectives
    2. Software used to collect assessment data
    3. Report Interpretation
  2. Reference
    1. Summary
    2. Monitored Devices
    3. Device Inventory
    4. WAN Interface Connections
    5. WAN Interface Utilization
    6. Cisco Device CPU Utilization
    7. Cisco Device Free RAM
  3. MOS Results
    1. MOS Score to Remote Devices
    2. Call Simulations
  4. Recommendations
    1. Discovered Network Issues
    2. WAN Interface Recommendations
    3. Half-Duplex Interfaces
    4. 10Meg Interfaces

I. Executive Summary

I.a. Explanation of objectives

The objective of this report is to identify network weaknesses throughout the infrastructure that will lead to call quality problems. All network devices and interfaces are checked for health and performance statistics every 00 hours, 05 minutes, 00 seconds. As a result, any network anomalies that occur anywhere in the network, any time of day will be detected and reported.

Call Simulations should also be performed from endpoint to endpoint to further validate that the network is properly handling QoS tagging for VoIP calls. Those reports should be attached to this report as part of a complete assessment.

I.b. Software used to collect assessment data

This assessment report has been generated by PathSolutions Network Monitor Release v4.2 (5265). Individual call simulator reports will idenfity the version of the Call Simulator used for each report individually.

I.c. Report Interpretation

This assessment report is divided into three main sections: Reference, MOS Results, and Work to be Done.

The Reference section identifies the network and its current condition and capabilities Each of these items should be investigated further to insure that the network is running according to understood parameters. Care should be taken with the interpretation of the reference information to determine additional network improvements that should be implemented prior to VoIP deployment.

The MOS Results section identifies MOS (Mean Opinion Score) results from the monitoring server out to the monitored devices as well as manually run Call Simulations from various points in the network. If the MOS results show drop-offs or lower quality ratings than expected, the root-causes of the problems may lie with the third section of this report: Recommendations.

The Work to be Done section outlines existing network faults and weaknesses that should be remedied prior to VoIP deployment. Improvements to this section of the report should yield better MOS Results with subsequent tests.

The best practices regarding performing a network assessment include fixing all of the discovered network faults and then running simulated VoIP calls through the network from various points. If the MOS Results from the simulated calls show problems, perform a call path mapping to determine where additional network weaknesses lie.

Optimally, the network should be continuously monitored for health and performance to prevent call quality issues from occurring, as a one-time network assessment tends to not be as valuable as continuous ongoing monitoring.

II. Reference

II.a. Summary

For this report, 22 devices and a total of 587 interfaces were monitored.

9 WAN links with bandwidth less than 10meg were discovered.

6 interfaces were discovered above theshold levels (80% utilization and 3% packet loss) in the past 24 hours.

II.b. Monitored Devices

The following network devices were monitored as part of the assessment.

Device Name IP Address Interfaces Location Contact
HQ SG 40/8 10.100.36.100 1 HQ Santa Clara ShoreTel
SF SG 40/8 10.100.37.100 1 San Francisco ShoreTel
VELMA 10.100.36.10 1

DAGWOOD 10.100.36.11 1 Santa Clara, CA Tim Titus
Bordeaux 10.100.37.2 26 Santa Clara Tim Titus x111
Chiraz 10.100.37.3 48 Santa Clara, CA Tim Titus
SC_Server 10.0.12.5 58 SC IT dlit@PathSolutions.com
SC_User_SW1 10.0.12.6 58 SC IT dlit@PathSolutions.com
SC_User_SW2 10.0.12.7 58 SC IT dlit@PathSolutions.com
QA_SW1 10.0.12.8 58 QA dlit@PathSolutions.com
QA_SW2 10.0.12.9 58 QA dlit@PathSolutions.com
PinotNoir 10.100.36.49 114 Santa Clara, CA Tim x111
Chardonnay 10.100.36.51 24 Sunnyvale, CA noc@pathsolutions.com
Gewurztraminer 10.100.36.55 24 Santa Clara, CA Tim Titus
Muscat 192.168.202.50 26 Santa Clara, CA Tim Titus
SCWANRTR 38.102.148.163 6

Internet 10.100.36.1 2 San Francisco, CA Tim Titus x4413
Denver 10.100.36.60 2 Denver, CO ttitus@pathsolutions.com
NewYork 192.168.201.2 2 New York, NY ttitus@pathsolutions.com
Atlanta 192.168.202.2 3 Atlanta, GA John Bejarano x4312
EdgeMarc4500 10.100.36.50 4 Sunnyvale, CA ttitus@pathsolutions.com
Router 10.100.36.65 12 Sunnyvale, CA noc@pathsolutions.com

II.c. Device Inventory

The following device internal descriptions are as follows.

Device OS
HQ SG 40/8
10.100.36.100
ShoreGear 40/8
SF SG 40/8
10.100.37.100
ShoreGear 40/8
VELMA
10.100.36.10
Hardware: x86 Family 6 Model 5 Stepping 2 AT/AT COMPATIBLE - Software: Windows 2000 Version 5.0 (Build 2195 Uniprocessor Free)
DAGWOOD
10.100.36.11
Hardware: x86 Family 15 Model 6 Stepping 4 AT/AT COMPATIBLE - Software: Windows Version 5.2 (Build 3790 Multiprocessor Free)
Bordeaux
10.100.37.2
FSM700S Managed Stackable Switch
Chiraz
10.100.37.3
Ethernet Switch 470-48T HW:#01 FW:3.6.0.7 SW:v3.6.4.08 BN:8 ISVN:2 (c) Nortel Networks
SC_Server
10.0.12.5
ProCurve J9147A 2910al-48G Switch, revision W.14.38, ROM W.14.04 (/sw/code/build/sbm(t4a))
SC_User_SW1
10.0.12.6
ProCurve J9147A 2910al-48G Switch, revision W.14.38, ROM W.14.04 (/sw/code/build/sbm(t4a))
SC_User_SW2
10.0.12.7
ProCurve J9147A 2910al-48G Switch, revision W.14.38, ROM W.14.04 (/sw/code/build/sbm(t4a))
QA_SW1
10.0.12.8
ProCurve J9147A 2910al-48G Switch, revision W.14.38, ROM W.14.04 (/sw/code/build/sbm(t4a))
QA_SW2
10.0.12.9
ProCurve J9147A 2910al-48G Switch, revision W.14.38, ROM W.14.04 (/sw/code/build/sbm(t4a))
PinotNoir
10.100.36.49
Ethernet Switch
Chardonnay
10.100.36.51
Cisco Internetwork Operating System Software IOS (tm) C2900XL Software (C2900XL-H2S-M), Version 12.0(5.1)XP, MAINTENANCE INTERIM SOFTWARE Copyright (c) 1986-1999 by cisco Systems, Inc. Compiled Fri 10-Dec-99 10:57 by cchang
Gewurztraminer
10.100.36.55
NetVanta 1224, Version: 13.07.04.SP4242, Date: Wed Jun 06 13:42:20 2007
Muscat
192.168.202.50
T-E24
SCWANRTR
38.102.148.163
Cisco IOS Software, 2800 Software (C2800NM-ADVIPSERVICESK9-M), Version 12.4(9)T7, RELEASE SOFTWARE (fc3) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2008 by Cisco Systems, Inc. Compiled Thu 10-Jan-08 16:35 by prod_rel_team
Internet
10.100.36.1
Cisco Internetwork Operating System Software IOS (tm) C2600 Software (C2600-J1S3-M), Version 12.3(19), RELEASE SOFTWARE (fc2) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2006 by cisco Systems, Inc. Compiled Fri 12-May-06 04
Denver
10.100.36.60
Cisco Internetwork Operating System Software IOS (tm) 1600 Software (C1600-SY-L), Version 12.2(40), RELEASE SOFTWARE (fc1) Copyright (c) 1986-2006 by cisco Systems, Inc. Compiled Thu 12-Oct-06 20:28 by pwade
NewYork
192.168.201.2
Cisco Internetwork Operating System Software IOS (tm) 1600 Software (C1600-SY-L), Version 12.2(40), RELEASE SOFTWARE (fc1) Copyright (c) 1986-2006 by cisco Systems, Inc. Compiled Thu 12-Oct-06 20:28 by pwade
Atlanta
192.168.202.2
Cisco Internetwork Operating System Software IOS (tm) 1600 Software (C1600-SY-M), Version 12.2(40), RELEASE SOFTWARE (fc1) Copyright (c) 1986-2006 by cisco Systems, Inc. Compiled Thu 12-Oct-06 20:28 by pwade
EdgeMarc4500
10.100.36.50
Linux EdgeMarc4500 2.4.31-uc0 #1 Fri Apr 10 00:17:24 PDT 2009 armv5b
Router
10.100.36.65
NetVanta 3448, Version: 17.03.01.00.E, Date: Mon Aug 04 10:18:52 2008

II.d. WAN Interface Connections

Queuing configuration on WAN interfaces is very important to insure proper prioritization of VoIP traffic. The following describes different queuing configurations that may exist on WAN interfaces:

  • FIFO:First In First Out
  • WFQ:Weighted Fair Queuing
  • PQ:Priority Queuing
  • MQC:Modular Qos Cli
  • CQ:Custom Queuing

The following WAN interfaces below 10mbps were detected on the network.

Device Name Interface Number IP Address Description Speed Queuing Configuration
Denver Int #2 192.168.201.1 Se0: Serial0 1,544,000 FIFO
NewYork Int #2 192.168.201.2 Se0: Serial0 1,544,000 WFQ

II.e. WAN Interface Utilization

The following WAN interfaces have bandwidth utilization below 10mbps and may have contention if they have high utilization and VoIP traffic passes along this link. QoS should be evaluated for these links to prevent packet loss during periods of high utilization.

Device & Interface Description
Denver
Int #2
Se0: Serial0
Speed: 1,544,000 bps
Peak Tx Rate: 0.547496%
Peak Rx Rate: 0.618307%
NewYork
Int #2
Se0: Serial0
Speed: 1,544,000 bps
Peak Tx Rate: 0.621762%
Peak Rx Rate: 0.544041%

II.f. Cisco Device CPU Utilization

CPU utilization is tracked for Cisco devices across the network. The health over the past 24hours is displayed below for each Cisco device.

Device Name Daily CPU Utilization
Chardonnay
10.100.36.51
SCWANRTR
38.102.148.163
Internet
10.100.36.1
Denver
10.100.36.60
NewYork
192.168.201.2
Atlanta
192.168.202.2

II.g. Cisco Device Free RAM

Free RAM is tracked for Cisco devices across the network. The health over the past 24hours is displayed below for each Cisco device.

Device Name Daily Free RAM
Chardonnay
10.100.36.51
SCWANRTR
38.102.148.163
Internet
10.100.36.1
Denver
10.100.36.60
NewYork
192.168.201.2
Atlanta
192.168.202.2

III. MOS Results

III.a. MOS Score to Remote Devices

The following devices were monitored for MOS (Mean Opinion Score) over the past 24 hours. If any score is below acceptable levels, further investigation should be performed.

Device Name Daily Graph
HQ SG 40/8
10.100.36.100
Min: 4.4 Avg: 4.4 Max: 4.4
SF SG 40/8
10.100.37.100
Min: 1.0 Avg: 4.2 Max: 4.4
VELMA
10.100.36.10
Min: 4.4 Avg: 4.4 Max: 4.4
DAGWOOD
10.100.36.11
Min: 4.4 Avg: 4.4 Max: 4.4
Bordeaux
10.100.37.2
Min: 4.3 Avg: 4.4 Max: 4.4
Chiraz
10.100.37.3
Min: 4.3 Avg: 4.4 Max: 4.4
SC_Server
10.0.12.5
Min: 4.4 Avg: 4.4 Max: 4.4
SC_User_SW1
10.0.12.6
Min: 4.4 Avg: 4.4 Max: 4.4
SC_User_SW2
10.0.12.7
Min: 4.4 Avg: 4.4 Max: 4.4
QA_SW1
10.0.12.8
Min: 4.4 Avg: 4.4 Max: 4.4
QA_SW2
10.0.12.9
Min: 4.4 Avg: 4.4 Max: 4.4
PinotNoir
10.100.36.49
Min: 4.4 Avg: 4.4 Max: 4.4
Chardonnay
10.100.36.51
Min: 4.4 Avg: 4.4 Max: 4.4
Gewurztraminer
10.100.36.55
Min: 4.4 Avg: 4.4 Max: 4.4
Muscat
192.168.202.50
Min: 4.2 Avg: 4.4 Max: 4.4
SCWANRTR
38.102.148.163
Min: 4.4 Avg: 4.4 Max: 4.4
Internet
10.100.36.1
Min: 4.4 Avg: 4.4 Max: 4.4
Denver
10.100.36.60
Min: 1.0 Avg: 3.8 Max: 4.0
NewYork
192.168.201.2
Min: 4.1 Avg: 4.3 Max: 4.3
Atlanta
192.168.202.2
Min: 3.1 Avg: 4.3 Max: 4.3
EdgeMarc4500
10.100.36.50
Min: 4.3 Avg: 4.4 Max: 4.4
Router
10.100.36.65
Min: 4.1 Avg: 4.4 Max: 4.4

III.b. Call Simulations

Simulated phone calls should be made across each WAN link for a 24hour period.

IV. Recommendations

IV.a. Discovered Network Issues

The following network issues were discovered on the network. These issues should be evaluated to determine if they should be resolved prior to deploying a VoIP phone system.

Device & Interface Description Peak Daily Error Rate Peak Daily Transmit Rate Peak Daily Receive Rate
Muscat
192.168.202.50
Int #3
swp02
  • Collisions exist on this interface
    This can be eliminated by configuring the interface and device to work in full-duplex mode. This may not be possible if more than one device is connected to this interface. If this interface is plugged into a single device, then full-duplex may be enabled (providing the network card can recognize full duplex). If this interface has a hub plugged in, then full-duplex operation cannot be enabled.
  • Deferred Transmissions exist on this interface
    Deferred transmissions are transmissions that could not be sent because the physical medium was busy. If this error occurs frequently, you should consider increasing the bandwidth of this link, or employing full duplex switches to reduce the traffic on this segment.
40.774% 0.100% 0.078%
Muscat
192.168.202.50
Int #13
swp12
  • FCS Errors exist on this interface
    FCS Errors can occur when a cable has noise or crosstalk. This may occur continuously if a poor quality cable is used, or intermittently if a network cable is located near an AC power source or heavy machinery that is used infrequently. A cable tester should be deployed on this segment to investigate the exact source of this problem.
  • Inbound Errors exist on this interface
    Inbound errors are packets that are mal-formed, but are enclosed in a valid frame. This can be caused by a bad NIC driver or protocol driver on the sending device. To track down this error, you will need to connect a packet analyzer in front of this interface to capture the actual mal-formed packet to determine which device is at fault.
40.680% 0.110% 0.091%
Router
10.100.36.65
Int #7
swx 0/3: swx 0/3: Fast Ethernet (BCM53xx)
  • Interface configured for half-duplex operation
    This interface should be configured for full-duplex operation to prevent collisions from occurring and error rates rising.
  • Collisions exist on this interface
    This can be eliminated by configuring the interface and device to work in full-duplex mode. This may not be possible if more than one device is connected to this interface. If this interface is plugged into a single device, then full-duplex may be enabled (providing the network card can recognize full duplex). If this interface has a hub plugged in, then full-duplex operation cannot be enabled.
  • Late Collisions exist on this interface
    This condition can occur with Ethernet switches that are configured for low-latency forwarding (like Fast-Forwarding or Fragment-Free forwarding) when multiple switches are daisy-chained together. If multiple switches are daisy-chained together, it is recommended that switches be configured for the "store-and-forward" method of forwarding to remedy this error.
15.857% 0.001% 0.003%
Chiraz
10.100.37.3
Int #11
ifc11 (Slot: 1 Port: 11): Nortel Ethernet Switch 470-48T Module - Port 11
  • Alignment errors exist on this interface
    There may be a Duplex problem on this interface. It appears that this interface is configured for full duplex. Alignment errors were detected with no collisions. There should be only one device connected to the other end of this interface, and that device should also be configured for full duplex. Note: Full duplex connections cannot function properly with a hub. If you need to use a hub, configure this interface for half-duplex. If there is no duplex problem on this interface, there may be a software driver problem. Check to see if all machines can communicate successfully on this segment.
  • FCS Errors exist on this interface
    FCS Errors can occur when a cable has noise or crosstalk. This may occur continuously if a poor quality cable is used, or intermittently if a network cable is located near an AC power source or heavy machinery that is used infrequently. A cable tester should be deployed on this segment to investigate the exact source of this problem.
  • FCS Errors exist on this interface
    FCS Errors can be caused by a duplex mismatch on an interface. You should check the duplex settings on this device as well as the device this interface is connected.
  • Inbound Discards exist on this interface
    Inbound packets had to be discarded because of a lack of available packet receive buffers. This can indicate that the device's internal CPU may be unable to process all of the inbound data that it is receiving.
  • Inbound Errors exist on this interface
    Inbound errors are packets that are mal-formed, but are enclosed in a valid frame. This can be caused by a bad NIC driver or protocol driver on the sending device. To track down this error, you will need to connect a packet analyzer in front of this interface to capture the actual mal-formed packet to determine which device is at fault.
3.365% 0.002% 0.001%

IV.b. WAN Interface Recommendations

  • Weighted Fair Queuing (WFQ) is employed
    Weighted Fair Queuing should not be utilized on links slower than 10megs in a VoIP environment, as it does not provide adequate prioritization for VoIP packets. Custom queuing or Modular Qos CLIshould be enabled to ensure bandwidth protection for VoIP packets.
  • First In First Out (FIFO) Queuing is employed
    FIFO Queuing should not be utilized on links slower than 10megs in a VoIP environment, as it does not provide any prioritization for VoIP packets. Custom queuing or Modular Qos CLIshould be enabled to ensure bandwidth protection for VoIP packets.

IV.c. Half-Duplex Interfaces

The following interfaces were shown to have collision counters or are set for half-duplex. They should be set for full-duplex on both sides of the connection to eliminate the possibility of collisions.

Device Name Interface Number Description Peak Daily Error Rate Duplex
Router Int #7 swx 0/3: swx 0/3: Fast Ethernet (BCM53xx) 15.857% Half
Internet Int #1 Fa0/0: FastEthernet0/0 (WAN side <FG726>) 14.462% Half
SC_User_SW2 Int #24 24: 24 (Path Solutions) 9.789% Half
Atlanta Int #3 Et1: Ethernet1 1.096% Half*
Gewurztraminer Int #3 eth 0/3: eth 0/3: Fast Ethernet (BCM56xx v17) 0.939% Half
Atlanta Int #2 Et0: Ethernet0 0.880% Half*
Internet Int #2 Fa0/1: FastEthernet0/1 0.820% Half
NewYork Int #1 Et0: Ethernet0 0.781% Half*
Denver Int #1 Et0: Ethernet0 0.509% Half*
Bordeaux Int #1 FastEthernet1 0.507% Half*
Gewurztraminer Int #14 eth 0/14: eth 0/14: Fast Ethernet (BCM56xx v17) 0.452% Half
SC_Server Int #19 19: 19 0.028% Half
* Estimated duplex setting. Device does not appear to support RFC2665 dot3StatsDuplexStatus

IV.d. 10Meg Interfaces

The following interfaces were shown to be operating at 10megabits per second. Most network adapters are capable of being set to 100mbps. These interfaces should be considered for upgrade to 100mbps to alleviate the possibility of bottlenecking.

Device Name Interface Number Description Speed Peak Daily Transmit Rate Peak Daily Receive Rate
SC_User_SW2 Int #7 7: 7 (43.1) 10,000,000 62.350% 74.450%
SC_User_SW2 Int #24 24: 24 (Path Solutions) 10,000,000 2.973% 1.957%
Internet Int #1 Fa0/0: FastEthernet0/0 (WAN side <FG726>) 10,000,000 2.045% 2.901%
Gewurztraminer Int #14 eth 0/14: eth 0/14: Fast Ethernet (BCM56xx v17) 10,000,000 0.113% 0.107%
Muscat Int #13 swp12 10,000,000 0.110% 0.091%
Denver Int #1 Et0: Ethernet0 10,000,000 0.105% 0.104%
Muscat Int #3 swp02 10,000,000 0.100% 0.078%
Bordeaux Int #1 FastEthernet1 10,000,000 0.075% 0.071%
NewYork Int #1 Et0: Ethernet0 10,000,000 0.086% 0.099%
Atlanta Int #3 Et1: Ethernet1 10,000,000 0.078% 0.090%
Atlanta Int #2 Et0: Ethernet0 10,000,000 0.071% 0.071%