In this month's post, I'll be explaining the new call grading mechanism. I would really like to hear your thoughts about this as well as encourage you to share this out to others for their input.
Mean opinion score is no longer the best way to grade audio quality. Concealment-related metrics with Cisco Prime™ Collaboration Assurance and Analytics offers many advantages including simplicity, better SLA enforcement, and codec independence. This blog explains the rationale behind replacing MOS with new call-quality metrics.
Cisco Prime Collaboration Assurance and Analytics uses mean opinion score (MOS, including Cisco® Voice Transmission Quality [Cisco VTQ] and the K-Factor algorithm) reported by IP phones to grade listener audio quality. The latest Cisco IP Phones (Cisco DX Series and Cisco IP Phone 7800 and 8800 Series devices) as well as future IP phones no longer report MOS. In addition to the reason for changing how we measure call quality, I'll explain:
The advantages of using concealed seconds (CS) and severely concealed seconds (SCS) metrics, a process that is more efficient and reliable on a typical enterprise environment
Usage of SCS Ratio (SCSR) (explained later) in Cisco Prime Collaboration Assurance and Analytics to grade audio quality
To enhance end-user experience, many Cisco IP Phones support wideband audio codecs (such as G.722, AAC-LD, and up-coming Opus). With this transition, MOS scores are no longer optimal to grade call quality:
MOS scores don't scale to wideband codecs.
The value of MOS scores is codec-dependent.
Wideband code uses the same algorithm as low-band code, but at an incompatible scale. A great, unimpaired wideband G.722 call will score lower than a corrupted, narrowband 711 call.
“Average MOS” over a 2-hour call has little or no meaning. Long calls will have an underweighted MOS score, short calls overweight.
The MOS score is not ideal for service-level agreement (SLA) enforcement because MOS scores can’t be added or averaged across calls of different duration or codecs.
Because of the nature of voice over IP (VoIP), loss and jitter are inevitable. If for any reason an IP phone doesn't have valid audio to play, packet lost concealment (PLC) is used to smoothly fill in the gap with “made-up” audio. Concealment statistics measure packet (frame) loss and its effect on voice quality in an impaired network.
All Cisco IP Phones report the following concealment-related metrics (refer to Table 1 and the call detail record (CDR) and call management record (CMR) administrator’s guide at this example).
CS: Concealed seconds represents the time during which some concealment is observed during a call.
SCS: Severely concealed seconds represents the time during which a significant amount of concealment is observed. If the concealment that is observed is usually greater than 50 milliseconds or approximately 5 percent, the speech probably does not seem very audible.
Cisco Prime Collaboration Assurance and Analytics 11.0 uses SCSR (that is, SCS ÷ call duration to grade good, acceptable, or poor) listening audio quality. The advantages of using SCSR include the following:
It is simple and intuitive because SCSR is based on raw, network-oriented, packet-related counts.
It is standardized; Cisco publishes values openly as a component of the draft RFC RTCP-HR (Real-Time Control Protocol - High Resolution).
It can add CS and SCS across thousands of calls, for SLA. It does not over- or under-weight calls based on duration.
It is codec-independent. It reflects dynamic quality of transmission, not the codec baseline.
SCSR grades calls with below-default configurable thresholds.
Note that all audio quality metrics listed in Table 1 are stored in the CMR. For phones that do not generate CMR, such as Cisco TelePresence® TC, CE, CTX, and IX endpoints; Jabber® platforms (all platforms); Cisco IP Communicator; and Cisco Unified Personal Communicator, grading will be “N/A”.
Let me know what you think about the new way of call grading. Are you implementing? Let's discuss. Check out more information about Cisco Prime Collaboration.