VoIP Testing with TEMS Investigation

PC-based Clients
Technical Paper

Prepared by:
Ascom Network Testing

Date:
5/6/2013

Document:
NT11-12850

Ascom (2013)
TEMS is a trademark of Ascom. All other trademarks are the property of their respective holders.
No part of this document may be reproduced in any form without the written permission of the copyright holder.
The contents of this document are subject to revision without notice due to continued progress in methodology, design and
manufacturing. Ascom shall have no liability for any error or damage of any kind resulting from the use of this document.

Contents

Ascom (2013)

Introduction ................................................................ 1

How VoIP Works: A Brief Technology

Overview ..................................................................... 1

Testing VoIP with TEMS Investigation ..................... 2

3.1
3.2
3.2.1
3.3
3.4
3.4.1
3.4.2
3.4.3
3.4.4
3.5
3.6

Physical Configuration .............................................................2

Scripting ....................................................................................3
Tips on Scripting .........................................................................5
Voice Quality Measurement .....................................................7
Output........................................................................................7
VoIP-specific Information Elements ............................................7
Other Information Elements of Interest .....................................10
VoIP Events ..............................................................................10
VoIP KPIs (Key Performance Indicators) ..................................10
Presentation in TEMS Investigation Windows ......................11
Ascom Test Setup ..................................................................11

Troubleshooting....................................................... 11

4.1
4.2

Problem: Script Activity Fails ................................................11

Problem: Bad Audio Quality (PESQ/POLQA Score Low) .....12

Limitations ................................................................ 12

Appendices............................................................... 13

6.1
6.1.1
6.1.2
6.1.3
6.1.4
6.1.5
6.1.6
6.1.7
6.2

SIP Response Codes ..............................................................13

Informational Responses ..........................................................13
Successful Responses..............................................................13
Redirection Responses .............................................................13
Client Failure Responses ..........................................................13
Server Failure Responses ........................................................15
Global Failure Responses.........................................................15
Extended Codes .......................................................................15
Abbreviations ..........................................................................16

Document:
NT11-12850

Introduction

VoIP, Voice over IP, is a technology for delivering voice communications

over IP networks such as the Internet. Being a voice-over-data service,
VoIP has characteristics from both realms. On one hand it is very delaysensitive (like circuit-switched voice); on the other it is subject to all of the
various challenges associated with packet-switched services, such as
keeping down packet loss and jitter. This makes VoIP a very complicated
service to optimize.
The present paper describes VoIP in general terms and tells how to test
and measure VoIP performance using TEMS Investigation PC-based
clients. The document does not cover VoIP testing with on-device clients,
which is another option available in TEMS Investigation.

How VoIP Works: A Brief Technology

Overview

VoIP (Voice over IP) is an umbrella term for technologies that enable
delivery of voice calls and multimedia sessions over IP networks, such as
the Internet, rather than the public switched telephone network (PSTN).
The voice signal is digitized and encoded using audio codecs, just as in
circuit-switched cellular telephony, and then divided into IP packets for
transmission over the packet-switched network. On the receiving side
similar steps are applied in the reverse order to reproduce the original voice
stream: reception and decoding of IP packets followed by digital-to-analog
conversion.
The range of audio codecs used differs between VoIP implementations;
some implementations rely on narrowband and compressed speech, while
others support high fidelity stereo codecs.
VoIP systems employ session control protocols to control the setup and
teardown of calls. Examples of such protocols are:

H.323

Media Gateway Control Protocol (MGCP)

Session Initiation Protocol (SIP)

Real-time Transport Protocol (RTP)

Session Description Protocol (SDP)

Of these, SIP and RTP have gained particularly widespread use, and these
protocols also figure in the present document.

Ascom (2013)

Document:
NT11-12850

1(16)

Testing VoIP with TEMS Investigation

3.1

Physical Configuration

PC-based VoIP testing is conducted using two instances of TEMS

Investigation, installed on two different PCs, each of which have a mobile
phone connected.1 This setup is necessary to enable end-to-end speech
quality measurement for VoIP.
The calling device (caller) is connected to one PC and the called device
(callee) to the other. Audio is sent in semi-duplex fashion between the
parties, that is, in both directions but only in one direction at a time.
TEMS Investigation has a built-in PJSIP VoIP client; the VoIP clients thus
reside in the PCs and not in the mobile devices.
It should be noted from the outset that no further devices running data
services can be connected to the PCs during VoIP testing. See also
chapter 5.

Callee
side

Caller
side
VoIP
server

VoIP
client

Mobile phones

VoIP
client

PCs running TEMS Investigation

Schematic diagram of physical configuration for VoIP testing.

Two TEMS Investigation licenses are thus also required, as well as a special
license option for VoIP.

Ascom (2013)

Document:
NT11-12850

2(16)

3.2

Scripting

Two scripts are needed in TEMS Investigation, one for the caller and one
for the callee. Predefined snippets, VoIP PC Dial and VoIP PC Answer,
are supplied with TEMS Investigation for this purpose.
It is worth underlining that the timing between caller and callee is essential.
The callee must be registered with the SIP server and finish its
preparations for answering before the caller dials the call. See step
below.
VoIP PC Dial snippet

VoIP PC Answer snippet

Screenshots of VoIP PC Dial snippet (left) and VoIP PC Answer snippet (right) as
displayed in the TEMS Investigation Service Control Designer. The numbering refers
to the step-by-step description that follows below.

Ascom (2013)

Document:
NT11-12850

3(16)

Network Connect
First, both parties need to have an active data session. This is done in
TEMS Investigation through the Network Connect activity.

SIP Registration
Before a VoIP call can begin, both caller and callee must register with the
SIP server to be used for VoIP. The SIP Register activity is used for this
purpose. Here you indicate the IP address or host name of the server to
use. If no special domain needs to be chosen, enter the server address
under Domain, and leave Proxy empty. If on the other hand you need to
specify a domain within the server, enter the server address in the Proxy
field and the domain in the Domain field. You also specify the user and
password the client should use when registering.

Answer
The callee must be ready to answer before the caller can initiate a call. To
this end the callee executes the script activity Answer with Call Type set to
VoIP PC. In this activity you select the audio codec and encoding rate the
callee should use. The callee will communicate these settings to the caller,
so that the parties agree on the same codec and rate.
To ensure that the callee has reached Answer before the caller dials the
call, you should insert a wait period in the callers script. See section 3.2.1.
This detail has been left out of the above diagram to keep things
straightforward.

Dial
The caller initiates the call by running the activity Dial, again with Call Type
set to VoIP PC. In this activity you indicate the codec the caller should
use, which must be the same as the callees designated codec (Answer
activity, see step ). You also specify the codec rate and the phone
number to call.

Voice Quality Measurement

Once the call has been connected, voice quality can be measured on both
sides using one out of several algorithms supported (see section 3.3 for
details on this matter). This is done with the Voice Quality activity, where
Audio Source is set to VoIP (PC). The call durations should preferably
differ between caller and callee, so that it can be controlled which side
hangs up the call. Compare step .
It is possible to store audio files containing the received audio. All audio
sentences having a MOS score lower than or equal to the MOS limit will be

Ascom (2013)

Document:
NT11-12850

4(16)

stored on the PC.2 If three consecutive voice quality measurement reports

with indication of silence are received, the VoIP call is terminated.
If for some reason you do not wish to measure voice quality, simply use a
Wait activity instead of Voice Quality. However, be aware that in this case
you will not obtain any other VoIP quality measurements either, such as
FER and jitter buffer metrics (see the list in section 3.4.1), nor will MTSI
Session Completion Failure events (dropped calls) be generated.

Hangup
One of the parties (the one with the duration of the Voice Quality activity
set lower) hangs up the call. This is to ensure that the party hanging up has
the time to do so before the other party unregisters; otherwise the hangup
will fail. This is done through the Hang Up activity (Call Type = VoIP PC).
In the above diagram, the caller performs the hangup.

SIP Unregister
Both sides unregister from the SIP server. This is done using the SIP
Unregister activity.

Network Disconnect
Finally the data session is terminated as each party performs a Network
Disconnect. If the snippet is executed in a loop, this activity is necessary to
force a disconnect from the network after each VoIP call (desirable for the
purpose of KPI calculation).

3.2.1

Tips on Scripting

The above description covered the key steps in conducting a VoIP call. In
practice, the scripts should be made slightly more sophisticated.
Suggested setup:

On both sides, run all activities within a while loop. Add an extra Wait
activity as the last item in each loop, with the callers wait period longer
than the callees, to make sure the callee is ready and waiting for the
call when the caller dials. Suggested wait durations are 30 s for the
caller and 10 s for the callee. Compare step in section 3.2 above.

To ensure that the timing becomes right for the first VoIP call, start the
script on the callee side first, then the caller script.

If the parties are not in sync, that is, if the callee is not registered when the
caller places the call, then the caller will generate an MTSI Session Setup
Failure event (see section 3.4.3).

Storage location: C:\Users\<user>\Documents\TEMS Product Files\TEMS

Investigation <version no.>\PESQ.

Ascom (2013)

Document:
NT11-12850

5(16)

Caller

Callee

Use of while loops with VoIP scripting: caller (left) and callee (right).

Ascom (2013)

Document:
NT11-12850

6(16)

3.3

Voice Quality Measurement

The Voice Quality script activity has an Algorithm parameter where the
following choices can be made:

PESQ P862.1

POLQA P863 Narrowband

POLQA P863 Super Wideband

Non-intrusive

PESQ and POLQA are industry standard algorithms for assessing voice
quality as perceived by a human listener, POLQA being a refinement of the
older PESQ algorithm. Their output is a value on the MOS scale ranging
from 1 (worst) to 5 (best). The score obtained is a function of the radio
environment, of the speech codec and codec rate used, and of other
factors. Both algorithms are dealt with at length in the document AQM in
TEMS Products (Including PESQ and POLQA).
Each of the two algorithms requires the purchase of a special license option
to be enabled in TEMS Investigation. Even without PESQ and POLQA,
however, a more basic estimate of voice quality is obtained in the form of
the quantity VoIP FER Combined Packet Loss (see section 3.4.1.3), which
is always computed. This is the meaning of the term Non-intrusive.
Note that both uplink and downlink must use the same voice quality
algorithm. It is not possible to use PESQ on one link and POLQA on the
other, even if you possess license options for both.

3.4

Output

3.4.1

VoIP-specific Information Elements

All of these are found in the Data information element category in TEMS
Investigation unless otherwise noted.
3.4.1.1 Jitter
VoIP RFC 1889 Jitter (ms)

Packet jitter or delay variation as defined in IETF RFC 1889, section 6.3.1:
An estimate of the statistical variance of the RTP data packet interarrival
time [...] The interarrival jitter is defined to be the mean deviation
(smoothed absolute value) of the difference in packet spacing at the
receiver compared to the sender for a pair of packets. As shown in the
equation below, this is equivalent to the difference in the relative transit
time for the two packets [...].
If is the RTP timestamp from packet , and is the time of arrival in
RTP timestamp units for packet , then for two packets and , may be
expressed as

Ascom (2013)

Document:
NT11-12850

7(16)

The interarrival jitter is calculated continuously as each data packet is

received [...] using this difference for that packet and the previous
packet
in order of arrival (not necessarily in sequence), according
to the formula

The quantity is what is output in the VoIP RFC 1889 Jitter information
element. The latter is updated once every second.
3.4.1.2 Jitter Buffer
A jitter buffer is used to mitigate the effects of packet jitter. The jitter buffer
holds the received voice packets briefly, reorders them if necessary, and
then plays them out at evenly spaced intervals to the decoder.
These elements are updated once every second.

Ascom (2013)

VoIP Decoding Errors (%)

Percentage of audio frames that could not be

decoded by the speech codec.

VoIP Jitter Buffer Lost

Packets (%)

Percentage of packets that were missing from the

audio reproduction because they were not
delivered from the jitter buffer to the decoder in
timely fashion.
Note that the packet need not have been lost on
the way to the receiving party; it may just have
been delayed too long, so that it was discarded by
the jitter buffer.

VoIP Jitter Buffer Playout

Delay Average (ms)

Average playout delay in ms: that is, the average

time the voice packets were held by the jitter
buffer.

VoIP Jitter Buffer Playout

Delay Maximum (ms)

Maximum playout delay in ms.

VoIP Jitter Buffer Playout

Delay Minimum (ms)

Minimum playout delay in ms.

VoIP Jitter Buffer Size

Increase (%)

Percentage of audio frames where the VoIP client

decided to increase the jitter buffer size (because
the jitter was found to be too high). This
procedure results in a period of silence in the
audio reproduction as the jitter buffer accumulates
packets without releasing any.

Document:
NT11-12850

8(16)

VoIP Jitter Buffer

Overruns (%)

Percentage of audio frames with overruns.

The VoIP client tries to keep the delays caused by
the jitter buffer reasonably low. When the buffer
becomes too long, the VoIP client will throw away
received packets to decrease the buffer size. This
is referred to as overruns, and it affects the audio
reproduction.
Usually occurs after underruns (see below).

VoIP Jitter Buffer

Underruns (%)

Percentage of audio frames where the jitter buffer

was empty and had no packets to deliver to the
speech decoder.

3.4.1.3 Audio Quality Related

Data category

VoIP FER Combined

Packet Loss (%)

Total percentage of packet loss that affects the

reproduction of the audio. Encompasses decoding
errors, underruns, overruns, and jitter buffer size
increases: compare the information elements in
section 3.4.1.2. Should in general correlate
closely to PESQ and POLQA.

VoIP Speech Codec

Speech codec selected for the VoIP client in the

governing script (Dial and Answer activities: see
section 3.2, steps and ).

Media Quality category (see also section 3.3):

PESQ Score Downlink

PESQ (ITU P.862.1) voice quality score.

For VoIP measurements the speech sentences
are 5.5 s in length. This means that a MOS score
will be calculated every 11 s (since transmissions
are done in semi-duplex). Note that a
performance degradation that occurs while the
measurement is done at the other end will not
register in the PESQ score.

POLQA NB Score Downlink

POLQA (ITU P.863.1) voice quality score for

narrowband.

POLQA SWB Score

Downlink

POLQA voice quality score for super-wideband.

In the real time presentation, the PESQ and POLQA scores appear the
moment they have been computed. When loading a logfile for analysis, on
the other hand, the PESQ and POLQA scores are moved backward in time
to the point when the corresponding speech sentence was received by the
VoIP client. That is, sentences are aligned in time with their quality scores.
This is not much of an issue for PESQ, which takes only a fraction of a
second to compute, but it can be for POLQA, whose computation may

Ascom (2013)

Document:
NT11-12850

9(16)

require several seconds (the worse the degradation of the signal, the more
complex POLQA is to evaluate).

3.4.2

Other Information Elements of Interest

Application throughput IEs (Data category).

RAN throughput at various protocol levels (details being dependent on

the cellular technology used; the IEs are found in the relevant category,
such as LTE, WCDMA).

3.4.3

VoIP Events

These events underlie the KPIs in section 3.4.4:

MTSI Registration Failure

One of the parties failed to register with the SIP

server.

MTSI Registration Time

Time required for the terminal to register with the

SIP server. Also functions as a success event.

MTSI Session Completion

Failure

A VoIP session that was successfully set up failed

to complete. Similar to dropped call for CS voice.

MTSI Session Completion

Time

Duration of the VoIP session. Also functions as a

success event.
Note: This event does not have an associated
KPI, since the VoIP session duration is not a
relevant performance measure.

MTSI Session Setup

Failure

The terminal failed in setting up a VoIP session.

Similar to blocked call for CS voice.

MTSI Session Setup Time

Time required to set up the VoIP session. Also

functions as a success event.

TEMS Investigation also generates the following VoIP events, which are
unrelated to KPI computation:
VoIP Start

A VoIP session was started.

VoIP End

A VoIP session ended normally.

VoIP Error

A VoIP session was aborted because of an error.

3.4.4

VoIP KPIs (Key Performance Indicators)

TEMS Investigation provides data for computation of the following KPIs.

The actual computation is done in TEMS Discovery or TEMS Automatic.
MTSI Registration Failure
Ratio (%)

Ascom (2013)

Denotes the probability that the terminal cannot

register towards IMS when requested.

Document:
NT11-12850

10(16)

MTSI Registration Time (s)

Denotes the time elapsing from the IMS

registration request until the terminal is registered
to IMS.

MTSI Session Setup

Failure Ratio (%)

Denotes the probability that the terminal cannot

set up an MTSI session. An MTSI session setup is
initiated when the user presses the call button
and concludes when the user receives, within a
predetermined time, a notification that the callee
has answered.

MTSI Session Setup Time

(s)

Denotes the time elapsing from initiation of an

MTSI session until a notification is received that
the session has been set up.

MTSI Session Completion

Failure Ratio (%)

Denotes the probability that a successfully set up

MTSI call is ended by a cause other than
intentional termination by either party.

3.5

Presentation in TEMS Investigation Windows

Suitable presentation windows for VoIP data:

VoIP Quality status window containing the information elements

described in section 3.4.1

VoIP Quality Line Chart tracking VoIP PESQ/POLQA scores and VoIP
FER Combined Packet Loss, also indicating MTSI events

VoIP AMR Codecs Usage status window

Data Reports message window

IP Protocol Reports message window.

3.6

Ascom Test Setup

The VoIP function in TEMS Investigation has been tested with TekSIP, a
SIP Registrar and SIP Proxy for Windows (www.teksip.com), as well as
with an Ericsson IMS server.

Ascom (2013)

Troubleshooting

4.1

Problem: Script Activity Fails

Check that caller and callee are synchronized, that is, that the callee
reaches Answer before the caller begins Dial. See section 3.2, step 3,
and section 3.2.1.

In the Events window, look for MTSI failure events.

In the Data Reports message window, look into the VoIP Error
Message category.

Document:
NT11-12850

11(16)

In the IP Protocol Reports message window, study the SIP messages.

If any other ports than SIP port 5060 and RTP port 4000 are used on
the SIP server, the corresponding settings have to be changed in the
file <TEMS Investigation install dir>\Application\Configuration\
Investigation.Voip.config.

If SIP response code 422 (Session interval too small) is received, set
DisableTimers="false" in the same file.

4.2

Investigate throughput and BLER values at different levels. Example:

For LTE, this includes the application layer, PDSCH, RLC, PDCP, and
MAC. Remember to look at both uplink and downlink.

Check channel quality indicators and serving/neighbor signal strength.

Example: In an LTE network, study CQI, Serving Cell RSRP, and
Neighbor Cell RSRP.

Check for excessively frequent handovers.

In the IP Protocol Reports message window, look into the RTP

messages.

Ascom (2013)

Problem: Poor Audio Quality (PESQ/POLQA

Score Low)

Limitations
You cannot have any other internet connections in parallel while running
VoIP measurements. That is, the PCs cannot be connected to any
further IP addresses, whether through other external devices, through
an Ethernet cable, or by other means. All network interfaces except
the testing devices, both fixed and wireless, must be disabled. It is
however possible to make CS voice calls with devices connected to the
PCs.

Document:
NT11-12850

12(16)

Appendices

6.1

SIP Response Codes

6.1.1

Informational Responses

100

Trying

180

Ringing

181

Call is being forwarded

182

Queued

183

Session progress

6.1.2

Successful Responses

200

OK

202

Accepted
Indicates that the request has been understood but actually cannot
be processed

6.1.3

Redirection Responses

300

Multiple choices

301

Moved permanently

302

Moved temporarily

305

Use proxy

380

Alternative service

6.1.4

Client Failure Responses

400

Bad request

401

Unauthorized
Used only by registrars or user agents. Proxies should use proxy
authorization 407

402

Payment required
Reserved for future use

403

Forbidden

404

Not found
User not found

Ascom (2013)

405

Method not allowed

406

Not acceptable

407

Proxy authentication required

Document:
NT11-12850

13(16)

408

Request timeout
Could not find the user in time

409

Conflict

410

Gone
The user existed once, but is no longer available here

412

Conditional request failed

413

Request entity too large

414

Request URI too long

415

Unsupported media type

416

Unsupported URI scheme

417

Unknown resource priority

420

Bad extension
Bad SIP protocol extension used, not understood by the server

421

Extension required

422

Session interval too small

423

Interval too brief

424

Bad location information

428

Use identity header

429

Provide referrer identity

433

Anonymity disallowed

436

Bad identity info

437

Unsupported certificate

438

Invalid identity header

480

Temporarily unavailable

481

Call/transaction does not exist

482

Loop detected

483

Too many hops

484

Address incomplete

485

Ambiguous

486

Busy here

487

Request terminated

488

Not acceptable here

489

Bad event

491

Request pending

493

Undecipherable
Could not decrypt S/MIME body part

494

Ascom (2013)

Security agreement required

Document:
NT11-12850

14(16)

6.1.5

Server Failure Responses

500

Server internal error

501

Not implemented
The SIP request method is not implemented here

502

Bad gateway

503

Service unavailable

504

Server timeout

505

Version not supported

The server does not support this version of the SIP protocol

513

Message too large

580

Precondition failure

6.1.6
600

Busy everywhere

603

Decline

604

Does not exist anywhere

606

Not acceptable

6.1.7

Ascom (2013)

Global Failure Responses

Extended Codes

701

The called party has hung up

702

VoIP socket error

703

Connection cancelled because of timeout

704

Connection interrupted because of a SIP error

705

SIP memory error

706

SIP transaction memory error

751

Busy tone: No codec match between the calling and called

party

810

General socket layer error

811

General socket layer error: Wrong socket number

812

General socket layer error: Socket is not connected

813

General socket layer error: Memory error

814

General socket layer error: Socket not available check IP

settings/connection problem/VoIP setting incorrect

815

General socket layer error: Illegal application on the socket

interface

922

No DNS server known

923

DNS name resolution failed

Document:
NT11-12850

15(16)

924

Insufficient resources for DNS name resolution

925

URL error

6.2

Ascom (2013)

Abbreviations

AMR-NB

Adaptive Multi Rate Narrowband

AMR-WB

Adaptive Multi Rate Wideband

BLER

Block Error Rate

CQI

Channel Quality Indicator

FER

Frame Erasure Rate

IMS

IP Multimedia Subsystem

IP

Internet Protocol

KPI

Key Performance Indicator

LTE

Long Term Evolution

MAC

Medium Access Control

MOS

Mean Opinion Score

MTSI

Multimedia Telephony Service for IMS

PDSCH

Physical Downlink Shared Channel

PESQ

Perceptual Evaluation of Speech Quality

POLQA

Perceptual Objective Listening Quality Assessment

PSTN

Public Switched Telephone Network

RAN

Radio Access Network

RSRP

Reference Signal Received Power

RTP

Real-time Transport Protocol

SIP

Session Initiation Protocol

VoIP

Voice over IP

Document:
NT11-12850

16(16)