Vous êtes sur la page 1sur 17

KPI Definitions in TEMS Products

NT13-25359, ver 2.0, 9/25/2013

Ascom. All rights reserved.


TEMS is a trademark of Ascom. All other trademarks are the property of their respective holders.
NT13-25359, ver 2.0, 9/25/2013

Contents
1

Introduction

General Aspects of KPI Data Collection and Computation

2.1

KPI Reporting.................................................................................................................... 1

2.2

ETSI Compliance .............................................................................................................. 1

2.3

Complete vs. Timed Measurements ................................................................................. 1

2.4

IP Capture Options ........................................................................................................... 2

KPIs by Service
3.1

3.2

3.3

KPIs for FTP Download .................................................................................................... 2


3.1.1

Timed Measurements .................................................................................................... 3

3.1.2

SFTP (Secure Shell FTP) .............................................................................................. 3

KPIs for FTP Upload ......................................................................................................... 3


3.2.1

Timed Measurements .................................................................................................... 3

3.2.2

SFTP (Secure Shell FTP) .............................................................................................. 4

KPIs for HTTP Get/Download ........................................................................................... 4


3.3.1

3.4

Timed Measurements .................................................................................................... 5

KPIs for Ping ..................................................................................................................... 5


3.5.1

3.6

Timed Measurements .................................................................................................... 4

KPIs for HTTP Post/Upload .............................................................................................. 4


3.4.1

3.5

Timed Measurements .................................................................................................... 5

KPIs for CS Voice ............................................................................................................. 5

Notes on Other Services: Email, Iperf, Streaming, CS Fallback

4.1

Email ................................................................................................................................. 6

4.2

Iperf................................................................................................................................... 7

4.3

4.4

NT13-25359, ver 2.0, 9/25/2013

4.2.1

TCP ............................................................................................................................... 7

4.2.2

UDP .............................................................................................................................. 7

Streaming over HTTP ....................................................................................................... 8


4.3.1

Streaming Player Download .......................................................................................... 8

4.3.2

Start of Streaming Video Replay .................................................................................... 9

4.3.3

Streaming Video Session .............................................................................................. 9

CS Fallback..................................................................................................................... 13

Introduction

This document describes the implementation of ETSI circuit-switched and packetswitched service KPIs, as well as some closely related non-ETSI KPIs, in TEMS
products.
The document is valid for the following TEMS product versions:

TEMS Automatic 10.1 and later

TEMS Investigation 16.0 and later

TEMS Symphony 7.4 and later

General Aspects of KPI Data Collection and


Computation

2.1

KPI Reporting

For PS data services, current TEMS products report KPI events, each of which
contains either

an instance of the KPI itself (such as a session duration or throughput figure), or

an instance of an occurrence whose probability is measured by the KPI (such as a


session failure).

These in turn underlie the aggregation of KPI statistics in TEMS Discovery or some
other post-processing tool. The probability KPIs obviously need to be based on a fair
number of sessions for a good estimate of the probability in question to be obtained.
The data-collecting TEMS products do not in themselves output values of these latter
KPIs; rather, they need to be calculated during post-processing.
TEMS Symphony users should note that earlier versions of TEMS Symphony (version
6.x and older) did not report data in this fashion, but only trigger points on which to
base the KPI computation: for example, session start and end times.
Regarding CS voice, see section 3.6.

2.2

ETSI Compliance

In TEMS products, timestamps for KPI computation are captured as close to the source
as possible: that is, from the packet capture driver within the operating system kernel.
Calculations adhere to the specification ETSI TS 102 250-2 V2.2.1 (2011-04) unless
otherwise stated. This specification is hereafter referred to as ETSI 102 250-2.

2.3

Complete vs. Timed Measurements

The standardized KPIs have been defined with complete transactions in mind.
Transactions that are aborted after a fixed duration specified by the TEMS product user
(timed measurements) have not been considered. In case of timed measurements, a
best-effort solution is provided. Where triggers do not fully match ETSI 102 250-2, they
have been aligned as closely as possible to their ETSI counterparts in a manner
NT13-25359, ver 2.0, 9/25/2013

compliant with ETSI TS 102 678 V1.2.1, except that the following item stated in
section 4.1 of that specification has not been implemented:
The connection should be checked to be still alive at the end of the transfer period td. The
reception of any data packet sent by the server on the data connection after the end of the
transfer period is a valid indicator that the data connection is still alive. An appropriate
timeout of e.g. 3 to 5 multiples of typical RTT in the measured network might be used while
waiting for the desired packet.

Session Time KPIs, of course, make sense only for sessions that have been allowed
to run to completion.
For details, see the various subsections of chapter 3.

2.4

IP Capture Options

The default IP data capture that is done by TEMS products during PS data service
testing is designed to obtain sufficient input for KPIs. No special activity needs to be
used in Service Control scripts for this purpose.
It is however also possible to conduct a more thoroughgoing IP sniffing by means of
the Start IP Sniffing script activity. This activity has a Filter parameter whose
possible settings are given here for convenience.
Filter: Type of filtering to apply to IP packets.
o

Optimized Performance: Capture of IP packets is reduced to the minimum needed to


compute KPIs. (Packets are filtered with respect to IP address, protocol [TCP/UDP], and
port.) Not supported for on-device measurement.

Headers Only: The first 54 bytes of every IP packet are captured.

None: No filtering of IP packets. Note: This means that all packets belonging to any
service used by the PC will be captured; packets are filtered with respect to IP address
only. Be aware that full packet capturing can cause crashes on high speed networks
where sustained high throughput is achieved.

KPIs by Service

This chapter deals with TEMS product KPIs for each service.

3.1

KPIs for FTP Download


ETSI-to-TEMS Mapping Table
ETSI KPI FTP
{Download|Upload}

Sect.

TEMS KPI Event

Service Non-Accessibility

6.1.1

FTP Download Service Not Accessible

Setup Time

6.1.2

FTP Download Setup Time

IP-Service Access Failure Ratio

6.1.3

FTP Download IP Service Access Failure

IP-Service Setup Time

6.1.4

FTP Download IP Service Setup Time

Session Time

6.1.6

FTP Download Data Transfer Time

NT13-25359, ver 2.0, 9/25/2013

ETSI-to-TEMS Mapping Table


ETSI KPI FTP
{Download|Upload}

Sect.

TEMS KPI Event

Mean Data Rate

6.1.7

FTP Download Mean Data Rate

Data Transfer Cut-off Ratio

6.1.8

FTP Download Data Transfer Cutoff

Note: Method A and Method B values are differentiated wherever applicable.


See ETSI 102 250-2 section 4.2.

3.1.1

Timed Measurements

For timed measurements, the end trigger is defined as Last incoming packet
containing content. This trigger is valid for both completed and aborted transfers.
Since this definition conforms to the ETSI standard, timed FTP downloads will have
ETSI compliant KPIs.

3.1.2

SFTP (Secure Shell FTP)

If SFTP is used for the file transfer rather than FTP proper, no KPIs are obtained. ETSI
102 250-2 does not define any KPIs for SFTP, and some metrics would not be possible
to obtain owing to the SSH encryption.

3.2

KPIs for FTP Upload


ETSI-to-TEMS Mapping Table
ETSI KPI FTP
{Download|Upload}

Sect.

TEMS KPI Event

Service Non-Accessibility

6.1.1

FTP Upload Service Not Accessible

Setup Time

6.1.2

FTP Upload Setup Time

IP-Service Access Failure Ratio

6.1.3

FTP Upload IP Service Access Failure

IP-Service Setup Time

6.1.4

FTP Upload IP Service Setup Time

Session Time

6.1.6

FTP Upload Data Transfer Time

Mean Data Rate

6.1.7

FTP Upload Mean Data Rate

Data Transfer Cut-off Ratio

6.1.8

FTP Upload Data Transfer Cutoff

Note: Method A and Method B values are differentiated wherever applicable.

3.2.1

Timed Measurements

For timed FTP uploads, the end trigger is defined as Reception of ACK for last packet
containing content. This trigger is not ETSI compliant: ETSI 102 250-2
(section 6.1.6.3) defines the trigger as Stop: Reception of the [FIN, ACK] for the last
data packet containing content. That is, the FIN indicator has been removed from the

NT13-25359, ver 2.0, 9/25/2013

condition. This is because the FIN indicator is sent during connection teardown, a
procedure which does not necessarily take place during a timed measurement.

3.2.2

SFTP (Secure Shell FTP)

If SFTP is used, no KPIs are obtained; compare section 3.1.2.

3.3

KPIs for HTTP Get/Download

This section deals primarily with HTTP downloads conducted with the TEMS browser.
The same KPI values apply also for Internet Explorer browser usage.
ETSI-to-TEMS Mapping Table
ETSI KPI HTTP

Sect.

Service Non-Accessibility

6.8.1

HTTP Service Not Accessible

Setup Time

6.8.2

HTTP Setup Time

IP-Service Access Failure Ratio

6.8.3

HTTP IP Service Access Failure

IP-Service Setup Time

6.8.4

HTTP IP Service Setup Time

Session Time

6.8.6

HTTP Data Transfer Time

Mean Data Rate

6.8.7

HTTP Mean Data Rate

Data Transfer Cut-off Ratio

6.8.8

HTTP Data Transfer Cutoff

TEMS KPI Event

Note: Method A and Method B values are differentiated wherever applicable.

3.3.1

Timed Measurements

There is no difference between timed and complete measurements; the same triggers
can be used in both scenarios.

3.4

KPIs for HTTP Post/Upload

ETSI does not define KPIs for HTTP Post/Upload. The TEMS product KPIs have been
designed to closely resemble the ETSI KPIs for other services, such as HTTP Get and
FTP (compare sections 3.13.3).
ETSI-to-TEMS Mapping Table
ETSI KPI HTTP (Get)

Sect.

TEMS KPI Event

Service Non-Accessibility

6.8.1

HTTP Service Not Accessible

Setup Time

6.8.2

HTTP Setup Time

IP-Service Access Failure Ratio

6.8.3

HTTP Post IP Service Access Failure

IP-Service Setup Time

6.8.4

HTTP Post IP Service Setup Time

Session Time

6.8.6

HTTP Post Data Transfer Time

Mean Data Rate

6.8.7

HTTP Post Mean Data Rate

NT13-25359, ver 2.0, 9/25/2013

ETSI-to-TEMS Mapping Table


ETSI KPI HTTP (Get)

Sect.

Data Transfer Cut-off Ratio

6.8.8

TEMS KPI Event


HTTP Post Data Transfer Cutoff

Trigger points for the HTTP Post session are as follows:

Start: First packet containing content.

End: Last incoming ACK packet for content.

No Method A vs. Method B distinction exists for this service.

3.4.1

Timed Measurements

Since ETSI does not define any KPIs for HTTP Post, the session end trigger was
defined in such a way as to work for both timed and complete measurements.

3.5

KPIs for Ping


ETSI-to-TEMS Mapping Table
ETSI KPI Ping

Sect.

Round trip time

TEMS KPI Event

6.3.1

Ping Roundtrip Time

Trigger points for Ping round trip:

Start: ICMP Echo Request sent.

End: ICMP Echo Reply received by the sender.

The round-trip time is that calculated by the Windows API function IcmpSendEcho2Ex.
This is the lowest-latency way available in Windows to send Ping requests and capture
replies. Reference:
msdn.microsoft.com/en-us/library/windows/desktop/aa366050(v=vs.85).aspx
The difference between the round-trip time measured by Windows and that derived
from the network trace is in the range 0.5 ms. Since the round-trip time is reported in
milliseconds in the KPI, the difference after rounding is not noticeable.
TEMS products also define a timeout error message Ping Timeout signifying a failed
Ping request.

3.5.1

Timed Measurements

Not applicable.

3.6

KPIs for CS Voice


ETSI-to-TEMS Mapping Table
ETSI KPI: Telephony

Service Non-Accessibility

NT13-25359, ver 2.0, 9/25/2013

Sect.
6.6.1

TEMS Data (see also below)


Blocked Call event

ETSI-to-TEMS Mapping Table


ETSI KPI: Telephony

Sect.

TEMS Data (see also below)

Setup Time

6.6.2

Call Setup event with call setup time

Speech Quality on Sample Basis

6.6.4

PESQ/POLQA information elements

Cut-off Call Ratio

6.6.5

Dropped Call event

For CS voice, no special KPI events are generated. However, call setup time is carried
as extra information by the Call Setup event, and speech quality scores are reported in
information elements belonging to the Media Quality category. Supported speech
quality measures are PESQ, POLQA NB and POLQA SWB.
TEMS products also have an additional, non-ETSI speech quality measure called
Speech Quality Index (SQI).
CS voice KPIs are computed for mobile-originated (MO) as well as mobile-terminated
(MT) calls.

Notes on Other Services: Email, Iperf,


Streaming, CS Fallback

For the following services, TEMS products do not have any KPIs defined. However,
they do exhibit certain information elements and/or events with similar content.
Average throughputs and all percentages apply to the period following the latest
network connect (triggering of Network Connect event) and are reset at network
disconnect (when the Network Disconnect activity has completed).

4.1

Email
IE Name

Range/Unit

Description

Email Receive
Average
Throughput

0 ... 350000
kbit/s

Average throughput for receiving of email.

Email Receive
Transfer Time

0 ... 172800
s
(= 48 h)

Elapsed time for current email receive


session. (Not an average.)

Email Send
Average
Throughput

0 ... 350000
kbit/s

Average throughput for sending of email.

Email Send
Transfer Time

0 ... 172800
s

Elapsed time for current email send


session. (Not an average.)

NT13-25359, ver 2.0, 9/25/2013

4.2

Iperf

Iperf testing is conducted with the Network Bandwidth activity in Service Control
scripts. The testing can be done over either TCP or UDP.

4.2.1

TCP
IE Name

Range/Unit

Description

TCP Download
Average
Throughput
(kbit/s)

0 ... 350000
kbit/s

Average throughput for TCP download.

TCP Download
Transfer Time

0 ... 172800
s

Elapsed time for current TCP download


session. (Not an average.)

TCP Packet Loss

0 ... 100
%

Retransmissions (in percent) on the


downlink over the TCP protocol during the
last second. (Not an average.)
Note: For this percentage to be correct, IP
sniffing must not be set to Optimized
Performance. See section 2.4.

TCP Upload
Average
Throughput
(kbit/s)

0 ... 350000
kbit/s

Average throughput for TCP upload.

TCP Upload
Transfer Time

0 ... 172800
s

Elapsed time for current TCP upload


session. (Not an average.)

4.2.2

UDP
IE Name

Range/Unit

Description

UDP Download
Average
Throughput
(kbit/s)

0 ... 350000
kbit/s

Average throughput for UDP download.

UDP Download
Jitter

Text

UDP jitter: the mean deviation (in ms) of


the difference in packet spacing at the
receiver compared to the sender, for a
pair of packets.

UDP Download
Packet Loss

0 ... 100
%

NT13-25359, ver 2.0, 9/25/2013

Retransmissions (in percent) on the


downlink over the UDP protocol. This
measurement is obtained once every
second. (Not an average.)

IE Name

Range/Unit

Description

UDP Download
Transfer Time

0 ... 172800
s

Elapsed time for current UDP download


session. (Not an average.)

UDP Upload
Average
Throughput
(kbit/s)

0 ... 350000
kbit/s

Average throughput for UDP upload.

UDP Upload
Transfer Time

0 ... 172800
s

Elapsed time for current UDP upload


session. (Not an average.)

4.3

Streaming over HTTP

4.3.1

Streaming Player Download

These information elements relate to the task of downloading the streaming player.
Please note that not all of the Streaming Player elements are obtained if the
streaming server is an HTTPS server, since the relevant packets are then encrypted.
IE Name

Range/Unit

Description

Streaming Player
Service IP Access
Failure Ratio

0 ... 100
%

Percentage of attempts to download the


streaming player that failed before the first
packet of the download was received (IP
access failure).

Streaming Player
Service IP Access
Time

0 ... 172800
s

Streaming Player
Download Data
Transfer Failure
Ratio

0 ... 100
%

Streaming Player
Download Data
Transfer Time

0 ... 172800
s

Streaming Player
Session Failure
Ratio

0 ... 100
%

Percentage of attempts to download the


streaming player that failed at some stage
(IP access or data transfer).

Streaming Player
Session Time

0 ... 172800
s

Total session time for streaming player


download (from user request to receipt of
last packet).

NT13-25359, ver 2.0, 9/25/2013

Time taken for IP access when


downloading the streaming player (from
user request to receipt of first packet).
Percentage of attempts to download the
streaming player that failed at the data
transfer stage (after receipt of first packet
and before receipt of last packet).
Time taken to download the streaming
player (from receipt of first packet to
receipt of last packet).

4.3.2

Start of Streaming Video Replay


IE Name

Range/Unit

Description

Streaming Service
Access Time

0 ... 172800
s

Streaming Service
Non Accessibility

0 ... 100
%

Percentage of attempts to request


streaming that failed (never receive
response from streaming host).

Streaming
Reproduction
Start Delay

0 ... 100
%

The duration of the delay after streaming


request was responded and before the
video replay was started.

Streaming
Reproduction
Start Failure

Text

The duration of a service access from


requesting the stream at the portal until
the reception of the first stream data
packet.

Indicate that there is error occurs after


streaming request was responded and
before the video replay was started.

Streaming Video
Play Start Failure
Ratio

0 ... 100
%

Percentage of attempts to start streaming


video replay that failed (playing never
started).

Streaming Video
Play Start Time

0 ... 172800
s

Time from user video replay request until


playing started.

Streaming
Reproduction Cut
off Ratio

0 ... 100
%

4.3.3

Percentage of failures throughout the


video playing session.

Streaming Video Session

Overall
IE Name

Range/Unit

Streaming Video
Session Failure
Ratio

0 ... 100
%

Streaming Video
Session Time

0 ... 172800
s

Description
Percentage of streaming video replay
session that failed at some point before
the last video packet was received.
Total time of streaming video replay
session from user request to receipt of last
video packet.

VQmon Metrics: Video and Audio Perceptual Quality


IE Name

NT13-25359, ver 2.0, 9/25/2013

Range/Unit

Description

IE Name

Range/Unit

Description

Streaming
Absolute MOS-V

0 ... 5
MOS

Average absolute video MOS for the


stream.

Streaming
MOS-A

0 ... 5
MOS

Average audio MOS for the stream.

Streaming
MOS-AV

0 ... 5
MOS

Average audiovideo MOS for the stream.

Streaming
Relative MOS-V

0 ... 5
MOS

Average relative video MOS for the


stream. The difference between relative
and absolute video MOS is that the
relative metric does not consider frame
resolution, thus constituting a score which
is relative to the ideal for the current video
format.

These average scores are also reported in a Streaming Quality MOS event at the end
of a streaming session.
Bandwidth
IE Name

Range/Unit

Description

Streaming
Average Audio
Received
Bandwidth

0 ... 350000
kbit/s

Average audio bandwidth excluding


transport packet overhead, FEC, and
retransmissions.

Streaming
Average Video
Received
Bandwidth

0 ... 350000
kbit/s

Average bandwidth for video transport


packets, excluding IP overhead, FEC, and
retransmissions.

Packet Loss, Packet Errors


IE Name

Range/Unit

Description

Streaming
Average Audio
Effective Packet
Loss Rate

0 ... 100
%

Percentage of audio packets lost or


discarded (minus those corrected).

Streaming
Average Video
Effective Packet
Loss Rate

0 ... 100
%

Percentage of video packets lost or


discarded (minus those corrected).

NT13-25359, ver 2.0, 9/25/2013

10

IE Name

Range/Unit

Description

Streaming Audio
Packet Corrected
Rate

0 ... 100
%

Percentage of audio packets received with


errors, but corrected using error correction
algorithms.

Streaming Audio
Packet Discarded
Rate

0 ... 100
%

Percentage of audio packets discarded by


the receiving jitter buffer.

Streaming Audio
Packet Out Of
Sequence Rate

0 ... 100
%

Percentage of audio packets identified as


out-of-sequence, possibly due to high jitter
levels or the use of load-sharing devices.

Streaming Video
Packet Corrected
Rate

0 ... 100
%

Percentage of video packets received with


errors, but corrected using error correction
algorithms.

Streaming Video
Packet Discarded
Rate

0 ... 100
%

Percentage of video packets discarded by


the receiving jitter buffer.

Streaming Video
Packet Out Of
Sequence Rate

0 ... 100
%

Percentage of video packets identified as


out-of-sequence, possibly due to high jitter
levels or the use of load-sharing devices.

Buffering and Rebuffering


IE Name

Range/Unit

Streaming Video
Interruption Count

Text

Description
Number of times the video stream replay
was interrupted for rebuffering.

Streaming Video
Interruption
Duration

0 ... 172800
s

Total duration of video stream replay


interruptions for reasons of rebuffering.

Streaming Playout
Buffer Delta Rate

0 ... 350000
kbit/s

Equal to Streaming Playout Buffer Empty


Rate minus Streaming Playout Buffer Fill
Rate.

Streaming Playout
Buffer Empty
Proportion

0 ... 100
%

Streaming Playout
Buffer Empty Rate

0 ... 350000
kbit/s

Rate at which data is emptied from the


streaming players playout buffer.

Streaming Playout
Buffer Fill Rate

0 ... 350000
kbit/s

Rate at which the streaming players


playout buffer is filled with new data.

NT13-25359, ver 2.0, 9/25/2013

Percentage of the total session time


during which the playout buffer was
empty.

11

IE Name

Range/Unit

Streaming Playout
Buffer Rebuffering
Proportion

0 ... 100
%

NT13-25359, ver 2.0, 9/25/2013

Description
Percentage of the total session time that
was spent rebuffering.

12

4.4

CS Fallback

The following events report on various aspects on CS fallback procedures:


Event Name
CSFB Blocked Call

Description
A CS fallback call was blocked. This can happen in
several ways:

The Extended Service Request timed out without


any response from network, or the network
responded with Service Reject. In this case the CS
fallback procedure never reaches the RAT change
stage.

RAT change to UTRAN/GERAN failed, as indicated


by the event EUTRAN RRC Connection Release
Redirected Failure.

After successful RAT change to UTRAN/GERAN,


the CS call setup failed (CS Blocked Call event
generated).

Extra information:

CSFB Call Setup

Call direction (MO/MT)

Block type

A CS fallback call was set up. Triggered by the Layer 3


message Alerting.
Extra information:

EUTRAN Reselection
Time After CSFB Call

Call direction (MO/MT)

Target technology (WCDMA/GSM)

Call setup time (measured from CSFB Call


Attempt, i.e. the first call attempt)

User setup time (measured from CSFB Call


Initiation, thus more accurately reflecting the userperceived setup time. Obtained only for MO calls in
scripts. If no CSFB Call Initiation event was
generated, the user setup time cannot be
computed.)

This event carries performance information on the


switch back to LTE after hangup of a CS fallback call.
Extra information:

NT13-25359, ver 2.0, 9/25/2013

Idle to LTE time: Time in seconds from entering


idle mode (in UMTS) to reception of System
Information Block on LTE.

SIB 19 to LTE time: Time in seconds from


reception of System Information Block Type 19
(UMTS) to reception of System Information Block
on LTE.

13

Event Name
PS Data Interruption
Time Due To CSFB

Description
This event reports the IP interruption time during RAT
change to UTRAN/GERAN due to initiation of a CS
fallback call.
Extra information: Interruption time in ms. Measured
from last received IP packet in EUTRAN to first
received IP packet in UTRAN/GERAN.

NT13-25359, ver 2.0, 9/25/2013

14

Vous aimerez peut-être aussi