LTE CONFORMANCE TESTING: EXPERIENCES IN A LONG

TERM
PROJECT
TTCN-3
User Conference 2011 Bled, Slovenia
Presented by MCC TF160: Wolfgang Seka
Authors: Wolfgang Seka, Hellen Griffiths, Shicheng Hu
wolfgang.seka@etsi.org , hellen.griffiths@etsi.org , shicheng.hu@etsi.org

ETSI 2011. All rights reserved

LTE Conformance Testing:

Experiences in a long term
project
The Project
Technical Issues
Common Aspects of Long Term Projects
Conclusions

ETSI 2011. All rights reserved

THE PROJECT
3

ETSI 2011. All rights reserved

MCC TF160 - General

Task Force Mobile Competence Centre: Project Group at ETSI

Pool of TTCN expertise used by 3GPP

3GPP: 3rd Generation Partnership Project (http://www.3gpp.org)

Telecommunication Standardisation Bodies
TSG RAN: Radio Access Network
WG RAN5: Mobile terminal conformance testing

Conformance Tests

Specification (Prose): RAN5

Implementation (TTCN): MCC TF160
Verification and Validation: Test Industry

MCC TF160: Signalling Conformance Tests for 3GPP (RAN5:

Testing)

Task: Develop Conformance Test Suites for UE world-wide certification

Conformance Tests for UMTS Signalling (TTCN-2)
since 2000
since 2006
2007..2008
2008..now

ETSI 2011. All rights reserved

Conformance Tests for IMS (TTCN-3)

Pre-evaluation of TTCN-3 for LTE Signalling
3GPP LTE/SAE UE Conformance Test

MCC TF160 LTE/SAE Project

Size:
17 experts all over the world
Duration:
since 2008 and at least until 2015 (LTE-advanced will
follow)
Deliveries
2011: 8 deliveries planned
Test cases: 264 approved
391 fully implemented
468 in total
80..100 planned for IMS
Code size:
nearly 200 Modules
nearly 190 000 Lines of code (TTCN-3)
Type Definitions
25 TTCN-3 modules, 3 ASN.1 modules, 6 XML modules
TTCN-3: > 17 500 lines of code
ASN.1: > 38 000 lines of code
XML: < 1 000 lines of code

Tools:
6 different compilers (all available at ETSI)
quality checks (naming conventions, template restrictions etc.)
code generation (top-level test case definitions, parameters, etc.)
code analysis (structure, approved objects)
TTCN-3 code is officially published and widely used
5

ETSI 2011. All rights reserved

Component Structure - 1
MTC
Start of PTCs, monitoring of done and killed
RAT independent interfaces (e.g. AT-/MMI-commands to control the UE)
In general no pass/fail verdicts
PTCs for each RAT (radio access technology)
LTE, UMTS, GSM/GPRS, CDMA2000
May be connected to any other RAT PTC
Only RAT specific interfaces
Assignment of pass/fail verdicts
PTCs for other protocols
IP data (e.g. DHCP, ICMPv6), IMS and protocol layer underneath LTE PTC (NAS

Emulator)
To keep system interface simple and deal with parallel behaviour
In general no test characteristic ( no pass/fail verdicts)

Ports and interfaces

Connections: only one-to-one

( no send to, receive from; no use of address data type)

No duplicated interfaces: e.g. only one interface for AT-/MMI-commands hosted by
the MTC
Message based communication only

ETSI 2011. All rights reserved

7 7

UTRAN
PTC

Coordination

C2K_Ut

CDMA2000
PTC

Upper Tester
(AT , MMI)

G_Ut

SYSIND

SYS

NASCTRL

SRB

GERAN
PTC

CTRL

TC_SRB

MTC

U_Ut

DRB

E_Ut

C2K

GERAN
PTC

UT

(LTE)

GERAN

EUTRA PTC

CDMA2000
PTC

UTRAN

Component Structure - 2

NAS Emulator

E_DRB

E_SYSIND

E_SYS

SYS_SRB

Ut

E_SRB

Ut

System Interface
ETSI 2011. All rights reserved

UTRAN
PTC

Configuration
Signalling
User Data

Test System
Host PC
Test Control,
Logging

System Simulator
HW

Test Executable

Codec

System/
Platform
Adaptor

system

complex configuration
delay of messages
no matter what test
purpose is

SUT
RF

UE

specific

AT/MMI

e.g. requirements regarding real-time behaviour for System Simulator and TTCN-3 code
8

ETSI 2011. All rights reserved

TECHNICAL ISSUES
9

ETSI 2011. All rights reserved

Real Time Issues - 1

Requirements

Time resolution at the air interface: 1ms

Due to significant delay between air interface and test case executable:
Timing needs to be monitored/controlled at the air interface (not at the
TRI)
All test equipment shall have similar behaviour at any time
Test results shall not depend on
how fast or slow a UE or system simulator is
specific combinations of UEs and system simulators

System Restrictions

Delay by codec and matching: up to some milliseconds;

typically < 5ms
Host system: typically Windows or Linux
no real-time OS
Architecture: Host-PC + System Simulator

Estimation

delay between test executable and air interface:

1ms < delay < 80ms

10

ETSI 2011. All rights reserved

Real Time Issues - 2

Impact on System Interface
LTE timing information
is part of all LTE primitives
is provided by the System Simulator for all received messages
is used to schedule sending of messages in the system simulator
can be retrieved from the system simulator

Impact on writing TTCN-3

Scheduled and non-scheduled sending shall not be mixed up
When there is no reaction (e.g. from the UE) it is hard to know in the TTCN code when
the message has been sent out
Scheduling vs. TTCN timers
TTCN timers are not as accurate as scheduling
Rules are required when to use TTCN timers or scheduled sending

Restrictions and Issues

No common system time: uncorrelated timing between different RATs
Legacy primitive definitions for UMTS and GSM/GPRS: less support of timing

11

information
Short duration only: For LTE, wrap-around after 10s
No bell mechanism (yet):
to schedule a wake-up call in the system simulator to send back a trigger after a
given time

ETSI 2011. All rights reserved

Race Conditions
Race conditions occur when

messages appear at the same time at different ports

messages appear in any order at one port

Impact on TTCN implementation and Design

considerations

TTCN implementation

interleave (in general not more than two messages)

check operation
evaluation of timing information
Clever port definitions (few ports rather than many)
Component structure
(e.g. signalling and data at different PTCs)

12

ETSI 2011. All rights reserved

COMMON ASPECTS OF LONG TERM

PROJECTS
13

ETSI 2011. All rights reserved

Extendibility
Extensions of TTCN-3 code

Modification of common objects shall not cause changes in the whole

test suite
reduced maintenance effort
Impact on writing TTCN-3
foresighted TTCN implementation
introduction of additional optional parameters
Example:
template MyType
c_MyType :=
{
field1 := 1,
field2 := 2
};

template MyType c_MyType(integer

p_Int := 1) :=
{
field1 := p_Int,
field2 := 2
};

Note: Optional Parameters vs. Modified Templates

Optional parameters need to be added to modified templates

14

too
modification of templates and addition of optional parameters
may serve the same purpose
possible conflicts, potential contradiction of both approaches
ETSI 2011. All rights reserved

Backward Compatibility - 1
Backward Compatibility at the System Interface

Modifications are caused by

Release upgrade for signalling messages may have impact on

configuration of the system simulator
New requirements (due to new test cases)
Corrections

Backward Compatibility Aspects

Old TTCN-3 code on new system simulator implementation

New TTCN-3 code on old system simulator implementation

Impact

system adaptor and codec

interface definition

top-level records or unions (even when not needed in the first place)
definitions of place holders

15

ETSI 2011. All rights reserved

Backward Compatibility - 2
Examples: Backward Compatibility at System
Interface
type union MyUnion_Type :=
{
R8_Type R8

type union MyUnion_Type :=

{
R8_Type R8,
R9_Type R9
};
};
New branch: branches are mutual exclusive; requires new templates

type record
MyRecord_Type := {
R8_Type R8Only

type record
MyRecord_Type := {
R8_Type R8Only,
R9Extension_Type R9Ext
};
optional
Extension: templates may be enhanced
};by optional parameter
16

ETSI 2011. All rights reserved

Tools
Large, long-term project

code reviews and manual checks are not sufficient anymore

Requirements

Quality checks
Approved objects: objects used by approved test cases shall not
be changed without change request
Analysis of module dependencies
Recursive selection

Enhanced tool support

Quality checks by T3Q (on behalf of ETSI)

Common tool to get dependencies based on T3D (on behalf of

ETSI)
Project-specific front-ends (approved objects, recursive selection)
Further project specific tools:
replace tabs, find non-ASCII characters in comments
17

ETSI 2011. All rights reserved

CONCLUSIONS
18

ETSI 2011. All rights reserved

Conclusions (TTCN-3 language)

Important TTCN-3 language features
optional parameters

backward compatibility, maintenance

template restrictions

in combination with naming conventions and appropriate tools

no problems with receive templates used in send direction anymore

encvalue/decvalue

maintenance (no external functions needed)

check operation, interleave

race conditions

testcase.stop

well-defined termination of a test case in case of unrecoverable error situations

visibility: "private", "friend

not used yet (may improve quality regarding module dependencies)

19

ETSI 2011. All rights reserved

Common Conclusions
Feedback of long-terms projects to the core language is vital
Tool Compatibility is needed
Shortened release cycle of TTCN-3 standards
TTCN-3 development is SW Development
SW Engineering
System Engineering
Long-term characteristic of a project needs to be considered from the
beginning
Conceptual errors will be expensive
Tool support is vital
Common and project-specific tools

20

ETSI 2011. All rights reserved

ADDITIONAL SLIDES
21

ETSI 2011. All rights reserved

Release Upgrades - 1
Release Upgrade for Protocol and Signalling
(Baseline Moving)

Typically every year

In general changes are backward compatible
Critical and non-critical extensions (TS 36.331
Annex A)
Critical extensions

redefinition of message content or parts of it

(branching, e.g. r8 and r9 branch)

sender needs to know which release the receiver supports

Non-critical extensions

Provision of additional information

e.g. new fields at the end of a record, new enumerated value
when not supported receiver shall skip the information
Messages in general contain hooks for critical and noncritical extensions

22

ETSI 2011. All rights reserved

Release Upgrades - 2
Impact on writing TTCN-3

Non-critical extensions

In general no code duplication is necessary

Enhancement of existing templates by new optional

parameters
(initialised with * for receiving and omit for sending)

Critical extensions

New templates are needed

Challenge: minimised code duplication
Example:

template of a plain RRC PDU for UMTS

up to several hundred lines of code
clever template structure reduces duplication of code
parameterised templates
modified templates

23

ETSI 2011. All rights reserved