79
OSI Conformance Testing ©
D. R A Y N E R
Leader of the Protocol Standards Group, Dioision of Information
Technology and Computin~ National Physical Laboratory, Ted-
dington, Middelsex TWI I OLW, UK
Now that many international standards for Open Systems
Interconnection (OSI) services and protocols have reached at
least the Draft International Standard stage, there is much
international activity in standardizing OSI conformance test
suites. This new activity is based on the emerging OSI confor-
mance testing methodology and framework standards being
progressed by ISO and CCITT. This paper presents the major
aspects of this methodologyand framework.
Keywords: Open Systems Interconnection, OSI, Confor-
mance Testing, Protocols, Test Methods, Test Suite Design.
Dr. D. Rayner has worked at NPL
since 1975 and has been leader of the
Protocol Standards Group at NPL
since 1980. The group has con-
centrated on protocol conformance
testing since 1979. From 1976 to 1982,
he was closely involved with the Net-
work Independent File Transfer Pro-
tocol (a fore-runner of the OSI FTAM
protocol), having been both secretary
and later chairman of the FTP Imple-
mentor's Group. He became both BSI
and ISO rapporteur for OSI confor-
mance testing in 1983 and as such has led the standardization
in this field.
© UK Crown.
North-Holland
Computer Networks and ISDN Systems 14 (1987) 79-98
O. Introduction
It is now widely accepted that OSI Confor-
mance Testing is crucial to the achievement of the
objective of OSI [1]. Standard test suites need to
be developed for each OSI protocol standard, for
use by suppliers, users, carriers or third-party test
centres. This should lead to the comparability of
results produced by different testers, and thereby
to the mutual recognition of results which will
minimize the need for repeated conformance test-
ing of the same system.
The standardization of conformance test suites
needs to be based upon a standard testing meth-
odology and approach to test suite specification.
The International Organization for Standardiza-
tion (ISO) and the International Consultative
Committee for Telephones and Telegraphs
(CCITT) are now working together to produce
just such a common methodology and framework
for conformance testing [2-6] applicable to OSI
standards and similar C C I T T X-series and T-series
recommendations (hereinafter just referred to as
'OSI standards'). ISO has been working on this
since 1984, basing its work on research and devel-
opment which began in 1979. As a consequence,
many of the ideas are now technically mature.
Indeed, the work is already being used as the basis
for standardization of test suites for X.25 termi-
nals, the OSI connection-oriented Transport Pro-
tocol (ISO 8073) and Message Handling Systems
(CCITT X.400 series). In April 1987, work also
began in ISO on the standardization of confor-
mance test suites for Session, Presentation, Associ-
ation Control Service Elements, and File Transfer
Access and Management protocols.
The OSI Conformance Testing Methodology
and Framework is being progressed as a 5 part
standard. Part 1 concerns general concepts and
Part 2 concerns abstract test suite specification
(i.e. the specification of tests in a form which is
independent of particular real testers and suitable
for standardization). Both of these have now
reached the Draft Proposal stage in ISO. They are
the parts required as the basis for conformance
test suite standardization. The other 3 parts are
less developed; they concern executable test de-
80 D. Rayner / OSI ConformanceTesting
rivation, requirements on protocol implementors,
and test laboratory operations.
This paper presents the major aspects of the
methodology and framework, based mostly on
matrial from Parts 1, 2 and 3 of the current drafts
of the multi-part standard.
1. The Meaning of Conformance in O S I
Conformance in the context of OSI is con-
cerned only with the conformance of implementa-
tions and real systems to those OSI standards
which specify applicable requirements. Those
standards include protocol standards (both
single-layer and multi-layer) and transfer syntax
standards.
1.1. Conformance Requirements
There are three ways of looking at conformance
requirements.
(1) The conformance requirements in a standard
can be:
(a) mandatory requirements: these are to be
observed in all cases;
(b) conditional requirements: these are to be
observed if the conditions set out in the
standard apply;
(c) options: these can be selected to suit the
implementation, provided that any re-
quirements applicable to the option are
observed.
(2) The statements of conformance requirements
in a standard can be stated
(a) positively: they state what shall be done;
(b) negatively (prohibitions): they state what
shall not be done.
(3) Conformance requirements fall into two
groups:
(a) static conformance requirements;
(b) dynamic conformance requirements.
These are defined below.
1.2. Static Conformance Requirements
Static conformance requirements are those that
define the allowed minimum capabilities of an
implementation, in order to facilitate interwork-
ing. These requirements may be at a broad level,
such as the grouping of functional units and op-
tions into protocol classes, or at a detailed level,
such as the range of values that must be supported
for specific parameters or timers.
They can come in two varieties:
(a) those which concern the capabilities to be
included in the implementation of the particu-
lar protocol;
(b) those which concern multi-layer dependencies
- placing constraints on the capabilities of the
underlying layers of the system.
If the static conformance requirements are in-
completely specified then all capabilities not ex-
plicitly stated as static conformance requirements
are to be regarded as optional.
1.3. Dynamic Conformance Requirements
Throughout most of the OSI protocol standard,
the requirements and options define the set of
allowable behaviours of an implementation or real
system in instances of communication. This set
defines the maximum capability that a conforming
implementation or real system can have within the
terms of the OSI protocol standard. These are the
dynamic conformance requirements.
Thus, dynamic conformance requirements are
all those requirements (and options) which de-
termine what observable behauiour is permitted by
the relevant OSI protocol standard(s) in instances
of communication.
1.4. Protocol Implementation Conformance State-
ment
The protocol implementation conformance
statement (PICS) is a statement made by the
supplier of an OSI implementation or system,
stating the capabilities and options which have
been implemented, and any features which have
been omitted. It is needed so that the implementa-
tion can be tested for conformance against rele-
vant requirements, and against those requirements
only.
1.5. A Conforming System
A conforming system or implementation is,
therefore, one which is shown to satisfy both static
a n d dynamic requirements, consistent with the
capabilities and options stated in the PICS.
 D. Rayner / OS1 ConformanceTesting
2. Types of Testing
In principle, the objective of conformance test-
ing is to establish whether the implementation
being tested conforms to the specification in the
relevant standard. Practical limitations make it
impossible to be exhaustive, and economic consid-
erations may restrict testing still further.
Therefore, four types of testing are dis-
tinguished, according to the extent to which they
provide an indication of conformance:
• basic interconnection tests, which provide prima
facie evidence that an implementation under
test (IUT) conforms;
• capability tests, which check that the observable
capabilities of the I U T are in accordance with
the static conformance requirements and the
capabilities claimed in the PICS;
• behaviour tests, which endeavour to provide as
comprehensive testing as possible over the full
range of dynamic conformance requirements
specified by the standard, within the capabil-
ities of the IUT;
• conformance resolution tests, which probe in de-
pth the conformance of an I U T to particular
requirements, to provide a definite y e s / n o
answer and diagnostic information in relation
to specific conformance issues; such tests are
not standardized.
2.1. Basic Interconnection Tests
These provide limited testing of an I U T in
relation to the main features in a standard, to
establish that there is sufficient conformance for
interconnection to be possible, without trying to
perform thorough testing.
Basic interconnection tests are appropriate:
• for detecting severe cases of non-conformance;
• as a preliminary filter before undertaking more
costly tests;
• to give a prima facie indication that an imple-
mentation which has passed full conformance
tests in one environment still conforms in a new
environment (e.g. before testing an (N)-imple-
mentation, to check that a tested ( N - 1 ) -
implementation has not undergone any severe
change when linked to the (N)-implementation);
• for use by users of implementations, to de-
termine whether the implementations appear to
be usable for communication with other confor-
81
ruing implementations, e.g. as a preliminary to
data interchange.
Basic interconnection tests are inappropriate:
• as a basis for claims of conformance by the
supplier of an implementation;
• as a means of arbitration to determine causes
for communications failure.
Basic interconnection tests should be standar-
dized as either a very small test suite or a subset of
the conformance test suite. They can be used on
their own or together with a conformance test
suite.
2.2. Capability Tests
These provide limited testing of each of the
static conformance requirements in a standard, to
ascertain what capabilities of the I U T can be
observed and to check that those observable capa-
bilities are valid with respect to the static confor-
mance requirements and the PICS.
Capability tests are appropriate:
• to check as far as possible the consistency of the
PICS with the IUT;
• as a preliminary filter before undertaking more
in-depth and costly testing;
• to check that the capabilities of the I U T are
consistent with the static conformance require-
ments;
• to enable efficient selection of behaviour tests
to be made for a particular IUT;
• to check that the capabilities of the I U T are
adequate to meet a particular user's needs;
• when taken together with behaviour tests, as a
basis for claims of conformance.
Capability tests are inappropriate:
• on their own, as a basis for claims of confor-
mance by the supplier of an implementation;
• for testing in detail the behaviour associated
with each capability which has been imple-
mented or not implemented;
• for resolution of problems experienced during
live usage or where other tests indicate possible
non-conformance even though the capability
tests have been satisfied.
Capability tests should be standardized within
a conformance test suite. They can either be sep-
arated into their own test group(s) or merged with
the behaviour tests.
82 D. Rayner / 0S1 Conformance Testing
2.3. Behaoiour Tests
These test an implementation as thoroughly as
is practical, over the full range of dynamic confor-
mance requirements specified in a standard. Since
the number of possible combinations of events
and timing of events is infinite, such testing can-
not be exhaustive. There is a further limitation,
namely that these tests are designed to be run
collectively in a single test environment, so that
any faults which are difficult or impossible to
detect in that environment are likely to be missed.
Therefore, it is possible that a non-conforming
implementation passes the conformance test suite;
one aim of the test suite design is to minimize the
number of times that this occurs.
Behaviour tests are appropriate, when taken
together with capability tests, as a basis for the
conformance assessment process.
Behaviour tests are inappropriate for resolution
of problems experienced during live usage or where
other tests indicate possible non-conformance even
though the behaviour tests have been satisfied.
Behaviour tests should be standardized as the
bulk of a conformance test suite.
2.4. Conformance Resolution Tests
These provide diagnostic answers, as near to
definite as possible, to the resolution of whether
an implementation satisfies particular require-
ments. Because of the problems of exhaustiveness
noted above, the definitive answers are gained at
the expense of confining tests to a narrow field.
The test architecture and test method will nor-
mally be chosen specifically for the requirements
to be tested, and need not be ones that are gener-
ally useful for other requirements; they may even
be ones that are regarded as being unacceptable
for generally specified conformance tests, e.g. in-
volving implementation-specific methods using,
say, the diagnostic and debugging facilities of the
specific operating system.
The distinction between behaviour tests and
conformance resolution tests may be illustrated by
the case of an event such as a Reset. The be-
haviour tests may include only a representative
selection of conditions under which a Reset might
occur, and may fail to detect incorrect behaviour
in other circumstances. The conformance resolu-
tion tests would be confined to conditions under
which incorrect behaviour was already suspected
to occur, and would confirm whether or not the
suspicions were correct.
Conformance resolution tests are appropriate:
• for providing a y e s / n o answer in a strictly
confined and previously identified situation (e.g.
during implementation development, to check
whether a particular feature has been correctly
implemented, or during operational use, to in-
vestigate the cause of problems);
• as a means for identifying and offering resolu-
tion for deficiencies in a current conformance
test suite.
Conformance resolution tests are inappropriate:
• as a basis for judging whether or not an imple-
mentation conforms overall;
• as a condition for procurement.
Conformance resolution tests are not standar-
dized.
2.5. Other Types of Testing
In addition to conformance testing, other types
of testing have been proposed including:
• inter-operability testing to determine whether
two IUTs will actually inter-operate and if not
why not;
• performance tests to measure the performance
characteristics of an IUT, such as its through-
put and responsiveness under various condi-
tions;
• robustness tests to determine how well an IUT
recovers from various error conditions.
There may be some overlap between these types
of test and conformance tests and where there is
then such tests could be extracted as a subset of a
full conformance test suite. However, where the
test purpose is not concerned with conformance,
then such tests fall outside the current scope of
standardization.
3. Protocol Implementation Extra Information for
Testing (PIXIT)
In order to test a protocol implementation, the
Test Laboratory will require information relating
to the IUT and its testing environment in addition
to that provided by the appropriate PICS. This
'Protocol Implementation eXtra Information for
Testing' (PIXIT) will be provided by the client
 D. Rayner / OSI Conformance Testing
submitting the implementation for testing.
The PIXIT may contain the following informa-
tion:
(a) information needed by the test operator in
order to be able to run the appropriate test
suite on the specific system (e.g. information
related to the test method to be used to run
the test cases, addressing information):
(b) information already mentioned in the PICS
and which needs to be made precise (e.g. a
timer value which is declared as a parameter
in the PICS should be specified in the PIXIT);
(c) information to help determine which capabil-
ities stated in the PICS as being supported are
testable and which are untestable;
(d) other administrative matters (e.g. the IUT
identifier, reference to the related PICS).
The PIXIT should not conflict with the ap-
propriate PICS.
As the type of information required will vary
considerably according to the needs of individual
real tester as well as test architectures and proto-
col layers, it is envisaged that responsibility for the
design of the PIXIT proforma will primarily re-
side with the Test Laboratory. However, guide-
lines may be provided by the test realizers and
protocol layer conformance testing standardiza-
tion groups.
4. Test Procedure Overview
It is important to understand how conformance
testing should relate to static and dynamic confor-
mance requirements and the PICS. There are many
possible ways of interleaving various phases of
testing, enabling the selection of later tests to be
based on the results of earlier tests and paper
studies. However, Fig. 1 is used by ISO and
CCITT to give an outline of the conformance test
procedure.
There are five main steps in this procedure. The
first step is the analysis of the PICS accompany-
ing the IUT. The PICS will be analysed for its
own consistency, and its consistency with the static
conformance requirements specified in the rele-
vant standard(s).
The second step is test selection, in which the
PICS and PIXIT are used to select the appropriate
test cases from the conformance test suite and
parameterize them using the PIXIT information.
83
Protocol
Standards
~ Dnalysi iorPlC~S ,
'
:1 Static
conformance
' I Requirements
Dynamic
Conformance
Test Selection Conformance
i ~ Test Suite
i
Basic Interconnection Testing i
(optional)
Capability Testing
........... Be h'avi~ ~%~st'i n'g........... /
'I Analysis of Results I
1
Final Conformance Review
Synthesis and Conclusion
Test Report Production
Fig. 1. Test Procedure Outline.
It is also necessary to convert the abstract test
cases into corresponding executable test cases sui-
table for the intended real tester. This convertion
can be done before or after the selection and
parameterization. The result of conversion, selec-
tion and parameterization is called a para-
meterized executable test suite, which is the actual
test suite to be run.
The third step is the execution of the para-
meterized executable test suite. This can be thought
of as comprising three phases: basic interconnec-
tion testing, which is optional, capability testing,
and behaoiour testing,. If executed, basic intercon-
nection testing can detect severe cases of noncon-
formance, and thus passing this phase gives confi-
dence that it is worthwhile engaging in more com-
prehensive (and therefore more expensive) testing.
Capability testing is designed to ascertain the
validity of the PICS with respect to the actual,
observable capabilities of the IUT. Behaviour test-
ing is, however, the main activity in this step,
concentrating on checking the correct behaviour
of the protocol implementation in a large repre-
sentative sample of instances of communication.
These will cover both simple and complex situa-
tions, involving both correct and erroneous be-
haviour on the part of the communicating partner.
It cannot be proved by testing that an implemen-
84 D. Rayner / OSI Conformance Testing
tation conforms dynamically in all instances of
communication. However, it can be shown by
testing that an implementation consistently con-
forms dynamically in representative instances of
communication.
The fourth step is the analysis of the results.
This may in fact be interleaved with the execution
of different groups of test cases, but it is easiest to
think of it coming after the test execution step is
over. The results will be analysed with respect to
both the test purpose of each test case and the
validity of the behaviour observed (i.e. with re-
spect to the relevant conformance requirements).
This is discussed in more detail below.
The fifth step is the final conformance review,
which involves a synthesis of the results of the
behaviour tests with what has been learned from
the analysis of the PICS and the results of the
capability tests. It can then be seen whether the
results are consistent with the capabilities stated
in the PICS. A conclusion on the conformance of
the I U T to the requirements of the standard(s) can
then be reached. These results are recorded in
standardized Conformance Test Reports, as dis-
cussed below.
At any state in this process the test laboratory
and its client can agree to abandon the process.
However, even if it is abandoned it is recom-
mended that some sort of test report should be
produced to record the results achieved, such as
they are, but it must be clear that such a test
report is not a full conformance test report.
4.1. Analysis of Results
The observed outcome is the series of events
which occurred during execution of a test case; it
includes all input to and output from the I U T at
the points of observation and control.
The foreseen outcomes are identified and de-
fined by the abstract test case specification taken
in conjunction with the protocol standard. For
each test case, there may be one or more foreseen
outcome(s). Foreseen outcomes are defined prim-
arily in abstract terms.
A verdict is a statement of pass, fail or incon-
clusive to be associated with every foreseen out-
come in the abstract test suite specification. The
analysis of results is performanced by comparing
the observed outcomes with foreseen outcomes.
The verdict assigned to an observed outcome is
that associated with the matching foreseen out-
come. If the observed outcome is unforeseen then
the abstract test suite specification will state what
default verdict shall be assigned.
The verdict will be pass, fail or inconclusive:
• pass means that the observed outcome satisfies
the test purpose and is completely valid with
respect to the conformance requirements;
• fail means that the observed outcome is invalid
with respect to the conformance requirements;
three categories of fail are used to distinguish
between those outcomes which fail with respect
to the test purpose, those which satisfy the test
purpose despite the error, and those which are
inconclusive with respect to the test purpose;
• inconclusive means that the observed outcome
is valid with respect to the conformance re-
quirements but inconclusive with respect to the
test purpose.
The verdicts made in respect of individual test
cases will be synthesised into an overall summary
for the I U T based on the test case executed.
The results will be documented in a set of
conformance test reports. These reports will be of
two types: a System Conformance Test Report
(SCTR), and a Protocol Conformance Test Report
(PCTR).
The SCTR, which will always be provided,
gives an overall summary of the conformance
status of the system under test (SUT), with respect
to its single- or multi-layer IUT. A proforma for
the SCTR will be standardized in Part 5 of the
Methodology and Framework standard [6].
The P C T R is specific to each protocol tested in
the SUT; it documents all of the results of the test
cases giving references to all input to and output
from the conformance assessment process. If it is
necessary to repeat the conformance assessment
process, this document should give all the infor-
mation necessary to carry out such a re-assess-
ment, and also give the results against which the
new results can be compared. A proforma for the
P C T R should be standardized with the related
conformance test suite.
5. Test Suite Production
5.1. Test Suite Structure
Test suites have a hierarchical structure (see
Fig. 2) in which a key level is the test case. Each
 D. Rayner / 0S1 Conformance Testing 85

test case has a narrowly defined purpose, such as (d) includes a description of the initial state in
that of verifying that the I U T has a certain re- which the test body should start, in lieu of a
quired capability (e.g. the ability to support cer- preamble;
tain packet sizes) or exhibit a certain required (e) need not describe the postamble.
behaviour (e.g. behave as required when a particu- A generic test suite (i.e. a test suite composed of
lar event occurs in a particular state). generic test cases) has the coverage of the set or a
Within a test suite, nested test groups are used superset of all possible abstract test suites for a
to provide a logical ordering of the test cases. Test particular protocol.
groups may be nested to an arbitrary depth. They An abstract test case is derived from a generic
may be used to aid planning, development, under- case and the relevant protocol specification; it:
standing or execution of the test suite. (a) specifies the test case in terms of a particular
Test cases are modularized by using named test method;
subdivisions called test steps. Each test case com- (b) adds a more precise specification for se-
prises at least one test step: the series of events quences of events which are only described
covered by the test purpose (test body). It may informally in the generic test case;
include further test steps to put the I U T into the (c) adds the sequences of test events required to
state required for the test body to start from (the achieve the objectives of the preamble and
preamble) or to tidy up after the test body has postamble of the generic test case using a
finished (the postamble). Test steps may be nested specialized notation.
to an arbitrary depth. An executable test case is derived from an ab-
All test steps consist of a series of test events stract test case, and is in a form which allows it to
(e.g. the transfer of a single P D U or ASP to or be run on a real tester for testing a real implemen-
from the IUT). tation.

5.2. Types of Test Case 5.3. Main Steps of Test Suite Production Process

Three types of test case are defined: generic, In order to present the requirements and gen-
abstract and executable. eral guidance for abstract test suite specification,
A generic test case is one which: it is useful to assume a normal form of the process
(a) provides a refinement of the test purpose; of test suite production. Abstract test suite desig-
(b) specifies the sequences of test events (paths) in ners are not required to follow this normal form
the test body which correspond to verdicts of exactly, but are required to produce abstract test
' pass', ' fail' and 'inconclusive', using a special- suites which could have been produced by this
ized notation; process. Thus, they are recommended to use a
(c) is used as the common root of corresponding similar process involving the same steps but possi-
abstract test cases for different abstract test bly in a different order.
suites for the same protocol; Thus, the abstract test suite production process
is assumed to be as follows:
Test Suite
(a) study the relevant standard(s) to determine
what the conformance requirements (including
, ,
options) are which need to be tested, and what
: : : Libraries of
Common needs to be stated in the PICS;
Test Steps
Test Groups (b) decide on the test groups which will be needed
lllt]ll
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . .
to achieve the appropriate coverage of the
Test Cases conformance requirements and refine the test
groups into sets of test purposes;
(c) specify generic test cases for each test purpose,
Test Steps using some appropriate test notation;
. . . . ll Test Events
lLI] (d) choose the test method(s) for which the com-
plete abstract test cases need to be specified,
Fig. 2. Test Suite Structure. and decide what restrictions need to be placed
86 D. Rayner / OSI Conformance Testing
on the assumed capabilities of the testers and
test coordination procedures;
(e) choose a test notation and in it specify the
complete abstract test cases, including the test
step structure to be used;
(f) specify the inter-relationships among the test
cases and those between the test cases and the
PICS, in order to determine the restrictions on
the selection of test cases for execution and on
the order in which they can be executed;
(g) consider the procedures for maintaining the
abstract test suite.
6. Determining the Conformance Requirements and
PICS
Before a conformance test suite can be speci-
fied, the test suite designer must first determine
what the conformance requirements are for the
relevant standard(s) and what is stated in the
PICS proforma concerning the implementation of
those standard(s).
Part 2 of the Methodology and Framework
standard [3] specifies the requirements to be met
by protocol specifiers as a prerequisite to the
production of a conformance test suite for a par-
ticular protocol. It also gives additional guidance
for the production of new OSI protocol standards
to improve the clarity of their conformance re-
quirements. These requirements and guidance were
based on the U K input of a published conference
paper [7].
In practice early OSI standards might not con-
tain a clear specification of all the relevant confor-
mance requirements. In particular, the static con-
formance requirements might be badly specified
or even omitted. In such cases, the test suite
designer is advised to try to progress an amend-
ment or addendum to the relevant standard(s) to
clarify the conformance requirements. In the ab-
sence of the conformance requirements being
clarified, then the test suite designer is recom-
mended to adopt as a short-term solution the
assumption that everything which is not explicitly
stated to be mandatory or conditional is optional.
The test suite standard then needs to state clearly
what the implications of this are.
If no PICS proforma is included in a relevant
protocol standard, the test suite designer must
provide a PICS proforma to be processed as an
addendum to that standard, or in exceptional cir-
cumstances as an annex to the conformance test
suite standard.
7. Abstract Test Suite Structure
7.1. Basic Requirements
An abstract conformance test suite comprises a
number of test cases. The test cases are grouped
into test groups, nested if necessary.
7.2. The Test Group Structure
In order to ensure that the resulting confor-
mance test suite provides adequate coverage of the
relevant conformance requirements, the test suite
designer is advised to design the test suite struc-
ture in terms of nested test groups in a top down
manner. There are many ways of doing this; no
one way is necessarily right and the best approach
for one test suite might not even be appropriate
for another. Nevertheless, the test suite designer
should ensure that the test suite includes test cases
for each of the following categories where rele-
vant:
(a) capability tests (for static conformance re-
quirements);
(b) behaviour tests of valid behaviour;
(c) behaviour tests of syntactically invalid be-
haviour;
(d) behaviour tests of inopportune behaviour
(i.e. syntactically valid events occurring at the
wrong time);
(e) tests focusing on PDUs sent to the IUT;
(f) tests focusing on PDUs received from the
IUT;
(g) tests focusing on interactions between what is
sent and received;
(h) tests related to each mandatory protocol fea-
ture;
(i) tests related to each optional feature which is
implemented;
(j) tests related to each protocol phase;
(k) variations in the test event occurring in a
particular state;
(1) timing and timer variations;
(m) PDU encoding variations;
(n) variations in values of individual parameters;
(o) variations in combinations of parameter val-
ues.
 D. R a y n e r / OSI Conformance Testing 87

Table 1

A. Capability tests
A.1 Mandatory features
A.2 Optional features said by the PICS to be supported
B, Behaviour tests: response to valid behaviour by peer
B.1 Connection establishment phase (if relevant)
B.I.1 Focus on what is sent to the IUT
B.I.I.1 Test event variation in each state
B.l.l.3 Timing/timer variation
B.1.1.3 Encoding variation
B.l.l.4 Individual parameter value variation
B.l.l.5 Combination of parameter values
B.1.2 Focus on what is received from the IUT
- s u b s t r u c t u r e d as B. 1,1
B.1.3 Focus on interactions
- s u b s t r u c t u r e d as B . 1 . 1
B.2 Data transfer phase
- s u b s t r u c t u r e d as B . 1
B.3 Connection release phase (if relevant)
- substructured as B.1
C. Behaviour tests: response to syntactically invalid behaviour by peer
C.1 Connection establishment phase (if relevant)
C.1.1 Focus on what is sent to the IUT
C.I.I.1 Test event variation in each state
C.1.1.2 Encoding variation of the invalid event
C.1.1.3 Individual invalid parameter value variation
C.1.1.4 Invalid parameter value combination variation
C.1.2 Focus on what the IUT is requested to send
C.1.2.1 Individual invalid parameter values
C.1.2.2 Invalid combinations of parameter values
C.2 Data transfer phase
- substructuredas C.1
C.3 Connection release phase (if relevant)
- s u b s t r u c t u r e d as C. 1
D. Behaviour tests: response to inopportune events by peer
D.1 Connection establishment phase (if relevant)
D.I.1 Focus on what is sent to the IUT
D.I.I.1 Test event variation in each state
D.1.1.2 Timing/timer variation
D.l.l.3 Special encoding variations
D.l.l.4 Major individual parameter value variations
D.1.1.5 Variation in major combination of parameter values
D.1.2 Focus on what is requested to be sent by the IUT
- s u b s t r u c t u r e d as D , I . 1
D.2 Data transfer phase
- s u b s t r u c t u r e d as D . 1
D.3 Connection release phase (if relevant)
- s u b s t r u c t u r e d as D . 1

This list is not exhaustive; additional categories
might be needed to ensure adequate coverage of
the relevant conformance requirements for a
specific test suite. Furthermore, these categories
overlap one another and it is the task of the test
suite designer to put them into an appropriate
hierarchical structure. Table 1 is an example of a
suitable structure for a single-layer test suite.
If the test suite is to cover more than one layer,
then a single-layer test suite structure such as this
could be replicated for each layer concerned. In
addition, a correspondingly detailed structure
could be produced for testing the capabilities and
behaviour of multiple layers taken as a whole,
88 D. Rayner / 0S1 Conformance Testing
including the interaction between the activities in
adjacent layers.
7.3. Test Case Purposes
The test suite designer must ensure that, for
each test case in a conformance test suite, there is
a clear statement of the test purpose. It is sug-
gested that these test purposes should be produced
as the next refinement of the test suite after its
structure in terms of test groups has been defined.
The test purposes could be produced directly from
studying those clauses in the relevant standard(s)
which are appropriate to the test group concerned
For some test groups the test purposes might be
derivable directly from the protocol state table;
for others, they might be derivable from the P D U
encoding definitions or the descriptions of particu-
lar parameters, or from text which specifies the
relevant conformance requirements. Alternatively,
the test suite designer could employ a formal
description of the protocol(s) concerned and de-
rive test purposes from that by means of some
automated approach.
Whatever method is used to derive the test
purposes, the test suite designer should ensure that
they provide an adequate coverage of the confor-
mance requirements of the standard(s) concerned.
There should be at least one test purpose related
to each distinct conformance requirement.
In addition, the following example gives gui-
dance on what 'adequate coverage' might mean:
(a) for capability test groups:
• at least one test purpose per relevant fea-
ture;
• at least one test purpose per relevant P D U
type and each major variation of each such
type, using ' n o r m a l ' or default values for
each parameter;
(b) for test groups concerned with test event vari-
ation in each state:
• at least one test purpose per relevant
state/event combination;
(c) for test groups concerned with timers and
timing:
• at least one test purpose concerned with the
expiry of each defined timer;
• at least one test purpose concerned with
very rapid response for each relevant type
of PDU;
• at least one test purpose concerned with
very slow response for each relevant type of
PDU;
(d) for test groups concerned with encoding varia-
tions:
• at least one test purpose for each relevant
kind of encoding variation per relevant P D U
type;
(e) for test groups concerned with valid individual
parameter values:
• for each integer parameter, test purposes
concerned with the boundary values and
one randomly selected mid-range value;
• for bitwise parameters, test purposes for as
many values as practical, but not less than
all the ' n o r m a l ' or common values;
• for other parameters, at least one test pur-
pose concerned with a value different from
what is considered ' n o r m a l ' or default in
other test groups;
(f) for test groups concerned with invalid individ-
ual parameter values:
• for each integer parameter, test purposes
concerned with invalid values adjacent to
the allowed boundary values plus one other
randomly selected invalid value;
• for bitwise parameters, test purposes for as
many invalid values as practical;
• for all other types of parameter, at least one
test purpose per relevant parameter;
(g) for test groups concerned with combinations
of parameter values:
• at least one test purpose per 'critical' value
pair (representing a multi-dimensional
boundary);
• at least one test purpose per pair of inter-re-
lated parameters to test a random combina-
tion of relevant values.
7.4. Test Notation
Once the test purposes have been defined, the
test cases can be written in a test notation. It is
recommended that generic test cases are produced
first. In these just the test body, the heart of the
test case, will be written in a test notation. Ab-
stract test cases, on the other hand, will be com-
pletely defined in a test notation.
ISO and C C I T T have defined an informal test
notation for this purpose: the Tree and Tabular
Combined Notation (TTCN) [8]. T T C N was origi-
nally designed for human readability, with the
 D. Rayner / OSI Conformance Testing
ability to define all the relevant paths through a
test case and assign verdicts to each. It has now
been enhanced by the addition of a BNF syntax
description which serves as both a means of clari-
fying the human readable tabular notation and
also as the basis of a machine-processable version
which can be used as a transfer syntax when test
suites are transferred between systems.
There are three main sections to a T T C N
specification: declarations, the dynamic part and
constraints. The declaration part is used to declare
test suite parameters, the points of control and
observation, and the test events. The dynamic part
includes behaviour descriptions which use a tree
notation, labels, references to constraints, verdict
assignments and synchronization requirements.
The constraints part can use either sequences of
numeric or character values, or the standard ab-
stract syntax notation, ASN.1 [9], to describe in
detail the encoding of PDUs and their parameters.
8. Abstract Test Methods
8.1. Control and Observation and Abstraction
The ISO abstract testing methodology is based
upon the OSI reference model. Abstract test meth-
ods are described in terms of what outputs from
the IUT are observed and what inputs to it can be
controlled. More specifically, an abstract test
method is described by identifying the points
closest to the IUT at which control and observa-
tion are specified.
The OSI protocol standards define allowed be-
haviour of a protocol entity (i.e. the dynamic
conformance requirements) in terms of the PDUs
and the abstract service primitives (ASPs) both
above and below that entity. Thus the behaviour
of an (N)-entity is defined in terms of the (N)-
ASPs and ( N - 1)-ASPs (the latter including the
(N)-PDUs). Each of these two sets of interactions
could be observed and controlled from several
different points, locally or externally.
The possible points of observation and control
are identified by three factors:
(a) whether it is the ASPs or PDUs which are
observed and controlled;
(b) the layer identity of the ASPs or PDUs con-
cerned;
(c) whether they are controlled and observed
89
within the system under test or in a system
remote from the system under test - if the
latter then the ASPs or PDUs are dis-
tinguished by the addition of a double-quote
character (").
Complete control and observation of the ( N -
1)-ASPs (or ( N - 1)-ASP"s) will include control
and observation of the ( N ) - P D U s (or ( N ) -
PDU"s), but not vice versa.
It is possible that the ASP activity above the
IUT might be neither controllable nor observable,
directly or indirectly, in which case this activity is
said to be hidden. It is, however, assumed that the
( N - 1 ) - A S P activity will at least be indirectly
observable and controllable, via some real means
of communication (e.g. via ( N - 1)-ASP"s). Thus
when the ( N - 1 ) - A S P s are neither controllable
nor observable directly, conformance testing can
only be carried out if the ( N - 1)-service is pro-
vided sufficiently reliably for control and observa-
tion to take place remotely.
It is important to make the distinction between
the interactions which are controlled and ob-
served, and the subset of those observations on
which judgement is passed. Judgement can only
be passed on whether or not the conformance
requirements are being met. This will usually apply
to the PDUs exchanged rather than the particular
realizations of ASPs. However, the reason for
wishing to control ASPs is to t r y to determine
more precisely which PDUs should be exchanged
in a particular test.
The virtue of expressing control in terms of
ASPs is that this is an abstract form of specifica-
tion which does not unduly hmit the freedom of
test laboratories to implement the tests in differ-
ent ways using whatever interfaces are accessible
externally, provided that the required degree of
control and observation is available.
Associated with this abstract view of control
and observation, there needs to be a correspond-
ingly abstract view of the testers which perform
the control and observation. They are referred to
as the lower and upper testers. The lower tester
(LT) is the means for providing control and ob-
servation of events which approximate to what the
IUT sees at its lower SAP (i.e. the ( N - 1)-ASPs,
( N - 1 ) - A S P " s , ( N ) - P D U " s , etc). The upper tes-
ter (UT) is the means for providing control and
observation of events which approximate to what
the IUT sees at its upper SAP (i.e. the (N)-ASPs,
90 1). Rayner / OSI Conformance Testing
( N + 1)-ASPs, etc). In addition, there is the con-
cept of test coordination procedures (TCPs) which
are the rules for cooperation between upper and
lower testers during testing.
Unfortunately, not all events or states defined
in protocol standards can be controlled and ob-
served by an upper tester which can only talk in
terms of ASPs. Therefore, the concept of abstract
local primitive (ALP) has been introduced. An
ALP is an abbreviation for a description of the
control a n d / o r observation to be performed by
the upper tester, which cannot be expressed in
terms of ASPs but which relates to the events or
states defined within the relevant protocol stan-
dard(s). A typical ALP might be an abbreviation
for 'do whatever is necessary to get the IUT to
send an XYZ-PDU', or 'put the IUT into the
busy state'. The PIXIT is used to determine which
ALPs can be supported by an SUT, and therefore
which of the otherwise hidden protocol states and
events are testable.
8.2. Test Method Overview
The two main classes of end-system test meth-
ods are local and external. Local test methods are
characterised by observation and control being
specified in terms of events occurring within the
SUT, at the layer boundaries immediately above
and below the IUT. External test methods, on the
other hand, are characterised by the observation
and control of PDUs taking place externally from
the SUT, on the other side of the underlying
service provider from the IUT. The local test
methods are really only applicable to in-house
testing by suppliers, whereas the external test
methods are also applicable to testing carried out
by users and third party test centres. They also
have the advantage of being applicable to a realis-
tic communications environment.
There are three types of external test method:
distributed, coordinated and remote. Each comes in
three variant forms: single-layer, multi-layer and
embedded. Single-layer methods are designed for
testing a single-layer without reference to the layers
above it. Multi-layer methods are designed for
testing a multi-layer IUT as a whole. Embedded
methods are designed for testing a single layer
within a multi-layer IUT, using the knowledge of
what protocols are implemented in the layers above
the layer being tested. In fact, the preferred way of
testing a multi-layer IUT is by incremental use of
embedded methods, starting at the lowest layer
and working up to the top.
The external test methods vary according to
their ability to define a test management protocol
(TMP) to carry out the test coordination proce-
dures, or to express the test coordination proce-
dures only in terms of requirements.
There may seem to be rather too many test
methods, but already in practice all the single-layer
methods and the external embedded methods are
being used. Furthermore, the standardization of
these methods aims to enable test suite designers
to use the most appropriate method for their
circumstances, rather than to restrict them to a
few possibly inappropriate methods.
8.3. Single-layer Test Methods for Single-layer IUTs
For single-layer test methods, the abstract
model of the IUT is called the (N)-entity under
test.
8. 3.1. The Local Single-Layer Test Method
The Local Single-layer (LS) abstract test method
defines the points of control and observation as
being at the service boundaries above and below
the (N)-entity under test. The test events are
specified in terms of (N)-ASPs above the IUT
and ( N - 1)-ASPs and ( N ) - P D U s below the IUT,
as shown in Fig. 3. In addition ALPs may be used
as test events. Abstractly, a lower tester is consid-
ered to observe and control the ( N - 1)-ASPs and
( N ) - P D U s while an upper tester observes and
controls the (N)-ASPs and ALPs. The require-
ments to be met by test coordination procedures
used to coordinate the realizations of the upper
and lower testers are defined in the abstract con-
formance test suites, although the test coordina-
tion procedures themselves are not.
TestI ~ASPs
](NI-PDUs
~ -ASPs
Fig. 3. The LS Test Method.
 D. Rayner / OSI Conformance Testing 91

and lower testers. There is only a single point of

control and observation, at the opposite side of
ASPs the ( N - 1)-Service provider from the (N)-entity
under test. Test events are specified in terms of
( N - 1)-ASP"s, ( N ) - P D U s and test management
Service Provider PDUs, as shown in Fig. 5. ALPs are not needed as
test events in this method because the same effect
Fig. 4. The DS Test Method. can come from a suitably defined TM-PDU.
For lower layers (1-3) where it may be unre-
alistic to specify observation and control of ( N -
8. 3.2. The Distributed Single-Layer Test Method
1)-ASP"s, the observation and control will be
The Distributed Single-layer (DS) abstract test
specified in terms of TM-PDUs, ( N ) - P D U s and
method defines the points of control and observa-
when necessary the state of the underlying connec-
tion as being at the service boundaries above the
tion.
(N)-entity under test and at the opposite side of
the ( N - 1)-Service Provider from the (N)-entity
8. 3.4. The Remote Single-layer Test Method
under test. The test events are specified in terms
The Remote Single-layer (RS) abstract test
of (N)-ASPs above the IUT and ( N - 1 ) - A S P " s
method defines the point of control and observa-
and ( N ) - P D U s remotely, as shown in Fig. 4. In
tion as being on the opposite side of the ( N - 1)-
addition ALPs may be used as test events. Ab-
Service Provider from the (N)-entity under test.
stractly, lower and upper testers are again consid-
The test events are specified in terms of the ( N -
ered to observe and control the behaviour at the
1)-ASP"s and ( N ) - P D U s remotely, as shown in
respective points. The requirements to be met by
Fig. 6. In addition ALPs may be used as test
the test coordination procedures are again defined
events. Some requirements for test coordination
in the abstract conformance test suites, although
procedures may be implied or informally ex-
the procedures themselves are not.
pressed in the abstract conformance test suites,
For lower layers (1-3) where it may be unre-
but no assumptions are made regarding their
alistic to specify observation and control of ( N -
feasibility or realization.
1)-ASP"s, the lower tester observation and control
will be specified in terms of ( N ) - P D U s and when For the lower layers (1-3) where it is unrealistic
to specify observation and control of ( N - 1 ) -
necessary the state of the underlying connection.
The observation and control to be performed ASP's, the observation and control will be speci-
by the lower tester can optionally be specified in fied in terms of ( N ) - P D U s and when necessary
the state of the underlying connection.
terms of (N)-ASP"s where this will reduce the size
ALPs are particularly important in this method,
of the test case specification without loss of re-
as they provide the only means of specifying up-
quired precision.
per tester control over IUT initiated PDUs. Also
the only observations above the IUT that can be
8.3.3. The Coordinated Single-layer Test Method
The Coordinated Single-layer (CS) abstract test expressed in this method are those expressable as
method is an enhanced version of the DS method, ALPs. This is a major difference between this and
using a standardized upper tester and the defini- the DS and CS test methods.
tion of a test management protocol to realize the
test coordination procedures between the upper

:~]~' I_P.
roceduresi ;:::
L__~-(NI-PDU-s~
l (~-')-"S~'' I I
Service Provider Service Provider

Fig. 5. The CS Test Method. Fig. 6. The RS Test Method.

92 D. Rayner / OSI ConformanceTesting
8.4. Multi-layer Test Methods for Multi-layer IUTs
(LM, DM, CM, RM)
Multi-layer testing consists of testing all the
layers of a multi-layer IUT as a whole, when the
combined allowed behaviour of the multi-layer
implementation must be known, without control-
ling or observing any of the inter-layer boundaries
within the IUT.
The Local Multi-layer (LM) test method is like
the LS test method but the IUT goes from layer
( N ) to ( N + n). Hence the test events are speci-
fied in terms of ( N + n ) - A S P s and ALPs above
the IUT and ( N - 1 ) - A S P s and ( N ) to ( N + n)-
PDUs below the IUT.
The three external multi-layer test methods
(DM, CM and RM) are like the corresponding
single-layer test methods (DS, CS and RS, respec-
tively) but for an IUT which goes from layer ( N )
to ( N + n). Thus for the DM test method, the test
events are specified in terms of ( N + n)-ASPs and
ALPs above the IUT and ( N - 1)-ASP"s and ( N )
to ( N + n ) - P D U s remotely. For the CM test
method, the test events are specified in terms of
( N - 1 ) - A S P " s , ( N ) to ( N + n)-PDUs and TM-
PDUs; the test management protocol should be
designed to operate over the ( N + n ) - S e r v i c e ,
where ( N + n) is the highest layer in the IUT. For
the RM test method, the test events are specified
in terms of ALPs above the IUT and ( N - 1 ) -
ASP"s and ( N ) to ( N + n)-PDUs remotely; some
requirements for test coordination procedures may
be implied or informally expressed, but no as-
sumptions are made regarding their feasibility or
realization.
8.5. Single-layer Testing of Multi-layer IUTs or
SUTs (Embedded Methods)
In embedded single-layer test methods testing
is specified for a single-layer within a multi-layer
IUT, including the specification of the protocol
activity in the layers above the one being tested
but without specifying control or observation at
service boundaries within the multi-layer IUT.
Thus in a multi-layer IUT from layer ( N ) to
( N + n), abstract test cases for testing layer ( N + i)
include the specification of the PDUs in layers
( N + i + 1) to ( N + n) as well as those in layer
(N+i).
The Local Single-layer Embedded (LSE) test
I I I (N) I I I I ON) I
(N-I )- Service Provider (N-l)- Service Provider
(a) Example of DSE test method: (b) Example of CSE test method:
testing (N+l)-protocol in testing (N+l)-protocol in
n-layer IUT n-layer rUT
Fig. 7. The EmbeddedTest Methods.
method uses the same points of control and ob-
servation as the LM test method for the same set
of layers. The test events will also be specified in
the same terms as for the LM test method. The
difference is that the LSE test method con-
centrates on a single-layer at a time, whereas the
LM test method tests the multi-layer IUT as a
whole.
In the Distributed Single-layer Embedded
(DSE) test method for layer ( N + i) within a
multi-layer IUT from the layer ( N ) to ( N + n) the
points of observation and control are at the service
boundary above the IUT and at the opposite side
of the service provider from the IUT, as illustrated
in Fig. 7(a). The test events are specified in terms
of ( N + n ) - A S P s and ALPs above the IUT and
( N + i - 1)-ASP"s and ( N + i) to ( N + n)-PDUs
remotely. For the top layer in the multi-layer IUT,
( N + n), this method is the same as the DS test
method.
The Coordinated Single-layer Embedded (CSE)
test method is a cross between the CM and DSE
test methods, in which the test events are specified
in terms of ( N + i - 1)-ASP"s, ( N + i) to ( N + n)-
PDUs and TM-PDUs, as illustrated in Fig. 7(b).
The test management protocol is designed to oper-
ate over the ( N + n)-Service.
The Remote Single-layer Embedded (RSE) test
method uses the same point of control and ob-
servation as the RS test method for the same
layer, but differs from the RS test method in that
( N + i + 1) to ( N + n)-PDUs are specified in test
cases for layer ( N + i).
Successive use of a single-layer embedded test
method (from layer ( N ) to ( N + n ) ) is called
 D. Rayner / OSI ConformanceTesting
incremental testing of a multi-layer IUT.
The embedded single-layer test methods are
defined for a single layer under test in a multi-layer
IUT. This does not mean that there cannot be
accessible service boundaries within the multi-layer
IUT, just that no such boundaries are used in the
test methods. Thus, all layers between the layer
under test and the highest layer for which abstract
test cases specify P D U s as test events are regarded
as being part of the multi-layer IUT.
9. Real Testers
9.1. Introduction
A real tester is a realization of the lower tester,
together with either a realization or specification
(as appropriate) of an upper tester and the specifi-
cation of the way in which the test coordination
procedures are to be realized.
It is important to recognise that there is not a
one-to-one mapping between abstract test meth-
ods and real testers. Each abstract test method can
be realized by more than one real tester and m a n y
real testers can each realize more than one ab-
stract test method.
A real tester can be used to realize a particular
abstract test method if an abstract test suite for
that abstract test method can be translated into an
executable test suite for that real tester. This re-
quires that the control and observation performed
by the realizations of the lower and upper testers
are at least as good as that specified in the ab-
stract test suite, and that the means of synchroniz-
ing the lower and upper tester activities is at least
capable of achieving the requirements of the test
coordination procedures specified in the abstract
test suite.
SUT
[(N-1)-Serv|ceProvider ] Tester
Fig. 8. External Lower Tester.
93
9.2. The Lower Tester
The realization of the lower tester should have
the ability to control the generation of ASPs and
PDUs, and to observe and store the outcomes, as
defined in the abstract test cases.
The realizations of lower testers can be classi-
fied architecturally as follows:
(a) external lower tester - observing and control-
ling the I U T on the other side of the underly-
ing service-provider - see Fig. 8;
(b) local lower tester - observing and controlling
the lower boundary of the I U T directly - see
Fig. 9;
(c) hybrid lower tester - capable of observing and
controlling the I U T both from the other side
of its underlying service provider and directly
at its lower boundary - this category is of
more theoretical than practical interest but
could be build by enabling an external lower
tester to interact with a probe within the SUT
positioned at the boundary in question - see
Fig. 10.
External and hybrid lower testers can also be
classified physically as follows:
(i) networked - connected to the SUT via a
network - see Fig. 11;
(ii) linked - directly linked to the SUT (close-
coupled) - see Fig. 12;
(iii) divided - consisting of two inter-communi-
cating parts, one directly linked to the SUT
and the other connected to the first via a
network - see Fig. 13;
Real lower testers (which realize the lower tester)
can also be classified internally as follows:
(1) encoder/de¢oder - just encodes and decodes
the ASPs and P D U s as required for the test
case being run, without being an implementa-
tion of the protocol(s) in question;
(2) enhanced implementation - an implementa-
SUT
IUT
I PCO
Lower
Fig. 9. Local Tower Tester.
 94 D. Rayner / OSI Conformance Testing
SUT
I (N-I) - Service Provider
Fig. 10. Hybrid Lower Tester.
tion of the protocol(s) concerned, modified by
the addition of an error generator, configura-
tion module or similar device to ensure that
invalid or unusual ASPs or PDUs can be
generated as required by the test case being
run.
The basis of an enhanced implementation is
often referred to as a 'reference implementa-
tion' but this term is found to be confusing. It
is not the protocol implementation which
should be regarded as the reference, but rather
the abstract test suite. Also the protocol im-
plementation does not need to include every
option as long as the additional features en-
able the desired control and observation to
take place. However, the protocol implementa-
tion could, if desired, be partially generated
automatically from a formal description of the
protocol.
All classes of external and hybrid lower tester
can be used to realize any of the external abstract
test methods. All classes of local and hybrid lower
tester can be used to realize any of the local
abstract test methods.
The functions of a lower tester which are com-
mon across all test methods are the following:
(a) executing the executable test cases;
(b) forming PDUs for submission to the underly-
ing layer service;
(c) operating the underlying layer service;
(d) analysing primitives received from the un-
derlying service, including interpretation of the
data content;
Lower
Tester SUT
i @ i l__ @
Fig. 11. Networked Lower Tester.
Lower
Tester SUT
I I
Link
Fig. 12. Linked Lower Tester.
(e) reconciling PDUs extracted from the received
underlying layer service primitives with the
appropriate protocol specification(s) to assess
the validity of the syntax of each PDU and the
validity of the relationships between received
PDUs and previously submitted PDUs;
(f) enabling the operation of the test coordination
procedures to synchronize behaviour of the
lower and upper testers, to the extent per-
mitted by the test method in use;
(g) reporting the observed behaviour in a way
which permits conformance to be assessed with
respect to the relevant standard(s), and which
permits comparability with the results of other
equivalent testing operations.
9.3. The Upper Tester
The ways in which the real tester can interact
with the IUT in the SUT are numerous in terms of
upper testers with or without synchronization, and
one can identify several common implementation
types:
(a) the upper layers of the SUT are used to realize
upper tester functionality - this can only be
used to realize the remote abstract test method
see Fig. 14;
-
(b) upper tester realized by a human being at a
user interface that maps onto the IUT service
boundary - see Fig. 15;
(c) upper tester realized by a portable remote
scenario interpreter, taking its instructions
from files, with a mapping region between it
and the IUT service boundary - see Fig. 16;
Networked
Part of Linked
Lower Part of SUT
Tester Lower
Tester
I I Link i
Fig. 13. Divided Lower Tester
 D. Rayner / OSI Conformance Testing 95

SUT
implementation Tester I
of
Ferry
Xigher Layer
Protocols Lower [1
Tester I [
Active I:
Ferry
Control j
I Protocol"
Passive
Ferry
IUT Mapping Region

L. . . . . . . . . . . . . . . . . . . . . . . . I, ..o!°°,,
ii =, .... t.t,on I
Ferry I
ChT,, IUT
PCO'

Fig. 14. Upper Tester Functionality Using Normal Protocols. I

( N - l ) - Service P r o v i d e r

\
Test Channel
User Interface
Fig. 19. Upper Tester realized using a Ferry.
IUT
l e s t Management Astride Test
Protocol Responder
L........................
Lower
Fig. 15. Upper Tester Ftmctionality by Human User. Mapping

_••
Tester Region
Control
Specified
in scenarios Test Management Mapping IUT
Remote on disc Protocol Channel
Scenario / Region
Observation
Interpreter on disc / I
Mapping (N-I)-ServiceProvider
Region
KN
TestChannel
IUT
Fig. 20. Upper Tester realized by an Astride Test Responder.

Fig. 16. Upper Tester realized as a Portable Remote Scenario

Interpreter. (d) upper tester realized by a portable test re-
sponder operating a test management pro-
Test Management Test tocol, with a mapping region between it and
Protocol Responder
the IUT service boundary - see Fig. 17;
Mapping (e) upper tester embedded inside the IUT, func-
Region
tioning either as a remote scenario interpreter
IUT or as a test responder - see Fig. 18,
(f) upper tester realized as a passive ferry [10]
with a mapping region between it and the I U T
service boundary - see Fig. 19;
Fig. 17. Upper Tester realized as a Portable Test Responder. (g) upper tester realized as an astride test re-
sponder, communicating with both the I U T
and the lower service boundary beneath the
IUT - see Fig. 20.
Test Management If the real tester is claimed to support the Local
TR
Protocol or Distributed abstract test methods, the upper
IUT tester will have the ability to control and observe
......................
test events, and either store the outcomes for
off-line analysis or transmit appropriate informa-
tion to the real tester for analysis. If the real tester
Fig. 18. Upper Tester realized within the IUT. is claimed to support the Coordinated abstract
96 D. Rayner / OSI Conformance Testing

test method, the upper tester will have the ability Abstract Executable
to control and observe test events, and to use the
required test management protocol to exchange Generic Test
appropriate information with the lower tester. For Suite
the Remote abstract test method, there are no
requirements on the upper tester, so any of the ~ T e s t Method
possible types of upper tester can be used.
I Real Tester
Abstract Test ! I ExecutablesuiteTest
9.4. Synchronization Suite

If the real tester is claimed to support the ~PICS ~---PICS

Local, Distributed or Coordinated abstract test
methods, the real lower tester must have the abil-
ity to synchronize with the realization of the upper
tester. For the Local method, this can be done
using inter-process communication. For the Coor-
dinated method, synchronization will be done
using the required test management protocol. For
the Distributed method, synchronization can be
done by any suitable test management protocol,
by a ferry protocol with inter-process communica-
tion, or through manual coordination (by use of a
telephone a n d / o r terminal connection to the SUT)
if this can be made precise enough to meet the
requirements for test coordination procedures de-
fined in the test cases.
Synchronization is optional for the Remote
method, but can be done either through manual
coordination or through actions of the SUT (in-
cluding the use of higher layer protocols).
10. Executable Test Suites
10.1. Executable Test Suite Derivation
An executable test suite is composed of a num-
ber of executable test cases. These executable test
cases are derived from abstract test cases con-
tained in standardized abstract test suites. This
derivation may take one of three forms shown in
Figure 21:
(a) the executable test suite can be derived from
the abstract test suite;
(b) the selected executable test suite can be
selected from the executable test suite as ap-
propriate for the PICS provided for the IUT;
alternatively, the selected executable test suite
can be derived from the selected abstract test
suite;
(c) the parameterized executable test suite can be
parameterized from the selected executable test
Real Testers, )
Selected Abstract Selected Executable
Test Suite Test Suite
~PIXIT ~---PIXIT
Parameterized
Abstract Test
P,,T Teter,l
Real
Parameterized
Executable Test
Suite Suite
Fig. 21. Test Suite Types.
suite as approximate for the PIXIT provided
for the IUT; alternatively, the parameterized
executable test suite can be derived from the
parameterized abstract test suite.
10.2. Executable Test Suite Realization
When deriving executable test cases from ab-
stract test cases, the executable test suite realizers
should ensure that the derivation process pre-
serves the meaning of each abstract test case and
the test method chosen.
In particular, the executable test suite realizers
must strictly maintain the test purpose of an ab-
stract test case in its executable version, thus
preserving the quality of the abstract test suite
coverage with respect to the protocol standard.
Test groups and other relevant subdivisions in
the abstract test suite, should be preserved in the
process of derivation. Thus, basic interconnection,
capability and behaviour test cases will be identi-
fied in the executable test suite to the same extent
that they are in the abstract test suite.
Additionally, the executable test suite realizers
must ensure that the assignments of verdicts to
outcomes are translated accurately. This means
that for every possible outcome of an executable
test case there must be a corresponding outcome
 D. Rayner / OSI Conformance Testing
in the abstract test case and the verdicts assigned
to each of the two must be the same.
The executable test suite realizer should ensure
that they understand the intent of the abstract test
case and implement this intent faithfully.
Furthermore, attention of executable test suite
realizers is drawn to the necessity of checking the
correctness and feasibility of test coordination
procedures, as they are expressed in the abstract
test case. In the case of the Remote method, this
may imply detailed knowledge of the SUT and the
testing environment.
If the test method or real tester involves the use
of a Test Management Protocol (whether stan-
dardized or not), it may be desirable to include in
the executable test suite additional test cases de-
signed to verify that the upper tester has com-
pletely and accurately implemented the required
Test Management Protocol.
In any case, the executable test suite realizers
should ensure that they realize the intent of the
test coordination procedures.
Conversely, the executable test suite realizers
need not maintain:
(a) a-one-to-one correspondence between abstract
and executable test cases; it is conceivable that
an abstract test case can be implemented as
several executable test cases, and vice versa,
provided that all of the above requirements
are met;
(b) all the hierarchical levels of the abstract test
suite structure;
(c) anything in the abstract test suite which is
explicitly defined as being implementation de-
pendent.
10.3. Test Selection
The Test Laboratory will acquire from a test
realizer an executable test suite for the chosen test
method and real tester for each relevant protocol
or set of protocols to be tested together.
This executable test suite should include cross-
references from each executable test case to each
abstract test case.
The Test Laboratory will select all those execu-
table test cases which correspond to abstract test
cases appropriate to the SUT, This means that the
Test Laboratory will select:
(a) all basic interconnection test cases that are
independent of the contents of the PICS;
97
(b) all basic interconnection test cases that require
the presence of a capability stated as present
in the PICS;
(c) all capability test cases for mandatory capabil-
ities;
(d) all capability test cases for optional or condi-
tional capabilities identified in the PICS as
being implemented in the SUT;
(e) all behaviour test cases that are mandatory;
(f) all behaviour test cases that assume the pres-
ence of optional or conditional capabilities
that are present in the SUT according to the
PICS;
(g) all behaviour test cases that assume the ab-
sence of optional or conditional capabilities
that are not present in the SUT according to
the PICS.
Once the Test Laboratory has selected the re-
quired test cases - the 'selected executable test
suite' - then the information provided in the
P I X I T should be used to parameterize those test
cases. The resulting 'parameterized executable test
cases' are then ready to be executed.
11. Conclusion
This paper has described the concepts con-
cerned with OSI conformance testing and the
specification of abstract conformance test suites,
as they are being standardized by ISO and CCITT.
The OSI Conformance Testing Methodology and
Framework standard will lead the way to the
production of standard conformance test suites
for OSI protocols and to the harmonization of
testing activities carried out by different test
centres, suppliers and users. This in turn should
enable OSI to succeed in its objective of
widespread interworking of open systems.
Acknowledgements
The author, as ISO rapporteur for OSI confor-
mance testing, is in a privileged position to pro-
duce this paper. He wishes to acknowledge the
help of both ISO and C C I T T rapporteur groups,
without which this paper could not have been
written. He also wishes to acknowledge the advice
and encouragement of his colleagues at N P L and
in the N C C C O M M S - A I D team at the National
98 D. Rayner/OSIConformance Testing
Computing Centre, Manchester, UK. His own
work at NPL is supported by (a) the Computing,
Software and Communications Committee of the
Technology Requirements Board of the Depart-
ment of Trade and Industry, (b) the UK con-
sortium for OSI conformance testing (currently
consisting of NCC, NPL, British Telecom, DEC,
IBM and ICL) and (c) the Corporation for Open
Systems.
References
[1] ISO/TC9/SC21, Information processing systems - Open
Systems Interconnection - Basic Reference Model, ISO
7498, 1984.
[2] ISO/TC97/SC21, OSI conformance testing methodology
and framework - Part 1: General Concepts, ISO 2nd DP
9646-1 revised text, edited by D. Rayner, Vancouver,
December 1987.
[3] ISO/TC97/SC21, OSI conformance testing methodology
and framework - Part 2: Abstract Test Suite Specifi-
cation, ISO 2nd DP 9646-2 revised text, edited by D.
Rayner, Vancouver, December 1987.
[4] ISO/TC97/SC21, OSI conformance testing methodology
and framework - Part 3: Executable Test Derivation,
edited by D. Rayner, Tokyo, June 1987.
[5] ISO/TC97/SC21, OSI conformance testing methodology
and framework - Part 4: Requirements on Suppliers and
Clients of Test Laboratories, edited by D. Rayner, Tokyo,
June 1987.
[6] ISO/TC97/SC21, OSI conformance testing methodology
and framework - Part 5: Test Laboratory Operations,
edited by D. Rayner, Tokyo, June 1987.
[7] D. Rayner, Towards an objective understanding of con-
formance, in: Protocol Specification, Testing and Verifica-
tion III, proc. 3rd international workshop on this subject,
held in Zurich on 31 May-2 June 1983, edited by H.
Rudin and C.H. West, North Holland, 1983, 477-492.
[8] ISO/TC97/SC21, The tree and tabular combined nota-
tion, Annex E of Part 2 of [3], edited by A. Wiles,
Vancouver, December 1987.
[9] ISO/TC97/SC21, Information processing systems - Open
Systems Interconnection - Specification of Abstract Syn-
tax Notation One (ASN.1), ISO 2nd DIS 8824, 1986.
[10] H.X. Zeng and D. Rayner, The impact of the ferry
concept on protocol testing, in: Protocol Specification,
Testing and Verification V, proc. 5th international
workshop on this subject, held in Moissac, Toulouse,
France, on 10-13 June 1985, edited by M. Diaz, North
Holland, 1985, 519-531.