203
Conformance Testing for OSI Protocols *
Richard J. L I N N , Jr.
National Institute of Standards and Technology, National Com-
puter Systems Laboratory, Gaithersburg, MD 20899 USA
Abstract. In the early 1980's, research and development was
initiated on methods to test the developing International
Standards Organization's Open Systems Interconnection
(ISO/OSI) communications protocols. In 1983, ISO initiated
work to develop standardized test methods. A tutorial overview
of the test methods and test notation named q'TCN which were
developed within ISO are presented. Issues regarding multi-
layer test methods are still not completely resolved. These
issues are identified and alternatives to ISO's methods are
explored. Application of the formal description techniques
named ASN.1 and Estelle to multi-layer test systems is il-
lustrated. Concluding remarks summarize the status of current
practice in conformance testing and status of evolving OSI
testing standards.
Keywords. ASN.1, communications protocols, conformance
testing, ferry methods, Open Systems Interconnection (OSI),
Estelle, test architectures, test methodology, Tree and Tabular
Combined Notation (TTCN).
Richard (Jerry) Lima is a computer
scientist and manager of the Auto-
mated Protocol Methods Program
within the National Computer Sys-
tems Laboratory, at the National In-
stitute of Standards and Technology
(NIST) in Maryland. Before joining
NIST, he was a senior systems en-
ginecr and managed groups support-
hag data communications, field en-
gineering, and systems engineering of
data acquisition and process control
systems used in research laboratories
at Virginia Tech.
His recent research and development activities at the NIST
have focused on the formal description techniques for OSI
protocols named Estelle and Abstract Syntax Notation One
(ASN.1) and conformance evaluation methodology for OSI
protocols. He has contributed to international standards in
both areas. Linn received his B.S. degree in Forestry in 1972
and an M.S. degree in Computer Science in 1981 from Virginia
Polytechnic Institute and State University.
* This work is a contribution of the National Institute for
Standards and Technology (formerly, the National Bureau
of Standards) and is not subject to copyright.
North-Holland
Computer Networks and ISDN Systems 18 (1989/90) 203-219
0169-7552/90/$3.50 1990, Elsevier Science Publishers B.V. (North-Holland)
1. I n t r o d u c t i o n
Given that we have international standards for
communication protocols which define their be-
havior and encoding of messages, testing imple-
mentations of protocols should be done in a uni-
form way and be relatively easy. This is the shared
goal of the standards bodies who define the proto-
cols and test methods, test laboratories, and
vendors who implement the standards in products.
(Consumers of communication products just want
them to work together!) But given the complexity
of Open Systems Interconnection (OSI) protocols
and the state of conformance evaluation methods,
"Is the goal a reality or a fantasy ... ?" The reality
is that conformance testing is a difficult technical
issue fraught with problems. So, we start with a
discussion of the issues and problems which in-
eludes at least the following:
- Initially, standards may be based upon a paper
design, and as a result, may contain errors,
omissions and ambiguities.
- Several options or alternative behaviors may be
allowed.
- Often, conformance criteria are not precisely
specified in a standard.
- In a layered protocol architecture, is a stack of
protocols to be tested as individual layers, as a
collective unit, or both?
-Individual layer testing implies exposed in-
terfaces must exist at each layer (interfaces
might not be exposed in a product).
-Multi-layer testing (a group of two or more
layers) raises issues of how layers should be
grouped for testing; i.e., at which layers will
interfaces exist and what services are actually
implemented?
- A a result of decisions made regarding
s
grouping, multi-layer testing may restrict the
amount of testing possible on lower layers
within the group.
- Is the testing to be conducted:
-locally (the test system has direct access to
the interfaces of the implementation), or
-remotely (the test system has access to the
implementation via underlying communica-
tions services)?
204 R.J. Linn, Jr. / Conformance testingfor OSI protocols
- What test coordination procedures are to be
used (if any) and what degree of coordination is
desirable a n d / o r acceptable?
- Are test coordination procedures to be auto-
mated? If so, what test coordination protocol
definitions are to be used in an automated
environment?
- May requirements beyond those specified in a
protocol standard be placed upon a supplier to
make a product testable?
- How are tests to be specified:
- test cases, interpreted/executed by a soft-
ware tool;
- by a suite of programs, each testing specific
functions;
- by a reference entity enhanced with features
for testing?
- Are the tests themselves to be standardized? If
so, are test cases generated by manual, semi-au-
tomated or fully automated methods?
- How is test suite coverage to be measured, and
how many tests are enough?
- How are test cases to be selected from a test
suite given optional features in a protocols and
valid implementation choices?
- What information must be recorded during test-
ing to support verdicts rendered either manu-
ally or automatically?
- What is the form and structure of the informa-
tion recorded?
- What is the format of test reports:
- what information must be reported;
- what information must be treated as con-
fidential;
- who owns the information; and
- when must a test report be made public?
- Are executable test systems to be standardized?
- Are testing procedures to be standardized?
- How are mutual recognition of test results and
test reports to be achieved?
The issues range from technical to political and
legal areas. The issues and the assumptions un-
derlying the issues influence answers to the ques-
tions. Different resolutions lead to different ways
to conduct conformance testing. Nonetheless, all
test methods attempt to assess the static and dy-
namic aspects of a protocol implementation with
respect to criteria specified in a standard. Exam-
pies of the static aspects of testing are assessing
the validity implementation choices regarding op-
tions and services offered, and assessing encodings
of protocol data units (PDUs). Examples of the
dynamic aspects of testing are assessing the valid-
ity of messages sent in response to messages re-
ceived and assessing error recovery mechanism.
The process of testing is outlined after ISO's test
methodology is introduced.
ISO has provided some answers in an evolving
standard composed of five parts [11-13]. The scope
of the standard includes definition of concepts
and terms, test methods, a test notation, and pro-
formas for use in protocol standards and test
reports. It also defines requirements for suppliers
of test systems and executable test suites, test
laboratories, and their clients. Since this work is
intended to be applied by other working groups
within ISO and CCITT, other answers are found
in the protocol standards themselves: annexes to
standards identify specific testing requirements
and solicit specific answers from suppliers of OSI
products. Additional answers are found in test
suites being developed by ISO and CCITT for
individual protocols or groups of protocols (e.g.,
what test methods will be standardized for a par-
ticular protocol).
This tutorial focuses on a subset of the techni-
cal issues. It begins with an overview of early
research and development. Then an overview of
the ISO work on conformance evaluation method-
ology presents a number of alternatives and
answers to some of the questions raised. Finally,
problems unanswered by current ISO work are
identified, and alternative solutions to the prob-
lems are proposed.
2 . B a c k g r o u n d
Within Europe in the early 1980's, formal col-
laborative research efforts were initiated to estab-
lish methods for testing implementations of com-
munications protocol, for conformance, as inter-
national standards were emerging under the
umbrella of Open System Interconnection (OSI).
Initial collaboration was between Agence de r l n -
formatique (ADI), Paris, France; Gesellschaft fiir
Mathematik und Datenverarbeitung (GMD),
Darmstadt, F R G ; and the National Physical
Laboratory (NPL), Teddington Middlesex, En-
gland. Each research laboratory had industrial
partners and focused upon a different aspect of
testing. ADI designed and implemented an X.25
 R.J. Linn, Jr. / Conformance testingfor OSl protocols
tester; GMD developed a language oriented ana-
lytic tool for passive monitoring and error detec-
tion for the ISO Session protocol; NPL developed
a test system for a Network Service. Since the
National Institute of Standards and Technology
(NIST) in the USA was developing a test system
for the ISO Transport Class 4 protocol, NIST was
invited to participate. Each of the tools developed
were based upon different design architectures
and philosophies. By 1984, other researchers from
Europe and Canada were invited to join the effort.
Much of the early work is reported in the proceed-
ings of the first through fifth IFIP workshops on
Protocol Specification, Testing and Verification
[1-5]. Early work considered a variety of test
architectures, test languages, and demonstrated
the viability of several test methods, automated
generation of test sequences from formal specifi-
cations, and specification languages for protocols.
In 1986, Bochmann [25] and his students surveyed
this work and identified tools for development of
formal specifications, code generation from formal
specifications and tools employed for testing.
Results and terminology coming out of the
initial research efforts were subsumed by ISO's
work on conformance testing methodology. Ini-
tiated in 1983, it subsequently became a joint
effort with the International Telegraph and Tele-
phone Consultative Committee (CCITT). A se-
quence of demonstrations between 1984 and 1988
showed the progress of this work. Among the
earhest demonstrations of OSI protocol was one
at the National Computer Conference in 1984.
For six months prior to the show, the National
Bureau of Standards and General Motors Corpo-
ration conducted testing on thirteen vendors' pro-
totype implementations of the Transport Class 4
and IEEE 802.3/4 protocols [34]. In 1985, the
Industrial Technology Institute conducted tests on
prototype implementations of the File Transfer
and Management, Session, Transport, Connec-
tionless Network and IEEE 802.3/4 protocols in
preparation for a demonstration at the AUTO-
FACT'85 trade show [34]. In 1988, the Enterprise
Networking Event (ENE '88) demonstrated Mes-
sage Handling Systems (electronic mail) as well as
the other protocols operating over local networks
and global interconnections via wide area net-
works. Corporation for Open Systems demon-
strated testing of OSI products at ENE '88. Pre-
ENE '88 testing employed tools which implement
205
some of the test methods described in the next
section. However, this does not imply that all
aspects of OSI protocol testing are mature. Test-
ing of lower layer protocols is the most mature
because they are older (e.g., X.25 and Transport);
upper layer protocols are newer and present unique
problems for two reasons:
(1) several layers of protocol may be embedded
in a single product; and
(2) use of Abstract Syntax Notation One
(ASN.1) [7-8] to describe message syntax permits
many encodings of the same message.
The next two sections 1 introduce ISO's test
methods and test notation (Tree and Tabular
Combined Notation--TTCN). The terminology of
ISO is adopted and is introduced in context.
3. ISO's Single Layer Conformance Methods
A brief overview of the methodology and
framework developed by ISO is presented. The
concepts and vocabulary used within ISO also
provide the framework to discuss other work. The
ISO documents [11-13] and Rayner [40-42] pro-
vide more detail. It is important to remember, this
discussion focuses on models or a framework
which are independent of implementation.
ISO's work defines a framework for the de-
scription of tests whose purpose is to provide
sufficient information to determine if an imple-
mentation of a protocol conforms to a standard. A
judgment is made based upon static and dynamic
aspects of an implementation. Static conformance
criteria focus on proper formation of data struc-
tures, coding of protocol data units and condi-
tional criteria which depend upon legitimate
choices by an implementor about services offered
by the Implementation Under Test (IUT). Dy-
namic conformance criteria focus on the behavior
of an IUT as it interacts with another implementa-
tion (or a test system).
3.1. The Local Method
The local method of conformance testing (Fig.
1) is important as the basis for a large body of
1 D u e to overlapping subject matter, parts of Sections 3, 4 and
6 are nearly identical to materials contained in another paper
[38].
206 R.J. Linn, Jr. / Conformance testingfor OSl protocols
Upper
Tester
Fig. 1. Local method.
terminology and concepts that are subsequently
applied to other models. A basic assumption of
the local method is that exposed interfaces exist
above and below the IUT. These interfaces serve
as points of control and observation (PCOs); i.e.,
points at which a real test system can control
inputs to and observe outputs from an IUT.
Using ISO's conventions, the layer under test is
referenced as the N-layer and the next lower layer
as the ( N - 1)-layer. Protocol service definitions
define abstract service primitives (ASPs) ex-
changed at the top interface of a protocol entity.
Abstractly, each protocol entity exchanges
( N - 1)-ASPs with an underlying service provider.
ASPs comprise a logical set of test events (with
parameters) which can be controlled and observed
at the interfaces of an IUT. The set of test events
exchanged at the top interface are denoted as
(Nt)-ASPs; t for top. The same is true of the
b o t t o m interface, and the test events are called
( N b - 1)-ASPs; b for bottom. Each protocol de-
fines a set of messages to be exchanged with a
peer entity; they are called Protocol Data Units
(PDUs). ( N ) - P D U s are exchanged as data in the
parameters of some of the ( N b - 1)-ASPs; other
( N b - 1)-ASPs convey control information to use
the services of the ( N - 1)-layer. The test method
includes two logically distinct elements which are
called the upper tester and lower tester because of
their relationship to the interfaces of the IUT.
In earlier work, an implementation of a lower
tester was called either a test driver or an en-
coder/decoder; an upper tester was called a test
responder. But, ISO did not adopt these terms.
The upper tester is assumed to generate and re-
ceive a set of test events complementary to the set
of (Nt)-ASPs generated and received by the I U T
at its top interface. Similar assumptions are made
for the b o t t o m interface of the I U T and the lower
tester. The model in Fig. 1 comprises a test har-
ness around the I U T which coordinates the ac-
tions of the upper and lower tester. The roles of
the upper and lower testers are to stimulate the
I U T by exchanging test events at the top and
b o t t o m interfaces of the IUT. The lower tester is
also assumed to record (log) test events so that the
behavior of the I U T m a y be assessed and there is
evidence supporting p a s s / f a i l (or possibly incon-
clusive) verdicts.
Test cases provide the means of specifying the
actions of the upper and lower testers. Test cases
specified in T T C N m a y be interpreted by a test
system (manually translated, or compiled into
some executable form). The statements of a test
case may make reference to named PCOs, (Nt)-
ASPs, ( N b - 1)-ASPs, ( N ) - P D U s and their com-
ponent fields. Thus, test cases define the actions of
the upper and lower testers which control and
observe test events exchanged at the upper and
lower interfaces of the IUT. In the local method,
test-coordination procedures are expressed ab-
stractly by the ASPs used to specify the actions of
the upper and lower testers.
Note, even though the lower tester is below the
IUT, it functions as a peer protocol entity. The
lower tester exchanges N - P D U s with the I U T
through its b o t t o m interface. Therefore, the lower
tester may emulate normal behavior of an N-layer
protocol entity during testing, or it may inject
errors to test for error recovery.
In summary, the local method of testing as-
sumes a test harness around the I U T and exposed
interfaces above and below the IUT. It may not be
applicable when conformance evaluation is done
by a test laboratory because these of assumptions.
3.2. Distributed Method
When an implementor (client) arranges for a
test laboratory to test a protocol implementation,
direct access to the b o t t o m interface of the I U T is
not likely to be available (except in the data link
layer where media standards require an exposed
interface). The distributed method (Fig. 2) is one
of three models which make no assumptions about
the existence of a PCO at the b o t t o m of the IUT.
 R.J. Linn, Jr. / Conformance testing for 0.9 protocols
c I
(N-l) Service Provider
Fig. 2. Distributed method.
In all methods employing a communications
service, IS0 denotes the complementary set of
abstract service primitives available to the lower
tester as (Nb - l)-ASP”s (double prime).
Note, the lower tester and IUT reside in two
different systems. The lower tester and IUT are
connected by an underlying OSI service which
offers an (N - 1)-service using lower-layer proto-
cols and the physical media connecting the sys-
tems. Despite its name, the lower tester is obvi-
ously a peer entity of the IUT in this figure. The
arrows between the IUT and OS1 service do not
imply a real interface, just the conceptual flow of
N-PDUs. Also note, an upper tester is part of the
model and an exposed top interface is assumed at
the PCO.
Several consequences flow from changed as-
sumptions:
(1) abstract test cases written for the local
method are not applicable-they must be rewrit-
ten to reflect the abstract service primitives availa-
ble to the lower interface of the lower tester.
Generally, they are complementary to those as-
sumed available at the bottom interface of the
IUT in the local method;
(2) the lower tester and IUT are physically
separated with the implication that they observe
the same test event at different times;
(3) data loss, delivery out of sequence, and
data corruption are possible, particularly at lower
layers due to the quality of some lower-layer
services;
201
(4) synchronization and control are more dif-
ficult because elements of the test system are
distributed over two systems,
Synchronization and control (test coordination
procedures) may be specified by ASPS exchanged
at PCOs (or possibly by a test management proto-
col which is not standardized). The distributed
method relies on the protocol being tested to
provide sufficient synchronization to achieve test
purposes. Therefore, judgments and verdicts for-
mulated depend upon behavior observed by the
lower tester. In practice, there must be some coor-
dination between upper and lower testers; the
same tests must be selected and executed concur-
rently. However, the distributed method does not
address how these issues are resolved.
In summary, the distributed method is a logical
equivalent to the local method with lower tester
and IUT interconnected by a communications
service. However, the structure of the local method
implicitly gives the capability to synchronize and
control the upper and lower testers because they
are elements of the same test harness. Information
collection and sharing is possible (although a local
issue). Thus, the distributed method is not a func-
tional equivalent to the local method.
3.3. Coordinated Method
Two features that distinguish the coordinated
method (Fig. 3) from the distributed method are:
(1) no exposed upper interface is necessary
within the IUT (although this is not precluded);
and
Fig. 3. Coordinated method.
208 R.J. Linn, Jr. / Conformance testing for OSl protocols
(2) a standardized test management protocol
(TMP) and test management protocol data units
(TMPDUs) are used to automate test management
and coordination procedures. Often it it is as-
sumed that the lower tester is the master and the
upper tester is a slave to minimize the effort in
realizing an upper tester.
This is the most sophisticated model. It allows
a very high degree of coordination and reporting
of information observed and collected at both the
upper and lower testers. Upper and lower testers
may be synchronized, information received by the
upper tester may be reported back to the lower
tester for formulation of verdicts, selection of test
cases by an operator may be conveyed from the
lower tester to the upper tester, and branching
logic may be implemented for selecting tests if a
previous test case fails. To date, ISO has assumed
minimal coordination (e.g., test cases will be ex-
ecuted in a predefined order with no branching if
the verdict of a test case is fail). This assumption
can lead to serious synchronization problems if
the test management protocol does not resynchro-
nize the upper and lower testers after a test fails.
Communications between upper and lower tes-
ters may be in-band (TMPDUs are carried as data
via the protocol being tested) or out-of-band (use
of a lower-layer protocol which is assumed to be
reliable enough to carry TMPDUs). Test manage-
ment using out-of-band services raises issues about
which layer services are assumed to be exposed.
Unfortunately, ISO conformance methodology
has not addressed the following issues:
- D e f i n i t i o n of a test-management-protocol
kernel which is independent of its application.
Without a standardized test-management-pro-
tocol kernel, standards groups will have to in-
vest the effort to invent their own. This has
happened with the Session and Transport ab-
stract test suites, and the test management pro-
tocols are distinctly different. If test laborato-
ries invented their own test-management proto-
cols, test services ~ould be incompatible which
is even worse. Currently, ISO is studying the
issue of a standardized kernel of test manage-
ment functions.
- Recommendations for in-band or out-of-band
communications. Both are feasible; both have
drawbacks. In-band communications may be
difficult at the application layer but is feasible
in some cases (e.g., electronic mail). Out-of-band
('3
Test Coordination
procedures
Lower
Tester ~ ~!~i~ Implementation
UnderTest
(Nb-])-ASP"s
t
(N-l) Service Provider
Fig. 4. Remote method.
communications below the network layer and
above the transport layer raises serious prob-
lems with exposed interfaces. Historically, it is
generally accepted that the transport layer will
have an exposed interface and be the basis of a
"test platform". Thus, the transport layer is a
logical candidate for providing out-of-band
communications for upper-layer testing.
3.4. Remote Method
The last method defined by ISO for single layer
testing is the remote method (Fig. 4). The signifi-
cant features are that no interface at the top of the
I U T is assumed, and no explicit test coordination
procedures are assumed. (Test coordination, if any,
is manual.) The method relies solely on the proto-
col being tested for synchronization of the lower
tester and the IUT. The method assumes that the
state of the I U T is known from actions specified
for the lower tester including knowledge of N-
PDUs transmitted and received via ( N b ) - A S P " s .
Verdicts must be formulated based upon stimulus
provided by the lower tester and the responses of
the IUT as observed by the lower tester. This
method is widely used for testing implementations
of X.25.
4. Tree and Tabular Combined Notation (TYCN)
The usual assumption regarding conformance
evaluation is: a test case (also called a test scenario)
is written with a particular test purpose in mind.
If test cases are to be the basis for I S O / O S I
 R.J. Linn, Jr. / Conformance testingfor OSI protocols
conformance testing, it is clear that a notation or
language is required to specify behaviors of a test
system and a protocol entity to be tested. The
language must contain sufficient features to de-
scribe tests for all OSI protocols. With these pur-
poses in mind, ISO has defined a notation which
is called Tree and Tabular Combined Notation
(TTCN). It is defined in the third document (called
Part 3 for brevity) of the ISO Conformance Test-
ing Methodology and Framework [12]. A test case
written in TTCN specifies, as a sequence of atomic
test events, the behaviors of a test system and a
protocol entity in sufficient detail to judge the
behavior of an implementation of the protocol to
formulate a verdict of pass/fail.
TTCN is intended to be applied by protocol
developers within CCITT and ISO. They will de-
fine abstract test suites. (An abstract test suite is
independent of the features of a protocol incorpo-
rated in a particular IUT, and it is independent of
any test system(s) employed by test laboratories to
conduct protocol testing. As such, abstract test
suites are not necessarily executable and TTCN
was not intended to be an executable test lan-
guage.) Thus, when developing an abstract test
suite, a standards body may assume a particular
test method from those outlined in the previous
section. However, standards bodies are not re-
sponsible for executable test suites or the imple-
mentation aspects of the methods employed to
describe tests.
To understand TTCN (and its name) one must
understand the structure of the notation and what
it is intended to express: TTCN is employed to
describe the dynamic behaviors and exchange of
messages between a test system and a protocol
entity. A graph of the dynamic behavior of a
protocol entity may be drawn with the nodes of
the graph labeled with inputs and edges labeled
with outputs (or vice versa). The resulting graph is
a tree reflecting the dynamic behavior of a proto-
col entity in response to its inputs. (Similar infor-
mation is often expressed as state graphs with
both inputs and outputs used to label the edges.)
Thus, TTCN is a language for describing trees of
behavior and actions associated with sending/
receiving test events and protocol data units
(PDUs).
TTCN's name is derived from the fact that the
graphic definitions in Part 3 use trees of behavior
specified in a tabular form to describe the dy-
209
namic aspects of a test case, and additional tables
to describe the static aspects of a test suite. For
example, tables are used to declare variables, iden-
tify points of control and observation, define ref-
erences to other standards where data types are
defined (structure and fields of protocol data
units), and to specify assumed constraints. (This
form is often called the graphics form, or TI'CN-
GR.) For every tabular element in TTCN-GR,
Part 3 also includes BNF syntax definitions for
the machine processable form of TTCN (TFCN-
MP). TTCN-MP is also called the transfer syntax
of TTCN because it allows for electronic exchange
of test suites.
The dynamic elements of TTCN include assign-
ment and arithmetic operators, predefined func-
tions (e.g., string manipulation), label declara-
tions, a goto statement (used to specify cyclic
behaviors), and input/output operations. The send
operation is denoted "!"; receive is denoted as
"?". Both may be qualified by the name of a point
of control and observation. These dynamic aspects
of TTCN are presented on paper in tabular form.
The column containing the text describing actions
is significant; i.e., blanks and tabs are significant
in the semantics of the language. When describing
a tree of possible input/output sequences, tabs are
used as a shorthand notation for the text in the
same column on previous lines; i.e., tabs are inter-
preted as if the text above was repeated.
An example may help. Consider a connection
oriented protocol which a lower tester employs as
an ( N - 1) service and assume the distributed test
method (Fig. 2). Five dynamic behaviors that
might be found in an input/output trace (log) are
tabulated below. They are presented in Table 1 as
sets of events and in Table 2 as a tree of behaviors
that may be observed at the PCO of the lower
tester (as they might be specified in TTCN).
(Nb - 1)-ASP's initiated by the lower tester are
Table 1
A set of dynamic behaviors for a connection oriented service
1 Con.Req Con.Conf Dat.Req Dat.Ind Dis.Req
2 Con.Req Con.Conf Dat.Req Dat.Ind Dis.Ind
3 Con.Req Con.Conf Dat.Req Dis.Ind
4 Con.Req Con.Conf Dis.Ind.
5 Con.Req Dis.Ind
Abbreviations: Con Connect Req Request
Dat Data Con Confirm
Dis Disconnect Ind Indication
210 R.J. Linn, Jr. / Conformancetestingfor OSl protocols
Table 2
A tree of dynamic behaviors observable by a lower tester
1 !Con.Req ?Con.Conf !Dat.Req ?Dat.Ind !Dis.Req
2 ?Dis.Ind
3 ?Dis.Ind
4 ?Dis.Ind
5 ?Dis.Ind
-TIME
Legend: ! Send
? Receive
in bold font in both tables; those observed by the
lower tester in response to actions taken by the
I U T are in R o m a n font. The numbers in the first
column of the two tables identify the correspond-
ing set of events. In Table 2, empty entries in the
left-hand columns of a row are interpreted as if
the text above it in the same column were copied
into the row. Table 2 is not a test case. In a test
case, the entries in row 1 of Table 2 could be put
into separate rows and each entry in the table
could be followed by additional test steps which
might include verdict assignment (verdicts are
specified in a separate column). Comments are
placed in the last column of a test case. A group
of test steps may be named and referenced as
group.
In T T C N , the scope of variables is global.
Behavioral expressions are deterministic; i.e., text-
ual order of test events in a test case is used to
resolve nondeterminism. Basic assumptions un-
derlying T T C N are that a protocol entity in an
I U T can be driven into an assumed " s t a t e " b y a
sequence of inputs and outputs (test steps). Once
reaching that state, another sequence of test steps
judges the behavior of a particular aspect of the
I U T and a verdict is formed. Finally, the I U T is
driven back to a known initial state by subsequent
actions specified in a test case. This suggests a
preamble, followed by a sequence of test steps,
followed by a postamble. The notions of preamble
and postamble are used in organization of a test
suite where certain ~aamed groups of test steps
may be used repeatedly (e.g., connection establish-
ment and termination). This functional grouping
of test steps leads to the notion of libraries of
named entries composing a test suite.
T T C N also facilitates organization of the ele-
ments of a test into larger units and, ultimately,
into a test suite; i.e., a hierarchical collection of
test cases. A test case m a y be composed of a
named preamble, body, and postamble; each may
be composed of one or more named elements.
(However, a p r e a m b l e and p o s t a m b l e are
optional.) Since all identifiers are global, individ-
ual elements of a test case m a y make reference to
previously identified objects (e.g., types, variables,
labels associated with a tree of behaviors). T T C N
allows reference to d e m e n t s of a test suite by
attaching a sequence of named test steps. Concep-
tually, this is equivalent to the "include" or " c o p y "
facilities of some programming languages; how-
ever text substitution follows specific recursive
rules. N a m e s m a y be qualified by other names
denoting the inclusion of hierarchically structured
elements of text.
An organization strategy for test suites sug-
gested by ISO is:
- basic interconnection tests which are intended
to establish that an I U T conforms sufficiently
to justify further testing (or identify severe cases
of non-conformance);
- capability tests which are intended to establish
that the static conformance requirements of a
protocol are met (but not probe the I U T for
detailed behaviors); and
behavior tests which are intended to test a full
-
range of dynamic behaviors which the supplier
of a product claims to support.
Together, capability and behavior tests are in-
tended to establish conformance or non-confor-
mance of an IUT. Rayner [42] describes the pur-
pose of this organization of tests in detail.
Operational semantics are being defined for
T T C N . (Early versions had no formal semantics.)
Without rigorous semantics:
(1) T T C N is subject to subtly different inter-
pretations; and
(2) it is possible for either humans or auto-
mated tools to interpret T T C N differently and
introduce errors when translating T T C N into an
executable test suite.
Wiles [46] describes an environment for T T C N
which includes both a syntax directed editor and
interpretive execution. Currently, this is the only
environment known which directly executes ab-
stract test cases. Probert et al. [39] are developing
an integrated environment for specification of
T T C N test suites which includes the facilities to
edit and transform canonical representations of
test cases into T T C N - G R , T T C N - M P and an
executable form. Others are also working on T T C N
 R.J. Linn, Jr. / Conformance testingfor OSI protocols
tool kits. Since machine translation and execu-
tion/interpretation of T T C N test suites is feasi-
ble, it is obvious that rigorous semantics are re-
quired if the intent of those individuals writing
test suites are to have the same interpretations.
5. Synthesis of the Concepts Presented
Before introducing new topics, let us sum-
marize the discussion thus far. ISO has defined
T T C N as a language for the specification of ab-
stract test suites for OSI protocols a n d I S D N
systems. T T C N was not intended to be an execu-
table language. Nonetheless, tools are being built
to directly interpret or translate T T C N into execu-
table tests. The g r a m m a r of T T C N - M P and oper-
ational semantics provide necessary definitions to
support these developments. Thus, T T C N must be
considered an executable language.
ISO has defined four single layer test methods
for OSI protocols. U p p e r and lower testers are
abstractions of elements that m a y be found in real
test systems; they serve as virtual interpreters of
T T C N within the constraints of a particular test
method•
These aspects are c o m m o n to the four test
methods:
- The role of the lower tester is to interpret
abstract test cases and formulate verdicts based
upon behaviors specified in a test case. (Note,
there is no requirement that elements of a real
test system formulate verdicts in real-time.)
- The lower tester acts as a peer entity of the I U T
and exchanges N - P D U s with the IUT.
These are the differences in the test methods:
- Control and observation by the lower tester is
specified in terms of:
- ( N b - 1)-ASPs in the local method;
- ( N b - 1 ) - A S P ' s in the other methods.
- The remote method assumes no control and
observation using an upper tester.
- The coordinated method assumes no upper in-
terface and employs T M P D U s to control the
upper tester.
- The local and distributed methods assume an
upper tester plus control and observation
specified in terms of (Nt)-ASPs.
- The distributed method m a y employ a non-
standard test management protocol (TMP).
211
However, a non-standard T M P is unlikely to be
used in a standardized test suite.
5.1• The Testing Process
Thus far, the process of testing an I U T has
been ignored; ISO calls this a test campaign• Fig-
ure 5 depicts how testing proceeds given some of
the elements discussed. It also introduces some
new elements. The following description is ideal-
ized: some steps m a y be combined, omitted
a n d / o r reordered.
The process model assumes a standards com-
mittee has done the following for each protocol to
be tested:
- developed at least one standardized abstract
test suite for the protocol using one of the test
methods described earlier;
- developed a Protocol Implementation Confor-
mance Statement (PICS) which defines a set of
questions that a product supplier answers be-
fore taking a product to a laboratory for test-
ing;
- developed a Protocol Implementation eXtra In-
formation for Testing (PIXIT) statement. The
P I X I T identifies implementation-specific infor-
mation regarding the I U T that is necessary for
testing (e.g., timer values required, number of
P D U s acknowledged in a single acknowledg-
ment, addressable elements in a stack of proto-
col entities).
The ISO test methodology identifies the latter two
items as required parts of a protocol standard. At
I of Static Analysis
Options
I
No > Verdict: Fail
Test Parameterization I
Test Execution ~ SCTR~
Results Analysis
Fig. 5. Test campaign.
212 R.J. Linn, Jr. / Conformance
this time, these elements do not exist for all OS1
protocols.
A test laboratory must either have the means to
interpret an abstract test suite directly or have
transformed it into an executable test suite by
some means. If a laboratory offers more than one
test method, it is the client’s choice which will be
used to test an IUT. The information contained in
PICS provided by a client identifies what options
and features of a protocol have been implemented.
The client’s PICS is used by a test laboratory for
static conformance assessment. Specifically, a
laboratory checks to assure that mandatory fea-
tures are included and conditional features (those
dependent upon other options) are also imple-
mented as required. The PICS is also used for test
selection; i.e., the laboratory eliminates tests for
features and options which are not implemented.
Information contained in the PIXIT is used to
parameterize a test suite. For example, timers and
addresses of each component protocol of the IUT
may have to be set before testing commences.
Tests are executed, results are analyzed, and
two reports are generated. A Protocol Confor-
mance Test Report (PCTR) contains sufficient
information to uniquely identify the system and
protocol tested, identifies the test suite and test
method employed, and contains verdicts for each
test case executed (or a note indicating that a test
was not executed) and identifies the conformance
log supporting verdicts rendered. Since more than
one protocol may be tested, there may be more
than one PCTR. A System Conformance Test
Report (SCTR) identifies a system which was
tested and gives a summary conformance state-
ment for each component protocol tested.
6. Embedded and Multi-layer Testing
Thus far, single layer test methods were pre-
sented. In the following subsections, we look at
ISO’s embedded test methodology, and then re-
port the status of multi-layer testing within ISO.
6.1. Embedded Methodr
Currently, IS0 has only defined what are known
as the embedded methods for testing one or more
layers of a multi-layer IUT without exposed layer
interfaces. Embedded testing focuses on one pro-
testingfor OSI protocols
tocol layer at a time within an IUT which is
embedded one or more layers down in a stack of
protocols within an IUT. Usually, testing proceeds
bottom-up within the stack, and upper layers are
tested incrementally as conformance of lowers is
established; i.e., focus of testing switches between
layers as testing proceeds. IS0 has defined em-
bedded testing for each of the four methods out-
lined in Section 3.
Conceptually, it is easy to consider lower layer
protocols as envelopes wrapped around upper
layer PDUs. This is certainly true for the data
transfer phase. At each successive layer, a header
prefixes “data” from the adjacent upper layer. It
is this relatively simple model that underlies the
notions of embedded testing. Assume that the
N-layer is to be tested and (N + l)-PDUs are to
be embedded in N-PDUs. Conceptually, two-layer
embedded testing defines a set of (N + l)-PDUs
and then envelopes this set of data in appropriate
N-PDUs. This model is depicted in Fig. 6.
The direct consequence of these assumptions is
that each test case must concurrently specify all
the actions of two layers (or at least a functional
subset of the (N + 1)-layer) in order to achieve a
test purpose. A test case must have a specific test
purpose and must reflect dynamic behaviors possi-
ble for two layers of protocol in the IUT. Ad-
ditionally, test cases must reflect the context of an
(N + l)-protocol in the lower tester (e.g., results of
negotiating options, or variables which record pro-
tocol control information that influence subse-
quent behavior) as the protocol progresses through
its phases (e.g., connection establishment, data
transfer, and connection termination). A test suite
Upper
Tester
Under Test
1 1 (Nb-I)-ASP?
I t I
I ’ (N-l)-Service
Fig. 6. Embedded method with test coordination.
 R.J. I.a'nn,Jr. / Conformance testingfor OSI protocols
consists of describing all possible behaviors be-
tween two peer protocol entities and reflecting the
context of the (N + 1)-protocol in order to pre-
vent a test from being aborted. In fact, this be-
comes quite complex very rapidly and becomes
progressively more difficult when attempting to
specify behaviors for more than two layers. At this
time, only limited experience exists with the
method. It is viable for two layers, but usually,
test cases use a relatively small subset of the
(N + 1)-protocol.
The advantage of an embedded method is that
it does not assume an exposed interface at every
layer of the IUT. The disadvantages are:
- test cases become more complex in a non-linear
relationship for each additional layer above the
layer to be tested;
- it is correspondingly more difficult to anticipate
all possible behaviors given that suppliers have
valid implementation options and choices for
each protocol that is involved in the test. Thus,
verdict assignment is more difficult and the
probability of either an inconclusive or invalid
verdict increases;
- test suites are defined assuming a particular set
of protocols; a new test suite must be written to
test the same layer if the combination of proto-
cols above N-layer change. Furthermore, if any
protocol in the combination changes, the entire
test suite must at least be checked, if not en-
tirely rewritten.
Due to the reasons identified above, embedded
testing presents some significant problems to those
defining abstract test suites for embedded meth-
ods. Test laboratories employing embedded meth-
ods face additional problems: they must obtain or
create, and subsequently maintain, executable test
suites and systems which reflect the intent of the
abstract test suites.
Given the possibility of single-layer and em-
bedded methods, ISO could produce test suites for
every possible combination of local, distributed,
coordinated and remote methods and groupings of
protocols. This would require significant resources
and add to the complexity of testing. In most
cases, standardized abstract test suites are likely to
be written employing one method per protocol (or
group of protocols). But, there are exceptions:
abstract test suites being developed by ISO for the
Session protocol include the single-layer coordi-
nated and single-layer embedded methods (under
213
the File Transfer and Management, and Message
Handling Systems protocols).
6.2. Multi-Layer Testing
Part I of the ISO documents defines multi-layer
testing as "Testing the behavior of a multi-layer
IUT as a whole, rather than testing it layer by
layer". However, no further guidance is given on
the topic and currently multi-layer testing is the
subject of research.
Advocates of the Ferry Clip method (intro-
duced in the next section) suggest that it may be
used for multi-layer testing. However, an assump-
tion they make is that the IUT has exposed inter-
faces at each layer which is not valid for many
commercial products. An example of one formal
method to conduct multi-layer testing is included
in the next section. While the methodology em-
ploys Estelle and ASN.1, the methodology may
also be exploited by proponents of LOTOS and
SDL.
7. O p e n Topics within I S O Conformance Method-
ology
While the following topics are under study by
ISO, we include them as part of an overview of
conformance testing for OSI. The first is ferry test
methods. Some argue that the ferry methods are
.simply a means to realize the test coordination
implied by either the distributed or coordinated
methods. Others believe they are distinct test
methods. Regardless of opinion, ISO has been
requested to consider the ferry methods as one
possibility of realizing test coordination as part of
a larger question: definition of a kernel set of test
management functions a n d / o r test management
protocol.
Second, ISO is studying the potential roles of
formal methods and formal description techniques
within conformance testing. The scope of this
topic includes generation of test sequences as well
as application of the formal description techniques
named Estelle [9], LOTOS [10] and SDL [18].
Since an explanation of test generation method-
ology is beyond scope of this paper and the topic
is surveyed in another paper [38], only some of the
most recent research results are noted below.
214 R.J. Linn, Jr. / Conformance testing for OSI protocols
Sabnani and Dahbura, and Aho et al. [43,23]
describe methods to optimize generation of test
sequences. A proposal was made to incorporate
results of optimization in the developing abstract
test suite for X.25. Sidhu and Leung [44] report on
the ability of several test generation methods to
detect faults. Ural [45] defined a method for selec-
tion of test cases based upon static control flow
and data flow analysis of descriptions of protocols
written in Estelle. Barbeau and Sarikaya have
developed a computer-aided design tool [24] which
can display both control flow and data flow graphs
of protocol specifications written in Estelle. For-
ghani and Sarikaya have created a tool to generate
TTCN test suites from EsteUe specifications [31].
To date, most applications of LOTOS have focused
on protocol specification and verification rather
than testing. Exceptions are the work of Brinksma
[27] and de Meer [30].
The following subsections report on topics
raised in ballots on the ISO documents and are
outstanding questions to be resolved.
7.1. Ferry Control and Ferry Clip Methods
Zeng [47] defined an alternative to both the
distributed and coordinated methods. The upper
tester is moved from the client's system to the test
laboratory's system. The upper tester in the client's
system is replaced by a ferry control protocol
entity (the name was originally derived from the
notion of a ferry boat). The model is depicted in
Fig. 7. Note, the block labeled "Test Coordination
Procedures" includes a peer ferry control entity
Ferry
Upper Control
Tester Protocol
I ; | communications is required. In his original work,
Test (Nt)-ASPs
Coordination:
I Procedures I I so-called "ferry control channel". In Fig. 7, some
Implementation
Lower
Tester N-PDUs~ Under Test
(Nb-1)-ASP"s
p,
(N-l) Service Provider
Fig. 7. Ferry method.
(although not explicitly identified). It is called an
"active" ferry entity because it assumes the role of
a master; the entity above the IUT is called a
"passive" entity because it responds to actions
taken by and directives issued by its peer. The
following summarizes the concepts found in the
original work and assumes in-band communica-
tions except as noted.
The ferry control entity serves as a carrier of
PDUs received from either the upper or lower
testers by retransmitting them to the other entity;
i.e., it functions in logical loop back mode.
Minimal state information and headers are re-
quired to realize a ferry control implementation,
and it can be designed in a relatively layer-inde-
pendent manner. The advantages of the ferry con-
trol protocol are:
(1) it places a relatively small burden on an
implementor,
(2) it is protocol independent,
(3) it is independent of test management proto-
col, and
(4) most test coordination problems are the
test laboratory's.
Its disadvantages are:
(1) an exposed interface is assumed at each
layer of the IUT,
(2) synchronization problems unique to the
ferry method may arise (it is a state based proto-
col entity),
(3) it requires an "interface adaptor" for each
layer, and
(4) it simply may not be applicable for Appli-
cation and Presentation layer protocols.
The ferry method does not resolve the in-band/
out-of-band communications issues described
earlier. Either may be used if the state of the IUT
is not changed by conveying data to and from the
upper and lower testers; otherwise, out-of-band
Zeng proposed out-of-band communications via a
reliable (N - k)-service may be used to realize the
ferry control channel (e.g., Transport protocol) if
there is an exposed lower layer interface. Other-
wise, the ferry control channel may have to be
derived by some other medium.
Extensions Zeng has proposed to his original
work are known as the ferry clip method [48]. In
brief, the ferry clip method assumes that exposed
interfaces exist at all layers of interest in the IUT,
 R.J. Linn, Jr. / Conformance testingfor OSI protocols
and that the passive ferry entity can be attached to
multiple PCOs in the IUT. Thus, the ferry entities
must multiplex streams of data t o / f r o m upper
and lower testers for multiple layers of protocol
within the IUT. Additionally, the ferry entities
must provide a reliable data transport mechanism
when testing lower layer protocols (e.g., network
and below) if the method is to provide a reliable
means of observation and control. It remains to be
seen if ISO will adopt either ferry method.
7. 2. Architectural Refinement
Current ISO methodology restricts the number
of points of control and observation in a lower
tester to exactly one PCO. Therefore, it is impossi-
ble to make reference to any PCO other than the
one immediately above the ( N - 1 ) service.
Specifically, within a test case specified in TTCN,
it is impossible to make reference to (Nt)-ASP"s
containing (N + 1)-PDUs .which might be gener-
ated by any other higher layer entity within a
lower tester; i.e., a "higher-layer test case" or
reference entity. For our purposes, assume a refer-
ence entity is:
(1) an implementation of a protocol which sup-
ports all functions and options specified in a pro-
tocol standard,
(2) can interoperate with any implementation
which realizes a valid subset of behaviors specified
in a standard, and
(3) has been tested sufficiently to demonstrate
the first two features.
While a reference entity may have been derived
from a formal description of a protocol by auto-
mated methods, this is not strictly necessary.
Whether or not an IUT is implemented with
interfaces at each layer, a test method may be
developed with layered components, each focusing
on a specific aspect of testing a multilayer imple-
mentation. Such a test method (system) acts as if
the IUT is layered, but it does not depend upon
the internal structure of the IUT as no assump-
tions regarding layering and interfaces are neces-
sary. Thus, the OSI Reference Model [6] suggests
an alternative to embedded methods. The Refer-
ence Model refines the problem of communica-
tions into layers of protocols, each with distinct
functions and services. This approach suggests
reference entities could also be used above the
N-layer in multi-layer test systems to maintain the
215
Lower Upper
Tester Test Mgmt Proel Tester
TMPDUs
-
Layer N+I •~ (N+I)-PDUs"~
Irnplementatior
UnderTest
Layer N ~ N-PDUs
"~(Nb-1 )-ASP"s
(N-1)-Service
Fig. 8. Architectural refinement.
protocol context for one or more layers above the
layer under test (Fig. 7). Indeed, Davis [29] de-
scribes the architecture of a test system which
employs reference entities above and below the
layer under test. Current practice at the Corpora-
tion for Open Systems (COS) demonstrates the
viability of the concept. COS employs a reference
implementation of the Transport Class 4 protocol
above the test entity for the CLNP (Protocol for
Providing the Connectionless Network Service).
Reasonable arguments support developing a
standardized test method which allows reference
entities or single-layer test entities to be inter-
changed at every layer of a real test system. For
example, ISO's FTAM and CCITT's MHS proto-
cols use different functional subsets of the Session
protocol and may run over different Classes of the
Transport protocol. Maintenance of test suites for
embedded methods will be expensive to ISO and
test laboratories. Perhaps, the strongest argument
is ISO's definition of multi-layer testing: "Testing
the IUT as a whole".
Given the approach depicted in Fig. 8 and a
multi-layer test method, the N-layer could be
tested with an arbitrary combination of protocols
above it. Note, this approach de-emphasizes the
role of test cases and puts the emphasis on refer-
ence entities. The consequences are:
(1) for purposes of testing, the test case/behav-
ior observed focuses on a single layer; and
(2) test scenarios play a significantly smaller
role and consequently become significantly sim-
pler.
216 R.J. Linn, Jr. / Conformancetestingfor OSl protocols
This architectural refinement of a "lower tester"
into a stack of reference entities above and below
the N-layer raises new issues:
- What role do test scenarios play and how do
they differ from single layer test scenarios?
- What points of control and observation:
- are relevant within a test system; and
- how is error recovery on the part of the IUT
tested?
- If any protocol entity above or below the layer
under test behaves invalidly:
- how can this be observed;
- what verdict should be assigned;
- what information should be logged; and
- how should reference entities be specified
a n d / o r realized?
If it is desirable to observe and report invalid
behavior observed on the part of the I U T in more
than one layer, nothing prohibits a test system
designer from extending a reference entity into a
test entity with this feature.
7.3. Formal Specification of Multi-layer Test Sys-
tems
CCITT has produced a Formal Description
Technique (FDT) named System Design Lan-
guage (SDL) [18]. Jointly, ISO and CCITT have
produced Abstract Syntax Notation One (ASN.1)
[7-8]. ISO has produced two FDTs, Estelle
[9,28,35] and LOTOS [10,26]. They became inter-
national standards in 1988; thus, their application
has been limited. To date, most OSI protocol
standards have used natural language and state
tables to define the standard. ASN.1 has been
employed to define message syntax for upper layer
protocols. If standardized FDTs are applied to
protocol specification, then it is quite natural to
augment their formal descriptions using the same
F D T to specify the behavior of a test entity. Since
translators for the FDTs exist, it is also quite
natural to realize executable test entities by trans-
lation.
However, testing requires mechanisms to inject
errors to test for error recovery on the part of the
IUT. A logical way to proceed is to simply extend
a standardized protocol specification by specifying
the behaviors necessary to invoke error recovery
on the part of the IUT. This implies three basic
changes to a formal description of a standardized
protocol:
(1) generation and transmission of valid PDUs
at points disallowed in the standard (inopportune
PDUs);
(2) transmission of parameters of PDUs which
are either out of range or semantically inconsistent
with the protocol standard; and
(3) invalid encoding of PDUs.
The first two cover most dynamic aspects of
testing error recovery mechanism in a standard.
The first can readily be achieved by defining ac-
tions in response to ( N t ) - A S P " s that are never
valid user sequences (e.g., request data transmis-
sion before a connection is established). The sec-
ond set can be achieved by defining additional test
services which override normal protocol con-
straints; i.e., define specific actions in response to
new abstract service primitives. Note, these ad-
ditional ( N t ) - A S P " s would not be invoked by a
"normal user" and therefore will have no effect
unless specifically invoked in either a single-layer
or multi-layer test system. The third set raises
pragmatic issues. Totally undecodable PDUs are
likely to generate inconclusive results unless the
protocol has specific error recovery mechanisms
for loss and corruption of PDUs. Exhaustive test-
ing in this domain is infeasible. Thus, the usual
approach is to intuitively construct a small set of
tests covering likely failures on the part of imple-
mentors.
The approach outlined above is more than the-
ory. Figure 9 depicts the refined architecture of a
test system developed at NIST for testing FTAM
through the Presentation layer protocols using
either the remote or coordinated methods. Thir-
teen modules are specified in Estelle: the Lower
Tester Manager down through a generic Session
interface. There are two major clusters of mod-
ules:
(1) the upper cluster of modules which were
designed to test F T A M [14-17] and the layers
below it, and
(2) the lower cluster of modules which are aug-
mented test entities that implement the FTAM,
ACSE and Presentation protocols. All are derived
from Estelle specifications of the corresponding
protocol entities. Estelle is used to specify and
define the dynamic aspects of the upper cluster of
modules and the stack of protocols in the lower
cluster. Augmented ASN.1 type definitions are
 R.J. La'nn,Jr. / Conformance testingfor OSI protocols
LOWER TESTER
Fig. 9. Multi-layertest architecturefor the FTAM,ACSEand
presentation protocols.
used to specify the encoder/decoders for the pro-
tocol stack and to define the test language of the
system.
Regardless of the internal structer of an IUT,
the components of the test system are able to
observe behavior at all three layers, detect and
report invalid behavior on the part of the IUT,
and log observed behavior (as ASN.1 values).
These components are also employed as part of a
system designed to test an application layer gate-
way for ISO's FTAM and DoD's File Transfer
Protocol (FTP). Translators for Estelle [22] and
ASN.1 [32,33] are used to realize implementations
of both test systems. Another test system for an
electronic mail gateway employs the same meth-
odology. Details of these systems [20,21] and the
methods employed to realize them are reported
elsewhere [36-38].
8. Concluding Remarks
We conclude with comments on fault detection,
tool development, progress of testing standards
and current testing practice. The fault detection
capacity of the test methods described is the sub-
217
ject of open debate. Fault detection is not only
related to the test method, but is related to fault
models, the test language employed, test suite
coverage, and synchronization issues. If all were
equal (and they are not), assumptions often made
are: from most powerful to least, the fault detec-
tion capacity of ISO's test methods is local, coor-
dinated, distributed and remote methods for single
layer testing. Embedded methods limit observabil-
ity and control and are generally considered
weaker than single layer methods. Proponents of
the ferry methods argue that ferry methods have
equal fault detection capacity as the local method.
The Conformance Testing Service (CTS) pro-
jects (CST-WAN and CTS-LAN), sponsored by
the Community of European Countries (CEC),
have produced a substantial number of test tools
and test suites. Many will be employed within
European test centers which have been established
to test OSI products for use throughout the CEC.
Information regarding the tools and test suites
may be ordered from the National Computing
Centre in the UK.
Parts 1, 2, 4, and 5 of the ISO testing methodol-
ogy should advance to international standard
status within a year; Part 3 is likely lag by up to
18 months. CCITT is applying TTCN to standard-
ized abstract test suites for the X.400 series recom-
mendations for electronic mail (Message Handling
Systems), X.25 and its components, and ISDN
D-channel testing. These are destined to become
CCITT recommendations by the end of the cur-
rent study period (1992). ISO has started develop-
ment of abstract test suites for Transport, Session
and upperlayer protocols in TTCN. If current
schedules within ISO are met, many of these test
suites will become international standards be-
tween 1990 and 1992.
Current testing practice is best described as a
patchwork quilt of old and new test systems (with
quite a few holes). The Corporation for Open
Systems (USA) and National Computing Centre
(UK) are using a mixture of older tools (adapta-
tions of 1985 and earlier vintage tools) to test
lower-layer protocols and newer tools developed
by the National Computing Centre (UK) for File
Transfer and Management (FTAM) and Message
Handling Systems (MHS). The gaping holes are
test suites.
Many test suites written in TTCN for MHS
and X.25 are available; FTAM test cases ~are
218 R.J. Linn, Jr. / Conformance testing for OSl protocols
scarce. Older tools use ad hoc test suites; but their
history of use makes them relatively complete.
References
IFIP: International Federation for Information Processing
[1] Protocol Testing - Towards Proof, D. Rayner and R.W.S.
Hale, eds., Vol. 1-2, INWG/NPL Workshop, National
Physical Laboratory, Teddington, Middlesex, T W l l OWL,
United Kingdom, 1981.
[2] Proc. Protocol Specification, Testing, and Verification II,
C. Sunshine, ed. (North-Holland, Amsterdam, 1982).
[3] Proc. Protocol Specification, Testing, and Verification III,
H. Rudin and C.H. West, eds. (North-Holland, Amster-
dam, 1984).
[4] Proc. Protocol Specification, Testing, and Verification IV,
Y. Yemini, R. Strom and S. Yemini, eds. (North-Holland,
Amsterdam, 1985).
[5] Proc. Protocol Specification, Testing, and Verification V,
M. Diaz, ed. (North-Holland, Amsterdam, 1986).
ISO: International Organization for Standardization, ISO
Secretariat for JTC1/SC21, ANSI, 1430 Broadway, New York,
NY 10018, USA.
[6] Information Processing Systems - Open Systems lntercon-
nection - Basic Reference Model, IS 7498, 1984.
[7] Information Processing Systems - Open Systems Intercon-
nection - Specification of Abstract Syntax Notation One
(ASN.1), IS 8824, 1987.
[8] Information Processing Systems - Open Systems Intercon-
nection - Specification of Basic Encoding Rules for Ab-
stract Syntax Notation One (ASN.1), IS 8825.2, 1987.
[9] Estelle: A Formal Description Technique Based on an Ex-
tended State Transition Model, IS 9074, 1988.
[10] Information Processing Systems - Open Systems lntercon-
nection - LOTOS - A Formal Description Technique
Based on Temporal Ordering of Observed Behavior, IS
8807, 1988.
[11] Information Processing Systems - 0S1 Conformance Test-
ing Methodology and Framework, ISO/IEC JCT 1/SC 21
DIS 9646, Parts 1-2, November 1988.
[12] Information Processing Systems - 05rI Conformance Test-
ing Methodology and Framework, ISO/IEC JCT 1/SC 21
N3077, Part 3, February 1989.
[13] Information Processing Systems - OSI Conformance Test-
ing Methodology and Framework, ISO/IEC JCT 1/SC 21
DIS 9646, Parts 4-5, March 1989.
[14] Information Processing Systems - Open Systems Intercon-
nection - File Transfer, Access and Management Part I:
General Introduction, IS 8571/1.
[15] Information Processing Systems - Open Systems Intercon-
nection - File Transfer, Access and Management Part H:
The Virtual Filestore, IS 8571/2.
[16] Information Processing Systems - Open Systems Intercon-
nection - File Transfer, Access and Management Part IH:
File Service Definition, IS 8571/3.
[17] Information Processing Systems - Open Systems Intercon-
nection - File Transfer, Access and Management Part IV:
File Protocol Profile, IS 8571/4.
Other Publications
[18] SDL, CCITT Recommendations Z.101-Z.104 (Blue Book
Series), Consultative Committee for International Tele-
graph and Telephone, International Telecommunications
Union, Place des Nations, CH 1211, Switzerland, 1988.
[19] Message Handling Systems, CCITT Recommendations
X.400, X.401, X.408, X.409, X.410, X.420, X.430 (Red
Book Series), 1984.
[20] A Test System for Implementation of M H S / S M T P Gate-
ways, National Institute of Standards and Technology,
National Computer and Systems Laboratory, ICST/SNA
87/5, Parts 1-3, September 1987.
[21] A Test System for Implementations of F T A M / F T P Gate-
ways, National Institute of Standards and Technology,
National Computer and Systems Laboratory, ICST/SNA
88/6, Parts 1-3, October 1988.
[22] Users Guide for the NBS Prototype Compiler for Estelle,
NBS, ICST, ICST/SNA 87/3, (Rev.) January 1989.
[23] A.V. Aho, A.T. Dahbura, D. Lee and M.U. Uyar, An
Optimization Technique for Protocol Conformance Test
Generation Based on UIO Sequences and Rural Chinese
Postman Tours, in: Proc. Protocol Specification, Testing
and Verification VIII (North-Holland, Amsterdam, 1989).
[24] M. Barbeau and B. Sarikaya, A Computer-Aided Design
Tool for Protocol Testing, in: Proc. INFOCOM '88 (IEEE,
New York, 1988) 86-95.
[25] G. Bochmann, Usage of Protocol Development Tools, in:
Proc. Protocol Specification, Testing and Verification VI1
(North-Holland, Amsterdam, 1988) 139-161.
[26] T. Bolognesi and E. Brinksma, Introduction to the ISO
Specification Language LOTOS, Comput. Networks ISDN
Systems 14 (1) (1987) 25-60.
[27] E. Brinksma, A Theory for the Derivation of Tests, in:
Proc. Protocol Specification, Testing, and Verification VIII
(North-Holland, Amsterdam, 1989).
[28] S. Budkowski and P. Dembinski, An Introduction to
Estelle: a Specification Language for Distributed Systems,
Comput. Networks ISDN Systems 14 (1) (1987) 3-24.
[29] W.B. Davis, Architecture and Design of a Portable OSI
Protocol Tester, in: Proc. 1st International Workshop on
Protocol Testing, Comput. Sci. Dept., Univ. British Col-
umbia Vancouver, B.C., Canada V6T 1W5 (1988).
[30] J. de Meer, Derivation and Validation of Test Scenarios
Based upon the Formal Specification Language LOTOS,
in: Proc. Protocol Specification, Testing, and Verification
111 (North-Holland, Amsterdam, 1987) 203-216.
[31] B. Forghani and B. Sarikaya, Automatic Dynamic Behav-
ior Generation in TTCN Format from Estelle Specifica-
tions, Dept. of Electrical and Computer Eng., Concordia
Univ., 1455 de Marsionneuve Blvd. W. 915, H3G 1M8,
Canada, 1889.
[32] P. Gaudette, S. Trus and S. Collins, An Object-Oriented
Model for ASN.1, in: K. Turner, ed., Proc. FORTE'88
(North-Holland, Amsterdam, 1989) 121-134; also Report.
No. ICST/SNA 88/4.
[33] P. Gaudette, S. Trns and S. Collins, ASN.1 Free Value
Tool, National Bureau of Standards, Institute for Com-
puter Sciences and Technology, ICST/SNA 88/2, January
1989.
[34] R.J. Linn, Testing to Assure Interworking of Implementa-
tions of ISO/OSI Protocols, Comput. Networks 11 (4)
(1986) 277-286.
 R.J. Linn, Jr. / Conformance testing for OSl protocols
[35] R.J. Linn, The Features and Facilities of Estelle, National
Bureau of Standards, Institute for Computer Sciences and
Technology, ICST/SNA 87/6, November 1988 (Rev.)
[36] R.J. Linn and J.P. Favreau, Application of Formal De-
scription Techniques to the Specification of Distributed
Test Systems, in: Proc. 1NFOCOM '88 (IEEE, New York,
1988) 96-109; also, Report No. ICST/SNA87/9.
[37] R.J. Lima, J.P. Favreau, L. Gebase and A. Iwabuchi, An
Overview of Formally Specified Multi-Layered Test Sys-
tems, National Bureau of Standards, Institute for Com-
puter Sciences and Technology, ICST/SNA 88/1, January
1988.
[38] R.J. Lima, Conformance Evaluation Methodology and
Protocol Testing, IEEE J. Select. Areas Comm. 7 (7)
(1989) 1143-1158.
[39] R.L. Probert, H. Ural and M.W.A. Hornbeek, An In-
tegrated Software Environment for Developing & Validat-
ing Standardized Conformance Tests, in: Proc. Protocol
Specification, Testing, and Verification VIII (North-Hol-
land, Amsterdam, 1989).
[40] D. Rayner, Towards an Objective Understanding of Con-
formance, in: Proc. Protocol Specification, Testing, and
Verification 111 (North-Holland, Amsterdam, 1984).
[41] D. Rayner, Towards Standardized OSI Conformance
219
Tests, in: Proc. Protocol Specification, Testing, and Verifi-
cation V (North-Holland, Amsterdam, 1986) 441-460.
[42] D. Rayner, OSI Conference Testing, Comput. Networks
ISDN Systems 14 (1) (1987) 79-98.
[43] K.K. Sabnani and A. Dahbura, A Protocol Test Genera-
tion Procedure, Comput. Networks 1SDN Systems 15 (4)
(1988) 285-297.
[44] D. Sidliu and T.K. Leang, Fault Coverage of Test Meth-
ods, in: Proc. INFOCOM "88 (IEEE, New York, 1988)
80-85.
[45] H. Ural, Test Sequence Selection Based on Static Data
Flow Analysis, Comput. Comm. 10 (5) (1987) 234-242.
[46] A. Wiles, B. Pehrson, ITEX-An Interactive TTCN Editor
and Executer, Swedish Institute of Computer Science,
Design Methodology Laboratory, Box 1263, S-163 13
Spanga, Sweden.
[47] H.X. Zeng and D. Rayner, The Impact of the Ferry
Concept on Protocol Testing, in: Proc. Protocol Specifica-
tion, Testing, and Verification V (North-Holland,
Amsterdam, 1986).
[48] H.X. Zeng, X.F. Du and C.S. He, Promoting the "Local"
Test Method with the New Concept "Ferry Clip", in:
Proc. Protocol Specification, Testin~ and Verification VIII
(North-Holland, Amsterdam, 1989).