Conformance Testing

Lynne S. Rosenthal
Software Diagnostic and Conformance Testing Division
(301)975-3353
lynne.rosenthal@nist.gov
Outline
 Software Standards and Testing
 Conformity assessment
 Conformance Testing
– what is it
– why bother
– how it works
Software Standards
 Promote portability & interoperability
 Success depends on:
– Efficiency of standard development process
– Quality of resulting standard
– Ability to ensure implementations conform to the
standard
Software Testing
 Testing - process of trying to find errors by
means of experimentation
 Testing of realistic system not exhaustive
– can show presence of errors not their absence
 Testing is either
– structural testing (white box)
– functional testing (black box)
• conformance testing
Conformance Testing

 Determineswhich areas of the specification*

are implemented correctly.
– tests capture the technical description of the
specification
– testing is bound in scope by the specification
– internal structure of product-under-test not
accessible

*publicly available standard

Why: conformity assessment
 ISO Guide 2: “any activity concerned with
determining directly or indirectly that relevant
requirements are fulfilled.”

 communication between buyer and seller

– buyers increased confidence
– sellers substantiate claims
Why:
 Improves quality of software
 Objective method for evaluating products
– ensures presence of required characteristics
 Promote portability and interoperability
How: building tests
 Falsification Testing
– find errors by means of experimentation
– show presence of errors not their absence
– prove non-conformance
 Tests based on:
– required functionality
– logical errors by misunderstanding requirements
– errors from boundary conditions and divergence
– common programming errors
 Requirements

Software

Conformance Tests
Test programs
 When:

 Develop in parallel with specification

– feedback to specification for clarification,
correction, etc.
 Developin partnership with product
developers
Summary: Conformance tests

 Determine if a product has required

characteristics
 Increases product confidence
 Communication between buyer and seller
 Improve quality of software