Académique Documents
Professionnel Documents
Culture Documents
Mark Skall
Lynne S. Rosenthal
National Institute of Standards and Technology
mark.skall@nist.gov
lynne.rosenthal@nist.gov
Introduction
NIST works with industry to develop
standards and tests to improve the quality of
software and achieve interoperable solutions
Many years experience with
– Formal standards organizations and Consortia
– Developing conformance test suites, tools,
reference implementations
– Developing validation and certification testing
programs
Today’s Objectives
Present basic information about conformance
and conformance related topics
Achieve a common understanding
Lead a discussion about how conformance
applies to ebXML
Present overview of TA Spec. Conformance
Clause
Offer our assistance to WG in discussing and
developing conformance for their Spec.
Conformance
Outline clause
Conformance Testing
Background Validation
Certification
Certificate Branding
Terminology s
– Testing Methodology
– Testing Ingredients
ebXML Issues
TA Conformance Guidance
Background
Standards not enough to ensure interoperability
– Standards only meaningful if implemented in a consistent way
Need to ensure that implementations adhere to the
standard
– What is expected of implementations in order to claim
conformance – i.e., what are the requirements?
– How will we know if an implementation conforms?
• test suites, test tools
Different ideas of what conformance is
– Past experience may have affected view of conformance
Terminology
Conformance
Falsification Testing
– find errors by means of experimentation
– show presence of errors not their absence
– prove non-conformance; can never prove conformance
Tests built to check for:
– required functionality has been implemented (basic tests)
– logical errors by misunderstanding requirements
– errors from boundary conditions and divergence
– common programming errors
Specification
Requirements
Implementation
Conformance Tests
Test programs
100% ????
non conforming conforming
Terminology - Testing Process
VALIDATION - process necessary to perform
conformance testing in accordance with a prescribed
procedure and official test suite
– ensures that testing can be repeatable and reproducible
– ensures that conclusions are consistent with facts presented
in the evaluation
Validation
Process - policy and procedures for testing
Conformance Testing
Test assertions
Test suite
(test software, test scripts, test criteria)
Standard
Conformance clause
ebXML Conformance Issues
Conformance Clause should address?
1. What Needs to Conform?
2. Profiles and Levels
3. Extensions
4. Implementation Defined Features
5. Alternate approaches
Scope –what should be addressed?
a. Assertions
b. Test Suite
c. Testing Service
d. Certification
ebXML Issues -
Conformance Clause address:
1. What needs to conform
– identify the ‘class of products’ that will be
developed
– what are the conditions that need to be met in
order to claim conformance
– Examples:
registry, messaging service, business service
interface, ebXML message, testing partner agreements
ebXML Issues -
Conformance Clause address:
2. Profiles and Levels
– Are profiles and/or levels needed?
– Should a profile have its own levels?
– minimal requirements (core)
Profile is a subset of the overall specifications that includes all of the functionality
necessary to satisfy the requirements of a particular community of users.
Levels are nested subsets of the specifications. Level 1 = core that all must implement;
subsequent levels include lower level + added functionality
ebXML Issues -
Conformance Clause address:
3. Extensions
– If allowed:
• shall not re-define semantics of existing functions
• shall not cause standard-conforming functions to execute incorrectly
– Standardized method for extensions?
• Define the mechanism for extending the specification, so extended in
a standard manner
– Register of extensions
• document parallel to but distinct from specification
• list of recognized, published extensions
• may eventually migrate into future versions
ebXML Issues -
Conformance Clause address:
4. Implementation Defined Features
– may want to address implementation dependent
ranges, min/max allowed values
5. Alternative Approaches
– specification may describe several different ways
to accomplish an operation (e.g., choice of
bindings)
– specify whether or not all approaches need to be
implemented
ebXML Issues - Scope
Should ebXML include or address:
a. Assertions (test requirements)
- Should assertions be part of the standard?
b. Test Suite
- Prescribe a test methodology?
- How test suite will be delivered/used (e.g., web based,
downloadable)?
c. Testing Service
- Guidelines for establishing a test service
d. Certification
- Guidelines for issuing certificates
Testing Program - Considerations
How formal should testing be?
– Self testing, 3rd party testing
– Branding/certificates
Testing program
– Who will ‘own’ the testing program
– Policy and procedures
– Testing laboratory
– Issue Certificates
– Control board
– Test suite maintenance
TA Conformance Guidance
General conformance guidelines for ebXML
All ebXML specs shall contain a conformance
clause
– Can have levels of ebXML conformance -
hierarchical
– Explains how conformance requirements can be
expressed (mandatory, conditional, optional)
Encourage use of publicly available test
suites
Discussion