Académique Documents
Professionnel Documents
Culture Documents
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Next step after requirements documentation Process
Inspection Guidelines
Natural Language
Result of this phase is a consolidated requirements document. Verification
Inspection Checklists
Automatic Assessment
11.2
Example: L4.verified
Example: L4.verified
Decision Table Verification
11.3
Requirements Quality Assurance Ren Witte
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.4
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.5
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
From Ferguson & Lami, Automated Natural Language Analysis of Requirements, http://www.incose.org/delvalley/data/INCOSE- preview- QuARS_21June05.ppt
11.6
Overview Ren Witte
Key Points
Requirements
Inspections and
Reviews
Techniques for Requirements Quality Assurance Requirements Inspection
Process
Inspection Guidelines
Inspections and Reviews: Auditing of the specification with a dedicated inspection Verification of Z
Animation-Based Validation: Visualize (semi-)formal requirements using a tool Challenges with Formal
Methods
Formal Verification: Check consistency, completeness, invariant properties, etc. Decision Table Verification
Natural Language
using formal methods Verification
Inspection Checklists
Automatic Assessment
11.7
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Copyright 2009 by John Wiley & Sons Ltd, [vL09]
Inspection Guidelines
Verification of Z
Specifications
Checklist based, using given list of questions and issues Notes and Further
Reading
Process based, using roles such as domain expert or developer
11.8
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
11.9
Inspection Guidelines Ren Witte
Key Points
Requirements
Inspections and
Follow the WHATWHOWHENWHERE guidelines Reviews
Requirements Inspection
Process
WHAT? Report should contain only facts, not opinions. Constructive, not Inspection Guidelines
WHO? Inspectors independent from authors. Include different stakeholders Model Checking and
Theorem Proving
(e.g., domain expert, developer, end user). May include QA specialist Consistency checks
WHEN? Not too early (RS still under development); not too late (design & Challenges with Formal
Methods
Natural Language
WHERE? Critical system aspects (e.g., safety/security), places with already Verification
Inspection Checklists
detected defects (and areas impacting/impacted by them). Automatic Assessment
11.10
Key Points
Requirements
Inspections and
1 Requirements Inspections and Reviews Reviews
Requirements Inspection
Process
Inspection Guidelines
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.11
Key Points
If this variable is declared as a partial function in this data schema, is there an Verification of Z
Specifications
invariant in the schema to specify the input domain where this function is Model Checking and
Natural Language
Is there a corresponding exception schema for the case where this Verification
Inspection Checklists
11.12
Verification of Z Specifications (II) Ren Witte
Key Points
Requirements
Inspections and
Reviews
Syntax Checking Requirements Inspection
Process
Inspection Guidelines
AddBirthday Verification of Z
Specifications
BirthdayBook Model Checking and
Natural Language
Verification
(declaration symbols cannot appear in unquantified predicates) Inspection Checklists
Automatic Assessment
11.13
Key Points
Requirements
Inspections and
Type Checking Reviews
Requirements Inspection
Consider the post-conditions: Process
Inspection Guidelines
BorrowedBy : BookCopy
7 Person Example: L4.verified
Decision Table Verification
Natural Language
Verification
Errors? Inspection Checklists
11.14
Key Points
Requirements
Inspections and
Static Semantics Checking Reviews
Requirements Inspection
Process
Verification of Z
each variable must have specified initial value Specifications
can be automatically checked through Zs import and initialization mechanisms Validity Checks
Inference
Challenges with Formal
Circularity Checking
Methods
Tool Support
11.15
Tool Support Ren Witte
Requirements
formatting Z specifications (using LATEX) Inspections and
Reviews
Verification of Z
Input Example Specifications
Natural Language
\begin{schema}{Enrol} Verification
Inspection Checklists
\Delta Club \\ Automatic Assessment
new?: PERSON
Notes and Further
\where Reading
members = members \cup new? \\
president = president
\end{schema}
11.16
Requirements
Inspections and
A club has a set of members and a president, who is one of the members: Reviews
Requirements Inspection
Process
Verification of Z
members : P PERSON Specifications
To enroll somebody in the club, we just add them to the set of members: Tool Support
Example: L4.verified
Decision Table Verification
Key Points
Errors in Example Schemas Requirements
Inspections and
The fuzz type checker will identify a number of issues: Reviews
Requirements Inspection
Process
11.18
Report of defined names and types Ren Witte
members: P PERSON
president: PERSON
new?: PERSON
End
11.19
Requirements
Inspections and
2 Verification of Z Specifications
Reviews
Requirements Inspection
Process
Inspection Guidelines
Example: L4.verified
Example: L4.verified
Decision Table Verification
11.20
Key Points
interpretation I such that valI (p) = T . The satisfying interpretation I is Challenges with Formal
Methods
Validity: A formula p is valid, written |= p, if it is true (T ) in all interpretations Decision Table Verification
Natural Language
(i.e., p is a tautology). Verification
Inspection Checklists
Unsatisfiability: A formula p is unsatisfiable (or contradictory) if it is false (F ) in all Automatic Assessment
11.21
Application to RE? Ren Witte
Key Points
Requirements
Inspections and
Consistency check Reviews
Requirements Inspection
Are the requirements consistent? Check set of all requirements Process
Inspection Guidelines
S = {r1 , r2 , . . . , rn } = r1 r2 . . . rn (often also called the knowledge base): Verification of Z
if we can find a model I such that I |= S, our requirements are consistent Specifications
Practice: Use the methods discussed in the lecture on requirements evaluation Decision Table Verification
Natural Language
Theory: Apply belief revision to find minimal change needed to restore Verification
Inspection Checklists
consistency in the knowledge base Automatic Assessment
11.22
Often, we need to check if a certain assertion logically follows from a set of Requirements
Inspections and
requirements. Reviews
Requirements Inspection
E.g., from the collected requirements S we want to show that whenever the
Process
Inspection Guidelines
|= S Tool Support
Example: L4.verified
Decision Table Verification
11.23
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
can give important insights into the modeled domain Validity Checks
Inference
check with stakeholders (domain experts) if this is what they expect (if Challenges with Formal
Methods
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.24
Challenge: Lack of World Knowledge Ren Witte
Key Points
Requirements
Inspections and
Common Problems Reviews
generated
Inspection Guidelines
Verification of Z
not all of them will be insightful Specifications
stored in a book
Inference
Challenges with Formal
but the theorem prover does not have such common sense knowledge, which can Methods
Tool Support
lead to spurious inconsistencies and irrelevant conclusions Example: L4.verified
Decision Table Verification
Encoding all this world knowledge is usually not feasible (in AI/Expert Systems this
Natural Language
is known as the knowledge acquisition bottleneck) Verification
Inspection Checklists
Requires requirements engineer trained in formal methods Automatic Assessment
11.25
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.26
Small logic core, can be used for propositional, predicate, higher-order logics Key Points
(HOL) Requirements
Inspections and
Reviews
Free/open source software (BSD license) Requirements Inspection
Process
Verification of Z
Developed at University of Cambridge, Technical University Munich, and Specifications
Has been used for many software verification projects Consistency checks
Validity Checks
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.27
Isabelle/jEdit (Modern Version) Ren Witte
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.28
Copyright 2013 Gottfried Barrow, Isabelle Mailing List
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.29
specification Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.30
The L4.verified project Ren Witte
Key Points
Some statistics (2009) Requirements
Inspections and
From http://ertos.nicta.com.au/research/l4.verified/ Reviews
Requirements Inspection
Code Size: 8,700 lines of C code plus 600 lines of ARM assembly code. Process
Inspection Guidelines
Verified Code Base: 7,500 lines of C code (excluding assembly code and boot Verification of Z
Specifications
loader) Model Checking and
Authors estimate doing it again now would cost about 10 person Challenges with Formal
Methods
Cost Comparison: Industry rule of thumb: software certification in the Common Decision Table Verification
Natural Language
Criteria process at Evaluation Level 6 is $10,000 per line of code Verification
Detected Bugs: 160 bugs in the C code in total, of those 144 found with formal Notes and Further
Reading
verification
11.31
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.32
Copyright 2009 NICTA, http://ertos.nicta.com.au/research/l4.verified/visual.pml
Key Points
Requirements
QA for RS using decision tables Inspections and
Reviews
Count number of columns/conditions: for N entry conditions, there must be 2N Requirements Inspection
Process
Verification of Z
Otherwise table is incomplete Specifications
Natural Language
Verification
Full braking activated X X Inspection Checklists
Automatic Assessment
Alarm generated to station computer X X X X Notes and Further
Reading
11.33
Decision Table Check (II) Ren Witte
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
Inconsistencies Verification of Z
Specifications
Two decision table entries are inconsistent if they define different actions for the
Model Checking and
same input conditions. Theorem Proving
Consistency checks
Validity Checks
Redundancies Inference
Two decision table entries are redundant if two columns have the same input- and
Challenges with Formal
Methods
Tool Support
output conditions. Example: L4.verified
Decision Table Verification
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.34
Example: L4.verified
Example: L4.verified
Decision Table Verification
11.35
Key Points
Requirements
Check for Defects in RD Inspections and
Reviews
Contradiction Statement consistent with the system objectives and Consistency checks
Validity Checks
constraints? Inference
Challenges with Formal
Statement consistent with the other related statements? Methods
Tool Support
Inadequacy Does this RD item formulate what stakeholders really expect? Example: L4.verified
Decision Table Verification
Ambiguity Can statement be interpreted differently (readers from different Natural Language
Verification
context/background)? Inspection Checklists
Automatic Assessment
Other statements using same terms with different meanings?
Notes and Further
Reading
11.36
Defect-Based Checklists (contd.) Ren Witte
Key Points
Requirements
Inspections and
Reviews
Immeasurability Is there a fit criterion associated with this quality requirement? Verification of Z
Specifications
Is the fit criterion measurable (quantities, protocol)? Model Checking and
Can test data be derived from this statement? Theorem Proving
Consistency checks
Overspecification Does this statement entail premature design choice? Decision Table Verification
11.37
Key Points
Requirements
Inspections and
Reviews
Poor Structuring Are statement structured in an apparent way? Challenges with Formal
Methods
Does an RD item mix requirements and domain properties? Decision Table Verification
11.38
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Check for Defects in RD (IV) Inspection Guidelines
Verification of Z
Forward Reference Is this concept, so far undefined, defined somewhere later? Specifications
Poor Modifiability Would any change to a single RD item require changes Validity Checks
Inference
throughout the RS? Challenges with Formal
Methods
11.39
Other Checklists Ren Witte
Key Points
Requirements
Inspections and
Reviews
Quality-specific checklists Requirements Inspection
Based on the specification language used (UML, templates, etc.). See [LW03, Natural Language
Verification
Chapter 29] for a number of checklists for UP artifacts. Inspection Checklists
Automatic Assessment
11.40
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
Anda, Bente, and Dag IK Sjberg. Towards an inspection technique for use case models. In Proceedings of the 14th international conference on Software engineering and knowledge engineering, pp. 127-134. ACM, 2002.
11.41
Key Points
Requirements
1 Requirements Inspections and Reviews Inspections and
Reviews
Requirements Inspection
Process
Verification of Z
Specifications
Natural Language
Verification
5 Notes and Further Reading Inspection Checklists
Automatic Assessment
11.42
Requirements in Practice Ren Witte
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Formal Inspection Guidelines
Specifications 10% Verification of Z
Specifications
Issues Model Checking and
Theorem Proving
Natural Language SRS are Consistency checks
Validity Checks
fragile & difficult to maintain Inference
Challenges with Formal
Natural Language
Verification
Inspection Checklists
Industry distribution of SRS [MFI04] Automatic Assessment
11.43
Verification of Z
hence costly (not very efficient) Specifications
11.44
Key Points
Requirements
Definition Inspections and
Reviews
Requirements Inspection
11.45
NASA ARM Tool Ren Witte
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.46
Verification of Z
options (can, may, optionally, . . .) Specifications
Natural Language
subjects (words preceding imperatives) Verification
Inspection Checklists
11.47
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Verification of Z
Designed to measure the comprehension and retention of text. Indicates the Specifications
number of years of formal education a reader would need to understand a block of Model Checking and
Theorem Proving
text. Consistency checks
Validity Checks
Inference
Originally developed to improve the readability of U.S. Navy technical documents, Tool Support
Example: L4.verified
the Kincaid metric is designed to measure the persistence and efficiency of text. Decision Table Verification
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.48
Readability Formulas Ren Witte
Key Points
Fog Requirements
Inspections and
Reviews
Verification of Z
Where: Specifications
ASL = Average sentence length using number of words. Model Checking and
Theorem Proving
HW = Number of words with more than two syllables Consistency checks
Validity Checks
Inference
Flesch-Kincaid Challenges with Formal
Methods
Tool Support
Natural Language
Where: Verification
Inspection Checklists
ASL = Average sentence length using number of words. Automatic Assessment
11.49
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.50
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.51
[William M. Wilson, Linda H. Rosenberg, Lawrence E. Hyatt, Automated Analysis of Requirement Specifications,, 19th International Conference on Software Engineering (ICSE97), 1997]
Semantic Assistants Ren Witte
Requirements
Services such as readability analysis are executed through a Web service Inspections and
Reviews
Results are displayed through client plug-ins (e.g., in OpenOffice, Eclipse, Requirements Inspection
Process
Verification of Z
New clients and services can be added and improved independently Specifications
...
NLP Service n
Notes and Further
Reading
Word Processor
NLP Service
Result
11.52
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.53
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.54
User Interface: Wiki-NLP Plug-in Ren Witte
Key Points
Requirements
Inspections and
Reviews
Input Text
Requirements Inspection
Process
Inspection Guidelines
Annotated
NLP results Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.55
NLP Pipelines
Each service corresponds to an NLP Pipeline Key Points
http://gate.ac.uk Process
Inspection Guidelines
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.56
Part-of-Speech Tagging
Add part-of-speech information (noun, verb, determiner, . . .)
Key Points
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.57
SA Service: Writing Quality Assessment Ren Witte
Key Points
Requirements
Features Inspections and
Reviews
Which parties (system, partner-application or stakeholders) are responsible for the Notes and Further
Reading
actions?
11.58
Key Points
Requirements
Features Inspections and
Reviews
Requirements Inspection
Measures readability of text through Flesch & Kincaid metrics Process
Inspection Guidelines
The system shall configure the frame-rate as specified by the camera provided Validity Checks
Inference
during initialization & the frame-rate shall be capable of being automatically Challenges with Formal
Methods
11.59
Key Points
Requirements
Features Inspections and
Reviews
Based on NASA ARM requirements metrics Requirements Inspection
Process
Verification of Z
Option Defects give implementer latitude Specifications
The system may approve customer & supplier requests in a timely manner" Example: L4.verified
Decision Table Verification
Natural Language
Rationale Verification
Inspection Checklists
may is ambiguous. Is request approval mandatory? For all stakeholders? Automatic Assessment
11.60
ReqSAC (Requirements Specification Ambiguity Checker) Ren Witte
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Automatic Ambiguity Detection Process
Inspection Guidelines
MSc project at Concordia (I. Hussain [HOK07]): Verification of Z
Specifications
Find ambiguous requirements sentences using a classifier
Model Checking and
Based on machine learning: Theorem Proving
Consistency checks
Natural Language
(Note: Not yet integrated into SA/ReqWiki.) Verification
Inspection Checklists
Automatic Assessment
11.61
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.62
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
Verification of Z
Specifications
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.63
Automatic Natural Language QA: Conclusions Ren Witte
Key Points
Instead of performing only manual checks, add automated support: Inspection Guidelines
Verification of Z
needs technology from Natural Language Processing (NLP), Text Mining, and Specifications
Will not replace inspection by a human requirements engineer (false positives, Challenges with Formal
Methods
But can significantly reduce workload and allow humans to focus on the difficult
Decision Table Verification
Natural Language
issues Verification
Inspection Checklists
Prediction: will become standard tool in the next 35 years (comparable to spell Automatic Assessment
11.64
Example: L4.verified
Example: L4.verified
Decision Table Verification
11.65
Verification of Z
Supplemental Specifications
11.66
Tools Ren Witte
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
Process
Inspection Guidelines
Tool References
Verification of Z
Specifications
The fuzz type checker for Z, http://spivey.oriel.ox.ac.uk/mike/fuzz/
Model Checking and
The Isabelle Interactive Theorem Prover, Theorem Proving
Consistency checks
Natural Language
Verification
Inspection Checklists
Automatic Assessment
11.67
Key Points
[CL13] Nathan Carlson and Phil Laplante. Requirements
The NASA automated requirements measurement tool: a Inspections and
Reviews
reconstruction. Requirements Inspection
Process
Innovations in Systems and Software Engineering, pages 115, 2013. Inspection Guidelines
http://dx.doi.org/10.1007/s11334-013-0225-8. Verification of Z
Specifications
[HOK07] Ishrar Hussain, Olga Ormandjieva, and Leila Kosseim. Model Checking and
Theorem Proving
Computer Society.
Decision Table Verification
Natural Language
http://users.encs.concordia.ca/~kosseim/Publications/2007QSIC.pdf. Verification
Inspection Checklists
11.68
[MFI04] Luisa Mich, Mariangela Franch, and Pierluigi Inverardi. Inspection Guidelines
[SARW12] Bahar Sateli, Elian Angius, Srinivasan Sembakkam Rajivelu, and Ren Natural Language
Verification
Witte. Inspection Checklists
Key Points
Requirements
Inspections and
Reviews
Requirements Inspection
[vL09] Axel van Lamsweerde. Process
Inspection Guidelines
Requirements Engineering: From System Goals to UML Models to Verification of Z
Software Specifications. Specifications
[WRH97] William M. Wilson, Linda H. Rosenberg, and Lawrence E. Hyatt. Validity Checks
Inference
Automated analysis of requirement specifications. Challenges with Formal
Methods
11.70