Intro to RSpec & Unit

Tests (Engineering Software as

a Service §8.1)
Armando Fox

© 2013 Armando Fox & David Patterson, all rights reserved

 Testing
can never
demonstrat
Debugging is
e the _____
absence
twice as hard
of errors in
as writing the
software,
code in the
only presence
their
first place.
_______
Therefore, if
you write the
code as
cleverly as
possible, you
are, by
definition, not
smart enough
to debug it.
 Testing Today
• Before
– developers finish code, some ad-hoc
testing
– “toss over the wall to Quality
Assurance [QA]”
– QA staff manually poke at software
• Today/Agile
– testing is part of every Agile iteration
– developers test their own code
– testing tools & processes highly
automated
 Testing Today
• Before
– developers finish code, some ad-hoc
testing
Software Quality is the result
– “toss over the wall to Quality
of a good
Assurance process, rather
[QA]”
than the responsibility
– QA people manually poke at of one
software
specific group
• Today/Agile
– testing is part of every Agile iteration
– developers responsible for testing own
code
– testing tools & processes highly
 BDD+TDD: The Big
Picture
• Behavior-driven design (BDD)
– develop user stories (the features you
wish you had) to describe how app will
work
– via Cucumber, user stories become
acceptance tests and integration tests
• Test-driven development (TDD)
– step definitions for new story,may
require new code to be written
– TDD says: write unit & functional tests
for that code first, before the code
 Cucumber & RSpec
• Cucumber
describes Failing (red) Cucumber
step
behavior via
features &
scenarios Failing (red) RSpec test

(behavior driven
design) Passing (green) RSpec test
• RSpec tests
individual
Passing (green)
modules that Cucumber step
contribute to
those behaviors 6
 Testing stacks revisited

Cucumber rspec-rails gem

Capybara RSpec

Selenium Rack::Test

Rack
browse
r
Rails app

web
server
SaaS Integration Unit & Funct.
app
Tests tests
Which are true about BDD & TDD:
a) requirements drive the
implementation
b) they can be used only in Agile
development
c)☐they
Onlyembrace
(a) & deal with change

☐ Only (a) & (b)

☐ Only (a) & (c)

☐ (a), (b) and (c)

8
END
9
FIRST, TDD, and Getting
Started With RSpec
(Engineering Software as a
Service §8.2)
Armando Fox

© 2013 Armando Fox & David Patterson, all rights reserved

 Unit tests should be
FIRST
• Fast

• Independent

• Repeatable

• Self-checking

• Timely
 Unit tests should be
FIRST
• Fast: run (subset of) tests quickly
(since you’ll be running them all the
time)
• Independent: no tests depend on
others, so can run any subset in
any order
• Repeatable: run N times, get same
result (to help isolate bugs and
enable automation)
• Self-checking: test can
 RSpec, a Domain-
Specific Language for
testing
• DSL: small programming language
that simpifies one task at expense
of generality
– examples so far: migrations, regexes,
SQL
• RSpec tests are called specs or
examples

• Run the tests in one file: rspec

filename
–
 RSpec Basics by
Example
x = Math.sqrt(9)
expect(x).to eq 3
expect(sqrt(9)).to
be_within(.5).of(3)

expect(x.odd?).to be_true
expect(x).to be_odd
expect(hash['key']).to be_truthy

m = Movie.new(:rating => 'R')

expect(m).to be_a_kind_of Movie
relishapp.com/rspec/rspec-expectations/docs/built-in-matchers
require 'ruby_intro.rb'

describe "BookInStock" do
it "should be defined" do
expect { BookInStock }.not_to raise_error
end

describe 'getters and setters' do

before(:each) { @book = BookInStock.new('isbn1', 33.8) }
it 'sets ISBN' do
expect(@book.isbn).to eq('isbn1')
end
it 'sets price' do
expect(@book.price).to eq(33.8)
end
it 'can change ISBN' do
@book.isbn = 'isbn2'
expect(@book.isbn).to eq('isbn2')
end
it 'can change price' do
@book.price = 300.0
expect(@book.price).to eq(300.0)
end
end
 More RSpec Basics by
Example
expect { m.save! }.
to raise_error(ActiveRecord::RecordInvalid)
m = (create a valid movie)
expect(m).to be_valid
expect { m.save! }.
to change { Movie.count }.by(1)

expect
{ lambda }.to(assertion)
expect(expression).to(assertio
Which kinds of code can be tested
Repeatably and Independently?
a) Code that relies on randomness (e.g.
shuffling a deck of cards)
b) Code that relies on time of day (e.g.
run backups every Sunday at midnight)
☐ Only (a)
☐ Only (b)

☐ Both (a) and (b)

☐ Neither (a) nor (b)

17
END
18
 RSpec on Rails
(Engineering Software as a
Service §8.2)
Armando Fox

© 2012 Armando Fox & David Patterson

Licensed under
Creative Commons Attribution-NonCommercial-ShareAl
ike 3.0 Unported License
 So what’s in rspec-rails?
• Additional methods mixed into
RSpec to test Rails-specific things
– e.g. get, post, put, … for controllers
– response object for controllers
• Matchers to test Rails apps’
behaviors
expect(response).to
render_template("movies/index")
• Support for creating various
doubles needed to test non-toy
 Example: calling TMDb
• New RottenPotatoes feature: add
movie using info from TMDb (vs.
typing in)
• How should user story steps behave?

When I fill in "Search Terms" with "Inception"

And I press "Search TMDb"
Then I expect to be on the RottenPotatoes homepage
...
Recall Rails Cookery #2:
adding new feature ==
 The Code You Wish You
Had
What should the controller method
do that receives the search form?
1. call a method that will search
TMDb for specified movie
2. if match found: select (new)
“Search Results” view to display
match
3. If no match found: redirect to
home page with message
http://pastebin.com/LcejGjYs
The method that contacts
TMDb to search for a movie
should be:

☐ A class method of the Movie

model
☐ An instance method of the Movie
model
☐ A controller method

☐ A helper method

23
cut
24
RSpec-Rails & TDD Cycle:
Red–Green–Refactor
(Engineering Software as a Service
§8.2-8.3)
Armando Fox

© 2013 Armando Fox & David Patterson, all rights reserved

 Test-First development
• Think about one thing the code
should do
• Capture that thought in a test, which
fails
• Write the simplest possible code
that lets the test pass
• Refactor: DRY out commonality
w/other tests
• Continue with next thing code
should do
 Preview: rspec-rails
• Additional methods mixed into
RSpec to test Rails-specific things
– e.g. get, post, put, … for controllers
– response object for controllers
• Matchers to test Rails apps’
behaviors
expect(response).to
render_template("movies/index")
• Support for creating various
doubles needed to test non-toy
 Example: calling TMDb
• New RottenPotatoes feature: add
movie using info from TMDb (vs.
typing in)
• How should user story steps behave?

When I fill in "Search Terms" with "Inception"

And I press "Search TMDb"
Then I expect to be on the RottenPotatoes homepage
...
Recall Rails Cookery #2:
adding new feature ==
The method that contacts
TMDb to search for a movie
should be:

☐ A class method of the Movie

model
☐ An instance method of the Movie
model
☐ A controller method

☐ A helper method

29
How to test something “in
isolation” if it has
dependencies that would
affect test?
 The Code You Wish You
Had
What should the controller method
do that receives the search form?
1. call a method that will search
TMDb for specified movie
2. if match found: select (new)
“Search Results” view to display
match
3. If no match found: redirect to
home page with message
http://pastebin.com/LcejGjYs
require 'rails_helper'

describe MoviesController do
describe 'searching TMDb' do
it 'calls the model method that performs TMDb search'
it 'selects the Search Results template for rendering'
it 'makes the TMDb search results available to that
template'
end
end
 TDD for the Controller
action: Setup
• Add a route to config/routes.rb
# Route that posts 'Search TMDb' form
post '/movies/search_tmdb'
– Convention over configuration will map
this to MoviesController#search_tmdb
• Create an empty view:
touch app/views/movies/search_tmdb.html.haml
• Replace fake “hardwired” method in
movies_controller.rb with empty
method:
def search_tmdb
 What model method?
• Calling TMDb is responsibility of the
model... but no model method exists to
do this yet!
• No problem...we’ll "fake" the call to the
code we wish we had (“CWWWH”),
Movie.find_in_tmdb
• Game plan:
– Simulate POSTing search form to controller
action.
– Check that controller action tries to call
Movie.find_in_tmdb with data from submitted
form. http://pastebin.com/kGXbxVq9
– The test will fail (red), because the (empty)
require 'rails_helper'

describe MoviesController do
describe 'searching TMDb' do
it 'calls the model method that performs TMDb search' do
expect(Movie).to
receive(:find_in_tmdb).with('hardware')
post :search_tmdb, {:search_terms => 'hardware'}
end
it 'selects the Search Results template for rendering'
it 'makes the TMDb search results available to that
template'
end
end
Which is FALSE about
expect(...).to_receive?

☐ It provides a stand-in for a real

method that doesn’t exist yet
☐ It would override the real method,
even if it did exist
☐ It can be issued either before or
after the code that should make
☐ the
It call
exploits Ruby's open classes
and metaprogramming to
“intercept” a method call at
testing time 36
cut
37
 Seams
(Engineering Software as a
Service §8.3)
Armando Fox

© 2013 Armando Fox & David Patterson, all rights reserved

 Seams
• A place where you can change app's
behavior without changing source code.
(Michael Feathers, Working Effectively With
Legacy Code)
• Useful for testing: isolate behavior of
some code from that of other code it
depends on.
• expect…to_receive uses Ruby’s open classes
to create a seam for isolating controller
action from behavior of (possibly buggy
or missing) Movie.find_in_tmdb
• Rspec resets all mocks & stubs after
 How to make this spec
green?
• Expectation says controller action
should call Movie.find_in_tmdb
• So, let’s call it!
def search_tmdb
Movie.find_in_tmdb(params[:search_terms])
end
The spec has driven the creation of
the controller method to pass the
test.
• Preview: But shouldn’t find_in_tmdb
return something?
 Test techniques we
know
expect(obj).to_receive(a).with(b)

Optional!
Eventually we will have to write a
real find_in_tmdb. When that
happens, we should:

☐ Replace the call to to_receive in our

test with a call to the real find_in_tmdb
☐ Ensure the API to the real find_in_tmdb
matches the fake one used by
☐
Keep the to_receive seam in the spec,
to_receive
but if necessary, change the spec to
match the API of the real find_in_tmdb
☐
Remove this spec (test case)
altogether since it isn’t really testing
anything anymore 42
cut
43
 Expectations
(Engineering Software as a
Service §8.4)
Armando Fox

© 2013 Armando Fox & David Patterson, all rights reserved

 Where we are & where
we’re going: “outside in”
development
• Focus: write expectations that
drive development of controller
method
– Discovered: must collaborate w/model
method
– Use outside-in recursively: stub model
method in this test, write it later
• Key idea: break dependency
between method under test & its
collaborators
• Key concept: seam—where you can
 The Code You Wish You
Had
What should the controller method
do that receives the search form?
1. call a method that will search
TMDb for specified movie
2. if match found: select (new)
“Search Results” view to display
match
3. If no match found: redirect to
home page with message
 The story so far
When I fill in "Search Terms" with "Inception"
And I press "Search TMDb"
• Resolved: the interaction with TMDb
is responsibility of a (fictitious) class
method, Movie.find_in_tmdb
• Resolved: controller action that
handles form submission should call
that method
• Achieved: controller spec that
verifies this by breaking dependency
using to_receive 47
 Inserting a seam for
find_in_tmdb
it 'calls the model method that performs TMDb search' do
expect(Movie).to
receive(:find_in_tmdb).with('hardware')
post :search_tmdb, {:search_terms => 'hardware'}
end
• Meanwhile back in the controller…
def search_tmdb
Movie.find_in_tmdb(params[:search_terms])
end
The spec has driven the creation of
the controller method to pass the
test.
• Shouldn’t find_in_tmdb return
Which of these, if any, is not
a valid expectation?

☐ expect(result).to_not be_empty

☐ expect(5).to be <=> result

☐ expect(result).to match /^D'oh!$/

☐ All of the above are valid

expectations
49
cut
50
 Expectations
continued (Engineering
Software as a Service §8.4)
Armando Fox

© 2013 Armando Fox & David Patterson, all rights reserved

 “it selects Search
Results view to display
match”
• Really 2 specs:
1.It renders Search Results view
– more important when different views
could be rendered depending on
outcome
2.It makes list of matches available
to view
• Basic expectation construct:
expect(obj).to match-condition
– Many built-in matchers, or define your
own
 Checking for rendering
• After post :search_tmdb, response()
method returns controller’s
response object
• render_template matcher can check
what view the controller tried to
render…
require 'rails_helper'

describe MoviesController do
describe 'searching TMDb' do
it 'calls the model method that performs TMDb search' do
fake_results = [mock('movie1'), mock('movie2')]
expect(Movie).to receive(:find_in_tmdb).with('hardware').
and_return(fake_results)
post :search_tmdb, {:search_terms => 'hardware'}
end
it 'selects the Search Results template for rendering' do
fake_results = [mock('Movie'), mock('Movie')]
Movie.stub(:find_in_tmdb).and_return(fake_results)
post :search_tmdb, {:search_terms => 'hardware'}
expect(response).to render_template('search_tmdb')
end
it 'makes the TMDb search results available to that template'
end
end
Note that this view has to exist!
post :search_tmdb will try to do whole MVC
flow, including rendering  functional (not
 Test techniques we
know
expect(obj).to_receive(a).with(b)

Optional!
 Test techniques we
know
expect(obj).to_receive(a).with(b)

expect(obj).to match-condition

Rails-specific extensions to RSpec:

response()
render_template()
to_receive combines _____ and
_____,
whereas stub is only _____.
A mock and an expectation;
☐ a mock
☐ A mock and an expectation;
an expectation
☐ A seam and an expectation;
an expectation
☐ A seam and an expectation;
a seam

57
cut
58
 Mocks & Doubles
(Engineering Software as a
Service §8.4)

© 2013 Armando Fox & David Patterson, all rights reserved

 It should make search
results available to
template
• Another rspec-rails addition:
assigns()
– pass symbol that names controller
instance variable
– returns value that controller assigned
to variable
• D’oh! our current code doesn’t set
any
@movie instance variables:
= Movie.find_in_tmdb(params[:search_terms])
def search_tmdb
Movie.find_in_tmdb(params[:search_terms])
end
• TCWWWH: list of matches in @movie
it 'makes search results available to template' do
@fake_results = [mock('Movie'), mock('Movie')]
Movie.stub(:find_in_tmdb).and_return(@fake_results)
post :search_tmdb, {:search_terms => 'hardware'}
expect assigns(:movies).to eq(@fake_results)
end
 New seam concept:
mock/double
• mock: “stunt double” object
– verify if a value got propagated
correctly
– need a “placeholder” value for code to
work, even though test doesn’t depend
on that value
– can even stub individual methods on a
double:
m=mock('movie1',:title=>'Rambo')

Goal: like all seams, enables just

     RSpec Cookery #1
• Each spec should test just one
behavior
• Use seams as needed to isolate
that behavior
• Determine what type of
expectation will check the behavior
• Write the test and make sure it fails
for the right reason
• Add code until test is green
• Look for opportunities to
 Unit vs. Functional tests
in SaaS apps
• Unit tests: behavior within a
method/class
– collaborator classes are mocked
– collaborator methods may be stubbed
out (in this class or collaborator
classes)
– both are sometimes generically called
doubles
• Functional test: behavior across
methods/classes (multiple classes
are exercised)
 Test techniques we
know
expect(obj).to_receive(a).with(b).and_return(c)
obj.stub(a).and_return(b)
Optional!
d = mock('impostor')

expect(obj).to match-condition

Rails-specific extensions to RSpec:

assigns(:instance_var)
response()
render_template()
to_receive combines _____ and
_____,
whereas stub is only _____.
A mock and an expectation;
☐ a mock
☐ A mock and an expectation;
an expectation
☐ A seam and an expectation;
an expectation
☐ A seam and an expectation;
a seam

66
cut
67
Fixtures and Factories
(Engineering Software as a
Service §8.5)
Armando Fox

© 2013 Armando Fox & David Patterson, all rights reserved

 Pitfall: mock trainwreck
• Goal: test searching for movie by its
director or by awards it received
expect(m.award.type).to eq 'Oscar'
expect(m.director.name).to eq 'Abrams'
• Mock setup:
a = mock('Award', :type => 'Oscar')
d = mock('Director',:name => 'JJ Abrams'
m = mock('Movie',
:award => a,
:director => d)
 When you need the real
thing
fake_movie = mock('Movie')
fake_movie.stub(:title).and_return('Casablanca')
fake_movie.stub(:rating).and_return('PG')
fake_movie.name_with_rating.should == 'Casablanca
(PG)'
Where to get a real object:
• Fixture: statically preload some
known data into database
tables
• Factory: create only what you
 Fixtures
• database wiped & reloaded with
fixtures before each spec
• Pros/uses
– truly static data, e.g. configuration info
that never changes
– easy to see all test data in one place
• Cons/reasons not to use
– may introduce dependency on fixture
data
 Fixtures—example
# in spec/fixtures/movies.yml
milk_movie:
id: 1
title: Milk
rating: R
release_date: 2008-11-26
documentary_movie:
id: 2
title: Food, Inc.
release_date: 2008-09-07

fixtures :movies
it 'finds movie by title' do
movie = movies(:milk_movie)
# etc.
end
end
 Factories
• Set up “helpers” to quickly create
objects with default attributes, as
needed per-test
• Pros/uses:
– Keep tests Independent: unaffected by
presence of objects they don’t care
about
• Cons/reasons not to use:
– Complex relationships may be hard to
set up (but may indicate too-tight
coupling in code!)
 Example: FactoryGirl
gem
# in spec/factories/movie.rb
FactoryGirl.define do
factory :movie do
title 'A Fake Title' # default values
rating 'PG'
release_date { 10.years.ago }
end
or create
end
# in your specs
it 'should include rating and year' do
movie = FactoryGirl.build(:movie, :title => 'Milk')
# etc.
end
Which of the following kinds
of data, if any, should not be
set up as fixtures?

☐ Movies and their ratings

☐ The TMDb API key

☐ The application’s time zone

settings
☐ Fixtures would be fine for all of
these

75
cut
76
 TDD for the Model
& Stubbing the Internet
(Engineering Software as a
Service §8.6–8.7)
Armando Fox

© 2013 Armando Fox & David Patterson, all rights reserved

 Explicit vs. implicit
requirements
• find_in_tmdb should call TmdbRuby
gem with title keywords
– If we had no gem: It should directly
submit a RESTful URI to remote TMDb
site
• What if TmdbRuby gem signals
error? (e.g. invalid API key)
 Where to stub in Service
Oriented Architecture?
movie.rb Movie.stub(:find_in_tmdb).
ruby-tmdb and_return(...) (or and_raise)
TmdbMovie.stub(:find).with(...).
Net::HTTP/Ope
Rule of thumb:
and_return(...)
nURI • for unit testing, stub
Net::HTTP.stub(:get).
with(...).
OS - TCP/IP nearby, for maximum
and_return(...)
isolation of class under
test (Fast, Independent)
Internet
• for integration testing,
stub far away, to test as
many interfaces as
Parse contrived request
TMDb possible
Return canned value(s)
TMDb-dev
Which statement(s) are
TRUE about Implicit
requirements?

☐ They are often, but not always,

derived from explicit
☐ requirements
They apply only to unit & functional
tests, not integration tests
☐ Testing them is lower priority than testing
explicit requirements, since they don’t come
from the customer
☐ All of the above are true

80
You’re integrating Stripe payments
with your SaaS app. They have a
“sandbox” server that sends canned
replies to specific requests. Which
kind(s) of tests should you create?
☐ Integration tests that can be run
against their sandbox server
☐ Integration tests that use Webmock
or similar to “stub out” the Internet
☐ Functional and/or unit tests that stub

payment functionality at the level of

theof
☐ All payment model
the above

81
cut
82
Coverage, Unit vs.
Integration Tests
(Engineering Software as a
Service §8.8)
Armando Fox

© 2013 Armando Fox & David Patterson, all rights reserved

 How much testing is
enough?
• Bad: “Until time to ship”
• A bit better: (Lines of test) / (Lines
of code)
– 1.2–1.5 not unreasonable
– often much higher for production
systems
• Better question: “How thorough is
my testing?”
– Formal methods
– Coverage measurement
–
 Measuring Coverage—
Basics
• S0: every method
class MyClass
called
def foo(x,y,z) • S1: every method
if x from every call site
if (y && z) then bar(0) end • C0: every statement
else
bar(1) – Ruby SimpleCov gem
end • C1: every branch in
end both directions
def bar(x) ; @w = x ; end
end
• C1+decision
coverage: every
subexpression in
conditional
• C2: every path
(difficult, and
 What kinds of tests?
• Unit (one e.g.
model Runs fast High
method/class) specs Fine resolution
coverage
Many mocks;
Doesn’t test interfaces

• Functional or eg,con-
troller
module (a few specs
methods/classes)

e.g. Few mocks;

Cuke
tests interfaces
• Integration/syste scena-
rios Runs slow
m Coarse resolution
Low coverage
 Going to extremes
דI kicked the tires, it works”
דDon’t ship until 100% covered &
green”
 use coverage to identify untested
or undertested parts of code
דFocus on unit tests, they’re more
thorough”
דFocus on integration tests, they’re
more realistic”
 each finds bugs the other misses
Which of these is POOR
advice for TDD?

☐Mock & stub early & often in

unit tests
☐ Aim for high unit test coverage

☐ Sometimes it’s OK to use

stubs & mocks in integration
☐ tests
Unit tests give you higher
confidence of system correctness
than integration tests 88
cut
89
Other Testing Concepts;
Testing vs. Debugging
(Engineering Software as a
Service §8.9, 8.12)
Armando Fox

© 2013 Armando Fox & David Patterson, all rights reserved

 Power RSpec: Don’t
change the database in
before(:all) !
• Each RSpec example is run in a
database transaction, rolled back
after example
– But before(:all) is outside the
transaction!
• Symptom: “fluttering specs” (pass,
then fail)
– or, specs pass when run alone but fail
when run in conjunction with other
specs (not Independent)
– rake db:test:prepare to clean DB, then
 Other testing terms you
may hear
• Mutation testing: if introduce
deliberate error in code, does some
test break?
• Fuzz testing: 10,000 monkeys throw
random input at your code
– Find ~20% MS bugs, crash ~25% Unix
utilities
– Tests app the way it wasn’t meant to
be used
• DU-coverage: is every pair <define
x/use x> executed?
 TDD vs. Conventional
debugging
Conventional TDD
Write 10s of lines, run, hit bug: Write a few lines, with test first;
break out debugger know immediately if broken
Insert printf’s to print variables Test short pieces of code using
while running repeatedly expectations
Stop in debugger, tweak/set Use mocks and stubs to control
variables to control code path code path
Dammit, I thought for sure I Re-run test automatically
fixed it, now I have to do this
•all Lesson
again 1: TDD uses same skills &
techniques as conventional debugging—but
more productive (FIRST)

• Lesson 2: writing tests before code takes

more time up-front, but often less time
 TDD Summary
• Red – Green – Refactor, and always
have working code
• Test one behavior at a time, using
seams
• Use it “placeholders” or pending to
note tests you know you’ll need
• Read & understand coverage
reports
• “Defense in depth”: don’t rely too
heavily on any one kind of test
cut
95
And now a word from
our sponsor:
TDD vs. Debugging
Armando Fox
and Your Peers from
Previous Offerings of CS
169
© 2013 Armando Fox & David Patterson, all rights reserved
 TDD vs. Debugging
Conventional TDD
Write 10s of lines, run, hit bug: Write a few lines, with test first;
break out debugger know immediately if broken
Insert printf’s to print variables Test short pieces of code using
while running repeatedly expectations
Stop in debugger, tweak/set Use mocks and stubs to control
variables to control code path code path
Dammit, I thought for sure I Re-run test automatically
fixed it, now I have to do this
•all Lesson
again 1: TDD uses same skills &
techniques as conventional debugging—but
more productive (FIRST)
• Lesson 2: writing tests before code takes
more time up-front, but often less time
overall
 4. TDD & Testing
• “TDD is hard at first, but it gets
easier”
• “Was great for quickly noticing
bugs [regression] and made them
easy to fix”
• “Helped me organize my thoughts
as well as my code” [The code you
wish you had]
• “Wish we had committed to it
earlier & more aggressively”
 5. Test coverage, code
quality
• “Good coverage gave us confidence
we weren’t breaking stuff when we
deployed new code”
• “Felt great to get high grade from
CodeClimate”
• “Pull-request model for constant
code reviews made our code
quality high”
• “Wish we had committed to TDD +
coverage measurement earlier”
Which non-obvious
statement about testing is
FALSE?

☐ Even 100% test coverage is not a

guarantee of being bug-free
☐ If you can stimulate a bug-causing
condition in a debugger, you can
☐ capture it in a test the need to
Testing eliminates
use a debugger
☐ When you change your code, you need
to change your tests as well

100
cut
101
 Plan-And-Document
Perspective on Software
Testing
(Engineering Software as a Service §8.10)
David Patterson

© 2013 Armando Fox & David Patterson, all rights reserved

102
 P&D Testing?
• BDD/TDD writes tests before code
– When do P&D developers write tests?
• BDD/TDD starts from user stories
– Where do P&D developers start?
• BDD/TDD developers write tests &
code
– Does P&D use same or different
people for testing and coding?
• What does the Testing Plan and
Testing Documentation look like? 103
 P&D Project Manager
• P&D depends on Project
Managers
• Document project
management plan
• Creates Software
Requirements Specification
(SRS)
– Can be 100s of pages
– IEEE standard to follow
• Must document Test Plan
104
 P&D Approach to Testing
• Manager divides SRS into
programming units
• Developers code units
• Developers perform unit
testing
• Separate Quality Assurance
(QA) team does higher level
tests:
– Module, Integration, System,
Acceptance
105
 3 QA Integration Options
1. Top-down integration
– Starts top of dependency graph
– High-level functions (UI) work
soon
– Many stubs to get app to
“work”
2. Bottom-up integration
– Start bottom of dependency
graph
– No stubs, integrate everything
in a module 106
 3 Integration Options
3. Sandwich integration
– Best of both worlds?
– Reduce stubs by
integrating some units
bottom up
– Try to get UI
operational by
integrating some units
top down

107
 QA Team Testing
• Next QA Team does System Test
– Full app should work
– Test non-functional requirements
(performance) + functional
requirements (features in SRS)
• When P&D System Testing Done?
– Organization policy
• Eg, test coverage level (all statements)
• Eg, all inputs tested with good and bad
data
• Final step: Customer or User 108
 Limits of Testing
• Program testing can be used to
show the presence of bugs, but
never to show their absence!
– Edsger W. Dijkstra

(received the 1972 Turing Award for

fundamental contributions to
developing programming languages)

(Photo by Hamilton Richards. Used

by permission under CC-BY-SA-3.0.)

109
 Formal Methods
• Start with formal specification
& prove program behavior
follows spec.
1. Human does proof
2. Computer via automatic
theorem proving
– Uses inference + logical axioms
to produce proofs from scratch
3. Computer via model checking
– Verifies selected properties by
exhaustive search of all possible 110
 Formal Methods
• Computationally expensive, so
use
– Small, fixed function
– Expensive to repair, very hard to
test
– Eg. Network protocols, safety
critical SW
• Biggest: OS kernel
10K LOC @ $500/LOC
– NASA SW $80/LOC
• This course: rapidly changing 111
SW Testing: P&D vs.
Agile
(Fig. 8.26)

112
Which statement regarding testing is
FALSE?

1. Formal methods are expensive but worthwhile

to verify important applications
2. P&D developers code before they write tests while
its vice versa for Agile developers

3. Agile developers perform module, integration,

system, & acceptance tests; P&D developers don’t
4. Sandwich integration in P&D aims to reduce
effort making stubs while trying to get general
functionality early
113
END
114