CSE 5324: Software Engineering I
(Analysis, Design, Creation)
Fall 1997
Software Testing
Testing Approaches (Strategies)
What is the goal of testing?
(Finding errors.)
Assess quality
Uncover errors
Ensure standards are met
"You can't test-in quality"
(Another sad story)
The story of The Tandy Computer Back Up
Tandy Company used to manufacture computer systems,
and provided system software (They made an OS, utilities,
and software to backup the disk)
The backup software would backup (copy) files to floppy
disks and cost $129.99
The box for the software clearly reminded users to back up
their files often, to avoid any data loss…
…
Testing is an important part of Software Quality
But not all:
Also included are:
Formal Technical Reviews
Software Engineering Methods
Standards and Procedures
Configuration Management and SQA
Metrics
All testing follows:
Begins at the module (or object) level and
works upwards
Different techniques are appropriate at different times
Testing is done by developer and independent group
Testing is not debugging
Validation and Verification
Verify: Building product right
Validate: Building right product
Boehm
There is a conflict when producers test their own product
They don't want to find errors
"Independent" Test Group
Fallacies:
Developers do no testing
Software test is hidden from developers
Testers start testing when code complete
(or schedule tells them to)
Software testing follows lifecycle (backwards)
Unit Test
Focus on implementation - source code
Heavy use of white box testing
Integration Testing
Design Oriented
Black Box mostly, some White Box
Validation Testing
Requirements Oriented
Black Box
Does system meet: functional, behavior,
performance requirements
System Test
(Not really SE)
Does overall system work
When is testing finished?
The question that all managers, developers
and testers ask
"When the schedule says so"
When we run out of time, when there are no more
bugs
An SE answer
Musa and Ackerman:
Can not be certain software does not fail but based
on a model with 98 % confidence the probability of
one year of failure-free operation is probabilistically
0.95
The logarithmic Poisson execution time model:
f(t) = (1/p) ln(l0 pt + 1)
f(t) is cumulative failures over time t
l0 is failures per unit time (failure intensity)
p is exponential reduction in intensity as repairs are
made
Instantaneous failure intensity, derivative of f(t)
l(t) = l0 (l0 pt + 1)
Can use this to predict error rate drop off
What is error rate increases
Gilb says success is:
Specify quantifiable product requirements before
testing
State testing objectives explicitly
Understand users
Rapid Cycle Testing - fast feedback
Build robust software
FTR
Continuous improvement of testing process
Testing:
Units:
Interface:
Parameters match? Units match?
Number and order of parmaters?
Files:
Open, close OK? Buffer sizes,
formats? EOF?
Data Structures:
Typing? Initialization or default?
Under/Over flow?
Drivers and stubs written
Integration Test:
Top Down
Depth first
Breadth first
Bottom Up
Atomic modules built to clusters
Regression testing
Use test suites
Validation testing:
What are "reasonable" expectations
Needs to be documented!
Configuration reviews (Audits)
Alpha test
Developers site by customer
Beta test
Customer site by end users
System tests:
Recover from faults
Security
Stress (over stressed)
Performance
Debugging
Brute force
printf
Backtrack
know where fault is, inspect code
Reason it out (cause elimination)
divide and conquer
IBM:
Mills found removing 60% of defects would
lead to 3% reliability improvement
Terms:
Failure - when executing
Fault - static, built in
Probability of Failure on Demand
Prob. system fails when request is made
When useful?
Rate of failure occurrence
frequency of unexpected behaviour
2/100, 2 times per 100 time units
MTTF - Mean time to Fail
Availability
0.96 in 100 hours available 96 hours
Failures:
Transient (only some input)
Permanent
Recoverable (no operator needed)
Corrupting (system state or data)
Reliability:
Fault Avoidance
Fault Tolerance
Fault Detection