Software Testing

Testing Approaches (Strategies)

What is the goal of testing?

(Finding errors.)

Assess quality

Uncover errors

Ensure standards are met

"You can't test-in quality"

(Another sad story)

The story of The Tandy Computer Back Up

Tandy Company used to manufacture computer systems,

and provided system software (They made an OS, utilities,

and software to backup the disk)

The backup software would backup (copy) files to floppy

disks and cost $129.99

The box for the software clearly reminded users to back up

their files often, to avoid any data loss…

…

Testing is an important part of Software Quality

But not all:

Also included are:

Formal Technical Reviews

Software Engineering Methods

Standards and Procedures

Configuration Management and SQA

Metrics

All testing follows:

Begins at the module (or object) level and

works upwards

Different techniques are appropriate at different times

Testing is done by developer and independent group

Testing is not debugging

Validation and Verification

Verify: Building product right

Validate: Building right product

Boehm

There is a conflict when producers test their own product

They don't want to find errors

"Independent" Test Group

Fallacies:

Developers do no testing

Software test is hidden from developers

Testers start testing when code complete

(or schedule tells them to)

Software testing follows lifecycle (backwards)

Unit Test

Focus on implementation - source code

Heavy use of white box testing

Integration Testing

Design Oriented

Black Box mostly, some White Box

Validation Testing

Requirements Oriented

Black Box

Does system meet: functional, behavior,

System Test

(Not really SE)

Does overall system work

When is testing finished?

The question that all managers, developers

"When the schedule says so"

When we run out of time, when there are no more

bugs

An SE answer

Musa and Ackerman:

Can not be certain software does not fail but based

The logarithmic Poisson execution time model:

f(t) = (1/p) ln(l₀ pt + 1)

f(t) is cumulative failures over time t

l₀ is failures per unit time (failure intensity)

p is exponential reduction in intensity as repairs are

Instantaneous failure intensity, derivative of f(t)

l(t) = l₀ (l₀ pt + 1)

Can use this to predict error rate drop off

What is error rate increases

Gilb says success is:

Specify quantifiable product requirements before

testing

State testing objectives explicitly

Understand users

Rapid Cycle Testing - fast feedback

Build robust software

FTR

Continuous improvement of testing process

Testing:

Units:

Interface:

Parameters match? Units match?

Number and order of parameters?

Files:

Open, close OK? Buffer sizes,

formats? EOF?

Data Structures:

Typing? Initialization or default?

Under/Over flow?

Drivers and stubs written

Integration Test:

Top Down

Depth first

Breadth first

Bottom Up

Atomic modules built to clusters

Regression testing

Use test suites

Validation testing:

What are "reasonable" expectations

Needs to be documented!

Configuration reviews (Audits)

Alpha test

Developers site by customer

Beta test

Customer site by end users

System tests:

Recover from faults

Security

Stress (over stressed)

Performance

Debugging

Brute force

printf

Backtrack

know where fault is, inspect code

Reason it out (cause elimination)

divide and conquer

IBM:

Mills found removing 60% of defects would

lead to 3% reliability improvement

Terms:

Failure - when executing

Fault - static, built in

Probability of Failure on Demand

Prob. system fails when request is made

1. 1 in 1000 service requests on avg. fail

Rate of failure occurrence

frequency of unexpected behavior

2/100, 2 times per 100 time units

MTTF - Mean time to Fail

Availability

0.96 in 100 hours available 96 hours

Failures:

Transient (only some input)

Permanent

Recoverable (no operator needed)

Corrupting (system state or data)

Reliability:

Fault Avoidance

Fault Tolerance

Fault Detection