Chapter 8 - Software Quality Assurance

 
Quality

• Difficult to define characteristic.
• Two categories for software
• Quality of design
• Requirements specification
• Analysis
• Design
• Quality of conformance
• Implementation
• Coding
• Testing
• Documentation
• Maintenance
• Quality control is achieved via a series of inspections, reviews, tests, throughout the development lifecycle.
• Quality assurance consists of the auditing and reporting functions of management.

• Prevention Costs : cost of planning, formal reviews, test equipment, test procedures, test software, training.
• Appraisal Costs: cost of inspections, testing
• Failure Costs: cost of rework (program rewrites), repair (bug fixing)
• External Failure Costs: cost of complaint resolution, product return/replacement, help lines, warranty support, lost sales, ruined reputation.
• The earlier in the lifecycle that the problem is resolved, the less the cost. (reference figure 8-1).

HUMOR. Thanks to the unknown author of these "Top 20 Replies by Programmers when Their Programs Do Not Work."
 

20. "That's weird..."
19. "It's never done that before."
18. "It worked yesterday."
17. "How is that possible?"
16. "It must be a hardware problem."
15. "What did you type in wrong to get it to crash?"
14. "There's something funky in your data."
13. "I haven't touched that module in weeks!"
12. "You must have the wrong version."
11. "It's just some unlucky coincidence."
10. "I can't test everything!"
 9. "THIS can't be the source of THAT."
 8. "It works, but it hasn't been tested."
 7. "Somebody must have changed my code."
 6. "Did you check for a virus on your system?"
 5. "Even though it doesn't work, how does it feel?"
 4. "You can't use that version on your system."
 3. "Why do you want to do it that way?"
 2. "Where were you when the program blew up?"
 1. "It works on my machine."

Design and Code Reviews

• Goals of Formal Technical Reviews:
• Uncover errors in function, logic, or implementation
• Verify requirements are being met
• Ensure design artifacts and code meet standards
• Achieve uniform development of code and designs (style, documentation)
• Make projects manageable
• Aid in communication among and between teams, serve as a training mechanism
• Review meeting concepts
• limit size to the individuals that created the item and 3-5  outside reviewers (members of QA department, members of other software teams on the same project)
• limit the amount of advance preparation of reviewers (or they won't do it)
• keep the review to under two hours
• have a set agenda and start on time
• agenda should include a checklist of things to be reviewed and detail of each item
• review the product not the producer (no personal comments)
• limit debate and rebuttal, instead, assign action items for investigation
• enunciate problems and list action times
• assign action items to individuals, and assign a completion date
• take written notes
• meeting usually chaired by a person from the QA department
• Format of Review Meeting
• Line by line
• each line/page of artifact is reviewed for comments. requires extensive preparation by reviewers.
• Presentation
• artifact is presented in by creators, explaining how it is to work, how it interfaces with other components.

• From Microsoft Advanced VB 5 Online Library Reference
• Has the design requirement been met?
• Does it conform to in-house development standards?
• Variable and procedure names
• Commenting style
• Does the code check for invalid or unreasonable parameters?
• Are all handles to resources being closed properly?
• If a routine has an early Exit subroutine or function call, is everything tidied up before it leaves?
• Are all function return codes being checked? If not, what is the point of the function being a function instead of a subroutine?
• Is the code commented sufficiently?
• Are Debug.Assert statements used to their best advantage?
• Are there any visible suggestions that infinite loops can occur?
• Is one variable used for different tasks within the same procedure?
• Are algorithms as efficient as possible?
• Are inefficient sort/search mechanisms used
• Are there statements within iteration loops that can be placed outside them?
• Would changing the design of the database or data structure improve performance?
• Other things to look for:
• Will equations result in a divide by zero or overflow a variable
• Is sufficient checking performed to insure that array references will not overflow?
• Are there memory leaks? I.e. are all items that are allocated also deleted?
• Are there Y2K errors?
• Are there any hybrid variables (those used for both data and control?)
• Are all error messages meaningful? And does the system close down gracefully or abort transactions completely after errors?
• Does the program handle out of memory or full disk errors gracefully?
• What effect is there for multiple users?
• Is the code easy to read (and therefore easy to maintain)?

Statistical Quality Assurance

• Attempt to become more quantitative about quality
• Gives management the ability to measure and compare quality between projects and over time
• Allows analysts and managers to pin point causes of defects and change procedures to reduce future defects

Basic Process

• Record all defects as they are uncovered
• Trace/categorize defect according to its underlying cause
• Incomplete/Erroneous specification
• Poor communications with customer
• Intentional deviation from specification
• Violation of programming standards
• Inconsistent module interface
• Error in design logic
• Incomplete testing
• Inaccurate documentation
• Programming error
• Ambiguous or poor human computer interface
• Classify each error as Serious, Moderate, Minor
• Record the step in the development process where the defect was uncovered (not where it occurred)
1. Survey Phase
2. Study Phase
3. Define Definition Phase
4. Configuration and Design Phase
5. Implementation Phase
6. Delivery Phase (installation, operation, maintenance)
• Build a table showing the total number of errors for each cause and the % of the total that cause represents (table 8.2)
• Use the "Pareto principle" to identify primary causes of errors (80% of the defects are caused by 20% of the causes)

• Calculate an Phase Index (PI) for each major step in the process
• The phase index is the weighted error total for that phase of the development process
• Weights are assigned to the classifications , serious (w_s) = 10, moderate (w_m) = 3, minor(w_t) = 1)
• Calculate the PI for each phase:
• PI_i = w_s(S_i/E_i) + w_m(M_i/E_i) + w_t(T_i/E_i)
• where:
• E_i = total errors for phase i
• S_i = total serious errors uncovered during phase i
• M_i = total moderate errors uncovered during phase i
• T_i = total minor errors uncovered during phase i
• Note:  the equation the author gives factors out the total number of errors at each step, therefore a better measure of the Phase Index  would be:
• PI_i = w_s(S_i) + w_m(M_i) + w_t(T_i)
• Determine the Product Size (PS) based on the number of lines of code, pages of documentation, etc.
• Calculate the Error Index for the project as a whole
• Error index can be used to measure overall quality of the project and gives managers the ability to compare projects
• EI = Sum (i x PI_i)/PS
• Note, errors uncovered during later phases are weighted more heavily.

 Software Reliability

• Defined : "the probability of failure free operation of a computer program in a specified environment for a specified time."
• This definition relies on an accurate definition of failure.
• Mean time between failures (MTBF) is one simple measure of reliability
• MTBF = MTTF + MTTR
• where:
• MTTF = mean time to fail
• MTTR = mean time to repair
• Availability = MTTF/(MTBF) x 100%

Study Guide:

Key words:

Problems : 2, 3, 6, 7, 8, 9, 11, 12