Knowing when it's good enough

A business uses cost-benefit analysis as one of its evaluation criteria when deciding whether to pursue an opportunity or not. One concept central to this is the assessment of potential risks. To me there are a few components to risk assessment:

1. Likelihood of the risk materializing. What's the probability of the risk becoming a problem?
2. Potential cost (monetary, compliance or reputation) to mitigate the risk. How much is it going to cost you? This also includes implementing any mitigation strategies.
   

Combined, these factors provide a value to each risk. Risk assessment can be applied in many different areas:

• Managing the risks encountered in a project.
• Understanding the importance of a defect or bug in a system.
• Managing financial assets.
  

For the purposes of this post I'm only going to write about the second item.

What do you do if you find a defect or bug in a system? You evaluate it to determine whether it's a show stopper or whether you can proceed despite it. Creating bug-free software is the ultimate goal but it has cost and time implications. Here's a quote from an article, They Write The Right Stuff, from Fast Company about a specific software system.

This software never crashes. It never needs to be re-booted. This software is bug-free. It is perfect, as perfect as human beings have achieved. Consider these stats: the last three versions of the program -- each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors.

The piece of software in question runs the NASA space shuttle. The consequences of failure could result in the deaths of the astronauts, the loss of a multi-billion dollar piece of hardware and many years of setbacks to the space program. Many of the systems we deal with in the business world do not have this level of criticality. In my past jobs I have worked with time sensitive trading systems where 1/2 second delays mean losses in the $10,000's of dollars. On other projects, variances in marketing and market research data were explainable ("Yes, someone did purchase 50 rolls of toilet paper and it skewed the numbers.") We need to understand when something is good enough as it is instead of always trying to be perfect.

Slides available: Testing & validation

The slide I used on my post, Testing & validation, is available for download at esnips.com. You can find it using this link.

Non-functional requirements & QA test strategies

I received an interesting comment on my index post for non-functional requirements, from leathej1. Below is a short clip.

I just posted something on my site that has been gnawing at me for a few years - the fact that in my mind there is no such thing as a non-functional requirement. As defined in the traditional RUP sense.

A link was provided to leathej1's blog that contained his rationale. I'm still mulling it over in my mind but his post called, The Fallacy of Non-Functional Requirements, contains an excellent QA test strategy covering off the testing of "non-functional" requirements. I encourage you to visit his blog and check it out.

Testing & validation

Today's post will focus on testing and validation procedures that are performed throughout a system development life cycle (SDLC.) As the different artifacts and deliverables are created, traceability and requirements management practices are used to ensure that the vision and direction for the project is maintained. Thus,

• Business requirements are expanded into functional requirements.
• Functional requirements lead to design documents.
• Design documents lead to code generation.

Refer to the diagram below, the names used are not important but the objects they represent are.

From the perspective of testing and validation, testing is done as follows:

• Code is reviewed.
• Components defined in the design documents are unit tested (e.g., individually tested.)
• Integrated tests are performed on components that work together (aligning with the functional requirements and architecture.)
• UAT testing is performed to ensure that the objectives of the project are realized.
• After the project goes live, performance against the key goals (e.g., improve throughput by 15%) are measured. Ultimately, this will be one of the factors that determines the ultimate success.

Testing & validation in detail

As new code is created, code reviews can be performed by other programmers and team members. Personally, I do not code (and am not familiar with this) so I'm going to gloss over this part.

Functional requirements are expressed as design documentation that support the requirements. Each component is tested independently (e.g., unit tested.) Inputs are simulated and the outputs examined to determine whether the component accomplishes its purpose.

Functional requirements align with their corresponding business requirements to identify the key characteristics and needs for a project. You must be able to drill back to the higher-level business requirements and down to the design documents to get the full benefits of traceability. When testing to see that functional requirements have been met, you must determine that the different functional components work with each other. Integrated testing is the key concept. Outputs from one component become inputs into the other related components. The terminology integrated system testing has been used to express this idea.

The business requirements define the objectives and measurement criteria for the project as a whole (see my post called, The sweet smell of success.) User acceptance testing (UAT) is done to ensure that the basic business objectives are satisfied (e.g., setup a marketing campaign.) Measurement against performance goals are done at this stage as well as after the project's deliverable goes live. You cannot determine if the performance goals have been attained until the end-users have been trained and passed through the ramp-up period.

Good Requirements - Part V - Be Verifiable

At some point in a project the question, "Did we meet the requirement?" will be asked. This seems like a simple yes / no question, but sometimes, there is no eloquent answer. At times, development staff may produce something similar but not as articulated while at other times the articulation of the requirement itself will make it difficult to verify success. In this posting, I will be focusing on how to articulate requirements to enhance the ability to verify that they have been met.

Consider the following two examples:

Example 1 - A traditional "prose-like" articulation of requirements
A sports car that goes from 0-60 mph within 5 seconds, meets local emission standards and has a 6 speed manual transmission system. There will be a 6 disc CD changer and a DVD player available.

Example 2 - An atomic articulation of requirements
Accelerates from 0-60 within 5 seconds.
Meets local emission standards.
Available with a 6-speed manual transmission.
Option for a 6-disc CD changer.
Option for a DVD player.

Now, suppose that the car accelerates from 0-60 in less than 5 seconds, meets emission standards (in the applicable locals), has a 5 speed manual transmission and has a 6 disc CD changer but no DVD player. In example one, is the requirement met or not? Fully? 60%? Yes? No?

In the second example it is very clear that the 1st, 2nd and 4th requirements have been meet while the 3rd and 5th requirements have not been met.

The point I am trying to express is that by stating requirements as simple atomic statements the ability to easily verify them is enhanced significantly. All requirements must be testable. At the end of the day, one must show a solution adheres to its specifications. This is imperative towards building credibility with one's clients.