Regression Testing Without a QA team - Part I

Many of you work in an environment where there are no QA people assigned to your software product. So when a new version is released, it's usually up to the developers or other non-QA staff to make sure the release is ready.

Usually, this ends up being the same people spending a few days with the new version to make sure nothing seems wrong. But wouldn't it be better to have some insight into what parts of the system were tested and what parts weren't?

Let's walk through a basic strategy for getting some insight into how you want to not only test your product, but how you want to report on those results. Even QA people forget to think about how they want to report their results and end up with a 1200 page document that nobody reads.

For the examples, I'll suppose you are on the team that tests the Calculator application that comes with Windows OS and you are preparing to get it ready for Windows Vista based on the Windows XP version. There are no new visible features to speak of. But there is a new API, Themes, etc. to take into consideration.

Step 1 - Develop a model of the system functionality

There is no single way to do this, but here are some suggestions and considerations.

Think of the 'ilities'

• Usability
• Functionality
• Security
• Performance
• Release (Backwards-Compatibility/Upgrade/Installation)

Think of how the product is organized:

• Are there specific workflows that are helpful to use to organize the list of funcitonality?
• Would it be helpful to organize by screen/page/mode? If it's a website, is it helpful to organize functionality by web page? If it is a command-line applicaiton, is it helpful to organize by comand-line switch?
• If there are multiple components, is it helpful to organize by component (or group of components)?

Keep in mind, that you will usually have a blend of strategies in the end.

For the calculator example, we need to include the scientific and statistical functions.

List of System Functionality (aka Test Requirements)

Functionality

- Perform Standard Calculation

- Perform Additional Scientific Calculations

- Perform Additional Statistical Calculations

- Perform Support Functions

- Support Copy/Paste

- Support Digit Grouping Function

- "About" functionality changes for Vista

Usability

- Support OS Themes

- Support OS Color schemes

- Support OS Font Size settings

- Support 'each' Vista version

NOTE

But there's also a question of detail. You can list test requirements in varying degrees of detail. I've been on both sides of the equation. Too many requirements can be difficult to maintain. Too few details can tend to be useless. You will have to determine what makes sense to you and be prepared to make changes accordingly.

You will also want to think of how you want to report these issues. It may be helpful to use some subset of these test requirements to group test results. Again, this will be dependent on your needs.

Step 2 - Identify 'high risk' areas of functionality

Not all features are created equal. Some workflows are used daily and others are used infrequently. Those used daily likely pose a higher risk to the product if they don't work properly. So-called 'core functionality' is good candidate as well. You will want to come up with a scale for the risk rating. In addition to the scale, it may be helpful to identify the criteria used to justify this risk rating.

For our example, here is the risk rating we could use.

1 - Without this feature, the product would be rejected by the users

2 - Without this feature, the product would be used rarely by the users

3 - Without this feature, the product would be used, but with reservations

4 - Unlikely to affect usage of this product

Here is how we could apply the risk rating to our test requirements.

Functionality

- Perform Standard Calculation (risk - 1)

- Perform Additional Scientific Calculations (risk - 2)

- Perform Additional Statistical Calculations (risk - 2)

- Perform Support Functions

- Support Copy/Paste (risk - 3)

- Support Digit Grouping Function (risk - 4)

- "About" functionality changes for Vista (risk - 4)

Usability

- Support OS Themes (risk - 3)

- Support OS Color schemes (risk - 3)

- Support OS Font Size settings (risk - 3)

- Support 'each' Vista version (risk - 2)

NOTE

You will want to make sure that there is somewhat of an even distribution in your risk levels. Too many items in one level means that you'll want to discriminate further items at that level. We will use this risk rating later.

Wow, this is getting long. I'll post what I have now and pick up on this in a day or so.