Using Pairwise Testing to Minimize Testing and Maintain High Feature Coverage

Problem
You may run across a situation where there are so many combinations of settings to test, it's virtually impossible to test every combination in the time available. Let's consider an example.

Suppose you are testing a product that has printing capabilities and you want to validate that the printing functionality works. You support multiple printers on multiple operating systems using multiple web browsers. In our ficticous product, let's use the following supported components:

Operating Systems
- Windows 7
- Windows Vista
- Windows XP SP 3
- Mac OSX 10.5 (Leopard)
- Mac OSX 10.6 (Snow Leopard)
- Mac OSX 10.7 (Lion)
- Ubuntu 11.0.4

Browsers
- Opera (latest)
- Firefox (latest)
- Safari (latest)
- Chrome (latest)
- IE 7
- IE 8

Printers (you have selected these to work on the supported OS versions)
- HP Inkjet
- HP Laser
- Epson Inkjet
- Epson Laser
- Lexmark Inkjet
- Lexmark Laser
- Canon Inkjet
- Canon Laser
- PDF (file)

User Role (Not testing has caused problems in the past and needs to be added)
- User
- Manager
- Administrator


To test every configuration that would be 7 OS versions x 6 browser versions x 9 printers x 3 roles = 1134 individual combinations. You may notice that IE won't work on non-Microsoft OS, but we'll discuss that in a bit.

Solution
That just won't do. That could be weeks of testing. Let's use the 'allpairs' tool , you could use anything from http://www.pairwise.org/ . I created an input file that listed each component type as columns and it identified 66 test cases that provide good coverage of the supported configurations.

Here is how my input file looked.

OS browser printer role
Windows 7 Opera (latest) HP Inkjet User
Windows Vista Firefox (latest) HP Laser Manager
Windows XP SP 3 Safari (latest) Epson Inkjet Administrator
Mac OSX 10.5(Leopard) Chrome (latest) Epson Laser
Mac OSX 10.6(Snow Leopard) IE 7 Lexmark Inkjet
Mac OSX 10.7 (Lion) IE 8 Lexmark Laser
Ubuntu 11.0.4 Canon Inkjet
  Canon Laser
  PDF (file)

If you enter your data in a spreadsheet, copy the data and paste into notepad, it will be in a format that allparis likes. Other tools will have different input methods and the results will be the same.

The output is another tab-delimited file and if you paste the results into excel, you'll see the specific tests.

case OS browser printer role pairings
1 Windows 7 Opera (latest) HP Inkjet User 6
2 Windows Vista Firefox (latest) HP Inkjet Manager 6
3 Windows XP SP 3 Safari (latest) HP Inkjet Administrator 6
4 Windows 7 Firefox (latest) HP Laser Administrator 6
5 Windows Vista Opera (latest) HP Laser User 5
…
63 Windows XP SP 3 ~Chrome (latest) Lexmark Laser ~Manager 1
64 Mac OSX 10.7 (Lion) ~Opera (latest) Canon Inkjet ~Manager 1
65 Mac OSX 10.7 (Lion) ~Firefox (latest) Canon Laser ~Administrator 1
66 Ubuntu 11.0.4 ~Opera (latest) PDF (file) ~Administrator 1

With this 94% reduction in the number of tests, you can expect to find the vast majority of defects.

Theory
From the pairwise.org site:

Pairwise (a.k.a. all-pairs) testing is an effective test case generation technique that is based on the observation that most faults are caused by interactions of at most two factors. Pairwise-generated test suites cover all combinations of two therefore are much smaller than exhaustive ones yet still very effective in finding defects.

If you look at the far-right column of the output table, you'll see a 'pairings' count. This count identifies how many unique pairings of configurations exist in that test. At the top of the list, there are 6 and 6 unique pairings. At the bottom of the list are only single pairings. You can use this information to reduce the number of tests even further. By eliminating tests that have few pairings, you lessen the impact on the overall testing coverage.

There are tools listed that will use more than two factors to limit tests and these can be used when there are lots of columns in your tables. So instead of finding pairs of configurations, you would be finding triples or quadruples of configurations. This increases the likelihood of finding defects, but you will have to contend with 'The Law of Dimishing Returns' in that you'll have to run lots more tests for smaller gains in coverage.