http://sysadminday.com/
The last Friday of July is the unofficial 'Sys Admin Day'. Go find the folks that ensure your servers stay running and give them a huge THANKS!
If that's you, keeping those systems up. I Thank You!
Friday, July 29, 2011
Thursday, July 28, 2011
Using Pairwise Testing to Minimize Testing and Maintain High Feature Coverage
Problem
You may run across a situation where there are so many combinations of settings to test, it's virtually impossible to test every combination in the time available. Let's consider an example.
Suppose you are testing a product that has printing capabilities and you want to validate that the printing functionality works. You support multiple printers on multiple operating systems using multiple web browsers. In our ficticous product, let's use the following supported components:
Operating Systems
- Windows 7
- Windows Vista
- Windows XP SP 3
- Mac OSX 10.5 (Leopard)
- Mac OSX 10.6 (Snow Leopard)
- Mac OSX 10.7 (Lion)
- Ubuntu 11.0.4
Browsers
- Opera (latest)
- Firefox (latest)
- Safari (latest)
- Chrome (latest)
- IE 7
- IE 8
Printers (you have selected these to work on the supported OS versions)
- HP Inkjet
- HP Laser
- Epson Inkjet
- Epson Laser
- Lexmark Inkjet
- Lexmark Laser
- Canon Inkjet
- Canon Laser
- PDF (file)
User Role (Not testing has caused problems in the past and needs to be added)
- User
- Manager
- Administrator
To test every configuration that would be 7 OS versions x 6 browser versions x 9 printers x 3 roles = 1134 individual combinations. You may notice that IE won't work on non-Microsoft OS, but we'll discuss that in a bit.
Solution
That just won't do. That could be weeks of testing. Let's use the 'allpairs' tool , you could use anything from http://www.pairwise.org/ . I created an input file that listed each component type as columns and it identified 66 test cases that provide good coverage of the supported configurations.
Here is how my input file looked.
If you enter your data in a spreadsheet, copy the data and paste into notepad, it will be in a format that allparis likes. Other tools will have different input methods and the results will be the same.
The output is another tab-delimited file and if you paste the results into excel, you'll see the specific tests.
With this 94% reduction in the number of tests, you can expect to find the vast majority of defects.
Theory
From the pairwise.org site:
If you look at the far-right column of the output table, you'll see a 'pairings' count. This count identifies how many unique pairings of configurations exist in that test. At the top of the list, there are 6 and 6 unique pairings. At the bottom of the list are only single pairings. You can use this information to reduce the number of tests even further. By eliminating tests that have few pairings, you lessen the impact on the overall testing coverage.
There are tools listed that will use more than two factors to limit tests and these can be used when there are lots of columns in your tables. So instead of finding pairs of configurations, you would be finding triples or quadruples of configurations. This increases the likelihood of finding defects, but you will have to contend with 'The Law of Dimishing Returns' in that you'll have to run lots more tests for smaller gains in coverage.
You may run across a situation where there are so many combinations of settings to test, it's virtually impossible to test every combination in the time available. Let's consider an example.
Suppose you are testing a product that has printing capabilities and you want to validate that the printing functionality works. You support multiple printers on multiple operating systems using multiple web browsers. In our ficticous product, let's use the following supported components:
Operating Systems
- Windows 7
- Windows Vista
- Windows XP SP 3
- Mac OSX 10.5 (Leopard)
- Mac OSX 10.6 (Snow Leopard)
- Mac OSX 10.7 (Lion)
- Ubuntu 11.0.4
Browsers
- Opera (latest)
- Firefox (latest)
- Safari (latest)
- Chrome (latest)
- IE 7
- IE 8
Printers (you have selected these to work on the supported OS versions)
- HP Inkjet
- HP Laser
- Epson Inkjet
- Epson Laser
- Lexmark Inkjet
- Lexmark Laser
- Canon Inkjet
- Canon Laser
- PDF (file)
User Role (Not testing has caused problems in the past and needs to be added)
- User
- Manager
- Administrator
To test every configuration that would be 7 OS versions x 6 browser versions x 9 printers x 3 roles = 1134 individual combinations. You may notice that IE won't work on non-Microsoft OS, but we'll discuss that in a bit.
Solution
That just won't do. That could be weeks of testing. Let's use the 'allpairs' tool , you could use anything from http://www.pairwise.org/ . I created an input file that listed each component type as columns and it identified 66 test cases that provide good coverage of the supported configurations.
Here is how my input file looked.
OS browser printer role
Windows 7 Opera (latest) HP Inkjet User
Windows Vista Firefox (latest) HP Laser Manager
Windows XP SP 3 Safari (latest) Epson Inkjet Administrator
Mac OSX 10.5(Leopard) Chrome (latest) Epson Laser
Mac OSX 10.6(Snow Leopard) IE 7 Lexmark Inkjet
Mac OSX 10.7 (Lion) IE 8 Lexmark Laser
Ubuntu 11.0.4 Canon Inkjet
Canon Laser
PDF (file)
If you enter your data in a spreadsheet, copy the data and paste into notepad, it will be in a format that allparis likes. Other tools will have different input methods and the results will be the same.
The output is another tab-delimited file and if you paste the results into excel, you'll see the specific tests.
case OS browser printer role pairings
1 Windows 7 Opera (latest) HP Inkjet User 6
2 Windows Vista Firefox (latest) HP Inkjet Manager 6
3 Windows XP SP 3 Safari (latest) HP Inkjet Administrator 6
4 Windows 7 Firefox (latest) HP Laser Administrator 6
5 Windows Vista Opera (latest) HP Laser User 5
…
63 Windows XP SP 3 ~Chrome (latest) Lexmark Laser ~Manager 1
64 Mac OSX 10.7 (Lion) ~Opera (latest) Canon Inkjet ~Manager 1
65 Mac OSX 10.7 (Lion) ~Firefox (latest) Canon Laser ~Administrator 1
66 Ubuntu 11.0.4 ~Opera (latest) PDF (file) ~Administrator 1
With this 94% reduction in the number of tests, you can expect to find the vast majority of defects.
Theory
From the pairwise.org site:
Pairwise (a.k.a. all-pairs) testing is an effective test case generation technique that is based on the observation that most faults are caused by interactions of at most two factors. Pairwise-generated test suites cover all combinations of two therefore are much smaller than exhaustive ones yet still very effective in finding defects.
If you look at the far-right column of the output table, you'll see a 'pairings' count. This count identifies how many unique pairings of configurations exist in that test. At the top of the list, there are 6 and 6 unique pairings. At the bottom of the list are only single pairings. You can use this information to reduce the number of tests even further. By eliminating tests that have few pairings, you lessen the impact on the overall testing coverage.
There are tools listed that will use more than two factors to limit tests and these can be used when there are lots of columns in your tables. So instead of finding pairs of configurations, you would be finding triples or quadruples of configurations. This increases the likelihood of finding defects, but you will have to contend with 'The Law of Dimishing Returns' in that you'll have to run lots more tests for smaller gains in coverage.
Tuesday, July 26, 2011
A Smart Battery That Was Too Smart
Recently, a security researcher found that apple's Smart Battery firmware could be compromised to infect the laptop it was used in. Yikes!
http://arstechnica.com/apple/news/2011/07/how-charlie-miller-discovered-the-apple-battery-hackhow-a-security-researcher-discovered-the-apple-battery-hack.ars
http://arstechnica.com/apple/news/2011/07/how-charlie-miller-discovered-the-apple-battery-hackhow-a-security-researcher-discovered-the-apple-battery-hack.ars
Monday, July 25, 2011
Bugs!
(image from http://www.clker.com/)
I'm not sure if this qualifies as an allegory or not, but here goes...
(Disclaimer: If you are concerned about the killing of bugs, please do not read further)
I came to work this morning to find a half dozen small ants at my desk. I had heard others having the same issue, but they were several desks away.
My first idea was to mash them. I quickly learned that this was pretty ineffective. I then spot-checked desks nearby. A desk on one side had as many (or more) than I did, but nobody else in our immediate vicinity did.
I decided that I needed to get something that would be more effective. Arriving at the store, I went to the home pesticide area. There were lots of pesticides, including about a dozen that would work on the ants.
Fortunately, there was someone who worked there that pointed out a couple products.
"This trap didn't work because they just wouldn't go in"
"I used this other one and it's been two years and they haven't been back"
Sold!
I considered other options while I was there. There were several types of traps that would attract the ants and then keep them from escaping. There were several products that would kill them outright and others that would feed them poison to take back to the colony to kill them and the colony. I stuck with the colony-killing product that was recommended and came back to work.
There were instructions, but I didn't use them :P . I put the poison in several places. As the ants kept coming, I squashed them. Over the course of the day, the ants at both desks dropped dramatically.
The only question remaining is what will my desk look like in the morning? Would they have scattered during the day anyway?
Thursday, July 21, 2011
Monday, July 18, 2011
Stuxnet
Thanks to my friend Frank, I was able to get to read up in Wired on how Stuxnet was reverse-engineered and what was found when they did so.
Be ready to spend some time reading, it's not short, but it's engaging the whole way.
Wednesday, July 13, 2011
Today in Computing History - July 13 - The Smart Ship That Went Dumb
https://sites.google.com/site/todayincomputinghistory/google-gadget
"In an attempt to promote the stability and utility of the Windows NT platform, a new 'Smart Ship' was fitted with a Windows NT-based network. When one of the systems crashes, a domino-effect takes down all critical systems, leaving the ship unable to move for about an hour."
This is one of the events that I added to my 'Today in Computing History' project. Have fun!
"In an attempt to promote the stability and utility of the Windows NT platform, a new 'Smart Ship' was fitted with a Windows NT-based network. When one of the systems crashes, a domino-effect takes down all critical systems, leaving the ship unable to move for about an hour."
This is one of the events that I added to my 'Today in Computing History' project. Have fun!
Security Links
First, let's start with something fun. This link is to a t-shirt design that shows bad code, why it's bad and what can happen if someone mis-uses it http://cwe.mitre.org/community/tshirt.html
Next, in order to make security testing / mitigation a priority, there needs to be some way to measure it. Here is a link to a group that is working on this http://measurablesecurity.mitre.org/ One standard that is currently used is CVSS http://www.first.org/cvss/cvss-guide.pdf
NIST (National Institute of Standards and Technology) and You
There are so many documents NIST has created that they need a document to list all the documents!http://csrc.nist.gov/publications/CSD_DocsGuide.pdf This list includes the "Technical Guide to Information Security Testing and Assessment" http://csrc.nist.gov/publications/nistpubs/800-115/SP800-115.pdf
There's even a database of various checklists at http://checklists.nist.gov/ One example is, .NET security configuraiton checklists http://web.nvd.nist.gov/view/ncp/repository/checklistDetail?id=7
All of this information stems from looking at a tool called Retina, from eEye.
This tool allows you to scan computers on the network to look for known vulnerabilities. The scans and results provided are best interpreted in light of the links above.
All of this information stems from looking at a tool called Retina, from eEye.
This tool allows you to scan computers on the network to look for known vulnerabilities. The scans and results provided are best interpreted in light of the links above.
Enjoy!
Wednesday, July 06, 2011
Trusted Identities
As part of my development as a Software Quality Assurance tester, I've been building my skills in the Software Security arena. I ran across this US national strategy initiative on Trusted Identities.
Consider a world where you can apply for a mortgage and sign for it digitally and use that same infrastructure to anonymously connect to a private online discussion. There are lots of problems yet to solve in regards to this infrastructure, but here are some useful links to look into this further.
http://www.nist.gov/nstic/NSTIC-Why-We-Need-It.pdf
http://www.nist.gov/nstic/
Consider a world where you can apply for a mortgage and sign for it digitally and use that same infrastructure to anonymously connect to a private online discussion. There are lots of problems yet to solve in regards to this infrastructure, but here are some useful links to look into this further.
http://www.nist.gov/nstic/NSTIC-Why-We-Need-It.pdf
http://www.nist.gov/nstic/
Subscribe to:
Posts (Atom)