Hybrid Manual/Automation Tests

I've had this idea before. Suppose you have a set of tests that are difficult to automate because some middle step needs either human interaction or would benefit from a human performing the validation.

Well, Amazon's Mechanical Turk is one option. Another option was just announced as well. With these tools, you can add in a human task to your automation tests to make your Manual / Automation hybrid tests.

So... Any tool authors out there that want to make this available?

Inspected by A-12345

(from http://www.qclabels.com/)

Every so often, I find a slip of paper in a new pair of jeans that reads "Inspected by 12" (or something similar). Then I realize that someone, or multiple someones, inspected this product for some level of quality before it was shipped.

What if you looked at the help menu entry for a particular feature and it would tell you who tested it? What do you think you'd notice? Would you attribute the good or bad quality of that feature with that tester's ability? If you were that tester, would you feel pride in a job well done?

I feel somewhat like I'm living 3Ms motto, "We don't make the products you use, we make the products you use better".

Structured Creativity Lessons?

I saw this book, published in 2007 called "Wreck This Journal". It is a series of journal entries where you are asked to basically demolish the book, one page at a time, in a very specific way.

The instructions include things like "Rub dirt on this page", "Paint with coffee", "Paste a picture here and change it"

It may seem totally random, but I expect that it will encourage creativity in a fun way.

Learn to Code With Friends

If you've wanted to learn how to write code, but didn't have a place to start. Check out http://www.codecademy.com/ . This is a way to learn to code where you are given feedback as you learn in a very intuitive way. You can also do this with friends to see how they are doing. Check it out!

Ow, I'm Too Safe!

http://blogs.msdn.com/b/oldnewthing/archive/2011/08/18/10197031.aspx

This is a short and funny story about being 'too safe'. Can we have 'too much quality' in the same way?

Robert's Rule of Risks

"When a risk manifests itself, it becomes an issue. For big issues, the real question is 'Was this from a known risk?' "

Is SmoothTeddy Really 'work-related'?

In the end, I came up with this banner as a sample for a 'logo contest' for our internal QA community. Let me give some background in how this came about.

I ran across 'Smooth Teddy' a few years ago as a drop-dead simple way to create interesting 3D figures that don't look like they came from a CAD system. Here is a bit more of the 3D 'bug' I was able to make. (Sure, it's not particularly 'buggy' but hopefully, you can see the scope of the things that can be done.)

If you end up using Smooth Teddy, be sure to read the documentation because some of the coolest features are not obvious. (The program itself is a bit unstable, so save often).

The remaining question is 'Is this 3D tool work-related'. Well, only if you want to allow people to find their way to learn. I'm certain it has nothing to do with the 'fun' factor of using the tool :)

Framework to Allow User to Recover Gracefully from Hung Applications

So some researchers were looking into how to get MS Word to recover when it gets hung and came up with a general framework to allow users to attempt to get hung applications unstuck.

http://arstechnica.com/science/news/2011/08/jolt-framework-lets-users-force-some-hung-programs-to-recover.ars

This is a great case of a truly useful tool that can be 'an enabler' for bad programming practices. Surely, this is a good tool for end-users. But developers providing this as a workaround for problematic code can be a bad strategy.

Removing Bias From Your Strategy

While this article is geared towards corporate strategy, many of the concepts can be applied to product strategy, test strategy, etc.

http://www.mckinseyquarterly.com/The_case_for_behavioral_strategy_2551 (requires free registration)

Here are some of the tips provided.
- The process of developing the strategy is one of the largest contributors to success
- Change your 'angle of vision' to see the issue from different perspectives. For example, Apple is famous for fully developing multiple ideas for the same feature to come up with a single implementation that fully implements the new feature.
- In most organizations, simply expressing high confidence in a plan is enough to get it approved. However, that does not result in success. Approval should be based on a clear recognition of the uncertainty involved.
- Shake things up, such as setting 'stretch goals' that are not possible given the current ways of doing things. This can spark innovation and efficiencies.
- Call out 'Silo thinking'. If departments hold onto their own departmental interests at the expense of company interests, call it out.
- Allow for 'Safe Debate'. Group-think can elimnate dissent, but sometimes dissent is needed. The key is to be able to do it without personalizing it.

There are more interesting topics on this site. Look around and enjoy.

Happy Sys Admin Day!

http://sysadminday.com/

The last Friday of July is the unofficial 'Sys Admin Day'. Go find the folks that ensure your servers stay running and give them a huge THANKS!

If that's you, keeping those systems up. I Thank You!

Using Pairwise Testing to Minimize Testing and Maintain High Feature Coverage

Problem
You may run across a situation where there are so many combinations of settings to test, it's virtually impossible to test every combination in the time available. Let's consider an example.

Suppose you are testing a product that has printing capabilities and you want to validate that the printing functionality works. You support multiple printers on multiple operating systems using multiple web browsers. In our ficticous product, let's use the following supported components:

Operating Systems
- Windows 7
- Windows Vista
- Windows XP SP 3
- Mac OSX 10.5 (Leopard)
- Mac OSX 10.6 (Snow Leopard)
- Mac OSX 10.7 (Lion)
- Ubuntu 11.0.4

Browsers
- Opera (latest)
- Firefox (latest)
- Safari (latest)
- Chrome (latest)
- IE 7
- IE 8

Printers (you have selected these to work on the supported OS versions)
- HP Inkjet
- HP Laser
- Epson Inkjet
- Epson Laser
- Lexmark Inkjet
- Lexmark Laser
- Canon Inkjet
- Canon Laser
- PDF (file)

User Role (Not testing has caused problems in the past and needs to be added)
- User
- Manager
- Administrator


To test every configuration that would be 7 OS versions x 6 browser versions x 9 printers x 3 roles = 1134 individual combinations. You may notice that IE won't work on non-Microsoft OS, but we'll discuss that in a bit.

Solution
That just won't do. That could be weeks of testing. Let's use the 'allpairs' tool , you could use anything from http://www.pairwise.org/ . I created an input file that listed each component type as columns and it identified 66 test cases that provide good coverage of the supported configurations.

Here is how my input file looked.

OS browser printer role
Windows 7 Opera (latest) HP Inkjet User
Windows Vista Firefox (latest) HP Laser Manager
Windows XP SP 3 Safari (latest) Epson Inkjet Administrator
Mac OSX 10.5(Leopard) Chrome (latest) Epson Laser
Mac OSX 10.6(Snow Leopard) IE 7 Lexmark Inkjet
Mac OSX 10.7 (Lion) IE 8 Lexmark Laser
Ubuntu 11.0.4 Canon Inkjet
  Canon Laser
  PDF (file)

If you enter your data in a spreadsheet, copy the data and paste into notepad, it will be in a format that allparis likes. Other tools will have different input methods and the results will be the same.

The output is another tab-delimited file and if you paste the results into excel, you'll see the specific tests.

case OS browser printer role pairings
1 Windows 7 Opera (latest) HP Inkjet User 6
2 Windows Vista Firefox (latest) HP Inkjet Manager 6
3 Windows XP SP 3 Safari (latest) HP Inkjet Administrator 6
4 Windows 7 Firefox (latest) HP Laser Administrator 6
5 Windows Vista Opera (latest) HP Laser User 5
…
63 Windows XP SP 3 ~Chrome (latest) Lexmark Laser ~Manager 1
64 Mac OSX 10.7 (Lion) ~Opera (latest) Canon Inkjet ~Manager 1
65 Mac OSX 10.7 (Lion) ~Firefox (latest) Canon Laser ~Administrator 1
66 Ubuntu 11.0.4 ~Opera (latest) PDF (file) ~Administrator 1

With this 94% reduction in the number of tests, you can expect to find the vast majority of defects.

Theory
From the pairwise.org site:

Pairwise (a.k.a. all-pairs) testing is an effective test case generation technique that is based on the observation that most faults are caused by interactions of at most two factors. Pairwise-generated test suites cover all combinations of two therefore are much smaller than exhaustive ones yet still very effective in finding defects.

If you look at the far-right column of the output table, you'll see a 'pairings' count. This count identifies how many unique pairings of configurations exist in that test. At the top of the list, there are 6 and 6 unique pairings. At the bottom of the list are only single pairings. You can use this information to reduce the number of tests even further. By eliminating tests that have few pairings, you lessen the impact on the overall testing coverage.

There are tools listed that will use more than two factors to limit tests and these can be used when there are lots of columns in your tables. So instead of finding pairs of configurations, you would be finding triples or quadruples of configurations. This increases the likelihood of finding defects, but you will have to contend with 'The Law of Dimishing Returns' in that you'll have to run lots more tests for smaller gains in coverage.

A Smart Battery That Was Too Smart

Recently, a security researcher found that apple's Smart Battery firmware could be compromised to infect the laptop it was used in. Yikes!





http://arstechnica.com/apple/news/2011/07/how-charlie-miller-discovered-the-apple-battery-hackhow-a-security-researcher-discovered-the-apple-battery-hack.ars

Bugs!

(image from http://www.clker.com/)

I'm not sure if this qualifies as an allegory or not, but here goes...

(Disclaimer: If you are concerned about the killing of bugs, please do not read further)

I came to work this morning to find a half dozen small ants at my desk. I had heard others having the same issue, but they were several desks away.

My first idea was to mash them. I quickly learned that this was pretty ineffective. I then spot-checked desks nearby. A desk on one side had as many (or more) than I did, but nobody else in our immediate vicinity did.

I decided that I needed to get something that would be more effective. Arriving at the store, I went to the home pesticide area. There were lots of pesticides, including about a dozen that would work on the ants.

Fortunately, there was someone who worked there that pointed out a couple products.

"This trap didn't work because they just wouldn't go in"

"I used this other one and it's been two years and they haven't been back"

Sold!

I considered other options while I was there. There were several types of traps that would attract the ants and then keep them from escaping. There were several products that would kill them outright and others that would feed them poison to take back to the colony to kill them and the colony. I stuck with the colony-killing product that was recommended and came back to work.

There were instructions, but I didn't use them :P . I put the poison in several places. As the ants kept coming, I squashed them. Over the course of the day, the ants at both desks dropped dramatically.

The only question remaining is what will my desk look like in the morning? Would they have scattered during the day anyway?

One careless strcat?

http://www.globalnerdy.com/2011/07/20/best-coding-practices-poster-1-only-you-can-prevent-buffer-overflows/

I saw this one yesterday and a new poster today. Very fun!

Stuxnet

Thanks to my friend Frank, I was able to get to read up in Wired on how Stuxnet was reverse-engineered and what was found when they did so.

http://www.wired.com/threatlevel/2011/07/how-digital-detectives-deciphered-stuxnet/all/1

Be ready to spend some time reading, it's not short, but it's engaging the whole way.

Today in Computing History - July 13 - The Smart Ship That Went Dumb

https://sites.google.com/site/todayincomputinghistory/google-gadget


"In an attempt to promote the stability and utility of the Windows NT platform, a new 'Smart Ship' was fitted with a Windows NT-based network. When one of the systems crashes, a domino-effect takes down all critical systems, leaving the ship unable to move for about an hour."

This is one of the events that I added to my 'Today in Computing History' project. Have fun!

Security Links

First, let's start with something fun. This link is to a t-shirt design that shows bad code, why it's bad and what can happen if someone mis-uses it http://cwe.mitre.org/community/tshirt.html

Next, in order to make security testing / mitigation a priority, there needs to be some way to measure it. Here is a link to a group that is working on this http://measurablesecurity.mitre.org/ One standard that is currently used is CVSS http://www.first.org/cvss/cvss-guide.pdf

NIST (National Institute of Standards and Technology) and You

There are so many documents NIST has created that they need a document to list all the documents!http://csrc.nist.gov/publications/CSD_DocsGuide.pdf This list includes the "Technical Guide to Information Security Testing and Assessment" http://csrc.nist.gov/publications/nistpubs/800-115/SP800-115.pdf

There's even a database of various checklists at http://checklists.nist.gov/ One example is, .NET security configuraiton checklists http://web.nvd.nist.gov/view/ncp/repository/checklistDetail?id=7

All of this information stems from looking at a tool called Retina, from eEye.

This tool allows you to scan computers on the network to look for known vulnerabilities. The scans and results provided are best interpreted in light of the links above.

Enjoy!

Trusted Identities

As part of my development as a Software Quality Assurance tester, I've been building my skills in the Software Security arena. I ran across this US national strategy initiative on Trusted Identities.

Consider a world where you can apply for a mortgage and sign for it digitally and use that same infrastructure to anonymously connect to a private online discussion. There are lots of problems yet to solve in regards to this infrastructure, but here are some useful links to look into this further.

http://www.nist.gov/nstic/NSTIC-Why-We-Need-It.pdf

http://www.nist.gov/nstic/

Job Posting - Sharepoint Admin, Oklahoma City

If you know of someone with Sharepoint experience, please send them Todd's way. You'll find his contact information at the bottom.

============================================

Sr. SharePoint Administrator - Oklahoma City, OK

Primary Duties & Responsibilities:
•Serve as Administrator for SharePoint 2007, SharePoint 2010
•Serve as Administrator for FAST for SharePoint 2010
•Plan and lead application system installations and upgrades.
•Monitor, support and diagnose application systems functioning to ensure specifications are met.
•Modify existing application systems to correct errors, allow it to adapt to new hardware, or to improve its performance.
•Develop procedures for application system testing and validation to ensure system integrity.
•Prepare and review appropriate documentation to record computer specifications, layouts and peripheral equipment modifications.
•Design and perform capacity planning processes to ensure performance is in line with business expectations.
•Analyze user needs and design software requirements within time and cost constraints.
•Develop and support business relationships with project team and business customers.
•Develop and implement policies, procedures and standards to ensure conformance with information systems goals and procedures.
•Report status, issues, and timelines to management staff and project teams.
•Lead multi-function teams in problem resolution, evaluation and implementation of solutions.
•Provide mentoring and training to associate systems administrators and system administrators.

•Strong knowledge of enterprise search concepts
•Strong knowledge of web server technologies/frameworks/concepts (HTTP, networking, Windows Server 2008 R2, Active Directory, PowerShell)
•Demonstrated ability to solve complex technical issues, working alone or in a team
•Demonstrated ability to work in a fast-paced, team environment
•Ability to communicate effectively with all levels of co-workers, clients and other external contacts
•Demonstrated ability to document system architecture, installation configuration, methods and procedures for cross training and troubleshooting
•Excellent written and verbal English communication skills
•Experience with Microsoft FAST Search is preferred
•Experience with Microsoft SharePoint is preferred
Education & Work Experience

Minimum Requirements:
•Bachelor's degree in Computer Science or Management Information Systems or equivalent experience.
•Experience generating or adapting equipment and technology to serve user needs.
Minimum of 4 years Systems Administration - SharePoint experience

Contact:

Todd Laduron
Sr. Technical Recruiter - CDI IT Solutions
602-508-6471

Did you ever...?

Did you ever find a site that you went to start to get slow, buggy or that people just seemed to stop responding to you as if they all agreed to do so at the same time?

It may not be just your imagination or that the site is having issues. You may be flagged as an abusive user and these techniques are intentional ways to convince you to calm down and re-think your approach.

I was reading about the stackoverflow.com site (and related sites) where something needed to be done with the small set of users that were making the site unpleasant for everyone else.

http://www.codinghorror.com/blog/2011/06/suspension-ban-or-hellban.html

But what does this have to do with QA? Well, consider what happens if your site behaves erratically, is slow and error prone. Whether it's intentional or not, it drives users away. Do you know how often this happens? What's the impact when it does? How do you bring this to light?

Just something to consider.