A Tester's Commitment

http://www.satisfice.com/blog/archives/652

James Bach has a good post on what testers commit to

Here's an excerpt:

I provide a service. You are an important client of that service. I am not satisfied unless you are satisfied.

I am not the gatekeeper of quality. I don’t “own” quality. Shipping a good product is a goal shared by all of us.

I will test your code as soon as I can after you deliver it to me. I know that you need my test results quickly (especially for fixes and new features).

I will strive to test in a way that allows you to be fully productive. I will not be a bottleneck.

I’ll make every reasonable effort to test, even if I have only partial information about the product.

I will learn the product quickly, and make use of that knowledge to test more cleverly.

I will test important things first, and try to find important problems. (I will also report things you might consider unimportant, just in case they turn out to be important after all, but I will spend less time on those.)

I will strive to test in the interests of everyone whose opinions matter, including you, so that you can make better decisions about the product.

I will write clear, concise, thoughtful, and respectful problem reports. (I may make suggestions about design, but I will never presume to be the designer.)

I will let you know how I’m testing, and invite your comments. And I will confer with you about little things you can do to make the product much easier to test.

I invite your special requests, such as if you need me to spot check something for you, help you document something, or run a special kind of test.

I will not carelessly waste your time. Or if I do, I will learn from that mistake.

Bugs and Battleship

I got this link from Peter Hall today, it's a great way to look at different ways we look for bugs.

http://blog.ezyang.com/2011/12/bugs-and-battleships/

Enjoy!

PCMM - Like CMM, but for people

There is lots of evidence that one of the major contributors to a successful software project is the right people with the right skills being motivated to perform well.

SEI has released (nearly 10 years ago) a method for managing this kind of management.

Check it out!

http://www.sei.cmu.edu/library/abstracts/reports/01mm001.cfm

QA Tester Position in Oklahoma

This is a 6 month contract to hire opportunity. Looking for somebody to start December 12^th but the earlier the better. They are definitely looking to bring somebody on full time. The first project is in Massachusetts and the individual will need to be on site the first 2 weeks. If the person is not in OKC they can work remotely from home and come to OKC once a week. If they are in OKC they will work in the office.

·         Test Designer with Keyword experience preferably someone with web interface experience.

·         DB queries would be a plus as would C#, .NET experience.

·         Automation testing is the biggest requirement

·         Will be using Selenium so if somebody has any experience with Selenium that would be a huge bonus!

Contact Info:

Taylor Brecher

GDH Consulting, Inc.

405-948-9022 Office

405-290-7505 Fax

972-839-1841 Mobile

tbrecher@gdhconsulting.com

www.gdhit.com

Become a fan of GDH Consulting on Facebook

Re-imagining the Test Plan

Some folks got together to discuss the relative lack of utility of most test plans. Their idea is to tie in streams of data (tests, test results, bugs, issues, checkins, etc.) to provide an ongoing look at areas of risk for your application in a visual way.

This is by no means a usable product at the moment. However, the ideas are interesting. Read more here, join the discussion and be a part of this project!

http://googletesting.blogspot.com/2011/10/google-test-analytics-now-in-open.html

Being Geek - another book review

I've posted another book review on StickyMinds.com. This one is on 'Being Geek'

http://www.stickyminds.com/s.asp?F=S1362_BOOK_4

Javascript Code Coverage Tool

The Google Testing Blog has an entry on a tool to calculate how much of the javascript on a page has been covered.

Now you can get details on each script on the page and how much is left to test!

http://googletesting.blogspot.com/2011/10/scriptcover-makes-javascript-coverage.html

Time-Life Series on 'Understanding Computers' (circa 1989)

My brother-in-law brought these over for me. They were going to be thrown out at his work and he offered to take them off their hands.

The one that caught my eye was 'The Software Challenge'. It's essentially about Software Engineering and the tools and techniques used in software projects.

At the time, the big idea was CASE tools. Computer-Aided Software Engineering. There's a whole section on that idea as well as testing concepts such as boundary-value testing, positive testing, negative testing, code path coverage and more.

Filters of Experience

Filters are great things, until they aren't...

I use my experience to filter lots of things. For example, I won't get started on optimizing how my email is stored because I've done that and it ultimately takes more time to maintain than I get value out of. Additionally, I will go out of my way to meet the maintenance people in my building because knowing them can quickly get issues fixed when they arise.

This experience I've developed over years has allowed me to see nuances in everyday activities to try to maximize the benefit to me or others.

But these filters of experience also hold me back sometimes. My son had a school project that would take several hours to do and he waited until the last minute to do it. I looked at the clock after dinner the night before the assignment was due and concluded that given my son's available time, he couldn't have finished it and would have to turn it in late. I thought that even if it were done, it wouldn't be done well. I was wrong. While he could have turned in a more polished final product, he did finish it reasonably well and only a little after his normal bedtime.

So if you are new in your career:

• Look for those with experience to see if you can work better
• Don't let that experience hold you back if you have the desire to see it through (even if you end up being wrong to ignore experience, you'll have just built up some of your own)

If you are well established in your career.

• Look to the younger crowd for energy and new possibilities
• Be sure to voice your experience in a way that allows everyone to feel successful. (even if you end up being wrong, it won't be the first time)

What do you do if you need to test a system with little to no documentation?

You need to identify what sources of information exist. The common term is a 'Test Oracle'. The Test Oracles will tell you how the system is supposed to work, but unfortunately, they are known to conflict with each other.

This is an old article, but still very relevant on what forms Test Oracles take and what their strengths and weaknesses are.

http://www.softwarequalitymethods.com/Papers/OracleTax.pdf

 

(thanks Peter for remining me that this site has been too silent for too long)

 

TestAPI for Win32 and .NET applications

I heard about this today and it looks promising... http://testapi.codeplex.com/

According to the site :

"TestApi is a library of test and utility APIs that enables developers and testers to create testing tools and automated tests for .NET and Win32 applications. TestApi provides a set of common test building blocks -- types, data-structures and algorithms -- in a simple, layered, componentized and documented stack."

Post a comment if you have any experience with this.

Hybrid Manual/Automation Tests

I've had this idea before. Suppose you have a set of tests that are difficult to automate because some middle step needs either human interaction or would benefit from a human performing the validation.

Well, Amazon's Mechanical Turk is one option. Another option was just announced as well. With these tools, you can add in a human task to your automation tests to make your Manual / Automation hybrid tests.

So... Any tool authors out there that want to make this available?

Inspected by A-12345

(from http://www.qclabels.com/)

Every so often, I find a slip of paper in a new pair of jeans that reads "Inspected by 12" (or something similar). Then I realize that someone, or multiple someones, inspected this product for some level of quality before it was shipped.

What if you looked at the help menu entry for a particular feature and it would tell you who tested it? What do you think you'd notice? Would you attribute the good or bad quality of that feature with that tester's ability? If you were that tester, would you feel pride in a job well done?

I feel somewhat like I'm living 3Ms motto, "We don't make the products you use, we make the products you use better".

Structured Creativity Lessons?

I saw this book, published in 2007 called "Wreck This Journal". It is a series of journal entries where you are asked to basically demolish the book, one page at a time, in a very specific way.

The instructions include things like "Rub dirt on this page", "Paint with coffee", "Paste a picture here and change it"

It may seem totally random, but I expect that it will encourage creativity in a fun way.

Learn to Code With Friends

If you've wanted to learn how to write code, but didn't have a place to start. Check out http://www.codecademy.com/ . This is a way to learn to code where you are given feedback as you learn in a very intuitive way. You can also do this with friends to see how they are doing. Check it out!

Ow, I'm Too Safe!

http://blogs.msdn.com/b/oldnewthing/archive/2011/08/18/10197031.aspx

This is a short and funny story about being 'too safe'. Can we have 'too much quality' in the same way?

Robert's Rule of Risks

"When a risk manifests itself, it becomes an issue. For big issues, the real question is 'Was this from a known risk?' "

Is SmoothTeddy Really 'work-related'?

In the end, I came up with this banner as a sample for a 'logo contest' for our internal QA community. Let me give some background in how this came about.

I ran across 'Smooth Teddy' a few years ago as a drop-dead simple way to create interesting 3D figures that don't look like they came from a CAD system. Here is a bit more of the 3D 'bug' I was able to make. (Sure, it's not particularly 'buggy' but hopefully, you can see the scope of the things that can be done.)

If you end up using Smooth Teddy, be sure to read the documentation because some of the coolest features are not obvious. (The program itself is a bit unstable, so save often).

The remaining question is 'Is this 3D tool work-related'. Well, only if you want to allow people to find their way to learn. I'm certain it has nothing to do with the 'fun' factor of using the tool :)

Framework to Allow User to Recover Gracefully from Hung Applications

So some researchers were looking into how to get MS Word to recover when it gets hung and came up with a general framework to allow users to attempt to get hung applications unstuck.

http://arstechnica.com/science/news/2011/08/jolt-framework-lets-users-force-some-hung-programs-to-recover.ars

This is a great case of a truly useful tool that can be 'an enabler' for bad programming practices. Surely, this is a good tool for end-users. But developers providing this as a workaround for problematic code can be a bad strategy.

Removing Bias From Your Strategy

While this article is geared towards corporate strategy, many of the concepts can be applied to product strategy, test strategy, etc.

http://www.mckinseyquarterly.com/The_case_for_behavioral_strategy_2551 (requires free registration)

Here are some of the tips provided.
- The process of developing the strategy is one of the largest contributors to success
- Change your 'angle of vision' to see the issue from different perspectives. For example, Apple is famous for fully developing multiple ideas for the same feature to come up with a single implementation that fully implements the new feature.
- In most organizations, simply expressing high confidence in a plan is enough to get it approved. However, that does not result in success. Approval should be based on a clear recognition of the uncertainty involved.
- Shake things up, such as setting 'stretch goals' that are not possible given the current ways of doing things. This can spark innovation and efficiencies.
- Call out 'Silo thinking'. If departments hold onto their own departmental interests at the expense of company interests, call it out.
- Allow for 'Safe Debate'. Group-think can elimnate dissent, but sometimes dissent is needed. The key is to be able to do it without personalizing it.

There are more interesting topics on this site. Look around and enjoy.

Happy Sys Admin Day!

http://sysadminday.com/

The last Friday of July is the unofficial 'Sys Admin Day'. Go find the folks that ensure your servers stay running and give them a huge THANKS!

If that's you, keeping those systems up. I Thank You!

Using Pairwise Testing to Minimize Testing and Maintain High Feature Coverage

Problem
You may run across a situation where there are so many combinations of settings to test, it's virtually impossible to test every combination in the time available. Let's consider an example.

Suppose you are testing a product that has printing capabilities and you want to validate that the printing functionality works. You support multiple printers on multiple operating systems using multiple web browsers. In our ficticous product, let's use the following supported components:

Operating Systems
- Windows 7
- Windows Vista
- Windows XP SP 3
- Mac OSX 10.5 (Leopard)
- Mac OSX 10.6 (Snow Leopard)
- Mac OSX 10.7 (Lion)
- Ubuntu 11.0.4

Browsers
- Opera (latest)
- Firefox (latest)
- Safari (latest)
- Chrome (latest)
- IE 7
- IE 8

Printers (you have selected these to work on the supported OS versions)
- HP Inkjet
- HP Laser
- Epson Inkjet
- Epson Laser
- Lexmark Inkjet
- Lexmark Laser
- Canon Inkjet
- Canon Laser
- PDF (file)

User Role (Not testing has caused problems in the past and needs to be added)
- User
- Manager
- Administrator


To test every configuration that would be 7 OS versions x 6 browser versions x 9 printers x 3 roles = 1134 individual combinations. You may notice that IE won't work on non-Microsoft OS, but we'll discuss that in a bit.

Solution
That just won't do. That could be weeks of testing. Let's use the 'allpairs' tool , you could use anything from http://www.pairwise.org/ . I created an input file that listed each component type as columns and it identified 66 test cases that provide good coverage of the supported configurations.

Here is how my input file looked.

OS browser printer role
Windows 7 Opera (latest) HP Inkjet User
Windows Vista Firefox (latest) HP Laser Manager
Windows XP SP 3 Safari (latest) Epson Inkjet Administrator
Mac OSX 10.5(Leopard) Chrome (latest) Epson Laser
Mac OSX 10.6(Snow Leopard) IE 7 Lexmark Inkjet
Mac OSX 10.7 (Lion) IE 8 Lexmark Laser
Ubuntu 11.0.4 Canon Inkjet
  Canon Laser
  PDF (file)

If you enter your data in a spreadsheet, copy the data and paste into notepad, it will be in a format that allparis likes. Other tools will have different input methods and the results will be the same.

The output is another tab-delimited file and if you paste the results into excel, you'll see the specific tests.

case OS browser printer role pairings
1 Windows 7 Opera (latest) HP Inkjet User 6
2 Windows Vista Firefox (latest) HP Inkjet Manager 6
3 Windows XP SP 3 Safari (latest) HP Inkjet Administrator 6
4 Windows 7 Firefox (latest) HP Laser Administrator 6
5 Windows Vista Opera (latest) HP Laser User 5
…
63 Windows XP SP 3 ~Chrome (latest) Lexmark Laser ~Manager 1
64 Mac OSX 10.7 (Lion) ~Opera (latest) Canon Inkjet ~Manager 1
65 Mac OSX 10.7 (Lion) ~Firefox (latest) Canon Laser ~Administrator 1
66 Ubuntu 11.0.4 ~Opera (latest) PDF (file) ~Administrator 1

With this 94% reduction in the number of tests, you can expect to find the vast majority of defects.

Theory
From the pairwise.org site:

Pairwise (a.k.a. all-pairs) testing is an effective test case generation technique that is based on the observation that most faults are caused by interactions of at most two factors. Pairwise-generated test suites cover all combinations of two therefore are much smaller than exhaustive ones yet still very effective in finding defects.

If you look at the far-right column of the output table, you'll see a 'pairings' count. This count identifies how many unique pairings of configurations exist in that test. At the top of the list, there are 6 and 6 unique pairings. At the bottom of the list are only single pairings. You can use this information to reduce the number of tests even further. By eliminating tests that have few pairings, you lessen the impact on the overall testing coverage.

There are tools listed that will use more than two factors to limit tests and these can be used when there are lots of columns in your tables. So instead of finding pairs of configurations, you would be finding triples or quadruples of configurations. This increases the likelihood of finding defects, but you will have to contend with 'The Law of Dimishing Returns' in that you'll have to run lots more tests for smaller gains in coverage.

A Smart Battery That Was Too Smart

Recently, a security researcher found that apple's Smart Battery firmware could be compromised to infect the laptop it was used in. Yikes!





http://arstechnica.com/apple/news/2011/07/how-charlie-miller-discovered-the-apple-battery-hackhow-a-security-researcher-discovered-the-apple-battery-hack.ars

Bugs!

(image from http://www.clker.com/)

I'm not sure if this qualifies as an allegory or not, but here goes...

(Disclaimer: If you are concerned about the killing of bugs, please do not read further)

I came to work this morning to find a half dozen small ants at my desk. I had heard others having the same issue, but they were several desks away.

My first idea was to mash them. I quickly learned that this was pretty ineffective. I then spot-checked desks nearby. A desk on one side had as many (or more) than I did, but nobody else in our immediate vicinity did.

I decided that I needed to get something that would be more effective. Arriving at the store, I went to the home pesticide area. There were lots of pesticides, including about a dozen that would work on the ants.

Fortunately, there was someone who worked there that pointed out a couple products.

"This trap didn't work because they just wouldn't go in"

"I used this other one and it's been two years and they haven't been back"

Sold!

I considered other options while I was there. There were several types of traps that would attract the ants and then keep them from escaping. There were several products that would kill them outright and others that would feed them poison to take back to the colony to kill them and the colony. I stuck with the colony-killing product that was recommended and came back to work.

There were instructions, but I didn't use them :P . I put the poison in several places. As the ants kept coming, I squashed them. Over the course of the day, the ants at both desks dropped dramatically.

The only question remaining is what will my desk look like in the morning? Would they have scattered during the day anyway?

One careless strcat?

http://www.globalnerdy.com/2011/07/20/best-coding-practices-poster-1-only-you-can-prevent-buffer-overflows/

I saw this one yesterday and a new poster today. Very fun!

Stuxnet

Thanks to my friend Frank, I was able to get to read up in Wired on how Stuxnet was reverse-engineered and what was found when they did so.

http://www.wired.com/threatlevel/2011/07/how-digital-detectives-deciphered-stuxnet/all/1

Be ready to spend some time reading, it's not short, but it's engaging the whole way.

Today in Computing History - July 13 - The Smart Ship That Went Dumb

https://sites.google.com/site/todayincomputinghistory/google-gadget


"In an attempt to promote the stability and utility of the Windows NT platform, a new 'Smart Ship' was fitted with a Windows NT-based network. When one of the systems crashes, a domino-effect takes down all critical systems, leaving the ship unable to move for about an hour."

This is one of the events that I added to my 'Today in Computing History' project. Have fun!

Security Links

First, let's start with something fun. This link is to a t-shirt design that shows bad code, why it's bad and what can happen if someone mis-uses it http://cwe.mitre.org/community/tshirt.html

Next, in order to make security testing / mitigation a priority, there needs to be some way to measure it. Here is a link to a group that is working on this http://measurablesecurity.mitre.org/ One standard that is currently used is CVSS http://www.first.org/cvss/cvss-guide.pdf

NIST (National Institute of Standards and Technology) and You

There are so many documents NIST has created that they need a document to list all the documents!http://csrc.nist.gov/publications/CSD_DocsGuide.pdf This list includes the "Technical Guide to Information Security Testing and Assessment" http://csrc.nist.gov/publications/nistpubs/800-115/SP800-115.pdf

There's even a database of various checklists at http://checklists.nist.gov/ One example is, .NET security configuraiton checklists http://web.nvd.nist.gov/view/ncp/repository/checklistDetail?id=7

All of this information stems from looking at a tool called Retina, from eEye.

This tool allows you to scan computers on the network to look for known vulnerabilities. The scans and results provided are best interpreted in light of the links above.

Enjoy!

Trusted Identities

As part of my development as a Software Quality Assurance tester, I've been building my skills in the Software Security arena. I ran across this US national strategy initiative on Trusted Identities.

Consider a world where you can apply for a mortgage and sign for it digitally and use that same infrastructure to anonymously connect to a private online discussion. There are lots of problems yet to solve in regards to this infrastructure, but here are some useful links to look into this further.

http://www.nist.gov/nstic/NSTIC-Why-We-Need-It.pdf

http://www.nist.gov/nstic/

Job Posting - Sharepoint Admin, Oklahoma City

If you know of someone with Sharepoint experience, please send them Todd's way. You'll find his contact information at the bottom.

============================================

Sr. SharePoint Administrator - Oklahoma City, OK

Primary Duties & Responsibilities:
•Serve as Administrator for SharePoint 2007, SharePoint 2010
•Serve as Administrator for FAST for SharePoint 2010
•Plan and lead application system installations and upgrades.
•Monitor, support and diagnose application systems functioning to ensure specifications are met.
•Modify existing application systems to correct errors, allow it to adapt to new hardware, or to improve its performance.
•Develop procedures for application system testing and validation to ensure system integrity.
•Prepare and review appropriate documentation to record computer specifications, layouts and peripheral equipment modifications.
•Design and perform capacity planning processes to ensure performance is in line with business expectations.
•Analyze user needs and design software requirements within time and cost constraints.
•Develop and support business relationships with project team and business customers.
•Develop and implement policies, procedures and standards to ensure conformance with information systems goals and procedures.
•Report status, issues, and timelines to management staff and project teams.
•Lead multi-function teams in problem resolution, evaluation and implementation of solutions.
•Provide mentoring and training to associate systems administrators and system administrators.

•Strong knowledge of enterprise search concepts
•Strong knowledge of web server technologies/frameworks/concepts (HTTP, networking, Windows Server 2008 R2, Active Directory, PowerShell)
•Demonstrated ability to solve complex technical issues, working alone or in a team
•Demonstrated ability to work in a fast-paced, team environment
•Ability to communicate effectively with all levels of co-workers, clients and other external contacts
•Demonstrated ability to document system architecture, installation configuration, methods and procedures for cross training and troubleshooting
•Excellent written and verbal English communication skills
•Experience with Microsoft FAST Search is preferred
•Experience with Microsoft SharePoint is preferred
Education & Work Experience

Minimum Requirements:
•Bachelor's degree in Computer Science or Management Information Systems or equivalent experience.
•Experience generating or adapting equipment and technology to serve user needs.
Minimum of 4 years Systems Administration - SharePoint experience

Contact:

Todd Laduron
Sr. Technical Recruiter - CDI IT Solutions
602-508-6471

Did you ever...?

Did you ever find a site that you went to start to get slow, buggy or that people just seemed to stop responding to you as if they all agreed to do so at the same time?

It may not be just your imagination or that the site is having issues. You may be flagged as an abusive user and these techniques are intentional ways to convince you to calm down and re-think your approach.

I was reading about the stackoverflow.com site (and related sites) where something needed to be done with the small set of users that were making the site unpleasant for everyone else.

http://www.codinghorror.com/blog/2011/06/suspension-ban-or-hellban.html

But what does this have to do with QA? Well, consider what happens if your site behaves erratically, is slow and error prone. Whether it's intentional or not, it drives users away. Do you know how often this happens? What's the impact when it does? How do you bring this to light?

Just something to consider.

Incredibly strong passwords that are easy to remember

Steve Gibson is an interesting fellow that has been around quite a while. One thing he does when not creating commercial products is doing security-related research. He has come up with a way to make strong passwords easy to remember.

https://www.grc.com/haystack.htm

Consider these two passwords:

B0P4LzSzVZ4GgWiSZ5z2

.......D1mw!t.......

According to his calculations, it would take a thousand times more time to crack the second password than the first, even though the first is easier to remember! (check it out for yourself by going to the link above) The idea is that if you take what we typically use for a secure, but memorable password, and padd the front and back with something, anything easy to remember.

For example, take H3!!0 and make it something like one of these:

...H3!!0........

[]H3!!0[][][][]

~~~~H3!!0~

***H3!!0****

<-><->H3!!0<-><->

H3!!0

This increases the length of the password and minimally increases the ability to memorize it.

Job Posting - Norman, OK

If you are in the Norman area and considering opportunities in Quality Assurance, here is one through Tek Systems. They have supported this group for several years and I am very appreciative of that support.

The contact information is at the bottom of the description, please feel free to contact Chris if you are interested.

-----------------------------------

The Quality Assurance Analyst is responsible for conducting quality control activities for the IT Services Department. The QA Analyst will analyze application systems, create test case documentation, and execute plans. The successful candidate will have proven experience ensuring that complex software applications are performing as designed. Running queries in SQL. Score well on SQL assessment. Needs to proficient and score 5-6 on a scale to 10. Open up SQL scripts and understand it, do joins, deletes, updates, executes in outer joins, unions. Have SQL Fundamentals.

• Develop and maintain test plans, manual and automated test scripts for user interface, functionality, system and "ad-hoc testing".
• Execute regression tests, functional tests and data tests.
• Document quality assurance practices.
• Provide analysis of test results and deliver solutions to problem areas.
• Ensure that testing activities will allow applications to meet business requirements and systems goals, fulfill end-user requirements, and identify and resolve systems issues.
• Ensure that any new software integration into company systems meets functional requirements, system compliance, and interface specifications.
• Create and execute test scripts, cases, and scenarios that will determine optimal system performance according to specifications.
• Conduct all types of application testing as needed, such as system, unit, regression, load, and acceptance testing methods.
• Analyze formal test results in order to discover and resolve defects, bugs, errors, configuration issues, and interoperability flaws.
• Analyze documentation and technical specifications of any new application under deployment or consideration to determine its intended functionality.
• Produce reports and documentation for all testing efforts, results, activities, data, logging, and tracking.
• Communicate test progress, test results, and other relevant information.

Chris Keller - Technical Recruiter
3501 NW 63RD suite #300, OKLAHOMA CITY, OK 73116
866.585.0552 T 405.254.1723
F 405.415.1460 M 405.819.1310

ckeller@teksystems.com

World IPV6 Day - June 8, 2011

There have been several tests of IPV6 around the world already and this is no different. Major web sites such as Microsoft and Google will offer thier content over IPV6.

Now that all the IPV4 address have been claimed by Regional Internet Registries, it's just a matter of time before they are all used up. There are still questions as to how the IPV6 switchover will occur, but knowing your way around IPV6 will be a new skill to learn.

Book Review - "Open Source Fuzzing Tools" by Gavi Evron

"Open Source Fuzzing Tools" is written by Gavi Evron and does an excellent job providing a broad range of tools used for fuzzing and their basic usage. There is even a chapter that walks you through building your own file fuzzing tool in perl.

Some specific points that I learned from this book:
- There are (at least) three flavors of fuzzing: 1)File fuzzing- where you create malformed files used by the application. 2) Network fuzzing - network traffic used by the application are malformed and sent and 3) Library fuzzing - calling libraries directly with malformed input.
- The more you know about the workings of the application, the better you can tune your fuzzer to generate malformed data.
- The more you know about assmebler, compilers, bytecode, and TCP/IP, the better. These are the building-blocks for fuzzing.

It's possible to run many of these tools, such as wsfuzzer, without a deep understanding of the application and the underlying technologies, but the more you know, the better you are able to understand the output and take appropriate action.

Impressive Presentation Tool

Here is a fun way to do presentations. Let me know what you think of my first one!

Fuzzing on Prezi

Levels of abstraction, from Order to Chaos and back again

"There is a theory which states that if ever anyone discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened." - Douglas Adams from 'The Hitchiker's Guide to the Galaxy'

For years, I've thought about the order and chaos of the Universe as a lens to look at work. There is order in the universe in that there is this one large blob. But it quickly devolves into chaos when you see each of the galaxies. But looking at our galaxy, there is a sense of order in the spiral arms that make it up and the line of stars in the night sky that we see. This devolves again into chaos as we see the scattering of star systems within the galaxy. But order shows itself again in the orbits of the planets around the sun and the seemingly-perfect spheres that the planets and moons make. But a closer look reveals the chaos of the surface of the planet with mountains and valleys. We can continue this journey to the infinitesimile, but you should get the idea by now :)

Now think about working in a company. The company has a simple goal of making a profit. Then there is the chaos of how the organization is split up But in business unit, there are often annual goals which are very specific. These are to be met by a variety of teams that must figure out a plan to meet these goals, which you as an individual have specific annual goals. However, on a day-do-day basis, your time is spent doing many things, only some of which is towards the annual goals directly.

If you are still reading, congratulations! (and thanks for bearing with me while I try to get to the point) So, where do you like spending your time, in the chaos or the order? Where do your peers like spending time? In which location does your company's culture want you to spend your time?

Just something to ponder...

Book Review - "Threat Modeling" by Frank Swiderski and Window Snyder

Even though this book is from 2004, the method it describes for approaching security testing is sound. The basic method is:

• Create a Threat Profile




• Idendity Targets


• Identify Entry/Exit Points


• Build Models based on this data


• Associate Risks with each Target


• Determine how these risks will be tested


• Perform tests and report results


• Repeat

This book is an easy read and there is little technical knowledge that is required to be able to understand the concepts presented, which allows it to be technology-agnostic. However, you will need a bit more technical skill to be able to perform some of the analysis and testing that is described. The examples are helpful and provide a good guide on how do document the models.

I would recommend this book to anyone that is responsible for building, testing or responsible for the security of an application.

FTP is 40 years old this week

The idea of transferring files between computers has been around since networking began, but this protocol is the one that has survived the longest.

Happy Birthday FTP!

Have Fun Learing!

Did you ever want to spend team time to learn a new skill or hone an existing one? Here is a site that lists lots of games to play that are centered on software development teams. http://tastycupcakes.org/ Enjoy!

Just missed TEDxOKC - Doh!

Someone just told me that a TED conference was going to be held in OKC and when I looked it up, I found I just missed it! Ack.

You can go to their website and sign up for their newsletter, twitter feed and facebook page to keep in touch.

Stop wasting time debugging - advice from a sage

"If you want more effective programmers, you will discover that they should not waste their time debugging - they should not introduce the bugs to start with" - Edsger Dijkstra 1972 "The Humble Programmer"

When Edsger Dijkstra said this, he was speaking about a vision of the future where programming practices would drastically change. In this article, he wrote about 'The Software Crisis' and the resulting revolution. He predicted that anyone wishing to deliver reliable software would find a way to do so more quickly, which led to the quote above.

How much have we progressed?

Adrenaline Junkies and Template Zombies: Understanding Patterns of Project Behavior - a review

I've been reading "Adrenaline Junkies and Template Zombies" by the whole Atlantic Systems Guild crew, Tom DeMarco, Peter Hruschka, Tim Lister, Steve McMenamin, James Robertson, and Suzanne Robertson. You'll recognize Tom DeMarco and Tim Lister as the authors of Peopleware.

This is an easy and enjoyable read where you can see different patterns (and anti-patterns) for people, teams and organizations. Anyone at any level of the organization that is interested in trying to understand how people and teams work would enjoy this book.

Among the great quotes they cite in the book is this one.

“The correct amount of anarchy on a project is not zero.” —Mike Mushet

You'll laugh when you see some of the bad behaviors "others" engage in and cry when you see yourself. The book is a few years old, but it is something I expect will stay on my bookshelf for a very long time.

The roots of Design Patterns go as back as far as the 1960s

If you are not familiar with design patterns, look here.

In 1964, Christopher Alexander wrote a book called 'Notes on the Synthesis of Form' for architects and civil engineers to help them cope with incomplete and contradictory requirements during their design process. He suggests they simplify the process by looking at just the abstract elements and build a model that meets the basic requirements. This is followed by his contribution to the 1977 book 'A Pattern Language' where these abstract elements are standardized into a pattern that is applied to the overall design. This second book strongly influenced a paper on some patterns for smaltalk in 1987 which influenced the seminal work, 'Design Patterns: Elements of Object-Oriented Software' in 1994.

Hands-on training for Software Security!

OWASP has a project called WebGoat that allows you to learn about software security by testing and fixing an intentionally insecure web site. http://www.owasp.org/index.php/Category:OWASP_WebGoat_Project Leave a comment if you've worked through this and let everyone know your experiences!

The Stack is 54 years old! (at least the German Patent for it is)

The following link is a copy of the Patent application for the use of a stack to perform mathematical operations. We take this idea for granted today as a method for processing data, but the idea can be traced back at least this far. http://www.fh-jena.de/~kleine/history/software/BauerSamelson-patent-de1094019.pdf

QA Comics

I posted a comic series a few years ago,

http://www.webcomicsnation.com/not-bob/sqa/series.php?view=single&ID=70234

While the following strip isn't the first, clicking on the link will take you to the beginning :), Enjoy!

"If it weren't so sad, it would be enough to make you laugh. But instead it makes you cry"

While the quote sounds like it came from Yogi Berra, it summarizes how you may feel when it takes more than $7 million dollars to cancel a project.

http://randallrice.blogspot.com/2011/03/minnesota-taxpayers-to-foot-725-million.html

Mr. Fails

Here is a link to a story about testing software in the form of a children's book

http://blog.softwaretestingclub.com/2010/12/mr-fails/

It's good, check it out!

The origins of 'Software Engineering'

The first of four articles on the origins of 'Software Engineering' was published today on the Software Testing Club blog. Here is the video that goes along with the article.

Proposal for QA Certification

Poking around, I found this image (click the image for the original blog) that was claiming the quality of a piece of code being described. I realized that lots of us do this, our only claim to the quality of our code is that on one particular day on one particular computer, the code worked as expected. We should formalize this by putting this logo on our products.

What do you think?

What if network lag happened in real life? (video)

http://www.youtube.com/watch?v=ZcP9jiCbakw&feature=player_embedded

"Reflections on Trusting Trust"

http://cm.bell-labs.com/who/ken/trust.html

This is an interesting read. It talks about security by working through the seemingly innocuous act of writing self-replicating programs.

New Language / Compiler Features

Here are some proposed compiler / language options that most modern programming languages can benefit from. These are based on common errors that have occured on systems that I have used and/or tested. Consider implementing these if you feel moved to do so.

Feel free to add your own in the comments.

Feature	Preventing
Find all files - When enabled, this will allow the program to search not only the local computer, but all computers on the network and then through files available through major search engines. One optional parameter is a timeout, given in hours.	"File Not Found" error
Instantiate all Objects - Each object referenced in the project will be instantiated and when the condition is detected where an object appears to have no instance provided, a suitable instance from the pre-instantiated collection of each object type will be retrieved. Care should be taken when this feature is used in conjuction with 'Universal Cast'	"Object reference not set to an instance of an object" error
Universal Cast - Allow the casting between any two types. How hard could that be?	"Cannot implicitly convert type 'a' to 'b'" and "Type Mismatch" error
Expect the Unexpected - When enabled, the compiler will just ignore anything it doesn't expect.	"Unexpected character "?"" error
Trust Everyone - As a society, we are trusing each other too little. This can cause all sorts of issues with regards to full and complete access. When this feature is enabled, all means (legal and otherwise) are used to obtain the information or resources requested.	"Access Denied" and "Unauthorized" errors
Pass Mac - Give the MAC a passing grade, even when it hasn't really tried all that hard. Eventually, this will be someone else's problem.	"Vewstate Mac Failed" errors
Handle all Exceptions - Surely this error has been seen before. When enabled, the program will search online for a solution related to handling this exception and do that.	"Unhandled Exception" errors
All Knowing - Nothing happens by accident. So finding out the error can't be all that difficult, right?	"Unknown Error" errors
Virtual Valium - Instead of freaking out when there's an issue, give the kernel some virtual valium when it starts to get confused.	"Kernel Panic" error (aka BSOD)
Allow Infinity - When enabled, allow infinity to be used in calculations. For the purposes of this feature, there is no distinction between a positive infinity and a negative infinity.	"Index out of Range" and "Attempt to Divide by Zero" errors
Ethics Committee - When this is enabled, the Heap is investigated for corruption prior to the corruption starting to ensure that all ethics guidelines are followed.	Heap Corruption error
Stack Bowl - when enabled, the stack will be placed in a bowl to catch any overflow. This bowl will also be partially filled to fill in the stack when 'underflow' occurs.	Stack Overflow\Underflow
Auto-initialize everything - For times when you think "I just declared that, why can't I start to use it?" (Thanks to Shmuel Gershon for this new feature)	Errors when using variables before initializing
Do/Catch - As Yoda says "Do or not do, there is no try". (Thanks to Dwain)	Try/Catch
.correctSpelling, .toPresentTense, .toPastTense, .toFutureTense, .toSingular, .toPlural, .toFirstPerson, .toSecondPerson, .toThirdPerson etc. - Allows you to validate responses (such as from security challenges) with users that have difficulty remembering the spelling, tense, etc. of their original answers. (Thanks to Chris/Chad)	Security challenge response validation issues.

Upgrading Windows Video

Here is a summary of upgrading to every major release of MS windows. Ah... memories...

http://www.youtube.com/watch?v=vPnehDhGa14

Sometimes I feel like the machine, sometimes the monster

"World History in UML Diagrams" - the book

In 18 months, this book should be released. I'm a bit at a loss for words.

http://www.amazon.ca/World-History-UML-Diagrams-Structural/dp/1908043083

Masterpiece Engineering - a commentary on 'Engineering' Software

The 1968 NATO conference on Software Engineering was the first for this area of study to use this name.

Not everyone was convinced that a formalization for producing software was possible or desirable. Here is a paper submitted to that conference which expresses some of the concerns.

http://www.bobbemer.com/DAVINCI.HTM

CHI to Amazon's Mechanical Turk and beyond

In 2005, Philipp Lessen posted an idea on his blog called CHI, where a system is created that automates asking questions to humans. This may seem dense, but the idea is that these questions are ones that humans are better at answering, such as , 'is this photo of a person a man or woman?' or 'Does this description match the photo?' or 'Does this description make sense?"

Six months later, Amazon implements this and it's now known as Amazon's Mechanical Turk. The idea is that tasks that are best done by humans are packaged up and given a price. Workers then can pick up those tasks and get their fee. There are controls in place to ensure that the workers do a good job and are actually qualified to do the work being requested.

Now there are similar sites that are generally known as 'Crowdsourcing'.

But what does this have to do with software?
Suppose you had a system that tied into your automated test framework that would offload some of the more tricky parts of your tests that would be simpler to do manually, but it was just one step out of dozens?

Suppose you wanted to automate the layout of a webpage on multiple browsers under different conditions as part of build testing. You could write the specific rules for what consituted 'improper' layout, but that could take lots of time and lots of tweaking. Wouldn't it be nice to put in your automation:

assertThat(baselineimage.looksLike(myImage), true)

and that call shows both images to you for the determination that they are equivalent?

Sure there are issues, you may not be ready to do these steps when they are being run. Or you may take too long to decide. It's not clear the full extent of the useful features, but there may be some thing here to assist with your testing.


What are your thoughts?

Model-based Testing Tools

Here is a handy chart of model-based testing tools to consider using if Visual Studio isn't in your development toolset.

http://www.cs.waikato.ac.nz/research/mbt/Tools.pdf

Links, Links and More Links

I ran across an article that bloomed with interesting links and such. Here are a few.

Enjoy!

Continuous Delployment
http://engineering.imvu.com/2011/01/19/buildbot-and-intermittent-tests/
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
http://eng.wealthfront.com/2010/05/deployment-infrastructure-for.html
http://codeascraft.etsy.com/2011/02/04/how-does-etsy-manage-development-and-operations/

Automated Testing
http://www.quora.com/What-kind-of-automated-testing-does-Facebook-do

Tracking Metrics
http://etsy.me/e1ULhO

Strata 2011 (Large Data strategies and tech)
http://www.youtube.com/view_play_list?p=EF277D84FE2A28D5

Perform Calculations on Encrypted Data

Here is an article on a plan for performing calculations on encrypted data. You could:
- farm out calculations to untrusted worker processes with little fear of revealing the contents!
- maintain data security in memory!
- much more!

http://www.acm.org/press-room/news-releases/2010/dd-award-09

The last batch of IPv4 addresses have been sent to registrars

It's expected that sometime later this year, there will be no more IPv4 addresses for ISPs to hand out. The last blocks of addresses were sent to regional registrars. Moving to IPv6 has started, but very slowly. We should see some visible tests by Yahoo! and Google in the next few months.

http://www.zdnet.com/blog/networking/don-8217t-panic-it-8217s-only-the-internet-running-out-of-addresses/656

Auto-generate Test Cases (and have specification documentation to boot!)

Suppose you could create specification documentation that could generate test cases automatically. Suppose that doing this would reduce your testing effort by roughly 40%. Suppose that you could also see requirements coverage.

What would you pay for this tool? How about $0?

Spec Explorer has been available for several years and is available for both Visual Studio 2010 and 2008. There is also an earlier stand-alone version that works well, though it lacks some of the integration features and newly-developed features.

Check it out, give it a try and see if it doesn't make your life easier.

Download version for Visual Studio 2010
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=a4649791-a77c-4498-b392-df2ad2b2413f

Stand-alone version of Spec Explorer and Introduction
http://redearthqa.blogspot.com/2009/06/spec-explorer-validate-your-model-and.html


Series of webinars on Spec Explorer
http://channel9.msdn.com/Blogs/nicok#tab_sortBy_recent

Cloud Computing - Getting Started

Regardless of your role in your development process, you may well be doing cloud compuing in the next few years. If you are using Microsoft's Azure for cloud computing, here is a good starting point.

http://windowsazurebootcamp.com/

QA Achievement Levels

In my previous post, I pondered whether measuring skill through pre-defined Achievement Levels was a good idea or not. After deciding to have some fun with this. Here is my list of QA Achievement Levels that could be used on your team.

What would you add to the list?

Holistic Detective - You identify a test case that identifies in a major defect which requires no less than 80% of all current functionality to reproduce.
Sisiphus - You have run through the same test cases for the last ten releases with no new defects found.
Epic - the test cases you identified for a feature have an execution time measured in person-years.
Nailed It - A feature you tested has been in production for at least a year with no defects reported by the users.
Guru - You submitted a defect that was fixed and verified without the need for any clarification by the developer.
Borg - More than half the defects you report are as a result of automated tests you have written or automated testing tools you have used.
Cassandra - You have correctly identified the modules that will cause the biggest support headaches when released and nobody believed you.
Dead Parrot - You have an extremely difficult time convincing the developer that their ‘fix’ does not, in fact, fix the issue. After several hours of showing all the ways that the issue still exists, you are offered a slug.
Jar Jar - Every bug you submit requires clarification. For this, you are made team lead.

Measuring Skill

While the following list of 'achievements' is funny, is this approach helpful?

http://blog.whiletrue.com/2011/01/what-if-visual-studio-had-achievements/

Can our skills be measured in such a cut-and-dry way?

You could think about it, or just have fun with the list.

Is it me, or am I getting old?

For kicks, I looked up the first set of web pages that I created, and archive.org had them.

http://web.archive.org/web/19961110051852/www.math.unt.edu/

This snapshot is from 1996, though the original pages were 1994 or 1995.

A couple notes about these pages.

- The buttons were all manually-created (with some long-forgotten editor, possibly a early version of Photoshop)

- The logo image was from some mac-based program (again, I forget the name of it)

- All the html code was hand-generated. If you look at the faculty page, you'll see the dl, dt, dd html tags, which are rarely (if ever) used. (these were intended for lists of words and their definitions)

I don't often need to use this knowledge to create new content, but it does help me understand why a page acts like it does when I'm testing.

The Happiness Metric

There are lots of studies (cited in the links below) that show that happy team members make projects successful. Jeff Sutherland talks about the 'Happiness Index' and how it is used to make sure the project is on track.

http://scrum.jeffsutherland.com/2010/11/happiness-metric-wave-of-future.html

http://scrum.jeffsutherland.com/2010/12/scrum-inc-sprint-2-retrospective.html