Don't worry, while there's more to do, we are well positioned to make the most efficient use of our time with the remainder of the tasks.
Step 3 - Identify test cases / test data
For the purposes of this discussion, let's limit the concept of a 'test case' to a 1-2 sentence that describes the goal of each test you want to achieve. Start with the high-risk areas first. Depending on your situation, you may do only the risk level 1 items, or you may have time to do more. The answer depends on available time. Once you've built some test cases, you'll want to take a step back and build some models of your system to see if there are any more test cases you could write.
You'll also want to think about test data. You'll want to consider valid data, invalid data, large quantities of data, lack of data, etc.
For our example with the calculator, here are some preliminary test cases.
- Perform valid basic arithmetic calculations
- Perform 'invalid' basic arithmetic calculations (divide by 0, invalid input, etc.)
- Perform scientific calculations
- Perform 'invalid' scientific calculations (log 0, tan 90, etc.)
- Perform statistical calculations
Hmmm.... While I was using the application to identify test cases, I realized that I've missed some of the features
- keyboard shortcuts (risk - 3)
- number format -decimal, octal, etc. (risk - 2)
- number type -degree, radian, grad (risk -2)
- use parentheses up to 25 levels at a time (risk -2)
The Model
Now that I've gotten into this a bit further, it seems that I'm ready to build a model to make sure that I'm covering these requirements. There are several types of models. Dataflow diagram, Workflow diagram, State Transition diagram, etc. At least one would be helpful, so here is one that I came up with.
State - Ready for a Number
Actions
- Enter a '('.
- Enter a 'binary' operator (a 'binary' operator takes two inputs like +, *, etc.)
- Enter a number,
State - Ready for an Operator
Actions
- Enter a ')'
- Enter a number
- Enter a 'unary' operator (a 'unary' operator takes one input like 1/x, sin x, etc.)
- Enter a 'binary' operator
As with all models, this model doesn't tell the whole story, but it is better than having no model. Among it's faults are that it does not cover the statisical mode calculations, the names of the states are not completely accurate, only valid actions are allowed, etc. You will have to get used to these sort of issues to be able to have a strong basis for testing available in a short time.
Planning for Test Cases
For now, let's assume that the starting state of the calculator is 'Ready for a Number'. I have three valid actions, enter a '(', a number or a binary operator. Suppose I choose to enter a number, then my state has changed to 'Ready for an Operator'. In this new state, I have four valid actions (listed above). You will quickly see that there are lots of paths I can follow where I select actions and that action may or may not change the state I'm in (though it will change the value of the calculation being displayed and stored). You can also see that there is no actual ending state, you can theoretically keep doing calculations idefinately.
Here's one example path to follow
- Enter a number
- Enter a binary operator
- Enter a number
In an actual testing situation, you may want to cycle through each binary operation to make sure they are correct. Similarly, you can do the same with unary operators.
It does get tricky when you start to plan for using parentheses. What test cases are best to use? This question will be your constant challenge. It's akin to asking "When are we done testing?". The answer is "It depends". It depends on how much time you have available, the level of quality that you are attempting to achieve, how much coverage of your test requirements you want to achieve, etc.
Now suppose that we've had to make these hard decisions and have come up with the following test cases based on using the model we have built. Note that these are not the actual tests, only the test cases. The tests (or test procedures) include specific data and specific validations.
- Verify unary operators
- Verify binary operators
- Verify nested calculations with both unary and binary operators
- one level of nested calculations (e.g. "500 - (45 / 5)" )
- two levels of nested calculations (e.g. "sqrt (a^2 + b^2)" )
- 25 levels of nested calculations
- verify calculations where operators are replaced by different operators (e.g. enter '2' then '+' then 'x' then '5' and verify it is interpreted as "2 x 5" )
There are certainly more test cases, we have several test requirments that are not covered by these tests. You'll want to make sure that you are covering all the requirements you have identified or be able to explain why tests are not warranted.
Step 4 - Write Tests
There are many different approaches to writing the tests. Depending on factors such as time available, will the same person write it as will run it, what is the knowledge of the person running the test, etc. I'm going to suggest the following format because it's simple and covers just enough to be helpful and not so much to be overly cubmersome.
In a spreadsheet, create the following columns
- Test Name
- Step
- Expected Results
- Actual Results
- Pass/Fail
Now under the row with these headings, but the name of the fist test case. Under that, describe the action you want the tester to take. The action should be unambiguous, specific and only contain an action and not any validation. In the Expected Results column, describe what you expect to happen. Again, be unambiguous and specific. Leave the last two columns for later.
Here is an example using our test cases from above.
You may have noticed that on the second test, I have a step without an Expected Result. In this case, I don't expect anything to happen. I could say that, or I can leave it empty. You will need to determine which way works best for you.
Those that have an opnion on the matter may take exception to these tests as being poor examples of test cases, but for now, they will do.
We're almost done. To be continued.
