Unagile Testing
Teaching commitments in New York prevented me from sticking around to hear most of the talks at STARWest this week after my own sessions were over. However I did listen to Wednesday’s keynotes from James Whittaker and Julie Gardiner. Reflecting on them, it struck me that while today’s programmers may be test infected, testers aren’t yet agile infected.
Whittaker’s keynote made the useful point that we need to investigate our bugs to see what we’re doing wrong, which is sensible. However he seems to believe that the only reason to do this is so testers can more easily find bugs. He doesn’t hold out any hope that programmers can find bugs or not put the bugs into the code in the first place. Because all software is buggy, he thinks no methodologies are worth anything. He claims we don’t know how to write perfect code, and therefore we can only hope to detect the bugs after the fact.
Certainly all software has bugs, but not all software is created equal. Some software is demonstrably better than other software; and we can learn good techniques from this software, even if it still has a few bugs in it for testers to find. He’s really suffering from the binary fallacy. The question is not whether software is perfect or imperfect. It’s a quantitative question of how imperfect any given piece of software is. We can strive for perfect software, but we can only achieve better and worse software. But we can achieve that, and it’s still worth studying and striving for. (In a brief hallway conversation after the talk, James did hint that he was deliberately overstating his point to pump up the audience, but there were no such caveats in the keynote itself; even when I raised these points in Q&A)
Julie Gardiner delivered the second keynote on Why “‘Risk’ is a Tester’s Favorite Four-Letter Word.” I don’t have a big background in test theory. I haven’t read a lot of books about testing. (She claims to own over 60, and to have read most of them.) so I may not have fully understood everything she said. However, it seemed to me that she was saying risk is dependent on time spent testing. The more time spent on after-the-fact testing the more confidence you had that the software was bug free. You decide when to release based on how much risk you are willing to accept.
There’s something to that but it’s not how I work, it’s not how I see companies like Microsoft working, and it’s certainly not agile. In agile development, the software is tested and the bugs found much sooner. An agile team normally has a very good handle on how many bugs they have; and they run at a very high confidence level that there are few undiscovered bugs. This is because testing is done simultaneously with coding rather than after the coding.
As far as releases go, a really agile team may release every two weeks. If there are undiscovered bugs they can be fixed and the fix rolled out very quickly. In the world of shrinkwrapped software such fast release cycles may not be possible. However then the question of whether to ship is normally driven by two questions:
1. Are sufficient features implemented to justify the release?
2. Are any known bugs or combination of bugs significant enough to delay the release?
The possibility of unknown bugs just doesn’t enter into it.
Note that I am not saying there’s no place for testers on an agile team. It’s just that I think the testers need to be brought in much earlier. Testing needs to be simultaneous with development, not subsequent to it.
November 21st, 2005 at 5:31 PM
The trouble with the agile no-bugs-ever mindset is that for any sufficiently complex system, there is at least one WONTFIX defect that can be created in the right system state (e.g win98+modem, solaris +jvm1.4.02 or winxp in french canadian)
Now, what do you do with a test that fails? ignore it, or halt until everything is fixed. Or just pretend it isnt there and turn the test off?
November 23rd, 2005 at 11:23 AM
I think the claim that agile processes give you a good handle on your bugs is true only for a highly restricted sense of “bug”. Most of the problems I have found in my software are not because the code fails to do what I intended, but because what I intended is palpably the Wrong Thing in the total context. The flakiest parts of TagSoup have to do, not surprisingly, with its SAX interface. Unit-test-first, though certainly a good policy in itself, would not have helped with these, since they depended on my failure to understand exactly what a SAX parser is supposed to do.
November 23rd, 2005 at 12:07 PM
It’s not just about unit testing and test first development, though. On agile teams that include testers, the testers need to start working on day one, not after the code is written. They need to write acceptance tests first, as part of the analysis and design and perhaps even the contract for the job. Unit tests don’t find everything, for sure. However, if the test suite includes system, acceptance, functional, and integration tests as well, and if these tests are written earlier rather than later, then the bugs will be found and fixed sooner; and the programmers are less likely to wander off in the wrong direction.
In general I think unit testing is vastly overemphasized in test driven development. Testing, testing first, and testing often is important; but I’ve never seen any evidence that unitarity is a critical part of the process, compared to broader tests. Some tests work well as unit tests. Some don’t. An excessive focus on making tests unitary sometimes contorts APIs and discourages essential tests. The problems Gardiner was talking about arise because her testers don’t start work until the coders are done. I want them in the process and writing tests before the coders start work. In fact, I want them in the room and writing tests before the development contract is signed. And then I want them testing and looking for bugs while the code is written. There’s a place for testers in the agile development process, and it’s a much bigger place than they have in the waterfall model. In agile development testers need to be working throughout the development cycle, not just at the end.
May 30th, 2006 at 1:42 PM
Elliot,
Someone mentioned to me recently that you had seen my keynote talk and I am pleased to have found your blog.
Maybe the British accent didn’t help when I was trying to convey my message. Because it won Best Presentation at STAREast I was not able to change the presentation and there was probably too much information for 800 people in the short time I had. I only had 120 in the room in Orlando and there was lots of discussion and more practical advice I could give to the delegates.
My message was the complete opposite of what you think to the presentation and to me. It was to say get involved at the beginning, you make sure that the product risks are understood by all parties and then everyone can move forward effectively because deadlines are sometimes imposed on us. Because risk is endemic within Agile projects, it is acknowledged from the outset and most of the approaches I spoke about would be seen to slow that lifecycle down. I have been involved in Agile as well as more traditional projects and only one of the approaches I mentioned could be considered for agile. However, in my presentation there were details on how testing (through risk) can influence development before it gets to test execution (after-the-fact testing as you call it) within system test; but equally I have also known test managers get involved at the end (which is disappointing). My personal experience has been to get testers involved from day one; if the management couldn’t see the value than it was my job (as test manager) to show what an effective and immediate contribution they could make. To say that my testers have only ever got involved after coding is false.
I would also like to say that a lot of companies are still in non-agile projects and from the feedback I have had from attendees of that talk it has helped get their message across about the value testing can provide on projects (and not just the after-the-fact testing) by using risk based approaches. So it was of some use to some attendees and I am sorry my experiences of risk based testing could not help you.
Whether traditional or agile, testers need to be involved throughout and one of the benefits agile has over other lifecycles is that testers are welcomed, their contribution acknowledged and value shown immediately by generating acceptance tests. The great thing about agile for me is that the scope of the iterations will be directed by business risk and technical risk and the unit tests & acceptance test have to pass before it’s considered ‘done’. That reduces the risk because we should deliver what the user wants.
BTW – my testing book collection is 64 and if you would like a copy of my list and my thoughts on them then please let me know. I am not a testing theorist and have been testing for 15 years, but I believe that no-body knows everything about testing and I never want to stop learning whether from books, people or the real world.
If you would like to discuss it further, please contact me on julie@qstc.co.uk
June 14th, 2006 at 2:39 PM
Please send me a list of your testing book collection (all 64 of them should be in the list).