I’ve struggled with the right level of automation for a long time and alluded to a level of salvation here. Since that post, I’ve moved on to engage automation using Given-When-Then (and its collateral ideas) much like a sailor engages a lighthouse at night. In my most recent experience, it was more like a sailor is on the bridge watching the light and I’m below deck providing interpretation and direction.
My testing team created an excellent suite of automated regression tests for the stored procedures used in our application. But I’m struggling with the number of tests created for this suite.
It might be that my definition of “regression tests” is more economical than that of my teammates and I own that communication error: I probably was not clear on what I thought this task should produce. But it also goes to the heart of what I believe about automation tests.
In 2007, I, like many others, was enamored with automated tests against the UI of my applications. A steady diet of automation tools over the succeeding years (starting with HP QTP) were both valuable and frustrating. I eventually gave into the frustration because the tests I, and many others, wrote were fragile and numerous. My response was to limit what I automated because inflicting such a level of maintenance on a product team was just wrong.
I watched as other teams continued to indulge in not only creating these tests but to develop frameworks around frameworks that provided some theoretical value and, in my opinion, dubious claims of a positive ROI. I was unimpressed with what appeared to be the emperor (in their “new clothes”) dining happily on over-heated porridge.
With training and some practice in Given-When-Then and its description of business outcomes, I started to see practicality again in test automation. A test that evaluates business outcome is, in my opinion, more stable, valuable, and maintainable than several poorly designed automated UI tests. Then, there are two questions. Always. Two. Questions.
Can I automate this test? Of course, I can automate just about any test. The avalanche of software utilities that pop up around tools for test automation is evidence of over-indulgence in coding. Not testing. The second question is most key: Should I automate this test?
I advocate for tests designed for simplicity and to exercise isolated business outcomes. The tests should be motivated by risk, value, and maintenance. Upon a review of these criteria, I stand the best chance of balancing tests (the most value) and the code (that which must be maintained to deliver the test).
My team and I will meet to review our regression suite. I look forward to exploring differing definitions of “regression test”. I welcome their perspectives that could help all of us discover the combination of test and code that is “just right”.