Test Automation is an Investment

I submit that test automation can improve your project’s pace.  I realize that some believe it helps detect defects or that it can be developed independently from project execution.  I recommend you reconsider those positions.

Ever since I read Specification By Example, my understanding and attitude towards test automation has changed.  It did not change how I design or build test automation, it changed how I see its purpose.  For this post, I refer to test automation in a spectrum from evaluating newly checked in code to that used in regression testing.

I’ve been pitching what I believe is a familiar notion around test automation in testing circles.  But it was suggested to me that I provide some context.  Put simply:

The purpose of test automation is to enable frequent evaluation of application behavior in the face of change.

I did not arrive at this conclusion by myself.  Adzic’s book was a great catalyst to think about test automation more, and to discuss it with other test engineers.  A context took shape for me and its present form is my declaration above.

Defect Detection is a Narrow Focus
Test automation costs too much to make it beneficial in just detecting defects in the short term.  Indeed, test automation developers find defects many times when they are developing (not executing) the automation.  That means that by the time the test automation is completed and verified, it’s possible someone else could have located that defect AND corrected it.

I recommend you invest in your test automation today.  You will realize the real value when it detects a defect in a succeeding iteration, or a succeeding project.  When it does, it demonstrates how recent changes have introduced errors in existing behavior.  In that manner, it is acting to preserve your business intent.

Independent Development Costs More
Let’s say you wait to develop test automation.  At some point in the future (the next iteration or the next project), the testing team is discovering defects in code that was completed AND working.  The current pace begins to slow:

  • Defects in existing code must be corrected so testing can evaluate the newly deployed code.
  • There is a good chance that correcting defects will introduce more defects.
  • When the code is operating again, you may find defects in the newly deployed code.

If there was an investment in test automation, frequent execution would detect changes to existing functionality at the time they were introduced.  That is, defects would be detected when developers execute the test automation, and they would correct them.  This saves time because the project team continues building products rather than fixing products; the project maintains its pace.

Your Investment Pays You Back
Test automation pays dividends in protecting your investments in building and expanding business functionality, and detecting unintentional changes to business behavior early.

  • Time is saved because the team executes test automation often to detect behavior changes early – before the products are deployed for testing.
  • System and integrated testing is more effective because simple defects are corrected before deployment for testing.

Lastly, the project team builds confidence in their products because the testing team can spend more time exploring products broadly, deeply, and for risk.


The Explorer

Imagine you are an explorer.  You often explore jungles and forests, and occasionally cities and villages.  People regard your exploring skill highly but with some reserve.  One day, a builder comes to you who wants you to explore a bit of pasture.  They are interested in changing the pasture so that many people might live there.

  • Given your skill and experience in exploring, when you go out to the pasture, then the builder wants to know your opinion on having people live there.

You decide to visit the pasture.

After a day’s journey, you arrive at the pasture.  From your vantage point, you can see it is bordered on one side by a large hill.  While there are many groups of healthy trees, there are also inviting clearings.  You think that if people lived here, they would enjoy the area.  As you walk around the pasture, you take notice of the types of trees, variety of animals and insects, and appreciate the small river.

Upon your return, you describe what you saw and what you found.  In your opinion, people would be able to live in the pasture.  The builder asks many questions and you respond with your experience in the pasture.  The builder asks if you found any bugs.  You reply that you found many bugs for they were plentiful on the ground and on the trees.  The builder thanks you and gives you a special token of their appreciation.

A few weeks later, the builder asks you to explore a city.  They are interested in making changes in the city.

  • Given your skill and experience in exploring, when you go to the city, then the builder wants to know your opinion on making changes in the city.

You decide to visit the city.

After a day’s journey, you arrive at the city.  From your vantage point, you can see a few buildings, neighborhoods, streets, and people.  Beyond the city limits, there is a forest.  You think that if the city had some changes, the city would readily adapt.  As you walk around the city, you take notice of the population, the clean streets, the commerce, and the services.  You recall the builder asked about bugs so you also pay special attention to the ground and the few trees you find.

Upon your return, you describe what you saw and what you found.  In your opinion, the city was ready for changes.  The builder asks many questions and you respond with your experience in the city.  Some of their questions about infrastructure and services were challenging so you express your regret in not gathering more information about them.  The builder asks if you found any bugs.  You reply that you found no bugs.

Surprised, the builder asks, “Where did you look for bugs?”

“On the ground and in the trees.  The ground was often covered in concrete and there were few trees.”

“Did you look under the concrete, or in the buildings?”

“I did not.  Since I found bugs on the ground and in the trees in the pasture, that is where I looked for them in the city.”

The builder explains how the city is different from the pasture.  While the bugs were easy to find in the pasture, bugs in the city may hide in buildings or houses.  They may also hide in the way services operate in the city.  The builder suggests you review information about how cities are different from pastures.

The builder expresses their disappointment in your report.  They believe the changes to the city will be delayed because your information may not be accurate.  They thank you and leave.

A few weeks later, the builder asks you to explore the city again.  Not to be disappointed, they ask about your understanding of cities.

“I have explored cities occasionally and they contain many things that I don’t understand.”

The builder thanks you and says they know of another explorer more familiar with cities.  They will ask them to explore the city.

Imagine YOU are the explorer.  What would you do next?



An Experiment for 2016

I missed writing.  I think about it every day but it’s like drooling over a pizza ad.  It’s nice but pizza with the family is much more satisfying, leaves an impression, and has more impact.  My inspiration to return came from Julie Zhuo reminding me of many great things I have read about writing – thank you Julie!

I focused a bunch of time on building a new web site for my hobby.  I like building it and I like testing it.  It gives me an opportunity to stretch my testing muscles lest they atrophy from the daily experience and negotiation of leading a testing team.  I enjoy the test lead role but when I need to do something to relax, I code and I test.

With a large and successful project behind me, I had an opportunity to catch up with our testing community.  Notably, I finally read Specification By Example (SBE) by Gojko Adzic (I realize I’m late to the party on this one).  I was seven chapters deep and realized I’ve miss so many opportunities to implement SBE ideas.  Then, there was a post on an internal blog recommending all testers read the book.  There was a momentarily satisfaction that my actions and the universe had aligned.

Ever since I started automating test cases, I was apprehensive about their existence beyond the end of the project.  We live in a very dynamic industry so the solution we implemented would inevitably change.  My tests would fail when change hit some critical mass.  I resolved to do better but better was to recommend caution around what was automated.  My constant challenge was writing automation that could survive change AND deliver a valid result.  I was a little successful but not always satisfied.

SBE crystallized my reasons for cautious automation.  I was unable to articulate why because I hadn’t found a suitable alternative.  But the logic and wisdom of automating business outcomes resonated with me.  The business outcome has a much lower frequency of change so automation makes sense.
It also defined why I had such apprehension.  What I have seen is automation of not only business outcome, but automation of test cases that provide little information just because they could be automated.  I also saw automation evaluating lower level functionality – which is valuable – but it was executed from the UI.  Both of these added technical debt and impeded project pace.

With the new year and a new project, I’m ready to experiment (it feels more like a hurricane of righteous purpose but I’ll calm down after a bit).  I discovered that our Requirements/Business Analyst community also favors the approach.  So far, I’ve suggested SBE to two other projects and the response is positive.



The Power of “I don’t know”

My teammate, an experienced and wise System Architect, related an experience from a meeting where they reviewed and discussed processes.  He provided feedback along with everyone else.  When they read back everyone’s contribution, his was missing or skipped.  He asked about it and they assured him his feedback was included.
When the facilitator emailed the minutes of the meeting to the group, he found that she placed his suggestions under the “Other” category.  Mildly frustrated, he responded simply “Sounds good”.  A few minutes later, the facilitator wrote back.  She said she recalled some of his comments but they were a little technical for her.  She was hoping he might provide more detail through email.  In other words, she didn’t understand what he said during the meeting, and did not feel comfortable enough to ask for clarification.

When I hear things like this, I get frustrated, and then motivated.  In this case, I was motivated to encourage people to express “I don’t know” more often.


I was overwhelmed and humbled at the response – thank you!

I want to encourage you again.  When you aren’t following along with a concept, or have trouble grasping an idea, or the language is unfamiliar, you have two opportunities: explore the limits of your fear, and help others who may be experiencing the same.

Form your response clearly in your mind, politely request the speaker’s attention, and say it: “I didn’t follow…”, or “I don’t understand how…”, or “I’m unfamiliar with some of those words, would you…”.  Make it clear that you “don’t know”, and that you really want to know!

The Flip Side
If you are the speaker and a participant in the meeting, the lecture, or the conference presentation bravely confronts their fear with strength and humility, you have, in my opinion, an obligation and an opportunity to engage them fully.  Welcome their comment as you would feedback from a peer for that is what it is.
Neither of you is uniformed/lost/dumb; both of you want to communicate and understand because you both found value in some expressed idea.  Messages between people are garbled for a variety of reasons.

“Thank you for your honesty”, I would start.  “Where might we begin again?”

The Most Important IT Skill

My wife and I attended Curriculum Night at the high school where our boys attend.  It had already been a busy week (and frustrating one if you count the removal of a poorly designed kitchen faucet).  But Curriculum Night is something I have enjoyed for many years and would not consider missing it!

My son is enrolled in AP Programming so I was very curious about the curriculum AND the teacher.  I was impressed with the cognitive level of assignments he had last year, and this AP course had my expectations high.  I sat through informative yet similar sounding sessions on Pre-Calc, English, and Physics before hearing about the AP Programming course.

When I arrived for the session on AP Programming, the teacher greeted me with a hardy handshake, an excited “Welcome!”, and invited me to sit where my son usually sits.  He was animated and nerdy (cool).  He had spent ten years in the industry before deciding to teach (nice).

He started talking about the curriculum, noted the students receive one new programming concept per week (they are working on arrays this week), and finished with how their work is evaluated.  My interest grew.

Students are expected to review each others’ work, note the differences between the implementations, and they are invited to speak in front of the class about their discoveries.  The teacher stated the ability to communicate was as important as learning how to write programs!

I was elated.  An experience in collaboration and empathy respectfully facilitated by the students is a great IT lesson.  In my journey through my testing career, I discovered that, as a Test Lead or Tester, testing is a very social activity.  The ability to convey and negotiate ideas, collaborate with multiple personalities, and develop a working team has been as rewarding as writing programs and tests.  Getting that experience in high school is like preventing defects early in the project!

Testing is not for the Shy

I recently joined the Test Design Exploration team.  This is one of many groups within our enterprise available to mentor testers.  Our charter is to explore test design techniques, provide opportunities to use them, and discuss how techniques might be used in projects.  We establish a schedule of test design topics and testers sign up to attend sessions.  For example, we’ll be exploring All Pairs in an upcoming session.

Our team has been meeting regularly to prepare the sessions and set up a schedule.  I like our group – they are very experienced and bring some fresh ideas to our topics.  Recently, we provided a status update and some clarifications around our scheduled start date.  With the business complete, someone suggested that our team create a SharePoint site where testers might ask questions of our team should they be to shy to ask during a session. I liked the idea of having a place to follow up from our sessions.  Accessible to everyone, it makes our team more transparent, and I think we are willing to support it.  What stuck with me was the description of a tester to shy to ask a question.

When I started as a developer in this organization a while back, the fact that there was a whole group of people dedicated to testing was new to me.  I thought it was pretty cool!  But I rarely saw them during a project and heard from them even less.  When I moved to a testing job family, I tended to be very vocal.  Actually, it might have been called pushy or obnoxious.  But with a lot of feedback and practice, I found a testing voice that is more collaborative.  I also learned that testing demands more of a presence from a tester. I suggested to my test design teammates that we should encourage testers to ask questions.  The sessions are a safe place to do this, asking questions in a group gives them great practice and confidence, and maybe we can deliver more than test design mentoring.  They all agreed.

Testing is not for the shy.  There are developers that will push you around with very technical language, dispute defects, question the credibility of your tests, or just ignore you.  There are project managers that will pressure you to reduce testing.  In my opinion, this is not the identity of a tester.  Take back your identity!

You are a tester and quite capable of

  • Learning C#, Java, or any other language
  • Defending your defects both at a technical level and, more importantly, at a business level
  • Establishing credibility through early collaboration and shared goals
  • Being a contributing member of any project team
  • Expressing the risks of reduced testing

I’m looking forward to exploring test design with testers in my organization.  But I think they can learn more by asking questions in the sessions or anywhere they go.  The bigger challenge for me is encouraging them to believe in their capabilities, and to have them respectfully act on the behalf of the quality in the products they evaluate.

My Holiday Binge

I confess.  During the holidays, I binged.  Not only did I binge, I ignored some work required for an upcoming presentation, I didn’t write or blog, I didn’t shave, and I didn’t get regular meals for almost four days.  As you might have guessed, I was profoundly indulgent in writing and testing software.

I know what you’re thinking.  I could have said no, I could have more self control, I could have asked for help.  But I saw my new laptop sitting there, Visual Studio Community Edition 2013 looking sleek and installed and all, and an engaging hobby project at the ready.  “I’ll try just a few experiments for feasibility”, I told myself.  “It won’t take long.”

I created my first project and successfully embedded Google Maps on the web page.  “Wow”, I thought, “this is easier than I thought.”  I opened another project to explore the use of a web service for retrieving data I could place on my map.  All I had to do was port some code I already had (no one would know and I would save time).

I copied the code into my web service project and gave it a try.  Working.  I added more code to accommodate the web service and gave it a try.  Working.  I needed more, these small successes are great.  More is good, right?  After a few iterations of coding and testing, it was working well enough to integrate with the first project.  “This is going pretty well – why stop now since I’m so close?”

I had my web site call the web service.  Error.  I Googled the error and found a feasible resolution.  A small success.  More, I need more.  I added a secure call, a database connection, and some simple UI interaction.  Working.

A welcome development buzz was setting in.  After weeks of thinking about this project, planning its construction, designing an interface, the pent up creativity was released.  The cycles of build and test fueled my desire to continue and laser-focused my attention.  The total design was becoming clear and I had to pursue it.

I had enough code working that I could deploy to my website.  “This was it!”, I realized, “I would see my dream a reality.”  I elevated all the files and navigated to the page with the map.  I entered my information and submitted it.  I had such anticipation to see the data on the map.

The failure was frustrating.  My ISP did not allow web services on shared accounts.  I should have known but I didn’t care. I was already considering alternative solutions.  Somewhere through my thoughts, I recalled the Raspberry Pi my son had.  “You could run a website on it, Dad,” echoed in my head.

I investigated the Pi and ordered my own.  “I can experiment with his – he won’t miss it.”  Suddenly, I was working, most clumsily I might add, in Linux (download a TTY application; I tested and coded).  The web service could be written in Java (download an editor; I coded and tested).  It had to be deployed through something called Tomcat (install on the Pi; I coded.  I tested.).  The database (pant, pant), I need a database (install MySQL on the Pi; I tested.  I coded.).

In the quiet twilight of a Sunday afternoon and with the benefit of my favorite (caffeinated) tea, I navigated to the page with the map.  I entered my information and submitted it.  In the brief milliseconds awaiting a response, the world was still and my eyes focused on a blinking cursor.  With the innate agility of a cat, the browser displayed my data on the map.  The final code/test/deploy cycle was successful and I was breathless.  I sat there looking at the map and my data.  “Wow!”, I whispered to only our cat.

I started programming as a product engineer and moved into IT 15 years ago.  The joy in my programming has often been matched or exceeded by testing my code.  The initial creative impulse coupled with the quick feedback of a running program makes for some long and satisfying weekend afternoons.
If you code, I encourage you to test.  There is certainly a very satisfying quality in implementing business requirements in short, organized, syntactically appealing and patterned phrases.  But, like music, the real beauty is in when code plays and plays well.