Category Archives: Software Testing

The New QA

Our applications are growing larger and more complex in business functionality, and they are available on multiple platforms.  Our business partners expect more functionality with better quality delivered in a timely fashion while our technical leaders attempt to simplify designs and make it all possible.

The ability to deliver value in an application AND making it simple to use challenges experience designers, developers, analysts, and especially testers.  An expectation to evaluate application behaviors in depth through just the user interface has not only become a fallacy, it has become insufficient and risky.

High profile projects in your organization must be delivered quickly and as a tester, you cannot wait for completed code to begin testing, and you cannot assure quality merely by testing through a UI.  The definition of testing must extend beyond exercising code to evaluate requirements and risks.  To expect high quality products at a good pace, you must foster and nourish quality as a team sport.  Your time is now and project teams are ready to hear you!

Testing now begins when work is assigned.  You are the Question Asker and you scrutinize acceptance criteria.
Soon after the assignment of a story card (or your method of work conveyance upon your team), the Three Amigos is your first opportunity to foster team quality, your first opportunity to “test”.  Your tests include

  • A check for acceptance criteria clarity
    • Does everyone have the same, deep understanding of WHAT is being built?
    • Have those understandings been verbally explored and vetted?
    • Have you vocalized your assumptions and had them clarified?
  • A check for validity
    • Is this product being constructed at the right time?
    • Are there dependencies that could prevent its timely completion?
  • A check for value
    • Some products are defined at some time in the past – does the present context of the project still support the work defined by this card?
    • Will the completed work still be valuable to the project and the product?

Testing continues as products are designed.  You are the Quality Advocate for testability in designs and implementations.
Most products require some thought before their construction.  The project team explores possibilities for implementation during design meetings.  Your testing continues at these meetings.  While you could also attend design reviews, a design review may imply that the design has been decided – this is too late.
When the design of a product is discussed, your primary goal is the testability of the product.  If the design addresses just the business intent of the product, then it is not testable.  As a tester, advocate for testability.  Make a request for error logging, for transaction transparency, and for data transparency.  An introspective product is a testable product.

During implementation, paired programming becomes paired collaboration.  You are the Quality Accomplice with simple tests at the ready and encouraging reflection on how the product in front of you will interact with the system.
Quality must be built into a product: the product is defined in a manner that is clear to everyone, the product is designed so that inspection is simple, and the product is constructed and checked multiple times during its implementation.  As a tester, you are the Quality Accomplice by having tests ready for execution at any time, and by working with the developer to help them create a quality product that meets the business intent.  To work effectively with a developer, you must understand the system and its components, and how the components interact to provide value.  With that understanding, you can ask questions how the code in component A will interact with the code in component B under multiple scenarios.  This collaboration helps everyone reflect on the integration of small parts defined in story card and their contribution to a product that operates correctly in multiple scenarios.

I encourage you to go forth and experiment with these ideas.




Test Automation is an Investment

I submit that test automation can improve your project’s pace.  I realize that some believe it helps detect defects or that it can be developed independently from project execution.  I recommend you reconsider those positions.

Ever since I read Specification By Example, my understanding and attitude towards test automation has changed.  It did not change how I design or build test automation, it changed how I see its purpose.  For this post, I refer to test automation in a spectrum from evaluating newly checked in code to that used in regression testing.

I’ve been pitching what I believe is a familiar notion around test automation in testing circles.  But it was suggested to me that I provide some context.  Put simply:

The purpose of test automation is to enable frequent evaluation of application behavior in the face of change.

I did not arrive at this conclusion by myself.  Adzic’s book was a great catalyst to think about test automation more, and to discuss it with other test engineers.  A context took shape for me and its present form is my declaration above.

Defect Detection is a Narrow Focus
Test automation costs too much to make it beneficial in just detecting defects in the short term.  Indeed, test automation developers find defects many times when they are developing (not executing) the automation.  That means that by the time the test automation is completed and verified, it’s possible someone else could have located that defect AND corrected it.

I recommend you invest in your test automation today.  You will realize the real value when it detects a defect in a succeeding iteration, or a succeeding project.  When it does, it demonstrates how recent changes have introduced errors in existing behavior.  In that manner, it is acting to preserve your business intent.

Independent Development Costs More
Let’s say you wait to develop test automation.  At some point in the future (the next iteration or the next project), the testing team is discovering defects in code that was completed AND working.  The current pace begins to slow:

  • Defects in existing code must be corrected so testing can evaluate the newly deployed code.
  • There is a good chance that correcting defects will introduce more defects.
  • When the code is operating again, you may find defects in the newly deployed code.

If there was an investment in test automation, frequent execution would detect changes to existing functionality at the time they were introduced.  That is, defects would be detected when developers execute the test automation, and they would correct them.  This saves time because the project team continues building products rather than fixing products; the project maintains its pace.

Your Investment Pays You Back
Test automation pays dividends in protecting your investments in building and expanding business functionality, and detecting unintentional changes to business behavior early.

  • Time is saved because the team executes test automation often to detect behavior changes early – before the products are deployed for testing.
  • System and integrated testing is more effective because simple defects are corrected before deployment for testing.

Lastly, the project team builds confidence in their products because the testing team can spend more time exploring products broadly, deeply, and for risk.

An Experiment for 2016

I missed writing.  I think about it every day but it’s like drooling over a pizza ad.  It’s nice but pizza with the family is much more satisfying, leaves an impression, and has more impact.  My inspiration to return came from Julie Zhuo reminding me of many great things I have read about writing – thank you Julie!

I focused a bunch of time on building a new web site for my hobby.  I like building it and I like testing it.  It gives me an opportunity to stretch my testing muscles lest they atrophy from the daily experience and negotiation of leading a testing team.  I enjoy the test lead role but when I need to do something to relax, I code and I test.

With a large and successful project behind me, I had an opportunity to catch up with our testing community.  Notably, I finally read Specification By Example (SBE) by Gojko Adzic (I realize I’m late to the party on this one).  I was seven chapters deep and realized I’ve miss so many opportunities to implement SBE ideas.  Then, there was a post on an internal blog recommending all testers read the book.  There was a momentarily satisfaction that my actions and the universe had aligned.

Ever since I started automating test cases, I was apprehensive about their existence beyond the end of the project.  We live in a very dynamic industry so the solution we implemented would inevitably change.  My tests would fail when change hit some critical mass.  I resolved to do better but better was to recommend caution around what was automated.  My constant challenge was writing automation that could survive change AND deliver a valid result.  I was a little successful but not always satisfied.

SBE crystallized my reasons for cautious automation.  I was unable to articulate why because I hadn’t found a suitable alternative.  But the logic and wisdom of automating business outcomes resonated with me.  The business outcome has a much lower frequency of change so automation makes sense.
It also defined why I had such apprehension.  What I have seen is automation of not only business outcome, but automation of test cases that provide little information just because they could be automated.  I also saw automation evaluating lower level functionality – which is valuable – but it was executed from the UI.  Both of these added technical debt and impeded project pace.

With the new year and a new project, I’m ready to experiment (it feels more like a hurricane of righteous purpose but I’ll calm down after a bit).  I discovered that our Requirements/Business Analyst community also favors the approach.  So far, I’ve suggested SBE to two other projects and the response is positive.



Testing is not for the Shy

I recently joined the Test Design Exploration team.  This is one of many groups within our enterprise available to mentor testers.  Our charter is to explore test design techniques, provide opportunities to use them, and discuss how techniques might be used in projects.  We establish a schedule of test design topics and testers sign up to attend sessions.  For example, we’ll be exploring All Pairs in an upcoming session.

Our team has been meeting regularly to prepare the sessions and set up a schedule.  I like our group – they are very experienced and bring some fresh ideas to our topics.  Recently, we provided a status update and some clarifications around our scheduled start date.  With the business complete, someone suggested that our team create a SharePoint site where testers might ask questions of our team should they be to shy to ask during a session. I liked the idea of having a place to follow up from our sessions.  Accessible to everyone, it makes our team more transparent, and I think we are willing to support it.  What stuck with me was the description of a tester to shy to ask a question.

When I started as a developer in this organization a while back, the fact that there was a whole group of people dedicated to testing was new to me.  I thought it was pretty cool!  But I rarely saw them during a project and heard from them even less.  When I moved to a testing job family, I tended to be very vocal.  Actually, it might have been called pushy or obnoxious.  But with a lot of feedback and practice, I found a testing voice that is more collaborative.  I also learned that testing demands more of a presence from a tester. I suggested to my test design teammates that we should encourage testers to ask questions.  The sessions are a safe place to do this, asking questions in a group gives them great practice and confidence, and maybe we can deliver more than test design mentoring.  They all agreed.

Testing is not for the shy.  There are developers that will push you around with very technical language, dispute defects, question the credibility of your tests, or just ignore you.  There are project managers that will pressure you to reduce testing.  In my opinion, this is not the identity of a tester.  Take back your identity!

You are a tester and quite capable of

  • Learning C#, Java, or any other language
  • Defending your defects both at a technical level and, more importantly, at a business level
  • Establishing credibility through early collaboration and shared goals
  • Being a contributing member of any project team
  • Expressing the risks of reduced testing

I’m looking forward to exploring test design with testers in my organization.  But I think they can learn more by asking questions in the sessions or anywhere they go.  The bigger challenge for me is encouraging them to believe in their capabilities, and to have them respectfully act on the behalf of the quality in the products they evaluate.

My Testing Platform – One Year Retrospecitve

In a previous post (opens in a new tab), I declared a Testing Platform and an intent to review it one year hence.   Since that post, I completed one project and started another.  In my new project, I am working with new people and on a different platform.
In a somewhat agile fashion, I offer my retrospective on what has gone well, and where I might improve.  My perspective is from my role as Test Lead.

Testing Purpose
During the first part of the year, we loosely pursued a testing purpose in our tests.  With the start of a new phase of the project and the addition of more testers, I request the tester describe an information objective for their assigned story cards.  Not every tester on my team has attended BBST Foundations so I mentor them in this concept.
I believe this helps frame the testing effort however we’re just a few iterations into the phase.  I don’t have enough information to determine if my team has seen benefit from describing a testing objective so I plan to be diligent.

I started using more automated unit tests for my home programming projects and experimented with creating an MVC-based website.  I feel the exposure to both of these technologies keeps my programming and testing skills sharp.  They provide a perspective I can use in testing and a foundation for interacting with developers.
As a Test Lead, thinking is still very valuable.  I frequently offer thoughts and impressions on implementation ideas, on testability, on maintenance, or on people.  In my experience, a Test Lead is part of the core team guiding the project, and should offer a testing perspective on many project aspects.
Lastly, I benefit from participating in this community.  Both blogs and books offer perspectives, ideas, and experience that I find useful.  I think I can do better in offering the same, and encourage other testers to participate as well.

Early in the project, I established good credibility with the core project team.  I engaged them in conversations about their experience and background, project objectives, and testing.
I find that conversations about testing are more of continuous journey to help others understand testing value.  Just when I think everyone is on the same page, the testing team suffers some small, unintended slight, and my journey begins again.
Everyone agrees that testing is required, but they soon forget the conversation.  While this is often frustrating, perseverance without arrogance, I believe, helps the testing team and the project.
To date, I must say my project team is one of the more considerate of testing, understanding of its benefits, and engaging in quality as a team sport than I usually experience.

Over the course of the project, I provided alternative opinions to bring clarity to a few requirements however, assisting with the direction of our project was the most rewarding.  I suggested a path of more learning which could result in a shorter final testing cycle.  A faster delivery often appeals to project managers however I believe a few of my tenets in this platform assisted in his decision to use my suggestion.
Early in the first phase of our project, I did a poor job of communication.  I made assumptions about a few tasks, actions, and ideas.  I failed to realize that I was working with new people.  They did not understand me as well as those on my previous project (where we had worked together for two years).  I made a post-it note to ask myself if I am making an assumption which assisted in clarifying my emails and preparing for conversations.  I learned that I should not get to comfortable in language, words, and conversations for these may be as context-driven as my next test.

My conversations advocating testability have grown into collaborations on how to make products testable.  Additionally, I encourage my testers to discuss testability with developers, and our design reviews include a review on testability.  In my opinion, succeeding in improving testability one project at a time is good; when my testers join new projects, I hope they spread the word.
In a new project I find that, in addition to introducing the idea of testability, I must also convey my thoughts on what testability means to me.  While it may sound like a simple concept, offering your thoughts to testers or developers will generate a good conversation, and everyone lands on a working definition for the project.

My experiment in estimates includes factors that impact testing work: environment stability and availability, developer testing, defect density, and product familiarity.  I use these to adjust a base estimate (e.g., a large defect density increases the estimate) and then apply a confidence factor.  I also set the expectation that I would change my estimate as I learn more about the products and project.
Again, I don’t have enough iterations to draw conclusions on this approach.

I’ve been experimenting with wireless technology for some embedded development, and researching motors and controllers for some ideas around robotics.  While each has unique challenges in testing, neither is associated with the testing I perform for my employer.  However, I often find the defocus valuable and, serendipitously, occasionally useful.

During the first phase of my current project, I found no opportunity to use automated testing.  The implementation was not complex so I could not justify the cost.
In our second phase, the implementation is more complex and has a few opportunities for automation.  Specifically, I expect to extend a tester’s reach by providing them the ability to explore product behaviors with a greater diversity of data combinations, and an opportunity to review more data in a shorter amount of time.

While the feedback I receive on collaboration has been good, I was elated to hear that our business sponsor would like to know more about what we test and how we test.  I look forward to sharing more than a report or a defect.  I hope to share testing stories and help them understand the impact to the products we create.

How was testing for you in the last year?


YOU Can Improve Testing in Your Enterprise

Can a single blog post describe how to make a change in your enterprise?  Maybe! If you came this far, then I believe you can be the catalyst.

You know your organization needs a change.  It’s a nagging awareness that things could be better, or an annoying realization that other companies are doing it better, or dream-like world where testing is integral to the project, part of a team, part of a solution.

Am I the Only One Who Knows?
Perhaps you’re asking, “Is it just me that knows?  Do others know?”  Maybe – but think about the conversations at lunch, in the hallways, at happy hour – has anyone ever vented about testing?  Perhaps someone was subtly dismissed by another team member, or another felt they let a project down in not testing enough.  In all the venting, has anyone said, “It can be different”?

Maybe you can say it.  Maybe you have said it at happy hour.  It might have felt as if you just said y’all gonna climb Mt Everest.  Some look at their drink, some take a drink, and some review jukebox selections.  Then, everyone went back to work the next week.

You Were Right
When you said, “It can be different”, you were right.  You were also suggesting everyone climb Mt Everest.  Making a change in an enterprise is an uphill climb, requires a dedicated team, must demonstrate benefit, and must demonstrate repeatability by anyone.  So, you should start today.

Even the first step towards a lofty summit begins with some small action, some small preparation.  You know you want something different, but what does different really mean?  It might mean more respect for some, more time for testing for another.  That small action could be to identify what could be different.  Make a list and look for themes.

Your Opportunity
Does your list suggest better team dynamics?  More test planning?  Support for training?

Two questions:

  1. Can you prioritize the list based on value to your testing?
  2. Does anything on the list align with what your enterprise needs?

Where there is alignment of your list and enterprise needs, YOU have an opportunity.  That opportunity is to demonstrate value in testing as a benefit to your enterprise.

Your Next Step
Pick something on your list and describe what you could do to achieve it.  Let’s say you pick more training.  Training is a broad topic so you want to take a small step.  What might be a small first step?  Read a testing magazine (there are many), read a testing book (there are many), or watch a testing training video (there are a few).

Regardless of the small step you take, apply what you learn and share what you learn!  By applying a new skill, you demonstrate to yourself that change is possible.  By sharing what you learn, you begin to influence others.

The Beginning
Your small step is a beginning.  A beginning that could embody the change or changes that echo off your office cubicles.  Good change starts from within, and it continues with good people.  You will need help and should not be afraid to ask for it.  Often.  Change is possible – find those like-minded testers, find a business sponsor, collaborate on a purpose and principle, and propose a beneficial, sustainable, and exciting change in testing for your enterprise.

What will you do today to be that change?

Learning to Read Code? Here’s my Short Primer!

I support the idea of testers learning to read code. It’s also a great opportunity to build rapport with developers. If you want to learn how to write programs, I support that also.
However, I think reading code will benefit you more when you know more about it. For example, when you read a mystery book, you look for certain type of characters (a detective), and certain types of activity (discovering a clue). Reading code is much the same. Here’s my primer.

Core Constructs
There are four basic code constructs supported by just about every language. These are important to understand because each has implications to your testing. The constructs are assignment, If statements, loops, and math. Programming is an act of combining these constructs to accomplish business goals.

Code Construct How to Identify It (in most languages)
Assignment Look for an equals sign (=) between words and numbers
dateOfBirth = 6/30/2001
If Statements Search for the word ‘if’
Loops Loops contain with the word ‘for’ or the word ‘while’
Math Most languages use common symbols to perform arithmetic. The exception is multiplication which is often represented by the asterisk (*).

In addition to the core constructs, code supports collecting and reporting of information (input/output), and storage of information.

Assignment sets the value of a variable. For example, when you enter a date into a program, the program probably assigns it to a variable (temporary storage – see below) so that it might act on it. This allows the program to verify the date is valid and other tests before it is used elsewhere.
Testing Implication: Information collected from a user by the program is always suspect. This includes data collected from a web page, an application form, a file, or a web service. A healthy dose of Domain testing is in order.

The If statement provides a program the ability to evaluate data, and to direct workflow. For example, you want to open a credit card account and the company has a business rule requiring applicants be at least 25 years old. If you provide your birthday as today, the program evaluates that date and rejects it.
Testing Implication: Business rules may have multiple outcomes. In the example above, there are at least three. Your testing may need to functionally evaluate the program for each outcome: (1)the applicant is 25 years old, and the applicant is (2)more than or (3)less than 25 years old. Additionally, the program’s workflow may be different for each outcome.

Loops allow the program perform the same code sequence many times. For example, when your music program displays a list of songs, the same code sequence executes to display each line but the data it uses, the song name, changes each time through the code sequence.
Testing Implication: The number of times a code sequence executes may depend on the data, it may execute a fixed number of times, it may change program behavior if there is an error, and in some cases it may never stop. Review programs for loops and evaluate what controls the execution of loop.

Programs perform basic math to trigonometry. For example, a spreadsheet performs many arithmetic functions.
Testing Implication: The arithmetic equations themselves should be evaluated in isolation. Once you have confidence it the equations, the inputs to those equations may require Domain analysis and testing.

We are probably most familiar with the input and output of programs. We enter data into a screen, and we see data displayed on a screen or printed report. Programs accept data from other programs, web services, files, or cameras. Conversely, programs deliver information as files or strings of characters.
Testing Implication: There are many forms of input and output and they vary based on source (input) and destination (output).
How to Recognize It: Theses are largely code sequences specific to the requirement of the program. This is a great opportunity to review the functionality with your developers, collaborate on the testability of inputs and outputs, and learn.

Programs store data in two forms: temporarily and permanently. Temporary storage (known as variables) allows the program to work with data to meet the business needs. Permanent storage provides for persistence of information over a long period of time (disk drives and databases are examples of permanent storage).
Testing Implication: Temporary storage may be an issue in embedded applications where memory is scarce. Permanent storage can improve program testability when used to create logs or journals of transactions.
How to Recognize It: Temporary storage is declared explicitly in many languages. The syntax includes a data type identifier and a variable name. Permanent storage usually requires code libraries to read and write information to a file or database – another opportunity to talk to your developer.

Now, go find some code and practice!  Leave some feedback on the examples you found!