Category Archives: Experience

The New QA

Our applications are growing larger and more complex in business functionality, and they are available on multiple platforms.  Our business partners expect more functionality with better quality delivered in a timely fashion while our technical leaders attempt to simplify designs and make it all possible.

The ability to deliver value in an application AND making it simple to use challenges experience designers, developers, analysts, and especially testers.  An expectation to evaluate application behaviors in depth through just the user interface has not only become a fallacy, it has become insufficient and risky.

High profile projects in your organization must be delivered quickly and as a tester, you cannot wait for completed code to begin testing, and you cannot assure quality merely by testing through a UI.  The definition of testing must extend beyond exercising code to evaluate requirements and risks.  To expect high quality products at a good pace, you must foster and nourish quality as a team sport.  Your time is now and project teams are ready to hear you!

Testing now begins when work is assigned.  You are the Question Asker and you scrutinize acceptance criteria.
Soon after the assignment of a story card (or your method of work conveyance upon your team), the Three Amigos is your first opportunity to foster team quality, your first opportunity to “test”.  Your tests include

  • A check for acceptance criteria clarity
    • Does everyone have the same, deep understanding of WHAT is being built?
    • Have those understandings been verbally explored and vetted?
    • Have you vocalized your assumptions and had them clarified?
  • A check for validity
    • Is this product being constructed at the right time?
    • Are there dependencies that could prevent its timely completion?
  • A check for value
    • Some products are defined at some time in the past – does the present context of the project still support the work defined by this card?
    • Will the completed work still be valuable to the project and the product?

Testing continues as products are designed.  You are the Quality Advocate for testability in designs and implementations.
Most products require some thought before their construction.  The project team explores possibilities for implementation during design meetings.  Your testing continues at these meetings.  While you could also attend design reviews, a design review may imply that the design has been decided – this is too late.
When the design of a product is discussed, your primary goal is the testability of the product.  If the design addresses just the business intent of the product, then it is not testable.  As a tester, advocate for testability.  Make a request for error logging, for transaction transparency, and for data transparency.  An introspective product is a testable product.

During implementation, paired programming becomes paired collaboration.  You are the Quality Accomplice with simple tests at the ready and encouraging reflection on how the product in front of you will interact with the system.
Quality must be built into a product: the product is defined in a manner that is clear to everyone, the product is designed so that inspection is simple, and the product is constructed and checked multiple times during its implementation.  As a tester, you are the Quality Accomplice by having tests ready for execution at any time, and by working with the developer to help them create a quality product that meets the business intent.  To work effectively with a developer, you must understand the system and its components, and how the components interact to provide value.  With that understanding, you can ask questions how the code in component A will interact with the code in component B under multiple scenarios.  This collaboration helps everyone reflect on the integration of small parts defined in story card and their contribution to a product that operates correctly in multiple scenarios.

I encourage you to go forth and experiment with these ideas.




Perry Mason and Assumptions

That’s twice!  Twice this week my assumptions caused a bit of trouble.  Last Saturday, I placed an order for parts.  They are not expensive but I’m careful about my selections and want to verify multiple times that I have the right parts.  On Saturday, I was confident in my selections and I placed the order.

On Sunday, I performed a test with a voltage regulator.  By design, voltage goes in and exactly 3.3 volts comes out.   I connected my 3.7 volt battery to the input and found 2.9 volts at the output.  My assumption was the battery voltage was greater than the output (that is, 3.7 > 3.3) so the regulator should work.  I didn’t have the right parts.  Fortunately, I was able to remedy this assumption by amending my order.

Today, I was updating a Java program.  I needed a string representation of date and time so I used the SimpleDateFormat object to create the representation.  I handed the format() method a Calendar object and started the program.  I was not getting the expected result.  Ignoring sarcastic little voices, I reviewed my changes and added some logging.  The voices were correct; it was the format() method.  My assumption was I needed only to provide the method a Calendar object.  It needed a DateTime object.  Fortunately, I was able to remedy this assumption.

Assumptions are sneaky.  Sometimes we know about them and discuss them, other times they drift in like a slow forming fog.  You don’t know its there but you discern some challenge.  It’s too subtle to change a direction but you’re cognizant of it.

I have many example of this experience through my recent Earl Stanley Gardner binge reading.  Gardner brought us Perry Mason.  If you are aware of Perry Mason only as a TV series, I strongly recommend the books.  They are engaging, humorous, and full of clues.  There are strong analogs to story cards.  When a story card goes awry, check assumptions and check the story.  One or two facts get misaligned and a defect is created.  Gardner weaves multiple facts and multiple explanations for facts.  It the end, Perry Mason, in the role of tester, fights through the fog of assumption, illuminates small details, and finds the one true story to exonerate his client.

As a tester, you will benefit greatly by having your assumptions dashed, and attention to detail challenged.  This week, two assumptions caused some delay in executing my hobby.  What assumptions might we be making that could cause a delay in our projects?


My GWT Identity Crisis

There are a few things I’ve not tried yet.  Sadly, possibly, I’ve not tried karaoke, and I’ve not tried kite boarding.  I am however getting my first experience in applying Given-When-Then in describing product behaviors on a real project.  I’m very excited at the possibility of merging business behavior with automation.

I engaged Specification By Example, I attended hands-on training, and I have participated in periodic drive-by creations of GWT scenarios.  I felt very ready.  Hurtling into some of our project’s first story cards, I was shocked to review GWT that contained behavior AND, take a deep breath, implementation details.

As an example, I offer a scenario from an API I wrote for a website.

GIVEN active flights exist,

WHEN the website requests a list of active flights,

THEN the reply includes all active flights

AND their unique identifier,

AND their altitude,

AND their last update.

This GWT represents my interpretation of what I have learned and, as noted above, my limited experience.  Here is how the same scenario might appear if I violate one of the most fundamental rules.

GIVEN the active_flight field indicates true,

WHEN the website executes GET against the activelist API,

THEN the response contains an HTML table detailing the flight’s GUID,

AND the flight’s altitude as an integer,

AND the last update in a UTC format.

In my opinion, this is more than a violation of the “no implementation details” rule.  This description locks the behavior to a specific column of a database, defines the verb and API name, prescribes a specific rendering method, and calls out individual field data types.  Its as if a developer provided the information they needed to implement the scenario to an analyst.

While this is clearly an extreme example, I recently read such a scenario.  When I asked, the reason implementation details were included was because this is a “technical story card” (different from a “business story card”), and because it is easier to provide all the information in one place.

(ahem) Bull.  Shit.

In my opinion, this method of practicing and segregating GWT ignores one of its most quintessential and valuable aspects: collaboration.  Yes, I read Specification By Example and attended training.  I learned the mechanics but in both cases I left with something more.

I perceived an overwhelming feeling that during a conversation about product details between an analyst, a tester, and a developer, these few people begin to share not only their ideas but their understanding.  It IS NOT the GWT itself that encourages automation or reduces defects rather it is those precious minutes where an honest, introspective, and respectful dialog clarifies the behavior of a simple product.

Just writing GWT provides a small window into a product.  Collaborating over the value of a specific business behavior encourages teamwork and quality where everyone can take pride and feel their contribution matters.


The Explorer

Imagine you are an explorer.  You often explore jungles and forests, and occasionally cities and villages.  People regard your exploring skill highly but with some reserve.  One day, a builder comes to you who wants you to explore a bit of pasture.  They are interested in changing the pasture so that many people might live there.

  • Given your skill and experience in exploring, when you go out to the pasture, then the builder wants to know your opinion on having people live there.

You decide to visit the pasture.

After a day’s journey, you arrive at the pasture.  From your vantage point, you can see it is bordered on one side by a large hill.  While there are many groups of healthy trees, there are also inviting clearings.  You think that if people lived here, they would enjoy the area.  As you walk around the pasture, you take notice of the types of trees, variety of animals and insects, and appreciate the small river.

Upon your return, you describe what you saw and what you found.  In your opinion, people would be able to live in the pasture.  The builder asks many questions and you respond with your experience in the pasture.  The builder asks if you found any bugs.  You reply that you found many bugs for they were plentiful on the ground and on the trees.  The builder thanks you and gives you a special token of their appreciation.

A few weeks later, the builder asks you to explore a city.  They are interested in making changes in the city.

  • Given your skill and experience in exploring, when you go to the city, then the builder wants to know your opinion on making changes in the city.

You decide to visit the city.

After a day’s journey, you arrive at the city.  From your vantage point, you can see a few buildings, neighborhoods, streets, and people.  Beyond the city limits, there is a forest.  You think that if the city had some changes, the city would readily adapt.  As you walk around the city, you take notice of the population, the clean streets, the commerce, and the services.  You recall the builder asked about bugs so you also pay special attention to the ground and the few trees you find.

Upon your return, you describe what you saw and what you found.  In your opinion, the city was ready for changes.  The builder asks many questions and you respond with your experience in the city.  Some of their questions about infrastructure and services were challenging so you express your regret in not gathering more information about them.  The builder asks if you found any bugs.  You reply that you found no bugs.

Surprised, the builder asks, “Where did you look for bugs?”

“On the ground and in the trees.  The ground was often covered in concrete and there were few trees.”

“Did you look under the concrete, or in the buildings?”

“I did not.  Since I found bugs on the ground and in the trees in the pasture, that is where I looked for them in the city.”

The builder explains how the city is different from the pasture.  While the bugs were easy to find in the pasture, bugs in the city may hide in buildings or houses.  They may also hide in the way services operate in the city.  The builder suggests you review information about how cities are different from pastures.

The builder expresses their disappointment in your report.  They believe the changes to the city will be delayed because your information may not be accurate.  They thank you and leave.

A few weeks later, the builder asks you to explore the city again.  Not to be disappointed, they ask about your understanding of cities.

“I have explored cities occasionally and they contain many things that I don’t understand.”

The builder thanks you and says they know of another explorer more familiar with cities.  They will ask them to explore the city.

Imagine YOU are the explorer.  What would you do next?



The Most Important IT Skill

My wife and I attended Curriculum Night at the high school where our boys attend.  It had already been a busy week (and frustrating one if you count the removal of a poorly designed kitchen faucet).  But Curriculum Night is something I have enjoyed for many years and would not consider missing it!

My son is enrolled in AP Programming so I was very curious about the curriculum AND the teacher.  I was impressed with the cognitive level of assignments he had last year, and this AP course had my expectations high.  I sat through informative yet similar sounding sessions on Pre-Calc, English, and Physics before hearing about the AP Programming course.

When I arrived for the session on AP Programming, the teacher greeted me with a hardy handshake, an excited “Welcome!”, and invited me to sit where my son usually sits.  He was animated and nerdy (cool).  He had spent ten years in the industry before deciding to teach (nice).

He started talking about the curriculum, noted the students receive one new programming concept per week (they are working on arrays this week), and finished with how their work is evaluated.  My interest grew.

Students are expected to review each others’ work, note the differences between the implementations, and they are invited to speak in front of the class about their discoveries.  The teacher stated the ability to communicate was as important as learning how to write programs!

I was elated.  An experience in collaboration and empathy respectfully facilitated by the students is a great IT lesson.  In my journey through my testing career, I discovered that, as a Test Lead or Tester, testing is a very social activity.  The ability to convey and negotiate ideas, collaborate with multiple personalities, and develop a working team has been as rewarding as writing programs and tests.  Getting that experience in high school is like preventing defects early in the project!

Testing is not for the Shy

I recently joined the Test Design Exploration team.  This is one of many groups within our enterprise available to mentor testers.  Our charter is to explore test design techniques, provide opportunities to use them, and discuss how techniques might be used in projects.  We establish a schedule of test design topics and testers sign up to attend sessions.  For example, we’ll be exploring All Pairs in an upcoming session.

Our team has been meeting regularly to prepare the sessions and set up a schedule.  I like our group – they are very experienced and bring some fresh ideas to our topics.  Recently, we provided a status update and some clarifications around our scheduled start date.  With the business complete, someone suggested that our team create a SharePoint site where testers might ask questions of our team should they be to shy to ask during a session. I liked the idea of having a place to follow up from our sessions.  Accessible to everyone, it makes our team more transparent, and I think we are willing to support it.  What stuck with me was the description of a tester to shy to ask a question.

When I started as a developer in this organization a while back, the fact that there was a whole group of people dedicated to testing was new to me.  I thought it was pretty cool!  But I rarely saw them during a project and heard from them even less.  When I moved to a testing job family, I tended to be very vocal.  Actually, it might have been called pushy or obnoxious.  But with a lot of feedback and practice, I found a testing voice that is more collaborative.  I also learned that testing demands more of a presence from a tester. I suggested to my test design teammates that we should encourage testers to ask questions.  The sessions are a safe place to do this, asking questions in a group gives them great practice and confidence, and maybe we can deliver more than test design mentoring.  They all agreed.

Testing is not for the shy.  There are developers that will push you around with very technical language, dispute defects, question the credibility of your tests, or just ignore you.  There are project managers that will pressure you to reduce testing.  In my opinion, this is not the identity of a tester.  Take back your identity!

You are a tester and quite capable of

  • Learning C#, Java, or any other language
  • Defending your defects both at a technical level and, more importantly, at a business level
  • Establishing credibility through early collaboration and shared goals
  • Being a contributing member of any project team
  • Expressing the risks of reduced testing

I’m looking forward to exploring test design with testers in my organization.  But I think they can learn more by asking questions in the sessions or anywhere they go.  The bigger challenge for me is encouraging them to believe in their capabilities, and to have them respectfully act on the behalf of the quality in the products they evaluate.

My Testing Platform – One Year Retrospecitve

In a previous post (opens in a new tab), I declared a Testing Platform and an intent to review it one year hence.   Since that post, I completed one project and started another.  In my new project, I am working with new people and on a different platform.
In a somewhat agile fashion, I offer my retrospective on what has gone well, and where I might improve.  My perspective is from my role as Test Lead.

Testing Purpose
During the first part of the year, we loosely pursued a testing purpose in our tests.  With the start of a new phase of the project and the addition of more testers, I request the tester describe an information objective for their assigned story cards.  Not every tester on my team has attended BBST Foundations so I mentor them in this concept.
I believe this helps frame the testing effort however we’re just a few iterations into the phase.  I don’t have enough information to determine if my team has seen benefit from describing a testing objective so I plan to be diligent.

I started using more automated unit tests for my home programming projects and experimented with creating an MVC-based website.  I feel the exposure to both of these technologies keeps my programming and testing skills sharp.  They provide a perspective I can use in testing and a foundation for interacting with developers.
As a Test Lead, thinking is still very valuable.  I frequently offer thoughts and impressions on implementation ideas, on testability, on maintenance, or on people.  In my experience, a Test Lead is part of the core team guiding the project, and should offer a testing perspective on many project aspects.
Lastly, I benefit from participating in this community.  Both blogs and books offer perspectives, ideas, and experience that I find useful.  I think I can do better in offering the same, and encourage other testers to participate as well.

Early in the project, I established good credibility with the core project team.  I engaged them in conversations about their experience and background, project objectives, and testing.
I find that conversations about testing are more of continuous journey to help others understand testing value.  Just when I think everyone is on the same page, the testing team suffers some small, unintended slight, and my journey begins again.
Everyone agrees that testing is required, but they soon forget the conversation.  While this is often frustrating, perseverance without arrogance, I believe, helps the testing team and the project.
To date, I must say my project team is one of the more considerate of testing, understanding of its benefits, and engaging in quality as a team sport than I usually experience.

Over the course of the project, I provided alternative opinions to bring clarity to a few requirements however, assisting with the direction of our project was the most rewarding.  I suggested a path of more learning which could result in a shorter final testing cycle.  A faster delivery often appeals to project managers however I believe a few of my tenets in this platform assisted in his decision to use my suggestion.
Early in the first phase of our project, I did a poor job of communication.  I made assumptions about a few tasks, actions, and ideas.  I failed to realize that I was working with new people.  They did not understand me as well as those on my previous project (where we had worked together for two years).  I made a post-it note to ask myself if I am making an assumption which assisted in clarifying my emails and preparing for conversations.  I learned that I should not get to comfortable in language, words, and conversations for these may be as context-driven as my next test.

My conversations advocating testability have grown into collaborations on how to make products testable.  Additionally, I encourage my testers to discuss testability with developers, and our design reviews include a review on testability.  In my opinion, succeeding in improving testability one project at a time is good; when my testers join new projects, I hope they spread the word.
In a new project I find that, in addition to introducing the idea of testability, I must also convey my thoughts on what testability means to me.  While it may sound like a simple concept, offering your thoughts to testers or developers will generate a good conversation, and everyone lands on a working definition for the project.

My experiment in estimates includes factors that impact testing work: environment stability and availability, developer testing, defect density, and product familiarity.  I use these to adjust a base estimate (e.g., a large defect density increases the estimate) and then apply a confidence factor.  I also set the expectation that I would change my estimate as I learn more about the products and project.
Again, I don’t have enough iterations to draw conclusions on this approach.

I’ve been experimenting with wireless technology for some embedded development, and researching motors and controllers for some ideas around robotics.  While each has unique challenges in testing, neither is associated with the testing I perform for my employer.  However, I often find the defocus valuable and, serendipitously, occasionally useful.

During the first phase of my current project, I found no opportunity to use automated testing.  The implementation was not complex so I could not justify the cost.
In our second phase, the implementation is more complex and has a few opportunities for automation.  Specifically, I expect to extend a tester’s reach by providing them the ability to explore product behaviors with a greater diversity of data combinations, and an opportunity to review more data in a shorter amount of time.

While the feedback I receive on collaboration has been good, I was elated to hear that our business sponsor would like to know more about what we test and how we test.  I look forward to sharing more than a report or a defect.  I hope to share testing stories and help them understand the impact to the products we create.

How was testing for you in the last year?