Use Serial Introspection to Improve System Quality

If your organization is still using Finish-to-Start (Finish/Start) ratios to drive website improvements, then as a Tester you have a big opportunity!

Website Finish/Start ratios represent a small slice of what your company does to satisfy customers. While a customer believes they will receive their order after they make a payment through your website, their satisfaction may depend on

• Receiving the right product or service promptly
• Having their credit card charged accurately
• Other internal system transactions

Typically, the Finish/Start ratio is confined to your website where the customer starts their shopping and finishes with a purchase. As a Tester, I encourage you to redefine “Finish” as the successful completion of a transaction in all your company’s systems. I believe this is the opportunity to improve systems quality at an enterprise level.

For example, sending a confirmation email might be a “Finish” for the website. How does your company log a “Finish” for shipping a product? Or for charging the correct payment? What other systems in your company need to “Finish” so that a customer’s transaction is complete? At an inter-system level, the Finish/Start ratio becomes a benchmark of quality representing all systems that participate in the transaction: from the Start on a web page to a Finish in the General Ledger.

Introspective Applications
Introspection is a practice of logging software application events and other information so the behavior of a system or systems might be analyzed. Developers have used this practice to help them both verify and troubleshoot their code (ask your developer how they insert temporary logging into their code for debugging).

When scaled for a large web application, introspection logs reside in databases or third-party tools such as Splunk. Many times, the goal of these databases is to assist troubleshooting issues in production environments. This type of information is not useful for meaningful behavioral analysis.

To become introspective, the application must log contextual information such as product or service type, a unique transaction name, or product serial number to name a few into a database dedicated to introspective analysis. The information that you log should be unique to your product or service. Most importantly, the log must contain a unique identifier for the transaction, AND that identifier must be a part of all introspective databases. This is what I call serial introspection. It provides a data analyst the ability to review transactions through all introspective databases.

For example, a product is sold through a website which generates a shipping order. The order is logged in the billing system and the accounting system. The unique transaction identifier, created when the customer started shopping, must be logged into the website, shipping, billing, and accounting introspective databases.

In this manner, you can assess system quality by searching for that unique transaction identifier in all the databases. If it isn’t found, it may indicate an issue or opportunity in that system. Lastly, if you make that assessment in real-time, system quality issues can be located quickly.

Serial Introspection Example
In the example above, let’s say the website introspective database shows 1000 transactions (each transaction receives a unique identifier). If the shipping introspective database shows that only 950 of those transactions have a matching unique identifier, that might indicate a communication issue between systems or data errors. The larger question is will those 50 orders get shipped to customers? A real-time assessment could resolve this quickly and prevent future issues.

The Tester’s Opportunity
As quality assessments mature in organizations, Testers are ideally skilled to recommend not only tests but application improvements that help assess quality in real time and in production environments. Redefining the Finish in the Finish/Start ratio is one of those improvements. I encourage you to work with a Data Analyst to help define information for the business context of your introspective databases.

Advertisement

Agile Methodology Fallacies

Think your team can deliver products faster using Agile methodologies? It’s possible but Agile is not about delivery pace.

In my opinion, there is strong misalignment of expectations between IT and Business teams with respect to the promise of Agile, and what value means. I also believe that there is a distinct difference in delivering new products and updating legacy products when using Agile.

Agile Expectations

The Agile Methodology embraces experimentation, feedback, and collaboration to deliver products. When your enterprise considers using Agile, keep in mind that:

• Agile is not about delivery speed rather it is about speed of learning. Rapid learning helps a project team build the right product.
• Story Cards are not Agile rather they help project teams be agile.
• Story Card completion rate is not Agile because it does not capture the learning and feedback needed to be agile.
• Clarity of what to build must be shared and understood by everyone.
• Daily participation of IT and Business is vital for the success of product development.

Agile Value

For each organization, there is a perspective on value. IT sees value in the completion of component parts during the project life. Business sees value when the product produces business value.

When your enterprise considers adopting an Agile methodology, consider what speed and value mean to each organization. Establishing an understanding for the enterprise creates a collaborative foundation for Agile projects.

New Product vs Legacy Updates

In my opinion, the adoption of Agile for use in updating Legacy products is like the story of Cinderella: everything seems good at the Ball but reality is different. Why is this? I believe it is because of dependencies.

When defining a change for a legacy product, it appears simple enough at the start because, in an Agile fashion, the change should be small enough to implement quickly so value is realized. However, when the project team begins researching the implementation, dependencies are discovered. Inevitably, the estimate for delivery changes as the simple change accommodates dependencies of technologies and other groups in the enterprise.

Testing is like…

This post was inspired by a thread on The Club at Ministry of Testing. Join the best testing community on the internet at Ministry of Testing!

Testing is like searching for clues to solve a mystery. It’s almost like those old black and white detective movies where the hero is up against some dangerous foe. The foe may have somehow compromised the forces of good causing corruption their work.

We join our hero staying late at the office as they are running their last test…

Without any motion detected in the office area, the lights turn off automatically and I’m left in the glow of my laptop’s screen. I stare into the web page thinking it holds clues to despicable behaviors. If our customers saw them, they’d leave in droves. It was up to me. I had to find the culprit; he was known as Bugsy.

Previously, Bugsy hinted at his mischievousness on multiple pages causing spelling errors, massive misalignment of fields on small screens, and API errors. We found his accomplices, the demented NoCodeReview and the slothful CutNPaste, earlier today. The project team arrested them and sent them away for good. I sought they who ventured into the new API and tainted its reputation.

I reviewed the log for any hint that would unveil the darkness brought into our innocent API. The timestamps aligned with my test but the log message said only “Transaction Started”. Nothing else. What would cause the API to just stop?

I held my head in my hands above the keyboard. There had to be something else. What have I forgotten?

With a start, I straighten up in my chair. The sudden motion tripped the office lights on. I logged into the source control and reviewed the configuration file.

“You vile demon!”

In a moment, I realized Bugsy had teamed up with the captivating enchantress iForgot. The developer must have surrendered to her siren song as the project team worked quickly to deploy the API to a new environment. With incorrect credentials in the configuration file, the API just stopped.

I called the developer and explained my scenario. They logged on, sheepishly apologized, and updated the credentials. I sent iForgot up the river. Bugsy, unfortunately, may have slipped away. I wondered if we would meet again.

Epilogue

I walked towards the exit with my backpack and a knowing smile. We would be a better team because of this experience. Together, I thought, we will continue to fight and succeed against Bugsy. Through the door and into the night, I welcomed a Summer evening’s cool air on my face and victory in my heart.

Test Engineering vs Automation Engineering – 1

I see a difference between Test Engineering and Automation Engineering. As a Test Engineer, you work with people who test to improve the efficiency and effectiveness of their testing. These improvements span a spectrum of solutions from improving testability to providing automation. In my opinion, an Automation Engineer has a smaller scope. They design, develop, and maintain test automation.

I recently reviewed an email sent to our in-house test engineering community. It asked for assistance with an automation tool. The tool was unable to detect a modal dialog in a desktop application. The email asked if anyone had a solution.

There were two possible solutions. First, there was a suggestion to investigate a specific debug mode. Second, there was a suggestion to improve the testability by changing the design so that the model dialog was not needed. Both suggestions are valid but have different perspectives, different costs, and different impacts on project team dynamics.

Rather than seeing the issue framed as a tool constraint, I suggest a consideration of the testing challenge could help illuminate other possibilities.

Sometimes, it is not a automation challenge to quell through technical prowess, it is a testability challenge to be explored with a project team.

Testing and Modeling

We have seen the news and read the articles on testing.  These include the amount of testing, the type of testing, the testing results, debates on how much testing is needed, and who should be tested.  I have been in testing for about a decade and the information I see and hear is occasionally frustrating in its characterization, and many times presented without the context that could provide clarity.

Testing
Testing is an information collection activity.  While that may be a simple explanation, it grows complex as you begin to explore what is tested, how testing is performed, and what test results mean.

The most important question in testing is: what do you want to learn?  There are many answers because there are many people asking the same question.  Some things to learn are:

• About the percentage of the population infected
• If a person is infected
• If some actions help reduce infection

Learning about the percentage of the population infected is important because it informs us of the spread of the infection.  Learning about this frequently is more valuable because it informs us on the pace of the spread.  That is, it can answer a secondary question of how fast the infection travels through the population.
Learning if I am infected is important because it informs me that my future actions could be harmful to others and I may become ill.  Learning about this frequently is more valuable because it informs us that our actions may be helping to reduce the pace of the spread.  Alternatively, it may inform me that my present actions are preventing infection in me.
Learning if my actions reduce infection, say in the case of drugs administered to address the infection, is important because it informs me that my actions are correct.  Learning about this frequently is more valuable because we can inform others who may benefit from the same actions.

Once you determine what you want to learn, you ask what should be tested.  Since the infection lives in bodily fluids (such as mucous or blood), the answer seems clear.  There are challenges to collecting fluids because fluids can become contaminated and render the test invalid.  Collection methods must be sterile and, very importantly, those who collect the fluids must be protected.
Once the fluid is collected, the test occurs.  But how is the fluid tested?  We hear some tests attempt to detect the infection, and some tests attempt to detect antibodies.  My guess is the detection technique between the two is different.  Each test provides different information and answers different questions.  One test helps you learn if you might be infected; the other helps you learn if you might have been infected.
Note that I said “might be” and “might have”.  An important aspect of testing is accuracy.  An accurate testing result depends on an uncontaminated fluid sample, collected in a sterile manner, and evaluated by a highly accurate method.  If a test for infection resulted in no infection and that result was wrong, it is called a “false positive.”  Manufacturers of these tests must demonstrate the ability to produce accurate results and this ability requires time.

Lastly, the infection progression inside the body seems to show symptoms after many days.  Since the growth rate appears slow, a test may show a negative result one day (not infected), and a positive result the next (infection present).  This demonstrates test sensitivity.  That is, there must be some threshold of infection present in the collected fluid for the test to render a positive result.  A better sensitivity may detect infection sooner.

Models
I have also read and heard information about models.  In Information Technology, we are familiar with models and their uses.  Recently, I have heard how wrong some models are.  This conclusion seems uninformed and amateurish.

Weather forecasters use models to predict weather.  The models are based on information collected over a long period of time.   This history of information helps build an understanding of weather patterns.  Presently, models predict local weather patterns with pretty good accuracy.  As we all know, the weather forecaster occasionally makes an inaccurate prediction.

Models for infections are built in a similar manner.  The primary difference between weather and a new infection is the amount of information available.  With very little information about an infection, the prediction accuracy varies greatly.  As more information is collected, the infection model is updated and accuracy may improve but it improves very slowly.

The model presents a number representing a prediction for infections.  The number is not, and was never meant to be, exact.  It is a guess, an estimate.  The model may be inaccurate but I would not consider it wrong.  The inaccuracy helps identify things to consider.  These things can improve the accuracy for the next model.

A Short WFH Primer

I admit that when I started working from home, it seemed like a break.  It also provided my wife some peace of mind knowing that one day per week (usually Friday), I was home with the kids.

For the first few times, I realized I didn’t have to be “in the office” until 7 AM so I could sleep until almost 6:55 and still make it “in”.  By the time I had breakfast, I probably wasn’t all that available or effective until 7:30 or later.  Also, I felt sluggish throughout the day.  I needed to try something different.

On my next WFH day, I did what I usually do: I got up early to exercise, showered, and ate breakfast.  I felt more ready when I “arrived” and more awake during the day.  As I grew into this, I found the speaker and microphone on laptops suck.  Having a headset or puck helps me hear the conversation better, and they can understand me.

Do I take breaks?  Yes!  After a meeting, I might go up and down the stairs a few times, or go get the mail.  Were I at the office, I would normally walk somewhere to get water, collaborate with a peer, or oblige nature calls.

By treating the experience as I would for any other work day, I found I was just as effective when Working From Home.

The Decline of Civil Conversation

Perhaps I’ve been in too many meetings but I would like to call a truce on conversation artifacts such as frequent interruptions, unanswered questions, and inattention.

I generally wait for a lull in a conversation to add new information or ask a clarification.  When someone else is speaking, I practice some active listening or I take a note.  Mostly, I wait.  What I’ve noticed is someone will be talking or I will be talking and someone interrupts.  Blatantly interrupts.  Sometimes they want to clarify something or add their information.  I welcome it but an interruption seems rude.

Many times the conversation is going along and someone asks a question.  I would expect a short period of silence as everyone considers the question and a response.  What I’ve noticed is someone will start speaking about the same topic or another topic apparently clueless that a question had been posed.  I want to hear what they want to say but many times I want to hear an answer to the question.  An unanswered question feels like friction to the conversation.

Lastly, during those lulls or short periods of silence, someone looks up from their laptop and begins talking about a topic already covered or asking a question already answered.  This is exceptionally frustrating and takes time to provide the information they missed.

I blame social media for these annoying artifacts.  Anyone can post an update on social media sites at anytime and this practice may be carrying over into everyday conversation.

Listen twice as much as you speak, consider questions carefully, close your laptop unless you are reviewing a shared screen.  Enjoy a topic, collaborate, and move your conversations forward faster.  Who knows, it may even shorten your meeting!

Necessary Conversations – 1

Thanks for meeting with me <fill in Project Manager’s name>!  I appreciate this opportunity to explore the testing approaches for this project.  I realize we have just started planning and have no product designs yet, but this is the best time to have this conversation.

Soon, our team will begin to absorb and understand business needs, and transform them into viable products.  The Testing team, Testers and Test Engineers, look forward to collaborating with the team on that effort.

Yes, I can see where you might believe there is nothing to test.  Many are mistaken in this belief that there must be a product before testing can execute.  I would argue that the business needs will require some scrutiny and I believe Testers are best equipped to ask questions about assumptions and details.  These questions will help clarify the needs, clarify product definitions, and reduce the number of defects resulting from misunderstood or poorly written definitions.

Why the Test Engineers, then?  I’m glad you asked.  They listen to conversations for opportunities to make testing smoother for everyone.  By everyone, I mean Testers and Developers.  Basically, we have been practicing Shift Left, that is, moving evaluations closer to product construction.  We believe it can make a difference on this project.

How?  When the project team delivers unit tests with their minimal viable products, we know requirements are probably met.  This helps the Testing team focus on risks.  The time to complete testing is likely shorter.  That improves Pace.

Also, unit tests provide a first look at the quality of the product.  Both Testers and Developers review these tests.  More scenarios may be added to the unit test suite so the quality improves.  Unit tests are a leading indicator of quality and improve the confidence of Quality in the product.

The unit tests execute with every build.  If the build fails, the project team stops until they know the Build passes.  No sense in letting defects deploy because when they do, it is both unplanned down time for the team and an administrative defect management exercise.

Lastly, a growing suite of unit tests become the regression test.  We don’t need a large testing event (saving time and improving pace).  Additionally, the product team can confidently refactor and maintain the product in the long term which supports the Team.

Thanks for indulging me!  I look forward to a great project!

Challenging Questions – 1

The challenge becomes one of changing the question from

Why would we need a tester at this design meeting?

To

Why isn’t the tester invited to the <any product discussion> meeting?

 

Unit Test Advocacy

Unit tests – those tests created, executed, and maintained by developers – are no longer optional or a luxury.  In a world that demands CI/CD environments, unit tests are a necessity that keeps product updates flowing into production.  Unit tests are not the domain of developers, they are the domain of quality advocacy.  They are activities that are part of a team that accepts and pursues quality products AS a team.

Unit Tests Impact Pace, Quality, the Build, and the Team

• Unit tests aid pace in detecting defects early and reducing post construction testing.
• Unit tests improve quality by encouraging collaboration with the testing team to identify multiple scenarios and executing them early.
• Unit tests help determine the status of the product by executing tests before check-in and after a build.
• Lastly, unit tests demonstrate respect for your future self and your teammates. Everyone depends on prompt, valid, and valuable information provided by unit tests.