Use Serial Introspection to Improve System Quality

If your organization is still using Finish-to-Start (Finish/Start) ratios to drive website improvements, then as a Tester you have a big opportunity!

Website Finish/Start ratios represent a small slice of what your company does to satisfy customers. While a customer believes they will receive their order after they make a payment through your website, their satisfaction may depend on

  • Receiving the right product or service promptly
  • Having their credit card charged accurately
  • Other internal system transactions

Typically, the Finish/Start ratio is confined to your website where the customer starts their shopping and finishes with a purchase. As a Tester, I encourage you to redefine “Finish” as the successful completion of a transaction in all your company’s systems. I believe this is the opportunity to improve systems quality at an enterprise level.

For example, sending a confirmation email might be a “Finish” for the website. How does your company log a “Finish” for shipping a product? Or for charging the correct payment? What other systems in your company need to “Finish” so that a customer’s transaction is complete? At an inter-system level, the Finish/Start ratio becomes a benchmark of quality representing all systems that participate in the transaction: from the Start on a web page to a Finish in the General Ledger.

Introspective Applications
Introspection is a practice of logging software application events and other information so the behavior of a system or systems might be analyzed. Developers have used this practice to help them both verify and troubleshoot their code (ask your developer how they insert temporary logging into their code for debugging).

When scaled for a large web application, introspection logs reside in databases or third-party tools such as Splunk. Many times, the goal of these databases is to assist troubleshooting issues in production environments. This type of information is not useful for meaningful behavioral analysis.

To become introspective, the application must log contextual information such as product or service type, a unique transaction name, or product serial number to name a few into a database dedicated to introspective analysis. The information that you log should be unique to your product or service. Most importantly, the log must contain a unique identifier for the transaction, AND that identifier must be a part of all introspective databases. This is what I call serial introspection. It provides a data analyst the ability to review transactions through all introspective databases.

For example, a product is sold through a website which generates a shipping order. The order is logged in the billing system and the accounting system. The unique transaction identifier, created when the customer started shopping, must be logged into the website, shipping, billing, and accounting introspective databases.

In this manner, you can assess system quality by searching for that unique transaction identifier in all the databases. If it isn’t found, it may indicate an issue or opportunity in that system. Lastly, if you make that assessment in real-time, system quality issues can be located quickly.

Serial Introspection Example
In the example above, let’s say the website introspective database shows 1000 transactions (each transaction receives a unique identifier). If the shipping introspective database shows that only 950 of those transactions have a matching unique identifier, that might indicate a communication issue between systems or data errors. The larger question is will those 50 orders get shipped to customers? A real-time assessment could resolve this quickly and prevent future issues.

The Tester’s Opportunity
As quality assessments mature in organizations, Testers are ideally skilled to recommend not only tests but application improvements that help assess quality in real time and in production environments. Redefining the Finish in the Finish/Start ratio is one of those improvements. I encourage you to work with a Data Analyst to help define information for the business context of your introspective databases.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s