Considerations for a More Effective User Acceptance Testing

Slalom Consultant Pranav Jhumkhawala

Pranav Jhumkhawala is a technical manager and a solutions architect with Slalom Consulting, who has wide ranging experience in the areas of application architecture, development and management, implementing solutions for large and complex systems environments that span across multiple business areas, technology landscapes, and architectural disciplines.

User Acceptance Testing (UAT) is a significant milestone in software development lifecycle. It is during UAT that the users get to see the system “in action”, in many cases for the first time. Their acceptance and sign off are required to proceed with the production deployment. How formal and extensive is the UAT activity depends on the size of the project, user audience as well as the software development approach adopted by the delivery organization.

Acceptance Criteria
A key first step in the planning of UAT is to understand and establish a well-defined acceptance criteria. There is a general tendency amongst teams to focus on the “testing” and not on the “acceptance” part of UAT. This typically leads to too much attention being given to defects in the system, rather than the functionality and user satisfaction with the system. The goal of UAT is to showcase the functionality that the system offers in order to get an acceptance to move ahead with implementation, rather than asking the users to test the system for us. Therefore, emphasize the user acceptance as the primary objective of this activity and plan the UAT scripts accordingly.

The acceptance criteria should include:

  • Successful verification of functionality of the system (functional requirements are met);
  • Validation of key system interfaces;
  • If applicable, validation of converted or migrated data;
  • List of known issues;
  • The acceptance of look-and-feel as well as usability of the system; and
  • System response time and performance.

The last two bullets are especially important if the system being rolled out is a new application which will potentially change the users’ existing business processes. In this situation, user perception of the new system needs to be carefully managed. That said, most likely performance of the system during UAT will not be a “deal breaker” and there are ways in which you can work around performance related issues (e.g. limiting the size of data in the UAT environment, have limited number of users etc.). Nevertheless, it is necessary to address the response time related issues if they exist prior to the inception of UAT and therefore is an important consideration during this activity.

In an ideal situation, the UAT should be performed by the business users who were identified as the stakeholders and provided functional and non-functional requirements sign off. More often than not, especially in larger organizations, we have seen business analysts (BAs) who liaise with the business users provide and sign-off on the requirements, but the UAT is performed by a representation of end users (or the BAs taking the system to end users for UAT). This approach invariably leads to missed or misinterpreted requirements being implemented and turns out to be a costly fix since it is discovered only when the users see the system for the first time during UAT.

A side-effect of this approach is that it significantly increases the risk of UAT activity becoming a “focus group” session where users, upon experiencing the system for the first time, will provide their feedback in terms of potential enhancements and changes they would like to see before they would provide sign off. It is very important that stakeholders who provided sign off on the requirements are the ones executing and signing off on the UAT.

A Final Comment
Understand the user expectations and establish thresholds for open issues such as cosmetic issues, performance, etc. Communicate these to level-set with the UAT audience prior to UAT commencement. In my experience I’ve come across both extremes, from users who expected system to be production-ready at the UAT time to users whose primary focus was to ensure that the system encapsulates the functionality properly. On one occasion, we were going through UAT and the system was set up in an environment dedicated for this purpose. The performance of course was not as good as it would be in the production environment. Looking at the sluggish performance, one of the users commented, “The paper process (that this system was replacing) was quicker”. We of course clarified and reassured the user regarding performance improvements prior to production release but needless to say, some damage was already done when it came to the user perception.

Slalom Consulting’s San Francisco office Slalom Consulting's Strategy focus
Learn more about our San Francisco office Learn more about Slalom Consulting Strategy

subscribe by emailSubscribe to follow new Software Development posts

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: