Recently the have been a number of well written articles that reference UAT [User Acceptance Testing] and automation of such tests. These have often resulted in questions about how such activities map into the testing approaches that I cover in various Presentations and postings. This post is intended to answer those questions in a common place rather than having the material distributed across a number of venues.
Much of the time when the word “User” [e.g. User Stories] is encountered, I take the expansive view that the “user” of your product is *anything* that consumes or interacts with the product; meaning it could be human or another system. This view does not really work when talking about UAT where there can be a clean differentiation between a Human with many subjective aspects and integration with other systems which may be handled objectively with “Integration Tests”. This subjective nature means that different people will have different reactions as well as the same person having different reactions in different scenarios or over a period of time.
Defining Acceptance as pass fail is also problematic. At the very low end, people may accept something because they have little choice [e.g. a business application used on the job], this can be considered a coerced result. At the opposite end, people may find something to be wonderful and their level of acceptance is to embrace the item. Between these two points there is a wide range of reactions that an be grouped as tolerated results.
When one has a suite of tests, then an analysis of both the individual results and various aggregates can be performed to determine if the level of acceptance meets the targeted goal. Determining the goal is complex, but it should be clear that achieving a level of embrace by all potential users (Again, people) across all situations is unrealistic. Even defining tests in terms of “do this, verify that” which will correlate with actual reactions of different populations is likely to be out of reach.
If one can not define the tests or an exact measurement or results to yield “Pass/Fail”, then taking an Exploratory approach is likely to be much more effective. By definition a “Go play with X, and let me know what you think” is something that must be done directly by a human, and can not be automated.