User Acceptance Testing
Outsourcing services is quite common in clinical development. We outsource everything…
- Lab analysis of subject samples (central clinical labs)
- Interpretation of subject medical images (central imaging readers)
- Data capture at clinical sites (EDC and ePRO)
- Randomization and management of investigational product (IVRS/IWRS)
- Trial master files (eTMF)
…and the list goes on and on.
These outsourced services rely on computer systems. The difference between the first 2 and the last 3 services answers the question:
When do you need to conduct UAT?
What makes the first 2 services different from the last 3? The users.
In the first 2 situations, the systems are used (almost) exclusively by the service provider, making the service provider responsible for their own UAT.
In the last 3 situations, the sponsor requires clinical trial subjects, clinical investigator site staff, or their own personnel to use the systems. The sponsor provides their requirements to the service provider, who develops and tests the systems internally.
Because the sponsor defines the intended use of the system, the sponsor is ultimately responsible for verifying the system is fit for intended use.
This means UAT.
You need an SOP
These outsourced situations often bypass many internal IT controls. When your IT department deploys a system with you, they have SOPs you can follow to perform UAT. Those SOPs include how you’ll interact with IT to report unexpected results and get them resolved.
What happens when you don’t have an SOP? You get well-meaning people doing happy-go-lucky testing with no documentation, no assurance that testing is appropriate, and no formal feedback loop to the provider.
Start with an SOP that requires documenting
- Requirements you’ll be testing
- Test cases
- Executing test cases and capturing evidence
- Handling unexpected results and their resolution
Benefits of UAT
2 of the biggest benefits of UAT are
- Having confidence in your data
- Knowing the system works as intended
There are additional benefits you may not have thought of before for your customers and your service providers.
Customers
Principal investigators are your customers. When you deploy unreliable, buggy systems to them, you create ill will, reluctance to do business with you in the future, and an environment that can result in complaints to Institutional Review Boards, Ethics Committees, and regulators.
Conversely, when you deploy reliable, bug-free systems to principal investigators, you instill good will, confidence, and a desire to do business with you again.
Which kind of investigator do you want interacting with a regulator during a site inspection?
Service Providers
It is, unfortunately, not uncommon for service providers to give you access to systems for UAT only for you to discover bugs. Sometimes showstoppers. Sometimes simply irritating. You may feel like you’re doing testing they should have already performed.
When you have an SOP for UAT that requires tracking unexpected results and how they are handled, you can provide clear feedback to your provider.
Your vendor quality management oversight program just got that much more effective!
Where is it written?
Let’s face it. An oft-voiced objection to putting this kind of effort into UAT is that it’s not an explicit regulatory requirement. And while that’s true, if the ideas of having data you can trust, metrics you can hold your providers accountable for, and systems that operate for your customers as intended aren’t enough, then
Know that the regulators are looking.
FDA inspectors follow Compliance Program Guidance Manual 7348.810 when inspecting sponsors, monitors, or CROs. Among other things, the Manual requires inspectors to
“Determine if the sponsor/CRO has procedures to demonstrate that the computerized system was tested for its intended use (e.g., documentation of user acceptance testing).”
What are your thoughts?
Have success stories and challenges to share? Insights into how other regulatory agencies view UAT? Please share them in the Comments box below.
From my background in development (in a different industry), I’m familiar with customers conducting UAT of new systems we would develop for them. What about UAT when a vendor provides a system that is new to the sponsor organization, but perhaps is an established system for the vendor, part of a tried-and-true solution? Do you advocate UAT in this situation? If so, any tips about the differences between UAT of new and established systems? Thanks!
Yes, I do recommend performing UAT in situations where the sponsor or their representatives/business partners will be using the system. The extent of UAT depends on many factors, including the impact of the records, the extent of configuration and customization, and how much you can leverage from the provider. Systems can radically change user processes too, so I’d leverage UAT to demonstrate the new process works. What do you think?