Situation
The customer brought Clario’s Science Advisory team into the early discussions, which is standard practice when we are called upon to provide rescue services. In our experience, errors in interpreting the protocol and its implications on system design are among the many possible sources of client dissatisfaction.
Clario’s Science Advisor focused on the functionality that had been implemented to collect what were described as “Adverse Events” (AE). “Adverse Events” can be interpreted in clinical research in different ways depending on the type of trial. In this instance, the inclusion of Adverse Events led to confusion among participants given their answer choices in the diary.
There are procedures for Adverse Event data that must be followed, especially due to the risk of an implementation that would force the eCOA system to be classified as a medical device. Upon close examination using the regulatory definitions for AEs, the Clario team determined that the instrument would not be interpreting AE data. The risk of medical device classification could be averted with the appropriate structure and naming conventions.
Upon close examination using the regulatory definitions for AEs, the Clario team determined that the instrument would not be interpreting AE data.
The initial eCOA endpoint provider had not provided any cautions or definition in the risk identification document about AEs. In contrast, Clario’s Science Advisors provide several important considerations in the course of consulting on protocol development – including the capture of potential AEs.
The Science Advisors’ detailed review exposed two questions in the previous implementation that provided sub-optimal instrument design, which would lead to problems with the sponsor’s data. The phrase “Adverse Events” was an answer choice in both questions, which had other unclear language that would yield poor quality data. This would likely fail to meet the identified needs of the sponsor (e.g., Acute Events). In consultation with the Project Team, Clario scientists suggested more involvement to overhaul the diary design, paying particular attention to retaining certain questions and the data most feasible to generate the desired data quality.
The Science Advisors made the following insights:
1. Answer choices hard to navigate
Participants would have difficulty distinguishing between Acute Events and Adverse Events. The terms that participants were asked to evaluate needed clarity in the interest of data quality, reliability, and validity.
2. The presence of free text fields
Clario discourages the use of free text, especially in case of AE reporting. An alert or notification can be triggered but automatic parsing of free text is complicated and imprecise, particularly when working in multiple languages.
3. Lack of navigational branching
The lack of branching based on previous answers led to participants skipping many questions and making errors about which questions to omit. This resulted in incomplete data and a loss of concurrent data collection.
4. Confusing wording of questions
Both the construction of the questions and the words used must be chosen carefully. Clario advised the sponsor to avoid clinical jargon or terms that may have specific meaning to the investigator and sponsor but not to a study participant. The term “adverse event” was such an example in this study.
5. Improper question sequencing
The order of questions did not align to the sequence of participant experience. This made it more difficult to navigate the questionnaires, which was exacerbated by the lack of navigational branching.
6. Collection of unnecessary detail
Several time-consuming questions collected information from participants that would never be used. For example, the sponsor of this infusion trial may wish to learn the number of infusion interruptions, the causes, and whether the interrupted infusion was later completed. The sponsor would want to know how much of the study drug went unused upon its return, as this gives insights to compliance. But the original eCOA implementation asked the participant to record exact stop and start times down to the minute for each infusion. This added unnecessary burden on respondents.
Solution
The key in this rescue project was understanding what the sponsor was trying to accomplish, then develop an instrument that answers these questions in the least burdensome manner for participants.
Having uncovered the six findings cited above, Clario’s scientific experts reduced a complex 46-question instrument to 16 essential questions. Moreover, the streamlined questions were presented in a sequence closely aligned with the process of drug administration. As a result, the eCOA instrument collected data with:
- More reliability
- Better quality
- Higher compliance
- Lower participant burden