Reader fatigue in radiology can lead to high medical errors. Out of more than 11,000 preventable hospital deaths in English NHS acute hospitals each year, over eight thousand deaths are attributed to diagnostic error.

Interpretation of medical images is a challenging task as it is repetitive in nature, and errors (false negatives) are relatively common. Radiologists are also subject to visual fatigue due to continuous and longtime exposure to computer screens for interpreting medical images. It has been found that radiological errors are more likely to occur at the end of shift as compared to the beginning.

Blinded Independent Central Review (BICR) is one method recommended by the FDA for registration oncology trials (1). The objective evaluation of clinical indicators involving radiological images is an important end-point factor in oncology trials. During BICR, readers are routinely monitored using different models to ensure quality reads.

“Reviewer Disagreement Index may be used as an early indicator for potential reader quality and fatigue if a particular reader is concurrently reading on multiple trials.”

– Manish Sharma, MD, VP Medical Imaging, Calyx

Reader Monitoring Approaches

Monitoring of reviewers is not just mandatory from a regulatory viewpoint but also critical for proactive intervention throughout the trial. Monitoring safeguards high quality radiological assessments as well as adherence to specific clinical trial protocols. There is always inherent variability in the BICR process due to the difference in backgrounds, training, and humanity of the reviewers. Despite the existing inherent variability, there is a lack of meaningful methods for tracking and proactively improving reviewer performance.

“Double read with adjudication” is one of the frequently used models in various oncology trials review. This method permits better radiological assessments than a single read model. Double read with adjudication makes the process of BICR robust enough to not let variability and even minor errors impact the study outcome, if monitored properly.

Adjudication Rate (AR) has been used as a metric to track reviewer performance but does not accurately identify reviewer performance issues. It is dependent on many external variables like adjudication trigger, end point, indication, tumor burden etc. Adjudication agreement rate (AAR) is a relative performance indicator for a given reviewer where a higher adjudicator agreement rate suggests better reader performance.

Evaluating Reader Fatigue with Reviewer Disagreement Index

In an earlier blog, we presented how Reviewer disagreement index (RDI) considers the subjects for which adjudicator disagreed with the reviewer and considers adjudicator disagreement relative to the total number of cases read. RDI considers both the overall AR for a study and the individual AAR for each reader. The RDI indicates the percentage of disagreed cases for a given reader across the total number of cases read, as defined in the equation below, where a low RDI value indicates better reader performance and high RDI value indicates poor reader performance.

RDI =
# of cases where adjudicator disagreed with given reader
Total # of all cases read

Not surprisingly, there might be a correlation between the number of studies a reader reads on and how the cases are distributed across all studies. If the cases are distributed across too many studies, it can be a major issue contributing to reader fatigue e.g., if the reader in a reading session of 2 hours reads a few cases each on 10 different trials. The best-case scenario is to have a fine balance that would allow a reader a majority of cases on the same study in a reading session.

The plots below show a weak correlation of decreasing timepoints (Y-axis) to increasing RDI (X-axis). RDI can be used to give an early insight to trends by monitoring the ratio of RDI on each study per week, or per reading session and its interplay with overall and other studies’ RDI.

A recent study suggests that RDI can also be used as a good surrogate for reader fatigue (3). RDI may be used on an effective daily / weekly / monthly basis against read volume for monitoring reader fatigue which affects variability and thus read quality. Increasing RDI trend may be used as an early indicator for potential reader quality and fatigue if a particular reader is concurrently reading on multiple trials.

Reference

1. Guidance for Industry Developing Medical Imaging Drug and Biologic Products. Part 3: Design, Analysis, and Interpretation of Clinical Studies. US Department of Health and Human Services. Food and Drug Administration. Center for Drug Evaluation and Research. Center for Biologics Evaluation and Research; 2004.

2. Clinical Trial Imaging Endpoints Process Standards Guidance for Industry Draft. US Department of Health and Human Services. Food and Drug Administration. Center for Drug Evaluation and Research. Center for Biologics Evaluation and Research. March 2015 Revision1.

3. Manish Sharma, Madhuri Madasu, Sree Sudha Kota, Surabhi Bajpai, Yibin Shao, Srinivas Pasupuleti, Michael O’Connor, “Using reader disagreement index as a tool for monitoring impact on read quality due to reader fatigue in central reviewers,” Proc. SPIE 12035, Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment, 120350J (4 April 2022); doi: 10.1117/12.2613082

Clinscience extends Calyx’s reliable, expedited imaging services to its global pharmaceutical customers

 Nottingham, England and Morrisville, NC – June 9, 2022 Calyx, the eClinical and Regulatory solutions and services provider relied on for solving complex data challenges in clinical research, today announced it has been named an approved provider of centralized medical imaging services by Clinscience, a full service CRO offering smart, end-to-end clinical trial services to the global pharmaceutical industry. 

“The Clinscience team understands what’s important to pharmaceutical and biotech organizations,” said Elizabeth Dalton, Vice President, Channel Partnerships Solutions, Calyx. “We’re honored to work with Clinscience and to deliver the reliable imaging services their customers need to accelerate and optimize their clinical trials.”

Clinscience joins a long list of global pharmaceutical companies and CROs who repeatedly leverage Calyx’s scientific expertise and experience to assess the safety, efficacy, and effectiveness of their compounds. To date, Calyx Medical Imaging has been used in over 2,600 clinical trials worldwide and has supported the approvals of over 270 medical treatments across a broad range of therapeutic areas.

Click here for more information on how CROs – and their clients – benefit from partnering with Calyx.

About Calyx

Through innovative eClinical and Regulatory solutions and services, Calyx turns the uncertain into the reliable, helping bring new medical treatments to market reliably. With deep expertise in clinical development and 30 years supporting trial sponsors and clinical research organizations, Calyx harnesses its intelligence and experience to solve complex problems, deliver fast insights, and get new drugs to market every day.

Medical Imaging | IRT | CTMS | EDC | RIM

Take your trials further with intelligent insights at Calyx.ai or at LinkedIn, Twitter, or Facebook.

About Clinscience

Clinscience is a global CRO company offering smart CRO Services, from protocol creation to final study report development. The Company has offices in Poland, Spain, Italy, Germany, and the US and provides its services in 6 European countries.  More than 150 Biotechnology companies across Europe and the US trust the Clinscience brand. The Company’s procedures and Data-driven approach, harnessed with the Genius Suite™ technology, work together to give the Clients power and flexibility to address even the most unique needs.  Clinscience is part of a publicly-traded parent company, Neuca Plc, an organization in the pharmaceutical wholesaling and healthcare industry that provides the structure to meet the needs of commercial and non-commercial studies.

Visit www.clinscience.com for more information.

Contact:

Christine Tobin | [email protected] | +1 412-628-8598

The role of the patients who participate in clinical trials cannot be underestimated. Yet, we often don’t think about the value they get from their involvement, and how it impacts their daily lives.

Here we talk to Rob Marshall about his dual perspective on clinical trials – from his role in Calyx’s R&D team and from being a clinical trial patient.

Hear directly from Rob as he tells his complete story here.

 

What clinical trials have you participated in?

I’ve participated in numerous trials for asthma treatments. In fact, in one of those trials, my symptom improvement led my doctor to change my prescription to the treatment being studied. And I’m still benefiting from that treatment today.

I’m also the parent of a child in clinical trials. My daughter was diagnosed with a nut allergy when she was very young and for years, we ran the risk of her falling into a life-threatening anaphylactic shock if she wasn’t careful about what and where she ate.

So, when we had an opportunity to enroll her in a trial, we jumped on it. Even though the investigative site was 100 miles away, causing us to travel at least twice/month, often including overnight stays. We knew that by having her participate in the study she would eventually benefit from new treatments being developed. And we were right. In the past ten years, it’s been a delight to watch her eat a chocolate bar with nuts or order a piece of cake in a restaurant without worrying about what ingredients were used. It’s truly been life changing.

“It’s an honor to be part of the clinical trials industry and I’ve seen firsthand how the eClinical solutions we develop are making a difference.”

– Rob Marshall, Sr. Director, Software Engineering, Calyx

Has the use of eClinical technologies impacted your clinical trial experience?

It’s improved the experience massively. For example, when I first started clinical trials in 2010, I had to go to the investigative site for spirometer tests to evaluate my exhalation levels. They’d cart in a big machine with a monitor, graphs, etc. that I’d have to blow into repeatedly (figure 1). If it wasn’t configured correctly (which was often the case), we’d have to start all over again, which lengthened the amount of time and effort for patients and clinicians involved in each visit.

Spirometer
Figure 1: Asthma patients previously had to visit investigative sites routinely for spirometer tests during clinical trials
Variety of trial data capture devices
Figure 2: eClinical technologies, like handheld spirometers and mobile apps, enable patients to provide important trial data remotely, saving them and the investigative sites time and hassle.

Now, 12 years later, I’m given a hand-held spirometer device that fits in my pocket and connects to an app on a phone. I can easily use the device to test my exhalation levels wherever I am, and the data is automatically uploaded to the study database. It’s remarkable how easy it is and saves a lot of time and hassle that I used to go through when having to go to the investigative site on a regular basis.

How does being a clinical trial participant affect your role as a Calyx Software Engineer?

I manage a couple of software engineering groups at Calyx, and we often have discussions around the features and outcomes of the products we’re developing. I’m in a unique position because I can offer insight into the end user’s experience, for example, what it’s like from the patient side to go through the randomization process. And I’m happy to give that context and to advance the conversation because not only am I helping engineer product technology that supports clinical trials, I’m also a user as well.

What motivates you to continue participating in trials, and in your day-to-day job at Calyx?

The life-changing outcomes my family and I have benefitted from drive my ongoing participation. It makes me proud to know that I’m doing my part, knowing it will benefit others who lives are disrupted by disease, just as mine and my daughter’s lives have been. I highly encourage everyone to get involved in trials for that reason.

As I go about my daily job, I try to challenge myself and my teams with how we can make the trial process better. How can we bridge the gap between developing our solutions and the investigative sites and patients that use them? My thinking is the more we understand what it’s like for our end users to participate in a trial, the better we can be at developing the solutions that will simplify their experiences, and ultimately contribute to the development of new treatment options that so many patients need.

A Conversation with Oliver Bohnsack, MD, PhD, MBA

In February 2022 the Journal for ImmunoTherapy of Cancer published “Comparison of tumor assessments using RECIST 1.1 and irRECIST, and association with overall survival.” This publication marks the first time immune-related criteria show a correlation with overall survival as the most meaningful endpoint in the treatment of cancer patients.

Calyx’s Head of Oncology, Dr. Oliver Bohnsack is considered one of the leading industry experts on oncology-imaging trial design and response criteria, having co-authored the immune-related response criteria (irRC, 2009), authored irRECIST (2014), and co-authored Comparison of Assessments using RECIST and irRECIST by Eggleton P. et al. (2020).

Here we discuss with Dr. Bohnsack the implications of these new response assessments, based on his contribution as a co-author of the 2022 paper, and what they mean for oncology research and treatment decision-making moving forward.

“In my opinion irRECIST easily can and shall replace outdated RECIST 1.1 on all solid tumor trials going forward.”

– Dr. Oliver Bohnsack, VP, Medical Imaging & Head of Oncology, Calyx

Q: Can you give us some background on how cancer treatment response is currently measured in clinical trials?

Response Evaluation Criteria In Solid Tumors (RECIST) is a set of published rules that define when cancer patients improve (“respond”), stay the same (“stable”) or worsen (“progression”) during treatments. The original RECIST was published in February 2000 by an international collaboration and updated to RECIST 1.1 in 2009.

Today, most clinical trials with imaging-based response or progression as an endpoint evaluating cancer treatments in solid tumors are using RECIST 1.1.

Q: Are there limitations to using RECIST 1.1 for tumor assessment?

Yes, there are. Patients treated with immune checkpoint inhibitors (ICIs), or any other form of immune-based treatment may experience pseudoprogression, which shows – at least on imaging – a tumor burden worsening before the treatment shows efficacy in and is visible on scans. In this situation these patients will be classified as progressive disease (PD) by RECIST V.1.1 and thus could lead to inappropriate treatment discontinuation.

As a result, immune-response criteria – irRECIST (Immune-related Response Evaluation Criteria In Solid Tumors) ‒ were developed and used in many clinical trials since to better capture novel response patterns seen with immune-based treatments and T-cell therapies.

Q: How does irRECIST differ from RECIST 1.1?

The irRECIST approach allows responses not typically observed in traditional systemic treatment to be identified and better documented. The guideline describes a standard approach to solid tumor measurement and definitions for objective change in tumor size which can be used not only in immunotherapy clinical trials. Where any new lesion seen with RECIST 1.1 defines a treatment failure with documented PD, irRECIST allows for a possible continuation and further evaluation taking new lesions and the whole tumor burden into consideration.

Q. Can you tell us about the current findings outlined in the 2022 paper you contributed to?

Working with researchers from Merck Healthcare KGaA and renown clinical institutions we pooled data from 1765 patients with 12 types of advanced solid tumors treated with avelumab (an anti-programmed death ligand 1 (PD-L1) monoclonal antibody) monotherapy in the JAVELIN Solid Tumor and JAVELIN Merkel 200 trials, conducted a comparative analysis of tumor assessments by investigators according to RECIST 1.1 and irRECIST, and evaluated the correlation between progression-free survival (PFS), continued patient benefit and overall survival (OS).

The use of irRECIST identified a subset of patients with a best overall response (BOR) of progressive disease by RECIST 1.1 but an irBOR of immune-related disease control by irRECIST with a distinctive survival curve, thereby providing more clinically relevant information and better treatment decision-making options than RECIST 1.1 alone.

The publication demonstrates the benefit to a subgroup of patients in each of its various analyzed tumor indications, who otherwise would have foregone treatment and survival benefit when relying solely on RECIST 1.1 instead of irRECIST.

Q. What are the implications for sponsors of ICI trials who use RECIST 1.1 for evaluating response to treatment in studies of solid tumors?

Because immune-targeted treatment can initially cause the tumor burden to look as if it is progressing, when in fact it is not, clinical trial sponsors using RECIST 1.1 to assess treatment response may end up discontinuing patients who would otherwise remain in the study. Using RECIST 1.1 could limit sponsors from recognizing the full treatment benefit of new therapies in development, and more importantly, may prevent patients from receiving potentially beneficial treatments.

Q: What can we expect moving forward, as the industry considers these findings and their implications?

Sponsors and regulatory agencies will have to consider whether based on this data RECIST 1.1 is still appropriate to be used and advocated for as the current assessment standard for physicians, aiding them in treatment decision making and whether to continue or discontinue the current immune oncology treatment of their patients. irRECIST includes and covers all that’s embedded in RECIST 1.1 already but now takes the entire tumor burden including new tumor growth into consideration. In my opinion irRECIST easily can and shall replace outdated RECIST 1.1 on all solid tumor trials going forward.

Expertise in medical imaging, IRT, and EDC solution delivery to benefit ProTrials’ worldwide customers

Nottingham, England and Morrisville, NC – March 21, 2022 Calyx, the eClinical and Regulatory solutions and services provider relied on for solving complex data challenges in clinical research, today announced it has been named an approved partner by ProTrials Research, Inc., a mid-sized full-service clinical research organization (CRO) specializing in delivering clinical operations services to the pharmaceutical, biotechnology, and medical device industries.

“We’re pleased to partner with Calyx and are confident that our customers will benefit from the scientific, medical, and clinical expertise they have honed during their 30 years of delivering reliable eClinical solutions to the clinical development industry,” said Christy Meyer, Director, Quality Assurance, ProTrials.

The partnership enables Calyx to extend its proven medical imaging, interactive response technology (IRT) and electronic data capture (EDC) solutions and services to help ProTrials’ worldwide customers achieve their clinical development objectives. ProTrials’ clients will leverage Calyx’s innovative technology to improve the reliability of clinical trial outcomes data, ultimately enabling them to deliver safe and effective medical treatments to patients in need.

“We look forward to delivering the important imaging and eClinical data ProTrials’ customers rely on as they evaluate the safety and efficacy of often life-saving medical treatments,” said Elizabeth Dalton, Vice President, Technical Solutions, Calyx. “We’re honored that ProTrials selected Calyx to support them as they deliver on their commitment to improve the health and extend the lives of patients worldwide.”

Click here for more information on how CROs – and their clients – benefit from partnering with Calyx.

About Calyx

Through innovative eClinical and Regulatory solutions and services, Calyx turns the uncertain into the reliable, helping bring new medical treatments to market reliably. With deep expertise in clinical development and 30 years supporting trial sponsors and clinical research organizations, Calyx harnesses its intelligence and experience to solve complex problems, deliver fast insights, and get new drugs to market every day.

Medical Imaging | IRT | CTMS | EDC | RIM

Take your trials further with intelligent insights at Calyx.ai or at LinkedIn, Twitter, or Facebook.

About ProTrials Research, Inc.

ProTrials Research, Inc., is a mid-sized full-service clinical research organization (CRO) headquartered in Los Gatos, CA, with clinical operations personnel located throughout North America and across the world. Since our launch in 1996 as a woman-owned business, we have been driven by a shared commitment to provide outstanding service to the clinical research industry. To this day, that commitment remains our guiding principle and is reflected in our high-repeat business rate. ProTrials works with sponsors in the pharmaceutical, biotechnology and medical device industries to deliver high-quality clinical operations services such as study management, investigative site monitoring, grant services, project and program management, biometrics including biostatistics, pharmacovigilance, and associated clinical development services. Our operations support expands into Europe, Asia-Pacific, and Latin American countries. To learn how ProTrials Research, Inc., can help advance your clinical trial success, please visit us at www.protrials.com or call 650.386.7712.

Contact:

Christine Tobin | [email protected] | +1 412-628-8598

Concerns over variability in the clinical interpretation of medical images performed by radiologists accompany most trials that involve imaging biomarkers. Such variability in clinical trial assessment outcomes is due to the inherent variability of interpreting an image, interpretation of assessment criteria, and definition of potentially measurable / non-measurable and/or equivocal / unequivocal disease.

Several studies have been performed at various academic centers to evaluate this variability in image interpretation among experienced radiologists who reviewed images in a standardized manner and all of them have concluded this variability is not a reliable predictor of read quality.

Having acknowledged the innate variability in image interpretation and the fact that this variability does not simply equate to a reader error, a deep dive into the factors causing this variability is crucial to identify an acceptable benchmark. Consistently monitoring the variability in reads and analyzing the trends gives an opportunity for data-driven monitoring and early intervention, if needed.

An FDA published guidance document, “Clinical Trial Imaging Endpoint Process Standards Guidance for Industry” further advocates the role of monitoring reader performance. Based on this there are several metrics defined to monitor reader performance, which are not just a measure of past performance but also a robust predictor or trends / bias.

“Reviewer Disagreement Index offers advantages of identifying the most discordant reviewer, which may otherwise be missed during traditional reviewer performance monitoring.”

– Manish Sharma, MD, Vice President, Medical Imaging, Calyx

Despite the well-established assessment criteria in many therapeutic areas (RECIST 1.1 for example), there is always going to be inherent variability in the blinded independent central review (BICR) process due to the differing backgrounds, training, and humanity of the reviewers. Despite this inherent variability, the industry has failed to produce meaningful methods for tracking and proactively improving reviewer performance.

Reviewer Adjudication Rate (AR), for example, has been used as a metric to track reviewer performance, but does not accurately identify reviewer performance issues. Adjudication in BICR is triggered when two reviewers’ assessments do not agree. For example, if the first reviewer performs their assessment perfectly and the second reviewer fails to assess correctly, an adjudication will be triggered. AR as a performance indicator would falsely identify the first reviewer as potentially having a performance issue.

Access additional information on how the assessment of imaging data by an independent entity reduces evaluation bias and improves assessment consistency in clinical trials.

READ NOW

Introducing the Reader Disagreement Index (RDI)

To improve the industry’s standard metrics used to assess and monitor central reviewers, a retrospective review of 20 oncology studies (7,136 subjects and 32,536 timepoints) using RECIST 1.0 or RECIST 1.1 rules was conducted. This review validated the increase in performance monitoring of two novel Key Performance Indicators (KPIs). These KPIs include:

Adjudication Agreement Rate (AAR)
  • a relative performance indicator for a given reviewer as compared to the other reviewers in a given study, with a higher adjudicator agreement rate suggesting better reader performance
Reviewer Disagreement Index (RDI)
  • considers the subjects for which adjudicator disagreed with the reviewer and considers adjudicator disagreement relative to the total number of cases read

RDI proves to be a more reliable quality indicator as compared to AR and AAR, as RDI can additionally identify the discordant reader, therefore improving its reliability. RDI offers advantages of identifying the most discordant reviewer that may be missed by analysis of AR and AAR alone for reviewer performance monitoring.

Adding automated analysis of all or selected discordant assessment pairs for each reviewer in a study further improves the ability to monitor reader interpretation performance at a detailed level. Once a probability of low AAR has been ‘flagged,’ it automatically prompts the imaging core lab to further evaluate the signal. Discrepancy grids/assessment pair grids improve the capability to monitor BICR reviewers’ performance in specific trials. These methods can be used to explore a reviewer having a low overall AAR or high RDI.

When deployed into a BICR workflow, these metrics enable a dynamic, responsive risk model that is more accurate than historical monitoring tools and automatically indicates the precise intervention needed to rectify reviewer performance issues. Critically, this reduces the number of reviewer performance issues by constantly monitoring the proximity to risk thresholds.

By monitoring the performance trajectory, Calyx uses these metrics to proactively intervene before an issue occurs, improving not only the validity of the BICR results, but also patient safety and outcomes.  The biggest advantage of the system is that once set up, the regression model can utilize supervised learning algorithms to improve its predictability, reducing repetitive programming needs, and also allow medical monitors actionable outliers.

The multi-dimensional, real-time monitoring metrics add dimension to Calyx’s quality assessment of reviewer performance, ultimately improving the quality and integrity of data that pharmaceutical companies rely on to determine the safety and efficacy of new medical treatments.

References
  1. Guidance for Industry Developing Medical Imaging Drug and Biologic Products. Part 3: Design, Analysis, and Interpretation of Clinical Studies. US Department of Health and Human Services. Food and Drug Administration. Center for Drug Evaluation and Research. Center for Biologics Evaluation and Research; 2004.
  2. Clinical Trial Imaging Endpoints Process Standards Guidance for Industry Draft. US Department of Health and Human Services. Food and Drug Administration. Center for Drug Evaluation and Research. Center for Biologics Evaluation and Research. March 2015 Revision1.

Stay Updated

Never miss a beat. Sign up to receive emails covering industry news and useful content to help you advance clinical development.

  • This field is for validation purposes and should be left unchanged.