Reader fatigue in radiology can lead to high medical errors. Out of more than 11,000 preventable hospital deaths in English NHS acute hospitals each year, over eight thousand deaths are attributed to diagnostic error.

Interpretation of medical images is a challenging task as it is repetitive in nature, and errors (false negatives) are relatively common. Radiologists are also subject to visual fatigue due to continuous and longtime exposure to computer screens for interpreting medical images. It has been found that radiological errors are more likely to occur at the end of shift as compared to the beginning.

Blinded Independent Central Review (BICR) is one method recommended by the FDA for registration oncology trials (1). The objective evaluation of clinical indicators involving radiological images is an important end-point factor in oncology trials. During BICR, readers are routinely monitored using different models to ensure quality reads.

Reader Monitoring Approaches

Monitoring of reviewers is not just mandatory from a regulatory viewpoint but also critical for proactive intervention throughout the trial. Monitoring safeguards high quality radiological assessments as well as adherence to specific clinical trial protocols. There is always inherent variability in the BICR process due to the difference in backgrounds, training, and humanity of the reviewers. Despite the existing inherent variability, there is a lack of meaningful methods for tracking and proactively improving reviewer performance.

“Double read with adjudication” is one of the frequently used models in various oncology trials review. This method permits better radiological assessments than a single read model. Double read with adjudication makes the process of BICR robust enough to not let variability and even minor errors impact the study outcome, if monitored properly.

Adjudication Rate (AR) has been used as a metric to track reviewer performance but does not accurately identify reviewer performance issues. It is dependent on many external variables like adjudication trigger, end point, indication, tumor burden etc. Adjudication agreement rate (AAR) is a relative performance indicator for a given reviewer where a higher adjudicator agreement rate suggests better reader performance.

A recent study suggests that RDI can also be used as a good surrogate for reader fatigue (3). RDI may be used on an effective daily / weekly / monthly basis against read volume for monitoring reader fatigue which affects variability and thus read quality. Increasing RDI trend may be used as an early indicator for potential reader quality and fatigue if a particular reader is concurrently reading on multiple trials.

The role of the patients who participate in clinical trials cannot be underestimated. Yet, we often don’t think about the value they get from their involvement, and how it impacts their daily lives.

Here we talk to Rob Marshall about his dual perspective on clinical trials – from his role in Calyx’s R&D team and from being a clinical trial patient.

What clinical trials have you participated in?

I’ve participated in numerous trials for asthma treatments. In fact, in one of those trials, my symptom improvement led my doctor to change my prescription to the treatment being studied. And I’m still benefiting from that treatment today.

I’m also the parent of a child in clinical trials. My daughter was diagnosed with a nut allergy when she was very young and for years, we ran the risk of her falling into a life-threatening anaphylactic shock if she wasn’t careful about what and where she ate.

So, when we had an opportunity to enroll her in a trial, we jumped on it. Even though the investigative site was 100 miles away, causing us to travel at least twice/month, often including overnight stays. We knew that by having her participate in the study she would eventually benefit from new treatments being developed. And we were right. In the past ten years, it’s been a delight to watch her eat a chocolate bar with nuts or order a piece of cake in a restaurant without worrying about what ingredients were used. It’s truly been life changing.

How does being a clinical trial participant affect your role as a Calyx Software Engineer?

I manage a couple of software engineering groups at Calyx, and we often have discussions around the features and outcomes of the products we’re developing. I’m in a unique position because I can offer insight into the end user’s experience, for example, what it’s like from the patient side to go through the randomization process. And I’m happy to give that context and to advance the conversation because not only am I helping engineer product technology that supports clinical trials, I’m also a user as well.

What motivates you to continue participating in trials, and in your day-to-day job at Calyx?

The life-changing outcomes my family and I have benefitted from drive my ongoing participation. It makes me proud to know that I’m doing my part, knowing it will benefit others who lives are disrupted by disease, just as mine and my daughter’s lives have been. I highly encourage everyone to get involved in trials for that reason.

As I go about my daily job, I try to challenge myself and my teams with how we can make the trial process better. How can we bridge the gap between developing our solutions and the investigative sites and patients that use them? My thinking is the more we understand what it’s like for our end users to participate in a trial, the better we can be at developing the solutions that will simplify their experiences, and ultimately contribute to the development of new treatment options that so many patients need.

In February 2022 the Journal for ImmunoTherapy of Cancer published “Comparison of tumor assessments using RECIST 1.1 and irRECIST, and association with overall survival.” This publication marks the first time immune-related criteria show a correlation with overall survival as the most meaningful endpoint in the treatment of cancer patients.

Calyx’s Head of Oncology, Dr. Oliver Bohnsack is considered one of the leading industry experts on oncology-imaging trial design and response criteria, having co-authored the immune-related response criteria (irRC, 2009), authored irRECIST (2014), and co-authored Comparison of Assessments using RECIST and irRECIST by Eggleton P. et al. (2020).

Here we discuss with Dr. Bohnsack the implications of these new response assessments, based on his contribution as a co-author of the 2022 paper, and what they mean for oncology research and treatment decision-making moving forward.

Q: Can you give us some background on how cancer treatment response is currently measured in clinical trials?

Response Evaluation Criteria In Solid Tumors (RECIST) is a set of published rules that define when cancer patients improve (“respond”), stay the same (“stable”) or worsen (“progression”) during treatments. The original RECIST was published in February 2000 by an international collaboration and updated to RECIST 1.1 in 2009.

Today, most clinical trials with imaging-based response or progression as an endpoint evaluating cancer treatments in solid tumors are using RECIST 1.1.

Q: Are there limitations to using RECIST 1.1 for tumor assessment?

Yes, there are. Patients treated with immune checkpoint inhibitors (ICIs), or any other form of immune-based treatment may experience pseudoprogression, which shows – at least on imaging – a tumor burden worsening before the treatment shows efficacy in and is visible on scans. In this situation these patients will be classified as progressive disease (PD) by RECIST V.1.1 and thus could lead to inappropriate treatment discontinuation.

As a result, immune-response criteria – irRECIST (Immune-related Response Evaluation Criteria In Solid Tumors) ‒ were developed and used in many clinical trials since to better capture novel response patterns seen with immune-based treatments and T-cell therapies.

Q: How does irRECIST differ from RECIST 1.1?

The irRECIST approach allows responses not typically observed in traditional systemic treatment to be identified and better documented. The guideline describes a standard approach to solid tumor measurement and definitions for objective change in tumor size which can be used not only in immunotherapy clinical trials. Where any new lesion seen with RECIST 1.1 defines a treatment failure with documented PD, irRECIST allows for a possible continuation and further evaluation taking new lesions and the whole tumor burden into consideration.

Q. Can you tell us about the current findings outlined in the 2022 paper you contributed to?

Working with researchers from Merck Healthcare KGaA and renown clinical institutions we pooled data from 1765 patients with 12 types of advanced solid tumors treated with avelumab (an anti-programmed death ligand 1 (PD-L1) monoclonal antibody) monotherapy in the JAVELIN Solid Tumor and JAVELIN Merkel 200 trials, conducted a comparative analysis of tumor assessments by investigators according to RECIST 1.1 and irRECIST, and evaluated the correlation between progression-free survival (PFS), continued patient benefit and overall survival (OS).

The use of irRECIST identified a subset of patients with a best overall response (BOR) of progressive disease by RECIST 1.1 but an irBOR of immune-related disease control by irRECIST with a distinctive survival curve, thereby providing more clinically relevant information and better treatment decision-making options than RECIST 1.1 alone.

Q. What are the implications for sponsors of ICI trials who use RECIST 1.1 for evaluating response to treatment in studies of solid tumors?

Because immune-targeted treatment can initially cause the tumor burden to look as if it is progressing, when in fact it is not, clinical trial sponsors using RECIST 1.1 to assess treatment response may end up discontinuing patients who would otherwise remain in the study. Using RECIST 1.1 could limit sponsors from recognizing the full treatment benefit of new therapies in development, and more importantly, may prevent patients from receiving potentially beneficial treatments.

Q: What can we expect moving forward, as the industry considers these findings and their implications?

Sponsors and regulatory agencies will have to consider whether based on this data RECIST 1.1 is still appropriate to be used and advocated for as the current assessment standard for physicians, aiding them in treatment decision making and whether to continue or discontinue the current immune oncology treatment of their patients. irRECIST includes and covers all that’s embedded in RECIST 1.1 already but now takes the entire tumor burden including new tumor growth into consideration. In my opinion irRECIST easily can and shall replace outdated RECIST 1.1 on all solid tumor trials going forward.