1. Performance Evaluation of In Vitro Diagnostic Medical Devices Under Regulation (EU) 2017/746 (IVDR)
May 26, 2022, marked a significant milestone for manufacturers of in vitro diagnostic medical devices. The European Union’s Regulation 2017/746, commonly known as IVDR, replaced the In Vitro-Diagnostic Medical Devices Directive 98/79/EC (IVDD), which was in force for less than two decades. The shift from IVDD to IVDR, coupled with compliance with new requirements, has posed challenges for manufacturers, often requiring increased human and financial resources. The IVDR aligns with global requirements but introduces more stringent restrictions than the IVDD. This updated regulation provides comprehensive and clear guidance for the performance testing of in vitro diagnostic medical devices.
Compared to IVDD, IVDR imposes rigorous requirements for collecting and analyzing clinical evidence, with stringency increasing proportionally to the device class. It also mandates manufacturers to compile and submit a performance evaluation of in vitro diagnostic medical devices, showcasing their safety and efficacy for the intended purpose specified by the manufacturer. The IVDR emphasizes the importance of ongoing performance evaluation by requiring the manufacturers to monitor post-market performance throughout the device’s lifecycle.
These changes aim to introduce more effective regulatory control mechanisms to improve patient safety and health protection.
2. IVDR Regulation: Articles 58-65 and Performance Studies Planning
The IVDR classifies studies according to patient safety. Special attention is paid to interventional clinical performance studies, surgically invasive sample-taking, and studies involving additional invasive procedures or other risks to participants.
In an interventional clinical performance study, the test results may influence patient management decisions or guide treatment. Another type is an invasive study, which includes surgical sample-taking and additional invasive procedures that put the patient at risk. In practice, any intentionally performed additional sample-taking for the purpose of a prospective performance study will be considered an invasive study.
The IVDR also distinguishes performance studies involving companion diagnostics. Additionally, the regulation states that testing leftover samples requires fewer procedures.
Article 58 outlines the role and requirements of investigators and the facilities where the performance study will be conducted. It states that the investigator must have relevant professional qualifications, scientific knowledge, and experience in patient care or laboratory medicine. These criteria apply to all members of the research team. Additionally, the facility should be similar to the facilities where the device is intended to be used. In most cases, it is advisable to plan the study in a diagnostic laboratory that routinely conducts analyses similar to the performance study.
Article 59 of the IVDR deals with the informed consent for performance studies. It outlines primary considerations for ensuring participant comfort and well-being.
Information provided to the participant or their legal representative should allow them to comprehend key aspects of the study, including its purpose, benefits, effects, and risks. Additionally, participants should understand their rights, including the right to withdraw from the study and the conditions under which the study will be conducted. This information should be clear and concise and cover details of any compensation issues. Participants must acquire all of this knowledge before the performance study. Hence, it is the responsibility of the investigative team to ensure that the participant comprehends the content of the signed documentation.
Performance studies involving participants who cannot provide informed consent, minors, and pregnant or nursing women are particularly sensitive projects where special attention must be paid to the protection and safety of participants and additional precautions implemented to ensure the comfort of participants.
Emergency studies are permitted if the requirements of Article 64 are met, including cases where it is impossible to obtain the legal representative’s consent during the therapeutic window, provided that the investigator confirms that he or she is unaware of any contraindications to participation in the performance study.
The insurance of performance studies carried out in Poland is governed by the Regulation of the Minister of Finance of 2 December 2002 on the compulsory insurance of the sponsor and the investigator for the conduct of a clinical trial of a device or a performance study of an in vitro diagnostic medical device. The regulation outlines the insurance amounts that vary based on the number of study participants.
3. Annex XIII – Performance Evaluation
Let us begin by emphasizing the critical nature of diagnostic tests in providing accurate results. The significance is evident, as it influences whether a patient receives the appropriate medication and how swiftly they recover. In the context of self-testing devices, it determines whether a patient seeks medical attention. On the one hand, a delayed response could have adverse consequences. On the other hand, too many patients with false-positive results can overload the healthcare system.
Nevertheless, it is crucial to acknowledge that classifying devices as strictly “good” or “bad” is not feasible. Financial considerations impact the accessibility of diagnostic medical devices for both non-professional users and specialists.
So, how do we discern between more and less reliable devices? We need to examine the device’s performance characteristics outlined in the instructions for use.
The procedure to demonstrate a device’s functionality and suitability for the European market is stipulated in the IVDR. The whole process is called performance evaluation. Annex XIII of the IVDR divides it into three parts: verification of scientific validity, verification of analytical performance, and verification of clinical performance of a device for its intended purpose, as stated by the manufacturer. Additionally, post-market performance follow-up and continuous data updates must be carried out throughout the product’s life cycle.
a. Scientific validity
Demonstrating scientific validity applies to both the device and the technology employed in detecting the intended analyte. We can achieve this validation by using existing data, where available, considering the generally acknowledged state-of-the-art. This involves a systematic review of literature data, results of other studies, expert opinions, positions from relevant professional associations, and proof of concept studies.
In cases where existing evidence is deemed insufficient or the product is innovative, such as new analytes or intended purposes, manufacturers must provide a scientific rationale and generate new or additional data. This necessitates conducting a gap analysis to determine the additional evidence required.
It is crucial to define a literature search protocol before commencing data collection. This strategy must be thorough and objective, identifying favorable and unfavorable data. The manufacturer is required to produce suitable documentation for the subsequent assessment of the methodology, verification of results, and replication of the search outcomes.
Following the collection and/or generation of evidence supporting the scientific validity, the manufacturer should evaluate, analyze, and report the obtained data.
b. Analytical performance
The next step in the performance evaluation involves laboratory testing, which assesses the analytical performance of the in vitro diagnostic device. By definition, analytical performance evaluation focuses on the device’s capability to detect or measure a specific analyte accurately. This evaluation is tailored to each device type, with the regulatory parameters outlined in Annex I and Annex II of the IVDR. Each device will require a complete set of performance tests.
Consider a device designed for the qualitative diagnosis of influenza, utilizing rapid immunochromatographic technology, yielding results in 10-15 minutes. The planned tests will include determining the device’s limit of detection (i.e., the minimum concentration that the device can detect), verifying the hook effect, assessing analytical specificity by testing cross-reactions or potential interferences affecting the test result, and establishing precision through multiple series of tests carried out at different locations by different operators.
However, planning tests for a more challenging product, such as reagents for a quantitative PCR reaction performed on multiple samples simultaneously with an analyzer, requires additional considerations. Beyond basic parameters such as measurement range or analytical specificity, assessing the linearity of the assay, carryover resistance, measurement bias, and determining the cut-off value become essential.
c. Clinical performance
The third and most crucial phase of the performance evaluation involves assessing the clinical performance of an in vitro diagnostic device. According to the IVDR, clinical evidence can stem from peer-reviewed scientific literature or data from published experience gained by routine diagnostic testing. Nevertheless, the primary source of clinical evidence remains a traditional clinical performance study conducted against a reference method. The clinical performance study report constitutes a fundamental part of the technical documentation of an IVD device.
Clinical performance studies often serve as the initial end-user verification for a device entering the market. The entire process, from specimen collection to result reading, undergoes rigorous testing.
An impartial investigator should conduct clinical performance studies at an independent facility. For specific reasons, this study is termed a performance study rather than a clinical trial. Typically, patients are only involved during the sample collection performed by trained medical personnel. The collected sample is divided into two parts, or two identical clinical samples are collected simultaneously. The samples are tested concurrently using two methods. The first method is the one used by the device under test, and the result obtained from this method is recorded for statistical calculations. Simultaneously, another instrument is used to perform a separate test. This parallel testing allows for a comprehensive comparison of the device’s performance against an alternative method.
The technology of the reference method should be similar to the method being tested, and this can be achieved by utilizing a device already on the market employing the same or similar technology. Alternatively, we can also indirectly compare the results of these tests with those acquired through other assays. The utmost priority is to establish the most reliable comparison between the results obtained with the tested device and the actual clinical condition of the patient.
Tests can vary, so let us revisit our example. For a cassette designed to detect the presence or absence of the influenza virus antigen through immunochromatography, we will design a study that involves collecting two nasal or nasopharyngeal swabs from patients displaying respiratory disease symptoms. One of these samples will be tested using the device under evaluation, while the other will be tested using an alternative immunochromatographic method or a more sensitive qualitative RT-PCR. We will compare the results and ascertain the parameters, including diagnostic sensitivity and specificity, positive and negative predictive value, and reliability index. Of course, the results will be subject to some uncertainty. The more accurately we select the size of the study group and the study population, the more reliable results we will report.
Let us analyze our second example, PCR reagents for quantifying an analyte like antibiotic-resistant Klebsiella pneumoniae bacteria. In this scenario, we will divide the sample obtained from the participant into two parts. Subsequently, we will conduct two PCR analyses—one using the test method and the other using the reference method. Then, we will compare the results from both analyses.
4. ISO 20916:2019 or Good Clinical Practice in Performance Studies
The final section explores the types of performance studies delineated by both IVDR and ISO 20916:2019. An interventional clinical performance study is the initial and riskiest type of performance study regarding the participant’s health. In this setup, healthcare professionals make treatment decisions based on the results of the patient’s test. This procedure is typically reserved for cases without an approved diagnostic method, such as dealing with a new, previously undiscovered pathogen. The interventional study is closely monitored. The reference analysis will include collecting indirect clinical evidence of the patient’s condition.
Another type of study outlined by ISO 20916:2019 is the invasive study, where patients face additional health risks. It corresponds with Article 58 of the IVDR, which outlines conditions for surgically invasive sample-taking exclusively for performance study or when the study involves additional invasive procedures or poses other risks to the participants. In such studies, a reference method can be easily identified, and the process can be designed so that decisions by healthcare professionals are based on an established test method. While the result of this test may not impact the clinical process, it holds significance for the patient, who will be exposed to additional risks associated with the study team taking an extra sample in a manner that is not always entirely comfortable or safe.
As per the ISO 20916:2019 standard, the last category of trials involves the utilization of archived samples or leveraging patients’ historical clinical data. This category also includes studies where a sample is not explicitly collected for the study; instead, they utilize residual biological material left over from the standard diagnostic cycle. According to ISO 20916:2019, this type of study does not require such strict monitoring. In some cases, just notifying the authority will suffice, and purchasing additional insurance for potential patient harm might not be necessary.
In summary, in performance studies, as in any clinical trial, the participants’ safety and comfort are paramount. The Principal Investigator, research team, sponsor, and Contract Research Organization (CRO) all adhere to the provisions of Good Clinical Practice, as specified for IVD devices in ISO 20916:2019 or ISO 14155:2020. Patient consent is necessary to use donated samples, and privacy concerns should be taken into account.