The validation of medical imaging (processing and acquisition) can be achieved in multiple ways, somewhat influenced by the context. There are three traps to avoid: First reliance on ground truth requires the knowledge of it before the end of the trial, second comparison to gold standards cannot show improvement and finally one needs to deal with confirmation bias. In this paper we discuss those topics and alternative validation schemes.
When new imaging technologies begin to be used in clinical settings, little is known about their potential to improve care. They are usually “sold” on technological or impressionistic criteria. Clinical validation is rare, because the consensus is that it would delay the application. In this paper we discuss the different methods for validation, not all of them which would cause implementation delay.
Ideally, for the sake of improving the appropriateness of medical imaging, one would hope for more rapid progression to what can be called scientific clinical research or technology assessment.
The first level of evaluation can be called diagnostic efficacy1. The appropriate research question to ask in this phase, while the technology is just beginning to diffuse into clinical practice or when a substantive advance in the technology occurs, is “How well does the new technology detect specific disease conditions?” The measures of efficacy are the operating characteristics (sensitivity, specificity). Effectiveness includes the positive and negative predictive value for a given prevalence of the disease in the study population and receiver-operating characteristic (ROC) analysis [
The defining diagnosis is really a type of taxonomy3 and defines the ground truth4. The assumption is that the result of the taxonomy is related to outcome or the result of a specific therapy. While diagnostic efficacy is principally of interest to radiologists, referring clinicians may be more interested in how the information derived from an imaging test affects how they care for patients, represented by the concepts of therapeutic thinking efficacy. For therapeutic thinking, the corresponding question relates to the effect of imaging on considerations of treatment.
The validation of defining diagnostic technique would be solipsistic. We will concentrate on non-defining diagnostic techniques and as much as we can on techniques based on computer processing of medical images. But why would we want a new diagnostic procedure? The global answer is that the existing one or the combination of existing ones is too costly or lacks in efficacy, effectiveness or efficiency. Costly should be understood as combining expenses in material and personnel, pain, danger, the lack of accuracy (strictly speaking unfavorable operating characteristics for the examined population: e.g. false positive rates), lack of predictive value
Not all diagnostic techniques aim to properly classify the patients, but rather to predict either the outcome, or the best therapy to obtain the desired outcome (measurements of plasma cholesterol do not provide a diagnosis but a prognosis, staging is not diagnostic, but predictive). Ultimately, the diagnostic technique should be evaluated in its role in the management of the patient, or the outcome5.
Outcome analysis is actually based on large population studies. Originally it was presented as the method to evaluate cost-effectiveness (e.g. did people live longer if there were more MRI scanners in the region). More pointedly, it has been used to look at the efficacy of screening studies. The analysis of those data is complicated by the concatenation of increased detection (an increase in incidence) and the fact that early detection may not always predict progression. For breast cancer and mammography it did take a long time to show the beneficial outcome [
Imaging usually represents only one or a few steps in a chain of diagnostic and therapeutic interventions, so how can we ascribe an outcome to any one of these? The performance of an imaging test may be excellent, but patients might still have adverse outcomes because the treatment was inappropriate or no adequate treatment exists. As a result of all of these factors, outcomes evaluations of imaging technologies are rare. In the case of screening for early detection, the outcome is affected only to the extent that the treatment is (relatively) effective in early stages and not in more advanced disease. Outcome studies measure effectiveness rather than efficacy. Outcome analysis as validation would take too long and prevent the introduction of new techniques.
A taxonomic exact diagnosis may not be predictive. Consider that in some diseases the median survival time is n years: fifty percent die earlier, 50% later. The prognosis is not necessarily well defined by the diagnosis. Staging refines the prognosis, or the expected response to a particular therapy. Other techniques can be used to predict earlier which therapy will fail or succeed so that alternatives can be used [
This is the most common type of validation for diagnostic techniques. The most relevant aspects of this approach are 1) that a ground truth is assumed to exist and be known and 2) that at some point there has to be a defining test or the next best thing (e.g. a gold standard)6.
The major problem is verification bias in the first case always and in the second case mostly. An example of Verification Bias is the evaluation of Myocardial Perfusion Scintigraphy (MPS). The gold standard for the presence of (significant) coronary artery diseases (CAD) was originally the coronary arteriogram (CA).The MPS study would select patients more likely to need the CA. However, trust came too early. A value was ascribed to MPS (prematurely) and soon the probability of a CA being performed following a negative MPS decreased while the probability of a CA being performed following a positive MPS increased. The result was verification bias: with an over-estimation of the sensitivity and an under-estimation of the specificity. The proper validation would have been to perform MPS only on patients who had a positive or negative CA and do it blindly.
There are ways to overcome verification bias: one of them is to look at populations with a known prevalence [
If existing populations with known prevalence exist, this is a fairly direct approach, but expensive to implement.
There are two concatenated conditions for a test to be discriminating: the metric has to have intrinsic discriminating value and the measurement has to be precise enough, so that variability in testing does not reach the magnitude of the difference between affected and unaffected.
25 patients with “early” CF and 10 control cases. Patients are defined by genetics or sweat test. Controls are non-affected siblings in the same age range and same sex distribution. Did the pulmonary function tests discriminate between both groups [
| RV/TLC | IC% | FVC% | SVC% | FEV1% | FEV1/FVC | FEF25-75% | FEFMax% | |
|---|---|---|---|---|---|---|---|---|
| Mean CF | 28.2 | 102.3 | 115.4 | 110.7 | 104.0 | 80.4 | 83.6 | 99.6 |
| STDV CF | 10.2 | 18.9 | 18.5 | 19.8 | 17.1 | 7.6 | 29.5 | 25.6 |
| Mean Nl | 21.6 | 97.0 | 111.6 | 112.3 | 106.6 | 84.1 | 103.9 | 100.5 |
| STDV NL | 4.0 | 6.3 | 15.1 | 13.3 | 13.7 | 6.4 | 26.5 | 20.9 |
| T-test | 0.011 | 0.226 | 0.568 | 0.814 | 0.668 | 0.189 | 0.068 | 0.926 |
| A1 | A2 | A3 | ||
|---|---|---|---|---|
| 25 CF | Mean | 16.16 | 9.83 | 4.50 |
| SD | 14.71 | 10.30 | 5.11 | |
| 10 NL | Mean | 5.22 | 2.27 | 0.82 |
| SD | 3.64 | 1.72 | 0.62 | |
| T-test | 0.0013 | 0.0012 | 0.0013 |
To evaluate quantitative air trapping measurements in children with mild cystic fibrosis (CF) lung disease during a one year double-blind placebo-controlled rhDNase intervention trial and compare results from quantitative air trapping with those from spirometry or visually scored HRCT scans of the chest [
In a certain sense discriminating power is the el dorado of image processing, because if the exactitude of the measurement can often be determined (e.g. the shape and size of an hearing aid determined from an image of the external auricular canal), this is not true in all cases: there is no life verification of early air trapping in children with Cystic Fibrosis and there is no life diagnosis of Alzheimer disease in elderly. In this case discriminating power overcomes the lack of ground truth and shows efficacy.
The prototype of internalization is automation of region of interest (ROI) definition. The creator or user of the routine assumes that the user knows if a ROI is correctly placed and limited. The question is whether the automated program yields a result acceptable to the observer, and how frequently.
Another is image processing that extracts an image feature, and hence not only facilitates interpretation, but makes it more reproducible. Again, the validation is an agreement with the observer.
The validation is internalized, not because of clinical criteria, but because it standardizes interpretation to the satisfaction of the user. It is a weak validation, but immediate and cheap.
Equivalence is based on the comparison with an established diagnostic procedure. The established procedure is sometimes referred to as “gold standard”, even if it is not a perfect procedure. More precisely this approach is referred to as a “no worse than” design. The “not worse than” denomination refers to the fact that at best the evaluated diagnostic procedure perfectly matches the “gold standard”, but cannot be shown to be better: all discrepancies are demonstrating a worse performance.
It takes different forms. In MPS the gold standard was the CA; however the metric was not the same: the arteriographic measurement of stenosis does not necessarily determine the relative decrease of flow in the dependent myocardium. In addition, a normal MPS predicts a lowering of risk for myocardial ischemic events independently of the CA findings [
In imaging a common study design is to compare the automatic analysis to the judgment of a panel of (experienced) experts (see also internal validation). Again, the performance cannot be shown to improve in the new modality since the human observers define the truth.
The equivalence design is not altogether valueless since the new procedure may globally decrease the cost (expenses in material and personnel, pain and danger). What can be demonstrated is an improvement in reproducibility of the interpretation if the method is automated or quantitative.
However, the use of gold standards, while easily performed, may be dangerous if the metric differ in a physiological important manner.
Cost effectiveness or efficiency evaluation would be the next step: Assuming that we reach a satisfactory clinical validation, we would also like to know how much a successful technology will cost. This allows us to see, regardless of the health benefit, whether society can afford to implement it on a broad scale. Since cost is rela-
| Metric | PulmozymeTM | Placebo | P |
|---|---|---|---|
| N | 11 | 14 | |
| A1 | −2.10 ± 44.80 | 34.40 ± 62.10 | 0.102 |
| A2 | −9.30 ± 42.90 | 43.40 ± 73.20 | 0.035 |
| A3 | −13.10 ± 40.50 | 48.20 ± 81.20 | −0.02 |
tive, researchers generally relate it to how much benefit is obtained for how much money. They have developed the ratio of cost per years of life saved and refer to this ratio as a technology’s cost-effectiveness [