Every pathology AI model is trained and evaluated against a reference standard, the set of labels taken as correct for each case. Discussion of these labels often centres on how many pathologists should assign them. The number of readers matters less than whether the reference standard suits the model's intended use, the degree to which experts are known to disagree on the task, and the consequences of a wrong label.

Disagreement between pathologists depends on the task

Disagreement between pathologists reading the same case, known as inter-observer variability, is well documented, although its size varies considerably with the task. In a study in which 115 pathologists each interpreted 60 breast biopsy cases, overall agreement with an expert consensus diagnosis was 75.3%. Agreement reached 96% for invasive carcinoma yet fell to 48% for atypia (Elmore et al., 2015).

Biomarker scoring shows a similar pattern. When 13 pathologists scored PD-L1 expression in 90 non-small cell lung carcinomas, agreement on tumour cell staining was high, with intraclass correlation coefficients of 0.83 to 0.88 across four assays, while agreement on immune cell staining was poor, at 0.17 to 0.23 (Rimm et al., 2017). For Ki-67 in breast cancer, eight laboratories scoring the same 100 cases reached only moderate agreement, with an intraclass correlation coefficient of 0.71 on centrally stained material, despite high reproducibility within each laboratory (Polley et al., 2013).

These studies show that pathologist reliability cannot be summarised by a single figure. A reference standard for the detection of invasive carcinoma and one for immune cell PD-L1 scoring carry very different levels of uncertainty, and the review design should reflect that difference.

When a single expert read is sufficient

A single read by a pathologist with expertise in the relevant subspecialty is a reasonable reference standard where agreement between experts is known to be high and the label is not the primary measure of the model's performance. Tumour region annotation for model training, quality control of whole-slide images and case selection against predefined eligibility criteria usually fall into this category. In biomarker annotation work for a global pharmaceutical company, for example, Pathology Nucleus annotated 1,314 whole-slide images under senior pathologist oversight and quality assurance, without independent duplicate reads of every slide. For these applications, routine duplicate review of every case may offer limited additional benefit relative to its cost, particularly where criteria are well defined and senior pathologist quality assurance is already in place.

A single read still requires defined criteria, an appropriately specialised reader and a record of each decision. With that documentation in place, an independent reviewer can follow how each label was assigned, and a single-reader standard can be justified for its purpose.

When independent reads and adjudication add value

Multiple independent reads are most useful where agreement is known to be poor, where borderline categories carry clinical consequences, and where the labels will form the test set against which a model is judged. Atypia in breast biopsies, dysplasia grading and immune cell scoring are examples. In these settings, a model that disagrees with a single pathologist's label cannot be distinguished from a model that is wrong, because model error and label error are confounded.

Independent reads followed by a predefined adjudication rule help to characterise and reduce this uncertainty, although they cannot remove it. Two or more pathologists read each case without knowledge of each other's assessment, their agreement is measured, and discordant cases are resolved by a designated adjudicator according to rules set before the review begins. The measured agreement between readers also provides a benchmark against which the model's own agreement with the reference standard can be interpreted.

Agreement between pathologists does not establish truth

Agreement between pathologists improves the reliability of a label without guaranteeing that the label is correct. Readers trained in the same tradition can share a systematic bias, and adjudication resolves disagreement without correcting an error that every reader makes. Where the intended use allows it, the reference standard should therefore draw on evidence beyond morphology, such as molecular confirmation of a mutation-defined entity, a validated assay result or clinical outcome on follow-up. A model intended to predict a molecular alteration from histology, for instance, should be evaluated against the molecular result, with pathology review confined to confirming that the tissue is adequate and representative.

Implications for study design

The appropriate number of readers follows from three questions that should be answered before labelling begins: what the model is intended to do, how much experts are known to disagree on the task, and what a wrong label would cost in that use. The answers determine whether a single expert read, independent reads with adjudication, or a reference standard incorporating molecular or clinical data is appropriate. Whichever design is chosen, the criteria, the readers' expertise and the handling of disagreement should be recorded in enough detail for an independent reviewer to follow how each label was assigned.

References

  1. Elmore JG, Longton GM, Carney PA, et al. Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA, 2015, 313(11), 1122-1132. doi:10.1001/jama.2015.1405
  2. Rimm DL, Han G, Taube JM, et al. A prospective, multi-institutional, pathologist-based assessment of 4 immunohistochemistry assays for PD-L1 expression in non-small cell lung cancer. JAMA Oncology, 2017, 3(8), 1051-1058. doi:10.1001/jamaoncol.2017.0013
  3. Polley MY, Leung SC, McShane LM, et al. An international Ki67 reproducibility study. Journal of the National Cancer Institute, 2013, 105(24), 1897-1906.