Inter-Observer Variability in the Interpretation of ⁶⁸Ga-PSMA PET-CT Scan according to PROMISE Criteria

Aims & Objectives: Aim of the study was to evaluate inter-observer variability in interpretation of Gallium-68 labeled Prostate Specific Membrane Antigen sub-type 11 (68Ga-PSMA-11) Positron Emission Tomography-Computed Tomography PET_CT scan according to Prostate Cancer Molecular Imaging Standardized Evaluation (PROMISE) criteria. Method and Materials: Forty-four consecutive patients of prostate cancer were prospectively studied between the duration of January 2021 to June 2021 at Institute of Nuclear Medicine and Oncology (INMOL), Lahore. All PET-CT scans were assessed by a researcher and 3 nuclear physicians and divided into two groups, interpreted in two phases. In the first phase, each group independently evaluated the scans while in the second phase, a consensus meeting was held and all the cases with discordance were discussed. Cohen’s Kappa test was used to measure interobserver variability with the cut-off of K’s alpha < 0.61 that was chosen to indicate substantial disagreement. Results: The study showed 41 out of 44 scans with positive PSMA findings while 03 scans were negative for any PSMA avid disease. In the first phase of image analysis, the level of agreement was slight in T stage (Kappa = 0.068, p = 0.65), moderate in the miN stage (Kappa = 0.46, p = 0.02) and substantial in miM stage (Kappa = 0.77, p ≤ 0.001) was seen. For PSMA score, overall agreement was substantial agreement (Kappa = 0.64, p < 0.01). In the second phase, miT stage and miN stage showed total agreement (Kappa = 1, p < 0.001), and miM stage showed strong agreement (Kappa = 0.90, p < 0.001) while PSMA score in all three stages showed total agreement (Kappa = 0.9, p < 0.01). Conclusion: Remarkable inter-observer agreement was seen in PROMISE criteria.

• ⁶⁸Ga-PSMA
• PET-CT
• PROMISE Criteria
• PSMA Score
• Inter-Observer Variability

1. Introduction

In men, prostate cancer (PC) is the second frequent neoplasm [1] and a primary cause of cancer-associated mortality [2]. It is a commonly diagnosed malignancy and shows a broad spectrum of disease presentation ranging from a subclinical/indolent to very aggressive advanced disease [3].

The imaging modalities which are conventionally used in the management of prostate cancer include: Transrectal Ultrasound (TRUS) which has a sensitivity and specificity of 40% - 50% [4], Magnetic Resonance Imaging (MRI), a gold standard for the detection of localized prostate cancer and which has sensitivity and specificity approaching 75% and 90% respectively [5]. Computed Tomography (CT) is usually used for nodal staging and has documented sensitivity of 40% [6]. ^99mTc-MDP bone scan which is the most used technique in the assessment of bone metastases in advanced PC [6] and Positron Emission Tomography & Computed Tomography (PET-CT) is rapidly evolving imaging technique that detects the functional status of tumors.

¹⁸F-FDG (fluorodeoxyglucose) PET-CT is used in the management of various tumors but is ineffective for the diagnosis of PC owing to the low metabolic glucose activity of tumor and the urinary excretion of FDG [7]. ¹⁸F-Fluciclovine PET-CT, an FDA approved leucine analog, useful in suspected recurrence and advanced metastatic disease, however, has poor specificity for primary tumors and low sensitivity for nodal disease [8] while various radiolabeled PSMA-ligands identify the location and extent of the recurrent disease better than any other anatomical and functional imaging techniques.

⁶⁸Ga PSMA-11 PET-CT has high sensitivity and specificity of 80% and 97%, respectively [9]. In newly diagnosed high-risk PC, it is used for the detection of occult regional & metastatic disease [10], prior to surgical procedures or planning external beam radiation [1]. ⁶⁸Ga PSMA is also performed before and during PSMA-directed therapy, for staging [11] or in biochemical recurrence if serum PSA levels are rising after curative treatment [12] or for localization of tumor tissue in recurrent PC [1].

Due to rapidly evolving role of PSMA PET-CT in management of PC and ever-increasing use of technique globally, consensus criteria have been considered vital to standardize interpretation of PSMA PET-CT scans. Such criterion is expected to reduce variability of terminology in interpretation, minimize differences in interpretation of degree of tracer uptake and also reduced false positive or false negative interpretations. Different interpretation criteria for the standardized reporting and interpretation of PSMA PET-CT have been proposed recently, including Prostate Cancer Molecular Imaging Standardized Evaluation (PROMISE) criteria by Eiber et al. PROMISE criteria comprise of molecular imaging TNM (miTNM) framework for PSMA-ligand PET-CT prostate cancer staging and PSMA score for the expression of PSMA in a tumor lesion. miTNM was proposed for standardized reporting of the presence, location, and extent of local prostate cancer and its regional spread and extra-pelvic metastases. The miPSMA score enables the standardized reporting of PSMA expression in PSMA ligand PET-CT. Expression categories are defined with mean uptake in the blood pool, liver, and parotid glands. Results are reported as 0, 1, 2, or 3 for no, low, intermediate, or high PSMA expression, respectively. Expression level is determined visually, and uptake measurements can be quantitatively analyzed to correctly assign a specific miPSMA score [13].

Before worldwide standardization, validation of PROMISE criteria for appropriate image interpretation and reproducibility between the readers, it is necessary to evaluate intra- and inter-observer variability, as it was based on their joint clinical experience. Many studies have assessed the inter- and intra-observer variability of PROMISE criteria along with other criteria used in the interpretation of PSMA scan, to evaluate their reproducibility among different centers and observers and to device a standard criterion for the interpretation of PSMA PET-CT including Demirci et al., Derwael et al., Tohaira et al. and Gultekin et al.

INMOL being the pioneer center of the country that started PSMA-11 PET-CT scans in November 2017, receives maximum referral for the scan from across the country. The center regularly performs the scans in PC patients and has recently adopted the PROMISE criteria for routine interpretation of scans.

Objective of the study was to evaluate interobserver variability in reading the ⁶⁸Ga PSMA-11 scans while applying the PROMISE criteria.

2. Materials and Methods

2.1. Patients

Forty-four patients’ scans were prospectively analyzed, performed for various indications at PET-CT Department INMOL during the period between January 2021 and June 2021. All those patients were excluded who had primary pathology of reference organs such as aorta, liver, or parotid glands. Patients who received LHRH agonists before staging of prostate cancer or those who underwent Chemotherapy within six weeks prior to the scan, or patients who received ¹⁷⁷Lu Radioligand therapy were also excluded from the study.

2.2. Instrumentation/Protocol

All scans were conducted on Discovery STE PET-CT system (GE, healthcare, USA) with 16 slice CT scanner. CT images were acquired before PET acquisition (3.8 mm slice thickness, 100 kV, 50 mA) and the time of acquisition was 5 seconds per bed position for seven to eight-bed positions while PET emission data were acquired at 3 - 4 minutes/bed position from the base of the skull to mid-thigh in three-dimensional mode with Bismuth Germanium scanner (BGO). PET images were reconstructed using CT data by iterative reconstruction.

PSMA 11 was obtained from ITM, Germany and the dose of ⁶⁸Ga-PSMA-11 was calculated as per body weight, on average 02 MBq per kg body weight was given. PET-CT scan was carried out after 45 - 60 minutes post PSMA administration.

2.3. Image Interpretation

All PET-CT scans were interpreted by a researcher and 3 nuclear physicians divided into two groups, in two phases according to templates provided in PROMISE criteria. All observers were blinded to patient’s clinical data. During first phase, each group separately evaluated the scans and in the second phase a consensus meeting was held where all the cases with discordance were discussed.

The images were displayed simultaneously as PET, CT, fused PET-CT in axial, sagittal, and coronal sections, and 3D MIP (maximum intensity projection). Image analysis was done by using two reporting systems i.e., ADW 4.4 workstation and MAC Osirix MD software.

Any focal area of increased ⁶⁸Ga PSMA uptake higher than the physiologic pattern of PSMA uptake or PSMA score ≥ 2, 3 were considered positive for the disease. The location of ⁶⁸Ga PSMA uptake within the prostate gland and its extracapsular extension, regional and distant nodes, skeletal and visceral involvement were described according to the miTNM template by Eiber et al., except for sextant anatomy template of prostate gland. Due to low dose CT, arbitrary landmarks were used, and localized lesion was classified as unifocal, multifocal or extra-prostatic extension.

For miPSMA score, visually as well as quantitatively SUVmax (maximum standardized uptake value) was measured in parotids, mediastinal blood pool, and liver. A two-dimensional Region of Interest (ROI) was drawn with the width of 1.5 cm on the right parotid, 2 cm on the aortic arch for mediastinal blood pool uptake, and a 3 cm ROI on the 6th segment of the liver. The abnormal uptake was measured with a ROI width of 1 cm for prostatic fossa, nodal and osseous lesions while the lesion sizes were measured on the corresponding CT images in two dimensions as well.

2.4. Interobserver Variability

The two groups independently recorded their findings and concluded miTNM staging as well as PSMA score and interobserver agreement was seen in each stage, i.e., miT, miN and miM stage along with PSMA score in the two phases.

2.5. Statistical Analysis

SPSS 25 was used for statistical analysis and Cohen’s Kappa test was used to measure interobserver variability. Values for alpha statistics range from −1 to 1, and a rough guideline for interpreting the degree of agreement adapted from kappa statistics is as follows: <0.00 = total disagreement, 0.00 - 0.20 = slight agreement, 0.21 - 0.40 = fair agreement, 0.41 - 0.60 = moderate agreement, 0.61 - 0.80 = substantial agreement, 0.81 - 1.00 = almost perfect agreement. A cut-off of K’s alpha < 0.61 was chosen to indicate substantial disagreement and a p value of ≤0.05 was considered as statistically significant [14].

3. Results

3.1. Patients Characteristics

In our study, the most common indication for PET-CT scan was the staging of prostate cancer in intermediate and high-risk patients and the mean age of patients was 69.18 ± 7.35 years with a serum PSA levels’ median value of 12.3 ± 103 ng/ml. The Gleason score is calculated using histological features and most common observed Gleason score was 7 in our study. Characteristics of patients are given in Table 1.

In parotid gland, the physiologic PSMA uptake ranged from SUVmax of 4.3 to 24 with a mean value of 14.1 ± 4.7. In the mediastinal blood pool, the PSMA

Table 1

Patient characteristics (n = 44)

Patient Characteristics		N (%)	Scans		N (%)
Clinical Indications	Staging	17 (39%)	Positive Scans		41 (93%)
	Biochemical Recurrence	15 (34%)	Negative Scans		3 (7%)
	Restaging after therapy	12 (27%)	Lesions*
Age (years) Range: 48 - 84 years, Mean: 69.18 ± 7.35 years	≤50	1 (2%)	Prostatic Fossa Involvement (T stage) 34 (83%)	T2u	13 (38%)
	51 - 60	3 (7%)
	61 - 70	26 (59%)		T2m	03 (9%)
	71 - 80	11 (25%)		T3b	09 (26%)
	≥81	3 (7%)		T4	07 (21%)
Serum PSA Level (ng/ml) Range: <0.025 - 544 ng/ml, Median 12.3 ± 103 ng/ml	≤10	23 (52%)	Regional Nodal Involvement (N stage) 17 (41%)	N1	05 (29%)
	11 - 20	7 (16%)
	21 - 100	9 (20%)		N2	11 (65%)
	≥101	5 (11%)	Distant Metastasis (M stage) 23 (56%)	M1a	11 (48%)
Gleason Score	<7	1 (2%)
	7	22 (50%)		M1b	17 (74%)
	8	15 (34%)		M1c	05 (22%)
	≥9	6 (14%)

*Total number of cases in which local, nodal, or distant metastatic lesions are seen.

uptake ranged from SUVmax of 0.25 to 2.1 with a mean value of 1.08 ± 0.4 and in the liver, it ranged from SUVmax of 0.48 to 11.48 with a mean of 5.2 ± 2.2. Following Figure 1 shows the graphical representation of the PSMA expression and its frequency in our study.

The frequency of PSMA Score observed was 7%, 18%, 18%, and 57% for Score 0, 1, 2, and 3 respectively. The most common observed PSMA score was 3, as shown in Figure 1.

Out of 44 scans, 13 (30%) cases had alone prostatic fossa involvement with/without extra prostatic extension. Involvement of regional lymph nodes was seen in 17 (38%) while distant metastatic nodes were seen in 11 (25%) of cases. Unifocal, oligometastatic, disseminated bone involvement osseous metastases and diffuse involvement of bone/marrow was seen was seen in 03 (18%), 03 (18%) cases, 10 (59%) and in 01 (6%) cases, respectively. Liver involvement was seen in 4 out of 5 cases and a single case of non-PSMA avid lung of total 22 % cases of visceral metastases. Summary of total lesion are given in Table 1.

Following Figure 2 shows some of the examples of PSMA scans on miTNM template from our study.

3.2. Interobserver Variability

The summary of interobserver agreement in both phases is shown in Figure 3 in miT, miN, miM and PSMA score.

3.3. First Phase of Image Analysis

The agreement in T stage was on 55% cases (24/44) between the two groups on the involvement of prostatic fossa while disagreement was on 45% (20/44) cases. Out of 24 cases in which agreement was seen, 16 were diseased and 08 cases were not diseased. Slight agreement was found in miT stage with Kappa of 0.068, and a p-value of 0.65.

In N stage, the agreement was on 77% (34/44) cases and disagreement was on 23% (10/44) cases. Out of 34 cases of agreement, 26 cases were not diseased and 8 were diseased. Moderate agreement was found in miN stage with Kappa = 0.46, and significant p-value (p value 0.02).

In M stage, the agreement was on 89% (39/44) cases and disagreement was on 11% (5/44) cases. Out of 39 cases in which agreement was achieved, 20 were diseased and 19 were not diseased. Substantial agreement was observed in miM stage (Kappa = 0.77, p ≤ 0.001) with a significant p-value.

For PSMA score overall agreement was seen in 77% (34/44) cases, disagreement was seen in 23% (10/44) cases. Substantial agreement was observed in PSMA score with Kappa value of 0.64 and significant p-value (p-value of <0.01).

3.4. Second Phase of Image Analysis

A consensus meeting was held and all the cases with disagreement were discussed in this phase. miT stage and miN stage showed total agreement (Kappa = 1) with a highly significant p-value (p < 0.001) while in the miM stage, a strong agreement was seen (Kappa = 0.90) with a significant p-value (p < 0.001) but in 2 (5%) cases; no agreement could be achieved.

Cases with no agreement consist of one case with staging of miT0N0M1c; non-PSMA avid lung nodule (HU of 16, measuring 1.8 cm × 1.7 cm) without any other avid disease, while other case had miT0N0M1a staging with a single PSMA avid hilar node of score 3 (SUVmax 33, measuring SD: 4.7 mm).

After the consensus meeting, overall total agreement (Kappa = 0.9, p < 0.01) was seen in the PSMA score in all three stages.

4. Discussion

The ⁶⁸Ga PSMA PET-CT scan has a significant role in the management of the prostate cancer, given to its highest sensitivity, specificity, and accuracy than other imaging modalities and conventional radiotracers. Hence the standardization of an interpretation criterion for the PSMA scan was and is very important need of time. Out of the all-proposed interpretation criteria for PSMA PET-CT scans, PROMISE criteria seem more reproducible due to the incorporation of TNM staging with PSMA expression in the form of PSMA score. But PROMISE criteria still requires validation and standardization for image interpretation with less pitfalls and better reproducibility among readers with different clinical experiences irrespective of different imaging techniques, workstations, and characteristics of studied sample. This study was done to evaluate the reproducibility of PROMISE criteria in our clinical setup among readers of different clinical experiences with different workstations.

This study demonstrated the better level of agreement in nodal and metastatic diseases as compared to localized prostatic uptake in the first phase of image analysis, while in the second phase of image analysis, total agreement was seen in miT & miN stages and strong agreement in miM stage, proving reproducibility of PROMISE criteria.

Demirci et al. showed substantial agreement in the miT, miN, and miM reporting of the readers, with the highest agreement level in miM and lowest agreement in miT [15]. In a study by Tohaira et al., 104 patients with prostate cancer were retrospectively reviewed on PROMISE along with other two interpretation criteria i.e., EANM, and PSMA RAD. In the PET/CT group, inter-reader agreements were substantial to almost perfect in any sites according to all the three criteria, except in the evaluation of distant metastases based on PSMA-RADS [16]. In another study, Derwael et al. reported the miT and miN classification with a substantial agreement, while the miM classification with an almost perfect agreement. The overall miTNM classification was concordant for 60% patients, and observers’ agreement was substantial (K = 0.64; 95% CI 0.48 - 0.76) [17]. In a study by Gultekin et al., the intraobserver agreement was substantial and almost perfect for miT, miN, and miM, respectively [18].

The studies mentioned above are in line with our findings. The lowest interobserver agreement seen in the miT stage of first phase of image analysis was because of physiological PSMA expression in the prostatic tissue as well as activity in the prostatic urethra and poor sensitivity of low dose CT for the pelvic structures. At the consensus meeting, all these possible factors were discussed and hence agreement was achieved.

The 2 cases of discordance in the miM stage were seen in a non-PSMA avid lung nodule with the borderline malignant features and a hilar lymph node of PSMA score 3 without any other PSMA avid lesion specific for prostatic cancer as shown in Figure 4.

PSMA uptake has been reported in various benign diseases and non-prostatic malignant tumors as well, including lung cancer. According to Keidar et al. study, the prevalence and degree of ⁶⁸Ga-PSMA avidity of non-malignant lymph nodes represents 38% out of all non-malignant findings while up to 10% of findings included focal or diffuse uptake in the thyroid, in lung lesions or ganglions [19]. In a study by Damjanovic et al., the imaging characteristics of pulmonary metastases and opacities on ⁶⁸Ga-PSMA-PET scan were investigated. Most of the lung metastases showed overexpression of PSMA, PSMA-negative metastases showed decreased uptake and were undetectable directly by ⁶⁸Ga-PSMA-PET, while pulmonary opacities showed a moderate tracer uptake. The PSMA-negative metastases observed cases in this study were 27.5%. The reason of heterogeneity of ⁶⁸Ga-PSMA uptake in lung metastases is because of diverse histological variants of prostate cancers metastases, e.g., neuroendocrine differentiation of the metastases [20].

The non-PSMA avid lung nodule observed in our study might be the non-adenocarcinoma variant of the PC that’s why no remarkable PSMA uptake was seen.

Dias et al. reported 2 cases of ⁶⁸Ga-PSMA PET-CT in PC patients with symmetrical bilateral involvement of mediastinal and hilar lymph and on endobronchial ultrasound guided biopsy, it was found to be involved by non-necrotic granulomas compatible with sarcoidosis. This demonstrated that false positive PSMA uptake can be seen in lymph nodes with active sarcoidosis [21]. While in a study by Afshar-Oromieh et al., mediastinal and paraaortic lymph nodes were evaluated both histologically and their imaging characteristics were compared to lymph nodes metastases of PC. It was observed that lymph nodes showing strongest PSMA expression had follicular hyperplasia on histopathology and mediastinal lymph nodes which most commonly are activated shows usually lower uptake at 1-hour post injection imaging [22].

The disagreed case of single remarkably PSMA avid hilar node in our study is still debatable as it could be of inflammatory origin leading to false positive PSMA uptake or could be metastatic lymph node given the PSMA score 3 at 1-hour imaging without preserved fatty hilum but the short axis diameter of 4 mm were other factors causing the disagreement. As recommended by Afshar-Oromieh et al. in such uncertain grading cases of PSMA-positive lymph nodes additional imaging should be done at 3-hour post injection and in cases of tumor-free PSMA avid mediastinal and paraaortic lymph nodes should be verified histologically [22].

For PSMA score interobserver variability, overall substantial agreement was observed with a significant p-value in our study, and after consensus meeting total agreement was seen. While in Derwael et al. study, miPSMA score agreement was substantial in T & M stage while moderate in the N stage [17].

Calculation of PSMA score is highly dependent on the visual assessment of appropriate intensity setting as well as on the system used. In our study we used 2 different systems for the interpretation of scans, and it may require some time before the observer becomes used to different display settings of the two systems. This may play a role in inter observer variability. Hence the discordance in PSMA score is seen and inter-reader agreement was overall calculated in our study.

Limitation of the study includes small sample size due to relatively new imaging modality with availability in limited centers, the conventional approach of managing prostate cancer; monitoring serum PSA levels in BCR or staging with conventional imaging techniques (Bone scan, CT, and MRI), the import of ⁶⁸Ge/⁶⁸Ga generator and the limited radioisotope supply was other limitation.

5. Conclusion

There is minimum interobserver variability in ⁶⁸Ga PSMA-11 PET-CT interpretation according to PROMISE criteria in our experience. Current prospective study of 44 patients with PC showed remarkable inter-observer agreement in all three stages of miTNM and PSMA score.

Acknowledgements

Author wants to thank the faculty of Department of Medical Sciences, PIEAS and the Department of Nuclear Medicine, Atomic Energy Commission Hospital, SINOR especially Dr. Shoab Shah for their help and encouragement.

Conflicts of Interest

There were no conflicts of interest.

References

1. Fendler, W.P., Eiber, M., Beheshti, M., et al. (2017) 68Ga-PSMA PET/CT: Joint EANM and SNMMI Procedure Guideline for Prostate Cancer Imaging: Version 1.0. European Journal of Nuclear Medicine and Molecular Imaging, 44, 1014-1024. https://doi.org/10.1007/s00259-017-3670-z

2. Kulkarni, M., Hughes, S., Mallia, A., et al. (2020) The Management Impact of 68Galliumtris(Hydroxypyridinone) Prostate-Specific Membrane Antigen (68Ga-THP-PSMA) PET-CT Imaging for High-Risk and Biochemically Recurrent Prostate Cancer. European Journal of Nuclear Medicine and Molecular Imaging, 47, 674-686. https://doi.org/10.1007/s00259-019-04643-7

3. Williams, S. (2020) Molecular Imaging of Newly Diagnosed Prostate Cancer. The Cancer Journal, 26, 43-47. https://doi.org/10.1097/PPO.0000000000000427

4. Chen, F.K., de Castro Abreu, A.L. and Palmer, S.L. (2016) Utility of Ultrasound in the Diagnosis, Treatment, and Follow-up of Prostate Cancer: State of the Art. Journal of Nuclear Medicine, 57, 13S-18S. https://doi.org/10.2967/jnumed.116.177196

5. Harvey, H. and deSouza, N.M. (2016) The Role of Imaging in the Diagnosis of Primary Prostate Cancer. Journal of Clinical Urology, 9, 11-17. https://doi.org/10.1177/2051415816656120

6. European Association of Urology (EAU) (2021) EAU-EANM-ESTRO-ESUR-SIOG Guidelines on Prostate Cancer.

7. Turkbey, B., Albert, P.S., Kurdziel, K., Albert, K.K. and Choyke, P.L. (2009) Imaging Localized Prostate Cancer: Current Approaches and New Developments. American Journal of Roentgenology, 192, 1471-1480. https://doi.org/10.2214/AJR.09.2527

8. O’Malley, J.P., Ziessman, H.A. and Thrall, J.H. (2020) Nuclear Medicine and Molecular Imaging: The Requisites. 5th Edition, Elsevier, Amsterdam.

9. Perera, M., Papa, N., Christidis, D., et al. (2016) Sensitivity, Specificity, and Predictors of Positive 68Ga-Prostate-specific Membrane Antigen Positron Emission Tomography in Advanced Prostate Cancer: A Systematic Review and Meta-Analysis. European Urology, 70, 926-937. https://doi.org/10.1016/j.eururo.2016.06.021

10. Jadvar, H. (2016) Is There Use for FDG-PET in Prostate Cancer? Seminars in Nuclear Medicine, 46, 502-506. https://doi.org/10.1053/j.semnuclmed.2016.07.004

11. Demirkol, M.O., Acar, &#214;., U&#231;ar, B., Ramazanotlu, S.R., Satlican, Y. and Esen, T. (2015) Prostate Specific Membrane Antigen-Based Imaging in Prostate Cancer: Impact on the Clinical Decision-Making Process. The Prostate, 75, 748-757. https://doi.org/10.1002/pros.22956

12. Davidson, T., Amit, U., Saad, A., Hahiashvili, M., Goshen, E., Portnoy, O., et al. (2019) Gallium-68 Prostate-Specific Membrane Antigen PET-CT and the Clinical Management of Prostate Cancer. Nuclear Medicine Communications, 40, 913-919. https://doi.org/10.1097/MNM.0000000000001047

13. Eiber, M., Herrmann, K., Calais, J., et al. (2018) Prostate Cancer Molecular Imaging Standardized Evaluation (PROMISE): Proposed miTNM Classification for the Interpretation of PSMA-Ligand PET/CT. Journal of Nuclear Medicine, 59, 469-478. https://doi.org/10.2967/jnumed.117.198119

14. McHugh, M.L. (2012) Interrater Reliability: The Kappa Statistic. Biochemia Medica, 22, 276-282. https://doi.org/10.11613/BM.2012.031

15. Demirci, E., Akyel, R., Caner, B., et al. (2020) Interobserver and Intraobserver Agreement on Prostate-Specific Membrane Antigen PET/CT Images according to the miTNM and PSMA-RADS Criteria. Nuclear Medicine Communications, 41, 759-767. https://doi.org/10.1097/MNM.0000000000001219

16. Toriihara, A., Nobashi, T., Baratto, L., et al. (2020) Comparison of 3 Interpretation Criteria for 68Ga-PSMA11 PET Based on Inter and Intrareader Agreement. Journal of Nuclear Medicine, 61, 533-539. https://doi.org/10.2967/jnumed.119.232504

17. Derwael, C., Lavergne, O., Lovinfosse, P., et al. (2020) Interobserver Agreement of [68Ga] GaPSMA-11 PET/CT Images Interpretation in Men with Newly Diagnosed Prostate Cancer. EJNMMI Research, 10, Article No. 15. https://doi.org/10.1186/s13550-020-0596-4

18. Gültekin, A., Yaylali, O., &#350;eng&#246;z, T., Yüksel, D. and &#350;ahin, B. (2019) Intraobserver and Interobserver Agreement for the Interpretation of 68Ga-Prostate-Specific Membrane Antigen-I&T Positron Emission Tomography/Computed Tomography Imaging. Nuclear Medicine Communications, 40, 1250-1255. https://doi.org/10.1097/MNM.0000000000001097

19. Keidar, Z., Gill, R., Goshen, E., et al. (2018) 68Ga-PSMA PET/CT in Prostate Cancer Patients—Patterns of Disease, Benign Findings, and Pitfalls. Cancer Imaging, 18, Article No. 39. https://doi.org/10.1186/s40644-018-0175-3

20. Damjanovic, J., Janssen, J., Prasad, V., et al. (2019) 68Ga-PSMA-PET/CT for the Evaluation of Liver Metastases in Patients with Prostate Cancer. Cancer Imaging, 19, Article No. 37. https://doi.org/10.1186/s40644-019-0220-x

21. Dias, A., Holn Vendelbo, M. and Bouchelouche, K. (2017) Prostate-Specific Membrane Antigen PET/CT: Uptake in Lymph Nodes with Active Sarcoidosis. Clinical Nuclear Medicine, 42, e175-e176. https://doi.org/10.1097/RLU.0000000000001528

22. Afshar-Oromieh, A., Sattler, L., Steiger, K., et al. (2018) Tracer Uptake in Mediastinal and Paraaortal Thoracic Lymph Nodes as a Potential Pitfall in Image Interpretation of PSMA ligand PET/CT. European Journal of Nuclear Medicine and Molecular Imaging, 45, 1179-1187. https://doi.org/10.1007/s00259-018-3965-8