Segmentation agreement and the reliability of radiomics features

Isabella Cama; Valentina Candiani; Luca Roccatagliata; Pietro Fiaschi; Giacomo Rebella; Martina Resaz; Michele Piana; Cristina Campi

doi:10.3934/acse.2023009

Article Contents

2023, Volume 1, Issue 2: 202-217. Doi: 10.3934/acse.2023009

This issue Previous Article A reduced basis stabilization for the unsteady Stokes and Navier-Stokes equations Next Article On the thermal welding of paper-based polylaminate packages: Modelling, numerical implementation and sensitivity analysis

Segmentation agreement and the reliability of radiomics features

1.
Università di Genova, Dipartimento di Matematica, via Dodecaneso 35, Genova, 16146, Italy
2.
Università di Genova, Dipartimento di Scienze della Salute, via Pastore 1, Genova, 16132, Italy
3.
Università di Genova, Dipartimento di Neuroscienze, Riabilitazione, Oftalmologia, Genetica e Scienze Materno-Infantili, Largo Paolo Daneo 3, Genova, 16132, Italy
4.
IRCCS Ospedale Policlinico San Martino, Largo Rosanna Benzi 10, Genova, 16132, Italy

^*Corresponding author: Cristina Campi
^*Corresponding author: Cristina Campi

Received: May 2023

Revised: June 2023

Early access: June 2023

Published: June 2023

Abstract / Introduction Full Text(HTML) Figure(7) / Table(1) Related Papers Cited by

Abstract

Radiomics features extracted from medical images have been shown to correlate with tumor histological biomarkers and patient clinical information. An accurate selection of reliable features is crucial for an efficient prediction/stratification analysis in clinical applications. This work introduces a computational method for evaluating the reliability of radiomics features with respect to image segmentation. To do so, we define four reliability scores that link segmentation variability to feature quality, consistency, robustness and instability. These scores can be used to establish a ranking that helps identifying the best features to employ for the application at hand, allowing an a-priori evaluation of radiomics reliability in clinical applications. We show the performance of the method with Magnetic Resonance images of meningioma patients. In this case, we identified three main groups of features displaying similar behaviors with respect to image segmentation variability.

Keywords:

Mathematics Subject Classification: Primary: 92C50, 92C55; Secondary: 62H20, 62P10.

Citation:

FullText(HTML)

Figure 1. Left panel: segmentation mask produced by the automated level set algorithm. Right panel: manual refinement

Download: Full-size image PowerPoint slide

Figure 2. Top panel: scatter plot of the relative error (in absolute value) on a specific feature ('Energy') with respect to the Dice coefficient. Bottom panel: two-dimensional reliability histogram $ H $ of the same feature. Each bin contains the rate of slices falling in the corresponding area of the scatter plot

Download: Full-size image PowerPoint slide

Figure 3. Reliability matrices. Top left panel: quality matrix. Top right panel: consistency matrix. Bottom left panel: robustness matrix. Bottom right panel: instability matrix. Each entry contains the binary weight we assign to the corresponding bin of the reliability histogram

Download: Full-size image PowerPoint slide

Figure 4. Distributions of the relative errors $ \varepsilon_{f_i} $ computed slice-wise on each feature $ f_i $, $ \varepsilon_{f_i} \in [-1,1] $. Outlier slices with relative error outside the range $ [-1,1] $ have been neglected for the sake of readability. Features are subdivided according to group membership. Yellow: global features. Red: non-texture. Green: GLCM. Blue: NGTDM. Magenta: GLRLM. Cyan: GLSZM

Download: Full-size image PowerPoint slide

Figure 5. Scatter plots for each feature, where each dot represents a segmented slice. For a fixed slice, the y-axis contains the relative error on the feature value and the x-axis the corresponding Dice coefficient. Outlier slices with relative error outside the range $ [0,1] $ have been neglected from the analysis. Vertical reference lines at $ \rm{Dice} = 0.8 $ and $ \rm{Dice} = 0.9 $. Horizontal reference lines at $ |\varepsilon_{f_i}| = 0.2 $ and $ |\varepsilon_{f_i}| = 0.3 $

Download: Full-size image PowerPoint slide

Figure 6. Stacked bar chart representing the values of the four reliability scores computed for each feature. The features are ordered according to increasing values of the instability score

Download: Full-size image PowerPoint slide

Figure 7. Result of the bootstrap experiment on ICC. Each boxplot represents the distribution of the ICC values computed over 100 random slice extractions (two-way mixed-effect model, single rater type (k = 2), absolute-agreement definition [15]). Yellow: global features. Red: non-texture. Green: GLCM. Blue: NGTDM. Magenta: GLRLM. Cyan: GLSZM

Download: Full-size image PowerPoint slide

Table 1. Radiomics features used in this study, taken from [37]

Feature type	Feature name
Global	Variance
	Skewness
	Kurtosis
Non texture	Area
Gray-level co-occurence matrix (GLCM) [11]	Energy
	Contrast
	Correlation
	Homogeneity
	Variance1
	Sum Average
	Entropy
	Dissimilarity
	AutoCorrelation
Neighbourhood gray-tone difference matrix (NGTDM) [1]	Coarseness
	Contrast1
	Busyness
	Complexity
	Strength
Gray-level run-length matrix (GLRLM) [9,5,6,34]	Short Run Emphasis (SRE)
	Long Run Emphasis (LRE)
	Gray-Level Non-uniformity (GLN)
	Run-Length Non-uniformity (RLN)
	Run Percentage (RP)
	Low Gray-Level Run Emphasis (LGRE)
	High Gray-Level Run Emphasis (HGRE)
	Short Run Low Gray-Level Emphasis (SRLGE)
	Short Run High Gray-Level Emphasis (SRHGE)
	Long Run Low Gray-Level Emphasis (LRLGE)
	Long Run High Gray-Level Emphasis (LRHGE)
	Gray-Level Variance (GLV)
	Run-Length Variance (RLV)
Gray-level size zone matrix (GLSZM) [9,34,5,6]	Small Zone Emphasis (SZE)
	Large Zone Emphasis (LZE)
	Gray-Level Non-uniformity (GLN)
	Zone-Size Non-uniformity (ZSN)
	Zone Percentage (ZP)
	Low Gray-Level Zone Emphasis (LGZE)
	High Gray-Level Zone Emphasis (HGZE)
	Small Zone Low Gray-Level Emphasis (SZLGE)
	Small Zone High Gray-Level Emphasis (SZHGE)
	Large Zone Low Gray-Level Emphasis (LZLGE)
	Large Zone High Gray-Level Emphasis (LZHGE)
	Gray-Level Variance (GLV)
	Zone-Size Variance (ZSV)

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	M. Amadasun and R. King, Textural features corresponding to textural properties, IEEE Trans. Syst., Man, Cybern., 19 (1989), 1264-1274. doi: 10.1109/21.44046.
[2]	B. Baeßler, K. Weiss and D. P. Dos Santos, Robustness and reproducibility of radiomics in magnetic resonance imaging: a phantom study, Investig. radiol., 54 (2019), 221-228. doi: 10.1097/RLI.0000000000000530.
[3]	L. Brunasso, G. Ferini, L. Bonosi, R. Costanzo, S. Musso, U. E. Benigno, R. M. Gerardi, G. R. Giammalva, F. Paolini, G. E. Umana, F. Graziano, G. Scalia, C. L. Sturiale, R. Di Bonaventura, D. G. Iacopino and R. Maugeri, A spotlight on the role of radiomics and machinelearning applications in the management of intracranial meningiomas: A new Perspective in Neuro-Oncology: A review, Life, 12 (2022), 586. doi: 10.3390/life12040586.
[4]	T. F. Chan and L. A. Vese, Active contours without edges, IEEE Trans. on Image Proc., 10 (2001), 266-277. doi: 10.1109/83.902291.
[5]	A. Chu, C. M. Sehgal and J. F. Greenleaf, Use of gray value distribution of run lengths for texture analysis, Pattern Recognit. Lett., 11 (1990), 415-419. doi: 10.1016/0167-8655(90)90112-F.
[6]	B. V. Dasarathy and E. B. Holder, Image characterizations based on joint gray level – run length distributions, Pattern Recognit. Lett., 12 (1991), 497-502. doi: 10.1016/0167-8655(91)80014-2.
[7]	L. R. Dice, Measures of the amount of ecologic association between species, Ecol., 26 (1945), 297-302. doi: 10.2307/1932409.
[8]	F. Fiz, C. Marini, R. Piva, M. Miglino, M. Massollo, F. Bongioanni, S. Morbelli, G. Bottoni, C. Campi, A. Bacigalupo, P. Bruzzi, F. Frassoni, M. Piana and G. Sambuceti, Adult advanced chronic lymphocytic leukemia: Computational analysis of whole-body CT documents a bone structure alteration, Radiol., 271 (2014), 805-813. doi: 10.1148/radiol.14131944.
[9]	M. M. Galloway, Texture analysis using gray level run lengths, Comp. Graph. and Image Process., 4 (1975), 172-179. doi: 10.1016/S0146-664X(75)80008-6.
[10]	R. W. Y. Granzier, N. M. H. Verbakel, A. Ibrahim, J. E. van Timmeren, T. J. A. van Nijnatten, R. T. H. Leijenaar, M. B. I. Lobbes, M. L. Smidt and H. C. Woodruff, MRI-based radiomics in breast cancer: feature robustness with respect to inter-observer segmentation variability, Sci. Rep., 10 (2020), 14163. doi: 10.1038/s41598-020-70940-z.
[11]	R. M. Haralick, K. Shanmugam and I. Dinstein, Textural features for image classification, IEEE Trans. on Syst., Man, and Cybern., SMC-3 (1973), 610-621. doi: 10.1109/TSMC.1973.4309314.
[12]	M. Hu and J. Tan, Adaptive osculatory rational interpolation for image processing, J. of Comp. and Appl. Math., 195 (2006), 46-53. doi: 10.1016/j.cam.2005.07.011.
[13]	F. Isensee, P. Kickingereder, W. Wick, M. Bendszus and K. H. Maier-Hein, Brain tumor segmentation and radiomics survival prediction: Contribution to the BRATS 2017 challenge, In Inter. MICCAI Brainlesion Workshop, Springer, 2017,287-297. doi: 10.1007/978-3-319-75238-9_25.
[14]	H. C. Kniep, F. Madesta, T. Schneider, U. Hanning, M. H. Schönfeld, G. Schön, J. Fiehler, T. Gauer, R. Werner and S. Gellissen, Radiomics of brain MRI: Utility in prediction of metastatic tumor type, Radiol., 290 (2019), 479-487. doi: 10.1148/radiol.2018180946.
[15]	T. K. Koo and M. Y. Li, A guideline of selecting and reporting intraclass correlation coefficients for reliability research, J. of Chiropr. Med., 15 (2016), 155-163. doi: 10.1016/j.jcm.2016.02.012.
[16]	P. Lambin, E. Rios-Velazquez, R. Leijenaar, S. Carvalho, R. G. P. M. Van Stiphout, P. Granton, C. M. L. Zegers, R. Gillies, R. Boellard, A. Dekker and H. J. W. L. Aerts, Radiomics: Extracting more information from medical images using advanced feature analysis, Eur. J. of Cancer, 48 (2012), 441-446. doi: 10.1016/j.ejca.2011.11.036.
[17]	J. Lao, Y. Chen, Z.-C. Li, Q. Li, J. Zhang, J. Liu and G. Zhai, A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme, Sci. Rep., 7 (2017), Article number: 1. doi: 10.1038/s41598-016-0028-x.
[18]	K. R. Laukamp, G. Shakirin, B. Baeßler, F. Thiele, D. Zopfs, N. Große Hokamp, M. Timmer, C. Kabbasch, M. Perkuhn and J. Borggrefe, Accuracy of radiomics-based feature analysis on multiparametric magnetic resonance images for noninvasive meningioma grading, World Neurosurgery, 132 (2019), e366-e390. doi: 10.1016/j.wneu.2019.08.148.
[19]	E. J. Limkin, R. Sun, L. Dercle, E. I. Zacharaki, C. Robert, S. Reuzé, A. Schernberg, N. Paragios, E. Deutsch and C. Ferté, Promises and challenges for the implementation of computational medical imaging (radiomics) in oncology, Annals of Oncol., 28 (2017), 1191-1206. doi: 10.1093/annonc/mdx034.
[20]	R. Liu, H. Elhalawani, A. S. Radwan Mohamed, B. Elgohari, L. Court, H. Zhu and C. D. Fuller, Stability analysis of CT radiomic features with respect to segmentation variation in oropharyngeal cancer, Clin. and Transl. Radiat. Oncol., 21 (2020), 11-18. doi: 10.1016/j.ctro.2019.11.005.
[21]	P. Lohmann, N. Galldiks, M. Kocher, A. Heinzel, C. P. Filss, C. Stegmayr, F. M. Mottaghy, G. R. Fink, N. J. Shah and K.-J. Langen, Radiomics in neuro-oncology: Basics, workflow, and applications, Methods, 188 (2021), 112-121. doi: 10.1016/j.ymeth.2020.06.003.
[22]	D. Mackin, X. Fave, L. Zhang, D. Fried, J. Yang, B. Taylor, E. Rodriguez-Rivera, C. Dodge, A. K. Jones and L. Court, Measuring computed tomography scanner variability of radiomics features, Investigative Radiology, 50 (2015), 757-765. doi: 10.1097/RLI.0000000000000180.
[23]	K. O. McGraw and S. P. Wong, Forming inferences about some intraclass correlation coefficients, Psychol. Methods, 1 (1996), 30-46. doi: 10.1037/1082-989X.1.1.30.
[24]	D. Mumford and J. Shah, Optimal approximations by piecewise smooth functions and associated variational problems, Commun. Pure Appl. Math., 42 (1989), 577-685. doi: 10.1002/cpa.3160420503.
[25]	J. E. Park, H. S. Kim, D. Kim, S. Y. Park, J. Y. Kim, S. J. Cho and J. H. Kim, A systematic review reporting quality of radiomics research in neuro-oncology: toward clinical utility and quality improvement using high-dimensional imaging features, BMC Cancer, 20 (2020), Article number: 29. doi: 10.1186/s12885-019-6504-5.
[26]	C. Parmar, E. Rios Velazquez, R. Leijenaar, M. Jermoumi, S. Carvalho, R. H. Mak, S. Mitra, B. U. Shankar, R. Kikinis, B. Haibe-Kains, P. Lambin and H. J. W. L. Aerts, Robust radiomics feature quantification using semiautomatic volumetric segmentation, PLOS ONE, 9 (2014), e102107. doi: 10.1371/journal.pone.0102107.
[27]	M. G. Poirot, M. W. A. Caan, H. G. Ruhe, A. Bjørnerud, I. Groote, L. Reneman and H. A. Marquering, Robustness of radiomics to variations in segmentation methods in multimodal brain MRI, Scientific Reports, 12 (2022), 16712. doi: 10.1038/s41598-022-20703-9.
[28]	S. Rizzo, F. Botta, S. Raimondi, D. Origgi, C. Fanciullo, A. G. Morganti and M. Bellomi, Radiomics: The facts and the challenges of image analysis, Eur. Radiol. Exp., 2 (2018), Article number: 36. doi: 10.1186/s41747-018-0068-z.
[29]	N. Saltybaeva, S. Tanadini-Lang, D. Vuong, S. Burgermeister, M. Mayinger, A. Bink, N. Andratschke, M. Guckenberger and M. Bogowicz, Robustness of radiomic features in magnetic resonance imaging for patients with glioblastoma: Multi-center study, Phys. and Imaging in Radiat. Oncol., 22 (2022), 131-136. doi: 10.1016/j.phro.2022.05.006.
[30]	G. Sambuceti, M. Brignone, C. Marini, M. Massollo, F. Fiz, S. M., A. Buschiazzo, C. Campi, R. Piva, A. M. Massone, M. Piana and F. Frassoni, Estimating the whole bone marrow asset in humans by a computational approach to integrated PET/CT imaging, Eur. J. Nucl. Med. Mol. Imag., 39 (2012), 1326-1338. doi: 10.1007/s00259-012-2141-9.
[31]	D. Schenone, A. Dominietto, C. Campi, F. Frassoni, M. Cea, S. Aquino, E. Angelucci, F. Rossi, L. Torri, B. Bignotti, A. S. Tagliafico and M. Piana, Radiomics and artificial intelligence for outcome prediction in multiple myeloma patients undergoing autologous transplantation: A feasibility study with ct data, Diagn., 11 (2021), 1759. doi: 10.3390/diagnostics11101759.
[32]	A. Stanzione, M. Gambardella, R. Cuocolo, A. Ponsiglione, V. Romeo and M. Imbriaco, Prostate MRI radiomics: A systematic review and radiomic quality score assessment, Eur. J. of Radiol., 129 (2020), 109095. doi: 10.1016/j.ejrad.2020.109095.
[33]	A. S. Tagliafico, M. Piana, D. Schenone, R. Lai, A. M. Massone and N. Houssami, Overview of radiomics in breast cancer diagnosis and prognostication, The Breast, 49 (2020), 74-80. doi: 10.1016/j.breast.2019.10.018.
[34]	G. Thibault, B. Fertil, C. Navarro, S. Pereira, P. Cau, N. Levy, J. Sequeira and J. L. Mari, Texture indexes and gray level size zone matrix: Application to cell nuclei classification, In 10th Int. Conf. on Pattern Recognit. and Inform. Process., PRIP 2009, 2009,140-145. https://www.thibault.biz/Doc/Publications/TextureIndexesAndGrayLevelSizeZoneMatrixApplicationToCellNucleiClassification_PRIP2009.pdf
[35]	A. Traverso, L. Wee, A. Dekker and R. Gillies, Repeatability and reproducibility of radiomic features: A systematic review, Int. J. of Radiat. Oncol. Biol. Phys., 102 (2018), 1143-1158. doi: 10.1016/j.ijrobp.2018.05.053.
[36]	L. Ugga, G. Spadarella, L. Pinto, R. Cuocolo and A. Brunetti, Meningioma radiomics: At the nexus of imaging, pathology and biomolecular characterization, Cancers, 14 (2022), 2605. doi: 10.3390/cancers14112605.
[37]	M. Vallières, C. R. Freeman, S. R. Skamene and I. El Naqa, A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities, Phys. Med. Biol., 60 (2015), 5471-5496. doi: 10.1088/0031-9155/60/14/5471.
[38]	J. E. van Timmeren, D. Cester, S. Tanadini-Lang, H. Alkadhi and B. Baessler, Radiomics in medical imaging-"how-to" guide and critical reflection, Insights into Imaging, 11 (2020), Article number: 91. doi: 10.1186/s13244-020-00887-2.
[39]	J. Wong, M. Baine, S. Wisnoskie, N. Bennion, D. Zheng, L. Yu, V. Dalal, M. A. Hollingsworth, C. Lin and D. Zheng, Effects of interobserver and interdisciplinary segmentation variabilities on CT-based radiomics for pancreatic cancer, Sci. Rep., 11 (2021), 16328. doi: 10.1038/s41598-021-95152-x.
[40]	G. Wu, Y. Chen, Y. Wang, J. Yu, X. Lv, X. Ju, Z. Shi, L. Chen and Z. Chen, Sparse representation-based radiomics for the diagnosis of brain tumors, IEEE Trans. on Med. Imaging, 37 (2018), 893-905. doi: 10.1109/TMI.2017.2776967.
[41]	C. Xue, J. Yuan, G. G. Lo, A. T. Y. Chang, D. M. C. Poon, O. L. Wong, Y. Zhou and W. C. W. Chu, Radiomics feature reliability assessed by intraclass correlation coefficient: a systematic review, Quant. Imaging in Med. and Surg., 11 (2021), 4431. doi: 10.21037/qims-21-86.