Novel evaluation approach for molecular signature-based deconvolution methods

Abstract

The tumoral immune microenvironment (TIME) plays a key role in prognosis, therapeutic approach and pathophysiological understanding over oncological processes. Several computational immune cell-type deconvolution methods (DM), supported by diverse molecular signatures (MS), have been developed to uncover such TIME interplay from RNA-seq tumor biopsies. MS-DM pairs were benchmarked against each other by means of different metrics, such as Pearson's correlation, R2 and RMSE, but these only evaluate the linear association of the estimated proportion related to the expected one, missing the analysis of prediction-dependent bias trends and cell identification accuracy. We present a novel protocol composed of four tests allowing appropriate evaluation of the cell type identification performance and proportion prediction accuracy of molecular signature-deconvolution method pair by means of certainty and confidence cell-type identification scores (F1-score, distance to the optimal point and error rates) as well the Bland-Altman method for error-trend analysis. Our protocol was used to benchmark six state-of-the-art DMs (CIBERSORTx, DCQ, DeconRNASeq, EPIC, MIXTURE and quanTIseq) paired to five murine tissue-specific MSs, revealing a systematic overestimation of the number of different cell types across almost all methods.