Criteria for Selecting a Quality Index for Full-Reference Image Quality Assessment

Abstract

In the past few decades, many image quality indices have been developed. However, they stem from different theoretical frameworks, application scenarios and purposes. Thus, users and researchers are often faced with the time-consuming task of deciding which quality index to choose when they require a reliable image quality index that is capable of emulating the human visual system (HVS). In this work, general criteria for selecting the most appropriate index from a given set of quality indices according to application needs are established. These criteria are based on the statistical coefficients of correlation and concordance. It is discussed why Kendall’s Tau and Spearman’s rank correlation coefficients—which are widely used to compare quality index performance—are not sufficient for this purpose; moreover, additional nonparametric tests and methods of agreement are incorporated: the concordance coefficients (Kendall’s w, Cohen’s kappa, Scott’s pi and Fleiss’ kappa) not explored so far, to determine the best procedures to compare digital images. The combination of all these strategies led to a more complete comparison method, from which a ranking of quality indices could be generated from any set of them. As an application, the performance and suitability of a large number of quality indices for various real-world scenarios is compared. Our experiments reveal that the indices are sensitive to the type of distortions. This work expanded previous studies by incorporating directional indices, which perform well in the numerical experiments developed using real datasets.