Do concern mining tools really help requirements analysts? An empirical study of the vetting process

Abstract

Software requirements are often described in natural language because they are useful to communicate and validate. Due to their focus on particular facets of a system, this kind of specifications tends to keep relevant concerns (also known as early aspects) from the analysts’ view. These concerns are known as crosscutting concerns because they appear scattered among documents. Concern mining tools can help analysts to uncover concerns latent in the text and bring them to their attention. Nonetheless, analysts are responsible for vetting tool-generated solutions, because the detection of concerns is currently far from perfect. In this article, we empirically investigate the role of analysts in the concern vetting process, which has been little studied in the literature. In particular, we report on the behavior and performance of 55 subjects in three case-studies working with solutions produced by two different tools, assessed in terms of binary classification measures. We discovered that analysts can improve “bad” solutions to a great extent, but performed significantly better with “good” solutions. We also noticed that the vetting time is not a decisive factor to their final accuracy. Finally, we observed that subjects working with solutions substantially different from those of existing tools (better recall) can also achieve a good performance.