Effective benchmarking of structured argumentation via the generation of synthetic DeLP knowledge bases

Abstract

Computational models of argumentation is an active research field within Artificial Intelligence, with growing recent interest in real-world applications due to its intuitive reasoning mechanism and similarities to human reasoning. In this work, we focus on structured argumentation, which differs from abstract argumentation frameworks in that they work with the internal structure of the arguments. Much of the research in this area is still theoretical in nature, mostly because of the lack of benchmarks (which have mostly been investigated for abstract argumentation)—this has proven to be an important hurdle to overcome in the path to experimental research. Towards addressing this issue, we develop a knowledge base generator, called DPG, for the Defeasible Logic Programming (DeLP) framework that is designed to build synthetic programs for the practical evaluation of several computational tasks based on a set of nine parameters. We also develop a set of seven metrics in order to adequately assess the structural complexity of the generated programs and the computational cost of querying them; we study the computational complexity of computing such metrics, showing that most of them are intractable. Finally, we carry out an experimental evaluation of DPG in two parts. First, we carry out a rigorous study of the relationship between DPG parameter settings and metrics, which leads to the identification of interesting correlations that yield insights into how parameters must be adjusted in order to generate programs of a desired nature. Second, by setting thresholds for the value of specific metrics, we can divide programs into ‘‘easy’’ or ‘‘hard’’, and we show that it is feasible to train binary classifiers that, given a specific set of values for the parameters of the generator, are able to predict whether or not the generated instances will be difficult or easy to solve in terms of the running time metric. We also investigate the use of autoencoders to support the fully automated generation of easy or hard programs, achieving promising results in this task as well. These results demonstrate that one can avoid the theoretically costly ‘‘generate and check’’ process.