The role of the information bottleneck in representation learning

Abstract

A grand challenge in representation learning is thedevelopment of computational algorithms that learn the differentexplanatory factors of variation behind high-dimensional data.Encoder models are usually determined to optimize performanceon training data when the real objective is to generalize well toother (unseen) data. Although numerical evidence suggests thatnoise injection at the level of representations might improve thegeneralization ability of the resulting encoders, an informationtheoretic justification of this principle remains elusive. In thiswork, we derive an upper bound to the so-called generalizationgap corresponding to the cross-entropy loss and show that whenthis bound times a suitable multiplier and the empirical riskare minimized jointly, the problem is equivalent to optimizingthe Information Bottleneck objective with respect to the empirical data-distribution. We specialize our general conclusionsto analyze the dropout regularization method in deep neuralnetworks, explaining how this regularizer helps to decrease thegeneralization gap.