Capítulo de Libro
Information Bottleneck and Representation Learning
Título del libro: Information-Theoretic Methods in Data Science
Fecha de publicación:
2020
Editorial:
Cambridge University Press
ISBN:
9781108616799
Idioma:
Inglés
Clasificación temática:
Resumen
A grand challenge in representation learning is the development of computational algorithms that learn the different explanatory factors of variation behind high-dimensional data. Representation models (usually referred to as encoders) are often determined for optimizing performance on training data when the real objective is to generalize well to other (unseen) data. The first part of this chapter is devoted to provide an overview of and introduction to fundamental concepts in statistical learning theory and the Information Bottleneck principle. It serves as a mathematical basis for the technical results given in the second part, in which an upper bound to the generalization gap corresponding to the cross-entropy risk is given. When this penalty term times a suitable multiplier and the cross entropy empirical risk are minimized jointly, the problem is equivalent to optimizing the Information Bottleneck objective with respect to the empirical data distribution. This result provides an interesting connection between mutual information and generalization, and helps to explain why noise injection during the training phase can improve the generalization ability of encoder models and enforce invariances in the resulting representations.
Palabras clave:
LEARNING
,
INFORMATION
,
RATE-DISTORTION
,
GENERALIZATION
Archivos asociados
Licencia
Identificadores
Colecciones
Capítulos de libros(CSC)
Capítulos de libros de CENTRO DE SIMULACION COMPUTACIONAL P/APLIC. TECNOLOGICAS
Capítulos de libros de CENTRO DE SIMULACION COMPUTACIONAL P/APLIC. TECNOLOGICAS
Citación
Piantanida, Pablo; Rey Vega, Leonardo Javier; Information Bottleneck and Representation Learning; Cambridge University Press; 2020; 330-358
Compartir