Information flow in Deep Restricted Boltzmann Machines: An analysis of mutual information between inputs and outputs | ||
| dc.conicet.organizacion | Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Parque Centenario. Centro de Simulación Computacional para Aplicaciones Tecnológicas | |
| dc.conicet.organizacion | Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Parque Centenario. Centro de Simulación Computacional para Aplicaciones Tecnológicas | |
| dc.conicet.organizacion | Centre National de la Recherche Scientifique | |
| dc.contributor.author | Vera, Matías Alejandro | |
| dc.contributor.author | Rey Vega, Leonardo Javier | |
| dc.contributor.author | Piantanida, Pablo | |
| dc.date.issued | 2022-10 | |
| dc.description.abstract | Empirical evidence suggests the existence of an entangled relationship between the information flow from inputs features to hidden representations of a deep neural network and its ability to generalize from training samples to unobserved data. For instance, regularization techniques often used to control statistical generalization, are expected to impact this information flow. In this work, we study MI (mutual information) between inputs and representation outputs, and its relationship with various regularization methods commonly used in Restricted Boltzmann Machines (RBM) and their generalizations: Deep Belief Networks and Deep Boltzmann Machines. Our theoretical findings show the existence of fundamental connections between the hyperparameters associated with the regularization and the MI, including relevant practical ingredients such as: network dimension, matrix norms and dropout probability, which are well-known to influence the generalization ability of the network. These results are experimentally corroborated on various visual datasets. | es |
| dc.description.fil | Fil: Vera, Matías Alejandro. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Parque Centenario. Centro de Simulación Computacional para Aplicaciones Tecnológicas; Argentina | |
| dc.description.fil | Fil: Rey Vega, Leonardo Javier. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Parque Centenario. Centro de Simulación Computacional para Aplicaciones Tecnológicas; Argentina | |
| dc.description.fil | Fil: Piantanida, Pablo. Centre National de la Recherche Scientifique; Francia | |
| dc.format | application/pdf | es |
| dc.identifier.citation | Vera, Matías Alejandro; Rey Vega, Leonardo Javier; Piantanida, Pablo; Information flow in Deep Restricted Boltzmann Machines: An analysis of mutual information between inputs and outputs; Elsevier Science; Neurocomputing; 507; 10-2022; 235-246 | es |
| dc.identifier.institucional | CONICET | |
| dc.identifier.issn | 0925-2312 | es |
| dc.identifier.repo | CONICET Digital | |
| dc.identifier.uri | http://hdl.handle.net/11336/193980 | |
| dc.journal.ciudad | Amsterdam | es |
| dc.journal.pagination | 235-246 | es |
| dc.journal.pais | Países Bajos | es |
| dc.journal.title | Neurocomputing | es |
| dc.journal.volume | 507 | es |
| dc.language.iso | eng | es |
| dc.publisher | Elsevier Science | es |
| dc.relation.alternativeid | info:eu-repo/semantics/altIdentifier/doi/https://doi.org/10.1016/j.neucom.2022.08.014 | |
| dc.relation.alternativeid | info:eu-repo/semantics/altIdentifier/url/https://www.sciencedirect.com/science/article/abs/pii/S0925231222009833 | |
| dc.rights | info:eu-repo/semantics/restrictedAccess | * |
| dc.rights.uri | https://creativecommons.org/licenses/by-nc-sa/2.5/ar/ | * |
| dc.subject | GRAPHICAL MODELS | es |
| dc.subject | MUTUAL INFORMATION | es |
| dc.subject | REGULARIZATION | es |
| dc.subject | RESTRICTED BOLTZMANN MACHINE | es |
| dc.subject | UNSUPERVISED LEARNING | es |
| dc.subject.classification | Ciencias de la Computación | es |
| dc.subject.classification | Ciencias de la Computación e Información | es |
| dc.subject.classification | CIENCIAS NATURALES Y EXACTAS | es |
| dc.title | Information flow in Deep Restricted Boltzmann Machines: An analysis of mutual information between inputs and outputs | es |
| dc.type | info:eu-repo/semantics/article | |
| dc.type | info:ar-repo/semantics/artículo | |
| dc.type | info:eu-repo/semantics/publishedVersion | |
Archivos
Bloque original
Cargando...
- Nombre:
- CONICET_Digital_Nro.49cd5807-7b9c-4b2b-9b9d-0eac5aca817b_E.pdf
- Tamaño:
- 1.71 MB
- Formato:
- Adobe Portable Document Format