Repository logo

Infoscience

  • English
  • French
Log In
Logo EPFL, École polytechnique fédérale de Lausanne

Infoscience

  • English
  • French
Log In
  1. Home
  2. Academic and Research Output
  3. Conferences, Workshops, Symposiums, and Seminars
  4. SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
 
Loading...
Thumbnail Image
conference paper

SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient

Mishkin, Aaron
•
Kunstner, Frederik
•
Nielsen, Didrik
Show more
January 1, 2018
Advances In Neural Information Processing Systems 31 (Nips 2018)
32nd Conference on Neural Information Processing Systems (NIPS)

Uncertainty estimation in large deep-learning models is a computationally challenging task, where it is difficult to form even a Gaussian approximation to the posterior distribution. In such situations, existing methods usually resort to a diagonal approximation of the covariance matrix despite the fact that these matrices are known to result in poor uncertainty estimates. To address this issue, we propose a new stochastic, low-rank, approximate natural-gradient (SLANG) method for variational inference in large, deep models. Our method estimates a "diagonal plus low-rank" structure based solely on back-propagated gradients of the network log-likelihood. This requires strictly less gradient computations than methods that compute the gradient of the whole variational objective. Empirical evaluations on standard benchmarks confirm that SLANG enables faster and more accurate estimation of uncertainty than mean-field methods, and performs comparably to state-of-the-art methods.

  • Details
  • Metrics
Type
conference paper
Web of Science ID

WOS:000461852000072

Author(s)
Mishkin, Aaron
•
Kunstner, Frederik
•
Nielsen, Didrik
•
Schmidt, Mark
•
Khan, Mohammad Emtiyaz  
Date Issued

2018-01-01

Publisher

NEURAL INFORMATION PROCESSING SYSTEMS (NIPS)

Publisher place

La Jolla

Published in
Advances In Neural Information Processing Systems 31 (Nips 2018)
Series title/Series vol.

Advances in Neural Information Processing Systems

Volume

31

Subjects

Computer Science, Artificial Intelligence

•

Computer Science

•

gaussian variational approximation

Peer reviewed

REVIEWED

Written at

EPFL

EPFL units
INDY1  
Event nameEvent placeEvent date
32nd Conference on Neural Information Processing Systems (NIPS)

Montreal, CANADA

Dec 02-08, 2018

Available on Infoscience
June 18, 2019
Use this identifier to reference this record
https://infoscience.epfl.ch/handle/20.500.14299/157265
Logo EPFL, École polytechnique fédérale de Lausanne
  • Contact
  • infoscience@epfl.ch

  • Follow us on Facebook
  • Follow us on Instagram
  • Follow us on LinkedIn
  • Follow us on X
  • Follow us on Youtube
AccessibilityLegal noticePrivacy policyCookie settingsEnd User AgreementGet helpFeedback

Infoscience is a service managed and provided by the Library and IT Services of EPFL. © EPFL, tous droits réservés