Presented By: U-M Industrial & Operations Engineering
SEMINAR: "Information-theoretic Generalization Bounds for Noisy, Iterative Learning Algorithms" — Daniel Roy
The Departmental Seminar Series is open to all. U-M Industrial and Operations Engineering graduate students and faculty are especially encouraged to attend.
Title:
Information-theoretic Generalization Bounds for Noisy, Iterative Learning Algorithms
Abstract:
Deep learning approaches dominate in many application areas. Our understanding of generalization (relating empirical performance to future expected performance) is however lacking. In some applications, standard algorithms like stochastic gradient descent (SGD) reliably return solutions with low test error. In other applications, these same algorithms rapidly overfit. There is, as yet, no satisfying theory explaining what conditions are required for these common algorithms to work in practice. In this talk, I will discuss standard approaches to explaining generalization in deep learning using tools from statistical learning theory, and present some of the barriers these approaches face to explaining deep learning. I will then discuss my group's recent work (NeurIPS 2019, 2020) on information-theoretic approaches to understanding generalization of noisy, iterative learning algorithms, such as Stochastic Gradient Langevin Dynamics, a noisy version of SGD.
Bio:
Daniel Roy is an Associate Professor in the Department of Statistical Sciences at the University of Toronto, with cross appointments in Computer Science and Electrical and Computer Engineering. He is also a CIFAR Canada AI Chair at the Vector Institute. Roy's research spans machine learning, mathematical statistics, and theoretical computer science. Roy is a recipient of an NSERC Discovery Accelerator Supplement, Tri-agency New Frontiers in Research grant, Ontario Early Research Award, and a Google Faculty Research Award. Prior to joining Toronto, Roy was a Research Fellow of Emmanuel College and Newton International Fellow of the Royal Society and Royal Academy of Engineering, hosted by the University of Cambridge. Roy completed his doctorate in Computer Science at the Massachusetts Institute of Technology, where his dissertation was awarded the MIT EECS Sprowls Award, given to the top dissertation in computer science in that year.
Title:
Information-theoretic Generalization Bounds for Noisy, Iterative Learning Algorithms
Abstract:
Deep learning approaches dominate in many application areas. Our understanding of generalization (relating empirical performance to future expected performance) is however lacking. In some applications, standard algorithms like stochastic gradient descent (SGD) reliably return solutions with low test error. In other applications, these same algorithms rapidly overfit. There is, as yet, no satisfying theory explaining what conditions are required for these common algorithms to work in practice. In this talk, I will discuss standard approaches to explaining generalization in deep learning using tools from statistical learning theory, and present some of the barriers these approaches face to explaining deep learning. I will then discuss my group's recent work (NeurIPS 2019, 2020) on information-theoretic approaches to understanding generalization of noisy, iterative learning algorithms, such as Stochastic Gradient Langevin Dynamics, a noisy version of SGD.
Bio:
Daniel Roy is an Associate Professor in the Department of Statistical Sciences at the University of Toronto, with cross appointments in Computer Science and Electrical and Computer Engineering. He is also a CIFAR Canada AI Chair at the Vector Institute. Roy's research spans machine learning, mathematical statistics, and theoretical computer science. Roy is a recipient of an NSERC Discovery Accelerator Supplement, Tri-agency New Frontiers in Research grant, Ontario Early Research Award, and a Google Faculty Research Award. Prior to joining Toronto, Roy was a Research Fellow of Emmanuel College and Newton International Fellow of the Royal Society and Royal Academy of Engineering, hosted by the University of Cambridge. Roy completed his doctorate in Computer Science at the Massachusetts Institute of Technology, where his dissertation was awarded the MIT EECS Sprowls Award, given to the top dissertation in computer science in that year.
Related Links
Explore Similar Events
-
Loading Similar Events...
