BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//UM//UM*Events//EN
CALSCALE:GREGORIAN
BEGIN:VTIMEZONE
TZID:America/Detroit
TZURL:http://tzurl.org/zoneinfo/America/Detroit
X-LIC-LOCATION:America/Detroit
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20070311T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20071104T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20250828T122801
DTSTART;TZID=America/Detroit:20251015T160000
DTEND;TZID=America/Detroit:20251015T170000
SUMMARY:Workshop / Seminar:Normalization effects on deep neural networks and deep learning for scientific problems.
DESCRIPTION:We study the effect of normalization on the layers of deep neural networks. A given layer i with N_{i} hidden units is normalized by 1/N_{i}^{γ_{i}} with γ_{i}∈[1/2\,1]. We study the effect of the choice of the γ_{i} on the statistical behavior of the neural network’s output (such as variance) as well as on the test accuracy and generalization properties of the architecture. We find that in terms of variance of the neural network’s output and test accuracy the best choice is to choose the γ_{i}’s to be equal to one\, which is the mean-field scaling. We also find that this is particularly true for the outer layer\, in that the neural network’s behavior is more sensitive in the scaling of the outer layer as opposed to the scaling of the inner layers. The mechanism for the mathematical analysis is an asymptotic expansion for the neural network’s output. An important practical consequence of the analysis is that it provides a systematic and mathematically informed way to choose the learning rate hyperparameters. Such a choice guarantees that the neural network behaves in a statistically robust way as the number of hidden units N_{i} grow. Time permitting\, I will discuss applications of these ideas to design of deep learning algorithms for scientific problems including solving high dimensional partial differential equations (PDEs)\, closure of PDE models and reinforcement learning with applications to financial engineering\, turbulence and more.
UID:138049-21881419@events.umich.edu
URL:https://events.umich.edu/event/138049
CLASS:PUBLIC
STATUS:CONFIRMED
CATEGORIES:Mathematics
LOCATION:East Hall - 1360
CONTACT:
END:VEVENT
END:VCALENDAR