Abstract: While overparameterization in machine learning models offers great benefits in terms of optimization and generalization, it also leads to increased computational requirements as model sizes grow. In this work, we demonstrate that we can reap the benefits of overparameterization without the computational burden. First, we develop theory showing that when training the parameters of a deep linear network to fit a low-rank or wide matrix, the gradient dynamics of each weight matrix are confined to an invariant low-dimensional subspace. This is done by carefully studying the gradient update step, which is the product of several matrix variables, and noticing the way low-rank structure passes from the low-rank target through the variables sequentially. Given this invariant subspace, we can construct and train compact, highly compressed factorizations possessing the same benefits as their overparameterized counterparts. For language model fine-tuning, we introduce a method called "Deep LoRA", which improves the existing low-rank adaptation (LoRA) technique. While this technique does not arise directly from our theory, it involves only a minor modification that is surprisingly effective and of great interest for future theoretical study.

Contact: Peter Miller