Applied Mathematics Colloquium: Dr. Mamikon Gulian

Title: Robust architectures, initialization, and training for deep neural networks via the adaptive basis interpretation

Abstract: Deep neural networks (DNNs) hold intriguing potential for scientific problems. Some of their advantages include simple implementation using off-the-shelf codes, straightforward ways to utilize both data and physics knowledge or other constraints, and their potential in very high dimensional problems for which standard numerical methods are not feasible. These advantages have been demonstrated in many prototypical examples, including reduced-order modeling using DNNs and physics-informed neural networks (PINNs). At the same time, scientific methods based on DNNs exhibit new and challenging problems, such as hyperparameter tuning, stagnation of approximation errors, instability of DNN training and initialization, and lack of theoretical guarantees. In this talk, we will introduce how DNNs are used for scientific problems, with a focus on comparing them to numerical methods. We introduce the adaptive basis interpretation of DNNs as well as a novel initialization and Least Squares/Gradient Descent optimizer that greatly reduce training error and yield consistent improvement in training error as the size of the DNN increases. Time permitting, we will also discuss deep Partition-of-Unity networks, a novel architecture which leverages the strengths of DNNs for classification to construct mesh-free partitions of space, which are combined with traditional higher-order methods to yield precise approximation.