A Data-Centric View on Reliable Generalization

Title: A Data-Centric View on Reliable Generalization

Abstract: Researchers have proposed many methods to make neural networks more reliable under distribution shift, yet there is still vast room for improvement. Are better training algorithms or training data the more promising way forward? In this talk, we study this question in the context of computer vision and OpenAI’s CLIP model for learning from image-text data.First, we survey the current robustness landscape based on a large-scale experimental study involving more than 200 different models and test conditions. The CLIP models stand out with unprecedented robustness gains on multiple challenging distribution shifts. To further improve CLIP, we then introduce new methods for reliably fine-tuning models by interpolating the weights of multiple models. Finally, we investigate the cause of CLIP’s robustness via controlled experiments to disentangle the influence of language supervision and training distribution. While CLIP leveraged large scale language supervision for the first time, its robustness actually comes from the pre-training dataset.Based on our findings, we will conclude with initial experiments to improve the pre-training dataset for CLIP models, and with thoughts on dataset design for machine learning.

Speaker Bio: Ludwig Schmidt is an assistant professor in the Paul G. Allen School of Computer Science & Engineering at the University of Washington. Ludwig’s research interests revolve around the empirical and theoretical foundations of machine learning, often with a focus on datasets, evaluation, and reliable methods. Ludwig completed his PhD at MIT under the supervision of Piotr Indyk and was a postdoc at UC Berkeley with Benjamin Recht and Moritz Hardt. Ludwig’s research received a new horizons award at EAAMO, a best paper award at ICML, a best paper finalist at CVPR, and the Sprowls dissertation award from MIT.