Learning from Noisy Labels and Imperfect Teachers

Abstract: Datasets used in machine learning and statistics are huge and often imperfect, e.g., they contain corrupted data, examples with wrong labels, or hidden biases. Existing approaches often produce unreliable results when the datasets are corrupted, are computationally inefficient, or come without any theoretical/provable performance guarantees. In this talk,  I will discuss the design of learning algorithms that are computationally efficient and provably reliable, and then present an application in knowledge distillation.

I will first focus on the theory of supervised learning settings with noisy labels. I will present efficient and optimal learners under the semi-random noise models of Massart and Tsybakov – where the true label of each example is flipped with probability at most 50% – and an efficient approximate learner under adversarial label noise – where a small but arbitrary fraction of labels is flipped – under structured feature distributions.

In the second part of the talk we consider applications of the label-noise framework in knowledge distillation with unlabeled examples: a powerful training paradigm for generating compact and lightweight student models in applications where the amount of labeled data is limited but one has access to a large pool of unlabeled data. I will present a new method for this problem called Student-Label Mixing (SLaM) and show that it consistently improves over prior approaches by evaluating it on several standard benchmarks.

Speaker Bio: Vasilis Kontonis is a postdoctoral fellow at IFML. His research focuses on the theoretical foundations of machine learning aiming to design algorithms with provable guarantees of efficiency and robustness.  He received his Ph.D. in Computer Science from University of Wisconsin-Madison in 2023 and his M. Eng. and B. Eng. in Electrical and Computer Engineering from the National Technical University of Athens (NTUA) in 2017.