On Solving Inverse Problems Using Latent Diffusion-based Generative Models

Abstract: Diffusion models have emerged as a powerful new approach to generative modeling. In this talk, we present the first framework that uses pre-trained latent diffusion models to solve linear inverse problems such as image denoising, inpainting, and super-resolution. Previously proposed algorithms (such as DPS and DDRM) only apply to pixel-space diffusion models. We theoretically analyze our algorithm showing provable sample recovery in a linear model setting. The algorithmic insight obtained from our analysis extends to more general settings often considered in practice. Experimentally, we outperform previously proposed posterior sampling algorithms in a wide variety of problems including random inpainting, block inpainting, denoising, deblurring, destriping, and super-resolution. Next, we present an efficient second-order approximation using Tweedie's formula to mitigate the bias incurred in the widely used first-order samplers. With this method, we devise a surrogate loss function to refine the reverse process at every diffusion step to address inverse problems and perform high-fidelity text-guided image editing. Based on joint work with Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, Yujia Chen, Abhishek Kumar, and Wen-Sheng Chu. Papers: https://arxiv.org/abs/2307.00619 , https://arxiv.org/abs/2312.00852

Bio: Sanjay Shakkottai is a Professor and holds the Cockrell Family Chair in Engineering # 15 in the Department of Electrical and Computer Engineering at The University of Texas at Austin. Sanjay Shakkottai received his Ph.D. from the ECE Department at the University of Illinois at Urbana-Champaign in 2002. He is with The University of Texas at Austin, where he is a Professor in the Department of Electrical and Computer Engineering, and holds the Cockrell Family Chair in Engineering #15. He received the NSF CAREER award in 2004 and was elected as an IEEE Fellow in 2014. He was a co-recipient of the IEEE Communications Society William R. Bennett Prize in 2021. His research interests lie at the intersection of algorithms for resource allocation, statistical learning and networks, with applications to wireless communication networks and online platforms.