Title: Text Diffusion Models: DiffuSeq, RDMs, and DoTs

Speaker: Prof. Lingpeng Kong @ HKU

Time : 15:00 ~ 16:00, June 3rd, 2024

Location: Online

• Online: https://us02web.zoom.us/j/85705374458?pwd=OEN6VWpZMXZPREhPN1oyTm5vSzUxQT09 

Language: English speech & English slides

Abstract: 

This talk explores three recent works from our group in text diffusion models: DiffuSeq, Reparameterized Discrete diffusion Models (RDMs), and Diffusion of Thoughts (DoTs). 

DiffuSeq extends the unconditional diffusion framework to conditional generation tasks, introducing partial noising and conditional denoising for high-quality, diverse text generation. RDMs reveal a latent routing mechanism in discrete diffusion, enabling more effective training and decoding strategies for a better runtime-performance tradeoff compared to existing language models. DoTs implement chain-of-thought reasoning in diffusion language models, allowing for single-pass and multi-pass thought refinement. Built on Plaid 1B, DoTs show strong performance on reasoning tasks while providing speed-ups and benefiting from self-consistency.

The talk covers the basics of diffusion processes, contrasting continuous and discrete diffusion. Results highlight how these methods are closing the gap with autoregressive models in generation quality while offering advantages in parallel generation and runtime. Overall, the talk showcases the rapid progress and potential of diffusion models for advancing state-of-the-art natural language generation.

Bio:

Lingpeng Kong is an assistant professor in the Department of Computer Science at the University of Hong Kong (HKU) and a co-director of the HKU NLP Lab. His work lies at the intersection of natural language processing (NLP) and machine learning (ML), with a focus on representation learning, structured prediction, and generative models. Before joining HKU, Kong was a research scientist at Google DeepMind. He obtained his Ph.D. from Carnegie Mellon University.