Podlipodcast player Webplayer

Stanford Online: University Courses & Lectures

Stanford Online: University Courses & Lectures

Stanford CME295 Transformers and LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs

Stanford Online: University Courses & Lectures · Nov 16, 2026 · 1:50:11

0:001:50:11

Listen in the Podli app 🎧

Follow your favourite podcasts, listen offline and in the car with CarPlay and Android Auto, and always pick up where you left off. Free to try.

For more information about Stanford’s graduate programs, visit: https://online.stanford.edu/graduate-educationTo follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/Chapters:00:00:00 Introduction00:06:35 RL setup00:09:32 Reward design00:12:23 Policy gradient00:17:12 Derivation of the gradient00:23:50 REINFORCE00:25:51 Importance sampling ratio00:33:22 Challenges of policy gradients00:40:34 Proximal Policy Optimization (PPO)00:42:49 Value function and advantage00:51:16 Policy ratio 00:53:03 Ratio clipping00:57:45 KL Penalty00:59:48 Connection with DPO01:07:29 Group Relative Policy Optimization (GRPO)01:09:42 Comparison of GRPO and PPO01:18:25 Limitations of GRPO01:22:41 DAPO, Dr. GRPO01:25:17 On-Policy Distillation (OPD)01:39:32 Self (OPSD) and Multi-Teacher (OPD) variantsUploaded 19-10-26Afshine Amidi is an Adjunct Professor at Stanford University.Shervine Amidi is an Adjunct Professor at Stanford University.Follow the playlist: https://youtube.com/playlist?list=PLakGfbKCei80&si=HF2tNywj7nXIQq8X
Learn more about your ad choices. Visit megaphone.fm/adchoices

Episodes: Stanford Online: University Courses & Lectures

PodliGet the free Podli app
↓ App