#4 Beren Millidge: Reinforcement Learning through Active Inference

Beren is a postdoc in Oxford with a background in machine learning and computational neuroscience. He is interested in Active Inference (related to the Free Energy Principle) and how the cortex can perform long-term credit assignment as deep artificial neural networks do. We start off with some shorter questions on the Free Energy Principle and its background concepts. Next, we get onto the exploration vs exploitation dilemma in reinforcement learning and Beren's strategy on how to maximize expected reward from restaurant visits - it's a long episode :=). We also discuss multimodal representations, shallow minima, autism, and enactivism. Then, we explore predictive coding going all the way from the phenomenon of visual fading, to 20-eyed reinforcement learning agents and the 'Anti-Grandmother Cell'. Finally, we discuss some open questions about backpropagation and the role of time in the brain, and finish the episode with some career advice about writing, publishing, and Beren's future projects!

Timestamps:

(00:00) - Intro

(02:11) - The Free Energy Principle, Active Inference, and Reinforcement Learning

(13:40) - Exploration vs Exploitation

(26:47) - Multimodal representation, shallow minima, autism

(36:11) - Biased generative models, enactivism, and representation in the brain?

(45:21) - Fixational eye movements, predictive coding, and 20-eyed RL

(52:57) - Precision, attention, and dopamine

(01:01:51) - Sparsity, negative prediction errors, and the 'Anti-Grandmother Cell'

(01:11:23) - Backpropagation in the brain?

(01:19:25) - Time in machine learning and the brain?

(01:25:32) - Career Questions

Beren's Twitter: 

Beren's Google Scholar

My Twitter

Papers

Deep active inference as variational policy gradients paper

Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs paper

Predictive Coding: a Theoretical and Experimental Review paper