Module 5

Active Inference

Active inference extends the Bayesian brain into action: the agent can either update its beliefs to match the world (perception) or act on the world to match its beliefs (action). Both are instances of minimising expected free energy, which decomposes into epistemic and pragmatic value.

1. Expected Free Energy

For a candidate policy π, expected free energy G(π) is the free energy averaged over predicted future observations:

\[ G(\pi) \;=\; \underbrace{\mathbb{E}_{Q(o|\pi)}[\log Q(o|\pi) - \log P(o)]}_{\text{pragmatic (prior preference)}} + \underbrace{\mathbb{E}_{Q(o,z|\pi)}[\log Q(z|o,\pi) - \log Q(z|\pi)]}_{-\text{epistemic (info gain)}} \]

The agent prefers policies with low G: either policies that realise preferred outcomes (high pragmatic value) or policies that reduce uncertainty about hidden states (high epistemic value). The natural balance between exploration and exploitation emerges without ad-hoc exploration bonuses.

2. Embodied Cognition

Active inference is inherently embodied: action closes the perception-control loop. A dropping hand’s prediction of “hand-at-target” is fulfilled by reflex arcs that move the hand to the target, minimising proprioceptive prediction error. The motor system can be reconstructed as a predictive controller whose predictions are carried out by motor neurons (Adams 2013).

Simulation: Active-Inference Bandit

Python

script.py43 lines

import numpy as np, matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

# 3-armed bandit via active inference: EFE = epistemic + pragmatic
rng = np.random.default_rng(0)
n_arms = 3
true_reward = np.array([0.2, 0.5, 0.8])
# Beliefs over each arm: Beta(a, b)
a = np.ones(n_arms); b = np.ones(n_arms)
T = 100
history = np.zeros((T, n_arms))

for t in range(T):
    # Pragmatic value: expected reward = a/(a+b)
    prag = a / (a + b)
    # Epistemic value: information gain, approx var = ab/((a+b)^2 (a+b+1))
    epis = a*b / ((a+b)**2 * (a+b+1))
    # Total EFE negative; we want to maximise (prag + epistemic weight)
    w_epi = 5.0   # weight on exploration
    score = prag + w_epi * epis
    arm = int(np.argmax(score))
    reward = rng.random() < true_reward[arm]
    if reward: a[arm] += 1
    else: b[arm] += 1
    history[t] = a / (a + b)

fig, ax = plt.subplots(figsize=(11, 5), facecolor='#0a0a1a')
ax.set_facecolor('#111827'); ax.tick_params(colors='#cbd5e1')
for s in ax.spines.values(): s.set_color('#334155')
for k in range(n_arms):
    ax.plot(history[:, k], lw=2, label=f'Arm {k} (true p={true_reward[k]})')
ax.axhline(0.8, color='#fbbf24', ls=':')
ax.set_xlabel('Trial', color='#cbd5e1')
ax.set_ylabel('Posterior mean reward', color='#cbd5e1')
ax.set_title('Active inference on 3-armed bandit',
             color='#c4b5fd', fontweight='bold')
ax.legend(facecolor='#1e293b', edgecolor='#334155', labelcolor='#cbd5e1')
ax.grid(True, color='#334155', alpha=0.3)
plt.tight_layout()
plt.savefig('output.png', dpi=120, bbox_inches='tight', facecolor='#0a0a1a')
print('Epistemic weight drives exploration of low-sample arms early')
print('Pragmatic weight drives exploitation once beliefs sharpen')

Click Run to execute the Python code

Code will be executed with Python 3 on the server

3. Comparison with RL

Reinforcement learning maximises expected cumulative reward; active inference minimises expected free energy. The two converge in many cases; active inference additionally offers a principled handling of epistemic exploration, a built-in world-model requirement, and a natural integration with perception. Recent work (Friston 2021, Sajid 2021) has begun bridging the two frameworks formally.

Key References

• Friston, K. et al. (2015). “Active inference and epistemic value.” Cogn. Neurosci., 6, 187–214.

• Parr, T., Pezzulo, G. & Friston, K. J. (2022). Active Inference: The Free Energy Principle in Mind, Brain, and Behavior. MIT Press.

• Sajid, N. et al. (2021). “Active inference: demystified and compared.” Neural Comput., 33, 674–712.

• Adams, R. A. et al. (2013). “Predictions not commands: active inference in the motor system.” Brain Struct. Funct., 218, 611–643.

Share:X Reddit LinkedIn

← Module 4 Module 6: Neural Implementation →