← All projectsResearch · multi-agent reinforcement learning

Adaptive MAPPO

When a team of agents learns together, should each one judge its moves by what the whole team sees, or by what it sees alone? We let a small network decide, state by state.

Role: Designed the adaptive critic, led the method
Team: Course project, ECE 567 · UMich
When: Winter 2026
Stack: PyTorch · TorchRL · BenchMARL
Scale: 4 tasks × 6 methods × 3 seeds = 72 runs

MAPPO

Adaptive MAPPO

Real outputMultiwalker: three bipedal walkers must carry a package together without dropping it. Each clip is that method’s best checkpoint.

01The question

One critic for the team, or one for each agent?

In PPO, a critic estimates how good the current situation is, so the policy can tell whether an action helped. With several agents there’s a choice to make.

MAPPO gives the critic the whole team’s state. It understands teamwork, but its signal is noisy for any single agent: did we win because of me? IPPO gives each agent a critic over its own view. It’s steadier, but blind to teammates. In our baselines neither wins everywhere; they trade places from task to task and even within one training run.

So instead of choosing, blend them. Drag α to see what the policy would learn from:

Team criticsees every agent0.80Own criticsees only me0.30Value PPO usesthe blend0.55

Drag α: 0.50

0 · IPPO (own only)1 · MAPPO (team only)

v = 0.50 × 0.8 + 0.50 × 0.3 = 0.55

IllustrativeThe two critic values are made up to show the mechanism; the blend formula is the real one.

MAPPO (team critic)IPPO (own critic)

Re-plotted from logsVMAS Balance, mean of 3 seeds with min–max band. The two critics trade the lead several times during training.

02Method

Let a small network choose α for every state

01Both critics score the stateThe team critic V_global and each agent’s own critic V_local.
02Six featuresBoth values (normalized), their disagreement, team variance, each agent’s deviation from the team, and their log-ratio.
03Tiny MLP → α6 → 16 → 1 with LayerNorm and a sigmoid, trained end-to-end with the PPO loss.learned per state
04Blend and trainv = α·V_global + (1−α)·V_local feeds GAE advantages for clipped PPO.

The adaptive critic. Everything else is standard clipped PPO with GAE.

# features from both critics (normalized on the team critic's scale)
v_g = (v_global - mean) / std
v_l = (v_local  - mean) / std
features = cat([v_g, v_l,
                |v_g - v_l|,             # disagreement
                var_team(v_l),           # how split the team is
                v_l - mean_team(v_l),    # this agent vs the team
                log(|v_global| / |v_local|)])

alpha = sigmoid(mlp(features))           # 6 -> 16 -> 1
v_mix = alpha * v_global + (1 - alpha) * v_local   # raw scale for the PPO target

We compared against pure MAPPO (α = 1), pure IPPO (α = 0), and three fixed blends (α = 0.25, 0.5, 0.75), on two PettingZoo tasks (Multiwalker, Simple World Comm) and two VMAS tasks (Navigation, Balance), three seeds each, 3M frames per run on the Great Lakes cluster.

03Results

α lands somewhere different on every task

MultiwalkerNavigationBalanceWorld Comm

Re-plotted from logsMean learned α across 3 seeds (bands: min–max). Nobody told it where to go: on Multiwalker it settles near the team critic, on Navigation near each agent’s own.

21.2%drop from peak reward on Multiwalkervs 49.4% for MAPPO and 141.3% for IPPO

1stmost stable on Multiwalker and Navigationof 6 methods, ranked by drop from peak

0.94 / 0.24final α on Multiwalker / Navigationsame algorithm, opposite choices

Real outputMultiwalker: how much reward each method lost between its best checkpoint and the end of training (lower is more stable). Mean of 3 seeds.

All four tasks, all six methods

Adaptive MAPPO ranks first on Multiwalker and Navigation, third on Balance (where α simply converges to pure MAPPO), and last on Simple World Comm, a task built around communication where α wandered and a low fixed α did best.

Rank by drop from peak reward (1 = most stable).

Source: aggregate of 72 runs (mean over 3 seeds).
Task	Method	Rank	Drop from peak	Best reward	Final reward
Multiwalker	Adaptive (ours)	1	21.2%	12.48	9.83
Multiwalker	MAPPO	2	49.4%	5.98	3.03
Multiwalker	IPPO	6	141.3%	9.74	-4.02
Multiwalker	Fixed α = 0.25	3	53.3%	34.19	15.98
Multiwalker	Fixed α = 0.50	5	110.2%	16.48	-1.68
Multiwalker	Fixed α = 0.75	4	78.4%	24.34	5.27
Navigation	Adaptive (ours)	1	15.2%	1.12	0.95
Navigation	MAPPO	3	16.8%	1.17	0.97
Navigation	IPPO	6	23.7%	1.14	0.87
Navigation	Fixed α = 0.25	5	20.3%	1.13	0.90
Navigation	Fixed α = 0.50	2	16.5%	1.12	0.94
Navigation	Fixed α = 0.75	4	17%	1.16	0.96
Balance	Adaptive (ours)	3	17.2%	53.85	44.56
Balance	MAPPO	2	16.1%	54.06	45.37
Balance	IPPO	4	25.8%	52.08	38.65
Balance	Fixed α = 0.25	1	13.9%	55.39	47.71
Balance	Fixed α = 0.50	6	27.3%	54.78	39.81
Balance	Fixed α = 0.75	5	27.1%	53.61	39.06
World Comm	Adaptive (ours)	6	43%	30.68	17.48
World Comm	MAPPO	3	24.5%	39.92	30.16
World Comm	IPPO	2	16.4%	38.38	32.08
World Comm	Fixed α = 0.25	1	2.6%	41.13	40.07
World Comm	Fixed α = 0.50	5	35.3%	39.91	25.82
World Comm	Fixed α = 0.75	4	33.3%	38.13	25.42

04Next

What I’d try next

Give α the agents’ observations, not just the two values, so it can react to what’s happening on screen.
Understand the communication task: does α need to differ for speakers and listeners?
Harder coordination benchmarks with more agents, where a team critic gets noisier.

Want to see agents learning live? The football simulation on my homepage is a separate, illustrative demo of multi-agent RL.

Building agents that have to cooperate? I’d like to hear what you’re working on.

Email me LinkedIn ↗GitHub ↗Résumé ↗