Learning across an agent population

There are 1,000 persistent agents, each with its own randomly initialized 7 → 8 → 3 tanh Q-network: 91 parameters per brain. Every matchmaking round shuffles the IDs and creates 500 peer matches. Each agent plays exactly once per round. There is no cloning of winners, replacement of weak agents, shared policy, or shared replay memory.

The worker computes these matches sequentially to keep resource use manageable. It is a league of random pairings, not a thousand paddles occupying one rink. Longer rallies give their participants more moves and updates, so equal match counts do not mean identical training exposure.

All agents use learning rate 0.02, exploration 20%, a 128-experience replay buffer, and two sampled updates per move. The game and reward shaping are the same as in the earlier hockey lesson. Compact networks and buffers let the full population run locally.

The reference predicts where the puck will reach its paddle, including reflections at the top and bottom walls. It obeys the same paddle speed and collision rules as the learners. On the final approach, it chooses among legal moves to return the puck toward a location far from the opponent’s current paddle.

This is a strong predictive controller, not a proven optimal strategy. It has exact state information, just like the coordinate-based learners, but uses a hand-written physics model instead of a learned policy. The dashed chart line is its measured result against itself on the same benchmark suite.

At round 0 and every 10 rounds, each agent plays 16 deterministic benchmark rallies against the fixed reference, with both playing sides and serve directions balanced. Exploration and learning are off during evaluation. The benchmark does not alter the agent’s weights, replay buffer, training RNG, or the peer-match RNG.

Match points measure success against the reference; return rate measures defensive skill. A learner can improve its defense while still rarely beating the reference. The fixed suite makes successive measurements comparable, but it is a small set of scenarios. It cannot establish universal skill or permanent stagnation.

The population band contains the 10th–90th percentiles at each checkpoint. It does not track the same agents at its edges: individuals can move through the ranking. Select an agent to follow a fixed identity.

Compare each agent with itself two checkpoints earlier. A change of at least 10 percentage points in either return rate or match points is flagged as improvement or regression. If the metrics move strongly in opposite directions, the label is mixed. If neither crosses that threshold, the label is flat recently.

A flat agent may already be near the reference’s performance ceiling, may need more experience, or may be in a temporary plateau. Check its absolute score and training moves. These labels describe this 20-round window; they do not predict that an agent will never learn. Repeated seeds and longer runs are needed for stronger conclusions.

“Load prerun data” fetches a recorded 200-round run with all 1,000 agents and 21 benchmark checkpoints, including the untrained population. The history player only reveals those saved measurements; it does not train or invent intermediate results.

The population file stores per-agent outcomes and training-move counts, not the weights of every agent at every checkpoint. Loaded population results are therefore for inspection. “Start fresh” creates a new live population. Earlier lessons’ loaders include final model weights so those trained players can be watched immediately; their replay memories restart if training is resumed.

The generation script is scripts/generate-population.ts, run with bun scripts/generate-population.ts 200 1000 public/preruns/hockey-population.json. It uses seed 42. The JSON includes the run configuration, generation time, and measured duration. Download the recorded results.

import { HockeyAgent, HockeyMatch, createMatchStats, PADDLE_HALF, PUCK_RADIUS, type Rink, type Side } from './pong-hockey';
import { randomSource } from './star-training';

export const POPULATION_SIZE = 1000;
export const BENCHMARK_RALLIES = 16;
export const POPULATION_CONFIG = { seed: 42, rate: .02, exploration: .2, width: 8, replay: 128, updates: 2, evaluateEvery: 10 };
const clampPaddle = (y: number) => Math.max(PADDLE_HALF, Math.min(1 - PADDLE_HALF, y));
export function reflectedHeight(y: number) {
  const length = 1 - 2 * PUCK_RADIUS, period = length * 2;
  const folded = ((y - PUCK_RADIUS) % period + period) % period;
  return PUCK_RADIUS + (folded <= length ? folded : period - folded);
}

// Fixed predictive controller. It obeys the same movement limits as learners.
export function referenceAction(rink: Rink, side: Side): number {
  const incoming = side === 0 ? rink.vx < 0 : rink.vx > 0;
  const paddle = rink.paddles[side];
  if (!incoming) {
    return [0, 1, 2].reduce((best, action) => Math.abs(clampPaddle(paddle + (action - 1) * .035) - .5) < Math.abs(clampPaddle(paddle + (best - 1) * .035) - .5) ? action : best, 1);
  }
  const line = side === 0 ? .065 : .935;
  const moves = Math.max(1, Math.ceil((line - rink.x) / rink.vx));
  const landing = reflectedHeight(rink.y + rink.vy * moves);
  let best = 1, bestScore = -Infinity;
  for (const action of [1, 0, 2]) {
    const next = clampPaddle(paddle + (action - 1) * .035);
    const error = Math.abs(next - landing);
    let score = -error;
    if (moves === 1 && error <= PADDLE_HALF + PUCK_RADIUS) {
      const velocity = (landing - next) / (PADDLE_HALF + PUCK_RADIUS) * .05 + (action - 1) * .005;
      const flight = Math.ceil(.87 / Math.min(.055, Math.abs(rink.vx) * 1.07));
      const destination = reflectedHeight(landing + velocity * flight);
      score = 100 + Math.abs(destination - rink.paddles[1 - side]);
    }
    if (score > bestScore) { bestScore = score; best = action; }
  }
  return best;
}

export type Benchmark = { wins: number; draws: number; losses: number; returns: number; shots: number };
export type PopulationCheckpoint = { round: number; agents: Benchmark[]; moves: number[] };
export type PopulationData = { version: 1; kind: 'hockey-population'; size: number; config: typeof POPULATION_CONFIG; benchmarkRallies: number; reference: Benchmark; checkpoints: PopulationCheckpoint[]; generatedAt?: string; durationSeconds?: number };
export const defense = (result: Benchmark) => result.shots ? result.returns / result.shots : 0;
export const points = (result: Benchmark) => (result.wins + .5 * result.draws) / (result.wins + result.draws + result.losses || 1);
const greedy = (agent: HockeyAgent, input: number[]) => { const q = agent.network.forward(input).values; return q.indexOf(Math.max(...q)); };

export function benchmark(agent: HockeyAgent | null, arena: HockeyMatch, count = BENCHMARK_RALLIES): Benchmark {
  const result: Benchmark = { wins: 0, draws: 0, losses: 0, returns: 0, shots: 0 };
  // The same held-out serves and balanced side/serve combinations at every check.
  arena.random = randomSource(90421); arena.serves = 0;
  for (let trial = 0; trial < count; trial++) {
    arena.newRally(); const score = createMatchStats();
    const side = (Math.floor(trial / 2) % 2) as Side;
    while (!arena.terminal) {
      const actions: [number, number] = [referenceAction(arena.rink, 0), referenceAction(arena.rink, 1)];
      if (agent) actions[side] = greedy(agent, arena.observation(side));
      arena.advance(actions, score);
    }
    result.returns += score.returns[side]; result.shots += score.shots[side];
    if (arena.winner === null) result.draws++;
    else if (arena.winner === side) result.wins++;
    else result.losses++;
  }
  return result;
}

export class HockeyPopulation {
  agents: HockeyAgent[];
  random: () => number;
  arena = new HockeyMatch();
  evaluator = new HockeyMatch();
  order: number[];
  pair = 0;
  round = 0;
  pairStarted = false;
  activeIds = [0, 1];
  pairScore = createMatchStats();
  data: PopulationData;
  evaluating: PopulationCheckpoint | null = null;
  evaluationIndex = 0;
  constructor(public size = POPULATION_SIZE, seed = 42) {
    if (size < 2 || size % 2) throw new Error('Population must have an even number of agents.');
    this.random = randomSource(seed + 7001);
    this.arena.random = randomSource(seed + 1231);
    this.agents = Array.from({ length: size }, (_, id) => new HockeyAgent('state', seed + 1009 * (id + 1), 1, { memoryLimit: 128, replaySamples: 2, width: 8 }));
    this.order = Array.from({ length: size }, (_, id) => id);
    this.data = { version: 1, kind: 'hockey-population', size, config: { ...POPULATION_CONFIG, seed }, benchmarkRallies: BENCHMARK_RALLIES, reference: benchmark(null, this.evaluator), checkpoints: [] };
    this.shuffle(); this.beginEvaluation();
  }
  shuffle() {
    for (let i = this.order.length - 1; i > 0; i--) { const j = Math.floor(this.random() * (i + 1)); [this.order[i], this.order[j]] = [this.order[j], this.order[i]]; }
  }
  beginEvaluation() {
    this.evaluating = { round: this.round, agents: [], moves: this.agents.map(agent => agent.steps) }; this.evaluationIndex = 0;
  }
  // One small work unit: one benchmark agent or one simultaneous peer-game move.
  // A worker schedules these units and yields between batches.
  work(): boolean {
    if (this.evaluating) {
      this.evaluating.agents.push(benchmark(this.agents[this.evaluationIndex++], this.evaluator));
      if (this.evaluationIndex === this.size) {
        this.data.checkpoints.push(this.evaluating); this.evaluating = null; return true;
      }
      return false;
    }
    if (!this.pairStarted) { this.arena.newRally(); this.pairScore = createMatchStats(); this.pairStarted = true; this.activeIds = this.order.slice(this.pair * 2, this.pair * 2 + 2); }
    const ids = [this.order[this.pair * 2], this.order[this.pair * 2 + 1]];
    const states = [this.arena.observation(0), this.arena.observation(1)];
    const actions = ids.map((id, side) => this.agents[id].choose(states[side], this.data.config.exploration)) as [number, number];
    this.arena.advance(actions, this.pairScore);
    ids.forEach((id, side) => this.agents[id].learn({ state: states[side], action: actions[side], reward: this.arena.rewards[side], next: this.arena.observation(side as Side), done: this.arena.terminal }, this.data.config.rate));
    if (this.arena.terminal) {
      this.pairStarted = false; this.pair++;
      if (this.pair === this.size / 2) {
        this.pair = 0; this.round++; this.shuffle();
        if (this.round % this.data.config.evaluateEvery === 0) this.beginEvaluation();
      }
    }
    return false;
  }
}

export type Trend = 'early' | 'improving' | 'flat' | 'regressing' | 'mixed';
export function agentTrend(checkpoints: PopulationCheckpoint[], index: number, id: number): Trend {
  if (index < 2) return 'early';
  const before = checkpoints[index - 2].agents[id], after = checkpoints[index].agents[id];
  const changes = [defense(after) - defense(before), points(after) - points(before)];
  const up = changes.some(value => value >= .1), down = changes.some(value => value <= -.1);
  return up && down ? 'mixed' : up ? 'improving' : down ? 'regressing' : 'flat';
}