Catch a star from pixels

Train a small neural network from grayscale frames. Compare rewards and performance as training proceeds.

Architecture, equations & code ↓

Ready to learn

The game

8× speed / move 0 of 12

Catch the bright pixel with the paddle.

Move toward the star. Catch it before it lands.Training: 0 / 0 catches

Turn the knobs

Changes apply live

Learning rate 0.010

How much each lesson changes the weights. Bigger steps can learn faster—or wobble.

Exploration 25%

Chance of trying a random move instead of the network’s favorite. This stays at your chosen value.

Training speed 8×

More game steps per second. The learning rules stay the same. Slower devices may run below this limit.

What the brain sees

168 pixels → 24 neurons → 3 actions

Last decision’s frame

24 hidden activations

Teal: positive. Purple: negative. Stronger color: larger activation.

Each cell is a brightness value from 0 to 1. All 168 values enter the network, row by row. No paddle position, star position, time, or gap is supplied as a separate input.

Left-0.22Action value

Stay-0.40Action value

Right+0.29Action value

Action values estimate future reward, not probabilities. Higher is better.

Is it learning?

Training only

—Catch rate · last 100

0Rounds completed

—Smoothed training loss

The horizontal axis always starts at round 0. Each point measures up to 100 recent rounds; older checkpoints are thinned to retain the full run. The first measurement appears at round 10. Random exploration can cause misses even after learning. Lower loss alone does not prove better play.

Try learning without coordinates

Give it more practice

Start at 50× and compare watch-mode catches after 1,000 and 3,000 training rounds. This network starts from random weights and has 4,131 parameters to learn. Results may wobble; watch mode freezes the weights and removes random exploration.

Pause keeps the current frame. Stop starts a fresh round and keeps learning. Reset clears the brain and scores with the same initial seed, keeping your settings. The game still supplies coaching rewards for moving closer; only the network’s observation has changed.

LESSON 02 / UNDER THE HOOD

What changes when the inputs are pixels?

In lesson 01, the brain gets four useful numbers. Here, it must learn from brightness across a small image. The movement, rewards, and Q-learning algorithm stay the same.

01 / The image is the observation

The game is rendered into a 12-column × 14-row grayscale framebuffer. Black is 0 and white is 1. The star occupies a row near the top and falls one row per move; the paddle stays on the bottom row. Horizontal pixel brightness accounts for fractional coverage, so moving less than one pixel still changes the image.

I_t\in[0,1]^{14\times12},\qquad x_{12r+c}=I_t[r,c],\qquad\mathbf{x}_t\in\mathbb{R}^{168}

Rows r and columns c are indexed from zero. Flattening simply reads rows left to right, top to bottom. It does not detect objects or calculate their positions. These 168 brightness values are the network’s entire observation.

The renderer necessarily uses game coordinates to draw the scene, just as any game engine does. The policy only receives the resulting image. The reward function still uses distance to provide the same coaching as lesson 01; this is not learning without engineered rewards.

One frame is enough for this game: the star always falls straight down, and its row indicates time remaining. Games with changing velocities often need multiple frames. On the terminal frame the star stays on row 11; terminal inputs are not used to choose another action or bootstrap a future reward.

02 / A bigger, fully connected brain

OBSERVE12 × 14 image

168 brightness values

LEARN FEATURES24 tanh neurons

Every pixel connects to each neuron

ACT3 action values

Left · stay · right

\begin{aligned}\mathbf{h}&=\tanh(W_1\mathbf{x}+\mathbf{b}_1),&W_1&\in\mathbb{R}^{24\times168}\\\mathbf{q}&=W_2\mathbf{h}+\mathbf{b}_2,&W_2&\in\mathbb{R}^{3\times24}\end{aligned}

(168\times24)+24+(24\times3)+3=4\,131

Compare that with 67 parameters in the coordinate lesson: about 62 times as many. Each hidden neuron can learn a different weighted combination of pixels. Tanh lets these combinations interact nonlinearly before the three output scores are computed.

This is a multilayer perceptron, not a convolutional neural network (CNN). Flattening retains each pixel’s fixed index but supplies no built-in rule that neighboring pixels are related. A pattern at a new location activates different weights. A convolutional layer reuses the same small set of weights across the image, so a shifted input produces a correspondingly shifted response (Goodfellow et al., 2016, §9.2). The tiny, fixed scene makes this manageable in a browser; larger images would benefit from a different architecture.

03 / The same rewards and learning rule

The network chooses its highest-valued move, except when the exploration setting asks for a random action. The paddle moves 8% of the field per step. A catch within 9.5% horizontal distance earns +1, a miss earns −1, and earlier moves earn twice the decrease in distance to the star.

y=r+0.9(1-\mathrm{done})\max_{a'}Q_{\theta^-}(\operatorname{vec}(I_{t+1}),a')

The target combines immediate feedback with estimated future reward from the delayed target network. The terminal flag removes the future term when the round ends. The loss compares the selected action’s value with that target:

\begin{aligned}e&=Q_\theta(\operatorname{vec}(I_t),a)-y\\L(e)&=\begin{cases}\tfrac12e^2,&|e|\leq1\\|e|-\tfrac12,&|e|>1\end{cases}\\\theta&\leftarrow\theta-\eta\nabla_\theta L,\quad\eta=\text{learning rate}/4\end{aligned}

Each training move stores two pixel frames in the replay experience. After the first 16 moves, training samples 16 experiences with replacement and applies 16 sequential gradient updates. The buffer keeps at most 2,000 experiences. These are actual weight updates from the game’s rewards, with no prerecorded policy or correct-action labels.

The first lesson’s walkthrough derives the forward pass and backpropagation step by step. The PyTorch DQN tutorial provides another implementation of replay and a target network, with a gradual target update instead of periodic copies (PyTorch DQN tutorial, n.d.).

04 / Compare the lessons

Reset the pixel brain. Run watch mode for a few rounds to see how it behaves before learning.
Switch watch mode off, reset again, and train at 50× for 1,000 rounds. Pause and enter watch mode to evaluate without random exploration.
Continue training toward 3,000 rounds and evaluate again. Use several hundred watch rounds when you want a less noisy catch-rate estimate.
Compare with the coordinate lesson at the same training-round counts. Keep learning rate and exploration equal, and remember that the architectures differ too.

The pixel model has more weights to learn, no explicit gap feature, and more computation per update. A low training catch rate can also come from exploration, so use the separate watch-mode score when assessing its learned behavior. Improvement can be uneven; a good run does not guarantee the next one.

The speed slider sets a maximum, not a guarantee. Training yields regularly to keep the page responsive, so slower devices may complete fewer moves per second. Reset returns to the same initial seed and keeps your controls; refreshing clears everything.

05 / Pixel rendering and model setup: the real code

The pixel trainer inherits the same environment and training loop as lesson 01. It replaces the observation and allocates a larger network. This source is included directly from the running implementation.

pixel-training.ts

import { Network, StarTrainer } from './star-training';

export const WIDTH = 12;
export const HEIGHT = 14;
export const PIXEL_HIDDEN = 24;

// Render the game into a grayscale framebuffer. Fractional coverage smooths
// motion below one pixel; no coordinates are appended to the observation.
export function renderPixels(paddle: number, star: number, tick: number) {
  const pixels = Array(WIDTH * HEIGHT).fill(0) as number[];
  const starX = star * (WIDTH - 1);
  const row = Math.min(11, tick);
  const left = Math.floor(starX);
  const fraction = starX - left;
  pixels[row * WIDTH + left] = 1 - fraction;
  if (left + 1 < WIDTH) pixels[row * WIDTH + left + 1] = fraction;
  const paddleX = paddle * (WIDTH - 1);
  const halfWidth = 0.095 * (WIDTH - 1);
  for (let x = 0; x < WIDTH; x++) {
    // Area covered by the paddle in this pixel, at 70% brightness.
    const coverage = Math.max(0, Math.min(x + 0.5, paddleX + halfWidth) - Math.max(x - 0.5, paddleX - halfWidth));
    pixels[(HEIGHT - 1) * WIDTH + x] = coverage * 0.7;
  }
  return pixels;
}

export class PixelTrainer extends StarTrainer {
  constructor(seed = 42) {
    super(seed);
    this.network = new Network(this.random, WIDTH * HEIGHT, PIXEL_HIDDEN);
    this.target = new Network(this.random, WIDTH * HEIGHT, PIXEL_HIDDEN);
    this.target.copyFrom(this.network);
    this.lastInput = this.input();
    const prediction = this.network.forward(this.lastInput);
    this.lastHidden = prediction.hidden;
    this.lastValues = prediction.values;
  }
  input() {
    return renderPixels(this.paddle, this.star, this.tick);
  }
}

06 / Shared network, backpropagation, and game loop

Network dimensions are configurable; their defaults remain the four-input, eight-neuron model used in lesson 01. Every policy decision, replay sample, and gradient update in the pixel lesson uses the overridden pixel observation.

star-training.ts

// A small DQN: tanh hidden layer, linear action values, experience replay,
// a delayed target network, and Huber-loss stochastic gradient descent.
export const ACTIONS = ['Left', 'Stay', 'Right'] as const;
export const HIDDEN = 8;
export const DEFAULTS = { learningRate: 0.01, exploration: 0.25, speed: 8 };
export type Settings = typeof DEFAULTS;
type Experience = { state: number[]; action: number; reward: number; next: number[]; done: boolean };
export function randomSource(seed: number) {
  return () => {
    seed |= 0; seed = seed + 0x6D2B79F5 | 0;
    let t = Math.imul(seed ^ seed >>> 15, 1 | seed);
    t = t + Math.imul(t ^ t >>> 7, 61 | t) ^ t;
    return ((t ^ t >>> 14) >>> 0) / 4294967296;
  };
}
const best = (values: number[]) => values.indexOf(Math.max(...values));

export class Network {
  w1: number[][];
  b1: number[];
  w2: number[][];
  b2 = [0, 0, 0];
  constructor(random = Math.random, inputs = 4, hiddenSize = HIDDEN) {
    this.b1 = Array(hiddenSize).fill(0);
    this.w1 = Array.from({ length: hiddenSize }, () => Array.from({ length: inputs }, () => (random() * 2 - 1) * 0.5));
    this.w2 = Array.from({ length: 3 }, () => Array.from({ length: hiddenSize }, () => (random() * 2 - 1) * 0.35));
  }
  forward(input: number[]) {
    const hidden = this.w1.map((row, j) => Math.tanh(row.reduce((sum, w, i) => sum + w * input[i], this.b1[j])));
    const values = this.w2.map((row, k) => row.reduce((sum, w, j) => sum + w * hidden[j], this.b2[k]));
    return { hidden, values };
  }
  copyFrom(other: Network) {
    this.w1 = other.w1.map(row => [...row]); this.w2 = other.w2.map(row => [...row]);
    this.b1 = [...other.b1]; this.b2 = [...other.b2];
  }
  train(input: number[], action: number, target: number, rate: number) {
    const { hidden, values } = this.forward(input);
    const error = values[action] - target;
    const gradient = Math.max(-1, Math.min(1, error));
    // Compute hidden gradients before changing the output weights.
    const hiddenGradient = hidden.map((h, j) => gradient * this.w2[action][j] * (1 - h * h));
    for (let j = 0; j < hidden.length; j++) {
      this.w2[action][j] -= rate * gradient * hidden[j];
      for (let i = 0; i < input.length; i++) this.w1[j][i] -= rate * hiddenGradient[j] * input[i];
      this.b1[j] -= rate * hiddenGradient[j];
    }
    this.b2[action] -= rate * gradient;
    return Math.abs(error) <= 1 ? error * error / 2 : Math.abs(error) - 0.5;
  }
}

export class StarTrainer {
  random: () => number;
  network: Network;
  target: Network;
  paddle = 0.5;
  star = 0.5;
  tick = 0;
  terminal = false;
  rounds = 0;
  catches = 0;
  steps = 0;
  loss = 0;
  reward = 0;
  action = 1;
  exploring = false;
  lastInput = [0, 0, 1, 0];
  lastHidden = Array(HIDDEN).fill(0) as number[];
  lastValues = [0, 0, 0];
  recent: number[] = [];
  history: { round: number; rate: number }[] = [];
  historyStride = 10;
  memory: Experience[] = [];
  memoryCursor = 0;
  evaluation = { rounds: 0, catches: 0 };
  constructor(seed = 42) {
    this.random = randomSource(seed);
    this.network = new Network(this.random);
    this.target = new Network(this.random);
    this.target.copyFrom(this.network);
    this.newRound();
  }
  newRound() {
    this.paddle = 0.1 + this.random() * 0.8;
    this.star = 0.1 + this.random() * 0.8;
    this.tick = 0; this.terminal = false;
  }
  input() {
    return [this.paddle * 2 - 1, this.star * 2 - 1, 1 - this.tick / 12, this.star - this.paddle];
  }
  get catchRate() { return this.recent.length ? this.recent.reduce((a, b) => a + b, 0) / this.recent.length : 0; }
  step(settings: Settings, train = true) {
    if (this.terminal) this.newRound();
    const state = this.input();
    const prediction = this.network.forward(state);
    this.lastInput = state; this.lastHidden = prediction.hidden; this.lastValues = prediction.values;
    this.exploring = train && this.random() < settings.exploration;
    this.action = this.exploring ? Math.floor(this.random() * 3) : best(prediction.values);
    const before = Math.abs(this.star - this.paddle);
    this.paddle = Math.max(0.08, Math.min(0.92, this.paddle + (this.action - 1) * 0.08));
    this.tick++;
    const after = Math.abs(this.star - this.paddle);
    this.terminal = this.tick === 12;
    const caught = after <= 0.095;
    // Dense feedback makes early learning visible; terminal success is still
    // rewarded separately. This is deliberately a shaped, easy environment.
    this.reward = this.terminal ? (caught ? 1 : -1) : 2 * (before - after);
    if (train) {
      this.steps++;
      const experience = { state, action: this.action, reward: this.reward, next: this.input(), done: this.terminal };
      if (this.memory.length < 2000) this.memory.push(experience);
      else { this.memory[this.memoryCursor] = experience; this.memoryCursor = (this.memoryCursor + 1) % 2000; }
      if (this.memory.length >= 16) {
        let loss = 0;
        for (let i = 0; i < 16; i++) {
          const sample = this.memory[Math.floor(this.random() * this.memory.length)];
          const target = sample.reward + (sample.done ? 0 : 0.9 * Math.max(...this.target.forward(sample.next).values));
          loss += this.network.train(sample.state, sample.action, target, settings.learningRate / 4);
        }
        this.loss = 0.95 * this.loss + 0.05 * loss / 16;
      }
      if (this.steps % 120 === 0) this.target.copyFrom(this.network);
    }
    if (this.terminal) {
      if (train) {
        this.rounds++; this.catches += Number(caught);
        this.recent.push(Number(caught)); if (this.recent.length > 100) this.recent.shift();
        if (this.rounds % this.historyStride === 0) {
          this.history.push({ round: this.rounds, rate: this.catchRate });
          // Preserve the beginning and full time span, thinning checkpoints
          // instead of discarding the start of a long training run.
          if (this.history.length > 120) {
            this.history = this.history.filter((_, i) => i % 2 === 0);
            this.historyStride *= 2;
          }
        }
      } else { this.evaluation.rounds++; this.evaluation.catches += Number(caught); }
    }
  }
}

History: from hand-designed inputs to pixels

Before the deep Q-network, most successful applications of reinforcement learning depended on input features designed by people who knew the problem (Sutton & Barto, 2018, §16.5). Gerald Tesauro’s backgammon program shows the trade-off. TD-Gammon 0.0 received an essentially raw description of the board and learned to play about as well as the best earlier backgammon programs. Adding specialized backgammon features produced TD-Gammon 1.0, which was substantially stronger (Tesauro, 1995) (Sutton & Barto, 2018, §16.1).

Mnih and colleagues trained a convolutional network with a variant of Q-learning to choose actions from raw Atari screen pixels, using the same architecture and learning algorithm for seven games (Mnih et al., 2013). The 2015 deep Q-network extended this to 49 games (Mnih et al., 2015). It still preprocessed the screen: each frame was reduced to an 84 × 84 array of luminance values, and the four most recent frames were stacked because a single frame does not show the full state of many games. Three convolutional layers and one fully connected hidden layer then produced one value per action (Sutton & Barto, 2018, §16.5).

This lesson is far smaller: one 12 × 14 frame, one fully connected hidden layer and no convolution. The rewards are still computed from game coordinates, so the network learns from pixels while being coached with information it cannot see. The first lesson’s history covers Q-learning, replay and target networks; the pixel hockey lesson stacks two frames so moving objects can be detected.

References

Ian Goodfellow, Yoshua Bengio & Aaron Courville (2016). Deep Learning. MIT Press, 1st ed.Open access · Read: Authors’ online edition: chapter 6 (deep feedforward networks), including §6.3 hidden units, §6.4.1 universal approximation and depth, §6.5 back-propagation and §6.6 historical notes; chapter 8 §8.2.5 long-term dependencies and §8.4 parameter initialization; chapter 9 §9.2 motivation for convolution · Checked 2026-09-13
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra & Martin Riedmiller (2013). Playing Atari with Deep Reinforcement Learning. arXiv:1312.5602 [cs.LG].Open access · Read: Abstract, §1 introduction and §4 deep reinforcement learning (Algorithm 1) · Checked 2026-09-13
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg & Demis Hassabis (2015). Human-level control through deep reinforcement learning. Nature 518(7540), 529–533.Publisher record; full text may require access · Read: Publisher page: summary, abstract and opening paragraphs; method details as described in Sutton & Barto (2018) §16.5 · Checked 2026-09-13
Adam Paszke & Mark Towers (n.d.). Reinforcement Learning (DQN) Tutorial. PyTorch Tutorials.Open access · Read: Existing lesson link; tutorial text on replay memory, Huber loss, epsilon-greedy selection and its soft target-network update · Checked 2026-09-13
Richard S. Sutton & Andrew G. Barto (2018). Reinforcement Learning: An Introduction. MIT Press, 2nd ed.Library record · Read: Authors’ free PDF of the second edition (2020 printing, incompleteideas.net): §2.2 action-value methods, §6.5 Q-learning, §11.3 the deadly triad, §16.1 TD-Gammon, §16.2 Samuel’s checkers player, §16.5 human-level video game play · Checked 2026-09-13
Gerald Tesauro (1995). Temporal difference learning and TD-Gammon. Communications of the ACM 38(3), 58–68.Publisher record; full text may require access · Read: Crossref bibliographic record; content as described in Sutton & Barto (2018) §16.1 · Checked 2026-09-13