Catch a star

Train a small neural network to position a paddle from coordinates. Adjust learning rate and exploration, then inspect the rewards.

Architecture, equations & code ↓

Ready to learn

The game

8× speed / move 0 of 12

Move toward the star. Catch it before it lands.Training: 0 / 0 catches

Turn the knobs

Changes apply live

Learning rate 0.010

How much each lesson changes the weights. Bigger steps can learn faster—or wobble.

Exploration 25%

Chance of trying a random move instead of the network’s favorite. This stays at your chosen value.

Training speed 8×

More game steps per second. The learning rules stay the same. Slower devices may run below this limit.

Inside the brain

4 inputs → 8 neurons → 3 actions

Lines show current weights: blue positive, purple negative; thicker means stronger. Hidden-node fill shows activation magnitude. Inputs and scores describe the last decision, before its learning update.

Left+0.00Action value

Stay+0.00Action value

Right+0.00Action value

Action values estimate future reward, not probabilities. Higher is better.

Is it learning?

Training only

—Catch rate · last 100

0Rounds completed

—Smoothed training loss

The horizontal axis always starts at round 0. Each point measures up to 100 recent rounds; older checkpoints are thinned to retain the full run. The first measurement appears at round 10. Random exploration can cause misses even after learning. Lower loss alone does not prove better play.

A lesson in four moves

01 / OBSERVE

Where is everything?

The network sees paddle position, star position, time left, and the gap between them. It gets numbers, not pixels.

02 / CHOOSE

Try a move.

Eight neurons turn the inputs into three action values. Pick the highest, or explore with a random move.

03 / GET FEEDBACK

Did that help?

A catch earns +1; a miss earns −1. Each earlier move earns twice the reduction in distance to the star. This extra coaching makes the game easier to learn.

04 / UPDATE

Adjust the connections.

The network revisits past moves and adjusts its weights through backpropagation, making its reward predictions a little less wrong.

Try this

Train for 300 rounds at 50×. Pause, switch on “Watch learned policy,” then resume. Reset and compare 0% exploration with 100%. A fully random player can still learn from experience—but you will only see its learned choices in watch mode.

What exactly runs here?

A real, small Deep Q-Network trained in your browser: 4 inputs, 8 tanh neurons, 3 linear outputs, and 67 trainable weights and biases. It replays 16 experiences per game step from a buffer of up to 2,000. A separate target network is copied every 120 training steps. The discount factor is 0.9; updates use Huber loss and stochastic gradient descent with a per-sample step size of learning rate ÷ 4. Replay starts after 16 moves.

The star falls straight down over 12 moves. The paddle moves 8% of the field per action and catches within 9.5% of the star’s horizontal position. Each round starts at random positions. This is a small, coached environment; it is not a demonstration of learning complex games from images.

Pause preserves the exact game state. Stop starts a fresh round and keeps the network and training history. Reset clears all learning and scores using the same starting seed, while keeping your knob settings. Changing modes starts a fresh round; entering watch mode clears watch scores. Reloading the page clears everything. At high speeds, the display samples the simulation rather than showing every move.

For the underlying method, see the PyTorch DQN tutorial ↗. This experiment implements the math in TypeScript and has no model download or server training.

UNDER THE HOOD

From a falling star to a weight update

Follow one decision through the architecture, the equations, and the actual TypeScript that runs above. Open each step when you want to go deeper.

01 The architecture: 4 → 8 → 3

This is a fully connected feedforward network: every input connects to every hidden neuron, and every hidden neuron connects to every output. One pass produces a score for each possible move.

INPUT4 numbers
Position, time, gap
HIDDEN LAYER8 tanh neurons
Weighted sums + nonlinearity
OUTPUT3 action values
Left · stay · right

The 67 trainable parameters
Part	Shape	Count	In the code
Input → hidden weights	8 × 4	32	`w1`
Hidden biases	8	8	`b1`
Hidden → output weights	3 × 8	24	`w2`
Output biases	3	3	`b2`

(8\times4)+8+(3\times8)+3=67

A weight controls the strength and sign of a connection. A bias shifts a neuron’s weighted sum, even when its inputs are zero. The hidden layer’s tanh activation bends the function so the network can learn nonlinear relationships.

Weights start randomly: first-layer weights between −0.5 and +0.5, second-layer weights between −0.35 and +0.35. Biases start at zero. The hidden layer is deliberately fixed at eight neurons so changing training settings compares the same architecture.

02 Inputs and the forward pass

Let p be the paddle’s horizontal position, s the star’s position, and t the number of moves already taken. Positions use field coordinates from 0 to 1. The state vector is:

\mathbf{x}_t=\begin{bmatrix}2p_t-1\\2s-1\\1-t/12\\s-p_t\end{bmatrix}\in\mathbb{R}^{4}

The first two inputs center positions around zero. The third is the fraction of moves remaining. The signed gap says which side the star is on. That gap is redundant information, but handing it to this tiny network helps it learn.

State encoding · StarTrainer.input

input() {
  return [this.paddle * 2 - 1, this.star * 2 - 1, 1 - this.tick / 12, this.star - this.paddle];
}

\begin{aligned}\mathbf{h}&=\tanh(W_1\mathbf{x}_t+\mathbf{b}_1)\in\mathbb{R}^{8}\\\mathbf{q}&=W_2\mathbf{h}+\mathbf{b}_2\in\mathbb{R}^{3}\end{aligned}

Here h holds the eight hidden activations and q holds the three action values. Tanh keeps each hidden activation between −1 and +1. The output layer is linear: reward estimates may be negative or greater than one, so there is no softmax or conversion into probabilities.

Forward pass · Network.forward

forward(input: number[]) {
  const hidden = this.w1.map((row, j) => Math.tanh(row.reduce((sum, w, i) => sum + w * input[i], this.b1[j])));
  const values = this.w2.map((row, k) => row.reduce((sum, w, j) => sum + w * hidden[j], this.b2[k]));
  return { hidden, values };
}

03 Choosing a move and earning a reward

The exploration slider sets ε (epsilon). With probability ε, choose uniformly from all three actions. Otherwise, choose the highest action value. This is called an epsilon-greedy policy (Sutton & Barto, 2018, §2.2).

a_t=\begin{cases}\text{uniform random action},&\text{with probability }\varepsilon\\\arg\max_a Q_\theta(\mathbf{x}_t,a),&\text{with probability }1-\varepsilon\end{cases}

Q_θ means the value predicted by the network with its current weights and biases, collectively called θ (theta). Actions 0, 1, and 2 mean left, stay, and right. A random choice can still pick the best action; tied best scores choose the first action in this implementation.

Exploration versus the network’s favorite · StarTrainer.step

this.exploring = train && this.random() < settings.exploration;
this.action = this.exploring ? Math.floor(this.random() * 3) : best(prediction.values);

The paddle moves 0.08 field widths per action and is clamped to [0.08, 0.92]. Write d_t = |s − p_t| for its distance from the star before moving:

r_t=\begin{cases}+1,&t=11\text{ and }d_{t+1}\leq0.095\\-1,&t=11\text{ and }d_{t+1}>0.095\\2(d_t-d_{t+1}),&t<11\end{cases}

The final move replaces the distance reward with catch or miss feedback. Earlier moves get positive feedback for closing the gap and negative feedback for widening it. This reward shaping supplies useful coaching before the end of the round.

Movement and reward · StarTrainer.step

const before = Math.abs(this.star - this.paddle);
this.paddle = Math.max(0.08, Math.min(0.92, this.paddle + (this.action - 1) * 0.08));
this.tick++;
const after = Math.abs(this.star - this.paddle);
this.terminal = this.tick === 12;
const caught = after <= 0.095;
// Dense feedback makes early learning visible; terminal success is still
// rewarded separately. This is deliberately a shaped, easy environment.
this.reward = this.terminal ? (caught ? 1 : -1) : 2 * (before - after);

04 Learning from replay and future rewards

Each training move stores an experience: current inputs, chosen action, reward, next inputs, and whether the round ended. The replay buffer holds the latest 2,000 experiences. Once it has 16, each new move triggers 16 random samples with replacement, so an experience can be selected more than once.

(\mathbf{x}_t,\ a_t,\ r_t,\ \mathbf{x}_{t+1},\ \mathrm{done}_t)

For each sampled experience, form a target y: the reward already received, plus a discounted estimate of what could come next. The target network’s parameters are written θ⁻. At the end of a round there is no future reward to add.

y=r_t+\gamma(1-\mathrm{done}_t)\max_{a'}Q_{\theta^-}(\mathbf{x}_{t+1},a'),\qquad\gamma=0.9

The done flag is 1 at the end of a round and 0 otherwise. The discount factor γ (gamma) makes more distant rewards count less. Sampling old moves reduces dependence on the latest trajectory; the delayed target copy helps avoid chasing a target that changes after every weight update. The deep Q-network (DQN) combined both with Q-learning; the history below says where they came from (Sutton & Barto, 2018, §16.5).

Replay, target values, and target-network refresh · StarTrainer.step

if (this.memory.length >= 16) {
  let loss = 0;
  for (let i = 0; i < 16; i++) {
    const sample = this.memory[Math.floor(this.random() * this.memory.length)];
    const target = sample.reward + (sample.done ? 0 : 0.9 * Math.max(...this.target.forward(sample.next).values));
    loss += this.network.train(sample.state, sample.action, target, settings.learningRate / 4);
  }
  this.loss = 0.95 * this.loss + 0.05 * loss / 16;
}
if (this.steps % 120 === 0) this.target.copyFrom(this.network);

These targets are estimates, not labels supplied by an expert player. This is why reinforcement learning can be noisy: the network learns partly from its own evolving predictions. The PyTorch DQN tutorial explains replay and a target network in a larger example; it updates its target network gradually rather than copying it at intervals (PyTorch DQN tutorial, n.d.).

05 Loss, backpropagation, and the weight update

Compare the chosen action’s prediction with its target. Write this error as e. Huber loss is quadratic for small errors and linear for large errors, which limits how strongly one large prediction error drives an update. Peter Huber introduced this function in 1964 for estimates that stay reliable when some observations contain gross errors (Huber, 1964).

\begin{aligned}e&=Q_\theta(\mathbf{x}_t,a_t)-y\\L(e)&=\begin{cases}\tfrac12e^2,&|e|\leq1\\|e|-\tfrac12,&|e|>1\end{cases}\\g&=\frac{dL}{de}=\operatorname{clip}(e,-1,1)\end{aligned}

Backpropagation applies the chain rule to find how each weight contributed to that error. For selected action a, hidden neuron j, and input i:

\begin{aligned}\frac{\partial L}{\partial W_{2,aj}}&=g\,h_j,&\frac{\partial L}{\partial b_{2,a}}&=g\\\delta_j&=g\,W_{2,aj}(1-h_j^2)\\\frac{\partial L}{\partial W_{1,ji}}&=\delta_j x_i,&\frac{\partial L}{\partial b_{1,j}}&=\delta_j\end{aligned}

The factor 1 − h_j² is the derivative of tanh. Hidden gradients must use the output weights before they are updated. Only the selected output row receives a direct gradient; shared hidden-layer changes can still affect every action value.

\theta\leftarrow\theta-\eta\nabla_\theta L,\qquad\eta=\frac{\text{learning-rate slider}}{4}

Each of the 16 replay samples updates the weights immediately. This is 16 sequential SGD updates, rather than a single update from an averaged batch gradient. The displayed slider value is divided by four for each update; at the default 0.01, the per-sample step size is 0.0025. The target is treated as fixed during differentiation.

Huber loss and backpropagation · Network.train

train(input: number[], action: number, target: number, rate: number) {
  const { hidden, values } = this.forward(input);
  const error = values[action] - target;
  const gradient = Math.max(-1, Math.min(1, error));
  // Compute hidden gradients before changing the output weights.
  const hiddenGradient = hidden.map((h, j) => gradient * this.w2[action][j] * (1 - h * h));
  for (let j = 0; j < hidden.length; j++) {
    this.w2[action][j] -= rate * gradient * hidden[j];
    for (let i = 0; i < input.length; i++) this.w1[j][i] -= rate * hiddenGradient[j] * input[i];
    this.b1[j] -= rate * hiddenGradient[j];
  }
  this.b2[action] -= rate * gradient;
  return Math.abs(error) <= 1 ? error * error / 2 : Math.abs(error) - 0.5;
}

06 Reading the charts and experimenting fairly

The loss display is a smoothed mean of the 16 sampled losses from each training move. Because the samples update the network sequentially, these losses are measured at slightly different weights.

\overline L_k=0.95\,\overline L_{k-1}+0.05\left(\frac1{16}\sum_{m=1}^{16}L_{k,m}\right)

Here k counts replay-training passes; the running value starts at zero. Early readings are therefore pulled toward zero. A small loss means sampled reward predictions fit their current targets. It does not guarantee a high catch rate.

Learning rate: compare how quickly the catch rate improves and how much it wobbles. A higher setting makes larger parameter updates.
Exploration: random actions create varied experience but can reduce the training catch rate. Exploration stays fixed at your chosen value; there is no automatic decay.
Speed: changes how many game moves run per second. Compare runs after the same number of completed training rounds, not after the same wall-clock time.
Watch mode: disables both exploration and weight updates. Its separate catch count is a clearer check of the learned policy.

Reset gives you the same initial seed and clears the network, buffer, and scores while retaining your slider settings. Set the controls before starting each comparison. Treat any single result cautiously: changing exploration also changes the random sequence of later experiences.

</> Read the complete training engine

This is the source used by the simulation, included directly at build time.

star-training.ts · complete source

// A small DQN: tanh hidden layer, linear action values, experience replay,
// a delayed target network, and Huber-loss stochastic gradient descent.
export const ACTIONS = ['Left', 'Stay', 'Right'] as const;
export const HIDDEN = 8;
export const DEFAULTS = { learningRate: 0.01, exploration: 0.25, speed: 8 };
export type Settings = typeof DEFAULTS;
type Experience = { state: number[]; action: number; reward: number; next: number[]; done: boolean };
export function randomSource(seed: number) {
  return () => {
    seed |= 0; seed = seed + 0x6D2B79F5 | 0;
    let t = Math.imul(seed ^ seed >>> 15, 1 | seed);
    t = t + Math.imul(t ^ t >>> 7, 61 | t) ^ t;
    return ((t ^ t >>> 14) >>> 0) / 4294967296;
  };
}
const best = (values: number[]) => values.indexOf(Math.max(...values));

export class Network {
  w1: number[][];
  b1: number[];
  w2: number[][];
  b2 = [0, 0, 0];
  constructor(random = Math.random, inputs = 4, hiddenSize = HIDDEN) {
    this.b1 = Array(hiddenSize).fill(0);
    this.w1 = Array.from({ length: hiddenSize }, () => Array.from({ length: inputs }, () => (random() * 2 - 1) * 0.5));
    this.w2 = Array.from({ length: 3 }, () => Array.from({ length: hiddenSize }, () => (random() * 2 - 1) * 0.35));
  }
  forward(input: number[]) {
    const hidden = this.w1.map((row, j) => Math.tanh(row.reduce((sum, w, i) => sum + w * input[i], this.b1[j])));
    const values = this.w2.map((row, k) => row.reduce((sum, w, j) => sum + w * hidden[j], this.b2[k]));
    return { hidden, values };
  }
  copyFrom(other: Network) {
    this.w1 = other.w1.map(row => [...row]); this.w2 = other.w2.map(row => [...row]);
    this.b1 = [...other.b1]; this.b2 = [...other.b2];
  }
  train(input: number[], action: number, target: number, rate: number) {
    const { hidden, values } = this.forward(input);
    const error = values[action] - target;
    const gradient = Math.max(-1, Math.min(1, error));
    // Compute hidden gradients before changing the output weights.
    const hiddenGradient = hidden.map((h, j) => gradient * this.w2[action][j] * (1 - h * h));
    for (let j = 0; j < hidden.length; j++) {
      this.w2[action][j] -= rate * gradient * hidden[j];
      for (let i = 0; i < input.length; i++) this.w1[j][i] -= rate * hiddenGradient[j] * input[i];
      this.b1[j] -= rate * hiddenGradient[j];
    }
    this.b2[action] -= rate * gradient;
    return Math.abs(error) <= 1 ? error * error / 2 : Math.abs(error) - 0.5;
  }
}

export class StarTrainer {
  random: () => number;
  network: Network;
  target: Network;
  paddle = 0.5;
  star = 0.5;
  tick = 0;
  terminal = false;
  rounds = 0;
  catches = 0;
  steps = 0;
  loss = 0;
  reward = 0;
  action = 1;
  exploring = false;
  lastInput = [0, 0, 1, 0];
  lastHidden = Array(HIDDEN).fill(0) as number[];
  lastValues = [0, 0, 0];
  recent: number[] = [];
  history: { round: number; rate: number }[] = [];
  historyStride = 10;
  memory: Experience[] = [];
  memoryCursor = 0;
  evaluation = { rounds: 0, catches: 0 };
  constructor(seed = 42) {
    this.random = randomSource(seed);
    this.network = new Network(this.random);
    this.target = new Network(this.random);
    this.target.copyFrom(this.network);
    this.newRound();
  }
  newRound() {
    this.paddle = 0.1 + this.random() * 0.8;
    this.star = 0.1 + this.random() * 0.8;
    this.tick = 0; this.terminal = false;
  }
  input() {
    return [this.paddle * 2 - 1, this.star * 2 - 1, 1 - this.tick / 12, this.star - this.paddle];
  }
  get catchRate() { return this.recent.length ? this.recent.reduce((a, b) => a + b, 0) / this.recent.length : 0; }
  step(settings: Settings, train = true) {
    if (this.terminal) this.newRound();
    const state = this.input();
    const prediction = this.network.forward(state);
    this.lastInput = state; this.lastHidden = prediction.hidden; this.lastValues = prediction.values;
    this.exploring = train && this.random() < settings.exploration;
    this.action = this.exploring ? Math.floor(this.random() * 3) : best(prediction.values);
    const before = Math.abs(this.star - this.paddle);
    this.paddle = Math.max(0.08, Math.min(0.92, this.paddle + (this.action - 1) * 0.08));
    this.tick++;
    const after = Math.abs(this.star - this.paddle);
    this.terminal = this.tick === 12;
    const caught = after <= 0.095;
    // Dense feedback makes early learning visible; terminal success is still
    // rewarded separately. This is deliberately a shaped, easy environment.
    this.reward = this.terminal ? (caught ? 1 : -1) : 2 * (before - after);
    if (train) {
      this.steps++;
      const experience = { state, action: this.action, reward: this.reward, next: this.input(), done: this.terminal };
      if (this.memory.length < 2000) this.memory.push(experience);
      else { this.memory[this.memoryCursor] = experience; this.memoryCursor = (this.memoryCursor + 1) % 2000; }
      if (this.memory.length >= 16) {
        let loss = 0;
        for (let i = 0; i < 16; i++) {
          const sample = this.memory[Math.floor(this.random() * this.memory.length)];
          const target = sample.reward + (sample.done ? 0 : 0.9 * Math.max(...this.target.forward(sample.next).values));
          loss += this.network.train(sample.state, sample.action, target, settings.learningRate / 4);
        }
        this.loss = 0.95 * this.loss + 0.05 * loss / 16;
      }
      if (this.steps % 120 === 0) this.target.copyFrom(this.network);
    }
    if (this.terminal) {
      if (train) {
        this.rounds++; this.catches += Number(caught);
        this.recent.push(Number(caught)); if (this.recent.length > 100) this.recent.shift();
        if (this.rounds % this.historyStride === 0) {
          this.history.push({ round: this.rounds, rate: this.catchRate });
          // Preserve the beginning and full time span, thinning checkpoints
          // instead of discarding the start of a long training run.
          if (this.history.length > 120) {
            this.history = this.history.filter((_, i) => i % 2 === 0);
            this.historyStride *= 2;
          }
        }
      } else { this.evaluation.rounds++; this.evaluation.catches += Number(caught); }
    }
  }
}

History of the method

Q-learning comes from Christopher Watkins’s 1989 PhD thesis. Watkins and Dayan later proved that it converges to the optimal action values with probability 1, provided every action keeps being tried in every state and the values are stored discretely, one entry per state and action (Watkins & Dayan, 1992). They also point out that Watkins had shown it may not converge correctly with other representations. A neural network is one of those: this lesson has no convergence guarantee. Function approximation, bootstrapped targets and training on replayed off-policy data can together make learning unstable, and this lesson combines all three (Sutton & Barto, 2018, §11.3).

Experience replay was first studied by Long-Ji Lin in 1992 (Sutton & Barto, 2018, §16.5). Mnih and colleagues combined it with Q-learning and a convolutional network that learned directly from Atari screen pixels (Mnih et al., 2013). The 2015 deep Q-network (DQN) learned 49 Atari games with one architecture and set of settings (Mnih et al., 2015). It also added a target network, copied from the learning network after a fixed number of updates, and clipped the error term to [−1, 1] (Sutton & Barto, 2018, §16.5). That clipping gives the same gradient as the Huber loss with threshold 1 used here.

This lesson keeps those three pieces at toy scale. DQN used minibatches of 32 transitions, the RMSProp optimizer and exploration that decreased over the first million frames (Sutton & Barto, 2018, §16.5). Here there are 67 parameters, four hand-designed inputs, distance-based coaching rewards, fixed exploration and one plain gradient step per sampled experience. The pixel lesson follows the history from hand-designed inputs to pixels.

References and further reading

References

Peter J. Huber (1964). Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics 35(1), 73–101.Abstract available · Read: Project Euclid abstract, which states the most robust M-estimator’s ρ function · Checked 2026-09-13
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra & Martin Riedmiller (2013). Playing Atari with Deep Reinforcement Learning. arXiv:1312.5602 [cs.LG].Open access · Read: Abstract, §1 introduction and §4 deep reinforcement learning (Algorithm 1) · Checked 2026-09-13
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg & Demis Hassabis (2015). Human-level control through deep reinforcement learning. Nature 518(7540), 529–533.Publisher record; full text may require access · Read: Publisher page: summary, abstract and opening paragraphs; method details as described in Sutton & Barto (2018) §16.5 · Checked 2026-09-13
Adam Paszke & Mark Towers (n.d.). Reinforcement Learning (DQN) Tutorial. PyTorch Tutorials.Open access · Read: Existing lesson link; tutorial text on replay memory, Huber loss, epsilon-greedy selection and its soft target-network update · Checked 2026-09-13
Richard S. Sutton & Andrew G. Barto (2018). Reinforcement Learning: An Introduction. MIT Press, 2nd ed.Library record · Read: Authors’ free PDF of the second edition (2020 printing, incompleteideas.net): §2.2 action-value methods, §6.5 Q-learning, §11.3 the deadly triad, §16.1 TD-Gammon, §16.2 Samuel’s checkers player, §16.5 human-level video game play · Checked 2026-09-13
Christopher J. C. H. Watkins & Peter Dayan (1992). Q-learning. Machine Learning 8(3–4), 279–292.Publisher record; full text may require access · open copy · Read: Author copy: abstract, introduction and statement of the convergence conditions · Checked 2026-09-13