↓ Skip to main content
  1. Posts/

I Bet You Can't Crash This Rocket

Author
Daniel Kawalsky
Learning Reinforcement Learning - This article is part of a series.
Part : This Article

Last time, I bet that you couldn’t land this rocket. If you tried it on “random” mode, chances are, you crashed.

So let’s flip it. This time, you’re not the pilot. I trained an AI model to fly it. Your job is to make it crash.

Try to crash it
#

Press anywhere in the sky, pull back, and let go. The rocket launches the opposite way, faster the further you pull. Then my pilot takes over. Throw as many as you like; up to twelve can be in the air at once.

Open the slingshot on its own

You’ll notice a blue box while you aim. That’s the fine print. More on it below.

What changed
#

The last post ended with a plan: teach it to hover first, then land. That went to plan, and then kept going.

The pilot now lands rockets thrown at it, even if they’re falling fast and tipped as far as 70°. It brakes, rights itself, drifts over the pad and sets down gently enough to pass the same touchdown checks you were held to.

Before it flew a single one of your rockets, it practiced about 61,000 flights. That’s roughly 111 hours in the air. How close could you have gotten to landing this rocket after 111 hours of practice?

How it learned
#

There are three pieces.

  1. The world: Gymnasium. Gymnasium is a toolkit for describing a task so a learning algorithm can practice it. Mine spells out what the pilot can sense (its position, speed, tilt, spin and the last command it gave), what it controls (throttle and engine angle, ten times a second) and what counts as a landing or a crash.

  2. The practice: PPO. PPO learns by trial and error. The pilot flies, earns reward for making progress toward a slow, upright landing over the pad, and loses reward for crashing or dawdling. After each batch of flights it nudges itself toward whatever earned more. I ran eight simulations side by side to speed things up, which I’d filed under “eventually” last time.

  3. The ladder. I didn’t start with the hard version. I built the challenge up in stages: first hovering, then landing from nearby, then from far away, then thrown in hard, with heavier and lighter rockets, weaker and stronger engines, and sudden gusts of wind. Each stage turned up a problem to fix before the next. The final pilot practiced on a mix of all of it, from gentle drops right above the pad to hard throws from the edge of the sky.

It’s flying in your browser
#

Nothing here talks to a server. The pilot is a small neural network, about 5,000 numbers in a 98 KB file, and your browser runs it ten times a second for every rocket in the air. The physics is the same engine it trained in, Box2D, running in JavaScript through Planck.js.

To make sure nothing got lost in translation, I flew the same 80 test flights in the browser and in the original Python. Every one ended the same way, and the two versions of each rocket never drifted more than a thousandth of a unit apart.

The fine print
#

When you pull back, a blue box shows the throws the pilot practiced: mostly aimed at the pad, not away from it. Pull outside the box and the aim turns gray. The rocket still goes the way you aimed, just slower.

The rocket you’re throwing is the standard one, in calm air. In training I also threw in wind and the full range of rocket weights and engine strengths, and there the pilot does crash now and then, usually when a heavy rocket is paired with a weak engine. Sometimes that combination just can’t brake in time, whoever is flying. (More on that next time.)

Anyways, if you do manage to crash it, send me a screenshot!

Next time
#

Getting here wasn’t smooth. At one point the pilot decided that hovering forever was safer than landing, and at another it flew a rocket differently from that rocket’s mirror image. Some rockets turned out to be impossible to save at all. In the next post I’ll walk through what went wrong and how we fixed it, including the details I stopped checking once I started directing this project at a higher level.

Learning Reinforcement Learning - This article is part of a series.
Part : This Article

Related