Training an agent takes millions of environment steps. Your environment is a Python class with a step() method that advances one instance per call, so those millions of calls happen one at a time, and the environment ends up costing you more wall-clock time than the GPU does. You can rewrite it to hold ten thousand instances in tensors and advance them all in one set of operations. On the two environments on this page that buys 13× and 92×, depending on how much work each step does. It also gives you code you can't read, and it can drift away from the environment you originally wrote without your training curve ever looking wrong. This page shows how to keep both and prove they still agree. The demos below run live in your browser.
It's an 8×8 grid with six berries on it, worth 0.10, 0.15 or 0.30 energy each. You start with 1.0 energy and lose 0.02 every step, so you have fifty steps if you don't eat. Each turn you choose one of four directions, move a square, and eat whatever berry is there.
15% of the time your move gets rotated 90° clockwise before it happens, so you aim north and go east. That's the only random thing in the environment.
Press Step a few times. It plays itself with random moves.
That's the whole environment. Teaching an agent to play it takes a few million steps, and you'll want ten thousand copies of that board running at once rather than one, because that's what keeps a GPU busy.
So your environment has to do more than work. Ten thousand copies of it have to run fast.
Training alternates between two jobs. The optimizer updates weights on the GPU, which is fast. The environment produces the experience the optimizer learns from, in Python, one step per call, which isn't. Run that a few million times and the environment is what you're waiting on, not the hardware you paid for.
Rewrite it so every copy advances in one tensor operation and the environment stops being the thing you wait on. These are the numbers from the two example environments in the repo:
At 8192 instances, which is the ten thousand the rest of this page keeps rounding to, toywalk's batched version does ninety-two times what its plain Python one does, and the curve hasn't flattened yet. Toggle to forager and the same rewrite buys 13×, because a forager step does more work on both sides. The multiplier depends on the environment; both curves are still climbing at the right edge of the chart.
Look at N = 1 though. The batched version is slower there, 13,567 steps/s against 385,199. You're not sprinkling speed on top of what you had; you're making a trade that doesn't pay off until you're running a few hundred copies at once. If you only ever run one environment at a time, the version you already have is the fast one.
Your original has one agent and an if. The batched one has ten thousand agents and no branches at all, because you can't take a branch per instance when they're all in the same tensor. Every if you had becomes arithmetic that works out both answers and throws one away.
# spec: slip negates the move if rng.draw_bernoulli( self.key, state.t, Slots.SLIP, SLIP_P): delta = -delta # spec: clamp to the line position = state.position + delta if position > L: position = L if position < -L: position = -L
# spec: slip negates the move slip = rng.draw_bernoulli_torch( self.keys, self.t, Slots.SLIP, SLIP_P) delta = torch.where(slip, -delta, delta) # spec: clamp to the line self.position = torch.clamp( self.position + delta, -L, L)
What most people do next is write the batched version, check that it looks right, and delete the original. It feels safe because you tested it, but the thing you tested it against was itself.
And a subtly wrong environment still gives you a training curve that goes down. Your agent does get better at something. You just don't find out it was the wrong something until much later, if you find out at all.
Write the environment twice. Once so you can read it, with explicit ifs, one instance, every line traceable back to a rule. Once so ten thousand copies can run it. You write both from a single spec.md, and never from each other.
Then you let a machine check, step by step, that the two produce bit-identical states, rewards, terminations and observations. If they agree, either you got the spec right twice or you made the same mistake twice in two completely different programming styles. The second is a lot less likely than getting it right once by luck.
You end up with a readable version you can still debug against, reason about and explain your results with, and a batched version that is provably the same environment. You're not giving up one for the other, you're keeping the first and adding the second.
Three things have to work for that to hold: the two versions have to draw identical randomness, a test has to compare them step by step, and a second test has to catch the bugs that only appear in a crowd. The rest of this page is those three, each one running live.
One limit worth stating up front: agreement proves the two programs implement the same spec, not that the spec describes the game you meant to build. A wrong rule, written down clearly, passes every test on this page twice.
You don't hand-roll any of this per project. It's a Python package: you install it from the repo, run simulacrum new myenv to get the skeleton, and fill in four files.
myenv/ spec.md you the single source of truth schema.json you what a serialized state looks like reference.py you readable, one instance fast.py you batched tensors render.py you optional, for playback tests/ conftest.py you ~20 lines naming your two factories test_battery.py from simulacrum.harness.battery import *
This is what kills most attempts before they get anywhere. Your readable version draws randomness one instance at a time. Your batched version draws for ten thousand at once and throws most of them away. If you're using a normal random generator, the kind that keeps a position in a stream, the draws you threw away still moved the stream forward, and your two programs come apart permanently on step one.
So simulacrum doesn't use a stream. Every random decision is a pure function of four numbers:
bits = hash(episode_key, step, slot, index)
There's no history, no cursor, nothing to keep in sync. The slot tells it which decision this is, and the index separates repeats of the same decision, one per berry for example. Slots aren't invented in the code; they're declared in the spec, and this is the level of precision a spec has to reach before either implementation exists:
| slot | name | used at | index | distribution | |------|------------|----------------|-------|---------------------| | 0 | AGENT_X | reset (step 0) | 0 | randint(G) | | 1 | AGENT_Y | reset (step 0) | 0 | randint(G) | | 2 | BERRY_X | reset (step 0) | k | randint(G) | | 3 | BERRY_Y | reset (step 0) | k | randint(G) | | 4 | BERRY_KIND | reset (step 0) | k | randint(N_KINDS) | | 5 | GUST | every step t | 0 | bernoulli(GUST_P) | Same slot + same step + same index = same draw, in both implementations. That is the whole differential-testing contract.
examples/forager/spec.md, unedited: every random decision named, keyed and typed before any code is written. Two of the bugs further down are what happens when a row of this table gets misread.Drag the sliders: the draw depends on those four numbers and nothing else. Then hit "draw them out of order", and nothing moves.
That's the whole trick, and it solves three hard problems at once. Your batched code can draw for all ten thousand instances and throw most away, because a draw you threw away was never part of a sequence. An episode that ends early can re-key and restart in place without telling anybody. And porting to another array library is mechanical, because there's no hidden state to port.
That last one isn't hypothetical: the grid you played with up top runs on a 90-line JavaScript port of this exact generator.
Below are two copies running the same seed and the same actions, readable on the left and batched on the right. One bug is already switched on so that you land on a real divergence. Untick it and 300 steps come back identical, then go break something else.
Each bug shows you the actual line of code that causes it and what it costs you at training time. None of these are invented. They're the mistakes people make turning a readable environment into tensors, and most of them leave you with an environment that trains perfectly well.
"The states diverge at step six" is easy to shrug at, so here's what it does to you. Take a berry-seeking policy, score it in the broken environment the way your training dashboard would, then run the same policy in the real one and see what you actually get.
These are live numbers, computed in your browser over 50 episodes as you click, and the bugs don't all fail the same way:
You get three different disasters, and only one of them looks like a disaster while it's happening. The berry bug is the loud one: your dashboard reads +11.5 so you ship, and the policy scores −3.3 in the real environment because it learned a map that doesn't exist. The reward bug is the quiet one that costs you a month: your dashboard reads −2.3 for a policy actually worth +9.2, so you decide your algorithm is broken and go tune hyperparameters that were never the problem. The float bugs change no behaviour at all. They just mean nobody, including you, can reproduce your run.
To be clear about what this is: a hand-written policy scored in both environments, not a trained agent. A real training run would also adapt to the broken dynamics, which usually widens these gaps rather than closing them.
Everything so far compares your batched version against your readable one, and your readable one only knows how to be a single instance. So that comparison runs at N = 1, and the whole reason you're doing this is to run at N = 8192. What you tested and what you ship aren't the same thing.
Somewhere in your batched code a mask shaped [N] meets something shaped [N, 1]. In NumPy and PyTorch that isn't an error, it's a broadcast, and it hands you an [N, N] matrix that a reduction downstream flattens back into a perfectly reasonable-looking [N]. It takes one keyword to do this:
# correct alive_count = self.alive.sum(-1) terminated = (alive_count == 0) | (self.energy <= 0.0) # [N] [N] -> [N] ✓ # one keyword added alive_count = self.alive.sum(-1, keepdim=True) terminated = (alive_count == 0) | (self.energy <= 0.0) # [N, 1] [N] -> [N, N] ✗
At one instance that accidental matrix is 1×1, which is exactly the right answer. So your differential test doesn't just miss this bug, it's mathematically incapable of seeing it at the only batch size it can run at. The bug shows up when you scale up, which is when you started caring.
So you need a second check that doesn't involve your readable version at all. Run instance i on its own. Run it again inside a batch of eight. Require the two to be bit-identical. Drag N up and watch the rows go red.
All of that machinery ships with the package: the counter-based RNG in a scalar and a tensor flavour that are guaranteed to agree bit-for-bit, a BatchedEnv base class that owns the step loop, auto-reset inside the tensors, terminal-state capture and invariant checking, and trajectory read/write against your schema. forager's fast.py is about sixty lines, because the base class handles everything that isn't the game itself.
So it isn't left up to you. The ten tests write a report, and your training script reads that report and refuses to start if it's missing, failing, or older than your source files. Tweak fast.py, go straight back to training, and you get stopped, because the report no longer describes the code you're about to run.
Below is a real one from simulacrum validate examples/forager, on the same environment you were playing with up top. It's recorded output, not computed in the browser.
fast.py and going straight back to training. Pass strict=True and it raises instead of warning. The one opaque name in the list, test_benchmark_factory_parity: if you train on a tweaked variant of the env, a torch.compile'd one say, it has to be bit-identical to the batched version that passed everything above, or none of this transfers to the thing that actually trains.You write a second implementation, and you need the discipline to write it from the spec rather than from the first one. What you get back is that your readable version stops being a liability you'll eventually delete and becomes the thing that certifies the batched one on every run.
If you already have an environment, you're most of the way there, because what you have is the reference. Write down its spec, add a batched implementation, and let the battery tell you where the two disagree. Every divergence it finds is a place your spec was ambiguous, which you wanted to know about anyway.
pip install -e . # from a clone of the repo simulacrum new myenv # spec.md, reference.py, fast.py, tests/ simulacrum validate myenv # ten tests, one report, a verdict
require_fresh_report("myenv", strict=True). No environments ship with the framework; yours stays in your own repo.Going fast shouldn't cost you the ability to explain what your environment does. Write it once for yourself and once for the hardware, then let a machine hold both to the same spec, so your speed comes with a receipt.
Simulacrum · Orbitope