Elevator dispatch is a solved problem — the rules that run most buildings are seventy years old and genuinely good. Solved, anyway, until you’re the one in the lobby watching every car idle on the wrong floor, wondering what the holdup could possibly be. That’s the wait that made me try to beat the status quo: a learned policy against the incumbent, on identical traffic, measured honestly — where it earns its keep, and where it doesn’t. Scroll to see what the agent sees, how it’s scored, how its brain is wired — and the one change that actually mattered.
The interactive visualizations need a wider screen — on mobile you'll get the prose and the tables.
Elevator dispatch is not an open problem. Buildings have run on hand-engineered rules for seventy years, and the rules are genuinely good — collective control (LOOK), cost-function dispatch, destination dispatch. Nobody is waiting for a neural network to rescue their lobby.
Take LOOK, the most common one. Each car sweeps in one direction, answering every call along the way, and turns around only when nothing is left ahead of it — like a bus that runs its route to the end and comes back. Unanswered hall calls get handed to whichever idle car is nearest. It needs no tuning, and in a quiet building it is very hard to beat. What it can't do is plan: the assignment is greedy and car-by-car, with no picture of what the other cars — or the next minute of arrivals — will do. (The mechanics, and its stronger cost-function cousin, are laid out in full further down.)
So the honest framing isn't “RL beats elevators.” It's narrower: a tuned rule set is a fixed strategy, and as buildings get bigger — more floors, more cars, more coordination between them — a fixed strategy should start leaving service on the table. That's the bet this project measures. In a small building I fully expect to lose. The interesting question is where the crossover is.
The agent's entire input is one flat vector of numbers — 98 of them in the small building, 254 in the mid-size, 648 in the large. But that vector is really a live status panel, and it's easiest to read one block at a time.
Where is each car? A one-hot row per car marks the floor it's on — simply its height in the shaft.
Is each car in service? One bit per car flags the ones switched off — they grey out and drop from the agent's control.
Which way is it moving, and where between floors? Travel direction — down, idle, up — plus a smooth fractional position.
How full is it? Riders over capacity, drawn as the fill gauge — the difference between a car that can still stop for you and one that can't.
Where do the people aboard want off? The in-car destination buttons light the floors its riders are headed to.
Which floors are calling? Now the demand: a lamp beside a floor, up or down, means someone is waiting there.
And how long have they waited? The call lamps warm from steel to coral as the oldest person there closes on the 45-second abandon limit.
Together, these seven blocks are the observation — everything the agent knows, and nothing else, flattened into that one vector.
And here's the blind spot. The agent sees that people are waiting, and for how long — but never where they want to go. A real hall call carries no destination, so it must infer demand from the calls alone. This limit comes back later in a way I didn't expect.
Every half-second decision earns one number, summed from seven forces. Delivering a passenger is the big prize: +10.
Underneath, two costs drip every second the system is congested — people riding, and, weighted three times heavier, people still waiting in the hall.
Let traffic overwhelm the cars and the catastrophic penalties bite: −8 when someone gives up, −5 when a full queue turns an arrival away.
There's also a small shaping nudge in the original design: ±0.4 per floor a loaded car moves toward (or away from) a rider's destination. It looks harmless. Keep it in mind — it turns out to be the most important coefficient in the whole project, just not in the way you'd think.
The simplest brain flattens everything into one lump — a plain MLP over the whole status panel, one action head per car. No structure, and the network widens with every floor and car you add.
The structured alternatives all encode a prior. A convolution slides one filter up the floors — the same weights reused at every height, the way tall buildings repeat themselves.
One shared mini-brain per car reads a common summary, so the weight set no longer grows with the fleet. And letting those per-car brains talk through attention targets the hardest part of dispatch directly: coordination, so two cars don't chase the same call.
Escalating structure, escalating theory. The expectation is obvious: the structured brains should win as the building grows. That expectation is tested — not assumed — in the results below.
The reward function is just seven numbers. Drag them in the panel and watch which behaviour wins — the coefficients are the goal, and small changes reorder the leaderboard.
Then open the live elevator simulator to run a full building yourself — arrivals, queues, and all six actions per car.
Every number here is a greedy policy scored on the same protocol as the classical baseline: hour-long episodes, five fixed seeds, identical passenger traffic for every method, wait times measured over delivered passengers. No cherry-picking against a hobbled opponent — LOOK plays its best game.
The thesis line held: in the small building RL ties the heuristic — there was nothing to win. In the mid-size building it cuts mean wait by a quarter with zero abandonment; in the large one it delivers more people while cutting abandonment nearly five-fold.
The twist: none of that came from a fancier network. Flip the reward selector between shaped and unshaped and watch the same flat MLP jump. The full story is below the fold.
The simulation is turn-based underneath: passengers arrive by a Poisson process whose rate follows a 48-bin daily profile, queue per floor and direction (capacity 12, abandon after 45 s), and board capacity-8 cars. The policy acts every 0.5 s of simulated time — six discrete actions per car (idle, up, down, board-up, board-down, unload) — while the physics ticks at 0.1 s underneath: a semi-MDP with reward accumulated between decisions. Three building sizes carry the experiments:
| rung | floors | cars | population | observation dim | joint actions |
|---|---|---|---|---|---|
| S | 8 | 3 | 360 | 98 | 6³ = 216 |
| M | 16 | 5 | 600 | 254 | 6⁵ = 7,776 |
| L | 30 | 8 | 960 | 648 | 6⁸ ≈ 1.7M |
A dispatcher is only as interesting as the demand it faces, so the arrivals aren't a fixed schedule — they're a stochastic process with structure. Every tick, each floor draws a Poisson count of new passengers from a rate that follows a twelve-hour daily profile (48 fifteen-minute bins), and each new passenger draws a destination from an origin-dependent distribution. The profile is the sum of three overlapping demand components, and it's their balance over the day that produces the classic traffic patterns:
Origins are lobby-weighted — all incoming traffic starts at floor 0 — and destinations follow a matrix: from the lobby, roughly uniform across the upper floors; from an upper floor, split between “back to the lobby” and “across to another upper floor.” A single intensity knob scales the whole thing to a target load (population × a per-slot factor). The tensor experiments run the interfloor / midday slice at nominal load — deliberately the least forgiving regime for the agent's blind spot, since demand is spread across floors rather than funnelled through the lobby, so the cars can't just camp at the bottom. And because the RNG is counter-based, the same seed replays the identical passenger tape for every method, so LOOK and the learned policy are graded on the same people arriving at the same instants.
This part matters for trusting the numbers, so let me be concrete. The elevator simulator began as a Unity project driven by ML-Agents — a full 3D build that steps one building at a time. For this work I rebuilt the entire simulation from scratch as a pure-Python tensor environment: identical rules and traffic, but the whole state lives in plain arrays, so hundreds of buildings step at once with no game engine in the loop. Every result on this page comes from that Python rebuild.
The payoff is speed. The tensor environment plus the PPO learner runs at ~46,000 agent-steps/s on a laptop CPU — about 37× the original Unity/ML-Agents pipeline, measured end-to-end on the same machine — so a five-million-step run drops from hours to roughly two minutes. That single change is why an experiment matrix this wide was even possible; the Unity build is now the thing I check against, not the thing I train in.
And to trust a rewrite you have to prove it matches, so I built the Python version twice: a slow, readable single-instance reference and the fast batched one, both from a single spec, sharing a counter-based RNG so every random draw is a pure function of (seed, episode, step, slot). A differential-test battery steps the two side by side and compares every value at zero tolerance — the state is all-integer by design, so there is nothing to be approximately right about. When a number moves in the results below, it's the experiment that changed, not a drifting simulator.
The trainer is PPO with one categorical head per car and action masking — a car mid-travel can only no-op; boarding is masked when the car is full. Observations are normalized online; entropy is annealed to zero over training and the best checkpoint by episode return is kept. Two details earned their place the hard way: evaluation is greedy (argmax, matching how deterministic dispatchers run), and a slightly under-converged stochastic policy can look competent while its argmax parks a full car on a legal no-op forever. Masking board-when-full and annealing entropy gently removed that failure on every seed. I also tried a value-based D3QN — natively greedy, perfectly robust, and ~1.7× LOOK's mean wait; PPO stayed.
Networks: 256×2 at S, 768×4 at M and L (the structured variants below use 512×3). Budgets: 5M steps at S (parity with the original Unity run), 10–15M at M/L. Evaluation protocol, identical for every method including LOOK: 3,600 s episodes with a 300 s warmup, seeds 1–5, identical traffic per seed, wait statistics over delivered passengers with p95 from a 64-bin histogram. Every RL number on this page is stated relative to LOOK on the same traffic tape.
Headline first — final policy (flat MLP, unshaped reward) versus LOOK, per building size, means over the five seeds:
| rung | method | delivered | wait mean (s) | wait p95 (s) | abandoned |
|---|---|---|---|---|---|
| S | LOOK | 245 | 6.0 | 16.5 | 0.2 |
| PPO | 245 | 5.85 | 16.2 | 0 | |
| M | LOOK | 418 | 9.1 | 26.1 | 3.4 |
| PPO | 422 | 6.86 | 18.0 | 0 | |
| L | LOOK | 678 | 10.4 | 30.3 | 14.6 |
| PPO | 686 | 9.83 | 28.3 | 3.0 |
Exactly the shape the thesis predicted: a tie where the heuristic is already near-optimal, a clear win once there's coordination to exploit — −25% mean wait and zero abandonment at M, more delivered with a fraction of the abandonment at L.
The reward has seven terms (scene 02). Shaped is the original design: it includes a small ±0.4-per-floor bonus/penalty for a loaded car moving toward (or away from) a rider's destination — a hint that pays the agent for looking like progress, before anyone is actually delivered. Unshaped deletes those two movement terms, so the agent is paid only for real outcomes: deliveries, time spent waiting and riding, and the abandon / reject penalties. Every other coefficient is identical. That one difference is the whole experiment below.
Now the part I got wrong. I built four architectures expecting structure to win at scale. Measured at M under the original (shaped) reward:
| architecture | training return | wait mean (s) | wait p95 (s) | abandoned |
|---|---|---|---|---|
| Flat MLP | 1,258 | 7.5 | 21.1 | 0.8 |
| Attention | 1,449 | 8.0 | 23.9 | 3.4 |
| Per-car | 1,366 | 8.9 | 27.0 | 5.0 |
| Conv | 1,326 | 8.9 | 31.6 | 26.0 |
Read those two columns against each other. Every structured network out-optimized the flat one on the training objective, and not one of them out-served it. They weren't failing to learn — they were learning the wrong thing better. That pattern — higher reward, worse service — points at exactly one suspect: the reward.
The suspect was the ±0.4 movement nudge from scene 02. It pays a loaded car for drifting toward destinations whether or not anyone gets delivered — a subsidy the more expressive networks learned to farm. Deleting those two coefficients and retraining the plain flat MLP:
| rung | reward | wait mean (s) | wait p95 (s) | abandoned |
|---|---|---|---|---|
| M | shaped | 7.5 | 21.1 | 0.8 |
| unshaped | 6.86 | 18.0 | 0 | |
| L | shaped | 9.45 | 28.4 | 19.8 |
| unshaped | 9.83 | 28.3 | 3.0 |
Strictly better where it counts — at L the abandonment drops from 20 to 3 for the same architecture. The single highest-leverage change in this project was deleting two numbers from the reward function. Months of architecture work resolved to an objective-alignment bug, which is either deflating or the most useful lesson here, depending on how you look at it. My take: measure the objective before you upgrade the model.