Director’s commentary · The Ham Sandwich

Hotspots you can Tab to

How a point-and-click game in WebGL stays playable by keyboard and screen reader: real buttons projected over the 3D scene, tap and long-press input, a polite live region, reduced motion, and a Read instead link from the first frame.

A point-and-click adventure is, by definition, a game about pointing and clicking, which is a problem if you can't do either. The rule I set in the spec was that keyboard and screen-reader users can do everything a mouse user can. The prologue is where that rule gets tested first, because it teaches every control in the game without a line of tutorial text: each bite of the sandwich waits until you've tried something new. If any of those controls only worked with a pointer, you'd be stuck on a doorstep with three bites left.

Real buttons in the right places

A WebGL canvas is one big picture as far as assistive technology is concerned. There's nothing in it to focus and nothing to read. So every visible hotspot is mirrored as a real button, positioned over the thing it stands for. Every frame, the layer takes the centre of each hotspot's bounding box, projects it through the camera into screen space and moves its button there. Anything behind the camera or off the edge of the frame is hidden.

const place = (key: string, pos: THREE.Vector3 | null) => {
  const el = refs.current.get(key);
  if (!el) return;
  if (!pos) return void (el.style.display = "none");
  v.copy(pos).project(cam);
  const off = v.z > 1 || v.x < -1.05 || v.x > 1.05 || v.y < -1.05 || v.y > 1.05;
  el.style.display = off ? "none" : "";
  el.style.transform = `translate(${rect.left + ((v.x + 1) / 2) * rect.width}px, ${rect.top + ((1 - v.y) / 2) * rect.height}px) translate(-50%, -50%)`;
};
for (const id of visible) {
  const obj = world.hotspots.get(id);
  if (!obj) continue;
  box.setFromObject(obj);
  place(id, box.isEmpty() ? obj.getWorldPosition(v.clone()) : box.getCenter(new THREE.Vector3()));
}

The buttons carry an accessible name built from the hotspot's verb and name, such as Eat Ham sandwich or Talk to Mum, and they respond the way the controls table in the spec says they should: Enter or Space to use, L or the context menu to look.

<button
  key={id}
  type="button"
  ref={(el) => {
    if (el) refs.current.set(id, el);
    else refs.current.delete(id);
  }}
  className="hotspot-btn"
  aria-label={names.get(id)}
  aria-disabled={busy || undefined}
  onFocus={() => ui.set({ focusedHotspot: id })}
  onBlur={() => ui.set({ focusedHotspot: null })}
  onClick={() => game.actOn(id, "use")}
  onContextMenu={(e) => {
    e.preventDefault();
    game.actOn(id, "look");
  }}
  onKeyDown={(e) => {
    if (e.key === "l" || e.key === "L") {
      e.preventDefault();
      game.actOn(id, "look");
    }
  }}
/>

They're invisible until they get keyboard focus, and then they show a phosphor-orange ring and a label underneath, so a sighted keyboard user can see where they are. They never intercept a click or a tap, either: pointer events pass straight through them to the canvas, so the mouse and touch paths are untouched.

Tap, long-press, right-click

Pointer input goes through one small class that turns raw pointer events into intentions: tap, look, hover and leave. A press that's released within 400 milliseconds is a tap. One that's held longer is a look, and the release that follows is swallowed. A press that wanders more than 8 pixels is a drag and counts as nothing, and a second finger cancels the lot, because that's a gesture rather than a tap.

/** The long-press timer fired: true if that's a look. The release that follows is swallowed. */
hold(): boolean {
  if (this.state !== "down") return false;
  this.state = "held";
  return true;
}

/** A release that beat the timer by less than longPressMs is a tap; a late one (timer lag) is a look. */
up(x: number, y: number, t: number): "tap" | "look" | null {
  this.move(x, y);
  const was = this.state;
  this.state = "idle";
  if (was !== "down") return null;
  return t - this.t < this.longPressMs ? "tap" : "look";
}
// …
private readonly onContextMenu = (e: MouseEvent) => {
  e.preventDefault();
  if ((this.press.pressed && this.touchPress) || performance.now() - this.lastTouch < TOUCH_ECHO_MS) return;
  this.endPress(); // ctrl-click on a Mac starts a left press first
  this.emit("look", e.clientX, e.clientY);
};

The context-menu handler is where the platforms disagree. Right-click is Look on a desktop, but on touch screens the browser fires its own context-menu event at the end of a long press, and on a Mac a Ctrl-click starts as a left press. So a context menu that arrives within 800 milliseconds of a touch is treated as the browser echoing a long press we've already handled, and ignored.

When a tap does land, picking tries hotspots before the floor, so a tap on the sandwich uses the sandwich rather than walking you to the spot underneath it. Anything with a hidden parent can't be picked, so a hotspot that's been put away stays put away.

Saying it out loud

Narration and dialogue go to a polite live region, so a screen reader announces each line without interrupting what it's already saying. Each announcement starts with who's speaking. A second region announces changes of mode, such as arriving at the disk box, and the keyboard covers the rest: Space advances a line, H asks the narrator for a hint, I opens the bag, and Escape closes whatever's open or goes back to the disk box.

/** Polite live region: narration, toasts and mode changes for screen readers (spec §9). */
export function LiveRegion() {
  const line = useUi((s) => s.line);
  const toast = useUi((s) => s.toast);
  const mode = useUi((s) => s.mode);
  const text = line ? `${line.kind === "hint" ? "Hint" : line.speakerName}: ${line.text}` : toast?.text ?? "";
  const modeText =
    mode === "diskbox" ? "The disk box. Choose a disk, then insert it into the drive." : mode === "loading" ? "Loading." : "";
  return (
    <>
      <div className="sr-only" role="status" aria-live="polite" aria-atomic="true">
        {text}
      </div>
      <div className="sr-only" aria-live="polite">
        {modeText}
      </div>
    </>
  );
}

Less motion, and a way out

Reduced motion follows the operating system unless you've set it yourself in the game's settings. With it on, camera glides and rails become cuts, the film grain freezes, and the fluorescent flicker in Disk 5 and the roll band on the CRT stop. Separately from that setting, nothing in the game flashes more than three times a second, ever: the flicker is built from slow sine waves, and a test checks every one of them is under 3 Hz.

reducedMotion(): boolean {
  const pref = useSave.getState().settings.reducedMotion;
  if (pref === "on") return true;
  if (pref === "off") return false;
  return typeof matchMedia !== "undefined" && matchMedia("(prefers-reduced-motion: reduce)").matches;
}

And from the very first frame, the first thing a keyboard or screen-reader user reaches is a link that says Read instead. It goes to the plain version of whichever chapter you're in: the same story as prose, server-rendered, with no WebGL at all. The budget checker fails the build if a WebGL renderer ever sneaks into the plain pages' JavaScript. Some people will never want the game, and they shouldn't have to fight it to get the story.

What I'd do differently

Tab order follows the order the scene registered its hotspots, not where they are on screen, so in a busy shot focus can jump from the left of the room to the right and back. Sorting the buttons by their projected position, per shot, would make tabbing through a scene read left to right like a sentence. The projection loop also runs every animation frame even when nothing's moving, which is cheap but not free, and it could wait for the camera to move instead.

Most of all, none of this has been through a proper audit with people who use screen readers every day. That's in the polish milestone, and I'd much rather hear what's wrong from them than guess. The accessibility statement says how to tell me.