# Scaling plan — multi-vehicle (target: 2000 vehicles)

Development plan for scaling the CreatorStudio Engine from a single-process,
few-hundred-vehicle setup to a fleet of ~2000 vehicles.

**TL;DR:** after Phase 0 hardening, a single **hardened** process now handles ~2000
vehicles comfortably on modest hardware (measured — see the ladder below). The remaining
walls are TTS throughput/cost and, beyond ~3000 on 2 cores, the single-process CPU ceiling.
In-memory caches and the inbound firehose have been bounded/optimised, so sharding is now an
availability/headroom play — not a functional necessity for 2000. Full fleet scale is
achievable with a real TTS tier (Phase 1) and, for headroom/availability beyond ~3000,
sharding + a clustered broker (Phases 2–3).

> **Multi-tenant note.** Across a larger (~5k-vehicle) multi-tenant broker, the realised scaling
> model is **one engine process per tenant**: the bridge pre-scopes to a tenant (`UPSTREAM_TENANT`)
> and the engine reads only its `(tenant, fleet)` config (`CONFIG_DB_TENANT_VALUE`). Each
> tenant's ~hundreds–2000 vehicles then fit a single hardened process per the ladder below, and
> tenants scale horizontally by process rather than by sharding one giant process.

---

## 1. Where it stands today

Single Node process:
- Subscribes `+/+/pis/0/#` (whole fleet) — or a bridge feeds it a selected/tenant-scoped subset.
- Holds each vehicle's `VehicleContext` **in memory**; `PtTriggerEngine` holds a
  per-vehicle snapshot **in memory** — both now idle-TTL-evicted (Phase 0).
- On inbound message: `parseTopic` → update context → `engine.ingest` (edge detection, **only
  on state-changing topics**) → on event: `deriveState` → `scheduler.offer` (per-vehicle
  priority `AnnouncementScheduler`) → bounded render semaphore → `renderPlaylist` → TTS
  (single-flight + layered cache) → publish ADT + metrics. A predictive pre-render warms
  upcoming stops on the same path.

### Per-subsystem assessment at 2000 vehicles

| Subsystem | Load at 2000 vehicles | Verdict |
| --- | --- | --- |
| **Inbound parse** | GNSS @1 Hz ≈ 2000 msg/s + other topics → ~3–6k msg/s of `JSON.parse` + context updates on one event loop | Feasible on a good core (measured OK to ~3000); GNSS gated so it doesn't drive detection |
| **Trigger eval** | Cached enabled-trigger condition map (rebuilt only on config change), `getCondition()` O(1), threshold math only when enabled, GNSS gated | ✅ Fixed (Phase 0) |
| **TTS** | ~15–30 announcements/s peak; Azure **F0 caps ~20 req/s** → 429s. Single-flight dedup + warm cache cut the *peak synth* rate materially | **Remaining wall** — needs paid tier or self-hosted for true peak |
| **Memory** | Mem LRU (`TTS_MEM_CACHE_MAX`) + on-disk cap (`CACHE_MAX_FILES`) + context idle-TTL sweep | ✅ Fixed (bounded) |
| **Concurrency** | Counting render semaphore (`MAX_CONCURRENT_RENDERS`) + per-vehicle priority scheduler + single-flight synth dedup | ✅ Fixed (backpressure + scheduler) |
| **Publish** | ~15–30 ADT payloads/s × ~30 KB ≈ ~1 MB/s | Fine |
| **Broker** | 2000 publishers + engine + dashboards | Mosquitto OK-ish; EMQX better for 2000+ |
| **Single process** | One core, single point of failure | Measured OK to ~3000 on 2 cores; still a SPOF — sharding is availability/headroom, not a functional need for 2000 |

---

## 2. Known issues to fix before scale (tech debt) — ✅ done in Phase 0

- [x] In-memory MP3 cache (`TtsCache.mem`) is an **unbounded `Map`** → LRU cap
      (`TTS_MEM_CACHE_MAX`, `stats()` for hit-rate).
- [x] On-disk FS cache layer was unbounded (one MP3/phrase forever) → oldest-mtime prune to
      `CACHE_MAX_FILES` (`TtsCache.pruneFs`, amortised off the hot path).
- [x] `VehicleContext`s are **never evicted** → idle-TTL sweep (`CONTEXT_TTL_MS`) releasing
      contexts, scheduler slots, prerenderer, and exterior-sign state.
- [x] `getCondition()` did `triggers.find()` **per message** → cached condition map,
      rebuilt only on config `version()` change.
- [x] Threshold conditions evaluated on **every** message → only computed when that
      trigger is enabled; full detection skipped for non-state topics (GNSS/`shape`).
- [x] Fire-and-forget renders with **no limit** → a counting render semaphore
      (`MAX_CONCURRENT_RENDERS`, `acquire`/`release` in `index.ts`) + a per-vehicle priority
      `AnnouncementScheduler` (interrupt/queue/drop; interrupt aborts the superseded render).
- [x] Concurrent identical synths stampeded the provider → **single-flight** `getOrSynth`
      collapses same-key (text+voice+format) synths to one in-flight render (`synthCoalesced`).
- [x] Cold synths at each stop → **predictive pre-rendering** (`src/engine/prerender.ts`) warms
      the next N stops so real triggers hit cache; shares the global render semaphore.
- [x] No runtime health metrics → `engine/health` every `HEALTH_INTERVAL_MS` (vehicles, served,
      renderScope, msg/s, events/s, inFlight, queued, dropped, interrupted, RSS, loop lag, cache
      hit-rate/size, synthCoalesced, prerendered, TTS breaker/fallbacks/retries), plus a separate
      retained `engine/bridge` health topic.

---

## 3. Phased plan

### Phase 0 — Measure & harden a single instance — ✅ DONE & VERIFIED
- [x] **Metrics**: `engine/health` publishes msg/s, events/s, event-loop lag, RSS,
      cache hit-rate/size, in-flight, dropped.
- [x] **Load generator**: `scripts/load-gen.ts` (`npm run load -- <count>`) — N
      synthetic vehicles (GNSS firehose + stop advances).
- [x] **Multi-vehicle proof**: `scripts/multi-demo.ts` (`npm run multi:demo -- N [select]`)
      drives N vehicles concurrently against one engine and asserts per-vehicle correctness, no
      cross-talk, bounded concurrency, and served-set isolation.
- [x] **Bound caches**: LRU on the in-memory MP3 map; oldest-mtime FS prune to `CACHE_MAX_FILES`;
      idle-TTL eviction of contexts.
- [x] **Optimize ingest**: cached enabled-trigger map; full detection only on
      state-changing topics (GNSS gated to speed/geofence triggers); threshold math
      only when enabled.
- [x] **Backpressure**: a counting semaphore (`MAX_CONCURRENT_RENDERS`) + per-vehicle priority
      scheduler + single-flight synth dedup (`getOrSynth`).
- [x] **Serving scope**: `RENDER_SCOPE` (default `selection` — only dashboard-selected vehicles,
      silent until opted in; `all` = every fully-tracked vehicle, load-test) bounds how many
      vehicles the engine renders for. Reported as `renderScope`/`served` on health.
- [x] **Upstream load reduction**: the bridge (`scripts/bridge.ts`) follows the selection
      (subscribes upstream to only selected vehicles), tenant-scopes fleet discovery, and drops
      byte-identical retained re-publishes (`UPSTREAM_DEDUP`) — a large cut to the inbound firehose.

**Verified** (isolated broker, 150 vehicles): health metrics flowing; GNSS gating
(msg/s ≫ events/s); event-loop lag ~30 ms under load; RSS stable ~110 MB; backpressure
sheds bursts; stale vehicles evicted after TTL. Offline LRU unit test also passes.

#### Measured ceiling (2026-07-06, load ladder 50 → 4000)

Run on a small 2-vCPU / 4 GB Linux box (aedes broker + engine + load-gen on the same
host), `TTS_PROVIDER=mock` with 30 ms simulated synth latency so the ladder measures
the *engine*, not the TTS vendor. GNSS @1 Hz per vehicle, stop advance every 12 s,
~30 s steady state per rung. Steady-state event-loop lag (p50) per fleet size:

| Vehicles | msg/s | lag p50 | RSS | dropped |
| --- | --- | --- | --- | --- |
| 500 | 500 | 20 ms | 181 MB | 0 |
| 1000 | 1000 | 21 ms | 186 MB | 0 |
| 2000 | 2000 | 29 ms | 250 MB | 0 |
| 3000 | 3000 | 86 ms | 289 MB | 0 |
| 4000 | 4000 | ~870 ms | 324 MB | 0 |

**Verdict: a single hardened instance handles 2000 vehicles comfortably even on a
2-core box** — cache hit-rate 100%, zero drops/queue overflow. The wall is ~3000–3500
on this hardware (lag grows superlinearly; 4000 saturates the loop). Real server
hardware moves the wall up, but 2000 no longer *requires* Phase 2 sharding — sharding
becomes an availability/headroom play, not a functional necessity.

Findings from the ladder:
- **Retained-topic startup flood**: when the engine (re)connects against a large fleet,
  the broker replays ~5 retained topics × N vehicles at once — multi-second lag spikes
  at 2000+. Worth throttling initial backfill (paced subscribe or ingest queue).
- **`MaxListenersExceededWarning`** at 4000: >1000 `drain` listeners piled up on the
  MQTT socket — publish backpressure needs a shared drain wait, not one listener per
  pending publish.
- **`engine/fleet` payload is O(fleet)** (~940 KB at 4000, republished every second) —
  should be gated off or decimated when no dashboard is subscribed.

Repeat with: `TTS_PROVIDER=mock npm run dev` + `npm run load -- 2000` (mock provider:
`src/tts/mock.ts`, `MOCK_TTS_LATENCY_MS` simulates per-miss synth cost). Also tune
`MAX_CONCURRENT_RENDERS` (renders are mostly fast cache hits, so it can be well above 8).

### Phase 1 — TTS scaling (the cost/throughput blocker)
- [ ] Move off Azure **F0** to a tier with adequate TPS, **or self-host Piper** for
      unlimited local throughput (no per-char cost, no rate limit). *(the only remaining gap)*
- [x] Shared **warm cache** keyed by sha256(`text+voice+model+format`): an **Azure Blob Storage**
      layer already backs the mem/FS cache (`CACHE_BLOB_SAS_URL`), and **pre-warming** is
      implemented (`prerender.ts` warms upcoming-stop phrases). Redis/S3 remain optional future
      layers. Stop-name phrases are finite per route, so the cache converges to cheap hits.
- [x] Retry with backoff on HTTP 429 (`ResilientTtsProvider` retries transient failures incl.
      429 with exponential backoff + jitter and a half-open circuit breaker; only 401/403 skip
      retry); provider concurrency is bounded by `MAX_CONCURRENT_RENDERS`.

**Exit criteria:** sustained render rate at peak with acceptable latency and no 429s;
cache hit-rate > ~90% in steady state.

### Phase 2 — Horizontal sharding (the actual 2000+ answer)
The engine keeps **per-vehicle state in memory**, so a vehicle's messages must stick
to one worker. Two options:

- **(a) Deterministic shard by key** — run N engines, each owning a slice of vehicles
  (by tenant, or a `shard` topic level the bridge adds:
  `mta-maryland/{shard}/{vehicleId}/pis/0/#`, each worker subscribing its shard).
  Sticky, simple, no shared state. **Cleanest if the bridge can add a shard level.**
- **(b) Stateless workers + Redis** — move `VehicleContext` + snapshots into Redis so
  any worker can process any message, and use **MQTT 5 shared subscriptions**
  (`$share/group/+/+/pis/0/#`) so the broker load-balances the fleet across the group.
  Elastic / auto-scaling, at the cost of a Redis read/write per message.

**Recommendation:** (a) now (2–4 shards cover 2000 comfortably); (b) when elasticity
beyond that is needed.

> Note: plain MQTT `+` wildcards match a whole topic level, not a prefix — you can't
> subscribe `mta-maryland/1*/...`. So sharding needs either a **shard topic level**
> from the bridge, a **router** that re-publishes to shard topics, or the
> shared-subscription + external-state model.

### Phase 3 — Operations
- [ ] Broker: **EMQX** (native clustering + shared subscriptions) instead of a single
      Mosquitto for 2000+.
- [ ] Kubernetes `Deployment` per shard (or HPA in the stateless model, scaling on
      event-loop lag / backlog).
- [ ] Observability: Prometheus + Grafana; alert on render latency, 429s, loop lag.
- [ ] Dedup / idempotency; at-least-once handling; graceful drain on shutdown.

---

## 4. Capacity estimate for 2000 vehicles

- **Compute:** 2–4 sharded/stateless workers give ample headroom for ~3–6k msg/s
  ingest and ~15–30 renders/s.
- **TTS:** the real sizing lever — a paid Azure tier *or* a couple of Piper instances;
  warm cache keeps it cheap.
- **Memory:** ~100–300 MB per worker **with bounded caches** (mem LRU + FS cap + context TTL — all in place). Measured RSS ~250 MB at 2000 vehicles.
- **Broker:** EMQX cluster (or a well-sized Mosquitto) + WebSocket listener for dashboards.

**Verdict:** achievable with Phase 0 hardening + a proper TTS tier + Phase 2a sharding
+ EMQX. **Not** by pointing today's single process at 2000 buses.

---

## 5. Suggested order of work

1. Phase 0 hardening (LRU caches, context eviction, ingest optimizations, backpressure)
   — safe, immediate wins on the current single instance.
2. Load generator + metrics — measure the real ceiling.
3. Phase 1 TTS (tier/self-host + shared cache) — remove the cost/throughput wall.
4. Phase 2a sharding — cross the 2000 line.
5. Phase 3 ops hardening — make it production-grade.