> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vikat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Running multiple replicas

> What is shared between gateway nodes, what is not, and what that means for limits.

<Warning>
  This page previously described a clustering system that does not exist: memberlist
  gossip on port 10101, a gRPC sync channel on 10102, six service-discovery
  backends, leader election and thirty replicated entity types. None of it was
  built. It has been replaced with what the gateway actually does.
</Warning>

## What is shared

Every replica points at the same config store and governance store. Virtual keys,
teams, budgets, rate limits, guardrail policy and audit rows all live there, so
every node reads the same configuration.

Three things are actively reconciled between nodes:

|                             | How                                                                               | Window             |
| --------------------------- | --------------------------------------------------------------------------------- | ------------------ |
| Virtual key revocation      | each node re-reads the live key set                                               | 30 seconds         |
| Budget and rate-limit usage | each node adds its own delta to a shared counter, then reads the fleet total back | one flush interval |
| Feature-flag toggles        | broadcast directly to peer gateways                                               | immediate          |

Feature-flag broadcast is configured with `VIKAT_CLUSTER_PEERS` (comma-separated
peer base URLs) and `VIKAT_CLUSTER_TOKEN`.

## What that means for limits

Budgets and rate limits are **convergent, not instantaneous**. Each node
contributes the spend it has accrued to a shared counter and reads the counter
back in the same transaction, so every node enforces against the fleet's total
as of its own last flush. Between flushes a node is behind by whatever the
others have spent in that window, and the fleet can overshoot a limit by that
much.

State it that way to a customer: *enforced fleet-wide within the flush interval*,
not a hard real-time cap. A gateway that promised the latter would be wrong, and
the person who found out would be an auditor.

<Note>
  Both halves of that sentence are load-bearing, and the read-back half was
  missing until it was measured. Writing a delta makes the shared *row* correct;
  it does nothing for what a node *enforces*, which reads an in-memory counter.
  Without the read-back, a $100 budget was enforced at $100 on every replica
  independently — two nodes admitted \$200 and the row faithfully recorded the
  overspend after the money was gone.

  `TestFleetBudgetHoldsAtTheConfiguredLimit` and
  `TestFleetRequestRateLimitHoldsAtTheConfiguredLimit` in
  `plugins/governance/fleetenforcement_test.go` run two governance stores against
  one real Postgres and hold this page to its word. They need
  `VIKAT_GOVERNANCE_TEST_DSN` and skip without it, so CI must fail if they skip.
</Note>

<Warning>
  **If you are upgrading a multi-replica deployment, limits get tighter.** A fleet
  of R replicas was previously admitting roughly R times every configured budget
  and rate limit. After this change it admits the configured value. Nothing about
  your configuration changed — the limits you already wrote down start being the
  limits you get, which for a 3-replica deployment means about a third of the
  throughput and spend you were seeing. Re-check any limit you tuned by watching
  production behaviour rather than by deciding a number.

  Each gateway logs this once at startup, the first time it observes spend from
  another replica: *"budgets and rate limits are now enforced against the FLEET
  total, not this node's share alone."*
</Warning>

## What is not shared

* **In-process caches** other than the three above. A model catalog refresh or a
  plugin reload reaches other nodes on their own schedule.
* **Sessions.** Console sessions live in the config store, so any node accepts
  them, but there is no sticky routing — a console user's requests may land on
  different nodes between clicks. That is fine, and it is why session state was
  moved out of memory.
* **Leader election.** There is none. Every node runs every background task, and
  the tasks that must not double up guard themselves with a conditional write
  (see budget threshold alerts).

## Deployment shape

Run replicas behind an ordinary load balancer. No inter-node port needs to be
open except the peer HTTP port if you use feature-flag broadcast.

Helm defaults to `replicaCount: 1`. Raise it once you have read the limits
section above and are comfortable with the convergence window.