Skip to main content
This page previously described a clustering system that does not exist: memberlist gossip on port 10101, a gRPC sync channel on 10102, six service-discovery backends, leader election and thirty replicated entity types. None of it was built. It has been replaced with what the gateway actually does.

​
What is shared

Every replica points at the same config store and governance store. Virtual keys, teams, budgets, rate limits, guardrail policy and audit rows all live there, so every node reads the same configuration. Three things are actively reconciled between nodes: Feature-flag broadcast is configured with VIKAT_CLUSTER_PEERS (comma-separated peer base URLs) and VIKAT_CLUSTER_TOKEN.

​
What that means for limits

Budgets and rate limits are convergent, not instantaneous. Each node contributes the spend it has accrued to a shared counter and reads the counter back in the same transaction, so every node enforces against the fleet’s total as of its own last flush. Between flushes a node is behind by whatever the others have spent in that window, and the fleet can overshoot a limit by that much. State it that way to a customer: enforced fleet-wide within the flush interval, not a hard real-time cap. A gateway that promised the latter would be wrong, and the person who found out would be an auditor.
Both halves of that sentence are load-bearing, and the read-back half was missing until it was measured. Writing a delta makes the shared row correct; it does nothing for what a node enforces, which reads an in-memory counter. Without the read-back, a 100budgetwasenforcedat100 budget was enforced at 100 on every replica independently — two nodes admitted $200 and the row faithfully recorded the overspend after the money was gone.TestFleetBudgetHoldsAtTheConfiguredLimit and TestFleetRequestRateLimitHoldsAtTheConfiguredLimit in plugins/governance/fleetenforcement_test.go run two governance stores against one real Postgres and hold this page to its word. They need VIKAT_GOVERNANCE_TEST_DSN and skip without it, so CI must fail if they skip.
If you are upgrading a multi-replica deployment, limits get tighter. A fleet of R replicas was previously admitting roughly R times every configured budget and rate limit. After this change it admits the configured value. Nothing about your configuration changed — the limits you already wrote down start being the limits you get, which for a 3-replica deployment means about a third of the throughput and spend you were seeing. Re-check any limit you tuned by watching production behaviour rather than by deciding a number.Each gateway logs this once at startup, the first time it observes spend from another replica: “budgets and rate limits are now enforced against the FLEET total, not this node’s share alone.”

​
What is not shared

  • In-process caches other than the three above. A model catalog refresh or a plugin reload reaches other nodes on their own schedule.
  • Sessions. Console sessions live in the config store, so any node accepts them, but there is no sticky routing — a console user’s requests may land on different nodes between clicks. That is fine, and it is why session state was moved out of memory.
  • Leader election. There is none. Every node runs every background task, and the tasks that must not double up guard themselves with a conditional write (see budget threshold alerts).

​
Deployment shape

Run replicas behind an ordinary load balancer. No inter-node port needs to be open except the peer HTTP port if you use feature-flag broadcast. Helm defaults to replicaCount: 1. Raise it once you have read the limits section above and are comfortable with the convergence window.