Fleet & Scaling

Overview

A fleet is any number of servers of one game that see each other. The same configuration runs one server or fifty: start one more copy of the server and it joins; stop one and its work moves on. The game code does not change, and neither does config.yml.

The principle stays the one that makes a game server fast: a unit of work (a match room, a lobby) lives on exactly one server, in memory. Servers never talk to each other on the engine's hot path. They only coordinate on the control plane — a heartbeat every two seconds, a lookup when a player connects or moves — through a shared backplane.

altruist:
  server:
    http: { host: "0.0.0.0", port: 8080 }
    capacity:
      max-load: 200
    drain:
      timeout: 360
    fleet:
      redis: "redis:6379"      # the shared backplane
      cluster: "my-game"

What you get:

• Queues gather. The players of a playlist wait on one server, so a fleet of ten does not split them into ten small queues. Their waiting time moves with them.
• Full servers overflow. A server at its max-load sends its queue to a server with room.
• Rejoin from anywhere. A player who reconnects through the load balancer to the wrong server is sent to the server that runs its room.
• Lobby codes work everywhere. A code typed on any server leads to the lobby's server.
• Graceful scale-down. A draining server hands its queues and lobbies to the others, lets its rooms finish, and sends newcomers away.
• One public address. Any server relays a client to any other (?node=<id>), so the load balancer needs no sticky sessions and the servers need no public addresses of their own.

With a single server — or with the default in-memory backplane — every decision stays local and nothing moves: the fleet code is inert until a second server appears.

How It Works

 client ──► load balancer ──► any server (A)
                                 │  queue: "duel"
                                 │  A publishes its queue lengths with its heartbeat
                                 ▼
             the servers agree where "duel" players gather: server B
                                 │  A writes a handoff (playlist, waited 4.2 s) for B
                                 │  A tells the client: redirect to B
 client ──► load balancer ──► any server (C) ──relay──► B   (?node=B)
                                 B takes the handoff: the player waits on, 4.2 s already counted
                                 B matches the players and runs the room, in memory

Registry and heartbeat

Every server publishes itself every heartbeat seconds (default 2): its id, region, internal and public address, state, load and budget (from its server node), and numbers its modules publish (for example the queue length per playlist). A server that misses three heartbeats is gone. A server leaves the fleet explicitly after its drain.

public sealed class Stats(IFleet fleet)
{
    public string Describe() => string.Join(", ",
        fleet.Nodes.Select(n => $"{n.NodeId} {n.State} {n.Load}/{n.MaxLoad} duel:{n.Stat("queue:duel")}"));
}

Claims: where things are

A server claims the units it runs: every player in one of its rooms (player), every lobby code (lobby). Any server can ask where a unit is (LocateAsync). Claims of a server that crashed point nowhere as soon as it drops out of the registry, so nobody is sent to a dead server.

Handoffs: work that moves with a player

When a server sends a player elsewhere, it writes a handoff for the target server first — the queue place with the time already waited, or the lobby code and host role — and only then tells the client. The target takes it when the player arrives. A handoff is bound to its target server and expires after a minute.

Redirects

A redirect is a packet your game defines (IRoomGame.Redirect, see below). The client closes its connection, reconnects to the same public address with ?node=<id> (the parameter name is fleet:node-param), and says "rejoin". The server it lands on relays it to the named server, which finds the handoff (or the room) and carries on.

If your game returns null from Redirect (the default), nobody is ever moved: the fleet then only balances new connections through your load balancer.

The relay

FleetRelayMiddleware runs in front of every server's routes. A WebSocket request with ?node=<id> for another live server is relayed to that server's internal address, frame by frame, both ways, close handshake included. A request for this server, for an unknown server, or for one that does not answer is served locally — the client always gets a working connection.

The relayed request carries the client's IP address, scheme and host in headers signed with HMAC (fleet:secret, default the security key). The receiving server restores them (HttpContext.Connection.RemoteIpAddress is the client again), so rate limits, bans and IP-bound tickets keep working. Relay headers without a valid, fresh signature are removed and ignored.

What the Built-in Modules Do

A player who just arrived is not moved again for FleetMoveCooldownSeconds (3 s), so stale heartbeats cannot bounce players around.

Wiring a Game

Bind the room host to the server node and the fleet (both are services), and give your game a redirect packet:

[Service]
[Service(typeof(IWorldStepper))]
public sealed class GameServer : IWorldStepper
{
    private readonly RoomHost<MatchSim, Input, Player> _host;

    public GameServer(IServerNode node, IFleet fleet, IStepScheduler scheduler, MyGame game, MyTransport transport)
    {
        _host = new RoomHost<MatchSim, Input, Player>(game, transport, new RoomHostOptions(), fixedHz: 60)
            .Use(new MatchmakingModule<MatchSim, Input, Player>(game, new MatchmakingOptions(), Playlists.All))
            .Use(new LobbyModule<MatchSim, Input, Player>(game, new LobbyOptions()))
            .UseScheduler(scheduler)   // rooms on every core (step-workers)
            .UseServer(node)           // capacity and drain
            .UseFleet(fleet);          // gathering, rejoin from anywhere, handoffs
    }
    // StepMode, FixedHz, BeforeSteps/FixedStep/AfterSteps delegate to _host ...
}

public sealed class MyGame : IRoomGame<MatchSim, Input, Player> /* , IMatchmakingGame, ILobbyGame */
{
    public IPacketBase? Redirect(RoomRedirect r) => new SRedirect
    {
        Node = r.NodeId,          // reconnect with ?{r.NodeParam}={r.NodeId}
        Address = r.PublicAddress, // or straight here, when the server has its own address
        Reason = r.Reason,        // rejoin | queue | lobby | drain
    };

    public IPacketBase? ServerDraining() => new SServerDraining();
    // ...
}

The client side of a redirect:

socket.onmessage = (m) => {
  const msg = decode(m.data);
  if (msg.type === 'redirect') {
    nodeParam = msg.node;                 // keep it for later reconnects too
    socket.close(1000);
    socket = open(`${base}?ticket=${await newTicket()}&node=${encodeURIComponent(msg.node)}`, { hello: 'rejoin' });
  }
};

Backplanes

A server that cannot reach the backplane keeps serving on its own (the others drop out of its view) and rejoins the fleet when the backplane is back. It does not block start-up. Implement IFleetBackplane for another store.

Besides the fleet's own state, the backplane holds what must be counted once for all servers: connection tickets, HTTP rate limits and failure lockouts with store: backplane. They use IncrementAsync(key, ttl, by), an atomic counter whose expiry is set by its first increment. A custom backplane without it (the default throws NotSupportedException) keeps those counters in each server's memory.

Addresses

The HTTP host must be reachable by the other servers (0.0.0.0 in containers), or the internal address must point to a proxy in front of it.

Deployment Recipes

Docker Compose

services:
  redis:
    image: redis:7-alpine
  game:
    image: my-game-server
    deploy: { replicas: 3 }
    environment:
      ALTRUIST__SERVER__HTTP__HOST: "0.0.0.0"
      ALTRUIST__SERVER__FLEET__REDIS: "redis:6379"
  lb:
    image: nginx
    ports: ["443:443"]   # round-robin over game:8080, no sticky sessions needed

docker compose up --scale game=5 adds servers; they join on their first heartbeat.

Kubernetes

A Deployment behind a Service, the probes from Capacity, Drain & Health, terminationGracePeriodSeconds above drain:timeout, and an HPA on the load metric:

spec:
  terminationGracePeriodSeconds: 380
  containers:
    - name: game
      env:
        - { name: ALTRUIST__SERVER__FLEET__REDIS, value: "redis:6379" }
      readinessProbe: { httpGet: { path: /altruist/server/ready, port: 8080 } }
      livenessProbe:  { httpGet: { path: /altruist/server/live,  port: 8080 } }

Scaling down is safe: Kubernetes sends SIGTERM, the server drains (its queue and lobbies move, its rooms finish), then exits.

Failure Modes

Configuration Reference

altruist:
  server:
    fleet:
      redis: ""                 # Redis connection string: a shared backplane (default: in-memory)
      cluster: "default"        # servers with the same name form one fleet
      region: ""                # published with the heartbeat
      heartbeat: 2              # seconds; a server missing 3 heartbeats is gone
      internal-address: ""      # default: machine address + HTTP port; "{ip}:8000" style allowed
      public-address: ""        # optional; "{node-id}" placeholder
      node-param: "node"        # query parameter for reaching a given server
      secret: ""                # relay signature key (default: altruist:security:key)

See also: Capacity, Drain & Health, Match Rooms, Multi-Channel Servers (independent shards instead of one fleet).