Bevygap – Multiplayer server management tools

This is a (WIP) toolkit for running multiplayer game servers on Edgegap, where the games are built with Bevy using the Lightyear networking library.

The goal is to have an easy-to-deploy system either yourself with docker-compose, or in the cloud, to use Edgegap to spin up gameservers on demand.

My testbed for this is bevygap-spaceships – a modified stand-alone version of the spaceships example from the Lightyear repo.

Live Demo

Is the live demo of bevygap-spaceships working? Hopefully!

Documentation / Quickstart

Documentation by book: The Bevygap Book

mdbook serve -o book/ if you want to view it locally.

Connection flow

• Game client talks to bevygap_matchmaker_httpd via websocket, asking to play, and ultimately expecting a response including a ConnectToken and gameserver ip:port.
• This makes a nats request to the matchmaker service
• An edgegap session is created, which corresponds to a gameserver instance (which may be autodeployed in response to this session creation)
• Matchmaker generates a Lightyear ConnectToken, and associates the token's ClientId with the edgegap session ID in NATS KV.
• Client uses the ConnectToken to connect to the gameserver
• Gameserver maintains a list of active connections in NATS KV, which the matchmaker watches.
• When a client disconnects, the gameserver removes their entry from active_connection, causing the matchmaker to delete the Edgegap session.

Components

bevygap_matchmaker

Clients wishing to connect to the game make a request to our matchmaker service, which then:

• Creates an Edgegap session (which may trigger a new server deployment)
• Creates a new Lightyear client id and connect token, associated with the session
• Stores the token/session mapping in NATS KV
• Returns the connect token, and gameserver IP and port to the client. (the gameserver ip+port will be a machine controlled by Edgegap, running your game server's docker image)
• Deletes edgegap sessions when clients disconnect, by watching active_connections in NATS KV

bevygap_matchmaker_httpd

An http endpoint to make "i want to play" requests to the matchmaker. The matchmaker itself only exposes a service to NATS, not http.

bevygap_client_plugin

A bevy plugin for the game client, to replace the normal lightyear commands.connect_client() call. The new commands.bevygap_connect_client() function will make a request to the matchmaker, then modify lightyear's config to set the supplied game server socket address and connect token, then call commands.connect_client() for you.

bevygap_server_plugin

A bevy plugin for the gameserver, which loads its deployment context from the edgegap API on boot, and connects to our NATS instance in order to lookup session information.

bevygap_shared

Shared code for some protocol and NATS stuff, used between the matchmaker and gameserver.

nats

NATS is the shared state and messaging backend between our various components.

edgegap-client

Autogenerated client for the edgegap API using openapi-generator. See gen-edgegap-client.sh.

bevy_nfws

a basic bevy websocket client, used to talk to the matchmaker.

NOTES / TODO

probably want to prefix all NATS subjects/buckets with the edgegap app name or something, so that multiple apps on edgegap can share the nats instance without conflicts.

should probably be restricted by the nats creds

WebTransport custom certificate requirements

https://w3c.github.io/webtransport/#dom-webtransportoptions-servercertificatehashes

https://w3c.github.io/webtransport/#verify-a-certificate-hash

Strict cert requirements, we autogenerate one on boot and report the digest to the client, so as long as the server doesn't stay up for more than 14 days, we're fine.

Chrome can't do CORS to http://localhost from http://lan-machine it seems, lan-machine would need to serve https. https://stackoverflow.com/questions/66534759/cors-error-on-request-to-localhost-dev-server-from-remote-site

Access to fetch at 'http://127.0.0.1:3000/wannaplay' from origin 'http://lan-machine:8008' has been blocked by CORS policy: The request client is not a secure context and the resource is in more-private address space local.

TODO

add httpd to ly servers that respond like the MM, generating a cert etc? httpd example from bevy_remote: https://github.com/bevyengine/bevy/blob/main/crates/bevy_remote/src/http.rs

move cert gen to LY?

implement the ConnectionRequestHandler properly to reject invalid connections

move matchmaker bevy/http bits into matchmaker crate too? flag to run bemw server and mm service in one binary, using a thread for mm?

proto id and ly pkey needed in docker-compose for MM stuff too, perhaps a move to a standard lightyear.env file which lightyear itself reads too? could default to sane zeros defaults if missing, for dev.

edgegap notes

• env vars limited to 255 bytes. hacky util script added to pass ca_contents as docker runtime arg
• can't use latest tag for container images, it's cached aggressively.