Feature request: Simple WebSockets Support Peer Review Issue

[touhidurrr]

Before creating

• This is NOT a gameplay feature from Civ VI, BNW, or outside - see Roadmap
• This is NOT a gameplay feature from Vanilla Civ V or from G&K - If so, it should be a comment in Missing features from Civ V - G&K #4697

Problem Description

We need WebSocket to reduce network traffic. Everyone knows that. Now I am here to bother the devs to implement it. But fear not, I made a demo server that you can test and a demo spec sheet also!

Related Issue Links

No response

Desired Solution

The design goal is simple. The initial WebSocket support should be something that has 0 conflict with current multiplayer API's and should be able co-exist with it. There should be no need to upgrade to a higher version of Unciv or increment save file version for it to work. Unciv should be able use it if available and fallback to files API if not. But a connection over WebSocket should be preferred over polling over HTTP, that's all.

Current Design (Theory)

Unciv

1. Unciv tries to connect to the websocket server at ws://${multiplayerServerURL}/ws if the server url starts with http:// or wss://${multiplayerServerURL}/ws is if the server url starts with https://.
2. If a connection is possible then proceeds all with future communication on WebSocket.

Multiplayer Server

1. When a new connection is opened, parse userId from the Auth headers that Unciv already sends (This can be changed to query string if requested). Internally subscribe Unciv to a channel named user:${userId}. (This can be changed to game:${gameId} if requested)
2. When someone makes a turn. The server will publish its data to all playerId's in gameParameters.players. This eliminates the necessity for the player to let the server know of every game it has been playing every time it makes a connection. From client side, this is received like any other message as usual.
3. Support fetching an specific game data over WebSocket.
4. Support sending an specific game data over WebSocket.

Notes: All messages exchanged should be valid in JSON format.

Current Design (Schemas)

Unciv (Client)

// a simple ping for no reason whatsoever
// Status: Server: Implemented, Client: Unknown
{
  "type": "Ping"
}
// request game data of 1 game
// Status: Server: Implemented, Client: Unknown
{
  "type": "GameInfo"
  "data" {
    "gameId": "<Some UUIDv4>"
  }
}
// upload some game to the server
// Status: Server: Implemented, Client: Unknown
{
  "type": "GameUpdate"
  "data" {
    "gameId": "<Some UUIDv4>",
    "content": "<Some save. The format is the same as `GET /files/${gameId}` response>"
  }
}
// a list of all gameIds and lastUpdated timestamp when a client connects / reconnects to the server
// After receiving this message the server will send GameData messages for games with lastUpdated timestamp less than server last update timestamp.
// Status: Server: Implemented, Client: Unknown
{
  "type": "SyncGames"
  "data" {
    "lastUpdatedList": [
       // this is an array of objects of following type
      {
        "gameId": "<Some UUIDv4>",
        "lastUpdated": 0 // number, contains lastUpdated UTC timestamp, the time when this game was last updated by the client
    ]
  }
}

Multiplayer Server (Server)

// if ping then pong.
// Status: Implemented
{
  "type": "Pong"
}
// return 1 game data
// server can also send this message arbitrarily if a game update is available
// client should update the update a games save file when this is received
// Status: Implemented
{
  "type": "GameData"
  "data" {
    "gameId": "<Some UUIDv4>",
    "content": "<Some save. The format is the same as `GET /files/${gameId}` response>"
  }
}
// And error has occurred!
// Status: Implemented
{
  "type": "Error"
  "data" {
    "message": "Some error message"
  }
}
// Some validation error (sent by the schema validator of UncivServer.xyz)
// Status: Comes by default from Elysia
{
  "type": "validation",
  // arbitary params that describes the validation error
  // useful for debugging what went wrong
}

UncivServer.xyz WebSocket Support PR

touhidurrr/UncivServer.xyz#53

Test Server

Endpoint: https://ws.uncivserver.xyz
WebSocket: wss://ws.uncivserver.xyz/ws

Note: Initial connection to wss://ws.uncivserver.xyz/ws may take some time as Render aggressively makes the server down on inactivity.

Alternative Approaches

You suggest I change the spec sheet. Peer reviewed, there you go!

Additional Context

No response

[touhidurrr]

Ok. Here are my tests using Java-Websockets library.

package org.example

import kotlinx.serialization.ExperimentalSerializationApi
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.SerializationException
import kotlinx.serialization.json.ClassDiscriminatorMode
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonClassDiscriminator
import org.java_websocket.client.WebSocketClient
import org.java_websocket.drafts.Draft
import org.java_websocket.handshake.ServerHandshake
import java.net.URI

@Serializable
data class GameInfoData(val gameId: String)

@Serializable
data class GameData(val gameId: String, val content: String)

@JsonClassDiscriminator("type")
@Serializable
sealed class Message

@Serializable
@SerialName("Ping")
data object PingMessage : Message()

@Serializable
@SerialName("GameInfo")
data class GameInfoMessage(val data: GameInfoData) : Message()

@Serializable
@SerialName("GameUpdate")
data class GameUpdateMessage(val data: GameData) : Message()

@Serializable
data class ErrorData(val message: String)

@JsonClassDiscriminator("type")
@Serializable
sealed class Response

@Serializable
@SerialName("Pong")
data object PongResponse : Response()

@Serializable
@SerialName("GameData")
data class GameDataResponse(val data: GameData) : Response()

@Serializable
@SerialName("Error")
data class ErrorResponse(val data: ErrorData) : Response()

@OptIn(ExperimentalSerializationApi::class)
val json = Json {
    // otherwise 'type' field is not included in serialized string
    classDiscriminatorMode = ClassDiscriminatorMode.ALL_JSON_OBJECTS
}

class EmptyClient : WebSocketClient {
    constructor(serverURI: URI) : super(serverURI)
    constructor(serverUri: URI, draft: Draft) : super(serverUri, draft)
    constructor(serverURI: URI, httpHeaders: Map<String, String>) : super(serverURI, httpHeaders)

    override fun onOpen(handshakedata: ServerHandshake?) {
        println("new connection opened")

        // try sending some messages
        send(json.encodeToString(PingMessage))
        send(json.encodeToString(GameInfoMessage(GameInfoData("b78948eb-452f-42ea-8425-93dab002bcdc"))))
    }

    override fun onClose(code: Int, reason: String, remote: Boolean) {
        println("closed with exit code $code additional info: $reason")
    }

    override fun onMessage(message: String) {
        try {
            when (val response: Response = json.decodeFromString(message)) {
                is PongResponse -> println("Received Pong")
                is GameDataResponse -> println("GameData: ${response.data}")
                is ErrorResponse -> println("Error: ${response.data.message}")
                else -> println("Unknown response: $message")
            }
        } catch (e: SerializationException) {
            println("Failed to deserialize message: $message")
            e.printStackTrace()
        }
    }

    override fun onError(ex: Exception) {
        System.err.println("an error occurred: $ex")
    }
}

fun main() {
    val url = "wss://ws.uncivserver.xyz/ws"
    val headers = mapOf("Authorization" to "Basic ZjNiMGE0OWEtMjkyMC00OTg0LWE5YmUtOTk0OWNkMmIxYzA4Cg==")
    val client = EmptyClient(URI(url), headers)
    client.connect()
}

[yairm210]

This looks fine, however I think practically it won't make much of a difference. Unciv isn't an "always open" app, turns take several minutes, and so chances are that the app is closed at the time of the update.

[touhidurrr]

This looks fine, however I think practically it won't make much of a difference. Unciv isn't an "always open" app, turns take several minutes, and so chances are that the app is closed at the time of the update.

You can query the statuses of existing games once when you first connect / reconnect via WebSocket. This should mitigate such issues. Maybe we should retain preview functionality for this. (GameInfo, GameUpdate & GameData accepting preview gameId requests).

[touhidurrr]

The demo code was for my testing. Any support should add functionality for reconnecting after connection close. Which may happen often in Android.

[touhidurrr]

Note: Initial connection to wss://ws.uncivserver.xyz/ws may take some time as Render aggressively makes the server down on inactivity.

[touhidurrr]

Ok, after much deliberation, I have come to several conclusions:

1. No _Preview files via WevSocket
2. Need some other way to ensure that the client is always synced with server when it reconnects and the information exchanged is plausibly minimum.

Thus I have included 1 more client message type, which is SyncGames.

{
  "type": "SyncGames"
  "data" {
    "lastUpdatedList": [
       // this is an array of objects of following type
      {
        "gameId": "<Some UUIDv4>",
        "lastUpdated": 0 // number, contains lastUpdated UTC timestamp, the time when this game was last updated by the client
    ]
  }
}

After receiving this message the server will send GameData messages for games with lastUpdated timestamp less than server last update timestamp. Specs updated.

[touhidurrr]

@yairm210 need review!!

[yairm210]

Not sure what to tell you, looks probably fine? I'm not a websockets guy
Let's try it and see