Skip to content

Latest commit

 

History

History
626 lines (468 loc) · 25.7 KB

README.md

File metadata and controls

626 lines (468 loc) · 25.7 KB
Raw

Express 🚄

A lightning-fast networking library for Garry's Mod that allows you to quickly send large amounts of data between server/client with ease.


FYI: Please consider testing the next release, it has significant improvements over the base. Read more here: #37


Seriously, it's really easy! Take a look:

-- Server
local data = file.Read( "huge_data_file.json" )
express.Broadcast( "stored_data", { data } )

-- Client
express.Receive( "stored_data", function( data )
    file.Write( "stored_data.json", data[1] )
end )
Compared to doing it yourself... 
-- Server
-- This is just an example!
-- It doesn't handle errors or clients joining, and it doesn't support multiple streams

util.AddNetworkString( "myaddon_datachunks" )
local buffer = ""

local function broadcastChunk()
    if #buffer == 0 then return end

    local chunkSize, isLast = math.min( 63000, #buffer ), false
    buffer = string.sub( buffer, chunkSize + 1 )

    if #pending <= chunkSize then
        buffer, isLast = "", true
    end

    net.Start( "myaddon_datachunks" )
    net.WriteUInt( chunkSize, 16 )
    net.WriteData( string.sub( pending, 1, chunkSize ), chunkSize )
    net.WriteBool( isLast )
    net.Broadcast()
end

function BroadcastFile( filePath )
    local fileData = file.Read( filePath, "DATA" )
    buffer = util.Compress( fileData )
end

local interval = engine.TickInterval() * 8
timer.Create( "MyAddon_DataSender", interval, 0, broadcastChunk )

BroadcastFile( "huge_data_file.json" )
-- Client
local buffer = ""
net.Receive( "myaddon_datachunks", function()
    buffer = buffer .. net.ReadData( net.ReadUInt( 16 ) )
    if not net.ReadBool() then return end

    local datas = util.Decompress( buffer )
    processData( datas )
end )

In this example, huge_data_file.json could be in excess of 100mb (soon) 25mb post-compression without Express even breaking a sweat. The client would receive the contents of the file as fast as their internet connection can carry it.

GLuaTest GLuaLint

Details

Instead of using Garry's Mod's throttled (<1mb/s!) and already-polluted networking system, Express uses unthrottled HTTP requests to transmit data between the client and server.

Doing it this way comes with a number of practical benefits:

  • 📬 These messages don't run on the main thread, meaning it won't block networking/physics/lua
  • 💪 A dramatic increase to maximum message size (~100mb, compared to the net library's <64kb limit)
  • 🏎️ Big improvements to speed in many circumstances
  • 🤙 It's simple! You don't have to worry about serializing, compressing, and splitting your table up. Just send the table!

Express works by storing the data you send on Cloudflare's Edge servers. Using Cloudflare workers, KV, and D1, Express can cheaply serve millions of requests and store hundreds of gigabytes per month. Cloudflare's Edge servers offer extremely low-latency requests and data access to every corner of the globe.

By default, Express uses gmod.express, the public and free API provided by CFC Servers, but anyone can easily host their own! Check out the Express Service README for more information.

Usage

Examples

Broadcast a message from Server

-- Server
-- `data` can be a table of (nearly) any size, and may contain (almost) any values!
-- the recipient will get it exactly like you sent it
local data = ents.GetAll()
express.Broadcast( "all_ents", data )

-- Client
express.Receive( "all_ents", function( data )
    print( "Got " .. #data .. " ents!" )
end )

Client -> Server

-- Client
local data = ents.GetAll()
express.Send( "all_ents", data )

-- Server
-- Note that .Receive has `ply` before `data` when called from server
express.Receive( "all_ents", function( ply, data )
    print( "Got " .. #data .. " ents from " .. ply:Nick() )
end )

Server -> Multiple clients with confirmation callback

-- Server
local meshData = prop:GetPhysicsObject():GetMesh()
local data = { data = data, entIndex = prop:EntIndex() }

-- Will be called after the player successfully downloads the data
local confirmCallback = function( ply )
    receivedMesh[ply] = true
end

express.Send( "prop_mesh", data, { ply1, ply2, ply3 }, confirmCallback )


-- Client
express.Receive( "prop_mesh", function( data )
    entMeshes[data.entIndex] = data.data
end )

📖 Documentation

express.Receive( string name, function callback )

Description

This function is very similar to net.Receive. It attaches a callback function to a given message name.

Arguments

  1. string name
    • The name of the message. Think of this just like the name given to net.Receive
    • This parameter is case-insensitive, it will be string.lower'd
  2. function callback
    • The function to call when data comes through for this message.
    • On CLIENT, this callback receives a single parameter:
      • table data: The data table sent by server
    • On SERVER, this callback receives two parameters:
      • Player ply: The player who sent the data
      • table data: The data table sent by the player

Example

Set up a serverside receiver for the "balls" message:

express.Receive( "balls", function( ply, data )
    myTable.playpin = data

    if not IsValid( ply ) then return end
    ply:ChatPrint( "Thanks for the balls!" )
end )

express.ReceivePreDl( string name, function callback )

Description

Very much like express.Receive, except this callback runs before the data has actually been downloaded from the Express API.

Arguments

  1. string name
    • The name of the message. Think of this just like the name given to net.Receive
    • This parameter is case-insensitive, it will be string.lower'd
  2. function callback
    • The function to call just before downloading the data.
    • On CLIENT, this callback receives:
      • string name: The name of the message
      • string id: The ID of the download (used to retrieve the data from the API)
      • int size: The size (in bytes) of the data
      • boolean needsProof: A boolean indicating whether or not the sender has requested proof-of-download
    • On SERVER, this callback receives:
      • string name: The name of the message
      • Player ply: The player that is sending the data
      • string id: The ID of the download (used to retrieve the data from the API)
      • int size: The size (in bytes) of the data
      • boolean needsProof: A boolean indicating whether or not the sender has requested proof-of-download

Returns

  1. boolean:
    • Return false to halt the transaction. The data will not be downloaded, and the regular receiver callback will not be called.

Example

Adds a normal message receiver and a pre-download receiver to prevent the server from downloading too much data:

express.Receive( "preferences", function( ply, data )
    ply.preferences = data
end )

express.ReceivePreDl( "preferences", function( name, ply, _, size, _ )
    local maxSize = maxMessageSizes[name]
    if size <= maxSize then return end

    print( ply, "tried to send a", size, "byte", name, "message! Rejecting!" )
    return false
end )

express.ClearReceiver( string name )

Description

Removes the callback associated with the given message name. Much like net.Receive( message, nil ).

Arguments

  1. string name
    • The name of the message. Think of this just like the name given to net.Receive
    • This parameter is case-insensitive, it will be string.lower'd

Example

Create a new Receiver when the module is enabled, and remove the receiver when it's disabled

local function enable()
    express.Receive( "example", processData )
end

local function disable()
    express.ClearReceiver( "example" )
end

express.Send( string name, table data, function onProof )

Description

The CLIENT version of express.Send. Sends an arbitrary table of data to the server, and runs the given callback when the server has downloaded the data.

Arguments

  1. string name
    • The name of the message. Think of this just like the name given to net.Receive
    • This parameter is case-insensitive, it will be string.lower'd
  2. table data
    • The table to send
    • This table can be of any size, in any order, with nearly any data type. The only exception you might care about is Color objects not being fully supported (WIP).
  3. function onProof() = nil
    • If provided, the server will send a token of proof after downloading the data, which will then call this callback
    • This callback takes no parameters

Example

Sends a table of queued actions (perhaps from a UI) and then allows the client to proceed when the server confirms it was received. A timer is created to handle the case the server doesn't respond for some reason.

local queuedActions = {
    { "remove_ban", steamID1 },
    { "add_ban", steamID2, 60 },
    { "change_rank", steamID3, "developer" }
}

myPanel:StartSpinner()
myPanel:SetInteractable( false )
express.Send( "bulk_admin_actions", queuedActions, function()
    myPanel:StopSpinner()
    myPanel:SetInteractable( true )
    timer.Remove( "bulk_actions_timeout" )
end )

timer.Create( "bulk_actions_timeout", 5, 1, function()
    myPanel:SendError( "The server didn't respond!" )
    myPanel:StopSpinner()
    myPanel:SetInteractable( true )
end )

express.Send( string name, table data, table/Player recipient, function onProof )

Description

The SERVER version of express.Send. Sends an arbitrary table of data to the recipient(s), and runs the given callback when the server has downloaded the data.

Arguments

  1. string name
    • The name of the message. Think of this just like the name given to net.Receive
    • This parameter is case-insensitive, it will be string.lower'd
  2. table data
    • The table to send
    • This table can be of any size, in any order, with nearly any data type. The only exception you might care about is Color objects not being fully supported (WIP).
  3. table/Player recipient
    • If given a table, it will be treated as a table of valid Players
    • If given a single Player, it will send only to that Player
  4. function onProof( Player ply ) = nil
    • If provided, the client(s) will send a token of proof after downloading the data, which will then call this callback
    • This callback takes one parameter:
      • Player ply: The player who provided the proof

Example

Sends a table of all players' current packet loss to a single player. Note that this example does not use the optional onProof callback.

local loss = {}
for _, ply in ipairs( player.GetAll() ) do
    loss[ply] = ply:PacketLoss()
end

express.Send( "current_packet_loss", loss, targetPly )

express.Broadcast( string name, table data, function onProof )

Description

Operates exactly like express.Send, except it sends a message to all players.

Arguments

  1. string name
    • The name of the message. Think of this just like the name given to net.Receive
    • This parameter is case-insensitive, it will be string.lower'd
  2. table data
    • The table to send
    • This table can be of any size, in any order, with nearly any data type. The only exception you might care about is Color objects not being fully supported (WIP).
  3. function onProof( Player ply ) = nil
    • If provided, each player will send a token of proof after downloading the data, which will then call this callback
    • This callback takes a single parameter:
      • Player ply: The player who provided the proof

Example

Sends the updated RP rules to all players

RP.UpdateRules( newRules )
    RP.Rules = newRules
    express.Broadcast( "rp_rules", newRules )
end

🎣 Hooks

GM:ExpressLoaded()

Description

This hook runs when all Express code has loaded. All express methods are available. Runs exactly once on both realms.

This is a good time to make your Receivers (express.Receive).

Example

Creates the Express Receivers when Express is available

-- cl_init.lua

hook.Add( "ExpressLoaded", "MyAddon_SetupExpress", function()
    express.Receive( "MyAddon_ObjectData", function( data )
        processData( data )
    end )
end )

GM:ExpressPlayerReceiver( Player ply, string message )

Description

Called when ply creates a new receiver for message (and, by extension, is ready for both net and express messages)

Once this hook is called, it is guaranteed to be safe to express.Send to the player.

Arguments

  1. Player ply
    • The player that registered a new Express Receiver
  2. string message
    • The name of the message that a Receiver was registered for
    • (Note: This will be string.lower'd before calling this hook, so expect it to always be lowercase)

Example

Sends an initial dataset to the client as soon as they're ready

-- sv_init.lua

hook.Add( "ExpressPlayerReceiver", "MyAddon_InitData", function( ply, message )
    if message ~= "myaddon_initdata" then return end
    express.Send( "myaddon_initdata", MyAddon.CurrentData, ply )
end )
-- cl_init.lua

hook.Add( "ExpressLoaded", "MyAddon_SetupExpress", function()
    express.Receive( "MyAddon_InitData", function( data )
        processData( data )
    end )
end )

Performance

We tested Express' performance against two other options:

  • Manual Chunking:
    • This is a bare-minimum example script that serializes, compresses, and splits the data up across as few net messages as possible. (This is typically what people do in smaller addons.)
    • Source
  • NetStream:
    • This library is very popular. It's the go-to choice for sending large chunks of data. It's currently used by Starfall, PAC3, AdvDupe2, etc.
    • Source

Test Details

Test Setup

Our findings are based on a series of tests where we generated data sets filled with random elements across a range of data types. (string, int, float, bool, Vector, Angle, Color, Entity, table)

We sent this data using each of the options, one at a time.

These test were performed on a moderately-specced laptop. The server was a dedicated base-branch server run in WSL2. The client was base-branch clean-install run on Windows.

For each test, we collected two key metrics:

  • Duration: The total time (in seconds) it took to complete each test. This includes compression, serialization, sending, and acknowledgement.
  • Message Count: The number of net messages sent during the transfer. Fewer is usually better.

References:

  • This is an example of the data sets that we use during the test runs.
  • You can view the raw test setup here.
Detailed Test Results
Test 1 (74.75 KB):

Summary: This data can fit in only two net messages. In this situation, Express loses out to just sending net messages (by almost a full second).

Data Size Compressed Size
194.97 KB 74.75 KB
Method Duration (s) Messages Sent
Manual Chunking 1.265 2
NetStream 2.273 11
Express 1.909 1
Test 2 (374.78 KB):

Summary: Requiring at least six net messages when sent normally, Express sends the data about 3x faster.

Data Size Compressed Size
988.2 KB 374.78 KB
Method Duration (s) Messages Sent
Manual Chunking 6.160 6
NetStream 10.303 51
Express 2.151 1
Test 3 (1.5 MB):

Summary: After passing the "1 megabyte" mark, Express' advantages bein really shining through, beating the next fastest option by 21 seconds (8x faster!)

Data Size Compressed Size
3.97 MB 1.5 MB
Method Duration (s) Messages Sent
Manual Chunking 24.325 24
NetStream 40.849 200
Express 2.897 1
Test 4 (11.22 MB):

Summary: With a much larger payload, it becomes abundantly clear how slow and prohibitive the built-in net library can be. Express sends this 11mb payload in under 20 seconds, while the net library is nearing 200 seconds.

Data Size Compressed Size
29.67 MB 11.22 MB
Method Duration (s) Messages Sent
Manual Chunking 181.491 180
NetStream 304.552 1,485
Express 18.993 1
Test 5 (11.96 KB): Summary: Because this payload only requires a single net message, Express falls way behind of the pack in terms of transfer speed.
Data Size Compressed Size
29.79 KB 11.96 KB
Method Duration (s) Messages Sent
Manual Chunking 0.306 1
NetStream 0.833 3
Express 1.333 1

Test Result Takeaways

  • Express sends data significantly faster than both Manual Chunking and NetStream when the data size exceeds a certain threshold (Roughly whenever 3 or more net messages would be required).
  • Express only sends up to 2 net messages per transfer, no matter the size of the data.
  • Despite its impressive performance with large data sizes, Express is less efficient than other methods for smaller data sizes.
  • (NetStream is surprisingly slow, regardless of data size)

Extra Notes

  • These results will depend heavily on networking conditions. For some people, lots of smaller messages may actually perform better than one large Express download.
  • Anything that uses the built-in net library (like NetStream) will be more reliable than a library like Express, even if they may be slower overall.
  • Express caches sends. This means that if you needed to send a dataset to more than one player, Express would only need to upload the data once, saving a significant amount of time and bandwidth. These savings aren't reflected in this test run.

These tests illustrate how Express can significantly improve data transfer speed and efficiency for large or even intermediate-scale data, but may underperform when handling smaller data sizes.

Understanding the trade-offs of Express can help you determine if it's a good fit for your project.

Case Studies

Intricate ACF-3 Tank dupe 🔫

 Here's a clip of me spawning a particularly detailed and Prop2Mesh-heavy ACF-3 dupe (both Prop2Mesh and Adv2 use Netstream to transmit their data).
gmod_cL5uWh9hTu.mp4

A few things to note:

  • It took ~20 seconds for the dupe to be transferred to the server via Netstream
  • It took an additional ~20 seconds for the Prop2Mesh data to be Netstreamed back to me
  • On the netgraph, you can see the in and out metrics (and the associated green horizontal progress bar) that shows Netstream sending each chunk
  • Netstream only processes one request at a time. This is important, because it means while Adv2 or Prop2Mesh are transmitting data, no other player can use any Netstream-based addon until it completes.

Using some custom backport code, I converted Prop2Mesh and Advanced Duplicator 2 to use Express instead of Netstream. Here's me spawning the same tank in the exact same conditions, but using Express instead:

gmod_5RiCPGLfFA.mp4

The entire process took under 15 seconds - that's over 60% faster! My PC actually lagged for a moment because of how quickly all of the meshes downloaded and were available to render.

Even better? This doesn't block any other player from spawning their dupes! Because this is using Express instead of Netstream, other players can freely spawn their dupes, Prop2Mesh, Starfalls, etc. without being blocked and without blocking others.

Prop2Mesh + Adv2 stress test 🧪

 I had someone who knew more about Prop2Mesh than me create a highly complex controller. Here are the stats:

XngzjRoTlZ

Nearly 1M triangles across 162 models! If you've ever worked with meshes before, you'll know those are crazy high numbers.

When spawning this dupe in a stock server with Adv2 and Prop2Mesh, it takes nearly 4 minutes! All the while, blocking other players from using any Netstream-based addon. I can't even upload the video here because it's too big. Hopefully this screenshot is informative enough:

image

Some metrics:

  • It took 1 minute and 50 seconds before the dupe was even spawnable (it had to send the full dupe over to the server first)
  • After an additional 3 minutes, the meshes were finally downloaded and rendered
  • Again, while this was happening, no other player could use Adv2, Prop2Mesh, or Starfall

With that same backport code, forcing Adv2 and Prop2Mesh to use Express, the entire process takes under 30 seconds! That's almost a 90% speed increase.

gmod_3CairQAogv.mp4

Credits

A big thanks to @thelastpenguin for his super fast pON encoder that lets Express quickly serialize almost every GMod object into a compact message.