Merge pull request #6 from lemois-1337/2023_stratum_vardiff_support

[Stratum][CI] Added vardiff support and Hive integration

README.md

@@ -9,19 +9,17 @@ macOS and HiveOS.
 
 * [srbminer](https://github.com/doktor83/SRBMiner-Multi/releases)
 
-No fee, forever.
-
 Discord discussions/issues: [here](https://discord.gg/pPNESjGfb5)
 
 Huge shoutout to https://github.com/KaffinPX/KStratum and
-https://github.com/onemorebsmith/karlsen-stratum-bridge for the
-inspiration.
+https://github.com/onemorebsmith/karlsen-stratum-bridge and
+https://github.com/rdugan/kaspa-stratum-bridge for the inspiration.
 
 Tips appreciated: `karlsen:qqe3p64wpjf5y27kxppxrgks298ge6lhu6ws7ndx4tswzj7c84qkjlrspcuxw`
 
 ## Hive Setup
 
-[detailed instructions here](hive-setup.md)
+[detailed instructions here](docs/hive-setup.md)
 
 # Features
 
@@ -34,23 +32,25 @@ Shares-based work allocation with miner-like periodic stat output:
  lemois         |       0.13GH/s |          3/0/0 |            0 |       6m48s
 -------------------------------------------------------------------------------
                 |       0.13GH/s |          3/0/0 |            0 |       7m20s
-========================================================= kls_bridge_v1.0.0 ===
+========================================================= kls_bridge_v1.1.0 ===
 ```
 
+## Variable difficulty engine (vardiff)
+
+Multiple miners with significantly different hashrates can be connected
+to the same stratum bridge instance, and the appropriate difficulty
+will automatically be decided for each one. Default settings target
+15 shares/min, resulting in high confidence decisions regarding
+difficulty adjustments, and stable measured hashrates (1hr avg
+hashrates within +/- 10% of actual). The minimum share difficulty is 64
+and optimized for GPUs.
+
 ## Grafana UI
 
 The grafana monitoring UI is an optional component but included for
 convenience. It will help to visualize collected statistics.
 
-[detailed instructions here](monitoring-setup.md)
-
-![Grafana Monitoring 1](images/grafana-1.png)
-
-![Grafana Monitoring 2](images/grafana-2.png)
-
-![Grafana KLSB Monitoring 1](images/grafana-3.png)
-
-![Grafana KLSB Monitoring 2](images/grafana-4.png)
+[detailed instructions here](docs/monitoring-setup.md)
 
 ## Prometheus API
 
@@ -87,48 +87,10 @@ kls_worker_job_counter{ip="192.168.0.65",miner="SRBMiner-MULTI/2.4.1",wallet="ka
 
 # Install
 
-## Docker (all-in-one)
-
-Note: This does requires that docker is installed.
-
-`docker compose -f docker-compose-all.yml up -d` will run the bridge with
-default settings. This assumes a local karlsend node with default port
-settings and exposes port 5555 to incoming stratum connections.
-
-This also spins up a local prometheus and grafana instance that gather
-stats and host the metrics dashboard. Once the services are up and
-running you can view the dashboard using `http://127.0.0.1:3000/d/x7cE7G74k/monitoring`
-
-Default grafana user: `admin`
-Default grafana pass: `admin`
-
-Most of the stats on the graph are averaged over an hour time period, so
-keep in mind that the metrics might be inaccurate for the first hour or
-so that the bridge is up.
-
-## Docker (non-compose)
-
-Note: This does not require pulling down the repo, it only requires that
-docker is installed.
-
-`docker run -p 5555:5555 karlsennetwork/karlsen_bridge:latest --log=false`
-
-This will run the bridge with default settings. This assumes a local
-karlsend node with default port settings and exposes port 5555 to incoming
-stratum connections.
-
-Advanced and customized configuration.
-
-`docker run -p {stratum_port}:5555 karlsennetwork/karlsen_bridge --log=false --karlsen={karlsend_address} --stats={false}`
-
-This will run the bridge targeting a karlsend node at {karlsend_address}.
-Stratum port accepting connections on {stratum_port}, and only logging
-connection activity, found blocks, and errors.
+## Build from source (native executable)
 
-## Non-Docker (manual build)
-
-Install go 1.18 using whatever package manager is appropriate for your
-system.
+Install go 1.18 or later using whatever package manager is approprate
+for your system, or from [https://go.dev/doc/install](https://go.dev/doc/install).
 
 ```
 cd cmd/karlsenbridge
@@ -150,3 +112,70 @@ cd cmd/karlsenbridge
 go build .
 ./karlsenbridge
 ```
+
+## Docker (all-in-one)
+
+Best option for users who want access to reporting, and aren't already
+using Grafana/Prometheus. Requires a local copy of this repository, and
+docker installation.
+
+[Install Docker](https://docs.docker.com/engine/install/) using the
+appropriate method for your OS. The docker commands below are assuming a
+server type installation - details may be different for a desktop
+installation.
+
+The following will run the bridge assuming a local karlsend node with
+default port settings, and listen on port 5555 for incoming stratum
+connections.
+
+```
+git clone https://github.com/karlsen-network/karlsen-stratum-bridge.git
+cd karlsen-stratum-bridge
+docker compose -f docker-compose-all-src.yml up -d --build
+```
+
+These settings can be updated in the [config.yaml](cmd/karlsenbridge/config.yaml)
+file, or overridden by modifying, adding or deleting the parameters in the
+`command` section of the `docker-compose-all-src.yml` file. Additionally,
+Prometheus (the stats database) and Grafana (the dashboard) will be
+started and accessible on ports 9090 and 3000 respectively. Once all
+services are running, the dashboard should be reachable at
+`http://127.0.0.1:3000/d/x7cE7G74k1/klsb-monitoring` with default
+username and password `admin`.
+
+These commands builds the bridge component from source, rather than
+the previous behavior of pulling down a pre-built image. You may still
+use the pre-built image by replacing `docker-compose-all-src.yml` with
+`docker-compose-all.yml`, but it is not guaranteed to be up to date, so
+compiling from source is the better alternative.
+
+## Docker (bridge only)
+
+Best option for users who want docker encapsulation, and don't need
+reporting, or are already using Grafana/Prometheus. Requires a local
+copy of this repository, and docker installation.
+
+[Install Docker](https://docs.docker.com/engine/install/) using the
+appropriate method for your OS. The docker commands below are assuming a
+server type installation - details may be different for a desktop
+installation.
+
+The following will run the bridge assuming a local karlsend node with
+default port settings, and listen on port 5555 for incoming stratum
+connections.
+
+```
+git clone https://github.com/karlsen-network/karlsen-stratum-bridge.git
+cd karlsen-stratum-bridge
+docker compose -f docker-compose-bridge-src.yml up -d --build
+```
+
+These settings can be updated in the [config.yaml](cmd/karlsenbridge/config.yaml)
+file, or overridden by modifying, adding or deleting the parameters in the
+`command` section of the `docker-compose-bridge-src.yml`
+
+These commands builds the bridge component from source, rather than the
+previous behavior of pulling down a pre-built image. You may still use
+the pre-built image by issuing the command `docker run -p 5555:5555 karlsennetwork/karlsen_bridge:latest`,
+but it is not guaranteed to be up to date, so compiling from source is
+the better alternative.

cmd/karlsenbridge/config.yaml

@@ -1,33 +1,64 @@
-# stratum_listen_port: the port that will be listening for incoming stratum traffic
-# Note `:PORT` format is needed if not specifiying a specific ip range 
+# stratum_listen_port: the port that will be listening for incoming stratum
+# traffic
+# Note `:PORT` format is needed if not specifiying a specific ip range
 stratum_port: :5555
 
 # karlsend_address: address/port of the rpc server for karlsend, typically 42110
-# For a list of public nodes, run `nslookup mainnet-dnsseed.daglabs-dev.com` 
+# For a list of public nodes, run `nslookup mainnet-dnsseed.daglabs-dev.com`
 # uncomment for to use a public node
 # karlsend_address: 46.17.104.200:42110
 karlsend_address: localhost:42110
 
-# min_share_diff: only accept shares of the specified difficulty (or higher) from 
-# the miner(s).  Higher values will reduce the number of shares submitted, thereby 
-# reducing network traffic and server load, while lower values will increase the
-# number of shares submitted, thereby reducing the amount of time needed for 
-# accurate hashrate measurements
-# min_share_diff: 4
+# min_share_diff: only accept shares of the specified difficulty (or higher)
+# from the miner(s).  Higher values will reduce the number of shares submitted,
+# thereby reducing network traffic and server load, while lower values will
+# increase the number of shares submitted, thereby reducing the amount of time
+# needed for accurate hashrate measurements.
+#
+# If var_diff is enabled, min_share_diff will be the starting difficulty.
+#
+# Default value is chosen to accomodate current top of the line GPUs. If you
+# don't want to change the default to match your device(s), the vardiff engine
+# will adjust to an appropriate diff for lower hashrate devices within a few
+# minutes.
+min_share_diff: 64
+
+# var_diff: if true, enables the auto-adjusting variable share diff mechanism.
+# Starts with the value defined by the 'min_share_diff' setting, then checks
+# every 10s whether each client is maintaining a 20 shares/minute submission
+# rate, and sends an updated min diff per client if necessary. Max tolerance
+# is +/- 5% after 4hrs.
+var_diff: true
+
+# shares_per_min: number of shares per minute the vardiff engine should target.
+# Default value is chosen to allow for 95% confidence in measurement accuracy,
+# which affects fidelity of difficulty update decisions, as well as hashrate
+# stability (measured 1hr avg hashrate should be within +/- 10% of actual, with
+# the noted confidence.)  Higher values will result in better vardiff engine
+# performance and increased hashrate stability.  Lower values will cause
+# vardiff to behave more erratically, while measured hashrate will display
+# larger variations.
+#
+# Example values and their resulting confidence levels:
+# 20 => 99%, 15 => 95%, 12 => 90%
+shares_per_min: 15
+
+# var_diff_stats: if true, print vardiff engine stats to the log every 10s
+var_diff_stats: false
 
 # block_wait_time: time to wait since last new block message from karlsend before
-# manually requesting a new block
-# block_wait_time: 500ms
+# manually requesting a new block. Examples are '500ms', '3s', '1m', etc.
+block_wait_time: 3s
 
-# extranonce_size: size in bytes of extranonce, from 0 (no extranonce) to 3. 
+# extranonce_size: size in bytes of extranonce, from 0 (no extranonce) to 3.
 # With no extranonce (0), all clients will search through the same nonce-space,
 # therefore performing duplicate work unless the miner(s) implement client
-# side nonce randomizing.  More bytes allow for more clients with unique 
-# nonce-spaces (i.e. no overlapping work), but reduces the per client 
+# side nonce randomizing.  More bytes allow for more clients with unique
+# nonce-spaces (i.e. no overlapping work), but reduces the per client
 # overall nonce-space (though with 1s block times, this shouldn't really
-# be a concern). 
+# be a concern).
 # 1 byte = 256 clients, 2 bytes = 65536, 3 bytes = 16777216.
-# extranonce_size: 0
+extranonce_size: 0
 
 # print_stats: if true will print stats to the console, false just workers
 # joining/disconnecting, blocks found, and errors will be printed
@@ -36,9 +67,9 @@ print_stats: true
 # log_to_file: if true logs will be written to a file local to the executable
 log_to_file: true
 
-# prom_port: if this is specified prometheus will serve stats on the port provided
+# prom_port: if specified, prometheus will serve stats on the port provided
 # see readme for summary on how to get prom up and running using docker
 # you can get the raw metrics (along with default golang metrics) using
 # `curl http://localhost:{prom_port}/metrics`
-# Note `:PORT` format is needed if not specifiying a specific ip range 
+# Note `:PORT` format is needed if not specifiying a specific ip range
 prom_port: :2114

cmd/karlsenbridge/main.go

@@ -6,7 +6,6 @@ import (
 	"log"
 	"os"
 	"path"
-	"time"
 
 	"github.com/karlsen-network/karlsen-stratum-bridge/src/karlsenstratum"
 	"gopkg.in/yaml.v2"
@@ -30,21 +29,17 @@ func main() {
 	flag.StringVar(&cfg.StratumPort, "stratum", cfg.StratumPort, "stratum port to listen on, default `:5555`")
 	flag.BoolVar(&cfg.PrintStats, "stats", cfg.PrintStats, "true to show periodic stats to console, default `true`")
 	flag.StringVar(&cfg.RPCServer, "karlsen", cfg.RPCServer, "address of the karlsend node, default `localhost:42110`")
-	flag.DurationVar(&cfg.BlockWaitTime, "blockwait", cfg.BlockWaitTime, "time in ms to wait before manually requesting new block, default `500`")
-	flag.UintVar(&cfg.MinShareDiff, "mindiff", cfg.MinShareDiff, "minimum share difficulty to accept from miner(s), default `4`")
+	flag.DurationVar(&cfg.BlockWaitTime, "blockwait", cfg.BlockWaitTime, "time in ms to wait before manually requesting new block, default `3s`")
+	flag.UintVar(&cfg.MinShareDiff, "mindiff", cfg.MinShareDiff, "minimum share difficulty to accept from miner(s), default `4096`")
+	flag.BoolVar(&cfg.VarDiff, "vardiff", cfg.VarDiff, "true to enable auto-adjusting variable min diff, default `true`")
+	flag.UintVar(&cfg.SharesPerMin, "sharespermin", cfg.SharesPerMin, "number of shares per minute the vardiff engine should target, default `15`")
+	flag.BoolVar(&cfg.VarDiffStats, "vardiffstats", cfg.VarDiffStats, "include vardiff stats readout every 10s in log, default `false`")
 	flag.UintVar(&cfg.ExtranonceSize, "extranonce", cfg.ExtranonceSize, "size in bytes of extranonce, default `0`")
 	flag.StringVar(&cfg.PromPort, "prom", cfg.PromPort, "address to serve prom stats, default `:2112`")
 	flag.BoolVar(&cfg.UseLogFile, "log", cfg.UseLogFile, "if true will output errors to log file, default `true`")
 	flag.StringVar(&cfg.HealthCheckPort, "hcp", cfg.HealthCheckPort, `(rarely used) if defined will expose a health check on /readyz, default ""`)
 	flag.Parse()
 
-	if cfg.MinShareDiff == 0 {
-		cfg.MinShareDiff = 4
-	}
-	if cfg.BlockWaitTime == 0 {
-		cfg.BlockWaitTime = 5 * time.Second // this should never happen due to kls 1s block times
-	}
-
 	log.Println("----------------------------------")
 	log.Printf("initializing bridge")
 	log.Printf("\tkarlsend:        %s", cfg.RPCServer)
@@ -53,6 +48,9 @@ func main() {
 	log.Printf("\tstats:           %t", cfg.PrintStats)
 	log.Printf("\tlog:             %t", cfg.UseLogFile)
 	log.Printf("\tmin diff:        %d", cfg.MinShareDiff)
+	log.Printf("\tvar diff:        %t", cfg.VarDiff)
+	log.Printf("\tshares per min:  %d", cfg.SharesPerMin)
+	log.Printf("\tvar diff stats:  %t", cfg.VarDiffStats)
 	log.Printf("\tblock wait:      %s", cfg.BlockWaitTime)
 	log.Printf("\textranonce size: %d", cfg.ExtranonceSize)
 	log.Printf("\thealth check:    %s", cfg.HealthCheckPort)

docker-compose-all-src.yml

@@ -0,0 +1,42 @@
+volumes:
+  prometheus_data: {}
+services:
+  kls_bridge:
+    build:
+      context: .
+      no_cache: true
+    container_name: kls_bridge
+    restart: unless-stopped
+    user: "0"
+    command:
+      - '-stats=false'
+      - '-karlsen=host.docker.internal:42110'
+    ports:
+      - 5555:5555
+      - 2114:2114
+    extra_hosts:
+      - host.docker.internal:host-gateway
+  grafana:
+    image: grafana/grafana-oss:latest
+    container_name: kls_grafana
+    restart: unless-stopped
+    user: "0"
+    volumes:
+      - ./docker/grafana:/var/lib/grafana
+#    env_file:
+#      - ./docker/grafana.env
+    ports:
+      - 3000:3000
+    extra_hosts:
+      - host.docker.internal:host-gateway
+  prometheus:
+    image: prom/prometheus:latest
+    container_name: kls_prom
+    restart: unless-stopped
+    volumes:
+      - prometheus_data:/prometheus
+      - ./docker/prometheus-internal.yml:/etc/prometheus/prometheus.yml
+    ports:
+      - 9090:9090
+    extra_hosts:
+      - host.docker.internal:host-gateway

docker-compose-all.yml

@@ -12,6 +12,8 @@ services:
       - '-stratum=:5555'
       - '-prom=:2114'
       - '-karlsen=host.docker.internal:42110'
+      - '-mindiff=64'
+      - '-vardiff=true'
     ports:
       - 5555:5555
       - 2114:2114

docker-compose-bridge-src.yml

@@ -0,0 +1,16 @@
+services:
+  kls_bridge:
+    build:
+      context: .
+      no_cache: true
+    container_name: kls_bridge
+    restart: unless-stopped
+    user: "0"
+    command:
+      - '-stats=false'
+      - '-karlsen=host.docker.internal:42110'
+    ports:
+      - 5555:5555
+      - 2114:2114
+    extra_hosts:
+      - host.docker.internal:host-gateway