Table of Content

• Overview
  • Measurements
• Test Cases
  • Brainfuck
    • bench.b
    • mandel.b
  • Base64
  • Json
  • Matmul
  • Primes
• Tests Execution
  • Environment
  • Using Docker
  • Manual Execution
    • Prerequisites
• Contribution
  • Makefile guide
    • Binary executables
    • Compiled artifacts
    • Scripting language
  • README update
  • Docker image update

Overview

The benchmarks follow the criteria:

• They are written as the average software developer would write them, i.e.

  • The algorithms are implemented as cited in public sources;
  • The libraries are used as described in the tutorials, documentation and examples;
  • The used data structures are idiomatic.

• The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.

• All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

• time spent for the benchmark execution (loading required data and code self-testing are not measured);
• memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
• energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM. Currently, only Intel CPU are supported via the powercap interface.

All values are presented as: median_{±median absolute deviation}.

UPDATE: 2024-11-13

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

• Verbose (default). Prints the output immediately.
• Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

Language	Time, s	Memory, MiB	Energy, J
Scala (Staged)	0.425±0.017	208.26±02.66 + 36.81±04.81	27.57±01.62
Racket (Staged)	0.885±0.000	105.98±00.09 + 0.00±00.00	34.27±00.03
Rust	1.012±0.000	2.00±00.00 + 0.00±00.00	42.99±00.30
C++/g++	1.095±0.002	3.38±00.00 + 0.00±00.00	45.08±00.23
C/gcc	1.111±0.001	1.50±00.00 + 0.00±00.00	46.73±00.25
D/gdc	1.118±0.002	6.75±00.00 + 0.00±00.00	48.63±00.23
C/clang	1.138±0.001	1.38±00.00 + 0.00±00.00	48.14±00.11
C++/clang++	1.166±0.000	2.88±00.00 + 0.00±00.00	49.24±01.17
D/ldc2	1.167±0.001	3.12±00.06 + 0.00±00.00	49.12±00.20
Nim/gcc	1.168±0.001	1.94±00.06 + 0.00±00.00	48.20±00.22
Java	1.188±0.000	42.25±00.11 + 0.81±00.06	48.12±00.33
V/gcc	1.198±0.001	2.12±00.00 + 0.00±00.00	47.85±00.12
Vala/gcc	1.210±0.001	5.38±00.06 + 0.00±00.00	50.36±00.52
Kotlin/JVM	1.227±0.002	44.92±00.09 + 0.88±00.12	51.69±00.13
Vala/clang	1.234±0.000	5.31±00.06 + 0.00±00.00	52.04±00.89
Go	1.247±0.001	3.62±00.06 + 0.00±00.00	51.44±00.38
Zig	1.266±0.003	1.25±00.00 + 0.00±00.00	53.30±01.05
Go/gccgo	1.288±0.000	23.88±00.12 + 0.00±00.00	54.12±00.51
C#/.NET Core	1.367±0.001	32.38±00.06 + 0.12±00.00	58.25±00.89
Julia	1.551±0.014	248.75±00.06 + 0.25±00.00	66.53±00.84
Nim/clang	1.582±0.000	2.12±00.00 + 0.00±00.00	64.79±00.92
F#/.NET Core	1.593±0.002	37.49±00.06 + 0.38±00.12	68.98±01.23
Crystal	1.635±0.004	3.44±00.06 + 0.00±00.00	68.71±00.54
OCaml	1.657±0.003	3.25±00.00 + 2.62±00.00	76.47±00.71
Chez Scheme	1.748±0.002	25.12±00.00 + 3.38±00.00	73.58±00.24
Racket	1.794±0.027	93.84±00.05 + 22.75±00.19	73.46±01.48
V/clang	1.891±0.019	2.06±00.06 + 0.00±00.00	82.57±01.03
C#/Mono	2.034±0.001	25.84±00.12 + 0.00±00.00	86.11±00.25
MLton	2.077±0.011	1.75±00.00 + 0.38±00.00	86.06±01.09
Scala	2.777±0.005	62.17±00.16 + 145.05±00.32	122.07±00.64
Node.js	3.150±0.005	46.78±00.00 + 6.12±00.00	131.99±01.28
Haskell (MArray)	3.236±0.003	4.12±00.00 + 4.12±00.00	134.34±00.29
D/dmd	3.328±0.001	3.50±00.00 + 0.00±00.00	125.22±00.34
Haskell (FP)	3.786±0.003	4.25±00.06 + 4.00±00.00	160.86±00.26
Ruby/truffleruby	4.669±0.152	201.88±00.69 + 814.31±49.81	225.04±10.28
Swift	5.593±0.008	17.38±00.00 + 0.00±00.00	210.94±02.31
Lua/luajit	5.687±0.024	2.62±00.00 + 0.00±00.00	234.94±01.49
Ruby/truffleruby (JVM)	5.884±0.028	390.44±02.98 + 468.57±35.67	289.91±02.92
Python/pypy	9.575±0.073	60.50±00.12 + 29.45±00.01	426.59±05.95
Idris	15.559±0.040	20.63±00.06 + 8.38±00.00	700.59±04.61
Elixir	20.530±0.047	71.27±00.45 + 0.00±00.00	816.30±01.74
Ruby (--jit)	31.953±0.050	21.47±00.07 + 4.88±00.00	1320.18±03.40
PHP	34.067±0.088	18.88±00.00 + 0.00±00.00	1442.70±16.86
Lua	37.027±0.060	2.62±00.00 + 0.00±00.00	1523.51±15.61
Python	54.076±0.669	11.88±00.00 + 0.00±00.00	2418.75±26.40
Ruby	71.836±0.205	11.31±00.06 + 0.00±00.00	3123.03±27.59
Ruby/jruby	79.903±1.524	203.90±02.51 + 222.73±32.77	3573.69±72.79
Tcl (FP)	193.372±0.720	4.62±00.12 + 0.00±00.00	8541.35±121.54
Perl	223.395±1.420	7.06±00.06 + 0.12±00.00	9755.30±45.94
Tcl (OOP)	382.403±3.408	4.62±00.12 + 0.12±00.00	16926.54±148.14

mandel.b

Mandel in Brainfuck

Language	Time, s	Memory, MiB	Energy, J
Scala (Staged)	7.751±0.157	211.17±04.45 + 132.95±12.75	465.58±11.30
C++/g++	10.076±0.018	3.50±00.00 + 0.62±00.00	416.16±01.64
C#/.NET Core	11.991±0.035	32.56±00.06 + 1.38±00.00	481.39±02.69
Java	12.441±0.041	42.18±00.16 + 2.18±00.07	500.84±03.37
C/gcc	12.597±0.005	1.62±00.00 + 0.25±00.00	509.44±01.23
C++/clang++	12.910±0.015	3.00±00.00 + 0.62±00.00	534.77±00.89
Kotlin/JVM	13.168±0.070	44.94±00.06 + 1.94±00.27	551.88±02.60
C/clang	13.205±0.013	1.62±00.00 + 0.25±00.00	581.89±04.42
Zig	13.314±0.028	1.38±00.06 + 1.00±00.00	558.65±04.76
F#/.NET Core	13.335±0.024	37.61±00.06 + 2.12±00.12	534.94±02.52
Go	14.113±0.014	3.62±00.06 + 0.00±00.00	568.61±02.43
Rust	14.242±0.020	1.88±00.00 + 0.25±00.00	569.99±01.08
D/ldc2	14.272±0.019	3.25±00.06 + 0.75±00.00	584.52±03.64
V/gcc	14.333±0.038	2.12±00.06 + 1.00±00.00	560.92±03.16
Vala/gcc	14.436±0.031	5.38±00.00 + 0.62±00.00	572.24±04.78
Racket (Staged)	14.758±0.159	105.70±00.14 + 71.38±01.87	584.52±06.50
Vala/clang	14.781±0.010	5.38±00.12 + 0.62±00.00	593.05±01.04
Crystal	14.942±0.019	3.50±00.00 + 0.25±00.00	631.98±02.50
Nim/gcc	15.017±0.014	2.06±00.06 + 1.50±00.00	608.83±01.49
D/gdc	15.433±0.016	6.75±00.00 + 0.88±00.00	642.43±03.76
Scala	16.477±0.112	62.29±00.09 + 144.88±00.19	717.49±03.80
Nim/clang	17.695±0.052	2.38±00.12 + 1.50±00.00	719.34±09.74
Swift	18.038±0.044	17.62±00.12 + 0.12±00.00	740.60±03.26
V/clang	18.356±0.084	2.12±00.06 + 1.00±00.00	806.70±06.57
Go/gccgo	18.686±0.314	24.06±00.06 + 0.00±00.00	773.76±12.67
OCaml	24.514±0.018	4.38±00.00 + 3.12±00.00	1178.94±06.82
Julia	30.494±0.068	248.88±00.00 + 0.25±00.00	1291.74±10.50
C#/Mono	31.075±0.053	25.81±00.15 + 0.75±00.00	1320.05±03.33
Chez Scheme	31.201±0.191	25.45±00.00 + 3.12±00.00	1372.33±16.13
Lua/luajit	31.900±0.061	2.50±00.06 + 0.25±00.00	1318.65±02.87
MLton	33.684±0.058	1.75±00.00 + 4.27±00.00	1527.54±10.09
Node.js	34.372±0.081	46.80±00.00 + 8.62±00.12	1380.19±03.68
Haskell (MArray)	35.178±0.039	4.00±00.06 + 5.12±00.00	1431.66±08.02
Racket	35.789±0.831	93.83±00.07 + 23.62±00.12	1591.23±45.98
D/dmd	37.954±0.004	3.38±00.06 + 0.88±00.00	1390.20±05.03
Python/pypy	41.239±0.111	60.38±00.12 + 30.13±00.05	1834.39±16.72
Ruby/truffleruby	48.778±0.439	201.31±00.94 + 734.44±18.06	2375.16±22.12
Ruby/truffleruby (JVM)	50.823±2.734	393.21±05.77 + 405.25±32.51	2255.45±41.59
Idris	66.743±0.120	21.82±00.00 + 9.25±00.00	2977.76±12.58
Haskell (FP)	78.765±0.269	4.00±00.00 + 76.00±00.00	3323.00±06.05

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Language	Time, s	Memory, MiB	Energy, J
C/clang (aklomp)	0.095±0.001	2.12±00.00 + 0.00±00.00	4.50±00.05
C/gcc (aklomp)	0.097±0.000	2.12±00.00 + 0.00±00.00	4.56±00.02
PHP	0.105±0.000	19.25±00.00 + 0.00±00.00	4.93±00.03
Go (base64x)	0.275±0.002	6.62±00.06 + 0.00±00.00	13.00±00.06
Node.js	0.663±0.004	45.19±00.02 + 40.96±00.16	29.74±00.50
Zig	0.703±0.000	1.75±00.00 + 0.12±00.00	26.41±00.10
Rust	0.875±0.000	2.38±00.00 + 0.12±00.00	36.15±00.19
C/clang	0.918±0.000	2.00±00.00 + 0.12±00.00	33.80±00.13
Nim/clang	0.944±0.002	2.81±00.06 + 4.94±00.06	37.54±00.31
Crystal	1.039±0.001	3.88±00.00 + 1.00±00.00	42.82±00.44
C/gcc	1.106±0.001	1.88±00.00 + 0.12±00.00	40.27±00.33
D/ldc2	1.179±0.001	3.75±00.00 + 3.38±00.00	48.73±00.37
Nim/gcc	1.347±0.001	2.62±00.00 + 4.62±00.00	54.54±00.35
Java	1.433±0.002	43.05±00.10 + 215.12±03.94	59.06±00.69
Ruby (--jit)	1.503±0.002	15.09±00.06 + 72.88±00.38	58.96±00.26
Scala	1.531±0.002	59.09±00.11 + 265.94±05.44	65.30±00.39
Kotlin/JVM	1.567±0.003	45.92±00.12 + 263.50±05.75	67.15±00.22
V/gcc	1.630±0.001	2.62±00.00 + 2379.12±01.50	59.57±00.39
Vala/gcc	1.645±0.001	5.75±00.00 + 0.12±00.00	63.19±00.43
Vala/clang	1.645±0.001	5.81±00.06 + 0.12±00.00	63.50±01.04
Go	1.659±0.005	4.25±00.00 + 0.00±00.00	68.14±00.65
C++/clang++ (libcrypto)	1.705±0.003	6.06±00.06 + 0.11±00.00	68.13±00.60
C++/g++ (libcrypto)	1.706±0.004	6.50±00.00 + 0.12±00.00	68.03±00.51
Ruby	1.738±0.002	11.75±00.06 + 39.74±00.39	68.37±00.46
Perl (MIME::Base64)	2.000±0.003	14.94±00.06 + 0.00±00.00	80.98±00.50
C#/.NET Core	2.494±0.039	33.85±00.03 + 17.63±01.91	97.82±01.66
F#/.NET Core	2.580±0.027	38.48±00.01 + 19.97±01.83	98.40±01.39
Tcl	2.753±0.005	5.12±00.00 + 0.12±00.00	114.23±00.59
Python	2.872±0.001	11.62±00.00 + 0.12±00.00	112.57±00.90
Julia	2.983±0.002	263.48±00.11 + 28.98±00.08	120.37±00.91
Go/gccgo	3.020±0.005	25.00±00.12 + 0.00±00.00	142.55±00.34
V/clang	3.248±0.001	2.75±00.00 + 2388.56±00.50	119.99±00.66
Python/pypy	3.249±0.009	60.25±00.12 + 31.31±00.05	143.38±01.06
Ruby/truffleruby (JVM)	3.443±0.007	389.16±05.72 + 294.50±17.44	173.84±01.07
Racket	3.903±0.006	93.95±00.11 + 25.88±00.12	155.48±00.43
D/gdc	3.920±0.001	7.06±00.06 + 3.75±00.06	162.11±01.78
D/dmd	4.020±0.002	3.62±00.00 + 3.38±00.00	164.62±00.95
C#/Mono	4.800±0.010	26.58±00.20 + 18.86±00.03	198.99±01.14
Ruby/jruby	6.210±0.070	196.29±01.99 + 144.32±10.97	273.42±04.78
Ruby/truffleruby	8.259±0.038	200.88±00.50 + 542.00±02.56	399.78±03.55
Perl (MIME::Base64::Perl)	10.394±0.088	16.25±00.06 + 0.21±00.03	457.56±03.07

Json

Testing parsing and simple calculating of values from a big JSON file.

Few notes:

• gason mutates input strings;
• simdjson requires input strings with batch of trailing zeros: a special zero padding for SIMD instructions;
• DAW JSON Link "NoCheck" skips some JSON structure correctness checks;
• DAW JSON Link, gason, default (not "Precise") RapidJSON, and D implementations except Mir-based have some inaccuracies in number parsing:
  • DAW JSON Link's number parsing issue
  • gason's number parsing issue
  • D stdlib number parsing issue

Json

Language	Time, s	Memory, MiB	Energy, J
C++/clang++ (simdjson On-Demand)	0.061±0.000	112.88±00.00 + 59.75±00.00	2.55±00.01
C++/g++ (simdjson On-Demand)	0.061±0.000	113.50±00.00 + 59.75±00.00	2.56±00.01
C++/clang++ (DAW JSON Link NoCheck)	0.071±0.000	112.71±00.00 + 0.00±00.00	2.95±00.03
C++/g++ (DAW JSON Link NoCheck)	0.083±0.000	113.21±00.00 + 0.00±00.00	3.39±00.02
C++/clang++ (DAW JSON Link)	0.092±0.000	112.77±00.06 + 0.00±00.00	3.84±00.03
C++/g++ (DAW JSON Link)	0.092±0.000	113.33±00.01 + 0.00±00.00	3.78±00.04
C++/clang++ (simdjson DOM)	0.103±0.001	112.81±00.06 + 174.75±00.81	4.67±00.06
C++/g++ (simdjson DOM)	0.105±0.001	113.38±00.00 + 172.62±00.31	4.82±00.05
Rust (Serde Custom)	0.107±0.000	111.62±00.00 + 0.00±00.00	4.54±00.03
Rust (Serde Typed)	0.112±0.000	111.62±00.00 + 12.12±00.00	4.73±00.07
C++/g++ (gason)	0.127±0.000	113.21±00.00 + 96.75±00.00	5.11±00.06
D/ldc2 (Mir Asdf DOM)	0.132±0.000	112.88±00.00 + 61.25±00.00	5.41±00.05
C++/clang++ (gason)	0.134±0.000	112.70±00.06 + 96.75±00.00	5.32±00.03
C++/g++ (RapidJSON)	0.144±0.002	113.34±00.06 + 125.54±01.73	6.15±00.12
Scala (jsoniter-scala)	0.153±0.001	279.53±00.11 + 26.48±00.73	8.43±00.17
Go (rjson custom)	0.194±0.000	113.94±00.06 + 0.00±00.00	7.52±00.02
C++/clang++ (RapidJSON)	0.195±0.001	112.83±00.00 + 128.62±00.00	8.20±00.03
C++/g++ (RapidJSON Precise)	0.202±0.001	113.34±00.00 + 128.75±00.00	8.61±00.08
Go (Sonic)	0.204±0.002	123.62±00.06 + 0.00±00.00	8.87±00.07
D/ldc2 (Mir Amazon's Ion DOM)	0.220±0.001	112.88±00.00 + 80.75±00.00	9.32±00.03
Go (rjson)	0.221±0.000	114.00±00.00 + 0.00±00.00	8.58±00.04
Zig	0.225±0.000	111.12±00.00 + 39.25±00.00	9.76±00.13
Go (goccy/go-json)	0.260±0.001	114.44±00.06 + 0.00±00.00	10.41±00.16
C++/clang++ (RapidJSON Precise)	0.262±0.001	112.71±00.00 + 128.75±00.00	11.06±00.14
C++/g++ (RapidJSON SAX)	0.351±0.001	113.21±00.00 + 0.00±00.00	15.53±00.10
C/clang (yajl)	0.362±0.001	111.31±00.06 + 0.00±00.00	15.71±00.27
C/gcc (yajl)	0.365±0.000	111.31±00.06 + 0.00±00.00	15.77±00.10
C++/g++ (Boost.JSON)	0.365±0.001	113.60±00.00 + 307.75±00.00	15.48±00.06
C++/clang++ (Boost.JSON)	0.375±0.002	112.96±00.12 + 307.75±00.00	16.06±00.10
Nim/clang (jsony)	0.382±0.001	112.06±00.06 + 154.12±00.00	16.10±00.29
Nim/gcc (jsony)	0.408±0.001	111.62±00.00 + 156.19±01.75	17.33±00.14
C++/clang++ (RapidJSON SAX)	0.417±0.001	195.08±00.00 + 0.00±00.00	17.70±00.04
C++/g++ (RapidJSON SAX Precise)	0.427±0.000	113.34±00.00 + 0.12±00.00	19.06±00.18
Node.js	0.437±0.001	155.54±00.06 + 201.31±03.75	21.57±00.11
C++/clang++ (RapidJSON SAX Precise)	0.514±0.000	195.08±00.00 + 0.12±00.00	22.66±00.11
Go (jsoniter)	0.542±0.001	114.25±00.12 + 0.00±00.00	22.00±00.31
C#/.NET Core (System.Text.Json)	0.549±0.002	490.00±00.08 + 140.62±00.19	24.45±00.29
Rust (Serde Untyped)	0.558±0.001	111.75±00.00 + 839.88±00.00	23.69±00.30
Java (DSL-JSON)	0.571±0.004	267.72±00.12 + 288.95±01.92	30.06±00.56
Python/pypy	0.614±0.000	280.79±00.01 + 125.21±00.00	26.83±00.15
V/clang	0.634±0.001	111.75±00.12 + 496.00±00.00	26.49±00.14
V/gcc	0.634±0.000	111.81±00.06 + 496.00±00.00	26.77±00.27
Crystal (Pull)	0.634±0.002	113.44±00.06 + 18.12±00.00	27.95±00.31
Crystal (Schema)	0.647±0.001	113.38±00.00 + 50.75±00.00	28.21±00.09
Nim/gcc (Packedjson)	0.673±0.001	112.00±00.00 + 294.19±00.06	28.78±00.44
Nim/clang (Packedjson)	0.675±0.001	112.38±00.12 + 294.25±00.00	29.45±00.42
CPython (UltraJSON)	0.720±0.002	123.90±00.00 + 476.31±00.88	28.65±00.08
Perl (Cpanel::JSON::XS)	0.772±0.006	125.44±00.06 + 402.88±00.00	32.07±00.29
PHP	0.805±0.002	128.44±00.06 + 517.88±00.00	34.83±00.19
Go	0.825±0.001	114.00±00.00 + 0.00±00.00	34.28±00.40
Python	0.895±0.004	121.73±00.06 + 325.88±00.00	37.83±00.56
Crystal	0.947±0.008	113.50±00.00 + 392.12±00.00	41.02±00.80
Nim/gcc	1.023±0.002	112.06±00.06 + 1001.25±00.00	42.77±00.18
C#/.NET Core	1.064±0.004	496.26±00.18 + 273.00±00.12	50.88±00.49
Nim/clang	1.091±0.002	112.38±00.00 + 999.00±00.00	45.69±00.46
Clojure	1.141±0.022	411.01±04.96 + 545.75±15.06	60.70±00.83
C++/g++ (json-c)	1.203±0.002	113.35±00.00 + 1215.88±00.00	50.53±00.81
C++/clang++ (Nlohmann)	1.205±0.001	112.89±00.06 + 359.88±00.00	50.78±00.19
C++/clang++ (json-c)	1.205±0.005	112.91±00.06 + 1215.88±00.00	50.10±00.29
Go/gccgo	1.227±0.002	139.38±00.06 + 0.00±00.00	50.43±00.12
C++/g++ (Nlohmann)	1.274±0.002	113.40±00.06 + 447.88±00.00	53.66±00.57
Ruby (--jit)	1.330±0.017	127.03±00.43 + 213.50±00.62	56.45±00.76
Ruby	1.360±0.004	121.25±00.00 + 212.75±00.00	57.49±00.53
F#/.NET Core (System.Text.Json)	1.498±0.006	498.96±00.06 + 231.38±04.12	68.65±00.71
D/ldc2	1.742±0.002	113.25±00.00 + 708.38±00.06	73.21±00.37
Ruby (YAJL)	1.745±0.005	121.38±00.12 + 219.06±00.06	73.96±00.69
C#/Mono	1.789±0.010	253.37±00.07 + 31.54±00.03	77.28±00.84
C3	1.962±0.004	111.88±00.00 + 795.88±00.00	81.93±00.35
Haskell	2.008±0.005	115.88±00.00 + 723.56±00.19	86.20±00.70
C++/g++ (Boost.PropertyTree)	2.515±0.009	113.40±00.06 + 1439.88±00.00	107.37±01.05
Rust (jq)	2.597±0.006	113.50±00.06 + 903.47±01.19	108.44±00.97
C++/clang++ (Boost.PropertyTree)	2.625±0.007	195.14±00.06 + 1232.62±00.00	112.22±00.99
Odin	2.824±0.003	111.44±00.06 + 20.00±00.00	117.93±00.75
Ruby/jruby	2.869±0.026	469.56±03.09 + 890.16±23.07	146.96±02.03
Vala/gcc	3.089±0.009	115.31±00.06 + 980.00±00.00	131.83±00.56
Vala/clang	3.094±0.006	115.38±00.00 + 980.00±00.00	132.11±00.59
D/dmd	3.101±0.003	113.38±00.00 + 708.56±00.06	133.40±01.11
D/gdc	3.493±0.022	116.62±00.06 + 708.75±00.00	148.35±00.78
Racket	3.819±0.025	320.69±00.42 + 225.56±00.12	159.99±01.22
Perl (JSON::Tiny)	9.297±0.067	126.00±00.00 + 528.69±00.01	409.51±01.80
Ruby/truffleruby (JVM)	10.154±0.186	483.81±04.05 + 2243.44±141.72	637.52±13.44
Ruby/truffleruby	10.192±0.089	398.88±02.12 + 1932.31±17.38	586.73±04.37

Matmul

Testing allocating and multiplying matrices.

Matmul

Language	Time, s	Memory, MiB	Energy, J
D/ldc2 (lubeck)	0.042±0.001	16.19±02.25 + 48.19±02.31	4.17±00.04
Python (NumPy)	0.066±0.001	39.59±02.13 + 52.95±02.12	5.61±00.04
Nim/gcc (Arraymancer)	0.073±0.002	10.69±03.00 + 53.19±03.06	5.81±00.10
Java (ND4J)	0.082±0.001	117.87±01.02 + 92.62±00.00	6.38±00.11
Rust (ndarray)	0.091±0.001	2.50±00.00 + 68.48±00.00	6.18±00.06
Nim/clang (Arraymancer)	0.112±0.027	20.44±02.25 + 44.06±02.25	7.99±01.36
Julia (threads: 2)	0.119±0.000	263.91±00.08 + 57.19±00.06	6.62±00.02
C++/g++ (Eigen)	0.147±0.000	4.21±00.00 + 85.54±00.00	7.23±00.09
C++/clang++ (Eigen)	0.147±0.000	4.75±00.00 + 85.42±00.00	7.31±00.03
Julia (threads: 1)	0.170±0.000	263.98±00.07 + 56.62±00.00	7.95±00.04
V/clang (VSL + CBLAS)	0.274±0.003	14.31±02.00 + 44.88±01.88	19.26±00.20
V/clang (VSL)	0.274±0.004	14.38±02.75 + 44.69±02.69	18.94±00.23
V/gcc (VSL + CBLAS)	0.469±0.006	12.94±02.44 + 46.31±02.38	35.29±00.27
V/gcc (VSL)	0.484±0.002	13.25±02.44 + 45.94±02.25	32.84±00.15
Julia (no BLAS)	1.110±0.018	263.50±00.00 + 51.88±00.00	48.27±00.89
D/gdc	1.504±0.000	7.12±00.06 + 4.00±00.00	56.83±00.29
D/ldc2	1.723±0.002	3.62±00.06 + 70.38±00.00	64.10±00.41
D/dmd	1.881±0.002	3.44±00.06 + 70.38±00.00	71.95±00.51
C/gcc	3.034±0.000	2.00±00.00 + 68.38±00.00	112.68±00.46
V/gcc	3.036±0.000	2.50±00.00 + 68.75±00.00	113.43±00.28
Vala/clang	3.061±0.001	5.66±00.03 + 68.38±00.00	107.97±01.66
Rust	3.064±0.001	2.25±00.00 + 68.50±00.00	107.36±01.14
V/clang	3.064±0.001	2.88±00.00 + 68.75±00.00	106.71±00.68
C/clang	3.066±0.000	2.00±00.00 + 68.38±00.00	105.48±00.31
Zig	3.068±0.002	1.75±00.00 + 68.62±00.00	109.29±00.23
Nim/gcc	3.094±0.002	2.50±00.00 + 57.88±00.00	115.51±00.14
Java	3.107±0.006	43.27±00.10 + 78.61±00.08	123.58±00.40
Swift	3.117±0.001	8.31±00.06 + 68.62±00.00	112.98±00.66
Nim/clang	3.121±0.001	2.88±00.12 + 57.88±00.00	108.74±00.94
Vala/gcc	3.130±0.001	5.53±00.03 + 68.50±00.00	115.26±00.63
Go	3.153±0.000	4.00±00.12 + 0.00±00.00	114.91±00.69
Crystal	3.154±0.001	4.00±00.06 + 59.50±00.00	113.86±01.39
Go/gccgo	3.160±0.000	24.38±00.06 + 0.00±00.00	112.72±00.89
Kotlin/JVM	3.197±0.002	43.74±00.09 + 78.32±00.17	124.82±00.65
Node.js	3.201±0.001	54.88±00.13 + 70.75±00.12	125.28±00.24
Python/pypy	3.263±0.002	61.00±00.00 + 68.75±00.12	136.35±00.22
Scala	3.295±0.004	61.33±00.10 + 150.06±00.06	121.09±00.77
C#/.NET Core	4.890±0.001	34.75±00.07 + 68.88±00.00	199.16±01.76
C#/Mono	7.394±0.002	26.13±00.15 + 69.62±00.00	307.51±01.02
Ruby/truffleruby	23.778±0.432	323.12±01.06 + 495.12±08.81	822.87±11.67
Ruby/truffleruby (JVM)	24.266±0.702	460.01±13.73 + 344.48±85.17	849.21±19.35
Python	131.064±0.887	11.94±00.06 + 68.62±00.00	6080.98±50.15
Perl	149.885±1.491	8.75±00.12 + 379.62±00.00	6523.98±63.31
Ruby (--jit)	153.769±0.077	21.87±00.03 + 68.44±00.06	6836.84±17.55
Tcl	205.038±0.931	7.50±00.00 + 400.25±00.00	9334.51±36.78
Ruby	213.180±0.222	11.62±00.00 + 69.25±00.00	9606.45±45.45
Ruby/jruby	378.246±8.095	269.65±12.79 + 1169.97±99.95	15729.25±399.84

Primes

Testing:

• generating primes using the optimized sieve of Atkin;
• prefix search for their decimal numbers using Trie data structure.

Notes:

• All languages but V and Python use unordered hashmaps (V and Python don't provide those out of box, and their hashmaps use keys in the insertion order);
• The results are always sorted (could be unstable or stable though).

Primes

Language	Time, s	Memory, MiB	Energy, J
Zig	0.060±0.000	1.50±00.00 + 47.88±00.25	2.52±00.02
C++/g++	0.064±0.001	3.62±00.12 + 85.86±00.56	2.50±00.02
Go	0.072±0.001	3.56±00.06 + 0.00±00.00	3.13±00.06
C++/clang++	0.075±0.000	3.12±00.12 + 64.66±00.12	2.85±00.03
V/clang	0.100±0.000	2.25±00.00 + 200.38±00.94	4.15±00.03
Rust	0.103±0.000	2.00±00.00 + 72.98±00.00	4.05±00.06
V/gcc	0.105±0.001	2.12±00.00 + 215.81±02.50	4.36±00.05
Java	0.136±0.003	41.95±00.12 + 126.09±05.43	7.73±00.17
Crystal	0.142±0.000	3.44±00.06 + 89.06±00.62	5.77±00.06
Scala	0.193±0.002	63.27±00.10 + 147.50±04.91	11.42±00.09
Node.js	0.195±0.000	44.05±00.00 + 147.14±00.26	10.22±00.04
Nim/gcc	0.290±0.002	1.75±00.00 + 596.00±04.81	11.18±00.16
Nim/clang	0.291±0.001	2.12±00.00 + 585.56±03.38	11.34±00.10
Lua/luajit	0.305±0.001	2.56±00.06 + 157.53±00.65	12.31±00.03
Julia	0.419±0.002	263.62±00.00 + 216.46±02.14	15.98±00.08
Python/pypy	0.636±0.003	60.00±00.12 + 248.82±00.07	26.09±00.26
Racket	0.751±0.001	111.73±00.41 + 246.77±00.46	29.66±00.18
Ruby/truffleruby	0.895±0.009	201.19±02.19 + 649.06±40.88	60.60±00.54
Ruby (--jit)	1.214±0.003	23.14±00.06 + 163.74±00.49	49.45±00.50
Lua	1.218±0.004	2.50±00.00 + 283.91±00.62	49.71±00.55
Ruby/truffleruby (JVM)	1.305±0.052	393.06±03.76 + 462.28±33.59	84.79±03.35
Ruby	1.964±0.007	11.38±00.06 + 172.75±00.31	82.09±00.37
Ruby/jruby	2.245±0.083	200.22±01.74 + 552.07±44.14	120.12±05.85
Python	2.275±0.007	11.62±00.00 + 172.88±00.69	98.93±01.08

Tests Execution

Environment

CPU: Intel(R) Xeon(R) E-2324G

Base Docker image: Debian GNU/Linux trixie/sid

Language	Version
.NET Core	8.0.403
C#/.NET Core	4.11.0-3.24468.6 (b4e5d1dd)
C#/Mono	6.12.0.200
C3	0.6.4
Chez Scheme	10.0.0
Clojure	"1.12.0"
Crystal	1.14.0
D/dmd	v2.109.1
D/gdc	14.2.0
D/ldc2	1.39.0
Elixir	1.14.0
F#/.NET Core	12.8.401.0 for F# 8.0
Go	go1.23.2
Go/gccgo	14.2.0
Haskell	9.8.2
Idris 2	0.6.0
Java	23.0.1
Julia	v"1.11.1"
Kotlin	2.0.21
Lua	5.4.6
Lua/luajit	2.1.1723681758
MLton	20210117
Nim	2.2.0
Node.js	v23.1.0
OCaml	5.2.0
Odin	dev-2024-11-nightly
PHP	8.2.24
Perl	v5.40.0
Python	3.12.6
Python/pypy	7.3.17-final0 for Python 3.10.14
Racket	"8.14"
Ruby	3.3.5p100
Ruby/jruby	9.4.8.0
Ruby/truffleruby	24.1.1
Rust	1.82.0
Scala	3.5.2
Swift	6.0.2
Tcl	8.6
V	0.4.8 2ab1523
Vala	0.56.17
Zig	0.13.0
clang/clang++	19.1.2 (1)
gcc/g++	14.2.0

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where <cmd> is:

• versions (print installed language versions);
• shell (start the shell);
• brainfuck bench (build and run Brainfuck bench.b benchmarks);
• brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
• base64 (build and run Base64 benchmarks);
• json (build and run Json benchmarks);
• matmul (build and run Matmul benchmarks);
• primes (build and run Primes benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a ./run.sh that could be used to simplify Docker usage:

• ./run.sh build (build the image);
• ./run.sh make versions (run the image with the versions command);
• sudo ./run.sh shell (run the image with the `shell' command, sudo is required to read energy levels).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

• Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

• NumPy for matmul tests (Debian package python3-numpy).
• UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

• Boost for JSON tests (Debian package libboost-dev).
• JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

• libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

• Coursier for downloading Maven artifacts.

For Haskell:

• network for TCP connectivity between the tests and the test runner.
• raw-strings-qq for raw string literals used in tests.

For Perl:

• cpanminus for installing modules from CPAN (Debian package cpanminus).

For Vala:

• JSON-GLib for JSON tests (Debian package libjson-glib-dev).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

• The test runner listens on localhost:9001;
• All messages are sent using TCP sockets closed immediately after the message has been sent;
• There are two messages sent from a test (it establishes the measurement boundary):
  1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
  2. The end message with any content (mostly it's "stop" for consistency).
• The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

• append a new target (the final binary location) into executables variable;
• append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

• append a new target (the final artifact location) into artifacts variable;
• append the proper target rule to compile the test;
• append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

• append run[<script_file>] into all_runners variable;
• append run[<script_file>] rule to run the test.

README update

TOC is regenerated using git-markdown-toc:

./run.sh toc