Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix subscription benchmarks by fixing create_table_for_test_with_the_works #2027

Merged
merged 1 commit into from
Dec 5, 2024

Conversation

Centril
Copy link
Contributor

@Centril Centril commented Nov 28, 2024

Description of Changes

Basically replace with ::Refs as needed.

The numbers, just to set a baseline for any future work, are as of this PR:

Benchmarking full-scan: Collecting 100 samples in estimated 5.6924 s (200
full-scan               time:   [27.842 ms 28.019 ms 28.249 ms]

Benchmarking full-join: Collecting 100 samples in estimated 5.7658 s (25k
full-join               time:   [222.02 µs 222.63 µs 223.53 µs]

Benchmarking incr-select: Collecting 100 samples in estimated 5.0003 s (3
incr-select             time:   [153.60 ns 159.05 ns 169.53 ns]

Benchmarking incr-join: Collecting 100 samples in estimated 5.0028 s (7.0
incr-join               time:   [700.16 ns 704.12 ns 711.24 ns]

Benchmarking query-indexes-multi: Collecting 100 samples in estimated 5.0
query-indexes-multi     time:   [640.48 ns 643.12 ns 648.56 ns]

Fixes #1993

API and ABI breaking changes

No, this just fixes tests and benches.

@Centril Centril requested a review from kazimuth November 28, 2024 20:47
@Centril Centril force-pushed the centril/fix-sub-benches branch from 68857bc to d3c7265 Compare November 28, 2024 21:14
@bfops bfops added the release-any To be landed in any release window label Dec 2, 2024
@Centril Centril requested a review from RReverser December 4, 2024 18:58
@Centril
Copy link
Contributor Author

Centril commented Dec 5, 2024

benchmarks please

Copy link

github-actions bot commented Dec 5, 2024

Benchmarking failed. Please check the workflow run for details.

Copy link

github-actions bot commented Dec 5, 2024

Callgrind benchmark results

Callgrind Benchmark Report

These benchmarks were run using callgrind,
an instruction-level profiler. They allow comparisons between sqlite (sqlite), SpacetimeDB running through a module (stdb_module), and the underlying SpacetimeDB data storage engine (stdb_raw). Callgrind emulates a CPU to collect the below estimates.

Measurement changes larger than five percent are in bold.

In-memory benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6432 6396 0.56% 6616 6532 1.29%
sqlite 5609 5609 0.00% 6095 6015 1.33%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 74706 74672 0.05% 75186 75420 -0.31%
stdb_raw u32_u64_str no_index 64 128 2 string 116948 116914 0.03% 117686 117940 -0.22%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25117 25081 0.14% 25767 25771 -0.02%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24084 24048 0.15% 24656 24776 -0.48%
sqlite u32_u64_str no_index 64 128 2 string 144415 144415 0.00% 145823 145837 -0.01%
sqlite u32_u64_str no_index 64 128 1 u64 123763 123763 0.00% 125009 124907 0.08%
sqlite u32_u64_str btree_each_column 64 128 1 u64 131080 131080 0.00% 132514 132504 0.01%
sqlite u32_u64_str btree_each_column 64 128 2 string 134222 134222 0.00% 135818 135832 -0.01%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 880651 881594 -0.11% 931027 927476 0.38%
stdb_raw u32_u64_str btree_each_column 64 128 1029166 1029361 -0.02% 1090016 1053101 3.51%
sqlite u32_u64_str unique_0 64 128 399360 399366 -0.00% 416482 418334 -0.44%
sqlite u32_u64_str btree_each_column 64 128 984611 984611 0.00% 1020973 1023405 -0.24%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 138485 138372 0.08% 138679 138482 0.14%
stdb_raw u32_u64_str unique_0 64 15910 15797 0.72% 16096 15915 1.14%
sqlite u32_u64_str unique_0 1024 1042718 1042718 0.00% 1046074 1046038 0.00%
sqlite u32_u64_str unique_0 64 74704 74710 -0.01% 75842 75806 0.05%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50180 50282 -0.20%
64 bsatn 25509 25509 0.00% 27787 27719 0.25%
16 bsatn 8200 8200 0.00% 9628 9560 0.71%
16 json 12188 12188 0.00% 14092 14160 -0.48%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 20567742 20130299 2.17% 21220382 20588343 3.07%
stdb_raw u32_u64_str unique_0 64 128 1289582 1287061 0.20% 1365650 1320957 3.38%
sqlite u32_u64_str unique_0 1024 1024 1802137 1802137 0.00% 1811311 1811273 0.00%
sqlite u32_u64_str unique_0 64 128 128540 128540 0.00% 131468 131336 0.10%
On-disk benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6437 6401 0.56% 6617 6549 1.04%
sqlite 5651 5651 0.00% 6163 6091 1.18%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 74711 74677 0.05% 75155 75349 -0.26%
stdb_raw u32_u64_str no_index 64 128 2 string 116974 116919 0.05% 117684 117861 -0.15%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25121 25085 0.14% 25747 25743 0.02%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24089 24053 0.15% 24649 24709 -0.24%
sqlite u32_u64_str no_index 64 128 1 u64 125702 125684 0.01% 127316 127200 0.09%
sqlite u32_u64_str no_index 64 128 2 string 146336 146336 0.00% 148096 148026 0.05%
sqlite u32_u64_str btree_each_column 64 128 2 string 136418 136418 0.00% 138466 138444 0.02%
sqlite u32_u64_str btree_each_column 64 128 1 u64 133176 133176 0.00% 135034 134988 0.03%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 830790 830143 0.08% 880736 875167 0.64%
stdb_raw u32_u64_str btree_each_column 64 128 979612 980922 -0.13% 1039548 1037274 0.22%
sqlite u32_u64_str unique_0 64 128 416914 416914 0.00% 433562 435364 -0.41%
sqlite u32_u64_str btree_each_column 64 128 1023158 1023158 0.00% 1058568 1062324 -0.35%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 138490 138377 0.08% 138668 138467 0.15%
stdb_raw u32_u64_str unique_0 64 15915 15802 0.72% 16093 15892 1.26%
sqlite u32_u64_str unique_0 1024 1045796 1045796 0.00% 1049640 1049528 0.01%
sqlite u32_u64_str unique_0 64 76486 76486 0.00% 77808 77830 -0.03%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50180 50282 -0.20%
64 bsatn 25509 25509 0.00% 27787 27719 0.25%
16 bsatn 8200 8200 0.00% 9628 9560 0.71%
16 json 12188 12188 0.00% 14092 14160 -0.48%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 19056199 19059120 -0.02% 19762863 19627540 0.69%
stdb_raw u32_u64_str unique_0 64 128 1240874 1240513 0.03% 1315652 1306115 0.73%
sqlite u32_u64_str unique_0 1024 1024 1809785 1809791 -0.00% 1818535 1818339 0.01%
sqlite u32_u64_str unique_0 64 128 132687 132687 0.00% 135771 135643 0.09%

@Centril Centril added this pull request to the merge queue Dec 5, 2024
Merged via the queue into master with commit 6f22261 Dec 5, 2024
10 checks passed
@Centril Centril deleted the centril/fix-sub-benches branch December 5, 2024 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-any To be landed in any release window
Projects
None yet
Development

Successfully merging this pull request may close these issues.

subscription benchmark is broken
3 participants