Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Python example #444

Merged
merged 11 commits into from
Oct 24, 2024
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,3 +124,17 @@ These primitives are only defined when the result itself is a pure rational.

Use double quotes to get a quote: `"Foo "" Bar"` is `Foo " Bar`.
No primitives defined.

## Benchmarks
saulshanabrook marked this conversation as resolved.
Show resolved Hide resolved

We run all of our "examples" [as benchmarks in codspeed](https://codspeed.io/egraphs-good/egglog). These are in CI
for every commit in main and for all PRs. It will run the examples with extra instrumentation added so that it can
capture a single trace of the CPU interactions ([src](https://docs.codspeed.io/features/understanding-the-metrics/)):

> CodSpeed instruments your benchmarks to measure the performance of your code. A benchmark will be run only once and the CPU behavior will be simulated. This ensures that the measurement is as accurate as possible, taking into account not only the instructions executed but also the cache and memory access patterns. The simulation gives us an equivalent of the CPU cycles that includes cache and memory access.

Since many of the shorter running benchmarks have unstable timings due to non deterministic behavior, we ["ignore"](https://docs.codspeed.io/features/ignoring-benchmarks/)
saulshanabrook marked this conversation as resolved.
Show resolved Hide resolved
them in codspeed. That way, we still capture their performance, but their timings don't show up in our reports by default.

We use 50ms as our cutoff currently, any benchmarks shorter than that are ignored. This number was selected to try to ignore
saulshanabrook marked this conversation as resolved.
Show resolved Hide resolved
any benchmarks with have changes > 1% when they haven't been modified.
3 changes: 1 addition & 2 deletions benches/example_benchmarks.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,7 @@ pub fn criterion_benchmark(c: &mut Criterion) {
for entry in glob::glob("tests/**/*.egg").unwrap() {
let path = entry.unwrap().clone();
let path_string = path.to_string_lossy().to_string();
// Skip python_array_optimize since it is too slow and doesn't even reflect the current python implementation
if path_string.contains("fail-typecheck") || path_string.contains("python_array_optimize") {
if path_string.contains("fail-typecheck") {
continue;
}
let name = path.file_stem().unwrap().to_string_lossy().to_string();
Expand Down
Loading
Loading