Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Setup benchmark suite #332

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

[WIP] Setup benchmark suite #332

wants to merge 2 commits into from

Conversation

timkpaine
Copy link
Member

@timkpaine timkpaine commented Jul 13, 2024

xref: #38
Following @trallard's suggestion, exploring using asv to build a benchmark suite, starting with @AdamGlustein's code in #322 (comment).

Example output, running on just my machine through a few commits:

Screenshot 2024-07-13 at 18 50 25 Screenshot 2024-07-13 at 18 50 34

@timkpaine timkpaine added tag: wip PRs that are a work in progress - converted to drafts tag: internal Issues and PRs for maintainance of the project - not interesting to external users type: feature Issues and PRs related to new features labels Jul 13, 2024
@timkpaine timkpaine force-pushed the tkp/bm branch 2 times, most recently from d1969cc to a09002a Compare July 13, 2024 21:44
@@ -104,6 +104,11 @@ csp/lib/
*.so
*.tsbuildinfo

# Benchmarks
.asv
ci/benchmarks/*
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get some standardized machines before putting in raw data

csp/benchmarks/stats/basic.py Outdated Show resolved Hide resolved
csp/benchmarks/stats/basic.py Outdated Show resolved Hide resolved
csp/benchmarks/stats/basic.py Outdated Show resolved Hide resolved
@ptomecek
Copy link
Collaborator

I had missed the original discussion at #322 , that explains where a lot of this came from. Would be good to tidy up in any case. There are a whole bunch of benchmark tests that I want to write after...

@timkpaine timkpaine force-pushed the tkp/bm branch 2 times, most recently from f5b9763 to 8790a83 Compare July 14, 2024 01:04
@timkpaine
Copy link
Member Author

timkpaine commented Jul 14, 2024

Example from other PR:
Screenshot 2024-07-14 at 00 09 53

timkpaine added a commit that referenced this pull request Jul 15, 2024
Signed-off-by: Tim Paine <timothy.paine@cubistsystematic.com>
@AdamGlustein
Copy link
Collaborator

I had missed the original discussion at #322 , that explains where a lot of this came from. Would be good to tidy up in any case. There are a whole bunch of benchmark tests that I want to write after...

That benchmark script was just something I hacked together, I didn't intend for it to be a canonical example.
If we are making standard benchmarks then we should use a more general framework.

@timkpaine
Copy link
Member Author

If we are making standard benchmarks then we should use a more general framework.

I did a first pass approximation of a more general framework, extnded to the quantile function to demonstrate https://github.com/Point72/csp/blob/fb36b1b6f448d8c39aaab4edfb4b0ba7eed82ed7/csp/benchmarks/stats/basic.py

@timkpaine
Copy link
Member Author

@arhamchopra would be good to see if we can use our benchmark infrastructure to run csp benchmarks as well, and maybe fix some of the ugliness I introduced in this PR if we can.

timkpaine and others added 2 commits September 1, 2024 20:03
Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>
…ogic

Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tag: internal Issues and PRs for maintainance of the project - not interesting to external users tag: wip PRs that are a work in progress - converted to drafts type: feature Issues and PRs related to new features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants