Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Add SQLMetric to record the overhead of D2H memory copy before ShuffleWrite #11878

Closed
sperlingxx opened this issue Dec 16, 2024 · 0 comments · Fixed by #11882
Closed

[FEA] Add SQLMetric to record the overhead of D2H memory copy before ShuffleWrite #11878

sperlingxx opened this issue Dec 16, 2024 · 0 comments · Fixed by #11882
Assignees
@sperlingxx
Labels
feature request New feature or request

Comments

@sperlingxx
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Under many circumstance, the D2H MemCpy before ShuffleWrite could be very expensive, such as query stages with high-cardinality Aggregate or No Aggregate. With this metric, it will be helpful for performance inspection over the ShuffleWrite.
@sperlingxx sperlingxx added ? - Needs Triage Need team to review and classify feature request New feature or request labels Dec 16, 2024
@sperlingxx sperlingxx self-assigned this Dec 16, 2024
@sperlingxx sperlingxx changed the title [FEA] Add SQLMetrics to record the overhead of D2H memory copy before ShuffleWrite [FEA] Add SQLMetric to record the overhead of D2H memory copy before ShuffleWrite Dec 16, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Dec 16, 2024
@sperlingxx sperlingxx mentioned this issue Dec 17, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sperlingxx sperlingxx

Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add metrics GpuPartitioning.CopyToHostTime sperlingxx/spark-rapids
2 participants
@mattahrens @sperlingxx