Add metrics GpuPartitioning.CopyToHostTime #11882
Conversation
Signed-off-by: sperlingxx <lovedreamf@gmail.com>
sperlingxx
force-pushed
the
add_h2d_metrics
branch
from
db876e4 to
967d345
Compare
|
build |
|
build |
| // The SQLMetric key for MemoryCopyFromDeviceToHost | ||
| val CopyToHostTime: String = "d2hMemCpyTime" |
There was a problem hiding this comment.
This should be in GpuMetric along with the description. Copy to host time is not a metric specific to partitioning, and we should be consistent about it.
There was a problem hiding this comment.
Moved it into GpuMetric
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuPartitioning.scala
Outdated
Show resolved
Hide resolved
| new NvtxRange("PartitionD2H", NvtxColor.CYAN)) | ||
| // Wait for copyToHostAsync | ||
| withResource(memCpyNvtxRange) { _ => | ||
| Cuda.DEFAULT_STREAM.sync() |
There was a problem hiding this comment.
This is not the only time spent on copy to host. The copyToHostAsync calls above are not guaranteed to be asynchronous (e.g.: when the copy is from pageable memory, and we're not guaranteed to be using pinned memory). Therefore the metric and NVTX range needs to cover the copyToHostAsync calls above.
There was a problem hiding this comment.
I refined the code to wrap them all.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuPartitioning.scala
Outdated
Show resolved
Hide resolved
| @@ -132,7 +135,15 @@ trait GpuPartitioning extends Partitioning { | |||
| } | |||
| } | |||
| withResource(hostPartColumns) { _ => | |||
| Cuda.DEFAULT_STREAM.sync() | |||
| lazy val memCpyNvtxRange = memCopyTime.map( | |||
| new NvtxWithMetrics("PartitionD2H", NvtxColor.CYAN, _)) | |||
There was a problem hiding this comment.
NvtxWithMetrics has an apply that already does this for you.
withResource(NvtxRange("PartitionD2H", NvtxColor.CYAN, memCopyTime)) { _ =>
...
}
…ing.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com>
…ing.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com>
|
build |
Close #11878
This PR is to add the GpuMetric
GpuPartitioning.CopyToHostTime. SinceGpuPartitioningis a GpuExpression rather than a GpuPlan, a specialized methodGpuPartitioning.setupMetricswas created for the setup of detailed GpuPartitioning metrics during the planning time.During the local test, the newly-added metric works well.