Integration with HPCToolkit

[adayton1]

Are there bindings to export Caliper annotations to HPCToolkit? If not, it would be very helpful to have (and my guess is that it would be fairly simple to add).

[adayton1]

@daboehme, is there any tutorial on how to add a service? I've looked at the following pages, and so far all have been too abstract or just a list of APIs rather than a tutorial to actually aid in implementing a service:

http://software.llnl.gov/Caliper/CaliperBasics.html#third-party-tool-support-nvidia-nsight-intel-vtune
http://software.llnl.gov/Caliper/ThirdPartyTools.html
http://software.llnl.gov/Caliper/workflow.html
http://software.llnl.gov/Caliper/services.html
http://software.llnl.gov/Caliper/DeveloperGuide.html

[adayton1]

The hpctoolkit interface is really simple:

hpctoolkit_sampling_start();
hpctoolkit_sampling_end();
hpctoolkit_sampling_is_active();

Basically, I'm looking to add a service that will work with region filtering so that hpctoolkit will only profile the regions I care about. http://software.llnl.gov/Caliper/RegionFiltering.html

Even if you could point me to a service that I could start from, that would be helpful. There's just so many and they don't all implement the same interface, so it's hard to know where to start from.

[daboehme]

Hi @adayton1, I don't have a good tutorial here, but for this type of service you can look at the nvtx, roctx, and vtune services. Essentially you'll derive from the AnnotationBinding class and overwrite the on_begin, on_end, and service_tag methods. Shouldn't be too difficult. The AnnotationBinding class takes care of the filtering. I'm happy to help if you run into any issues.

[adayton1]

@daboehme, I'll give that a try. Thanks!

[adayton1]

@daboehme, can you explain the difference between a properly nested attribute and a non-nested attribute?

https://github.com/LLNL/Caliper/blob/master/src/services/nvtx/Nvtx.cpp#L122

[daboehme]

@daboehme, can you explain the difference between a properly nested attribute and a non-nested attribute?

Hi @adayton1 , the nested attributes form the "main" region hierarchy. The regions created by the Caliper annotation macros all fall into this category. You can create additional non-nested attributes that can overlap the main hierarchy, but it's a relatively rarely used feature. By default the AnnotationBinding class only forwards nested attributes.

[adayton1]

Is a service class a singleton? Do I have to worry about thread safety in on_begin and on_end?

[adayton1]

Also, is any error checking done behind the scenes (e.g. CALI_MARK_END("meh") where there was no CALI_MARK_BEGIN("meh"))?

[daboehme]

Hi @adayton1, the service class gets one object for each Caliper channel where it's active, but there is typically only one. In fact it doesn't make sense to have a binding service active multiple times. The on_begin/on_end functions can be invoked from multiple threads depending on how the target code is instrumented, so yes a good implementation should make sure to be thread safe. And yes, Caliper does check if the regions match with the CALI_MARK_BEGIN/END macros and error out if they don't.

[adayton1]

I'm not sure the AnnotationBinding class is going to work. NVTX and RocTX have a stack like API. HPCToolkit does not - it just has start and stop. So on an exclude region, I would need to call stop on the begin marker and potentially start on the end marker if an outer level is supposed to be timed. But if I understand correctly, on_begin and on_end for my service won't even be called for an excluded region. Or am I misunderstanding something?

[daboehme]

Correct, on_begin and on_end won't be called for excluded regions. But you're right, you'd have to make sure to just start/stop once at the correct stack level.