What resources are needed to visualize a large h5 file (~387 GB) on Perlmutter

[etpalmer63]

Hi VisIt Developers,

I'm helping a user running VisIt on Perlmutter. They're running into issues when trying to view a large h5 file. Following this process (https://docs.nersc.gov/applications/visit/) they tried with up to 4 nodes and 512 MPI tasks, but VisIt still seems to crash. Is there something we could do to confirm that this job needs additional resources? Or do people here have an estimate of what to expect in this kind of situation?

Any advice or ideas is appreciated!

Thanks,

Erik

[cyrush]

@etpalmer63 Thanks for reaching out.
It depend on the nature of the H5 file. What flavor of HDF5 is in use?

VisIt usually uses the same parallel domain decomposition of the code that create the dataset.

If you open the file with VisIt before plotting anything, you can use File Menu -> File Information to get some info about the data VisIt understands. Or you can create a plot w/o drawing and click on the small venn diagram. This will open up the Subset window, which will show you how many domains VisIt sees.

VisIt will distribute the domains across MPI tasks. If there is only 1 domain, using more tasks won't help.

[etpalmer63]

Thanks Cyrus! Let me check with the user and get back to you.

[etpalmer63]

In regards to the flavor of HDF5, I didn't and don't have access to the build. I can tell you the shared library it seems to be using is libhdf5.so.9.0.0 and libhdf5_hl.so.9.0.0. I wasn't sure how to extract more information of those files.

I'll ask the user to follow your instructions and report back.

Thank you!

[etpalmer63]

It's probably HDF5 version 1.8

[etpalmer63]

Update from the User:

There is only one domain. The simulation that generated this data file used 100 GPUs, so they were expecting 100 domains but got only one (see screenshot).

As for the flavor, they are using the "Vs" file type.

Thanks,

Erik

[cyrush]

Thanks for the update with these details. The Vs File format does support multiple blocks - so this may be an issue with choices in how file was written. Not sure what to suggest about the file, but this explains VisIt not being able to leverage extra MPI tasks.

[etpalmer63]

Do you have an estimate of how much memory you would expect a file like this to use?

[cyrush]

It's difficult to say. The HDF5 file could be using zlib compression, so the file size may not reflect the data arrays, they could be larger. Do we know how big the grid was, is a uniform grid or an unstructured grid, how many field variables there are?
If you have high memory nodes, trying one of those with a long time window would be what I suggest.

[Jwsembower]

Hi, I'm the user. The grid is uniform. Here is the h5ls dump info:

E Dataset {3, 6960, 1576, 1576}
derivedVariables Group
globalGridGlobal Group
globalGridGlobalLimits Group
runInfo Group
time Group

There are 3 field variables (x, y and z components).