Who needs to define local variables and attributes?

[mohrmarc]

Hi,

I only recently came across ADIOS2, but find it quite fascinating. We have already successfully implemented an ADIOS2-based data exporter for data visualisaton with ParaView in our HyTeG FEM framework.

However, now I started working on a checkpoint/restore functionality and ran into strange problems which could be based on misconceptions on my side. Thus, I wanted to clarify some details that did not become fully clear to me after going through the documentation. I have some suspicions, but would like to cross-check.

• My first question is related to local variables. Do all MPI processes within the communicator passed to the top-level ADIOS2 object need to define a local variable? Or is it also possible that only a single process defines and puts a variable?
• Similarly w.r.t. attributes. From experience it seems alright, if all processes define a variable of the same name with the identical data/information. However, would it be preferable, like in the case of a variable for a "Global Single Value", that only one process does this?

I hope these questions aren't too trivial or that I just overlooked the answers.

Cheers
Marcus

[pnorbert]

Hi Marcus,
These are excellent questions.

Q1: Local (and global) arrays can be defined on a subset of processes of an MPI group that's used in an IO. Moreover, one process can contribute more than one block (with multiple Put() calls in a step).

Q2: The fact that you can define a Global Value, and any attribute on every processor, is a convenience. ADIOS has to filter out the duplicates at some point, either in writing or at reading, since only a single value is presented to the reader. Of course this has a performance and memory cost, not much in a few instances, but with thousands of attributes/global values, on thousands of processes, this could lead to an observable cost. In short, if you can, define attributes and global values on a single process, but you are allowed to do so on all processes.

Best regards
Norbert

[mohrmarc]

Dear Norbert,

thanks for answering my two questions. The way I have setup my checkpoint output at the moment the program will create a large number of local variables each of which is only defined on a single MPI process, which should be legal then. However, what I experience is that in some cases the resulting BP "file" does not contain the correct number of variables (each with one block), but instead fewer variables each or some of which contain multiple blocks.

I have compiled an MWE (see below) and can deterministically reproduce this. The program tries to create the number of variables given as input evenly distributed over the MPI ranks. I haven't found a clear connection between number of variables/per process and number of processes, but in general it typically works sequentially and/or for a small number of process, then stops working, and works fine again with large process counts.

Any thing that I might be doing wrong or could that be an issue on the ADIOS side?

Cheers
Marcus

#MPI processes	#total variables	#variables in BP file
1	3,000	3,000
2	3,000	1,500
3-10	3,000	3,000
1	4,000	4,000
2	4,000	4,000
3	4,000	2,667
4-6	4,000	4,000
103	4,000	4,000
1	20,000	20,000
2	20,000	10,000
3	20,000	13,334
8	20,000	2,500
9-10	20,000	20,000
103	20,000	20,000

#include <iostream>
#include <iomanip>
#include <mpi.h>
#include <adios2.h>

std::string createVariable( adios2::IO& io, int counter, int rank, long unsigned int bufferSize ) {
  std::stringstream sStream;
  sStream << "Var-" << std::setfill('0') << std::setw(3) << counter << std::setw(3) << "_Rank-" << rank;
  adios2::Variable< double > var = io.DefineVariable< double >( sStream.str(), {}, {}, {bufferSize} );
  return sStream.str();
}

int main( int argc, char ** argv ) {

  std::string outFile = "./checkpoint.bp";

  // -----------
  //  MPI setup
  // -----------
  MPI_Init( &argc, &argv );
  int commSize;
  int commRank;
  MPI_Comm_size( MPI_COMM_WORLD, &commSize );
  MPI_Comm_rank( MPI_COMM_WORLD, &commRank );

  if( commRank == 0 ) {
    std::cout << "Executing test with " << commSize << " MPI processes" << std::endl;
  }

  // determine number of variables to create
  if( argc != 2 ) {
    MPI_Abort( MPI_COMM_WORLD, -1 );
  }

  int numVarTotal = atoi( argv[1] );
  int numVarLocal = numVarTotal / commSize;

  if( commRank == 0 ) {
    std::cout << "Total variables asked for = " << numVarTotal << std::endl;
    std::cout << "Local variables to create = " << numVarLocal << std::endl;
    numVarLocal += numVarTotal - numVarLocal * commSize;
  }

  // -------------
  //  ADIOS setup
  // -------------
  adios2::ADIOS adios( MPI_COMM_WORLD );
  adios2::IO output = adios.DeclareIO( "Writer" );
  output.SetEngine( "BP5" );
  adios2::Engine writer = output.Open( outFile, adios2::Mode::Write );

  int counter = 0;

  unsigned long int bufferSize = 100u;
  std::vector< double > buffer( bufferSize, 2.0 );

  for( int k = 1; k <= numVarLocal; ++k ) {
    double* address = buffer.data();
    std::string var = createVariable( output, counter++, commRank, bufferSize );
    writer.Put< double >( var, address );
  }

  int numVariables = counter;
  MPI_Reduce( &counter, &numVariables, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD );
  if( commRank == 0 ) {
    std::cout << "Total variables created = " << numVariables << std::endl;
  }

  // ----------
  //  Shutdown
  // ----------
  writer.Close();
  MPI_Finalize();

}

[pnorbert]

@lizdulac @dmitry-ganyushin Can you please check this code? Thanks

[eisenhauer]

I've taken a quick look, but am unable to reproduce this behaviour. Unless I missed it, you don't specify exactly how you're determining the number of variables in the output file, but sampling some of the degrees of parallelism and output counts that you listed and using bpls as a means of counting variables I get the expected results. For example:

eisen@Endor build % mpirun -n 8 bin/Test 20000
Executing test with 8 MPI processes
Total variables asked for = 20000
Local variables to create = 2500
Total variables created = 20000
eisen@Endor build % bin/bpls *.bp | wc -l
bin/bpls *.bp | wc -l
   20000
eisen@Endor build % bin/bpls *.bp 
  double   Var-000_Rank-0   [1]*{100}
  double   Var-000_Rank-1   [1]*{100}
  double   Var-000_Rank-2   [1]*{100}
  double   Var-000_Rank-3   [1]*{100}
  double   Var-000_Rank-4   [1]*{100}
  double   Var-000_Rank-5   [1]*{100}
  double   Var-000_Rank-6   [1]*{100}
  double   Var-000_Rank-7   [1]*{100}
  double   Var-001_Rank-0   [1]*{100}
  double   Var-001_Rank-1   [1]*{100}
  double   Var-001_Rank-2   [1]*{100}
  double   Var-001_Rank-3   [1]*{100}
  double   Var-001_Rank-4   [1]*{100}
  double   Var-001_Rank-5   [1]*{100}
  double   Var-001_Rank-6   [1]*{100}
  double   Var-001_Rank-7   [1]*{100}
  double   Var-002_Rank-0   [1]*{100}
  double   Var-002_Rank-1   [1]*{100}
  double   Var-002_Rank-2   [1]*{100}
  double   Var-002_Rank-3   [1]*{100}

I don't have a good guess that might explain what you're seeing, but maybe if you tell us something about your read/verification method, we can go from there.

[mohrmarc]

Thanks for taking a look and you are right, I forgot to specify how I checked the number of variable. But it is as you assumed. I simply did a

bpls checkpoint.bp | wc -l

to obtain the numbers in the table above. However, originally I encountered the issue, when I tried to import an existing checkpoint and InquireVariable() was not able to find one of the variables that should have been in the BP file.

If it helps, I did run the tests for the table with the following setups

• clang 11.0.1 + OpenMPI 4.1.5 + Adios 2.9.0
• gcc 13.1.0 + OpenMPI 4.1.5 + Adios 2.9.0

In both cases the results look identical, i.e. as in the table. Examinging the contents of the BP file, e.g. for the mpirun -n 8 bin/Test 20000 case, bpls only finds variables with names generated by process 0

  double   Var-000_Rank-0   [8]*{100}
  double   Var-001_Rank-0   [8]*{100}
  double   Var-002_Rank-0   [8]*{100}
  double   Var-003_Rank-0   [8]*{100}
  double   Var-004_Rank-0   [8]*{100}
  double   Var-005_Rank-0   [8]*{100}
  double   Var-006_Rank-0   [8]*{100}
  double   Var-007_Rank-0   [8]*{100}
  double   Var-008_Rank-0   [8]*{100}
  double   Var-009_Rank-0   [8]*{100}
  double   Var-010_Rank-0   [8]*{100}
  ...

So in total the number of blocks (8x2,500) does correspond to the number of expected variables (20,000).

I'll next try to run the MWE test on another system to ensure that the problem does not result from the software stack on my test machine.

[eisenhauer]

Interesting... That kind of points to some kind of problem with the variable naming and somehow always using 0 instead of the appropriate rank. Not sure how that would happen from looking at the code, but it's something to look at.

BTW, it would be much more efficient in terms of ADIOS metadata overhead to use the same variable name across all ranks and instead pull the appropriate block out (block 0 will be from rank 0, block 1 from rank 1, etc. Assuming that all ranks write one block.)

[mohrmarc]

I just checked and I seem to not be able to reproduce the problem when I change the engine type from BP5 to BP4.

Concerning your comment. Yes, that is something I also had imagined. However, our application is such that each process holds a very small (typically 1 and maybe up to 5 at most) containers. For each container we need to store a larger (potentially in the low 2-digit range) number of data buffers. When the checkpoint is imported again we need to associate the data buffers with the corresponding containers again. The containers are associated to the mesh we use and, thus, cannot simply be recreated. They carry e.g. connecticvity information. So my first approach was to generate a variable for each data buffer composed of the unique container ID and a unique buffer ID. This makes importing them quite painless. We haven't tested, yet, whether this then is a showstopper performance-wise.

I am open for other suggestions, though. E.g. I was wondering, if it would help to work with the group API? Haven't figured out what that is intended for, yet. But would it make a difference to generate variable names as

containerID/bufferID

[eisenhauer]

I don't think the group API is likely to help because it just manipulates the names internally. How about this: Try changing the naming convention. There's some funky name manipulation that happens in BP5 to add internal information as well as the variable name and manipulate them as a set. It's possible that there's a bug there. Maybe instead of underscores or dashes, try different characters.

[mohrmarc]

The variable names indeed have quite an influence. Changing the MWE to just use <rank>+<counter> as string, for example, makes
the 20,000 variables case work with 8 MPI processes, but then fail with 9 procs:

==> bpls checkpoint.bp | wc -l
4446

==> bpls checkpoint.bp
  ...
  double   0+994   [1]*{100}
  double   0+995   [1]*{100}
  double   0+996   [1]*{100}
  double   0+997   [1]*{100}
  double   0+998   [1]*{100}
  double   0+999   [1]*{100}
  double   1+0     [8]*{100}
  double   1+1     [8]*{100}
  double   1+10    [8]*{100}
  double   1+100   [8]*{100}
  double   1+1000  [8]*{100}
  double   1+1001  [8]*{100}
  double   1+1002  [8]*{100}
  double   1+1003  [8]*{100}
  double   1+1004  [8]*{100}
  ...

So this might indeed be where the issue comes from. [If it is not my installion, that is.]

[eisenhauer]

Hmm. While there's always a possibility that there's a bug in the code that encodes and then decodes the variable name plus additional characteristics into a single string, I'm having trouble imagining a scenario where that bug would evidence itself in a non-deterministic way across platforms (like for you, but not for me running the same code). And the whole "works for 8, but not for 9" is bewildering. Have you tried printing out commRank on every rank? Maybe at the beginning and again at the end? I can't imagine that MPI_Comm_rank() is broken, but I don't have a good explanation for what is going on...

[mohrmarc]

Hi,

I managed to run my original mwe on two other systems:

1. CI of our gitlab instance with gcc 12.2.0 and OpenMPI 4.1.5
2. Linux Cluster of our compute center Leibniz Supercomputing Center with Intel oneAPI 2023.1.0 and Intel MPI 2021.9

In both cases I can reproduce the issue when using BP5:

#MPI processes	#total variables	#variables in BP file
2	20,000	10,000
3	20,000	13,334
4	20,000	5,000
5	20,000	4,000
6	20,000	6,668
7	20,000	5,715
8	20,000	2,500
9	20,000	20,000
10	20,000	20,000

And, yes, I agree that the behaviour looks very strange. But I am now quite confident that it's not a problem of my local software stack.

Cheers
Marcus

[mohrmarc]

Hi,

I installed ADIOS2 on Debian GNU/Linux 11 (bullseye) via Spack using the system compiler GCC 10.2.1 and system OpenMPI 4.1.0. However, the result of the 8 mpi procs, 20K vars test is not different from before. 2,5 K variables with 8 blocks in the BP5 file.

One observable difference, though, is that with the older compiler/MPI library the output only contains one data.? file. While with GCC 13.1.0 + OpenMPI-4.1.5 there are eight data.[0-7]. I ran the test on a single machine , so from the description of Aggregation in BP5 I'd only expect one file, correct?

Cheers
Marcus

[eisenhauer]

Well, let me install a debian 11 VM and see if I can reproduce it that way. If not, then probably the next step is for me to put some diagnostics into some branch that you can download and try. Kind of grasping at straws here...

[eisenhauer]

OK, with a debian 11 VM, gcc 10.2.1 and openmpi 4.1.0, I get perfectly normal results running your test code (and with the tests that I added to our testing trying to reproduce the situation). This was compiling and building ADIOS out of github master. I noticed that you mentioned spack above. I tend to avoid spack like the plague, mostly because I don't trust that I know what it's going to do. However, I'm initiating a spack build of ADIOS on this VM and I'll see what happens. In the meantime, you might try a non-spack build of ADIOS, either pulling out of master or snagging the latest release, 2.9.1 (https://github.com/ornladios/ADIOS2/releases/tag/v2.9.1).

[eisenhauer]

I have not yet gotten back to trying this with a spack build. Have you been able to try it with a build outside of spack?

[mohrmarc]

No problem. Was busy myself. I cloned ADIOS2 and built 2.9.1 on Debian 11/bullseye with the system compiler and libraries. Linking my mwe against these did not result in changed behaviour for me. I still get 2,500 variables with 8 blocks instead of 20,000 with one block.

Running cmake and make caused no issues, but only 97% tests passed, 51 tests failed out of 1646. Had no time to look into the details of this, yet. Being sceptical, though, I also checked on another system running Ubuntu 22.04.3 LTS (so that's GCC 11.4.0 and OpenMPI 4.1.2). There 100% of tests passed, but the behaviour for me remains the same.

[pnorbert]

Installed debian on WSL, which is Debian 12.2, gcc (Debian 12.2.0-14) 12.2.0, mpirun (Open MPI) 4.1.4 and saw the same problem.

Specifically, from 9776 variables, I get 1/8th of that but 8 blocks per variable. At 9768 variables, I get 9768 variables with 1 block each. Even if I rename the variables to xxx-_RRRR-.

If I increase the length of name by one extra character to xxxx-*, the issue start to show up at 9600 variables. With five x, the limit is 9432. With a single x in the name, the limit is 32320.

I printed the list of variables per process, and they generate different variable names as expected. I also changed the function to return the adios2::Variable variable instead of std::string and used that in Put() to make sure adios does not find the wrong variable based on a string.

[eisenhauer]

Debug or Release build? I've tried dozens of ways to make this happen, but totally failed. The MPI version shouldn't matter, but the compiler may be important.

[pnorbert]

Debug

…

On Fri, Oct 20, 2023, 8:55 PM Greg Eisenhauer ***@***.***> wrote: Debug or Release build? I've tried dozens of ways to make this happen, but totally failed. The MPI version shouldn't matter, but the compiler may be important. — Reply to this email directly, view it on GitHub <#3822 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYYYLOPQEU3XYALQKEOKSTYAMMRRAVCNFSM6AAAAAA5DLFFIOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGNBUGE4DA> . You are receiving this because you commented.Message ID: ***@***.***>

[eisenhauer]

Well, I still haven't been able to duplicate this anywhere, which is a bit of a mystery, but I'm sufficiently convinced that it's a hash collision problem to proceed with that as a fix. @mohrmarc , would you please clone https://github.com/eisenhauer/ADIOS2.git and try branch "ExtremeVars" in your situation? This branch changes the FFS internal format hash to SipHash. This isn't ready to be merged yet for several reasons. Most importantly, I expect that any change in the hashing algorithm to change the set of situations in which we see collisions, so just checking that things now work in some circumstance where they didn't before isn't sufficient. What I'd like to see is a bit more of a torture test, and I'd also like to run some numbers on the computational overhead of using a cryptographic hash as opposed to the non-crypto hash we've been using. That's going to take more time than I can spend at the moment, but perhaps this a stopgap solution.

[mohrmarc]

Hi,

I cloned your repo and built that version. I did not do extensive testing, but I can confirm that with this version the mwe works fine for the 8 procs / 20,000 vars test. I get BP5 output that contains 20,000 different variables each with a single block 🙂. This works both on Debian 11 and Ubuntu 22.04.

Thanks
Marcus

[eisenhauer]

I'm sufficiently convinced that the new hash won't have severe consequences and that it solves your issue without creating additional issues (as far as I can see so far), so I have PRs pending for both our master and release_29 branches. This should be in release 2.9.2 when it comes out (soon-ish).

[eisenhauer]

OK, I have an update here because @pnorbert gave me a circumstance where I could reproduce the failure and actually find the root cause. Circumstances were a bit more complicated than I thought. The root cause was an error in the calculation of the length of a data block over which we should perform a hash. That length was encoded into two separate 16-bit values, but when they were extracted and combined, the upper bits were shifted down by 16 (leaving zero) instead of up by 16 (where they should be). As a result we were only doing a hash over a length defined by the lower 16 bits of the actual length. This often happened to work out OK, but if the length mod 2^16 happened to be a small number, then we were only doing a hash over the first short bits of a block (and our metadata blocks include some standard elements at the beginning). So depending upon the actual length of the encoded thing we either guaranteed or had a high chance of a hash collision. Fixing it involved changing a ">> 16" to "<< 16". The confusing bit was that this error was spotted and fixed in the FFS subpackage months ago, and that version of FFS was merged with ADIOS master in July. However ADIOS 2.9 branched off in May and the FFS update was a new version, so it was inappropriate to include in an ADIOS patch release. Presumably this explains why I was never able to duplicate the problem when building ADIOS from github. Current ADIOS PR #3875 represents a fix and we still expect to get it merged into ADIOS release 2.9.2. I don't have good workaround for 2.9.0 and 2.9.1 releases.

That said, generating variable names that are different on each rank is not a good way to use ADIOS and it works against what BP5 is trying to exploit to reduce metadata overhead. (It's not the worst way. The worst way would be to generate variable names that changed with each timestep. But it is the 2nd worst way.) Happy to talk about alternatives...

[eisenhauer]

I'm going to close this and it can be reopened if there are further questions. The summary is that there was a bug in FFS where only the format body hash was calculated only over a portion of the format body, with that portion being determined by the value of the bottom 16 bits of the body length. This meant that for larger formats (many variables), the possibility of hash collision was higher. This bug exists in all versions of ADIOS through 2.9.x, but is not in current GitHub master and will not be in 2.10. This should only be a problem if you 1) are writing thousands of variables, and 2) change the variable names in a structured way across ranks (such as naming the variables based on the MPI rank. If either of these things are not true, your odds of a hash collision are still 1 in 2^62 or so. You also won't get a hash collision if the format body length mod 2^16 is bigger than 300 or so, but you have little control over this at the application level (except maybe by adding dummy variables).