HDF5 Backend

The HDF5 backend allows simulation developers to write simulation results into HDF5 data format  files asynchronously. HDF5 backend, can be used in two different modes via Damaris, which are:

  1. File-Per-Core: In this mode, simulation results in each node are aggregated by dedicated cores and stored asynchronously at the end of each iteration.
  2. Collective I/O: In this mode, all simulation results are written into one single file for each iteration using Parallel HDF5. Although having one single file makes the data post-processing more convenient, synchronizing all the processes to write on a single file is typically more costly.

Below, you can find more about how to configure Damaris to use the HDF5 backend.

Configuration

To start using the HDF5 capability, you should first compile Damaris with HDF5 support. To do this, you need an installed version of the HDF5 library. Then you can initialize the HDF5_ROOT variable in the CMake command line to configure Damaris like this:

1

cmake <other options> -DENABLE_HDF5=ON -DHDF5_ROOT=/path/to/hdf5/installation/folder

Please note that Collective mode requires an HDF5 library that has been compiled with MPI support. Many Linux distributions have a version of the HDF5 library with this support available through their package manager (apt, dnf/yum, pacman, zypper etc.) and supercomputing centers will typically provide a MPI/IO enabled version via their modules system. To compile HDF5 with MPI/IO support from source please see the HDF website

With the compilation of Damaris with HDF5 support being successful, you should define your storage in the Damaris XML configuration file. To do so, you use the <storage> tag at the root of your XML file. An example <storage> tag is shown here:

1
2
3
4
5
6
7

<storage>
    <store name="MyStore" type="HDF5">
        <option key="FileMode">FilePerCore</option>
        <option key="XDMFMode">NoIteration</option>
        <option key="FilesPath"></option>
   </store>
</storage>

Inside the <storage> tag, you should define another tag, called <store>. This tag and all of its attributes define a storage backend, e.g. HDF5 backend. For each <store> tag you should define two important attributes:

  • Name: That is an arbitrary name that you should put for your store.
  • Type: For HDF5 backend, it should be HDF5. In future and for defining new storage backend, you should put proper names here.

Each storage backend may have its own parameters to configure the underlying storage technology. For HDF5, these options are available currently:

  • FileMode: That either can be set to FilePerCore or Collective. As described earlier, in the former case each dedicated core writes to its own file, but in the later all the dedicated cores are writing into a single HDF5 file at the end of each iteration.
  • XDMFMode: Not integrated to this version yet. But setting this option to NoIteration, FirstIteration or LastIteration configure Damaris in such a way that build the XDMF file on different times during running of the simulation.
  • FilesPath: represents the path in which the result HDF5 files should be stored.
    N.B. Paths require a single forward slash at the end to denote a directory e.g. <option key="FilesPath">this/is/my/path/</option> and the path must exist before the first file is written.

After defining a <store> tag in the XML file, you need to specify that a variable of your simulation is stored using this store type. For example the variable space in the definition below is configured to be stored in the MyStore storage.

1
2

 <variable name="space" type="scalar" layout="cells" mesh="mesh" 
    centering="nodal" store="MyStore" />

In the case that the collective mode is selected, for each variable, in addition to the local dimensions of the data, the global dimensions should be determined as well. As an example, the below layout:

1
2
3
4

<parameter name="WIDTH"  type="int" value="16" />
<parameter name="HEIGHT" type="int" value="16" />
<layout name="cells"  type="int" dimensions="WIDTH/size+3,HEIGHT+5" 
    global="WIDTH,HEIGHT" ghosts="2:1,3:2" />

shows that although a 16×16 grid is going to be processed by simulation, but each process is working with a smaller dataset. The height of this local dataset is equal to 16+5 (5, i.e. 2+3, is the total length of ghost zones in both directions) and its width is equal to to 16/size+3 (3, i.e. 1+2 is the total length of ghost zones). As mentioned earlier, size is the number of clients. To see more examples of HDF5 storage, you can check this page.

The Collective mode requires more effort from the programmer as it implies that each block of a variable is part of a larger (collective) dataset. So, each variable block has a place in a larger (possibly multi-dimensional) whole. To specify where the block belongs requires offsets to be specified using the damaris_set_position() or damaris_set_block_position() API calls. These calls provide Damaris with the information of where the variable block data starts in a varaiable, which is needed to join the data together as a whole in the HDF5 file.

File Names

There are some points about storing simulation variables in HDF5 files:

  • At the end of each iteration, the values of variables are written in separate HDF5 files. If the HDF5 backend is configured to store collectively, one file is created for each iteration. In the file-per-dedicated-core mode, one file is created for each dedicated core at the end of each iteration.
  • If a variable is empty in some iterations, the variable is not written to the files. For example, the coordinate variables are usually written at the first iteration. In this case, these variables are appeared only in the files of the first iteration.
  • If no data is written in an iteration, no file will be created for that iteration.

In addition, for naming the created files, Damaris HDF5 backend uses the simulation name (defined in the name attribute of the <simulation> tag) as the base and adds some postfixes at the end of the file. Here are some examples:

  • In the collective mode, only one file is created at the end of each iteration, so the name of the file will be simulation_ItXX.h5, wherein the simulation is the name of the simulation defined in the XML file and XX is the iteration number kept by Damaris. For example, multi-physics_It550.h5 is the name of a file for a simulation, namely multi-physics and on its 55th iteration, defined in the Damaris configuration XML file as <simulation name=”multi-physics” … >.
  • In the file-per-dedicated-core mode, one file will be created at the end of each iteration for each dedicated core. The files are created with a pattern like simulation_ItXX_PrYY.h5. Like the collective case, simulation and XX represent the simulation name and the iteration number. In addition, YY represents the rank of the dedicated process that has written its own variables into the file.

File Structure

According to the configuration of the XML file, the names and format of the HDF5 files may vary. For the examples below, we suppose that a 16×16 mesh is processes by 8 clients and 4 dedicated cores, using a total of 12 processes running on a node.

  • In the collective I/O case, the HDF5 file will have the same name as the simulation name and contain a dataspace, as large as the dimensions defined in the layout of the variable (In the global attribute as described above). So, the dataspace of the mentioned example will be something like this:
1
2
3
4
5
6

GROUP "/" {
   DATASET "space" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SIMPLE { ( 16, 16 ) / ( 16, 16 ) }
   }
}
  •  In the file-per-dedicated-core mode with one domain per process (domain = 1), the HDF5 will contain a group with the name of the stored variable and then a set of groups with the name PZ under the variable group. In this case, Z represents the client rank that has sent this block of memory to the dedicated core. As an example the dataspace for one of the created files could be like this:
1
2
3
4
5
6
7
8
9
10
11
12

GROUP "/" {
   GROUP "space" {
      DATASET "P0" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 2, 16 ) / ( 2, 16 ) }
      }
      DATASET "P1" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 2, 16 ) / ( 2, 16 ) }
      }
   }
}
  •  In the file-per-dedicated-core mode with more than one domain per process (domain > 1), the structure of the HDF5 will be the same as the previous case. The only difference is that PZ elements are created as groups, and new datasets are created for each block, e.g. a dataset BT is created in which B stands for block and T represents the block number. For example, if the clients write data twice in each iteration (domain =2 and therefore blocks =2), then the file structure will be like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

GROUP "/" {
   GROUP "space" {
      GROUP "P0" {
         DATASET "B0" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 2, 8 ) / ( 2, 8 ) }
         }
         DATASET "B1" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 2, 8 ) / ( 2, 8 ) }
         }
      }
      GROUP "P1" {
         DATASET "B0" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 2, 8 ) / ( 2, 8 ) }
         }
         DATASET "B1" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 2, 8 ) / ( 2, 8 ) }
         }
      }
   }
}

HDF5 Select and Reorder Variable Data

The ability to select specific data points from a variable array and save them in a dataset is possible (since Damaris v1.8.0). This capability uses the H5Sselect_elements() HDF5 API call. It is currently only available in Collective mode and is described in detail on the Damaris Gitlab Wiki page.

Comments are closed.