Storage Backends

In many HPC applications, I/O is concurrently performed by all processes, which leads to I/O bursts. This causes resource contention and substantial variability of I/O performance, which significantly impacts the overall application performance and, most importantly, its predictability over time. To solve the mentioned issues, Damaris can be used to leverages dedicated I/O cores on each multicore SMP node, along with the use of shared memory, to efficiently perform asynchronous data processing and I/O in order to hide this variability. This means that, all simulation processes could pass the simulation results to the dedicated core(s) through shared memory. Afterwards, the dedicated core(s) could write the results asynchronously to desired data formats. Using this technique can have several advantages:

  • It fully hides jitter as well as all I/O-related costs, which makes simulation performance predictable;
  • It increases the sustained write throughput compared to standard approaches;
  • It allows almost perfect scalability of the simulation (validated for up to over 9,000 cores) as opposed to state-of-the-art approaches which fail to scale;
  • It enables in situ data analysis (e.g. compression) leading to a major reduction of storage requirements.

Currently, only a HDF5 backend is supported in Damaris. You may check this link for more information about HDF5 backend and this link to view information about the HDF5 examples that are part of the Damaris distribution.

Comments are closed.