Instrumenting a Simulation

This part of the documentation will show you how to instrument a simulation in order to use Damaris. Damaris requires the simulation to be based on MPI.

Minimal Configuration File

Before actually instrumenting your code, an XML configuration file has to be created. This configuration file will contain some information about the simulation, the system, the data and the plug-ins that you want to use with Damaris. In the below sample, you can find a minimal configuration file.

1
2
3
4
5
6
7
8
9
10
11
12
13
<?xml version ="1.0"?>
<simulation name="mysimulation" language="fortran" xmlns="http://damaris.gforge.inria.fr/damaris/model">
    <architecture>
        <domains count="1"/>
        <dedicated cores="1" nodes="0"/>
        <buffer name="buffer" size="67108864" />
        <queue name="queue" size="100" />
    </architecture>
<data>
</data>
<actions>
</actions>
</simulation>

The simulation name is used in different backends, for example to name trace files or to be added on figures when using in situ visualization. The language (“c”, “cpp”, or “fortran”) is optional (default is “c”), it indicates in which language the simulation is written, so that fortran array layouts can be converted into C layouts. The domain section provides the number of domains that a single process will handle. Some simulations indeed split their variables into blocks (or domains) and distribute several blocks per process. If the number of blocks is not the same on every process, it should correspond to the maximum number of blocks that a single process may have to handle. The dedicated section provides the number of dedicated cores in a node, and the number of dedicated nodes. These different configurations are explained in more details later.

The buffer section provides the name and the size (in bytes) of the buffer that will be used for the simulation processes to communicate with the dedicated cores or dedicated nodes. It can take an additional type attribute which can be either “posix” (by default) or “sysv“, representing the type of share memory to be used. POSIX shared memory will use the shm_open function, while SYS-V shared memory will use shmget. Some HPC operating systems indeed don’t have one or the other type available. Finally, an additional blocks attribute can be added to the buffer, taking an integer (e.g. blocks=”3″) to indicate that several blocks of shared memory must be opened of the given size. This feature is useful in order to open a shared memory buffer with a size bigger than the maximum allowed by the operating system (most operating systems limit this size to 1 or 2 GB). However, a variable can never have a size bigger than the block size (since blocks may not be contiguous).

The queue section provides the name and size (in number of messages) of the message queues used to communicate between clients and dedicated resources. This is the remnant of older versions of Damaris in which the queue was handled in shared memory as well. Simply keeping the default name and size works just fine. Finally the data and actions sections will be described later.

Instrumenting a simulation

Now that a minimal configuration file has been provided, let’s instrument our code.

Writing the code

The below listings present the minimal code instrumentation for a C and a Fortran code respectively. All Damaris function calls must be placed after a first call to the damaris_initialize function (resp. damaris_initialize_f in Fortran). This function takes as parameter an XML configuration file and a MPI communicator (generally MPI_COMM_WORLD). The call to damaris_initialize should be placed after the MPI initialization, as it uses MPI functions. damaris_initialize involves collective MPI communications: process 0 in the provided communicator will load the XML file and broadcast its content to other processes in the communicator, thus all the processes in the involved communicator should call this function.

Symmetrically a call to damaris_finalize (resp. damaris_finalize_f in fortran) frees the resources used by Damaris before the simulation finishes. It should be called before finalizing MPI.

These functions only prepare Damaris but do not start the dedicated cores or nodes. To start the dedicated resources, a call to damaris_start (resp. damaris_start_f) is necessary. This function takes a pointer to an integer as parameter. Damaris will automatically analyze the platform on which the simulation runs and selects cores to be either clients or servers. On client processes, this function returns and the integer is set to a positive value. On server processes, this function blocks and runs the server loop. It will only return when the clients have all called damaris_stop (resp. damaris_stop_f), and the integer will be set to 0. This integer can thus be used to select whether or not to run the simulation loop on a particular process.

Inside the simulation’s main loop, a call to damaris_client_comm_get gives a communicator that the clients can use to communicate with one another (i.e., a replacement for the MPI_COMM_WORLD that now includes dedicated resources). It is crucial that the simulation only uses this communicator and not MPI_COMM_WORLD, as global communicator.

damaris_end_iteration (resp. damaris_end_iteration_f) informs the dedicated cores that the current iteration has finished. Damaris keeps track of the iteration number internally: this number is equal to 0 before the first call to damaris_end_iteration. A call to damaris_end_iteration will update potentially connected backends such as VisIt. As it involves collective communications, all the clients have to call this function. These main functions are summarized in tables below.

Ref. Function name (C,C++) Parameters
1. damaris_initialize const char* configfile, MPI Comm comm
2. damaris_start int* is_client
3. damaris_client_comm_get MPI_Comm* comm
4. damaris_end_iteration
5. damaris_stop
6. damaris_finalize

 

Ref. Function name (Fortran) Parameters
1. damaris_initialize_f const char* configfile, MPI Comm comm, integer ierr
2. damaris_start_f int* is_client, integer ierr
3. damaris_client_comm_get_f MPI_Comm* comm
4. damaris_end_iteration_f integer ierr
5. damaris_stop_f integer ierr
6. damaris_finalize_f integer ierr
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#include "Damaris. h"
 
 
void sim_main_loop (MPI Comm comm)
{
    int i ;
    for ( i =0; i < 100; i++) {
        // do something using comm as global communicator
        damaris_end_iteration( ) ;
    }
}
 
int main ( int argc , char argv )
{
    MPI_Init (&argc , &argv ) ;
    int id = 0 ;
    int  err, is_client;
    MPI_Comm global ;
    err = damaris_initialize("config.xml" , MPI_COMM_WORLD) ;
    damaris_start(&is_client) ;
    if ( is_client) {
        damaris_client_comm_get (&global) ;
        sim_main_loop(global) ;
        damaris_stop();
    }
    damaris_finalize() ;
    MPI_Finalize();
    return 0;
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
program sim
include 'mpi f.h'
 
    integer ::  ierr , is_client , comm
 
    call MPI_ INIT(ierr )
    call damaris_initialize_f ("config.xml" ,MPI_COMM_WORLD, ierr )
    call damaris_start_f ( is_client ,  ierr )
 
    if ( is_client.gt.0 ) then
        call damaris_client_comm_get (comm, ierr )
        call sim_main_loop(comm)
        call damaris_stop_f (ierr)
    endif
 
    call damaris_finalize_f (ierr)
    call MPI_FINALIZE( ierr )
end program sim
 
subroutine sim_main_loop (comm)
    integer comm
    integer i , ierr
    do i = 1 , 100
        ! do something
        call damaris_end_iteration_f (ierr )
    end do
end subroutine

Note: Damaris is not thread-safe yet. If your application uses an hybrid model where different threads run on the same node, only one thread should call damaris initialize (and other Damaris functions), and all the threads on a node should act as one single client.

Compiling

An instrumented software must be able to compile and link with the Damaris library. There are various options for this –

  1. Command line compilation
  2. Integrate with Makefile project
  3. Integrate with CMake project
  4. Integrate with Autotools project
  1. Command line compilation
    In order to compile your instrumented simulation, the Damaris header files must be provided, as well as the dependencies’ headers. The following options must be passed to the compiler:

    1
    2
    
    -I/damaris/install/prefix/include 
    -I/path/to/dependencies/include

    If all dependencies and Damaris have been installed in $HOME/local, for example, it becomes

    1
    
    -I$HOME/local/include

    Fortran programs may look for damaris.mod, which is also in the include directory where Damaris
    has been installed. The linker will need the Damaris library le libdamaris.a as well as other dependent libraries. The following provides the necessary options for the linker:

    1
    2
    3
    4
    5
    6
    
    -rdynamic
    -L/damaris/install/prefix/lib -ldamaris -L/path/to/dependencies/lib \
    -Wl,--whole-archive,-ldamaris,--no-whole-archive \
    -lxerces-c \
    -lboost_filesystem -lboost_system -lboost_date_time \
    -lstdc++ -lrt -ldl

    The -rdynamic option asks the linker to add all symbols, not only used ones, to the dynamic
    symbol table. This option is needed for some uses of dlopen in Damaris. It is possible that the
    option differs with some linkers.

    Running

    The deployment of a Damaris-enabled simulation is that of a normal simulation:

    1
    
    mpirun -np X -machinefile FILE ./sim

    Where X is the number of processes (in total), and FILE lists the hostnames on which to start the
    program.

  2. Integrate with Makefile project
    Hard coding the above in a Makefile is possible, however Damaris provides a damaris.pc file for use with the pkg-config utility. N.B. this file currently is malformed, in particular when Damaris is compiled with ParaView Catalyst support (see
  3. Integrate with CMake project
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    
     
    # Add a CMAke configure time switch to turn on and off use of Damaris
    option(USE_DAMARIS_LIB "Use the Damaris library for asynchronous I/O?" OFF)
    ...
     
    # If the switch is ON
    if (USE_DAMARIS_LIB)
      # Use the CMake find_package() directive to search for Damaris
      # The cmake command line may need to specify where to find Damaris if it is not in a typical system area.
      # Use the following on the command line: -DCMAKE_PREFIX_PATH=/path/to/damaris/
      # The <em>/path/to/damaris/</em> is the parent directory to the <em>lib</em> and the <em>share</em> directories, where damaris has been installed
      find_package(Damaris 1.5 )
      if (Damaris_FOUND)
        message(STATUS "The Damaris library was found: ${Damaris_VERSION}")
        if (Damaris_HAS_HDF5)
          message(STATUS "The Damaris library has HDF5 support: ${HDF5_VERSION}")
        else()
          message(STATUS "The Damaris library does NOT have HDF5 support")
        endif()
        if (Damaris_HAS_VISIT)
          message(STATUS "The Damaris library has VisIt support: ${VisIt_VERSION}")
        else()
          message(STATUS "The Damaris library does NOT have VisIt support")
        endif()
        if (Damaris_HAS_CATALYST)
          message(STATUS "The Damaris library has Catalyst support: ${Catalyst_VERSION}  ${Paraview_VERSION}")
        else()
          message(STATUS "The Damaris library does NOT have Catalyst support")
        endif()
        include_directories(${Damaris_INCLUDE_DIRS})
     
      else()
        message(STATUS "The Damaris library was NOT found")
      endif()
    endif()
     
    ...
    if (USE_DAMARIS_LIB)
        # Adds the libraries used - Damaris and its dependencies to the link line
        target_link_libraries(mysim PUBLIC ${Damaris_LIBRARIES})
        # This creates a definition of a pre-processor variable  that can be used as a guard in the C/C++ code:
        target_compile_definitions(mysim PRIVATE OPM_HAVE_DAMARIS)
    endif()
  4. Integrate with Autotools project

Note: Damaris doesn’t use any a priori knowledge about rank-to-cores binding and is able to
start the proper number of servers at the right location even if process ranks are not consecutive in
the nodes. It leverages the processor’s name to group processes, or platform specific information such
as the host’s personality on BG/P. Yet, it is important that the same number of cores are used in each
node.

Next: Data Description

Comments are closed.