Instrumenting a Simulation

This part of the documentation will show you how to instrument a simulation in order to use Damaris. Damaris requires the simulation to be based on MPI.

Minimal Configuration File

Before actually instrumenting your code, an XML configuration file has to be created. This configuration file will contain some information about the simulation, the system, the data and the plug-ins that you want to use with Damaris. In the below sample, you can find a minimal configuration file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<?xml version ="1.0"?>
<simulation name="mysimulation" language="fortran" xmlns="http://damaris.gforge.inria.fr/damaris/model">
    <architecture>
        <domains count="1"/>
        <dedicated cores="1" nodes="0"/>
        <buffer name="buffer" size="67108864" />
        <placement />
        <queue name="queue" size="100" />
    </architecture>
<data>
</data>
<actions>
</actions>
</simulation>

The <simulation> name is used in different backends, for example to name trace files or to be added on figures when using in situ visualization. The language (“c”, “cpp”, or “fortran”) is optional (default is “c”), it indicates in which language the simulation is written, so that fortran array layouts can be converted into C layouts. The <domain> section provides the number of domains that a single process will handle. Some simulations indeed split their variables into blocks (or domains) and distribute several blocks per process. If the number of blocks is not the same on every process, it should correspond to the maximum number of blocks that a single process may have to handle. The <dedicated> section provides the number of dedicated cores in a node, and the number of dedicated nodes. These different configurations are explained in more details later.

The <buffer> section provides the name and the size (in bytes) of the buffer that will be used for the simulation processes to communicate with the dedicated cores or dedicated nodes. The <buffer name=> is important, as it will be used to name the shared memory file. The name must be unique if multiple simulations are running on the same machine or node. It can take an additional type attribute which can be either “posix” (by default) or “sysv“, representing the type of share memory to be used. POSIX shared memory will use the shm_open function, while SYS-V shared memory will use shmget. Some HPC operating systems indeed don’t have one or the other type available. Finally, an additional blocks attribute can be added to the buffer, taking an integer (e.g. blocks=”3″) to indicate that several blocks of shared memory must be opened of the given size. This feature is useful in order to open a shared memory buffer with a size bigger than the maximum allowed by the operating system (most operating systems limit this size to 1 or 2 GB). However, a variable can never have a size bigger than the block size (since separate blocks may not be contiguous). This being said, larger shared memory segments are possible on modern HPC operating systems, and the value for size is stored as a 64 bit integer so can cope with larger allocations if applicable.

The <queue> section provides the name and size (in number of messages) of the message queues used to communicate between clients and dedicated resources. This is the remnant of older versions of Damaris in which the queue was handled in shared memory as well. Simply keeping the default name and size works just fine. Finally the data and actions sections will be described later.

The <placement /> section is a placeholder for future capabilities of setting where the Damaris core will be within a multicore architecture.

Instrumenting a simulation

Now that a minimal configuration file has been provided, let’s instrument our code.

Writing the code

The below listings present the minimal code instrumentation for a C and a Fortran code respectively. All Damaris function calls must be placed after a first call to the damaris_initialize function (resp. damaris_initialize_f in Fortran). This function takes as parameter an XML configuration file and a MPI communicator (generally MPI_COMM_WORLD). The call to damaris_initialize should be placed after the MPI initialization, as it uses MPI functions. damaris_initialize involves collective MPI communications: process 0 in the provided communicator will load the XML file and broadcast its content to other processes in the communicator, thus all the processes in the involved communicator should call this function.

Symmetrically a call to damaris_finalize (resp. damaris_finalize_f in fortran) frees the resources used by Damaris before the simulation finishes. It should be called before finalizing MPI.

These functions only prepare Damaris but do not start the dedicated cores or nodes. To start the dedicated resources, a call to damaris_start (resp. damaris_start_f) is necessary. This function takes a pointer to an integer as parameter. Damaris will automatically analyze the platform on which the simulation runs and selects cores to be either clients or servers. On client processes, this function returns and the integer is set to a positive value. On server processes, this function blocks and runs the server loop. It will only return when the clients have all called damaris_stop (resp. damaris_stop_f), and the integer will be set to 0. This integer can thus be used to select whether or not to run the simulation loop on a particular process.

Inside the simulation’s main loop, a call to damaris_client_comm_get gives a communicator that the clients can use to communicate with one another (i.e., a replacement for the MPI_COMM_WORLD that now includes dedicated resources). It is crucial that the simulation only uses this communicator and not MPI_COMM_WORLD, as global communicator.

damaris_end_iteration (resp. damaris_end_iteration_f) informs the dedicated cores that the current iteration has finished. Damaris keeps track of the iteration number internally: this number is equal to 0 before the first call to damaris_end_iteration. A call to damaris_end_iteration will update potentially connected backends such as VisIt. As it involves collective communications, all the clients have to call this function. These main functions are summarized in tables below.

Ref. Function name (C,C++) Parameters
1. damaris_initialize const char* configfile, MPI Comm comm
2. damaris_start int* is_client
3. damaris_client_comm_get MPI_Comm* comm
4. damaris_end_iteration —
5. damaris_stop —
6. damaris_finalize —
Ref. Function name (Fortran) Parameters
1. damaris_initialize_f const char* configfile, MPI Comm comm, integer ierr
2. damaris_start_f int* is_client, integer ierr
3. damaris_client_comm_get_f MPI_Comm* comm
4. damaris_end_iteration_f integer ierr
5. damaris_stop_f integer ierr
6. damaris_finalize_f integer ierr 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

#include "Damaris. h"
 
 
void sim_main_loop (MPI Comm comm)
{
    int i ;
    for ( i =0; i < 100; i++) {
        // do something using comm as global communicator
        damaris_end_iteration( ) ;
    }
}
 
int main ( int argc , char argv )
{
    MPI_Init (&argc , &argv ) ;
    int id = 0 ;
    int err, is_client;
    MPI_Comm global ;
    err = damaris_initialize("config.xml" , MPI_COMM_WORLD) ;
    damaris_start(&is_client) ;
    if ( is_client) {
        damaris_client_comm_get (&global) ;
        sim_main_loop(global) ;
        damaris_stop();
    }
    damaris_finalize() ;
    MPI_Finalize();
    return 0;
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

program sim
include 'mpi f.h'
 
    integer :: ierr , is_client , comm
 
    call MPI_ INIT(ierr )
    call damaris_initialize_f ("config.xml" ,MPI_COMM_WORLD, ierr )
    call damaris_start_f ( is_client , ierr )
 
    if ( is_client.gt.0 ) then
        call damaris_client_comm_get (comm, ierr )
        call sim_main_loop(comm)
        call damaris_stop_f (ierr)
    endif
 
    call damaris_finalize_f (ierr)
    call MPI_FINALIZE( ierr )
end program sim
 
subroutine sim_main_loop (comm)
    integer comm
    integer i , ierr
    do i = 1 , 100
        ! do something
        call damaris_end_iteration_f (ierr )
    end do
end subroutine

Note: Damaris is not thread-safe yet. If your application uses an hybrid model where different threads run on the same node, only one thread should call damaris initialize (and other Damaris functions), and all the threads on a node should act as one single client.

Compiling

An instrumented software must be able to compile and link with the Damaris library. There are various options for this –

  1. Command line compilation
  2. Integrate with Makefile project
  3. Integrate with CMake project
  4. Integrate with Autotools project
  1. Command line compilation
    In order to compile your instrumented simulation, the Damaris header files must be provided, as well as the dependencies’ headers. The following options must be passed to the compiler:

    1
2

    -I/damaris/install/prefix/include 
-I/path/to/dependencies/include

    If all dependencies and Damaris have been installed in $HOME/local, for example, it becomes

    1

    -I$HOME/local/include

    Fortran programs may look for damaris.mod, which is also in the include directory where Damaris
    has been installed. The linker will need the Damaris library le libdamaris.a as well as other dependent libraries. The following provides the necessary options for the linker:

    1
2
3
4
5
6

    -rdynamic
-L/damaris/install/prefix/lib -ldamaris -L/path/to/dependencies/lib \
-Wl,--whole-archive,-ldamaris,--no-whole-archive \
-lxerces-c \
-lboost_filesystem -lboost_system -lboost_date_time \
-lstdc++ -lrt -ldl

    The -rdynamic option asks the linker to add all symbols, not only used ones, to the dynamic
    symbol table. This option is needed for some uses of dlopen in Damaris. It is possible that the
    option differs with some linkers.

    Running

    The deployment of a Damaris-enabled simulation is that of a normal simulation:

    1

    mpirun -np X -machinefile FILE ./sim

    Where X is the number of processes (in total), and FILE lists the hostnames on which to start the
    program.

  2. Integrate with Makefile project
    Hard coding the above in a Makefile is possible, however Damaris provides a damaris.pc file for use with the pkg-config utility.
    N.B. The damaris.pc file has caused problems, in particular when Damaris is compiled with ParaView Catalyst support.
  3. Integrate with CMake project 
    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

    # Add a CMAke configure time switch to turn on and off use of Damaris
option(USE_DAMARIS_LIB "Use the Damaris library for asynchronous I/O?" OFF)
...
 
# If the switch is ON
if (USE_DAMARIS_LIB)
  # Use the CMake find_package() directive to search for Damaris.
  # The CMake system is looking for the DamarisConfig.cmake file and
  # may require a hint as to where to find it if it is not in a standard system path. 
  # Use the following on the command line: -DDamaris_ROOT=/path/to/damaris/
  # The <em>/path/to/damaris/</em> is the parent directory to the <em>lib</em> 
  # and the <em>share</em> directories, where Damaris has been installed
  find_package(Damaris 1.8)
  if (Damaris_FOUND)
    include_directories(${Damaris_INCLUDE_DIRS})
  else()
    message(STATUS "The Damaris library was NOT found")
  endif()
endif()
 
...
if (USE_DAMARIS_LIB)
    # Adds the libraries used - Damaris and its dependencies to the link line
    target_link_libraries(mysim PUBLIC ${Damaris_LIBRARIES})
    # This creates a definition of a pre-processor variable (PROG_HAS_DAMARIS)
    # that can be used as a guard in the C/C++ code:
    target_compile_definitions(mysim PRIVATE PROG_HAS_DAMARIS)
endif()
  4. Integrate with Autotools project
    An example of using Autotools is available via the Code_Saturne integration code, which is available through github (account jcbowden) and also available internally in https://gitlab.inria.fr/Damaris/Simulations. Look for the m4/cs_damaris.m4 file in the cs_with_damaris branch. The m4 file tests if Damaris is present on a system and then defines variables: DAMARIS_LDFLAGS DAMARIS_LIBS DAMARISRUNPATH, which are subsequently used in Makefile.am files of source code subdirectories that use the Damaris API.

Note: Damaris doesn’t use any a priori knowledge about rank-to-cores binding and is able to start the proper number of servers at the right location even if process ranks are not consecutive in the nodes. It leverages the processor’s name to group processes, or platform specific information such as the host’s personality on BG/P. Yet, it is important that the same number of cores are used in each node.

Next: Data Description

Comments are closed.