ParaView Connector

In order to benefit from ParaView connector that has been implemented in Damaris, it is necessary to build Damaris on a machine that ParaView Catalyst is already installed. You may use this link to install ParaView in your machine. Please note that if you are going to have your simulation and visualization in two different nodes, it is recommended not to install ParaView GUI on simulation nodes. Also, please ensure that the version of Paraview that Damaris is compiled against matches the Paraview being used for visualization of the simulation exactly.

Note 1: The first version of ParaView backend for Damaris (Damaris 1.3.0), has been developed and tested with ParaView 5.5.0. Damaris v1.3.1+ has been tested with 5.5.0 and 5.8.0 of Paraview.

Note 2: In situ visualization with VisIt and ParaView in Damaris are quite similar. As a result, it is recommended to check the in situ visualization with VisIt documentation before working through this tutorial.

Example code

A working example that closely matches the following description is available in the /examples/paraview directory, of either the git repository or installed if Damaris was configured with -DENABLE_EXAMPLES=ON and is named image.cpp. It has an accompanying Catalyst script code (image.py) and Damaris configuration file (image.xml).

ParaView Options

When using ParaView Catalyst in support of in situ data analysis and visualization without Damaris, the instrumented simulation starts loading a set of Python scripts. These scripts, include necessary steps to produce images, generate plots, and also extract derived information such as polygonal data in support of in situ processing and visualization. During the simulation execution, any analysis and visualization output is generated in synchronous fashion (i.e., while the simulation is running).

The mentioned Python scripts are used in the same way within Damaris too. In this context, simulation processes pass the datasets to the dedicated core(s) using shared memory. Afterwards, the dedicated core(s) load the Python scripts and use the steps inside each script to process data.

In the context of Damaris, ParaView should be enabled in the xml configuration files. The Python scripts should be introduced in the same file as well. To do that, it is necessary to create a <paraview> tag in the xml configuration file and mention the Python script files like this:

1
2
3
4
<paraview>
    <script>/path/to/the/first/script1.py</script>
    <script>/path/to/the/first/script2.py</script>
</paraview>

Description of meshes, fields and variables

Definition of meshes, fields and variables is as same as the ones that were described for VisIt connector here. We will describe them once more in this tutorial with an example. Please note that in the current version of ParaView backend for Damaris (version 1.3.0), only rectilinear and curvilinear meshes are supported.

Example

Suppose that you are going to decompose a 3D array with 100x100x100 dimensions over some processes (let’s say the number of processes is size). The decomposition will happen over the Z axis, as a result, each process will work on a 3D array with the dimensions of (100 , 100 , 100/size). To do so, doing the following steps is necessary:

1. Parameters 

To make the xml configuration file more flexible, it is better to define every important data item as an parameter. To do so, we can define the following parameters in the xml file:

1
2
3
4
<parameter name="WIDTH" type="int" value="100" />
<parameter name="HEIGHT" type="int" value="100" />
<parameter name="DEPTH" type="int" value="100" />
<parameter name="size" type="int" value="2" />

As the parameter names imply, WIDTH, HEIGHT and DEPTH are defined as the dimensions of the main data to be written. The parameter size is used for passing the number of simulation processes (not dedicated ones) to Damaris. This parameter can be set in the simulation code and at runtime using API function damaris_parameter_set() .

2. Variables

Variables are those data that are written to Damaris at the end of each iteration. In this case, we should define a variable with its attributes like this:

1
<variable name="pressure" type="scalar" layout="cells" mesh="mesh" centering="zonal" />

In this definition, pressure is the name of the variable, that is going to be written at each iteration. Scaler is the type of the data. In addition, layout and the mesh of this variable are defined as cells and mesh. These two items are discussed later in this document. Finally, the centering attribute determines that is variable should be visualized using nodes of the mesh (zonal) or points of the mesh (nodal).

3. Layout

The layout of every variable defines the structure of its data in memory. The definition of each layout includes the number of its dimensions, extents along for each direction, type, etc. For the mentioned variable, its layout is defined as:

1
2
<layout name="cells" type="double" dimensions="DEPTH/size,HEIGHT,WIDTH"
global="DEPTH,HEIGHT,WIDTH" ghosts="0:0,0:0,0:0" />
  • Each layout should have a name. Here, the name is selected as cells.
  • The type of the layout determines the type of each data element. In this case, the type is defined as double.
  • The local dimensions of the data is defined as (WIDTH,HEIGHT,DEPTH/size). This means that each process handles a data with these dimensions.
  • The global tag defines the global dimensions of the data, like (WIDTH,HEIGHT,DEPTH).
  • The ghosts attribute indicates the existence of ghost values in each direction. Ghosts are not implemented in the current version of Damaris (1.5.0) yet.
4. Mesh

A mesh (or grid) defines how data elements in a variable should be distributed in a 2D or 3D space. Each mesh in Damaris configuration file includes a name, its type, its dimensions and the list of variables associated to each dimension. For the mentioned example in this document, the mesh should be defined as:

1
2
3
4
5
<mesh name="mesh" type="rectilinear" topology="3">
    <coord name="coord/x" label="x"/>
    <coord name="coord/y" label="y"/>
    <coord name="coord/z" label="z"/>
</mesh>

In this definition, the mesh name is mesh, its type is rectilinear, its topology is 3 and the variable that contain the coordinates of each dimension are coord/x, coord/y and coord/z. Each of these coordinate variables indicates that it is located in a group (the name of the group is coord). The definition of these coordinate variables should be defined as:

1
2
3
4
5
<group name="coord">
    <variable name="x" layout="rmeshX" visualizable="false" time-varying="false" />
    <variable name="y" layout="rmeshY" visualizable="false" time-varying="false" />
    <variable name="z" layout="rmeshZ" visualizable="false" time-varying="false" />
</group>

As the above definition shows, the group name is coord and 3 different variables, namely x, y and z are defined inside it. As a result, the full name of each coordinate variable should be coord/x, coord/y and coord/z. Each of these variables are not time varying and visualizable. This means that the value of these variables are not changed during different iteration of the simulation and hence are not visualizable. The shape of each variable is defined as three different layouts as rmeshX, rmeshY and rmeshZ. The definition of these meshes are defined as below:

1
2
3
<layout name="rmeshX" type="float" dimensions="WIDTH+1" global="WIDTH+1" ghosts="0:0" />
<layout name="rmeshY" type="float" dimensions="HEIGHT+1" global="HEIGHT+1" ghost="0:0" />
<layout name="rmeshZ" type="float" dimensions="DEPTH/size+1" global="DEPTH+1" ghost="0:0" />

As shown in the definition of coordinate layouts, each dimension of each layout equals the dimension of the data plus one. This means that for visualization of a data block with (WIDTH, HEIGHT , DEPTH/size) dimensions, a mesh with dimensions of (WIDTH+1, HEIGHT+1 , DEPTH/size +1) is needed.

Instrumenting the simulation

For instrumenting a simulation for in situ visualization with ParaView Catalyst, the first steps are the standard ones to initialize, start, stop and finalize Damaris in your simulation. These steps are discussed in another page of Damaris documentation (found here). After those steps, you can proceed with the following steps described below:

Writing coordinates

For in situ visualization with ParaView (and also VisIt) in Damaris, it is necessary to write the coordinate variables first. For example, for the mentioned above example, the user should initialize the value of coordinate variables specified in the xml file, that are coord/x, coord/y and coord/z. The below code, shows how the coordinate variables should be passed to Damaris:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
int X,Y,Z;
int size = GetSize();  // The number of client MPI ranks
int rank = GetRank();
 
// Pass the specific size parameter back to Damaris
damaris_parameter_set("size",&ampsize,sizeof(int));
 
damaris_parameter_get("WIDTH",&X,sizeof(int));
damaris_parameter_get("HEIGHT",&Y,sizeof(int));
damaris_parameter_get("DEPTH",&Z,sizeof(int));
 
int Z_local = Z/size;
 
float* XCoord = new float[X+1];
float* YCoord = new float[Y+1];
float* ZCoord = new float[Z_local +1];
 
for(int i=0; i<=X ; i++)
XCoord[i] = i;
 
for(int j=0; j<=Y ; j++)
YCoord[j] = j;
 
for(int k=0; k<=Z_local ; k++)
ZCoord[k] =  rank*Z_local +k;
 
damaris_write("coord/x" , XCoord);
damaris_write("coord/y" , YCoord);
damaris_write("coord/z" , ZCoord);

Regarding the above code example, please note that:

  • Each process should pass its local coordinates to Damaris. Because of this, the value of Z is divided by size in the code.
  • For visualizing a block of 100x100x100 elements in the cells of a mesh, we need a mesh with 101x101x101 points. Because of this, each coordinate array (e.g. XCoord, YCoord and ZCoord) have been created with the size equal to X+1, Y+1 and Z_local+1;
  • If coordinates of a grid are not changing over time, it is better to defines the coordinates as non-time-varying and also set the visualizable attribute as false.
Writing the positions and the data

Unlike the VisIt plug-in for Damaris, for ParaView plug-in it is necessary to write the position of each block of data into Damaris. The position means where exactly ParaView should visualize the passed data block. Writing the position, can happen while the variable array data is written to Damaris. Take a look at the below example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
double* pressure_data = new double[X*Y*Z_local];
 
for(int i=0; i<X; i++)
  for(int j=0; j<Y; j++)
    for(int k=0; k<Z_local; k++)
        // fill the local pressure_data array
 
int64_t pos[3];
 
pos[0] = 0;
pos[1] = 0;
pos[2] = rank*Z_local;
 
damaris_set_position("pressure" , pos);
damaris_write("pressure" , pressure_data);
damaris_end_iteration();

Considering the above source code, please note that:

  • The main variable in the defined XML file is called pressure. As a result, in the damaris functions that need a variable name, the passed value is “pressure”.
  • In the layout of the main variable (i.e. pressure) that is called “cells”, the type of the layout is defined as double. As a result, the data that is passed to damaris_write is a block of double data.
  • Do not forget to call damaris_end_iteration when all of the data are written at the end of each iteration.

Running the simulation

Running the simulation while ParaView is enabled is like running other examples of Damaris. Just do not forget that when you have N dedicated cores and the simulation uses M processes, you should run the simulation with N+M processes, that is:

1
mpirun -np <N+M> <config-file.xml>

You will also need to ensure the python script specified in the configuration xml file defines a Catalyst pipeline and that it specifies the hostname of the machine that the Paraview GUI is running on. The value can be localhost for simulations running on the same node as the Paraview GUI, otherwise a FQDN should be specified.

2
3
# Live Visualization, if enabled.
coprocessor.DoLiveVisualization(datadescription, "localhost", 22222)

Connection to ParaView

After running the instrumented simulation, it is the time to show the result. To this aim:

  • Open ParaView GUI and from the Catalyst menu select “connect…”.
  • Afterwards, a dialog is shown to enter the port number for connecting to the simulation. (The default port is 22222, but you may change it in the Python script of your simulation.)
  • When ParaView GUI is connected to the simulation, you will receive a message about the connection in the GUI.
  • The final step is to go to the pipeline browser in ParaView GUI (in the left pane) and select what you want to do with the received information from simulation.

Example of unstructured-mesh mesh type

Since Damaris version 1.5, an unstructured mesh data type has been supported for use with the Paraview connector. VisIt support is hoped to follow soon.
The unstructured mesh data representation is more complex than other mesh types and is built up of 5 variables that need to be declared and linked with the <mesh type=”unstructured” … /> XML element. The variables used each need a layout and the layouts are parameterized using the parameters seen.
The mesh type declaration is highlighted here, where the name component is set to an available variable defined in the XML and filled with data by the simulation client code.

1
2
3
4
5
6
7
8
<data>
     <mesh name="fluid_domain_umesh" type="unstructured" topology="3" comment="These arrays can be ragged, i.e. each rank could have a different number of sections and each section can have a different number of vertices etc" >
          <coord                name="umesh_vars/unstructured_mesh_xyz"  unit="m"    />
          <vertex_global_id     name="umesh_vars/unstructured_gid"       offset="-1" />
          <section_types        name="umesh_vars/section_types"                      />
          <section_sizes        name="umesh_vars/section_sizes"                      />
          <section_connectivity name="umesh_vars/section_connectivity"               />
     </mesh>

A full example which includes the mesh variables and layouts and the field data and their layouts is shown below. The field variables are:

  • pressure (single float per field)
  • velocity (3 component vector per field element)
  • mpi_rank_id (requires an integer per field)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
<data>
               <parameter name="cs_glob_n_ranks"      type="int" value="4" /> 
               <parameter name="mesh_dim"             type="int" value="3" /> 
               <parameter name="n_sections_total"     type="int" value="1" /> 
               <parameter name="n_vertices_total"     type="int" value="1" />
               <parameter name="n_connectivity_total" type="int" value="1" /> 
               <parameter name="n_elements_total"     type="int" value="1" /> 
               <parameter name="n_sections_local"     type="int" value="1" /> 
               <parameter name="n_vertices_local"     type="int" value="1" />
               <parameter name="n_connectivity_local" type="int" value="1" /> 
               <parameter name="n_elements_local"     type="int" value="1" /> 
               <parameter name="n_elements_offset"    type="int" value="1" />
 
               <layout name="zonal_layout_usmesh"             type="double" dimensions="n_elements_local"   global="n_elements_total"   comment="For the field data e.g. Pressure"  />
               <layout name="zonal_layout_usmesh_int"         type="int"    dimensions="n_elements_local"   global="n_elements_total"   comment="For the integer based MPI rank data"  />
               <layout name="zonal_layout_usmesh_vect3"       type="double" dimensions="3,n_elements_local" global="3,n_elements_total" comment="For vectorised (3 component) field data e.g. Velocity "  />
 
               <layout name="unstructured_vertex_layout_xyz"  type="double"  dimensions="mesh_dim, n_vertices_local"   global="mesh_dim, n_vertices_total"   />
               <layout name="unstructured_gid_layout"         type="long"    dimensions="n_vertices_local"     global="n_vertices_total"     />
               <layout name="mesh_n_sections"                 type="int"     dimensions="n_sections_local"     global="n_sections_total"     />
               <layout name="connectivity_size"               type="long"    dimensions="n_connectivity_local" global="n_connectivity_total" />
 
            <group name="umesh_vars">    
               <variable name="unstructured_mesh_xyz" layout="unstructured_vertex_layout_xyz" type="vector" visualizable="false" time-varying="false" comment="The x,y,(z) coordinates of each vertex"  />
               <variable name="unstructured_gid"      layout="unstructured_gid_layout"        type="scalar" visualizable="false" time-varying="false" comment="The global id's of each vertex" />
 
               <variable name="section_types"         layout="mesh_n_sections"    type="scalar" visualizable="false"  time-varying="false"  comment="The section geometry type - VTK_QUAD VTK_HEXAHEDRON etc." />
               <variable name="section_sizes"         layout="mesh_n_sections"    type="scalar" visualizable="false"  time-varying="false"  comment="The number of elements in each section- soe equiv to the number of field values per section" />
               <variable name="section_connectivity"  layout="connectivity_size"  type="scalar" visualizable="false"  time-varying="false"  comment="n_connectivity += section->n_elements * vtk_type_stride ;"/>
 
               <mesh name="fluid_domain_umesh" type="unstructured" topology="3" comment="These arrays can be ragged, i.e. each rank could have a different number of sections and each section can have a different number of vertices etc" >
                  <coord                name="umesh_vars/unstructured_mesh_xyz"  unit="m"    />
                  <vertex_global_id     name="umesh_vars/unstructured_gid"       offset="-1" />
                  <section_types        name="umesh_vars/section_types"                      />
                  <section_sizes        name="umesh_vars/section_sizes"                      />
                  <section_connectivity name="umesh_vars/section_connectivity"               />
               </mesh>
            </group>
 
            <group name="fields">     
              <variable name="pressure"    layout="zonal_layout_usmesh"       mesh="fluid_domain_umesh" type="scalar"  visualizable="true"                      unit="Pa"   centering="zonal" />
              <variable name="velocity"    layout="zonal_layout_usmesh_vect3" mesh="fluid_domain_umesh" type="vector"  visualizable="true" vectorlength="3"     unit="m/s"  centering="zonal" />
              <variable name="mpi_rank_id" layout="zonal_layout_usmesh_int"   mesh="fluid_domain_umesh" type="scalar"  visualizable="true" time-varying="false" unit="rank" centering="zonal" comment="Compare store=MyStore with Paraview csv output" />
            </group>
      </data>

The simulation client code needs to fill the mesh variables with the correct data, also matching the type of the data, i.e. int float double etc.
A client side code example will be added here to extend this tutorial.

Comments are closed.