Getting Started

This section walks you through a brief introduction to using incflo.

Downloading the code

incflo is built on top of the AMReX framework. In order to run incflo, you must download separate git modules for incflo, AMReX and AMReX-Hydro.

First, make sure that git is installed on your machine.

Download the AMReX repository by typing:
```
git clone https://github.com/AMReX-Codes/amrex.git
```
This will create a folder called amrex/ on your machine. For additional information on downloading the AMReX source code see AMReX user guide - Downloading the code.
Download the AMReX-Hydro repository by typing:
```
git clone https://github.com/AMReX-Fluids/AMReX-Hydro.git
```
This will create a folder called AMReX-Hydro/ on your machine.
Download the incflo repository by typing:
```
git clone https://github.com/AMReX-Fluids/incflo.git
```
This will create a folder called incflo/ on your machine.

You will want to periodically update each of these repositories by typing git pull within each repository.

Next, you can choose to either Building with GNU Make or Building with CMake.

Building with GNU Make

Here, we walk you through compiling an incflo executable. Decide whether you want to build the executable in 2-D vs 3-D, with or without EB (embedded boundaries / cut cells), and choose the corresponding directory: test_2d, test_3d, test_no_eb_2d, or test_no_eb. For this example, let’s say you want to build in 2-D with cut cells. Then to build the code:

cd to the desired build directory; e.g.
```
cd incflo/test_2d
```

Edit the GNUmakefile:

Set AMREX_HOME to be the path to the directory where you have put amrex. NOTE: when setting AMREX_HOME in the GNUmakefile, be aware that ~ does not expand, so AMREX_HOME=~/amrex/ will yield an error.

Alternatively, the path to AMReX can be set up as an environment variable, AMREX_HOME, on your machine to point to the path name where you have put AMReX. For example, if you are using the bash shell, you can add this to your .bashrc as:

export AMREX_HOME=/path/to/amrex

alternatively, in tcsh one can set

setenv AMREX_HOME /path/to/amrex

Common options that you can set in the GNUMakefile include

Option name	Description	Possible values	Default value
COMP	Compiler (gnu or intel)	gnu / intel	None
USE_MPI	Whether to enable MPI	TRUE / FALSE	FALSE
USE_OMP	Whether to enable OpenMP	TRUE / FALSE	FALSE
USE_CUDA	Whether to enable CUDA	TRUE / FALSE	FALSE
DEBUG	Whether to use DEBUG mode	TRUE / FALSE	FALSE
PROFILE	Include profiling info	TRUE / FALSE	FALSE
TINY_PROFILE	Include tiny profiling info	TRUE / FALSE	FALSE
COMM_PROFILE	Include comm profiling info	TRUE / FALSE	FALSE
TRACE_PROFILE	Include trace profiling info	TRUE / FALSE	FALSE

Note

Do not set both USE_OMP and USE_CUDA to true.

Additional options and information on using other compilers can be found in the AMReX documentation at https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html .

Make the executable:

Now type
```
make
```
The name of the resulting executable (generated by the make system) encodes several of the build characteristics, including dimensionality of the problem, compiler name, and whether MPI and/or OpenMP were linked with the executable. Thus, several different build configurations may coexist simultaneously in a problem folder. For example, the default build in incflo/test_2d will look like incflo2d.gnu.MPI.EB.ex, indicating that this is a 2-d version of the code, made with COMP=gnu, USE_MPI=TRUE and support for embedded boundaries (EB).

Building with CMake

CMake build is a two-steps process. First cmake is invoked to create configuration files and makefiles in a chosen directory (builddir). Next, the actual build is performed by invoking make from within builddir. The CMake build process is summarized as follows:

mkdir /path/to/builddir
cd    /path/to/builddir
cmake [options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] /path/to/incflo
make

In the above snippet, [options] indicates one or more options for the customization of the build, as described later in this section. If the option CMAKE_BUILD_TYPE is omitted, CMAKE_BUILD_TYPE=Release is assumed.

There are two modes to build incflo with CMake:

o SUPERBUILD (recommended): AMReX and AMReX-Hydro are built as part of the incflo build process. This method is strongly encouraged as it ensures that the configuration options are consistent.

o STANDALONE: incflo source code is built separately and linked to an existing AMReX installation. This is ideal for continuous integration severs (CI) and regression testing applications. AMReX library version and configuration options must meet incflo requirements.

Note

incflo requires CMake 3.14 or higher.

SUPERBUILD Instructions (recommended)

In this mode, incflo CMake inherents AMReX CMake targets and configuration options, as well as AMReX-Hydro CMake targets; that is, incflo, AMReX and AMReX-Hydro are configured and built as a single entity.

First, make sure you’ve downloaded all the source codes (see Downloading the code).

Next, build the incflo executable:

> cd incflo
> mkdir build
> cd build
> cmake -DAMREX_HOME=/path/to/amrex/source/directory [amrex options] -DAMREX_HYDRO_HOME=/path/to/AMReX-Hydro/source/directory -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] ..
> make -j

where AMREX_HOME is a CMake variable pointing to the top-level source directory of the AMReX distribution you downloaded earlier; [amrex options] is a list of any of the AMReX configuration options listed in the AMReX user guide - Building with CMake; AMREX_HYDRO_HOME is a CMake variable pointing to the top-level source directory of the AMReX-Hydro distribution you downloaded earlier; and .. is the relative path pointing to the top-level source directory of incflo or this example.

STANDALONE instructions

For a stand alone build, incflo CMake will look for an existing AMReX installation on the system and link the incflo binaries against it.

Building AMReX

First, make sure you’ve downloaded all the source codes (see Downloading the code).

Next, configure, build and install AMReX as follows:

> cd /path/to/amrex
> mkdir build
> cd build
> cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel]  [amrex options] -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir -DAMREX_HYDRO_HOME=/path/to/AMReX-Hydro/source/directory ..
> make install

[amrex options] is a list of any of the AMReX configuration options listed in the AMReX user guide - Building with CMake. We suggest to always use the option -DUSE_XSDK_DEFAULTS=yes when building AMReX for incflo. AMREX_HYDRO_HOME is a CMake variable pointing to the top-level source directory of the AMReX-Hydro distribution you downloaded earlier

Building incflo

Build incflo:

> cd /path/to/incflo
> mkdir build
> cd build
> cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] [incflo options] -DAMReX_ROOT=/absolute/path/to/amrex/installdir ..
> make -j

Passing -DAMReX_ROOT=/absolute/path/to/amrex/installdir instructs CMake to search /absolute/path/to/amrex/installdir before searching system paths for an available AMReX installation. AMReX_ROOT can also be set as an environmental variable instead of passing it as a command line option.

[incflo options] indicates any of the configuration option listed in the table below.

Option name	Description	Possible values	Default value
CMAKE_CXX_FLAGS	User-defined C++ flags	valid C++ compiler flags	None
CMAKE_CUDA_FLAGS	User-defined CUDA flags	valid CUDA compiler flags	None
INCFLO_DIM	Dimensionality of the build	2/3	3
INCFLO_MPI	Enable build with MPI	no/yes	yes
INCFLO_OMP	Enable build with OpenMP	no/yes	no
INCFLO_CUDA	Enable build with CUDA	no/yes	no
INCFLO_EB	Build Embedded Boundary support	no/yes	no
INCFLO_HYPRE	Enable HYPRE support	no/yes	no
INCFLO_FPE	Build with Floating-Point Exceptions checks	no/yes	no

Few more notes on building incflo

The system default C++ compiler can be overwritten as follows:

> cmake -DCMAKE_CXX_COMPILER=<c++-compiler>  [options]  ..

When building on a platform that uses the module utility, use either the above command (with full path to the compilers) or the following:

> cmake -DCMAKE_CXX_COMPILER=CC [options] ..

incflo uses the same compiler flags used to build AMReX, unless CMAKE_CXX_FLAGS is explicitly provided, or the environmental variables CXXFLAGS is set.

Building incflo for Summit (OLCF)

For the Summit cluster at OLCF, you first need to load/unload modules required to build incflo.

> module unload xalt
> module unload darshan
> module load gcc
> module load cmake/3.14.0

Now incflo can be built following the SUPERBUILD Instructions (recommended).

To build incflo for GPUs, you need to load cuda module:

> module load cuda/10.1.105

To compile for GPUs:

> cd incflo
> mdkir build
> cd build
> cmake -DCMAKE_CXX_COMPILER=g++ -DINCFLO_CUDA=yes
> make -j

An example of a submission_script for using the GPUs on Summit can be found in incflo/summit_script.sh. For more information about Summit cluster: https://www.olcf.ornl.gov/for-users/system-user-guides/summit/

Running the code

incflo takes an input file as its first command-line argument. The file may contain a set of parameter definitions that will override defaults set in the code. Many example inputs files can be found in the incflo/test_XXX directories. For example, to run the Taylor-Green vortices example in incflo/test_2d, assuming your executable is named incflo.ex and is located in the test_2d directory, then you can simply type:

> cd incflo/test_2d
> ./incflo.ex benchmark.taylor_green_vortices

For more information on inputs options, see the Run-time Inputs section.

incflo can generate subfolders in the current folder named plt00000, plt00010, etc, and chk00000, chk00010, etc. These are called plotfiles and checkpoint files. The plotfiles are used for visualization of derived fields; the checkpoint files are used for restarting the code. For more information see the AMReX Documentation on I/O (Plotfile, Checkpoint).

Visualizing the Results

Options for visualizing the data are discussed in the AMReX Documenation on Visualization.