.. _Chap:GettingStarted: Getting Started *************** This section walks you through a brief introduction to using incflo. .. alternative to `` `` is :code:` ` .. _sec:Download: Downloading the code ==================== incflo is built on top of the AMReX framework. In order to run incflo, you must download separate git modules for incflo, AMReX and AMReX-Hydro. First, make sure that git is installed on your machine. #. Download the AMReX repository by typing: .. code:: shell git clone https://github.com/AMReX-Codes/amrex.git This will create a folder called ``amrex/`` on your machine. For additional information on downloading the AMReX source code see `AMReX user guide - Downloading the code `_. #. Download the AMReX-Hydro repository by typing: .. code:: shell git clone https://github.com/AMReX-Fluids/AMReX-Hydro.git This will create a folder called ``AMReX-Hydro/`` on your machine. #. Download the incflo repository by typing: .. code:: shell git clone https://github.com/AMReX-Fluids/incflo.git This will create a folder called ``incflo/`` on your machine. You will want to periodically update each of these repositories by typing ``git pull`` within each repository. Next, you can choose to either :ref:`sec:GNUMake` or :ref:`sec:CMake`. .. _sec:GNUMake: Building with GNU Make ====================== Here, we walk you through compiling an incflo executable. Decide whether you want to build the executable in 2-D vs 3-D, with or without EB (embedded boundaries / cut cells), and choose the corresponding directory: ``test_2d``, ``test_3d``, ``test_no_eb_2d``, or ``test_no_eb``. For this example, let's say you want to build in 2-D with cut cells. Then to build the code: #. ``cd`` to the desired build directory; e.g. .. code:: shell cd incflo/test_2d #. Edit the ``GNUmakefile``: Set AMREX_HOME to be the path to the directory where you have put amrex. NOTE: when setting ``AMREX_HOME`` in the ``GNUmakefile``, be aware that ``~`` does not expand, so ``AMREX_HOME=~/amrex/`` will yield an error. Alternatively, the path to AMReX can be set up as an environment variable, ``AMREX_HOME``, on your machine to point to the path name where you have put AMReX. For example, if you are using the bash shell, you can add this to your ``.bashrc`` as: .. highlight:: bash :: export AMREX_HOME=/path/to/amrex alternatively, in tcsh one can set .. highlight:: tcsh :: setenv AMREX_HOME /path/to/amrex Common options that you can set in the GNUMakefile include +-----------------+------------------------------+------------------+-------------+ | Option name | Description | Possible values | Default | | | | | value | +=================+==============================+==================+=============+ | COMP | Compiler (gnu or intel) | gnu / intel | None | +-----------------+------------------------------+------------------+-------------+ | USE_MPI | Whether to enable MPI | TRUE / FALSE | FALSE | +-----------------+------------------------------+------------------+-------------+ | USE_OMP | Whether to enable OpenMP | TRUE / FALSE | FALSE | +-----------------+------------------------------+------------------+-------------+ | USE_CUDA | Whether to enable CUDA | TRUE / FALSE | FALSE | +-----------------+------------------------------+------------------+-------------+ | DEBUG | Whether to use DEBUG mode | TRUE / FALSE | FALSE | +-----------------+------------------------------+------------------+-------------+ | PROFILE | Include profiling info | TRUE / FALSE | FALSE | +-----------------+------------------------------+------------------+-------------+ | TINY_PROFILE | Include tiny profiling info | TRUE / FALSE | FALSE | +-----------------+------------------------------+------------------+-------------+ | COMM_PROFILE | Include comm profiling info | TRUE / FALSE | FALSE | +-----------------+------------------------------+------------------+-------------+ | TRACE_PROFILE | Include trace profiling info | TRUE / FALSE | FALSE | +-----------------+------------------------------+------------------+-------------+ .. note:: **Do not set both USE_OMP and USE_CUDA to true.** Additional options and information on using other compilers can be found in the AMReX documentation at https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html . #. Make the executable: Now type .. code:: shell make The name of the resulting executable (generated by the make system) encodes several of the build characteristics, including dimensionality of the problem, compiler name, and whether MPI and/or OpenMP were linked with the executable. Thus, several different build configurations may coexist simultaneously in a problem folder. For example, the default build in ``incflo/test_2d`` will look like ``incflo2d.gnu.MPI.EB.ex``, indicating that this is a 2-d version of the code, made with ``COMP=gnu``, ``USE_MPI=TRUE`` and support for embedded boundaries (EB). .. _sec:CMake: Building with CMake =================== CMake build is a two-steps process. First ``cmake`` is invoked to create configuration files and makefiles in a chosen directory (``builddir``). Next, the actual build is performed by invoking ``make`` from within ``builddir``. The CMake build process is summarized as follows: .. highlight:: console :: mkdir /path/to/builddir cd /path/to/builddir cmake [options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] /path/to/incflo make In the above snippet, ``[options]`` indicates one or more options for the customization of the build, as described later in this section. If the option ``CMAKE_BUILD_TYPE`` is omitted, ``CMAKE_BUILD_TYPE=Release`` is assumed. There are two modes to build incflo with CMake: o **SUPERBUILD (recommended):** AMReX and AMReX-Hydro are built as part of the incflo build process. This method is strongly encouraged as it ensures that the configuration options are consistent. o **STANDALONE:** incflo source code is built separately and linked to an existing AMReX installation. This is ideal for continuous integration severs (CI) and regression testing applications. AMReX library version and configuration options must meet incflo requirements. .. note:: **incflo requires CMake 3.14 or higher.** .. _sec:build:superbuild: SUPERBUILD Instructions (recommended) ------------------------------------- In this mode, incflo CMake inherents AMReX CMake targets and configuration options, as well as AMReX-Hydro CMake targets; that is, incflo, AMReX and AMReX-Hydro are configured and built as a single entity. First, make sure you've downloaded all the source codes (see :ref:`sec:Download`). Next, build the incflo executable: .. code-block:: console > cd incflo > mkdir build > cd build > cmake -DAMREX_HOME=/path/to/amrex/source/directory [amrex options] -DAMREX_HYDRO_HOME=/path/to/AMReX-Hydro/source/directory -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] .. > make -j where ``AMREX_HOME`` is a CMake variable pointing to the top-level source directory of the AMReX distribution you downloaded earlier; ``[amrex options]`` is a list of any of the AMReX configuration options listed in the `AMReX user guide - Building with CMake `_; ``AMREX_HYDRO_HOME`` is a CMake variable pointing to the top-level source directory of the AMReX-Hydro distribution you downloaded earlier; and ``..`` is the relative path pointing to the top-level source directory of incflo or this example. .. _sec:build:standalone: **STANDALONE** instructions --------------------------------------------------------------------- For a stand alone build, incflo CMake will look for an existing AMReX installation on the system and link the incflo binaries against it. Building AMReX ~~~~~~~~~~~~~~~~~~~ First, make sure you've downloaded all the source codes (see :ref:`sec:Download`). Next, configure, build and install AMReX as follows: .. code:: shell > cd /path/to/amrex > mkdir build > cd build > cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] [amrex options] -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir -DAMREX_HYDRO_HOME=/path/to/AMReX-Hydro/source/directory .. > make install ``[amrex options]`` is a list of any of the AMReX configuration options listed in the `AMReX user guide - Building with CMake `_. We suggest to always use the option ``-DUSE_XSDK_DEFAULTS=yes`` when building AMReX for incflo. ``AMREX_HYDRO_HOME`` is a CMake variable pointing to the top-level source directory of the AMReX-Hydro distribution you downloaded earlier Building incflo ~~~~~~~~~~~~~~~~~ Build incflo: .. code:: shell > cd /path/to/incflo > mkdir build > cd build > cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] [incflo options] -DAMReX_ROOT=/absolute/path/to/amrex/installdir .. > make -j Passing ``-DAMReX_ROOT=/absolute/path/to/amrex/installdir`` instructs CMake to search ``/absolute/path/to/amrex/installdir`` before searching system paths for an available AMReX installation. ``AMReX_ROOT`` can also be set as an environmental variable instead of passing it as a command line option. ``[incflo options]`` indicates any of the configuration option listed in the table below. +-----------------+------------------------------+------------------+-------------+ | Option name | Description | Possible values | Default | | | | | value | +=================+==============================+==================+=============+ | CMAKE\_CXX\ | User-defined C++ flags | valid C++ | None | | _FLAGS | | compiler flags | | +-----------------+------------------------------+------------------+-------------+ | CMAKE\_CUDA\ | User-defined CUDA flags | valid CUDA | None | | _FLAGS | | compiler flags | | +-----------------+------------------------------+------------------+-------------+ | INCFLO\_DIM | Dimensionality of the build | 2/3 | 3 | | | | | | +-----------------+------------------------------+------------------+-------------+ | INCFLO\_MPI | Enable build with MPI | no/yes | yes | | | | | | +-----------------+------------------------------+------------------+-------------+ | INCFLO\_OMP | Enable build with OpenMP | no/yes | no | | | | | | +-----------------+------------------------------+------------------+-------------+ | INCFLO\_CUDA | Enable build with CUDA | no/yes | no | | | | | | +-----------------+------------------------------+------------------+-------------+ | INCFLO\_EB | Build Embedded Boundary | no/yes | no | | | support | | | +-----------------+------------------------------+------------------+-------------+ | INCFLO\_HYPRE | Enable HYPRE support | no/yes | no | | | | | | +-----------------+------------------------------+------------------+-------------+ | INCFLO\_FPE | Build with Floating-Point | no/yes | no | | | Exceptions checks | | | +-----------------+------------------------------+------------------+-------------+ Few more notes on building incflo ----------------------------------- The system default C++ compiler can be overwritten as follows: .. code:: shell > cmake -DCMAKE_CXX_COMPILER= [options] .. When building on a platform that uses the ``module`` utility, use either the above command (with full path to the compilers) or the following: .. code:: shell > cmake -DCMAKE_CXX_COMPILER=CC [options] .. incflo uses the same compiler flags used to build AMReX, unless ``CMAKE_CXX_FLAGS`` is explicitly provided, or the environmental variables ``CXXFLAGS`` is set. Building incflo for Summit (OLCF) ----------------------------------- For the Summit cluster at OLCF, you first need to load/unload modules required to build incflo. .. code:: shell > module unload xalt > module unload darshan > module load gcc > module load cmake/3.14.0 Now incflo can be built following the :ref:`sec:build:superbuild`. To build incflo for GPUs, you need to load cuda module: .. code:: shell > module load cuda/10.1.105 To compile for GPUs: .. code:: shell > cd incflo > mdkir build > cd build > cmake -DCMAKE_CXX_COMPILER=g++ -DINCFLO_CUDA=yes > make -j An example of a *submission_script* for using the GPUs on Summit can be found in ``incflo/summit_script.sh``. For more information about Summit cluster: ``_ Running the code ================ incflo takes an input file as its first command-line argument. The file may contain a set of parameter definitions that will override defaults set in the code. Many example inputs files can be found in the ``incflo/test_XXX`` directories. For example, to run the Taylor-Green vortices example in ``incflo/test_2d``, assuming your executable is named ``incflo.ex`` and is located in the ``test_2d`` directory, then you can simply type: .. code:: shell > cd incflo/test_2d > ./incflo.ex benchmark.taylor_green_vortices For more information on inputs options, see the :ref:`Chap:Inputs` section. incflo can generate subfolders in the current folder named ``plt00000``, ``plt00010``, etc, and ``chk00000``, ``chk00010``, etc. These are called plotfiles and checkpoint files. The plotfiles are used for visualization of derived fields; the checkpoint files are used for restarting the code. For more information see the AMReX Documentation on :ref:`amrex:Chap:IO`. Visualizing the Results ======================= Options for visualizing the data are discussed in the AMReX Documenation on :ref:`amrex:Chap:Visualization`.