Runtime Options

Time Stepping

The inputs below do not take a prefix. Note that if the first two are both specified, both criteria are used and the simulation still stop when the first criterion is hit. The simulation will stop when either the number of steps reaches max_step or time reaches stop_time. Note that if the code reaches stop_time then the final time step will be shortened so as to end exactly at stop_time, not past it.

	Description	Type	Default
max_step	Maximum number of time steps to take	Int	-1
stop_time	Maximum time to reach	Real	-1.0
stop_when_steady	Stop when steady state is reached	Bool	false
steady_tol	Specify tolerance to define steady state	Real	1e-10

The inputs below must be preceded by “ns.”

	Description	Type	Default
fixed_dt	Value of fixed dt if > 0	Real	-1.
cfl	CFL constraint (dt < cfl * dx / u) if fixed_dt not > 0	Real	0.5
init_shrink	Factor by which to shrink the initial time step	Real	1.0
max_change	Factor by which time step can grow in subsequent steps	Real	1.1
dt_cutoff	Time step below which the simulation will abort	Real	0.0

If you want to fix the dt, simply set ns.fixed_dt = XXX to set the fluid time step at level 0. Note that init_shrink can be used in conjunction with fixed_dt and the initial time steps will be reduced.

If you want to let the code determine the appropriate time step using the advective CFL condition, then set ns.cfl = 0.7 for example, and the fluid time step will be computed to be dt = 0.7 * dx / max(vel).

Note that the cfl defaults to 0.5 so it does not have to be set in the inputs file. If neither ns.cfl nor fixed_dt is set, then default value of cfl will be used. If ns.fixed_dt is set, then it will override the cfl option whether ns.cfl is set or not.

As an example, consider:

ns.cfl = 0.9
ns.init_shrink = 0.01
ns.change_max = 1.1
ns.dt_cutoff = 1.e-20

This defines the cfl parameter to be 0.9, but sets (via init_shrink) the first timestep we take to be 1% of what it would be otherwise. This allows us to ramp up to the hydrodynamic timestep at the start of a simulation. The change_max parameter restricts the timestep from increasing by more than 10% over a coarse timestep. Note that the time step can shrink by any factor; this only controls the extent to which it can grow. The dt_cutoff parameter will force the code to abort if the timestep ever drops below \(10^{-20}\). This is a safety feature—if the code hits such a small value, then something likely went wrong in the simulation, and by aborting, you won’t burn through your entire allocation before noticing that there is an issue.

Output Options

Here we give an overview of some of the output options for IAMR. For more detailed information on I/O, please see Parallel I/O and AMReX’s documentation I/O (Plotfile, Checkpoint)

Plotfiles

The following inputs must be preceded by “amr.” and control frequency, naming, and content of the plotfiles.

	Description	Type	Default
plot_int	Frequency of plotfile output; if -1 then no plotfiles will be written at this frequency	Int	-1
plot_per	Time period of plotfile output (approximate); does not modify dt. If -1 then no plotfiles will be written at this frequency	Real	-1
plotfile_on_restart	Should we write a plotfile when we restart (only used if plot_int>0)	Bool	False
plot_file	Prefix to use for plotfile output	String	plt
plot_vars	State variables to include in plotfile	String	ALL
derive_plot_vars	Derived variables to include in plotfile	String	NONE
plot_nfiles	Max number of files to use in parallel output if the number of processors exceeds this number, else the two are equal. If -1, number of files is always set equal to number of processors.	Int	64

The following inputs must be preceded by “ns.”

	Description	Type	Default
set_plot_coveredCell_val	For EB, set all covered cells? If 1, sets covered cells to zero. If 0, show covered cells without setting a value for the plotfile; for debugging, as covered cells are not used in the algorithm.	Int	1

Note:

amr.plot_per will write a plotfile at the first timestep whose ending time is past an integer multiple of this interval. The timestep is not modified to match this interval, so you won’t get a plotfile at exactly the time you requested, unless you’ve also set up ns.fixed_dt.
You can specify both amr.plot_int or amr.plot_per if you so desire; the code will print a warning in case you did this unintentionally. If both are non-negative values, you will get plotfiles at integer multiples of amr.plot_int timesteps and at multiples of amr.plot_per simulation time intervals.
All the options for amr.derive_plot_vars are kept in derive_lst in NS_setup.cpp. Feel free to look at it and see what’s there.

As an example:

amr.plot_file = plt_run
amr.plot_int = 10

means that plotfiles (really directories) starting with the prefix plt_run will be generated every 10 level-0 time steps. The directory names will be plt_run00000, plt_run00010, plt_run00020, etc.

If instead you specify

amr.plot_file = plt_run
amr.plot_per = 0.1

then plotfiles (really directories) starting with the prefix plt_run will be generated every 0.1 units of simulation time. The directory names may be something like plt_run00000, plt_run00043, plt_run00061, etc, where 43 level-0 steps is the first point when simulation time \(>= 0.1\), and 61 level-0 steps is the first point when simulation time \(>=0.2\), etc.

Checkpointing and Restarting

The following inputs must be preceded by “amr.” and control checkpoint/restart.

	Description	Type	Default
restart	If present, then the name of file to restart from	String	None
check_int	Frequency of checkpoint output by level 0 time steps; if -1 then no checkpoints will be written	Int	-1
check_per	Frequency of checkpoint output by (approximate) simulation time; if -1 then no checkpoints will be written. (See note below)	Real	-1.0
checkpoint_files_output	Should we write checkpoint files? (Takes precedent over check_int and check_per)	Int	1
check_file	Prefix to use for checkpoint output	String	chk
checkpoint_on_restart	Write a checkpoint immediately after restarting? (If 1 yes, if 0 no)	Int	0
checkpoint_nfiles	Max number of files to use in parallel output if the number of processors exceeds this number, else the two are equal. If -1, number of files is always set equal to number of processors.	Int	64

Note:

amr.check_per will write a checkpoint at the first timestep whose ending time is past an integer multiple of this interval. The timestep is not modified to match this interval, so you won’t get a checkpoint at exactly the time you requested, unless you’ve also set up ns.fixed_dt.
You can specify both amr.check_int or amr.check_per if you so desire; the code will print a warning in case you did this unintentionally. If both are non-negative values, you will get checkpoints at integer multiples of amr.check_int timesteps and at multiples of amr.check_per simulation time intervals.

As an example,

amr.check_file = chk_run
amr.check_int = 10

means that restart files (really directories) starting with the prefix chk_run will be generated every 10 level-0 time steps. The directory names will be chk_run00000, chk_run00010, chk_run00020, etc.

To restart from chk_run00061, for example, then set

amr.restart = chk_run00061

Particles Output

Checkpoint Files

The particle positions and velocities are stored in a binary file in each checkpoint directory. This format is designed for being read by the code at restart rather than for diagnostics.

Plot Files

The “derived quantity” particle_count represents the number of particles in a grid cell. To visualize the particle locations as represented on the grid, add particle_count to the list of derived quanties in amr.derive_plot_vars = in the inputs file.

If particles.write_in_plotfile = 1 in the inputs file, then the particle positions and velocities will be written in a binary file in each plotfile directory. This allows the use of the AMReX tools such as the particle comparison tool found in amrex/Tools/Postprocessing/C_Src/, and/or amrex/Tools/Py_util/amrex_particles_to_vtp to generate a vtp file you can open with ParaView.

ASCII Particle Files

To generate an ASCII file containing the particle positions and velocities, one needs to restart from a checkpoint file from a run with particles, but one doesn’t need to run any steps. For example, if chk00350 exists, then one can set:

amr.restart = chk00350
max\_step = 350
particles.particle\_output\_file = *particle\_output*

This tells the code to restart from chk00350, not to take any further time steps, and to write an ASCII-format file called particle_output. This file has the same format as the ASCII input file:

number of particles
x y z

Log Files

The following inputs must be preceded by “amr.”

	Description	Type
grid_log	If present, then the name of file to write grids; otherwise nothing written.	String
run_log	If present, then the name of file to write run output; otherwise nothing written.	String
run_log_terse	If present, then the name of file to write number of steps, time, and dt; otherwise nothing written.	String

As an example:

amr.grid_log = grdlog
amr.run_log = runlog

Every time the code regrids it prints a list of grids at all relevant levels. Here the code will write these grids lists into the file grdlog. Additionally, every time step the code prints certain statements to the screen, such as (if amr.v = 1):

STEP = 1 TIME = 1.91717746 DT = 1.91717746
PLOTFILE: file = plt00001

The run_log option will write these statements into the file runlog as well.

Terser output can be obtained via:

amr.run_log_terse = runlogterse

This file, runlogterse differs from runlog, in that it only contains lines of the form

10  0.2  0.005

in which “10” is the number of steps taken, “0.2” is the simulation time, and “0.005” is the level-0 time step. This file can be plotted very easily to monitor the time step.

Gridding and Load Balancing Inputs

The details of the regridding strategy are described in AMR Grids; here we cover how the input parameters can control the gridding.

These parameters can have a large impact on the performance of IAMR, so taking the time to experiment with is worth the effort. Having grids that are large enough to coarsen multiple levels in a V-cycle is essential for good multigrid performance.

Gridding

The following inputs must be preceded by “amr.” and determine how we create the grids and how often we regrid. (Additional information can also be found in AMReX documentation at AmrCore Source Code: Details.)

	Description	Type	Default
regrid_int	How often to regrid (in number of steps at level 0) if regrid_int = -1 then no regridding will occur	Int	-1
regrid_on_restart	Should we regrid immediately after restarting?	Int	0
max_grid_size_x	Maximum number of cells in each grid in x-direction, for all levels	Int	32
max_grid_size_y	Maximum number of cells in each grid in y-direction, for all levels	Int	32
max_grid_size_z	Maximum number of cells in each grid in z-direction, for all levels	Int	32
max_grid_size	Maximum number of cells in each grid in all directions. Specify multiple values to give levels a different max_grid_size	Int	32
blocking_factor_x	Each grid must be divisible by blocking_factor_x in x-direction	Int	8
blocking_factor_y	Each grid must be divisible by blocking_factor_y in y-direction	Int	8
blocking_factor_z	Each grid must be divisible by blocking_factor_z in z-direction	Int	8
blocking_factor	Each grid must be divisible by blocking_factor in all directions. Specify multiple values to give levels a different blocking_factor Must be a power of 2 at every level and the domain size must be a multiple of blocking_factor at level 0.	Int	8
grid_eff	grid efficiency (must be between 0 and 1)	Real	0.7
n_error_buf	radius of additional tagging around already tagged cells	Int	1
refine_grid_layout	refine grids more if # of processors \(>\) # of grids	Int	1
regrid_file	Name of file from which to read the grids, if specifying fixed grids	Text	None

Note that if regrid_file is set (e.g. amr.regrid_file = fixed_grids), then the list of grids at each fine level are read in from this file during the gridding procedure. These grids must not violate the amr.max_grid_size criterion. The rest of the gridding procedure will not occur if amr.regrid_file is set.

Note also that amr.ref_ratio, amr.n_error_buf, amr.max_grid_size and amr.blocking_factor can be read in as a single value which is assigned to every level, or as multiple values, one for each level.

As an example, consider:

amr.regrid_int = 2 2
amr.grid_eff = 0.9
amr.max_grid_size = 64
amr.blocking_factor = 32

tells the code to regrid every 2 steps. Thus in this example, new level 1 grids will be created every 2 level-0 time steps, and new level 2 grids will be created every 2 level-1 time steps. If amr.regrid_int \(<\) 0 for any level, then regridding starting at that level will be disabled. If amr.regrid_int = -1 only, then we never regrid for any level. Note that this is not compatible with amr.regrid_on_restart = 1.

The grid efficiency, amr.grid_eff, means that during the grid creation process, at least 90% of the cells in each grid at the level at which the grid creation occurs must be tagged cells. A higher grid efficiency means fewer cells at higher levels, but may result in the production of lots of small grids, which have inefficient cache and OpenMP performance and higher communication costs.

The amr.max_grid_size parameter means that the final grids will be no longer than 64 cells on a side at every level. Alternately, we could specify a value for each level of refinement as: amr.max_grid_size = 64 32 16, in which case our final grids will be no longer than 64 cells on a side at level 0, 32 cells on a side at level 1, and 16 cells on a side at level 2. The amr.blocking_factor means that all of the final grids will be multiples of 32 at all levels. Again, this can be specified on a level-by-level basis, like amr.blocking_factor = 32 16 8, in which case the dimensions of all the final grids will be multiples of 32 at level 0, multiples of 16 at level 1, and multiples of 8 at level 2.

Tiling

For details on IAMR’s approach to tiling see Parallelization.

The following inputs determine how we create the logical tiles and must be preceded by “fabarray_mfiter.” :

	Description	Type	Default
tile_size	Maximum number of cells in each direction for (logical) tiles. (Default for 3D CPU-only)	IntVect	1024000 (1024000,8,8)

Verbosity

Different classes control their own verbosity. In some cases, values > 1 will generate additional verbosity. Here is some of the more frequently used options:

	Description	Type
ns.v	Verbosity in IAMR routines	Int
ns.amr	Verbosity in AMR routines	Int
particles.verbose	Verbosity in particle routines	Int
nodal_proj.verbose	Verbosity in nodal projection	Int
mac_proj.verbose	Verbosity in MAC projection	Int

Multigrid Inputs

IAMR uses AMReX’s multigrid functionality to perform the nodal projection (which enures the cell-centered velocity field obeys the constraint), the MAC projection (which ensures that the edge-based velocity field used in advection obeys the constraint), and the diffusive solves.

Here we go over some inputs parameters that can be used to control these solves. For more information on AMReX’s linear solvers, see Linear Solvers

Nodal Projection

These control the nodal projection and must be preceded by “nodal_proj.”:

	Description	Type	Default
verbose	Verbosity in nodal projection	Int	0
bottom_verbose	Verbosity of the bottom solver in nodal projection	Int	0
proj_tol	Relative tolerance in nodal projection	Real	1.e-12
sync_tol	Relative tolerance in sync projection	Real	1.e-10
proj_abs_tol	Absolute tolerance in nodal & sync projections	Real	1.e-16
maxiter	Maximum number of iterations in the nodal projection	Int	100
bottom_maxiter	Maximum number of iterations in the nodal projection bottom solver if using bicg, cg, bicgcg or cgbicg	Int	100
mg_max_coarsening_level	Maximum number of coarser levels to allow in the nodal projection If set to 0, the bottom solver will be called at the current level	Int	30
bottom_solver	Which bottom solver to use in the nodal projection Options are bicgcg, bicgstab, cg, cgbicg, smoother or hypre	String	bicgcg

MAC Projection

These control the MAC projection and must be preceded by “mac_proj.”:

	Description	Type	Default
verbose	Verbosity of multigrid solver in MAC projection	Int	0
bottom_verbose	Verbosity of bottom solver in MAC projection	Int	0
mac_tol	Relative tolerance in MAC projection	Real	1.e-12
mac_sync_tol	Relative tolerance in MAC sync projection	Real	1.e-10
mac_abs_tol	Absolute tolerance in MAC & sync projection	Real	1.e-16
maxiter	Maximum number of iterations in the MAC projection	Int	200
bottom_maxiter	Maximum number of iterations in the MAC projection bottom solver if using bicg, cg, bicgcg or cgbicg	Int	200
mg_max_coarsening_level	Maximum number of coarser levels to allow in the MAC projection If set to 0, the bottom solver will be called at the current level	Int	100
bottom_solver	Which bottom solver to use in the MAC projection Options are bicgcg, bicgstab, cg, cgbicg, smoother or hypre	String	bicgcg

Viscous and Diffusive Solve

These control the diffusion solver and must be preceded by “diffusion.”:

	Description	Type	Default
v	Verbosity of linear solver for diffusion solve	Int	0

Initializing the Calculation

These options determine how we initialize the data for the calculation. The data initialization process ensures that the initial state is consistent with the constraint, and if applicable, the various AMR levels are consistent with each other.

The following inputs must be preceded by “ns.”

	Description	Type	Default
do_init_proj	Do the initial projections? False is primarily for debugging.	Bool	True
init_iter	How many pressure iterations before starting the first timestep.	Int	3
init_vel_iters	How many projection iterations to ensure the velocity satisfies the constraint. Set = 0 to skip this part of the initialization.	Int	3

Checking Conservation

IAMR has the option to sum up the total mass (and a few other quantities) at a user-defined interval.

	Description	Type	Default
ns.sum_interval	How often (in level-0 time steps) to compute and print integral quantities. If <= 0, do nothing.	Int	-1

The integral quantities include total mass, tracer, kinetic energy and magnitude of vorticity. They are summed over the entire the domain every ns.sum_interval level-0 steps. The print statements have the form

TIME= 1.91717746 MASS= 1.792410279e+02

for example.

Note that by default the tracer not conservative. To conservatively advect the tracer, that option must be set in the inputs (see Conservative vs. Non-conservative).