rdb_comm_env Module

Wraps pic_mpi_lib for MPI init/finalize, rank/size queries, broadcast, abort, and per-node GPU binding.

Two-phase initialisation: 1. comm_env_init() — basic MPI, cache world rank/size 2. comm_env_setup_roles(use_io_server) — split comms, assign roles, bind GPU

When use_io_server is enabled, the last rank on each node becomes a dedicated I/O server (no GPU, no solver). All solver/decomp/halo operations use compute_comm (excludes I/O ranks). When disabled, compute_comm = comm_world (current behavior, no overhead).

comm_env_push_compute_comm / comm_env_pop_compute_comm are a TEST/TOOLING-ONLY seam: a one-level override of what comm_env_compute_comm() returns, for tests that need a collective to run over something other than the real job communicator (see their docstrings below). Production code never calls them.


Uses

  • module~~rdb_comm_env~~UsesGraph module~rdb_comm_env rdb_comm_env iso_fortran_env iso_fortran_env module~rdb_comm_env->iso_fortran_env module~rdb_constants rdb_constants module~rdb_comm_env->module~rdb_constants pic_mpi_lib pic_mpi_lib module~rdb_comm_env->pic_mpi_lib pic_types pic_types module~rdb_constants->pic_types

Used by

  • module~~rdb_comm_env~~UsedByGraph module~rdb_comm_env rdb_comm_env module~rdb_driver rdb_driver module~rdb_driver->module~rdb_comm_env module~rdb_halo rdb_halo module~rdb_driver->module~rdb_halo module~rdb_ocean_state rdb_ocean_state module~rdb_driver->module~rdb_ocean_state module~rdb_ocean_console_stats rdb_ocean_console_stats module~rdb_driver->module~rdb_ocean_console_stats module~rdb_ocean_dyn rdb_ocean_dyn module~rdb_driver->module~rdb_ocean_dyn module~rdb_ocean_engine rdb_ocean_engine module~rdb_driver->module~rdb_ocean_engine module~rdb_halo->module~rdb_comm_env module~rdb_ocean_api rdb_ocean_api module~rdb_ocean_api->module~rdb_comm_env module~rdb_handle rdb_handle module~rdb_ocean_api->module~rdb_handle module~rdb_ocean_diag_derived rdb_ocean_diag_derived module~rdb_ocean_api->module~rdb_ocean_diag_derived module~rdb_ocean_diag_fills rdb_ocean_diag_fills module~rdb_ocean_api->module~rdb_ocean_diag_fills module~rdb_ocean_api->module~rdb_ocean_dyn module~rdb_ocean_api->module~rdb_ocean_engine module~rdb_ocean_fold_apply rdb_ocean_fold_apply module~rdb_ocean_api->module~rdb_ocean_fold_apply module~rdb_ocean_fold_exchange rdb_ocean_fold_exchange module~rdb_ocean_fold_exchange->module~rdb_comm_env module~rdb_ocean_halo rdb_ocean_halo module~rdb_ocean_halo->module~rdb_comm_env module~rdb_ocean_state->module~rdb_comm_env module~rdb_continuity rdb_continuity module~rdb_ocean_state->module~rdb_continuity module~rdb_ocean_state->module~rdb_ocean_dyn module~rdb_ocean_data_forcing rdb_ocean_data_forcing module~rdb_ocean_state->module~rdb_ocean_data_forcing module~rdb_barotropic_substep rdb_barotropic_substep module~rdb_barotropic_substep->module~rdb_ocean_fold_exchange module~rdb_barotropic_substep->module~rdb_ocean_halo module~rdb_continuity->module~rdb_ocean_fold_exchange module~rdb_continuity->module~rdb_ocean_halo module~rdb_continuity->module~rdb_ocean_fold_apply module~rdb_handle->module~rdb_ocean_state module~rdb_handle->module~rdb_ocean_engine module~rdb_ice_evp rdb_ice_evp module~rdb_ice_evp->module~rdb_ocean_halo module~rdb_ice_transport rdb_ice_transport module~rdb_ice_transport->module~rdb_halo module~rdb_ice_transport->module~rdb_ocean_halo module~rdb_ice_transport->module~rdb_continuity module~rdb_ocean_halo_state rdb_ocean_halo_state module~rdb_ice_transport->module~rdb_ocean_halo_state module~rdb_ocean_bt_wide rdb_ocean_bt_wide module~rdb_ocean_bt_wide->module~rdb_ocean_halo module~rdb_ocean_bt_wide->module~rdb_barotropic_substep module~rdb_ocean_chksum rdb_ocean_chksum module~rdb_ocean_chksum->module~rdb_halo module~rdb_ocean_console_stats->module~rdb_halo module~rdb_ocean_diag_derived->module~rdb_ocean_state module~rdb_ocean_diag_derived->module~rdb_ocean_diag_fills module~rdb_ocean_diag_fills->module~rdb_ocean_state module~rdb_ocean_dyn->module~rdb_ocean_halo module~rdb_ocean_dyn->module~rdb_barotropic_substep module~rdb_ocean_dyn->module~rdb_continuity module~rdb_ocean_dyn->module~rdb_ocean_bt_wide module~rdb_ocean_dyn->module~rdb_ocean_chksum module~rdb_ocean_dyn->module~rdb_ocean_console_stats module~rdb_ocean_dyn->module~rdb_ocean_fold_apply module~rdb_ocean_dyn->module~rdb_ocean_halo_state module~rdb_ocean_engine->module~rdb_halo module~rdb_ocean_engine->module~rdb_ocean_fold_exchange module~rdb_ocean_engine->module~rdb_ocean_halo module~rdb_ocean_engine->module~rdb_ocean_state module~rdb_ocean_engine->module~rdb_ice_evp module~rdb_ocean_engine->module~rdb_ice_transport module~rdb_ocean_engine->module~rdb_ocean_diag_derived module~rdb_ocean_engine->module~rdb_ocean_diag_fills module~rdb_ocean_engine->module~rdb_ocean_dyn module~rdb_ocean_engine->module~rdb_ocean_fold_apply module~rdb_ocean_engine->module~rdb_ocean_halo_state module~rdb_ocean_setup rdb_ocean_setup module~rdb_ocean_engine->module~rdb_ocean_setup module~rdb_ice_ocean_coupler rdb_ice_ocean_coupler module~rdb_ocean_engine->module~rdb_ice_ocean_coupler module~rdb_ocean_engine->module~rdb_ocean_data_forcing module~rdb_ocean_fold_apply->module~rdb_ocean_fold_exchange module~rdb_ocean_halo_state->module~rdb_ocean_halo module~rdb_ocean_halo_state->module~rdb_ocean_fold_apply module~rdb_ocean_setup->module~rdb_halo module~rdb_ocean_setup->module~rdb_ocean_fold_exchange module~rdb_ocean_setup->module~rdb_ocean_halo module~rdb_ocean_setup->module~rdb_ocean_state module~rdb_ocean_setup->module~rdb_ocean_dyn module~rdb_ocean_setup->module~rdb_ocean_fold_apply module~rdb_ocean_setup->module~rdb_ocean_halo_state proc~ocean_cavity_mass_step ocean_cavity_mass_step proc~ocean_cavity_mass_step->module~rdb_halo module~rdb_ice_ocean_coupler->module~rdb_ocean_halo_state module~rdb_ocean_data_forcing->module~rdb_ocean_halo_state

Variables

Type Visibility Attributes Name Initial
integer, private, parameter :: COMPUTE_COMM_OVERRIDE_MAX_DEPTH = 1

TEST/TOOLING-ONLY. One-level stack depth for comm_env_push_compute_comm / comm_env_pop_compute_comm — see their docstrings. Kept at 1 deliberately: nothing in production code nests the override, and a deeper stack would hide a forgotten pop instead of failing loud on the very next push.
integer, private, save :: cached_compute_rank = -1
integer, private, save :: cached_compute_size = -1
integer, private, save :: cached_io_server_rank = -1

World rank of this node’s I/O server (-1 if none)
logical, private, save :: cached_is_io_server = .false.
integer, private, save, allocatable :: cached_node_compute_ranks(:)

World ranks of compute processes on this node
integer, private, save :: cached_node_n_compute = 0

Number of compute ranks on this node
integer, private, save :: cached_rank = -1
integer, private, save :: cached_size = -1
type(comm_t), private, save :: comm_compute

Compute-only communicator (excludes I/O ranks)
type(comm_t), private, save :: comm_global

Global MPI communicator (cached after init)
type(comm_t), private, save :: compute_comm_override

TEST/TOOLING-ONLY override payload (valid only while compute_comm_override_depth == 1).
integer, private, save :: compute_comm_override_depth = 0

TEST/TOOLING-ONLY override stack depth, 0 or 1.
logical, private, save :: env_initialised = .false.

True between comm_env_init and comm_env_finalize. Makes init idempotent and finalize a no-op when MPI was never initialised (e.g. a test binary that never inits the comm-env), so the shared per-test main can call finalize unconditionally at exit.

Functions

public function comm_env_compute_comm() result(comm)

Return the compute-only communicator Falls back to comm_world() if comm_env_init has not been called (e.g. tests that use raw MPI_Init).

Read more…

Arguments

None

Return Value type(comm_t)

public function comm_env_compute_rank() result(rank)

Return this process’s rank in the compute communicator

Arguments

None

Return Value integer

public function comm_env_compute_size() result(nprocs)

Return number of compute ranks (excludes I/O ranks)

Arguments

None

Return Value integer

public function comm_env_global_comm() result(comm)

Return the cached global (world) communicator.

Read more…

Arguments

None

Return Value type(comm_t)

public function comm_env_io_server_rank() result(rank)

Return world rank of this node’s I/O server (-1 if none)

Arguments

None

Return Value integer

public function comm_env_is_io_server() result(is_io)

Return .true. if this rank is a dedicated I/O server

Arguments

None

Return Value logical

public function comm_env_node_compute_ranks() result(ranks)

Return world ranks of compute processes on this node

Arguments

None

Return Value integer, allocatable, (:)

public function comm_env_node_n_compute() result(n)

Return number of compute ranks on this node

Arguments

None

Return Value integer

public function comm_env_rank() result(rank)

Return this process’s world MPI rank

Arguments

None

Return Value integer

public function comm_env_size() result(nprocs)

Return total number of MPI processes (including I/O ranks)

Arguments

None

Return Value integer


Subroutines

public subroutine comm_env_abort(code)

Abort all MPI processes with given error code

Arguments

Type IntentOptional Attributes Name
integer, intent(in) :: code

public subroutine comm_env_bcast_real(val)

Broadcast a single real(wp) scalar from compute rank 0 Uses compute communicator so I/O server ranks don’t participate.

Arguments

Type IntentOptional Attributes Name
real(kind=wp), intent(inout) :: val

public subroutine comm_env_finalize()

Finalise MPI. No-op if the comm-env was never initialised, so it is safe to call unconditionally at program exit – the shared per-test main does this for the few tests that init MPI, and it costs nothing for the rest (and on serial builds via the stub).

Arguments

None

public subroutine comm_env_init()

Phase 1: Initialise MPI, cache world rank/size Call this before reading config. Idempotent: a second call is a no-op, so multiple entry points / testdrive cases can call it.

Arguments

None

public subroutine comm_env_pop_compute_comm()

TEST/TOOLING-ONLY. Pops the override pushed by comm_env_push_compute_comm, restoring comm_env_compute_comm() to the real compute communicator. Fails loud on pop-without-push (a bug in the caller, not a state this seam should absorb silently). Does NOT finalize the popped communicator – the pusher owns that (e.g. freeing a self-comm built for the push).

Arguments

None

public subroutine comm_env_push_compute_comm(comm)

TEST/TOOLING-ONLY. Temporarily overrides what comm_env_compute_comm() returns to comm, until the matching comm_env_pop_compute_comm().

Read more…

Arguments

Type IntentOptional Attributes Name
type(comm_t), intent(in) :: comm

public subroutine comm_env_setup_roles(use_io_server)

Phase 2: Assign roles (I/O server vs compute), bind GPU Call this after reading config, before decomp/solver init.

Arguments

Type IntentOptional Attributes Name
logical, intent(in) :: use_io_server

Dedicate one rank per node as I/O server

private subroutine print_gpu_binding(requested_gpu)

Per-rank one-shot diagnostic: world rank -> requested device num / actual device num reported by the OpenACC runtime. Lets us confirm that mpirun is binding each rank to its own GPU rather than serialising N ranks on device 0.

Arguments

Type IntentOptional Attributes Name
integer, intent(in) :: requested_gpu