Floating-point addition is not associative, so Sum x_i depends on the
order of summation – the reduction tree shape, the rank count, the
decomposition. This module maps each real onto a fixed-point integer
vector (efp_t) so accumulation becomes plain integer(int64)
addition, which IS associative and exact. A base exponent P
(EFP_PREC_WIDTH) splits the mantissa into EFP_DIGITS = 6 bins of
weight 2**(P*(3-n)) for n = 1..6; the decomposition
(efp_decompose) is a pure function of the input value, so two ranks
decomposing the same value produce identical bins, and summing bins
(efp_plus) is then order-invariant by construction.
This module is host-only, pure-arithmetic, and depends on nothing but
rdb_constants (for wp) and iso_fortran_env – no MPI, no grid, no
state – so src/comm/ can consume it (halo_allreduce_efp_list)
with no dependency cycle, and rdb_console_stats (shared coastal +
ocean) can too.
Parameter choice: EFP_PREC_WIDTH = 36 (MOM6 uses 46). Roundabout
picks a smaller P because the cross-rank transport (see
halo_allreduce_efp_list in src/comm/) has no integer(int64)
MPI allreduce available (pic_mpi_lib binds only dp/sp/i32
allreduce overloads) and instead transports the bins as exactly-
represented real64 values summed by MPI_SUM – exact only while
every partial sum stays within the 53-bit double mantissa. At
P = 36 that bounds the rank count to EFP_MAX_RANKS = 2**17 =
131072, far beyond any Roundabout run, while EFP_MAX_SUMMANDS = 2**27
~= 1.34e8 per local reduction block comfortably covers a single
k-slab of an 11500^2 grid. See docs/CAPABILITIES_AND_LIMITATIONS.md
for the full bound table.
Do NOT widen EFP_PREC_WIDTH without re-deriving EFP_MAX_RANKS
and EFP_MAX_SUMMANDS – both are parameters computed FROM it, and
test_efp_bounds pins the arithmetic.
Non-finite propagation. A fixed-point decomposition has nowhere to
put a NaN or +-Inf – int(NaN, int64) is compiler-undefined, and
the naive fallback (zero the bins, matching a NaN’s “no value”
intuition) is exactly the wrong choice for a REDUCTION: it makes a
poisoned summand silently vanish, and the reconstructed total comes
back as a plausible finite number instead of aborting – the same
hazard class as CLAUDE.md’s NaN-blind if/else clamp gotcha, just in
the reduction layer. efp_t therefore carries a poison counter
alongside its bins (see the type’s own docstring): every entry point
(efp_decompose/efp_from_real/efp_plus/efp_minus/efp_to_real/
efp_to_transport/efp_from_transport) propagates it, so a single
NaN/+-Inf/bin-1-overflow summand ANYWHERE in an accumulation –
local or cross-rank, through halo_allreduce_efp_list – makes
efp_to_real return a quiet NaN, identically on every rank, instead
of laundering the corruption into “0.000” or a saturated-but-finite
value. This was a real regression, not a hypothetical: once
&ocean_diag_nml reproducing_sums became the console default
(a0755ada1), the in-module device duplicate
rdb_ocean_console_stats::efp_decompose_impl had NO non-finite
handling at all, so a blown-up run’s console printed finite-looking
En/Salt/Temp columns instead of the NaN a diverged run must
show – see that module’s efp_decompose_impl docstring and
tests/test_efp.F90’s test_efp_poison_propagates_nan/_inf.
Nodes of different colours represent the following:
Solid arrows point from a submodule to the (sub)module which it is
descended from. Dashed arrows point from a module or program unit to
modules which it uses.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a submodule to the (sub)module which it is
descended from. Dashed arrows point from a module or program unit to
modules which it uses.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Variables
Type
Visibility
Attributes
Name
Initial
integer,
public,
parameter
::
EFP_DIGITS
=
6
Number of fixed-point bins per efp_t value.
integer,
public,
parameter
::
EFP_GUARD_WIDTH
=
63-EFP_PREC_WIDTH
= 27. int64 has 63 usable bits (sign-magnitude use here, not
two’s-complement range); this is the headroom for accumulating
multiple summands into one bin before it could overflow.
integer,
public,
parameter
::
EFP_MAX_RANKS
=
2**(53-EFP_PREC_WIDTH)
= 131072. Upper bound on the number of MPI ranks the double-
precision transport (halo_allreduce_efp_list) can combine
exactly: after a local efp_carry, bins 2..6 satisfy
|e(n)| < 2**P, so a partial sum over EFP_MAX_RANKS ranks stays
<= 2**53, the largest exactly-representable double integer.
integer(kind=int64),
public,
parameter
::
EFP_MAX_SUMMANDS
=
2_int64**EFP_GUARD_WIDTH
= 134217728 (~1.34e8). Local (single-rank, single k-slab) upper
bound on the number of values that may be accumulated into one
bin via efp_plus before an efp_carry is required to keep
|bin| < 2**63. Enforced fail-loud at the kernel call site
(rdb_ocean_console_stats), not silently.
integer,
public,
parameter
::
EFP_PREC_WIDTH
=
36
Bits per bin (P). See the module docstring for the derivation.
integer,
public,
parameter
::
EFP_TRANSPORT_WIDTH
=
EFP_DIGITS+1
Reals transported per efp_t value by efp_to_transport/
efp_from_transport: the EFP_DIGITS fixed-point bins plus ONE
extra slot for the non-finite “poison” counter (efp_t%poison,
see its docstring). Callers that size their own send/recv buffers
(halo_allreduce_efp_list) MUST use this, not EFP_DIGITS,
or the poison slot silently aliases the next value’s bin 1.
real(kind=real64),
private,
parameter
::
EFP_IPR1
=
1.0_real64/EFP_PR1
real(kind=real64),
private,
parameter
::
EFP_IPR2
=
1.0_real64/EFP_PR2
real(kind=real64),
private,
parameter
::
EFP_IPR3
=
1.0_real64/EFP_PR3
real(kind=real64),
private,
parameter
::
EFP_IPR4
=
1.0_real64/EFP_PR4
real(kind=real64),
private,
parameter
::
EFP_IPR5
=
1.0_real64/EFP_PR5
real(kind=real64),
private,
parameter
::
EFP_IPR6
=
1.0_real64/EFP_PR6
real(kind=real64),
private,
parameter
::
EFP_PR1
=
2.0_real64**(2*EFP_PREC_WIDTH)
real(kind=real64),
private,
parameter
::
EFP_PR2
=
2.0_real64**(1*EFP_PREC_WIDTH)
real(kind=real64),
private,
parameter
::
EFP_PR3
=
1.0_real64
real(kind=real64),
private,
parameter
::
EFP_PR4
=
2.0_real64**(-1*EFP_PREC_WIDTH)
real(kind=real64),
private,
parameter
::
EFP_PR5
=
2.0_real64**(-2*EFP_PREC_WIDTH)
real(kind=real64),
private,
parameter
::
EFP_PR6
=
2.0_real64**(-3*EFP_PREC_WIDTH)
integer(kind=int64),
private,
parameter
::
EFP_PREC_I64
=
2_int64**EFP_PREC_WIDTH
2**P as an int64 – the bin-2..6 magnitude bound used by
efp_carry / efp_regularize.
An order-invariant fixed-point value: Sum_n v(n) * pr(n). The
component is PUBLIC (unlike MOM6’s private EFP_type%v) so the
comm facade (halo_allreduce_efp_list) can pack/unpack it without
an accessor procedure – Roundabout splits the arithmetic module
(here) from the collective (in src/comm/), so the layering
requires v to be reachable from both. Tests asserting bit-
identity compare v(:) directly, never the reconstructed real.
Components
Type
Visibility
Attributes
Name
Initial
integer(kind=int64),
public
::
poison
=
0_int64
Non-finite “poison” counter – the number of NaN / +-Inf /
bin-1-overflow summands folded into this value so far (0 =
clean). efp_decompose/efp_from_real seed it from is_nan
.or. is_ovf; efp_plus/efp_minus add it forward (never
reset it); efp_to_real returns a quiet NaN whenever it is
nonzero, INSTEAD OF reconstructing a value from v(:). This
is the fix for the EFP fixed-point path laundering a non-finite
summand into a plausible finite number (0 comes out of a NaN
decompose’s zeroed bins, a saturated bin 1 comes out of an
overflowing one) – the exact hazard class CLAUDE.md’s
NaN-blind-clamp gotcha describes, just in the reduction layer
instead of a clamp. efp_to_transport/efp_from_transport
carry it in the SAME collective as the bins (see
EFP_TRANSPORT_WIDTH), summed by the same MPI_SUM: one
poisoned rank makes the transported count nonzero on every
rank, so every rank’s efp_to_real reports NaN identically –
never a rank-dependent branch. A plain count (not a saturating
flag) because it costs nothing extra (still an exact double
under MPI_SUM at any realistic magnitude) and is simpler to
reason about than a boolean OR chain.
Bin 1 is NOT bounded by efp_carry (only bins 2..6 are – see the
module docstring), so the double-precision transport needs its
own guard: after summing nranks local values, the partial sum
in bin 1 must stay <= 2**53 / nranks for the MPI_SUM-on-doubles
combine to remain exact (the analogue of MOM6’s
prec_error = huge(1_int64) / num_PEs, but for the int64-as-
double transport rather than int64 transport). .false. ⇒ the
caller must error stop, never silently proceed.
real64 -> efp_t. Wraps efp_decompose; unlike the pre-fix
version, the NaN / overflow flags are NOT discarded – they seed
a%poison (nonzero iff is_nan .or. is_ovf), so a poisoned
summand still taints every later efp_plus/efp_to_real even
though the flags themselves aren’t returned here (callers needing
the raw flags call efp_decompose directly).
Exact bin-wise integer subtraction, regularised. See efp_plus
for why regularisation (not mere carry) is required for
bit-for-bit order invariance, and for the poison propagation
(additive here too – subtraction of a poisoned operand is still
poisoned, never “cancels” back to clean).
Exact bin-wise integer addition, REGULARISED (not merely
carried). Order-invariant BIT-FOR-BIT:
efp_plus(a,b)%v == efp_plus(b,a)%v, and a running fold
acc = efp_plus(acc, x_i) over any permutation of the x_i
converges to the SAME raw bins (not merely the same reconstructed
real) – test_efp_order_invariant asserts this on %v(:)
directly, per the plan’s §9.2.
real64 = efp_to_real(efp_minus(a, b)) – the difference of two
~1e21-scale EFP totals resolved to the EFP quantum (2**-3P), NOT
to ulp(1e21) as a double subtraction would give. This is the
fix documented in the plan’s SS2.2: an implementer who converts
both operands to real64 FIRST and subtracts loses the whole
benefit of this module.
efp_t -> real64. Regularises a LOCAL COPY of a (never mutates
the argument) – pure with intent(in), per
FORTRAN_STYLE.md’s “default new procedures to pure” (MOM6’s
EFP_to_real instead mutates its intent(inout) argument).
Renormalise bins 6..2 into (-2**P, 2**P), propagating the excess
into the next-more-significant bin, without changing the
represented value. Bin 1 is left untouched (unbounded by
construction; see the module docstring on EFP_MAX_RANKS).
Mirrors MOM6 carry_overflow, which loops
EFP_DIGITS..2 for the same reason.
Arguments
Type
Intent
Optional
Attributes
Name
integer(kind=int64),
intent(inout)
::
e(EFP_DIGITS)
public pure subroutine efp_decompose(r, e, is_nan, is_ovf)
Greedy sign-magnitude fixed-point decomposition of r into six
int64 bins of weight pr(n), n = 1..6. Unrolled (no loop, no
array indexing over pr/I_pr) so this is a template a
!$acc routine seq in-module duplicate can mirror exactly (see
rdb_ocean_console_stats::efp_decompose_impl, which pins against
this procedure in test_efp_impl_matches_canonical).
Inverse of efp_to_transport: unpack a flat
real64(EFP_TRANSPORT_WIDTH*n) buffer (post-collective, still
exact integers as doubles) back into efp_t values, converting
each bin AND the summed poison counter back to int64 and
carrying the bins. ok = .false. iff any unpacked double is not
an exact integer (would indicate the transport-exactness bound
was violated) – the caller (halo_allreduce_efp_list) turns that
into a fail-loud error stop, never a silent truncation.
efp_carry plus: force every bin to share the overall sign, so a
single well-conditioned FP accumulation (efp_to_real) can form
Sum pr(n)*e(n) without alternating-sign cancellation error.
Mirrors MOM6 regularize_ints.
Pack a list of efp_t values into a flat
real64(EFP_TRANSPORT_WIDTH*n) buffer for a single collective
(halo_allreduce_efp_list). Each bin, PLUS the poison counter,
is transported as an EXACTLY-representable double (see
EFP_MAX_RANKS):
buf((i-1)*EFP_TRANSPORT_WIDTH + n) = real(list(i)%v(n)) for
n = 1..EFP_DIGITS, and
buf((i-1)*EFP_TRANSPORT_WIDTH + EFP_DIGITS+1) = real(list(i)%poison).
Summing poison through the SAME MPI_SUM collective as the bins
is what makes a poisoned rank’s contribution reach every other
rank identically – see efp_t’s docstring.