Performance Tuning
genvis fixed-sky: memory and threads
Each serval genvis fixed-sky task takes two resource settings:
--memory-per-task: the memory the task plans for (default 4 GB). The planner sizes each chunk group, i.e. how many frequency channels of one baseline's integrator cache are held at once, so that the cache slab, the prepared beams, the voltage-beam grids and the visibility blocks fit in it. A larger value means fewer, larger groups and fewer cache reads. Check what a value gives with--plan-only.--threads: the threads of the task (default: the cores the task may run on, i.e. its CPU affinity, which under SLURM is the task's cpuset). The contraction against the integrator cache uses all of them.
The remaining knobs are derived from --threads and rarely need setting:
| knob | default |
|---|---|
--power-beam-workers |
threads // 12, at least 1 |
--max-prepared-power-beams (fixed-sky) |
16 per power-beam worker |
--power-beam-nthreads (from-voltage-beams) |
threads // power-beam-workers |
--voltage-beam-workers (from-voltage-beams) |
the number of power-beam workers, at least 1 |
--max-cached-voltage-beams (from-voltage-beams) |
all |
Power-beam workers derive power beams from voltage beams (or load them, for
from-power-beams) while the contraction runs; voltage-beam workers synthesise the voltage
beam grids those workers combine. Both kinds of worker run --power-beam-nthreads threads
each. The workers, the grid workers and the contraction run at the same time, so a task
deliberately holds more threads than cores. A run is refused only when one of these stages
alone asks for more threads than the task's cores.
The workers' spherical harmonic transforms run on ducc0's thread pool, which is sized when ducc0
is imported, from DUCC0_NUM_THREADS, else OMP_NUM_THREADS, else the task's cores. A task
reports OMP_NUM_THREADS and the pool size when it starts, and warns when the pool is smaller
than the threads per power-beam worker: an OMP_NUM_THREADS=1 inherited from a cluster
environment otherwise makes beam preparation several times slower.
--max-prepared-power-beams is the queue between the workers and the contraction: too short
and workers wait for a free slot, too long and the queued beams take memory from the chunk
groups. --max-cached-voltage-beams limits the voltage-beam grids held per chunk group, for
arrays whose baselines touch more voltage beams than fit in memory at once.
Note
The derivation rules come from one machine: an AMD EPYC 9654 node of ETH Zurich's Euler cluster (192 cores in 8 NUMA domains). There the defaults give the production settings: 16 power-beam workers of 12 threads, 16 voltage-beam workers, and 16 prepared beams per worker (256).
What to tune
genvis logs, per chunk group and for the whole task, how long it waited on each stage:
Genvis task summary (blocking only): Integrator cache load: ... | Beam prep.: ... | Vis. gen.: ... | Vis. write: ... | Final write drain: ...
Tune the stage that dominates:
- Integrator cache load: each baseline's cache is read once, whatever the chunk group
size. Run several beam sets in one invocation (repeat
--voltage-beamsor--power-beams, and--store-location), so each read serves every set, or keep the cache on faster storage. - Beam prep.: the contraction is waiting for power beams. First raise
--max-prepared-power-beams, which costs only memory; if that does not help, raise--power-beam-workerswith fewer--power-beam-nthreadseach, and--voltage-beam-workers. - Vis. gen.: the contraction itself, already on all
--threads; this is the floor of the run. - Vis. write and Final write drain: one group's write stays in flight while the next group runs, so time here means the filesystem is slower than the computation.
genvis direct
The direct commands take --memory-per-task, --threads, --power-beam-workers and the
voltage-beam knobs above, with the same defaults, and --freq-batch-size.
--max-prepared-power-beams does not apply: a baseline's beams are prepared together for
one call that computes them all.
- Channel ranges. Every baseline uses the same split of the band into channel ranges of
one width, so each range of the sky model is read once, with all its Stokes components up
to the largest sky bandlimit among the task's baselines, and serves all the baselines and
beam sets. A range must hold the sky slice, a beam set's voltage-beam grids, one
baseline's power beams, its baseline coefficients and its visibility blocks within
--memory-per-task, after setting aside the transform grids (with--aberrate-baseline, also the grid of the phasor map the coefficients are analysed from, one channel at a time), which do not grow with the range; the heaviest baseline sets that limit. The planner takes the fewest ranges that fit, raised to a multiple of the task count;--plan-onlyprints each component per baseline. - Tasks. Chunk groups are (channel range, baseline), in channel-range order, and each task
takes a contiguous run of them: whole ranges, while the band has a channel per task. Every
range holds every baseline, so the tasks do equal work, each reads only its own channels
of the sky, and with many tasks each holds much less than
--memory-per-task. --freq-batch-size: the channels transformed together in one set of transform calls (default 1). Each channel of a batch holds about three Gauss-Legendre grids of the baseline's sky bandlimit, so memory grows with it.- Threads. The transforms run on all
--threads; the power-beam and voltage-grid workers run as for fixed-sky. The baseline coefficients of each channel are formed on one thread before its transforms, so on a node with many cores several tasks of fewer threads each keep more of them busy than one task; each task holds its own channel range, so the node's memory grows with the task count.
The log reports Sky load where fixed-sky reports the integrator cache load. Tune the stage that dominates:
- Sky load: once per channel range per task, so each task reads its own channels once, and several beam sets in one invocation share each read.
- Beam prep.: the transforms wait for power beams; raise
--power-beam-workersand--voltage-beam-workers. - Vis. gen.: the transforms themselves, already on all
--threads.
gencache
serval gencache runs one contraction at a time, whose OpenMP team is set by
OMP_NUM_THREADS (all cores when unset). Its memory target is --memory-per-task, as for
genvis.
Threads in Python code
Spherical harmonic transforms in SERVAL (ducc0) and its alm layout conversions take an
nthreads argument; at 0 each call uses all cores. A call made from a thread pool forms a
team of that size per calling thread, so a pool of W threads, e.g. one generating beams
with TIRSVoltageBeam.from_cst_farfields, can hold W times the cores. Set the threads
per call for the whole process before starting the pool:
An explicit nthreads argument still takes precedence. SERVAL warns once when an SHT call
from a non-main thread runs at the default.
Avoid OMP_NUM_THREADS=1 for this purpose: it also makes the contraction serial. Never set
OMP_PROC_BIND or OMP_PLACES in a process that also runs ducc0, BLAS or Python thread
pools: libgomp then binds the initial thread, the pools' threads inherit its binding, and
everything runs on a few cores.
For reference, ducc0 sizes its own pool from DUCC0_NUM_THREADS, then OMP_NUM_THREADS,
then the cores; OpenBLAS from OPENBLAS_NUM_THREADS, then OMP_NUM_THREADS.