Skip to content

Performance Tuning

genvis fixed-sky: memory and threads

Each serval genvis fixed-sky task takes two resource settings:

  • --memory-per-task: the memory the task plans for (default 4 GB). The planner sizes each chunk group, i.e. how many frequency channels of one baseline's integrator cache are held at once, so that the cache slab, the prepared beams, the voltage-beam grids and the visibility blocks fit in it. A larger value means fewer, larger groups and fewer cache reads. Check what a value gives with --plan-only.
  • --threads: the threads of the task (default: the cores the task may run on, i.e. its CPU affinity, which under SLURM is the task's cpuset). The contraction against the integrator cache uses all of them.

The remaining knobs are derived from --threads and rarely need setting:

knob default
--power-beam-workers threads // 12, at least 1
--max-prepared-power-beams (fixed-sky) 16 per power-beam worker
--power-beam-nthreads (from-voltage-beams) threads // power-beam-workers
--voltage-beam-workers (from-voltage-beams) the number of power-beam workers, at least 1
--max-cached-voltage-beams (from-voltage-beams) all

Power-beam workers derive power beams from voltage beams (or load them, for from-power-beams) while the contraction runs; voltage-beam workers synthesise the voltage beam grids those workers combine. Both kinds of worker run --power-beam-nthreads threads each. The workers, the grid workers and the contraction run at the same time, so a task deliberately holds more threads than cores. A run is refused only when one of these stages alone asks for more threads than the task's cores.

The workers' spherical harmonic transforms run on ducc0's thread pool, which is sized when ducc0 is imported, from DUCC0_NUM_THREADS, else OMP_NUM_THREADS, else the task's cores. A task reports OMP_NUM_THREADS and the pool size when it starts, and warns when the pool is smaller than the threads per power-beam worker: an OMP_NUM_THREADS=1 inherited from a cluster environment otherwise makes beam preparation several times slower.

--max-prepared-power-beams is the queue between the workers and the contraction: too short and workers wait for a free slot, too long and the queued beams take memory from the chunk groups. --max-cached-voltage-beams limits the voltage-beam grids held per chunk group, for arrays whose baselines touch more voltage beams than fit in memory at once.

Note

The derivation rules come from one machine: an AMD EPYC 9654 node of ETH Zurich's Euler cluster (192 cores in 8 NUMA domains). There the defaults give the production settings: 16 power-beam workers of 12 threads, 16 voltage-beam workers, and 16 prepared beams per worker (256).

What to tune

genvis logs, per chunk group and for the whole task, how long it waited on each stage:

Genvis task summary (blocking only): Integrator cache load: ... | Beam prep.: ... | Vis. gen.: ... | Vis. write: ... | Final write drain: ...

Tune the stage that dominates:

  • Integrator cache load: each baseline's cache is read once, whatever the chunk group size. Run several beam sets in one invocation (repeat --voltage-beams or --power-beams, and --store-location), so each read serves every set, or keep the cache on faster storage.
  • Beam prep.: the contraction is waiting for power beams. First raise --max-prepared-power-beams, which costs only memory; if that does not help, raise --power-beam-workers with fewer --power-beam-nthreads each, and --voltage-beam-workers.
  • Vis. gen.: the contraction itself, already on all --threads; this is the floor of the run.
  • Vis. write and Final write drain: one group's write stays in flight while the next group runs, so time here means the filesystem is slower than the computation.

genvis direct

The direct commands take --memory-per-task, --threads, --power-beam-workers and the voltage-beam knobs above, with the same defaults, and --freq-batch-size. --max-prepared-power-beams does not apply: a baseline's beams are prepared together for one call that computes them all.

  • Channel ranges. Every baseline uses the same split of the band into channel ranges of one width, so each range of the sky model is read once, with all its Stokes components up to the largest sky bandlimit among the task's baselines, and serves all the baselines and beam sets. A range must hold the sky slice, a beam set's voltage-beam grids, one baseline's power beams, its baseline coefficients and its visibility blocks within --memory-per-task, after setting aside the transform grids (with --aberrate-baseline, also the grid of the phasor map the coefficients are analysed from, one channel at a time), which do not grow with the range; the heaviest baseline sets that limit. The planner takes the fewest ranges that fit, raised to a multiple of the task count; --plan-only prints each component per baseline.
  • Tasks. Chunk groups are (channel range, baseline), in channel-range order, and each task takes a contiguous run of them: whole ranges, while the band has a channel per task. Every range holds every baseline, so the tasks do equal work, each reads only its own channels of the sky, and with many tasks each holds much less than --memory-per-task.
  • --freq-batch-size: the channels transformed together in one set of transform calls (default 1). Each channel of a batch holds about three Gauss-Legendre grids of the baseline's sky bandlimit, so memory grows with it.
  • Threads. The transforms run on all --threads; the power-beam and voltage-grid workers run as for fixed-sky. The baseline coefficients of each channel are formed on one thread before its transforms, so on a node with many cores several tasks of fewer threads each keep more of them busy than one task; each task holds its own channel range, so the node's memory grows with the task count.

The log reports Sky load where fixed-sky reports the integrator cache load. Tune the stage that dominates:

  • Sky load: once per channel range per task, so each task reads its own channels once, and several beam sets in one invocation share each read.
  • Beam prep.: the transforms wait for power beams; raise --power-beam-workers and --voltage-beam-workers.
  • Vis. gen.: the transforms themselves, already on all --threads.

gencache

serval gencache runs one contraction at a time, whose OpenMP team is set by OMP_NUM_THREADS (all cores when unset). Its memory target is --memory-per-task, as for genvis.

Threads in Python code

Spherical harmonic transforms in SERVAL (ducc0) and its alm layout conversions take an nthreads argument; at 0 each call uses all cores. A call made from a thread pool forms a team of that size per calling thread, so a pool of W threads, e.g. one generating beams with TIRSVoltageBeam.from_cst_farfields, can hold W times the cores. Set the threads per call for the whole process before starting the pool:

import serval

serval.set_num_threads(sht=max(1, serval.available_cores() // W))

An explicit nthreads argument still takes precedence. SERVAL warns once when an SHT call from a non-main thread runs at the default.

Avoid OMP_NUM_THREADS=1 for this purpose: it also makes the contraction serial. Never set OMP_PROC_BIND or OMP_PLACES in a process that also runs ducc0, BLAS or Python thread pools: libgomp then binds the initial thread, the pools' threads inherit its binding, and everything runs on a few cores.

For reference, ducc0 sizes its own pool from DUCC0_NUM_THREADS, then OMP_NUM_THREADS, then the cores; OpenBLAS from OPENBLAS_NUM_THREADS, then OMP_NUM_THREADS.