Inserts a loop index into a command if specified by eg. “{}” inside the command. Then evaluates the command as an f-string to resolve possible (e.g. mathematical) expressions.
command – command string
index – loop index
command with loop index inserted at all places specified by curly brackets and optional format specifications and afterwards as f-string evaluated
Extracts and stores global variables from the parameter object in epipe file
step – epipe object instance
:return : Dictionary holding the global variables
epipe main function. Accepts a epipe yaml configuration file and runs the pipeline.
args – Command line arguments
Adds the dependencies to the base command given in the pipeline Also tries to replace variables indicated by [] brackets.
base_cmd – The command string given in pipeline file
name – The job name
deps – The dependencies of the job
job_dict – The jobs dictionary
parameters – A dictionary holding global variables. Function attempts to replace them in the command string
Complete submission string
Processes one of the epipe items in the pipeline
step – The job item
job_dict – The previous job id dictionary
index – Loop index if within a loop
loop_dependence – The overall dependencies of the loop structure if within a loop
parameters – A dictionary holding global variables. Function attempts to replace them in the command string
assert_ids – whether to check that job ids were successfully obtained
An updated dictionary containing the already submitted jobs and the newly submitted job
Change to current working directory to the local scratch if set.
verbosity – Verbosity level (0 - 4).
Checks which of the indices are missing in the file containing the finished indices
indices – Job indices that should be checked
finished_file – The file from which the jobs will be read
exe – Path to executable
check_indices_file – If True adds indices from index file otherwise only use check_missing function
verbosity – Verbosity level (0 - 4).
Returns the indices that are missing
Transforms decimal hours into the hh:mm format
dec_hours – decimal hours, float or int
string in the format hh:mm
Execution of local MPI job
cmd_string – The command string to run
Wrapper that converts a generator to a function.
generator – A generator
Constructs the dependency string which handles which other jobs this job is dependent on.
function – The type o function to submit
jobids – Dictionary of the jobids for each job already submitted
ext_dependencies – If external dependencies are given they get added to the dependency string (this happens if epipe is used)
system – The type of the queing system of the cluster
verbosity – Verbosity level (0 - 4).
A sting which is used as a substring for the submission string and it handles the dependencies of the job
Parses the jobids from the tasks string.
tasks – The task string, which will get parsed into the job indices
A list of the jobids that should be executed
Parses the jobids from the tasks string. Performs load-balance splitting of the jobs and returns the indices corresponding to rank. This is only used for job array submission.
tasks – The task string, which will get parsed into the job indices
n_jobs – The number of cores that will be requested for the job
rank – The rank of the core
A list of the jobids that should be executed by the core with number rank
Builds the filenames of the stdout and stderr log files for a given job name and a given function to run.
log_dir – directory where the logs are stored
job_name – Name of the job that will write to the log files
function – Function that will be executed
filenames for stdout and stderr logs
Construct the path of the file containing the finished indices
log_dir – directory where log files are stored
job_name – name of the job for which the indices will be store
path of the file for the finished indices
Construct the path of the esub log file
log_dir – directory where log files are stored
job_name – name of the job that will be logged
path of the log file
Builds the command to source a given file if the file exists, otherwise returns an empty string.
source_file – path to the (possibly non-existing) source file, can be relative and can contain “~”
verbosity – Verbosity level (0 - 4).
command to source the file if it exists or empty string
Imports the functions defined in the executable file.
exe – path of the executable
executable imported as python module
Creates the submission string which gets submitted to the queing system
function – The name of the function defined in the executable that will be submitted
source_file – A file which gets executed before running the actual function(s)
n_jobs – The number of jobs that will be requested for the job
n_cores – The number of cores that will be requested for the job
tasks – The task string, which will get parsed into the job indices
mode – The mode in which the job will be ran (MPI-job or as a jobarray)
job_name – The name of the job
function_args – The remaining arguments that will be forwarded to the executable
exe – The path of the executable
main_memory – Memory per core to allocate for the main job
main_time – The Wall time requested for the main job
main_scratch – Scratch per core to allocate for the main job
watchdog_memory – Memory per core to allocate for the watchdog job
watchdog_time – The Wall time requested for the watchdog job
watchdog_scratch – Scratch to allocate for the watchdog job
merge_memory – Memory per core to allocate for the merge job
merge_time – The Wall time requested for the merge job
log_dir – log_dir: The path to the log directory
merge_scratch – Scratch to allocate for the merge job
dependency – The dependency string
system – The type of the queing system of the cluster
main_name – name of the main function
batchsize – If not zero the jobarray gets divided into batches.
max_njobs – Maximum number of jobs allowed to run at the same time.
add_args – Additional cluster-specific arguments
add_bsub – Additional bsub arguments to pass
discard_output – If True writes stdout/stderr to /dev/null
verbosity – Verbosity level (0 - 4).
The submission string that wil get submitted to the cluster
Creates the part of the submission string which handles the allocation of ressources
function – The name of the function defined in the executable that will be submitted
main_memory – Memory per core to allocate for the main job
main_time – The Wall time requested for the main job
main_scratch – Scratch per core to allocate for the main job
watchdog_memory – Memory per core to allocate for the watchdog job
watchdog_time – The Wall time requested for the watchdog job
watchdog_scratch – Scratch to allocate for the watchdog job
merge_memory – Memory per core to allocate for the merge job
merge_time – The Wall time requested for the merge job
merge_scratch – Scratch to allocate for the merge job
system – The type of the queing system of the cluster
verbosity – Verbosity level (0 - 4).
A string that is part of the submission string.
Remove a file or directory if existing
path – path to possible non-existing file or directory
This function runs an MPI job locally
exe – Path to executable
n_cores – Number of cores
function_args – A list of arguments to be passed to the executable
index – Index number to run
logger – logger instance for logging
main_name – Name of main function in executable
verbosity – Verbosity level (0 - 4).
main_name –
Executes an MPI job locally, running each splitted index list on one core.
exe – The executable from where the main function is imported.
n_jobs – The number of cores to allocate.
function_args – The arguments that will get passed to the main function.
tasks – The indices to run on.
function – The function name to run
Write a string to a file, with the file being locked in the meantime.
path – path of file
str_to_write – string to be written
mode – mode in which file is opened
Based on arguments gets the submission string and submits it to the cluster
tasks – The task string, which will get parsed into the job indices
mode – The mode in which the job will be ran (MPI-job or as a jobarray)
exe – The path of the executable
log_dir – The path to the log directory
function_args – The remaining arguments that will be forwarded to the executable
function – The name of the function defined in the executable that will be submitted
source_file – A file which gets executed before running the actual function(s)
n_jobs – The number of jobs that will be requested for the job
n_cores – The number of cores that will be requested for the job
job_name – The name of the job
main_memory – Memory per core to allocate for the main job
main_time – The Wall time requested for the main job
main_scratch – Scratch per core to allocate for the main job
watchdog_memory – Memory per core to allocate for the watchdog job
watchdog_time – The Wall time requested for the watchdog job
watchdog_scratch – Scratch to allocate for the watchdog job
merge_memory – Memory per core to allocate for the merge job
merge_time – The Wall time requested for the merge job
merge_scratch – Scratch to allocate for the merge job
dependency – The jobids of the jobs on which this job depends on
system – The type of the queing system of the cluster
main_name – name of the main function
test – If True no submission but just printing submission string to log
batchsize – If number of cores requested is > batchsize, break up jobarrays into jobarrys of size batchsize
max_njobs – Maximum number of jobs allowed to run at the same time
add_args – Additional cluster-specific arguments
add_bsub – Additional bsub arguments to pass
discard_output – If True writes stdout/stderr to /dev/null
verbosity – Verbosity level (0 - 4).
The jobid of the submitted job