oasislmf.execution.bash¶

Bash script generation for the pytools loss calculation pipeline.

This module generates bash scripts that orchestrate the pytools loss calculation pipeline. The generated scripts coordinate multiple concurrent processes connected via named pipes (FIFOs) to compute ground-up losses (GUL), insured losses (IL), reinsurance losses (RI), reinsurance gross losses (RL), and optionally fully-correlated (FC) and load-balanced (LB) streams.

The high-level flow of a generated bash script is:

Initialisation – set shell options, logging, error traps, and the completion-check function.

Directory and FIFO setup – create or clean work, output, and FIFO directories; create named pipes for every process/summary combination.

Consumer commands (written in reverse pipeline order so readers are ready before writers start): a. do_ord – per-process ORD output consumers (eltpy, pltpy). b. do_tees – tee commands that fan summary streams out to

work folders and ORD FIFOs.

do_summarycalcs – summarypy commands that read per-runtype FIFOs and write per-summary FIFOs.

Main GUL pipeline commands – evepy | modelpy | gulpy (or gulmc) piped through fmpy for IL/RI, writing into the FIFOs consumed above.

Waits – wait on all background PIDs from steps 3-4.

Kat concatenation – katpy merges per-process binary ORD files into single output files.

Post-wait aggregation – aalpy and lecpy run over the collected work-folder data to produce ALT and EPT/PSEPT outputs.

Cleanup – remove working directories and temporary FIFOs.

Completion check – verify every started process logged a successful finish.

Run types:

gul – ground-up loss

il – insured (direct) loss

ri – reinsurance net loss

rl – reinsurance gross loss

fc – fully correlated ground-up loss

lb – load-balanced stream (intermediary between GUL and FM)

Entry points:

genbash() – legacy single-call interface that writes a complete bash script.

bash_params() – builds a parameter dict consumed by create_bash_analysis() and create_bash_outputs().

Functions¶

`get_check_function`([custom_gulcalc_log_start, ...])	Creates a bash function to check the logs to ensure same number of process started and finsished.
`process_range`(max_process_id[, process_number])	Creates an iterable for all the process ids, if process number is set
`get_modelcmd`(→ str)	Gets the construct model command line argument for the bash script.
`get_gulcmd`(gulmc, gul_random_generator, ...[, ...])	Get the ground-up loss calculation command.
`get_fmcmd`([fmpy_low_memory, fmpy_sort_output])	Build the fmpy (financial module) command string.
`print_command`(command_file, cmd)	Writes the supplied command to the end of the generated script
`ord_enabled`(summary_options, ORD_SWITCHES)	Checks if ORD leccalc is enabled in a summaries section
`do_post_wait_processing`(runtype, analysis_settings, ...)	Write post-wait aggregation commands (aalpy, lecpy) to the bash script.
`get_fifo_name`(fifo_dir, producer, producer_id[, consumer])	Build the standardised path for a named pipe (FIFO).
`do_fifo_exec`(producer, producer_id, filename, fifo_dir)	Write a single FIFO create or remove command to the bash script.
`do_fifos_exec`(runtype, max_process_id, filename, fifo_dir)	Write FIFO create/remove commands for every process in the range.
`do_fifos_exec_full_correlation`(runtype, ...[, ...])	Write FIFO create/remove commands for full-correlation sumcalc and fmcalc pipes.
`do_fifos_calc`(runtype, analysis_settings, ...[, ...])	Write FIFO create/remove commands for all summary and ORD output pipes.
`create_workfolders`(runtype, analysis_settings, filename)	Write `mkdir -p` commands for work sub-directories needed by summary consumers.
`do_kats`(runtype, analysis_settings, max_process_id, ...)	Write `katpy` concatenation commands to merge per-process ORD files.
`do_summarycalcs`(runtype, analysis_settings, ...[, ...])	Write a `summarypy` command for one process to the bash script.
`do_tees`(runtype, analysis_settings, process_id, ...[, ...])	Write `tee` commands that fan each summary stream to ORD and work FIFOs.
`do_tees_fc_sumcalc_fmcalc`(process_id, filename, ...)	Write a `tee` command that splits the full-correlation GUL output.
`get_correlated_output_stems`(fifo_dir)	Return a dict of FIFO path stems for the full-correlation pipeline.
`do_ord`(runtype, analysis_settings, process_id, ...[, ...])	Write ORD output consumer commands (`eltpy`, `pltpy`) for one process.
`get_ri_inuring_priorities`(analysis_settings, ...)	Build the list of inuring priority dicts for reinsurance net loss.
`get_rl_inuring_priorities`(num_reinsurance_iterations)	Build the list of inuring priority dicts for reinsurance gross loss.
`rl`(analysis_settings, max_process_id, filename, ...[, ...])	Write all reinsurance gross loss (RL) consumer commands.
`ri`(analysis_settings, max_process_id, filename, ...[, ...])	Write all reinsurance net loss (RI) consumer commands.
`il`(analysis_settings, max_process_id, filename, ...[, ...])	Write all insured loss (IL) consumer commands.
`do_gul`(analysis_settings, max_process_id, filename, ...)	Write all ground-up loss (GUL) consumer commands.
`do_gul_full_correlation`(analysis_settings, ...[, ...])	Write GUL consumer commands for the full-correlation path.
`do_waits`(wait_variable, wait_count, filename)	Add waits to the script
`do_pwaits`(filename, process_counter)	Add pwaits to the script
`do_awaits`(filename, process_counter)	Add awaits to the script
`do_lwaits`(filename, process_counter)	Add lwaits to the script
`do_kwaits`(filename, process_counter)	Add kwaits to the script
`do_jwaits`(filename, process_counter)	Add jwaits to the script (join-summary-info processes)
`get_getmodel_cmd`(number_of_samples, gul_threshold, ...)	Gets the GUL pipeline command (gulpy/gulmc) for a single process
`add_pid_to_shell_command`(cmd, process_counter)	Add a variable to the end of a command in order to track the ID of the process executing it.
`get_main_cmd_ri_stream`(cmd, process_id, il_output, ...)	Gets the fmpy command for the reinsurance stream
`get_main_cmd_il_stream`(cmd, process_id, il_alloc_rule)	Gets the fmpy command for the insured losses stream
`get_main_cmd_gul_stream`(cmd, process_id[, fifo_dir, ...])	Gets the command to output ground up losses
`get_complex_model_cmd`(custom_gulcalc_cmd, ...)	Return a custom GUL command function for complex (third-party) models.
`do_computes`(outputs)	Execute a list of deferred compute operations.
`get_main_cmd_lb`(num_lb, num_in_per_lb, num_out_per_lb, ...)	Yield `load_balancer` commands that redistribute streams across FIFOs.
`get_pla_cmd`(pla, secondary_factor, uniform_factor)	Determine whether Post Loss Amplification should be implemented and issue
`bash_params`(analysis_settings[, max_process_id, ...])	Build the parameter dict consumed by `create_bash_analysis()` and `create_bash_outputs()`.
`bash_wrapper`(filename, bash_trace, stderr_guard[, ...])	Context manager that wraps the script body with header and footer boilerplate.
`create_bash_analysis`(process_counter, max_process_id, ...)	Write the main analysis section of the bash script.
`create_bash_outputs`(process_counter, fifo_tmp_dir, ...)	Write the output-aggregation and cleanup section of the bash script.
`genbash`(max_process_id, analysis_settings[, ...])	Generates a bash script containing pytools calculation instructions for an
`add_server_call`(call[, analysis_pk, socket_server])	Inject WebSocket or socket-server flags into a GUL command string.

Module Contents¶

oasislmf.execution.bash.logger[source]¶

oasislmf.execution.bash.RUNTYPE_GROUNDUP_LOSS = 'gul'[source]¶

oasislmf.execution.bash.RUNTYPE_LOAD_BALANCED_LOSS = 'lb'[source]¶

oasislmf.execution.bash.RUNTYPE_INSURED_LOSS = 'il'[source]¶

oasislmf.execution.bash.RUNTYPE_REINSURANCE_LOSS = 'ri'[source]¶

oasislmf.execution.bash.RUNTYPE_REINSURANCE_GROSS_LOSS = 'rl'[source]¶

oasislmf.execution.bash.RUNTYPE_FULL_CORRELATION = 'fc'[source]¶

oasislmf.execution.bash.REINSURANCE_RUNTYPES[source]¶

oasislmf.execution.bash.INTERMEDIATE_INURING_PRIORITY_PREFIX = 'IP'[source]¶

oasislmf.execution.bash.WAIT_PROCESSING_SWITCHES[source]¶

oasislmf.execution.bash.ORD_EPT_OUTPUT_SWITCHES[source]¶

oasislmf.execution.bash.ORD_PSEPT_OUTPUT_SWITCHES[source]¶

oasislmf.execution.bash.ORD_LECCALC[source]¶

oasislmf.execution.bash.ORD_ALT_OUTPUT_SWITCHES[source]¶

oasislmf.execution.bash.ORD_ALT_MEANONLY_OUTPUT_SWITCHES[source]¶

oasislmf.execution.bash.ORD_PLT_OUTPUT_SWITCHES[source]¶

oasislmf.execution.bash.ORD_ELT_OUTPUT_SWITCHES[source]¶

oasislmf.execution.bash.ORD_SELT_OUTPUT_SWITCH[source]¶

oasislmf.execution.bash.OUTPUT_SWITCHES[source]¶

oasislmf.execution.bash.EVE_SHUFFLE_OPTIONS[source]¶

oasislmf.execution.bash.TRAP_FUNC = Multiline-String[source]¶

Show Value

"""
touch $LOG_DIR/stderror.err
oasis_exec_monitor.sh $$ $LOG_DIR & pid0=$!

exit_handler(){
   exit_code=$?

   # disable handler
   trap - QUIT HUP INT KILL TERM ERR EXIT

   kill -9 $pid0 2> /dev/null
   if [ "$exit_code" -gt 0 ]; then
       # Error - run process clean up
       echo 'Kernel execution error - exitcode='$exit_code

       set +x
       group_pid=$(ps -p $$ -o pgid --no-headers)
       sess_pid=$(ps -p $$ -o sess --no-headers)
       script_pid=$$
       printf "Script PID:%d, GPID:%s, SPID:%d
" $script_pid $group_pid $sess_pid >> $LOG_DIR/killout.txt

       ps -jf f -g $sess_pid > $LOG_DIR/subprocess_list
       PIDS_KILL=$(pgrep -a --pgroup $group_pid | awk 'BEGIN { FS = "[ \t\n]+" }{ if ($1 >= '$script_pid') print}' | grep -v celery | egrep -v *\\.log$  | egrep -v *startup.sh$ | sort -n -r)
       echo "$PIDS_KILL" >> $LOG_DIR/killout.txt
       kill -9 $(echo "$PIDS_KILL" | awk 'BEGIN { FS = "[ \t\n]+" }{ print $1 }') 2>/dev/null
       exit $exit_code
   else
       # script successful
       exit 0
   fi
}
trap exit_handler QUIT HUP INT KILL TERM ERR EXIT"""

oasislmf.execution.bash.get_check_function(custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None)[source]¶

Creates a bash function to check the logs to ensure same number of process started and finsished.

Args:: custom_gulcalc_log_start (str): Custom message printed to the logs when a process starts. custom_gulcalc_log_finish (str): Custom message printed to the logs when a process ends.

oasislmf.execution.bash.BASH_TRACE = Multiline-String[source]¶

Show Value

"""
# --- Redirect Bash trace to file ---
bash_logging_supported(){
    local BASH_VER_MAJOR=${BASH_VERSION:0:1}
    local BASH_VER_MINOR=${BASH_VERSION:2:1}

    if [[ "$BASH_VER_MAJOR" -gt 4 ]]; then
        echo 1; exit
    fi
    if [[ $BASH_VER_MAJOR -eq 4 ]] && [[ $BASH_VER_MINOR -gt 3 ]]; then
        echo 1; exit
    fi
    echo 0
}
if [ $(bash_logging_supported) == 1 ]; then
    exec   > >(tee -ia $LOG_DIR/bash.log)
    exec  2> >(tee -ia $LOG_DIR/bash.log >& 2)
    exec 19> $LOG_DIR/bash.log
    export BASH_XTRACEFD="19"
    set -x
else
    echo "WARNING: logging disabled, bash version '$BASH_VERSION' is not supported, minimum requirement is bash v4.4"
fi """

oasislmf.execution.bash.process_range(max_process_id, process_number=None)[source]¶

Creates an iterable for all the process ids, if process number is set then an iterable containing only that number is returned.

This allows for the loss generation to be ran in different processes rather than accross multiple cores.

Parameters:

max_process_id – The largest process number
process_number – If set iterable only containing this number is returned

Returns:

iterable containing all the process numbers to process

oasislmf.execution.bash.get_modelcmd(server=False, peril_filter=[]) → str[source]¶

Gets the construct model command line argument for the bash script.

Args:: server: (bool) if set then enable ‘TCP’ ipc server/client mode peril_filter: (list) list of perils to include (all included if empty)

oasislmf.execution.bash.get_gulcmd(gulmc, gul_random_generator, gulmc_effective_damageability, gulmc_vuln_cache_size, modelpy_server, peril_filter, model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False)[source]¶

Get the ground-up loss calculation command.

Args:: gulmc (bool): if True, return the combined (model+ground up) command name, else use ‘modelpy | gulpy’ .
Returns:: str: the ground-up loss calculation command

oasislmf.execution.bash.get_fmcmd(fmpy_low_memory=False, fmpy_sort_output=False)[source]¶

Build the fmpy (financial module) command string.

Args:

fmpy_low_memory (bool): If True, append the -l flag to enable: low-memory mode.
fmpy_sort_output (bool): If True, append the --sort-output flag: so that output records are sorted.

Returns:

str: The assembled fmpy command with any requested flags.

oasislmf.execution.bash.print_command(command_file, cmd)[source]¶

Writes the supplied command to the end of the generated script

Parameters:

command_file – File to append command to.
cmd – The command to append

oasislmf.execution.bash.ord_enabled(summary_options, ORD_SWITCHES)[source]¶

Checks if ORD leccalc is enabled in a summaries section

Parameters:

summary_options (dict) – Summaies section from an analysis_settings file
ORD_SWITCHES (dict) – Options from the analysis_settings ‘Summaies’ section to search

Example: {

“id”: 1, “ord_output”: {

“ept_full_uncertainty_aep”: true, “ept_full_uncertainty_oep”: true, “ept_mean_sample_aep”: true, “ept_mean_sample_oep”: true, “ept_per_sample_mean_aep”: true, “ept_per_sample_mean_oep”: true, “psept_aep”: true, “psept_oep”: true, “return_period_file”: true

}

}

Returns:: True is leccalc is enables, False otherwise.

oasislmf.execution.bash.do_post_wait_processing(runtype, analysis_settings, filename, process_counter, work_sub_dir='', output_dir='output/', stderr_guard=True, inuring_priority=None, join_summary_info=False)[source]¶

Write post-wait aggregation commands (aalpy, lecpy) to the bash script.

These commands run after all per-process wait calls have completed, operating on the binary work-folder data produced by tee during the main pipeline. Specifically this handles:

PALT (aalpy) – Period Average Loss Table output.
ALT mean-only (aalpy) – Average Loss Table (mean only) output.
EPT / PSEPT (lecpy) – Exceedance Probability Table and Per-Sample Exceedance Probability Table outputs.

Args:

runtype (str): The run type identifier (e.g. 'gul', 'il',: 'ri', 'rl').

analysis_settings (dict): The full analysis settings dictionary. filename (str): Path to the bash script being generated. process_counter (dict): Mutable counter dict tracking background PIDs

(keys lpid_monitor_count, etc.).

work_sub_dir (str): Relative path prefix for work sub-directories. output_dir (str): Directory where final output files are written. stderr_guard (bool): If True, wrap commands in a sub-shell that

redirects stderr to the log.

inuring_priority (str or None): Inuring priority label to embed in: file names, or None / empty for the final priority.
join_summary_info (bool): If True, append join-summary-info: commands to enrich outputs with summary metadata.

oasislmf.execution.bash.get_fifo_name(fifo_dir, producer, producer_id, consumer='')[source]¶

Build the standardised path for a named pipe (FIFO).

The naming convention is <fifo_dir><producer>_<consumer>_P<id> when a consumer is specified, or <fifo_dir><producer>_P<id> otherwise.

Args:: fifo_dir (str): Base directory for FIFOs (e.g. 'fifo/'). producer (str): Run type that produces the stream (e.g. 'gul'). producer_id (int): Process number used as a suffix (P<id>). consumer (str): Optional consumer identifier appended between the

producer and the process id.
Returns:: str: The fully-qualified FIFO path.

oasislmf.execution.bash.do_fifo_exec(producer, producer_id, filename, fifo_dir, action='mkfifo', consumer='')[source]¶

Write a single FIFO create or remove command to the bash script.

Args:

producer (str): Run type that produces the stream. producer_id (int): Process number for the FIFO name. filename (str): Path to the bash script being generated. fifo_dir (str): Base directory for FIFOs. action (str): Shell command to execute ('mkfifo' to create,

'rm' to remove).

consumer (str): Optional consumer identifier for the FIFO name.

oasislmf.execution.bash.do_fifos_exec(runtype, max_process_id, filename, fifo_dir, process_number=None, action='mkfifo', consumer='')[source]¶

Write FIFO create/remove commands for every process in the range.

Iterates over all process IDs and emits one mkfifo (or rm) command per process for the given run type.

Args:

runtype (str): The run type identifier (e.g. 'gul', 'il'). max_process_id (int): Upper bound of process IDs. filename (str): Path to the bash script being generated. fifo_dir (str): Base directory for FIFOs. process_number (int or None): If set, only that single process ID

is used instead of the full range.

action (str): Shell command ('mkfifo' or 'rm'). consumer (str): Optional consumer identifier for FIFO names.

oasislmf.execution.bash.do_fifos_exec_full_correlation(runtype, max_process_id, filename, fifo_dir, process_number=None, action='mkfifo')[source]¶

Write FIFO create/remove commands for full-correlation sumcalc and fmcalc pipes.

For each process ID two FIFOs are created: one for the summarycalc input (<runtype>_sumcalc_P<id>) and one for the fmcalc input (<runtype>_fmcalc_P<id>).

Args:: runtype (str): The run type identifier (typically 'gul'). max_process_id (int): Upper bound of process IDs. filename (str): Path to the bash script being generated. fifo_dir (str): Base directory for FIFOs. process_number (int or None): If set, restrict to a single process. action (str): Shell command ('mkfifo' or 'rm').

oasislmf.execution.bash.do_fifos_calc(runtype, analysis_settings, max_process_id, filename, fifo_dir='fifo/', process_number=None, consumer_prefix=None, action='mkfifo')[source]¶

Write FIFO create/remove commands for all summary and ORD output pipes.

For every (process, summary) pair this creates:

A summary FIFO (and its .idx companion when leccalc or ALT output is enabled).
One FIFO per active ORD output type (plt, elt, selt).

Args:

runtype (str): The run type identifier. analysis_settings (dict): The full analysis settings dictionary. max_process_id (int): Upper bound of process IDs. filename (str): Path to the bash script being generated. fifo_dir (str): Base directory for FIFOs. process_number (int or None): If set, restrict to a single process. consumer_prefix (str or None): Optional prefix prepended to consumer

names (used for inuring priority labelling).

action (str): Shell command ('mkfifo' or 'rm').

oasislmf.execution.bash.create_workfolders(runtype, analysis_settings, filename, work_dir='work/', inuring_priority=None)[source]¶

Write mkdir -p commands for work sub-directories needed by summary consumers.

Creates directories that tee will write binary summary data into for subsequent aggregation by lecpy (leccalc) and aalpy (aalcalc).

Args:: runtype (str): The run type identifier (e.g. 'gul', 'il'). analysis_settings (dict): The full analysis settings dictionary. filename (str): Path to the bash script being generated. work_dir (str): Root working directory (e.g. 'work/'). inuring_priority (str or None): Inuring priority label to embed in

directory names, or None / empty for the final priority.

oasislmf.execution.bash.do_kats(runtype, analysis_settings, max_process_id, filename, process_counter, work_dir='work/kat/', output_dir='output/', sort_by_event=False, process_number=None, inuring_priority=None, join_summary_info=False)[source]¶

Write katpy concatenation commands to merge per-process ORD files.

After the main pipeline wait has completed, per-process binary output files sit in the kat work directory. katpy reads them all and produces a single sorted (or unsorted) output file per summary/ORD-table combination.

Args:

runtype (str): The run type identifier. analysis_settings (dict): The full analysis settings dictionary. max_process_id (int): Upper bound of process IDs. filename (str): Path to the bash script being generated. process_counter (dict): Mutable counter dict tracking background PIDs

(key kpid_monitor_count).

work_dir (str): Kat work directory containing per-process files. output_dir (str): Directory for final concatenated output files. sort_by_event (bool): If True, katpy sorts output by event id. process_number (int or None): If set, restrict to a single process. inuring_priority (str or None): Inuring priority label for file names. join_summary_info (bool): If True, append join-summary-info

commands after each concatenation.

Returns:

bool: True if any katpy commands were emitted, False otherwise.

oasislmf.execution.bash.do_summarycalcs(runtype, analysis_settings, process_id, filename, fifo_dir='fifo/', stderr_guard=True, num_reinsurance_iterations=0, gul_full_correlation=False, inuring_priority=None)[source]¶

Write a summarypy command for one process to the bash script.

Reads the per-process run-type FIFO and fans the stream out to one summary FIFO per summary set defined in the analysis settings.

Args:

runtype (str): The run type identifier (e.g. 'gul', 'il'). analysis_settings (dict): The full analysis settings dictionary. process_id (int): The process number for this command. filename (str): Path to the bash script being generated. fifo_dir (str): Base directory for FIFOs. stderr_guard (bool): If True, wrap the command in a sub-shell that

redirects stderr to the log.

num_reinsurance_iterations (int): Total number of reinsurance: iterations (used for directory switching).
gul_full_correlation (bool): If True, read from the: _sumcalc FIFO variant used by the full-correlation path.
inuring_priority (dict or None): Inuring priority dict with keys: 'text' and 'level', or None for non-reinsurance runs.

oasislmf.execution.bash.do_tees(runtype, analysis_settings, process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', inuring_priority=None)[source]¶

Write tee commands that fan each summary stream to ORD and work FIFOs.

For each summary set, a tee reads from the summary FIFO and duplicates the stream to:

ORD output FIFOs (for eltpy/pltpy consumers).
Work-folder binary files (for aalpy/lecpy post-wait processing).

When leccalc or ALT output is enabled, a companion .idx tee is also emitted.

Args:

(key pid_monitor_count).

fifo_dir (str): Base directory for FIFOs. work_dir (str): Root working directory. inuring_priority (str or None): Inuring priority label for file and

FIFO names, or None / empty for the final priority.

oasislmf.execution.bash.do_tees_fc_sumcalc_fmcalc(process_id, filename, correlated_output_stems)[source]¶

Write a tee command that splits the full-correlation GUL output.

Duplicates the correlated GUL stream into two FIFOs: one for the summarycalc path and one for the fmcalc (financial module) path.

Args:: process_id (int): The process number. filename (str): Path to the bash script being generated. correlated_output_stems (dict): FIFO path stems returned by

get_correlated_output_stems().

oasislmf.execution.bash.get_correlated_output_stems(fifo_dir)[source]¶

Return a dict of FIFO path stems for the full-correlation pipeline.

The returned dict contains three keys:

'gulcalc_output' – stem for the GUL output FIFO.
'fmcalc_input' – stem for the fmcalc input FIFO.
'sumcalc_input' – stem for the summarycalc input FIFO.

Each value is a string ending with _P so that a process ID can be appended directly.

Args:: fifo_dir (str): Base directory for FIFOs (e.g. 'fifo/').
Returns:: dict: Mapping of stem names to FIFO path prefixes.

oasislmf.execution.bash.do_ord(runtype, analysis_settings, process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, inuring_priority=None)[source]¶

Write ORD output consumer commands (eltpy, pltpy) for one process.

For each summary set and each active ORD output type, reads from the corresponding FIFO and writes per-process binary output files into the kat work directory for later concatenation by katpy.

Args:

runtype (str): The run type identifier. analysis_settings (dict): The full analysis settings dictionary. process_id (int): The process number. filename (str): Path to the bash script being generated. process_counter (dict): Mutable counter dict tracking background PIDs. fifo_dir (str): Base directory for FIFOs. work_dir (str): Root working directory (kat sub-directory is used). stderr_guard (bool): If True, wrap commands in a sub-shell that

redirects stderr to the log.

inuring_priority (str or None): Inuring priority label for file and: FIFO names.

oasislmf.execution.bash.get_ri_inuring_priorities(analysis_settings, num_reinsurance_iterations)[source]¶

Build the list of inuring priority dicts for reinsurance net loss.

Returns intermediate inuring priorities specified in the analysis settings plus the final priority (the full reinsurance iteration count).

Args:

analysis_settings (dict): The full analysis settings dictionary,: checked for the 'ri_inuring_priorities' key.
num_reinsurance_iterations (int): Total number of reinsurance: iterations.

Returns:

list[dict]: Each dict has 'text' (file-name prefix) and 'level' (iteration number) keys.

oasislmf.execution.bash.get_rl_inuring_priorities(num_reinsurance_iterations)[source]¶

Build the list of inuring priority dicts for reinsurance gross loss.

Unlike get_ri_inuring_priorities(), every iteration from 1 to num_reinsurance_iterations is included (there is no “final” entry).

Args:

num_reinsurance_iterations (int): Total number of reinsurance: iterations.

Returns:

list[dict]: Each dict has 'text' (file-name prefix) and 'level' (iteration number) keys.

oasislmf.execution.bash.rl(analysis_settings, max_process_id, filename, process_counter, num_reinsurance_iterations, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None)[source]¶

Write all reinsurance gross loss (RL) consumer commands.

Iterates over each RL inuring priority and emits do_ord, do_tees, and do_summarycalcs commands for every process.

Args:

analysis_settings (dict): The full analysis settings dictionary. max_process_id (int): Upper bound of process IDs. filename (str): Path to the bash script being generated. process_counter (dict): Mutable counter dict tracking background PIDs. num_reinsurance_iterations (int): Total number of reinsurance

iterations.

fifo_dir (str): Base directory for FIFOs. work_dir (str): Root working directory. stderr_guard (bool): If True, wrap commands in a sub-shell that

redirects stderr to the log.

process_number (int or None): If set, restrict to a single process.

oasislmf.execution.bash.ri(analysis_settings, max_process_id, filename, process_counter, num_reinsurance_iterations, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None)[source]¶

Write all reinsurance net loss (RI) consumer commands.

Iterates over each RI inuring priority and emits do_ord, do_tees, and do_summarycalcs commands for every process.

Args:

iterations.

fifo_dir (str): Base directory for FIFOs. work_dir (str): Root working directory. stderr_guard (bool): If True, wrap commands in a sub-shell that

redirects stderr to the log.

process_number (int or None): If set, restrict to a single process.

oasislmf.execution.bash.il(analysis_settings, max_process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None)[source]¶

Write all insured loss (IL) consumer commands.

Emits do_ord, do_tees, and do_summarycalcs commands for every process in the range.

Args:

analysis_settings (dict): The full analysis settings dictionary. max_process_id (int): Upper bound of process IDs. filename (str): Path to the bash script being generated. process_counter (dict): Mutable counter dict tracking background PIDs. fifo_dir (str): Base directory for FIFOs. work_dir (str): Root working directory. stderr_guard (bool): If True, wrap commands in a sub-shell that

redirects stderr to the log.

process_number (int or None): If set, restrict to a single process.

oasislmf.execution.bash.do_gul(analysis_settings, max_process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None)[source]¶

Write all ground-up loss (GUL) consumer commands.

Emits do_ord, do_tees, and do_summarycalcs commands for every process in the range.

Args:

analysis_settings (dict): The full analysis settings dictionary. max_process_id (int): Upper bound of process IDs. filename (str): Path to the bash script being generated. process_counter (dict): Mutable counter dict tracking background PIDs. fifo_dir (str): Base directory for FIFOs. work_dir (str): Root working directory. stderr_guard (bool): If True, wrap commands in a sub-shell that

redirects stderr to the log.

process_number (int or None): If set, restrict to a single process.

oasislmf.execution.bash.do_gul_full_correlation(analysis_settings, max_process_id, filename, process_counter, fifo_dir='fifo/full_correlation/', work_dir='work/full_correlation/', stderr_guard=None, process_number=None)[source]¶

Write GUL consumer commands for the full-correlation path.

Similar to do_gul() but operates on the full_correlation/ sub-directories and only emits do_tees (no do_ord or do_summarycalcs).

Args:: analysis_settings (dict): The full analysis settings dictionary. max_process_id (int): Upper bound of process IDs. filename (str): Path to the bash script being generated. process_counter (dict): Mutable counter dict tracking background PIDs. fifo_dir (str): FIFO directory for full-correlation pipes. work_dir (str): Work directory for full-correlation outputs. stderr_guard: Unused (kept for interface consistency). process_number (int or None): If set, restrict to a single process.

oasislmf.execution.bash.do_waits(wait_variable, wait_count, filename)[source]¶

Add waits to the script

Parameters:

wait_variable (str) – The type of wait
wait_count (int) – The number of processes to wait for
filename (str) – Script to add waits to

oasislmf.execution.bash.do_pwaits(filename, process_counter)[source]¶: Add pwaits to the script

oasislmf.execution.bash.do_awaits(filename, process_counter)[source]¶: Add awaits to the script

oasislmf.execution.bash.do_lwaits(filename, process_counter)[source]¶: Add lwaits to the script

oasislmf.execution.bash.do_kwaits(filename, process_counter)[source]¶: Add kwaits to the script

oasislmf.execution.bash.do_jwaits(filename, process_counter)[source]¶: Add jwaits to the script (join-summary-info processes)

oasislmf.execution.bash.get_getmodel_cmd(number_of_samples, gul_threshold, use_random_number_file, gul_alloc_rule, item_output, process_id, max_process_id, correlated_output, eve_shuffle_flag, modelpy_server=False, peril_filter=[], gulmc=False, gul_random_generator=1, gulmc_effective_damageability=False, gulmc_vuln_cache_size=200, model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False, **kwargs)[source]¶: Gets the GUL pipeline command (gulpy/gulmc) for a single process :param number_of_samples: The number of samples to run :type number_of_samples: int :param gul_threshold: The GUL threshold to use :type gul_threshold: float :param use_random_number_file: flag to use the random number file :type use_random_number_file: bool :param gul_alloc_rule: back allocation rule for gulcalc :type gul_alloc_rule: int :param item_output: The item output :type item_output: str :param eve_shuffle_flag: The event shuffling rule :type eve_shuffle_flag: str :param model_df_engine: The engine to use when loading dataframes :type model_df_engine: str :return: The generated getmodel command

oasislmf.execution.bash.add_pid_to_shell_command(cmd, process_counter)[source]¶

Add a variable to the end of a command in order to track the ID of the process executing it. Each time this function is called, the counter process_counter is incremented.

Args:: cmd (str): the command whose process ID is to be stored in a variable. process_counter (Counter or dict): the number of process IDs that are being tracked.
Returns:: cmd (str): the updated command string.

oasislmf.execution.bash.get_main_cmd_ri_stream(cmd, process_id, il_output, il_alloc_rule, ri_alloc_rule, num_reinsurance_iterations, fifo_dir='fifo/', stderr_guard=True, from_file=False, fmpy_low_memory=False, fmpy_sort_output=False, step_flag='', process_counter=None, ri_inuring_priorities=None, rl_inuring_priorities=None)[source]¶: Gets the fmpy command for the reinsurance stream :param cmd: either gulcalc command stream or correlated output file :type cmd: str :param process_id: ID corresponding to thread :type process_id: int :param il_output: If insured loss outputs required :type il_output: Boolean :param il_alloc_rule: insured loss allocation rule for fmcalc :type il_alloc_rule: int :param ri_alloc_rule: reinsurance allocation rule for fmcalc :type ri_alloc_rule: int :param num_reinsurance_iterations: number of reinsurance iterations :type num_reinsurance_iterations: int :param fifo_dir: path to fifo directory :type fifo_dir: str :param stderr_guard: send stderr output to log file :type stderr_guard: bool :param from_file: must be true if cmd is a file and false if it can be piped :type from_file: bool :param ri_inuring_priorities: Inuring priorities where net output has been requested :type ri_inuring_priorities: dict :param rl_inuring_priorities: Inuring priorities where gross output has been requested :type rl_inuring_priorities: dict

oasislmf.execution.bash.get_main_cmd_il_stream(cmd, process_id, il_alloc_rule, fifo_dir='fifo/', stderr_guard=True, from_file=False, fmpy_low_memory=False, fmpy_sort_output=False, step_flag='', process_counter=None)[source]¶: Gets the fmpy command for the insured losses stream :param cmd: either gulcalc command stream or correlated output file :type cmd: str :param process_id: ID corresponding to thread :type process_id: int :param il_alloc_rule: insured loss allocation rule for fmcalc :type il_alloc_rule: int :param fifo_dir: path to fifo directory :type fifo_dir: str :param stderr_guard: send stderr output to log file :type stderr_guard: bool :param from_file: must be true if cmd is a file and false if it can be piped :type from_file: bool :return: generated fmcalc command as str

oasislmf.execution.bash.get_main_cmd_gul_stream(cmd, process_id, fifo_dir='fifo/', stderr_guard=True, consumer='', process_counter=None)[source]¶: Gets the command to output ground up losses :param cmd: either gulcalc command stream or correlated output file :type cmd: str :param process_id: ID corresponding to thread :type process_id: int :param fifo_dir: path to fifo directory :type fifo_dir: str :param stderr_guard: send stderr output to log file :type stderr_guard: bool :param consumer: optional name of the consumer of the stream :type consumer: string :return: generated command as str

oasislmf.execution.bash.get_complex_model_cmd(custom_gulcalc_cmd, analysis_settings)[source]¶

Return a custom GUL command function for complex (third-party) models.

If custom_gulcalc_cmd is explicitly provided it must exist in PATH; otherwise the function infers a command name from <supplier>_<model>_gulcalc and checks for its presence.

Args:

custom_gulcalc_cmd (str or None): Explicit custom GUL binary name,: or None to attempt auto-detection.
analysis_settings (dict): The full analysis settings dictionary: (used to infer the binary name via model_supplier_id and model_name_id).

Returns:

callable or None: A function with the same signature as get_getmodel_cmd() that builds the custom GUL command string, or None if no custom binary was found.

Raises:

OasisException: If custom_gulcalc_cmd is explicitly set but: cannot be found on PATH.

oasislmf.execution.bash.do_computes(outputs)[source]¶

Execute a list of deferred compute operations.

Each entry in outputs is a dict with keys 'compute_fun', 'compute_args', and 'loss_type'. The function calls compute_fun(**compute_args) for every entry, preceded by a comment header identifying the loss type.

Args:

outputs (list[dict]): Deferred compute descriptors built up by the: caller (e.g. create_bash_analysis()).

oasislmf.execution.bash.get_main_cmd_lb(num_lb, num_in_per_lb, num_out_per_lb, get_input_stream_name, get_output_stream_name, stderr_guard)[source]¶

Yield load_balancer commands that redistribute streams across FIFOs.

Each load balancer reads from num_in_per_lb input FIFOs and writes to num_out_per_lb output FIFOs. Process IDs are assigned sequentially across all load balancers.

Args:

num_lb (int): Number of load balancer processes. num_in_per_lb (int): Number of input FIFOs per load balancer. num_out_per_lb (int): Number of output FIFOs per load balancer. get_input_stream_name (callable): Function accepting producer_id

and returning the input FIFO path.

get_output_stream_name (callable): Function accepting producer_id: and returning the output FIFO path.
stderr_guard (bool): If True, wrap commands in a sub-shell that: redirects stderr to the log.

Yields:

str: A shell command string for each load balancer process.

oasislmf.execution.bash.get_pla_cmd(pla, secondary_factor, uniform_factor)[source]¶

Determine whether Post Loss Amplification should be implemented and issue plapy command.

Args:

pla (bool): flag to apply post loss amplification secondary_factor (float): secondary factor to apply to post loss

amplification

uniform_factor (float): uniform factor to apply across all losses

Returns:

pla_cmd (str): post loss amplification command

oasislmf.execution.bash.bash_params(analysis_settings, max_process_id=-1, number_of_processes=-1, num_reinsurance_iterations=0, model_storage_json=None, fifo_tmp_dir=True, gul_alloc_rule=None, il_alloc_rule=None, ri_alloc_rule=None, num_gul_per_lb=None, num_fm_per_lb=None, stderr_guard=True, bash_trace=False, filename='run_kools.sh', _get_getmodel_cmd=None, custom_gulcalc_cmd=None, custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None, custom_args={}, fmpy_low_memory=False, fmpy_sort_output=False, event_shuffle=None, gulmc=True, gul_random_generator=1, gulmc_effective_damageability=False, gulmc_vuln_cache_size=200, process_number=None, remove_working_files=True, model_run_dir='', model_py_server=False, join_summary_info=False, peril_filter=[], exposure_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False, **kwargs)[source]¶

Build the parameter dict consumed by create_bash_analysis() and create_bash_outputs().

Validates analysis settings (checking output/summary coherence, reinsurance iteration requirements), resolves default allocation and event-shuffle rules, sets up directory paths (FIFO, work, output), and determines the full-correlation and complex-model flags.

Args:

analysis_settings (dict): The full analysis settings dictionary. max_process_id (int): Number of parallel processes (defaults to CPU

count when <= 0).

number_of_processes (int): Alias used elsewhere (defaults to CPU: count when <= 0).

num_reinsurance_iterations (int): Number of reinsurance iterations. model_storage_json (dict or None): Optional model storage metadata. fifo_tmp_dir (bool): If True, create FIFOs under /tmp/. gul_alloc_rule (int or None): GUL allocation rule override. il_alloc_rule (int or None): IL allocation rule override. ri_alloc_rule (int or None): RI allocation rule override. num_gul_per_lb (int or None): GUL streams per load balancer. num_fm_per_lb (int or None): FM streams per load balancer. stderr_guard (bool): Wrap commands with stderr redirection. bash_trace (bool): Enable bash -x tracing. filename (str): Output script filename. _get_getmodel_cmd (callable or None): Custom GUL command builder. custom_gulcalc_cmd (str or None): Explicit custom GUL binary name. custom_gulcalc_log_start (str or None): Custom log-start marker. custom_gulcalc_log_finish (str or None): Custom log-finish marker. custom_args (dict): Extra arguments forwarded to downstream functions. fmpy_low_memory (bool): Enable low-memory mode in fmpy. fmpy_sort_output (bool): Sort fmpy output. event_shuffle (int or None): Event shuffle rule override. gulmc (bool): Use gulmc (Monte Carlo sampler). gul_random_generator (int): Random number generator selector. gulmc_effective_damageability (bool): Enable effective damageability. gulmc_vuln_cache_size (int): Vulnerability cache size for gulmc. process_number (int or None): Single chunk number (distributed mode). remove_working_files (bool): Clean up work dirs after completion. model_run_dir (str): Root directory for the model run. model_py_server (bool): Launch a servedata server. join_summary_info (bool): Append summary metadata to outputs. peril_filter (list): Peril filter list. exposure_df_engine (str): DataFrame engine for exposure data. model_df_engine (str): DataFrame engine for model data. dynamic_footprint (bool): Enable dynamic footprint mode.

Returns:

dict: Parameter dictionary ready for unpacking into create_bash_analysis() and create_bash_outputs().

Raises:

OasisException: If no valid output settings are found or an unknown: event shuffle rule is specified.

oasislmf.execution.bash.bash_wrapper(filename, bash_trace, stderr_guard, log_sub_dir=None, process_number=None, custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None)[source]¶

Context manager that wraps the script body with header and footer boilerplate.

On entry, writes the bash shebang, shell options (set -euET), log directory setup, optional bash tracing, the error-trap function, and the completion-check function.

On exit (after the yield), writes the footer: either a check_complete call (single-script mode) or a chunk-validation block that verifies no output files are empty (distributed-chunk mode).

Args:

filename (str): Path to the bash script being generated. bash_trace (bool): If True, enable bash -x tracing. stderr_guard (bool): If True, install the error trap and write the

completion-check function.

log_sub_dir (str or None): Sub-directory under log/ for chunk: mode logging.
process_number (int or None): Chunk number when running in: distributed mode.
custom_gulcalc_log_start (str or None): Custom log-start marker for: the completion check.
custom_gulcalc_log_finish (str or None): Custom log-finish marker for: the completion check.

Yields:

None: Control is yielded to the caller to write the script body.

oasislmf.execution.bash.create_bash_analysis(process_counter, max_process_id, num_reinsurance_iterations, fifo_tmp_dir, bash_trace, filename, _get_getmodel_cmd, custom_args, fmpy_low_memory, fmpy_sort_output, process_number, remove_working_files, model_run_dir, fifo_queue_dir, fifo_full_correlation_dir, stderr_guard, work_dir, work_kat_dir, work_full_correlation_dir, work_full_correlation_kat_dir, output_dir, output_full_correlation_dir, gul_alloc_rule, il_alloc_rule, ri_alloc_rule, num_gul_per_lb, num_fm_per_lb, event_shuffle, eve_shuffle_flag, kat_sort_by_event, gul_threshold, number_of_samples, use_random_number_file, full_correlation, gul_output, il_output, ri_output, rl_output, need_summary_fifo_for_gul, analysis_settings, gulmc, gul_random_generator, gulmc_effective_damageability, gulmc_vuln_cache_size, model_py_server, peril_filter, model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False, **kwargs)[source]¶

Write the main analysis section of the bash script.

This is the core function that assembles the calculation pipeline. It:

Sets up output, FIFO, and work directories.
Creates FIFOs for every (process, summary, ORD-type) combination.
Creates work-folder directories for post-wait aggregation.
Writes consumer commands in reverse pipeline order (ORD consumers, tees, summarycalcs) so readers are ready before writers.
Writes the main GUL pipeline commands (evepy | modelpy | gulpy/gulmc piped through fmpy for IL/RI), optionally with load balancers.
Adds wait calls for all background processes.
Cleans up FIFOs.

All parameters are typically supplied by unpacking the dict returned from bash_params().

oasislmf.execution.bash.create_bash_outputs(process_counter, fifo_tmp_dir, filename, remove_working_files, fifo_queue_dir, stderr_guard, work_dir, work_full_correlation_dir, output_dir, output_full_correlation_dir, full_correlation, gul_output, il_output, ri_output, rl_output, analysis_settings, num_reinsurance_iterations, num_gul_per_lb, num_fm_per_lb, max_process_id, work_kat_dir, kat_sort_by_event, work_full_correlation_kat_dir, join_summary_info, **kwargs)[source]¶

Write the output-aggregation and cleanup section of the bash script.

This function runs after the main analysis wait and handles:

Kat concatenation – katpy merges per-process binary ORD files into single output files for each run type.
Post-wait aggregation – aalpy (PALT/ALT) and lecpy (EPT/PSEPT) run over collected work-folder data.
Cleanup – removes working directories and temporary FIFOs.

All parameters are typically supplied by unpacking the dict returned from bash_params().

oasislmf.execution.bash.genbash(max_process_id, analysis_settings, num_reinsurance_iterations=0, fifo_tmp_dir=True, gul_alloc_rule=None, il_alloc_rule=None, ri_alloc_rule=None, num_gul_per_lb=None, num_fm_per_lb=None, stderr_guard=True, bash_trace=False, filename='run_kools.sh', _get_getmodel_cmd=None, custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None, custom_args={}, fmpy_low_memory=False, fmpy_sort_output=False, event_shuffle=None, gulmc=True, gul_random_generator=1, gulmc_effective_damageability=False, gulmc_vuln_cache_size=200, model_py_server=False, peril_filter=[], join_summary_info=False, base_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', model_df_engine=None, dynamic_footprint=False, analysis_pk=None, socket_server=None)[source]¶

Generates a bash script containing pytools calculation instructions for an Oasis model.

Parameters:

max_process_id (int) – The number of processes to create
analysis_settings (dict) – The analysis settings
filename (string) – The output file name
num_reinsurance_iterations (int) – The number of reinsurance iterations
fifo_tmp_dir (boolean) – When set to True, Create and use FIFO quese in /tmp/[A-Z,0-9]/fifo, if False run in ‘./fifo’
gul_alloc_rule (Int) – Allocation rule (None or 1) for gulcalc, if not set default to coverage stream
il_alloc_rule (Int) – Allocation rule (0, 1 or 2) for fmcalc
ri_alloc_rule (Int) – Allocation rule (0, 1 or 2) for fmcalc
num_gul_in_calc_block (Int) – number of gul in calc block
num_fm_in_calc_block (Int) – number of gul in calc block
get_getmodel_cmd (callable) – Method for getting the getmodel command, by default GenerateLossesCmd.get_getmodel_cmd is used.
base_df_engine (str) – The engine to use when loading dataframes.
model_df_engine (str) – The engine to use when loading model dataframes.

oasislmf.execution.bash.add_server_call(call, analysis_pk=None, socket_server=False)[source]¶

Inject WebSocket or socket-server flags into a GUL command string.

If the environment variables OASIS_WEBSOCKET_URL and OASIS_WEBSOCKET_PORT are set and analysis_pk is provided, the gulmc/gulpy invocation within call is augmented with --socket-server and --analysis-pk flags. Otherwise, only the --socket-server flag is added.

Args:: call (str): The full pipeline command string. analysis_pk (int or None): Analysis primary key for WebSocket mode. socket_server (bool): Default socket-server flag value.
Returns:: str: The (possibly modified) command string.

`logger`
`RUNTYPE_GROUNDUP_LOSS`
`RUNTYPE_LOAD_BALANCED_LOSS`
`RUNTYPE_INSURED_LOSS`
`RUNTYPE_REINSURANCE_LOSS`
`RUNTYPE_REINSURANCE_GROSS_LOSS`
`RUNTYPE_FULL_CORRELATION`
`REINSURANCE_RUNTYPES`
`INTERMEDIATE_INURING_PRIORITY_PREFIX`
`WAIT_PROCESSING_SWITCHES`
`ORD_EPT_OUTPUT_SWITCHES`
`ORD_PSEPT_OUTPUT_SWITCHES`
`ORD_LECCALC`
`ORD_ALT_OUTPUT_SWITCHES`
`ORD_ALT_MEANONLY_OUTPUT_SWITCHES`
`ORD_PLT_OUTPUT_SWITCHES`
`ORD_ELT_OUTPUT_SWITCHES`
`ORD_SELT_OUTPUT_SWITCH`
`OUTPUT_SWITCHES`
`EVE_SHUFFLE_OPTIONS`
`TRAP_FUNC`
`BASH_TRACE`

oasislmf.execution.bash¶

Attributes¶

Functions¶

Module Contents¶