oasislmf.execution.bash

Bash script generation for the pytools loss calculation pipeline.

This module generates bash scripts that orchestrate the pytools loss calculation pipeline. The generated scripts coordinate multiple concurrent processes connected via named pipes (FIFOs) to compute ground-up losses (GUL), insured losses (IL), reinsurance losses (RI), reinsurance gross losses (RL), and optionally fully-correlated (FC) and load-balanced (LB) streams.

The high-level flow of a generated bash script is:

  1. Initialisation – set shell options, logging, error traps, and the completion-check function.

  2. Directory and FIFO setup – create or clean work, output, and FIFO directories; create named pipes for every process/summary combination.

  3. Consumer commands (written in reverse pipeline order so readers are ready before writers start): a. do_ord – per-process ORD output consumers (eltpy, pltpy). b. do_tees – tee commands that fan summary streams out to

    work folders and ORD FIFOs.

    1. do_summarycalcs – summarypy commands that read per-runtype FIFOs and write per-summary FIFOs.

  4. Main GUL pipeline commands – evepy | modelpy | gulpy (or gulmc) piped through fmpy for IL/RI, writing into the FIFOs consumed above.

  5. Waits – wait on all background PIDs from steps 3-4.

  6. Kat concatenation – katpy merges per-process binary ORD files into single output files.

  7. Post-wait aggregation – aalpy and lecpy run over the collected work-folder data to produce ALT and EPT/PSEPT outputs.

  8. Cleanup – remove working directories and temporary FIFOs.

  9. Completion check – verify every started process logged a successful finish.

Run types:

  • gul – ground-up loss

  • il – insured (direct) loss

  • ri – reinsurance net loss

  • rl – reinsurance gross loss

  • fc – fully correlated ground-up loss

  • lb – load-balanced stream (intermediary between GUL and FM)

Entry points:

Attributes

logger

INTERMEDIATE_INURING_PRIORITY_PREFIX

WAIT_PROCESSING_SWITCHES

ORD_EPT_OUTPUT_SWITCHES

ORD_PSEPT_OUTPUT_SWITCHES

ORD_LECCALC

ORD_ALT_OUTPUT_SWITCHES

ORD_ALT_MEANONLY_OUTPUT_SWITCHES

ORD_PLT_OUTPUT_SWITCHES

ORD_ELT_OUTPUT_SWITCHES

ORD_SELT_OUTPUT_SWITCH

OUTPUT_SWITCHES

EVE_SHUFFLE_OPTIONS

FIFO_CHECK_FUNC

TRAP_FUNC

BASH_TRACE

WAIT_FUNC

Functions

get_check_function([custom_gulcalc_log_start, ...])

Creates a bash function to check the logs to ensure same number of process started and finsished.

process_range(max_process_id[, process_number])

Creates an iterable for all the process ids, if process number is set

get_modelcmd(→ str)

Gets the construct model command line argument for the bash script.

get_gulcmd(gulmc, gul_random_generator, ...[, ...])

Get the ground-up loss calculation command.

get_fmcmd([fmpy_low_memory, fmpy_sort_output])

Build the fmpy (financial module) command string.

print_command(command_file, cmd)

Writes the supplied command to the end of the generated script

ord_enabled(summary_options, ORD_SWITCHES)

Checks if ORD leccalc is enabled in a summaries section

do_post_wait_processing(runtype, analysis_settings, ...)

Write post-wait aggregation commands (aalpy, lecpy) to the bash script.

get_fifo_name(fifo_dir, producer, producer_id[, consumer])

Build the standardised path for a named pipe (FIFO).

do_fifo_exec(→ str)

Write a single FIFO create or remove command to the bash script.

do_fifos_exec(→ list)

Write FIFO create/remove commands for every process in the range.

do_fifos_exec_full_correlation(runtype, ...[, ...])

Write FIFO create/remove commands for full-correlation sumcalc and fmcalc pipes.

do_fifos_calc(runtype, analysis_settings, ...[, ...])

Write FIFO create/remove commands for all summary and ORD output pipes.

create_workfolders(runtype, analysis_settings, filename)

Write mkdir -p commands for work sub-directories needed by summary consumers.

do_kats(runtype, analysis_settings, max_process_id, ...)

Write katpy concatenation commands to merge per-process ORD files.

do_summarycalcs(runtype, analysis_settings, ...[, ...])

Write a summarypy command for one process to the bash script.

do_tees(runtype, analysis_settings, process_id, ...[, ...])

Write tee commands that fan each summary stream to ORD and work FIFOs.

do_tees_fc_sumcalc_fmcalc(process_id, filename, ...)

Write a tee command that splits the full-correlation GUL output.

get_correlated_output_stems(fifo_dir)

Return a dict of FIFO path stems for the full-correlation pipeline.

do_ord(runtype, analysis_settings, process_id, ...[, ...])

Write ORD output consumer commands (eltpy, pltpy) for one process.

get_ri_inuring_priorities(analysis_settings, ...)

Build the list of inuring priority dicts for reinsurance net loss.

get_rl_inuring_priorities(num_reinsurance_iterations)

Build the list of inuring priority dicts for reinsurance gross loss.

rl(analysis_settings, max_process_id, filename, ...[, ...])

Write all reinsurance gross loss (RL) consumer commands.

ri(analysis_settings, max_process_id, filename, ...[, ...])

Write all reinsurance net loss (RI) consumer commands.

il(analysis_settings, max_process_id, filename, ...[, ...])

Write all insured loss (IL) consumer commands.

do_gul(analysis_settings, max_process_id, filename, ...)

Write all ground-up loss (GUL) consumer commands.

do_gul_full_correlation(analysis_settings, ...[, ...])

Write GUL consumer commands for the full-correlation path.

do_waits(wait_variable, wait_count, filename)

Add waits to the script

do_pwaits(filename, process_counter)

Add pwaits to the script

do_awaits(filename, process_counter)

Add awaits to the script

do_lwaits(filename, process_counter)

Add lwaits to the script

do_kwaits(filename, process_counter)

Add kwaits to the script

do_jwaits(filename, process_counter)

Add jwaits to the script (join-summary-info processes)

get_getmodel_cmd(number_of_samples, gul_threshold, ...)

Gets the GUL pipeline command (gulpy/gulmc) for a single process

add_pid_to_shell_command(cmd, process_counter)

Add a variable to the end of a command in order to track the ID of the process executing it.

get_main_cmd_ri_stream(cmd, process_id, il_output, ...)

Gets the fmpy command for the reinsurance stream

get_main_cmd_il_stream(cmd, process_id, il_alloc_rule)

Gets the fmpy command for the insured losses stream

get_main_cmd_gul_stream(cmd, process_id[, fifo_dir, ...])

Gets the command to output ground up losses

get_complex_model_cmd(custom_gulcalc_cmd, ...)

Return a custom GUL command function for complex (third-party) models.

do_computes(outputs)

Execute a list of deferred compute operations.

get_main_cmd_lb(num_lb, num_in_per_lb, num_out_per_lb, ...)

Yield load_balancer commands that redistribute streams across FIFOs.

get_pla_cmd(pla, secondary_factor, uniform_factor)

Determine whether Post Loss Amplification should be implemented and issue

bash_params(analysis_settings[, max_process_id, ...])

Build the parameter dict consumed by create_bash_analysis() and create_bash_outputs().

bash_wrapper(filename, bash_trace, stderr_guard[, ...])

Context manager that wraps the script body with header and footer boilerplate.

create_bash_analysis(process_counter, max_process_id, ...)

Write the main analysis section of the bash script.

create_bash_outputs(process_counter, fifo_tmp_dir, ...)

Write the output-aggregation and cleanup section of the bash script.

genbash(max_process_id, analysis_settings[, ...])

Generates a bash script containing pytools calculation instructions for an

add_server_call(call[, analysis_pk, socket_server_port])

Inject WebSocket or socket-server flags into a GUL command string.

Module Contents

oasislmf.execution.bash.logger[source]
oasislmf.execution.bash.INTERMEDIATE_INURING_PRIORITY_PREFIX = 'IP'[source]
oasislmf.execution.bash.WAIT_PROCESSING_SWITCHES[source]
oasislmf.execution.bash.ORD_EPT_OUTPUT_SWITCHES[source]
oasislmf.execution.bash.ORD_PSEPT_OUTPUT_SWITCHES[source]
oasislmf.execution.bash.ORD_LECCALC[source]
oasislmf.execution.bash.ORD_ALT_OUTPUT_SWITCHES[source]
oasislmf.execution.bash.ORD_ALT_MEANONLY_OUTPUT_SWITCHES[source]
oasislmf.execution.bash.ORD_PLT_OUTPUT_SWITCHES[source]
oasislmf.execution.bash.ORD_ELT_OUTPUT_SWITCHES[source]
oasislmf.execution.bash.ORD_SELT_OUTPUT_SWITCH[source]
oasislmf.execution.bash.OUTPUT_SWITCHES[source]
oasislmf.execution.bash.EVE_SHUFFLE_OPTIONS[source]
oasislmf.execution.bash.FIFO_CHECK_FUNC = Multiline-String[source]
Show Value
"""
check_fifos() {
    local has_error=0
    for f in "$@"; do
        [ -e "$f" ] || { echo "[ERROR] Expected FIFO not found: $f"; has_error=1; continue; }
        [ -p "$f" ] || { echo "[ERROR] Not a FIFO: $f"; has_error=1; }
    done
    [ "$has_error" -eq 0 ] || false
}"""
oasislmf.execution.bash.TRAP_FUNC = Multiline-String[source]
Show Value
"""
touch $LOG_DIR/stderror.err
oasis_exec_monitor.sh $$ $LOG_DIR & pid0=$!

exit_handler(){
   exit_code=$?

   # disable handler
   trap - QUIT HUP INT KILL TERM ERR EXIT

   kill -9 $pid0 2> /dev/null
   [ -n "${spid:-}" ] && kill -9 "$spid" 2>/dev/null || true
   if [ "$exit_code" -gt 0 ]; then
       # Error - run process clean up
       echo 'Kernel execution error - exitcode='$exit_code

       set +x
       group_pid=$(ps -p $$ -o pgid --no-headers)
       sess_pid=$(ps -p $$ -o sess --no-headers)
       script_pid=$$
       printf "Script PID:%d, GPID:%s, SPID:%d
" $script_pid $group_pid $sess_pid >> $LOG_DIR/killout.txt

       ps -jf f -g $sess_pid > $LOG_DIR/subprocess_list
       PIDS_KILL=$(pgrep -a --pgroup $group_pid | awk 'BEGIN { FS = "[ \t\n]+" }{ if ($1 >= '$script_pid') print}' | grep -v celery | egrep -v \.log$  | egrep -v startup.sh$ | sort -n -r)
       echo "$PIDS_KILL" >> $LOG_DIR/killout.txt
       kill -9 $(echo "$PIDS_KILL" | awk 'BEGIN { FS = "[ \t\n]+" }{ print $1 }') 2>/dev/null
       exit $exit_code
   else
       # script successful
       exit 0
   fi
}
trap exit_handler QUIT HUP INT KILL TERM ERR EXIT"""
oasislmf.execution.bash.get_check_function(custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None)[source]

Creates a bash function to check the logs to ensure same number of process started and finsished.

Parameters:

  • custom_gulcalc_log_start (str) – Custom message printed to the logs when a process starts.

  • custom_gulcalc_log_finish (str) – Custom message printed to the logs when a process ends.

oasislmf.execution.bash.BASH_TRACE = Multiline-String[source]
Show Value
"""
# --- Redirect Bash trace to file ---
bash_logging_supported(){
    local BASH_VER_MAJOR=${BASH_VERSION:0:1}
    local BASH_VER_MINOR=${BASH_VERSION:2:1}

    if [[ "$BASH_VER_MAJOR" -gt 4 ]]; then
        echo 1; exit
    fi
    if [[ $BASH_VER_MAJOR -eq 4 ]] && [[ $BASH_VER_MINOR -gt 3 ]]; then
        echo 1; exit
    fi
    echo 0
}
if [ $(bash_logging_supported) == 1 ]; then
    exec   > >(tee -ia $LOG_DIR/bash.log)
    exec  2> >(tee -ia $LOG_DIR/bash.log >& 2)
    exec 19> $LOG_DIR/bash.log
    export BASH_XTRACEFD="19"
    set -x
else
    echo "WARNING: logging disabled, bash version '$BASH_VERSION' is not supported, minimum requirement is bash v4.4"
fi """
oasislmf.execution.bash.WAIT_FUNC = Multiline-String[source]
Show Value
"""
exec_wait(){
    local BASH_VER_MAJOR=${BASH_VERSION:0:1}
    local BASH_VER_MINOR=${BASH_VERSION:2:1}
    if [[ "$BASH_VER_MAJOR" -gt 5 ]] || { [[ "$BASH_VER_MAJOR" -eq 5 ]] && [[ "$BASH_VER_MINOR" -ge 1 ]]; }; then
        local pid_exitcode
        wait -p pid_exitcode "$@"
    else
        wait "$@"
    fi
}"""
oasislmf.execution.bash.process_range(max_process_id, process_number=None)[source]

Creates an iterable for all the process ids, if process number is set then an iterable containing only that number is returned.

This allows for the loss generation to be ran in different processes rather than accross multiple cores.

Parameters:

  • max_process_id – The largest process number

  • process_number – If set iterable only containing this number is returned

Returns:

iterable containing all the process numbers to process

oasislmf.execution.bash.get_modelcmd(server=False, peril_filter=[]) → str[source]

Gets the construct model command line argument for the bash script.

Parameters:

  • server – (bool) if set then enable ‘TCP’ ipc server/client mode

  • peril_filter – (list) list of perils to include (all included if empty)

oasislmf.execution.bash.get_gulcmd(gulmc, gul_random_generator, gulmc_effective_damageability, gulmc_vuln_cache_size, modelpy_server, peril_filter, model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False)[source]

Get the ground-up loss calculation command.

Parameters:

  • gulmc (bool) – if True, return the combined (model+ground up) command name, else use ‘modelpy | gulpy’ .

  • gul_random_generator (int) – random number generator to use in gulmc or gulpy (0: Mersenne-Twister, 1: Latin Hypercube, 2: Latin Hypercube on Philox4x32-7)

  • gulmc_effective_damageability (bool) – if True, use the effective damageability to draw loss samples instead of the full Monte Carlo method

  • gulmc_vuln_cache_size (int) – size in MB of the cache for the vulnerability calculations

  • modelpy_server (bool) – if True, run gulmc against the ‘TCP’ ipc data server

  • peril_filter (list) – list of perils to include (all included if empty)

  • model_df_engine (str) – the dataframe reading engine to use when loading model files

  • dynamic_footprint (bool) – if True, enable dynamic footprint processing

Returns:

the ground-up loss calculation command

Return type:

str

oasislmf.execution.bash.get_fmcmd(fmpy_low_memory=False, fmpy_sort_output=False)[source]

Build the fmpy (financial module) command string.

Parameters:

  • fmpy_low_memory (bool) – If True, append the -l flag to enable low-memory mode.

  • fmpy_sort_output (bool) – If True, append the --sort-output flag so that output records are sorted.

Returns:

The assembled fmpy command with any requested flags.

Return type:

str

oasislmf.execution.bash.print_command(command_file, cmd)[source]

Writes the supplied command to the end of the generated script

Parameters:

  • command_file – File to append command to.

  • cmd – The command to append

oasislmf.execution.bash.ord_enabled(summary_options, ORD_SWITCHES)[source]

Checks if ORD leccalc is enabled in a summaries section

Example: {

“id”: 1, “ord_output”: {

“ept_full_uncertainty_aep”: true, “ept_full_uncertainty_oep”: true, “ept_mean_sample_aep”: true, “ept_mean_sample_oep”: true, “ept_per_sample_mean_aep”: true, “ept_per_sample_mean_oep”: true, “psept_aep”: true, “psept_oep”: true, “return_period_file”: true

}

}

Parameters:

  • summary_options (dict) – Summaies section from an analysis_settings file

  • ORD_SWITCHES (dict) – Options from the analysis_settings ‘Summaies’ section to search

Returns:

True is leccalc is enabled, False otherwise.

oasislmf.execution.bash.do_post_wait_processing(runtype, analysis_settings, filename, process_counter, work_sub_dir='', output_dir='output/', stderr_guard=True, inuring_priority=None, join_summary_info=False)[source]

Write post-wait aggregation commands (aalpy, lecpy) to the bash script.

These commands run after all per-process wait calls have completed, operating on the binary work-folder data produced by tee during the main pipeline. Specifically this handles:

  • PALT (aalpy) – Period Average Loss Table output.

  • ALT mean-only (aalpy) – Average Loss Table (mean only) output.

  • EPT / PSEPT (lecpy) – Exceedance Probability Table and Per-Sample Exceedance Probability Table outputs.

Parameters:

  • runtype (str) – The run type identifier (e.g. 'gul', 'il', 'ri', 'rl').

  • analysis_settings (dict) – The full analysis settings dictionary.

  • filename (str) – Path to the bash script being generated.

  • process_counter (dict) – Mutable counter dict tracking background PIDs (keys lpid_monitor_count, etc.).

  • work_sub_dir (str) – Relative path prefix for work sub-directories.

  • output_dir (str) – Directory where final output files are written.

  • stderr_guard (bool) – If True, wrap commands in a sub-shell that redirects stderr to the log.

  • inuring_priority (str or None) – Inuring priority label to embed in file names, or None / empty for the final priority.

  • join_summary_info (bool) – If True, append join-summary-info commands to enrich outputs with summary metadata.

oasislmf.execution.bash.get_fifo_name(fifo_dir, producer, producer_id, consumer='')[source]

Build the standardised path for a named pipe (FIFO).

The naming convention is <fifo_dir><producer>_<consumer>_P<id> when a consumer is specified, or <fifo_dir><producer>_P<id> otherwise.

Parameters:

  • fifo_dir (str) – Base directory for FIFOs (e.g. 'fifo/').

  • producer (str) – Run type that produces the stream (e.g. 'gul').

  • producer_id (int) – Process number used as a suffix (P<id>).

  • consumer (str) – Optional consumer identifier appended between the producer and the process id.

Returns:

The fully-qualified FIFO path.

Return type:

str

oasislmf.execution.bash.do_fifo_exec(producer, producer_id, filename, fifo_dir, action='mkfifo', consumer='') → str[source]

Write a single FIFO create or remove command to the bash script.

Parameters:

  • producer (str) – Run type that produces the stream.

  • producer_id (int) – Process number for the FIFO name.

  • filename (str) – Path to the bash script being generated.

  • fifo_dir (str) – Base directory for FIFOs.

  • action (str) – Shell command to execute ('mkfifo' to create, 'rm' to remove).

  • consumer (str) – Optional consumer identifier for the FIFO name.

Returns:

The FIFO path that was operated on.

Return type:

str

oasislmf.execution.bash.do_fifos_exec(runtype, max_process_id, filename, fifo_dir, process_number=None, action='mkfifo', consumer='') → list[source]

Write FIFO create/remove commands for every process in the range.

Iterates over all process IDs and emits one mkfifo (or rm) command per process for the given run type.

Parameters:

  • runtype (str) – The run type identifier (e.g. 'gul', 'il').

  • max_process_id (int) – Upper bound of process IDs.

  • filename (str) – Path to the bash script being generated.

  • fifo_dir (str) – Base directory for FIFOs.

  • process_number (int or None) – If set, only that single process ID is used instead of the full range.

  • action (str) – Shell command ('mkfifo' or 'rm').

  • consumer (str) – Optional consumer identifier for FIFO names.

Returns:

The FIFO paths that were operated on.

Return type:

list[str]

oasislmf.execution.bash.do_fifos_exec_full_correlation(runtype, max_process_id, filename, fifo_dir, process_number=None, action='mkfifo')[source]

Write FIFO create/remove commands for full-correlation sumcalc and fmcalc pipes.

For each process ID two FIFOs are created: one for the summarycalc input (<runtype>_sumcalc_P<id>) and one for the fmcalc input (<runtype>_fmcalc_P<id>).

Parameters:

  • runtype (str) – The run type identifier (typically 'gul').

  • max_process_id (int) – Upper bound of process IDs.

  • filename (str) – Path to the bash script being generated.

  • fifo_dir (str) – Base directory for FIFOs.

  • process_number (int or None) – If set, restrict to a single process.

  • action (str) – Shell command ('mkfifo' or 'rm').

oasislmf.execution.bash.do_fifos_calc(runtype, analysis_settings, max_process_id, filename, fifo_dir='fifo/', process_number=None, consumer_prefix=None, action='mkfifo', summarypy_low_memory=False)[source]

Write FIFO create/remove commands for all summary and ORD output pipes.

For every (process, summary) pair this creates:

  • A summary FIFO (and its .idx companion when leccalc or ALT output is enabled and summarypy_low_memory is True — only then will summarypy -m open the .idx writer that the companion consumer reads from).

  • One FIFO per active ORD output type (plt, elt, selt).

Parameters:

  • runtype (str) – The run type identifier.

  • analysis_settings (dict) – The full analysis settings dictionary.

  • max_process_id (int) – Upper bound of process IDs.

  • filename (str) – Path to the bash script being generated.

  • fifo_dir (str) – Base directory for FIFOs.

  • process_number (int or None) – If set, restrict to a single process.

  • consumer_prefix (str or None) – Optional prefix prepended to consumer names (used for inuring priority labelling).

  • action (str) – Shell command ('mkfifo' or 'rm').

  • summarypy_low_memory (bool) – Whether summarypy -m will be used. Gates creation of the .idx companion FIFO so we don’t leave a tee consumer blocked on an unwritten pipe (only the legacy ktools leccalc/aalcalc binaries consume that pipe; lecpy/ aalpy only read .bin work files).

oasislmf.execution.bash.create_workfolders(runtype, analysis_settings, filename, work_dir='work/', inuring_priority=None)[source]

Write mkdir -p commands for work sub-directories needed by summary consumers.

Creates directories that tee will write binary summary data into for subsequent aggregation by lecpy (leccalc) and aalpy (aalcalc).

Parameters:

  • runtype (str) – The run type identifier (e.g. 'gul', 'il').

  • analysis_settings (dict) – The full analysis settings dictionary.

  • filename (str) – Path to the bash script being generated.

  • work_dir (str) – Root working directory (e.g. 'work/').

  • inuring_priority (str or None) – Inuring priority label to embed in directory names, or None / empty for the final priority.

oasislmf.execution.bash.do_kats(runtype, analysis_settings, max_process_id, filename, process_counter, work_dir='work/kat/', output_dir='output/', sort_by_event=False, process_number=None, inuring_priority=None, join_summary_info=False)[source]

Write katpy concatenation commands to merge per-process ORD files.

After the main pipeline wait has completed, per-process binary output files sit in the kat work directory. katpy reads them all and produces a single sorted (or unsorted) output file per summary/ORD-table combination.

Parameters:

  • runtype (str) – The run type identifier.

  • analysis_settings (dict) – The full analysis settings dictionary.

  • max_process_id (int) – Upper bound of process IDs.

  • filename (str) – Path to the bash script being generated.

  • process_counter (dict) – Mutable counter dict tracking background PIDs (key kpid_monitor_count).

  • work_dir (str) – Kat work directory containing per-process files.

  • output_dir (str) – Directory for final concatenated output files.

  • sort_by_event (bool) – If True, katpy sorts output by event id.

  • process_number (int or None) – If set, restrict to a single process.

  • inuring_priority (str or None) – Inuring priority label for file names.

  • join_summary_info (bool) – If True, append join-summary-info commands after each concatenation.

Returns:

True if any katpy commands were emitted, False otherwise.

Return type:

bool

oasislmf.execution.bash.do_summarycalcs(runtype, analysis_settings, process_id, filename, fifo_dir='fifo/', stderr_guard=True, num_reinsurance_iterations=0, gul_full_correlation=False, inuring_priority=None, summarypy_low_memory=False)[source]

Write a summarypy command for one process to the bash script.

Reads the per-process run-type FIFO and fans the stream out to one summary FIFO per summary set defined in the analysis settings.

Parameters:

  • runtype (str) – The run type identifier (e.g. 'gul', 'il').

  • analysis_settings (dict) – The full analysis settings dictionary.

  • process_id (int) – The process number for this command.

  • filename (str) – Path to the bash script being generated.

  • fifo_dir (str) – Base directory for FIFOs.

  • stderr_guard (bool) – If True, wrap the command in a sub-shell that redirects stderr to the log.

  • num_reinsurance_iterations (int) – Total number of reinsurance iterations (used for directory switching).

  • gul_full_correlation (bool) – If True, read from the _sumcalc FIFO variant used by the full-correlation path.

  • inuring_priority (dict or None) – Inuring priority dict with keys 'text' and 'level', or None for non-reinsurance runs.

  • summarypy_low_memory (bool) – If True, append the -m flag to summarypy so it writes a .idx side-file alongside each summary stream. The .idx is consumed by the legacy ktools leccalc and aalcalc binaries via the parallel tee chain (see do_tees). The Python aggregators (lecpy, aalpy, eltpy, pltpy) glob *.bin work files and do not read .idx – so they don’t require this flag.

oasislmf.execution.bash.do_tees(runtype, analysis_settings, process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', inuring_priority=None, summarypy_low_memory=False)[source]

Write tee commands that fan each summary stream to ORD and work FIFOs.

For each summary set, a tee reads from the summary FIFO and duplicates the stream to:

  • ORD output FIFOs (for eltpy/pltpy consumers).

  • Work-folder binary files (for aalpy/lecpy post-wait processing).

When summarypy_low_memory is True and leccalc or ALT output is enabled, a companion .idx tee is also emitted. Only the legacy ktools leccalc/aalcalc binaries consume the .idx chain; lecpy and aalpy only glob *.bin work files and don’t need it.

Parameters:

  • runtype (str) – The run type identifier.

  • analysis_settings (dict) – The full analysis settings dictionary.

  • process_id (int) – The process number.

  • filename (str) – Path to the bash script being generated.

  • process_counter (dict) – Mutable counter dict tracking background PIDs (key pid_monitor_count).

  • fifo_dir (str) – Base directory for FIFOs.

  • work_dir (str) – Root working directory.

  • inuring_priority (str or None) – Inuring priority label for file and FIFO names, or None / empty for the final priority.

  • summarypy_low_memory (bool) – Whether summarypy -m is being used. When False the .idx companion is skipped to avoid leaving a tee blocked on a pipe nothing writes to.

oasislmf.execution.bash.do_tees_fc_sumcalc_fmcalc(process_id, filename, correlated_output_stems)[source]

Write a tee command that splits the full-correlation GUL output.

Duplicates the correlated GUL stream into two FIFOs: one for the summarycalc path and one for the fmcalc (financial module) path.

Parameters:

  • process_id (int) – The process number.

  • filename (str) – Path to the bash script being generated.

  • correlated_output_stems (dict) – FIFO path stems returned by get_correlated_output_stems().

oasislmf.execution.bash.get_correlated_output_stems(fifo_dir)[source]

Return a dict of FIFO path stems for the full-correlation pipeline.

The returned dict contains three keys:

  • 'gulcalc_output' – stem for the GUL output FIFO.

  • 'fmcalc_input' – stem for the fmcalc input FIFO.

  • 'sumcalc_input' – stem for the summarycalc input FIFO.

Each value is a string ending with _P so that a process ID can be appended directly.

Parameters:

fifo_dir (str) – Base directory for FIFOs (e.g. 'fifo/').

Returns:

Mapping of stem names to FIFO path prefixes.

Return type:

dict

oasislmf.execution.bash.do_ord(runtype, analysis_settings, process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, inuring_priority=None)[source]

Write ORD output consumer commands (eltpy, pltpy) for one process.

For each summary set and each active ORD output type, reads from the corresponding FIFO and writes per-process binary output files into the kat work directory for later concatenation by katpy.

Parameters:

  • runtype (str) – The run type identifier.

  • analysis_settings (dict) – The full analysis settings dictionary.

  • process_id (int) – The process number.

  • filename (str) – Path to the bash script being generated.

  • process_counter (dict) – Mutable counter dict tracking background PIDs.

  • fifo_dir (str) – Base directory for FIFOs.

  • work_dir (str) – Root working directory (kat sub-directory is used).

  • stderr_guard (bool) – If True, wrap commands in a sub-shell that redirects stderr to the log.

  • inuring_priority (str or None) – Inuring priority label for file and FIFO names.

oasislmf.execution.bash.get_ri_inuring_priorities(analysis_settings, num_reinsurance_iterations)[source]

Build the list of inuring priority dicts for reinsurance net loss.

Returns intermediate inuring priorities specified in the analysis settings plus the final priority (the full reinsurance iteration count).

Parameters:

  • analysis_settings (dict) – The full analysis settings dictionary, checked for the 'ri_inuring_priorities' key.

  • num_reinsurance_iterations (int) – Total number of reinsurance iterations.

Returns:

Each dict has 'text' (file-name prefix) and 'level' (iteration number) keys.

Return type:

list[dict]

oasislmf.execution.bash.get_rl_inuring_priorities(num_reinsurance_iterations)[source]

Build the list of inuring priority dicts for reinsurance gross loss.

Unlike get_ri_inuring_priorities(), every iteration from 1 to num_reinsurance_iterations is included (there is no “final” entry).

Parameters:

num_reinsurance_iterations (int) – Total number of reinsurance iterations.

Returns:

Each dict has 'text' (file-name prefix) and 'level' (iteration number) keys.

Return type:

list[dict]

oasislmf.execution.bash.rl(analysis_settings, max_process_id, filename, process_counter, num_reinsurance_iterations, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None, summarypy_low_memory=False)[source]

Write all reinsurance gross loss (RL) consumer commands.

Iterates over each RL inuring priority and emits do_ord, do_tees, and do_summarycalcs commands for every process.

Parameters:

  • analysis_settings (dict) – The full analysis settings dictionary.

  • max_process_id (int) – Upper bound of process IDs.

  • filename (str) – Path to the bash script being generated.

  • process_counter (dict) – Mutable counter dict tracking background PIDs.

  • num_reinsurance_iterations (int) – Total number of reinsurance iterations.

  • fifo_dir (str) – Base directory for FIFOs.

  • work_dir (str) – Root working directory.

  • stderr_guard (bool) – If True, wrap commands in a sub-shell that redirects stderr to the log.

  • process_number (int or None) – If set, restrict to a single process.

  • summarypy_low_memory (bool) – Enable summarypy -m (write .idx side-files for downstream seek-by-event consumers).

oasislmf.execution.bash.ri(analysis_settings, max_process_id, filename, process_counter, num_reinsurance_iterations, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None, summarypy_low_memory=False)[source]

Write all reinsurance net loss (RI) consumer commands.

Iterates over each RI inuring priority and emits do_ord, do_tees, and do_summarycalcs commands for every process.

Parameters:

  • analysis_settings (dict) – The full analysis settings dictionary.

  • max_process_id (int) – Upper bound of process IDs.

  • filename (str) – Path to the bash script being generated.

  • process_counter (dict) – Mutable counter dict tracking background PIDs.

  • num_reinsurance_iterations (int) – Total number of reinsurance iterations.

  • fifo_dir (str) – Base directory for FIFOs.

  • work_dir (str) – Root working directory.

  • stderr_guard (bool) – If True, wrap commands in a sub-shell that redirects stderr to the log.

  • process_number (int or None) – If set, restrict to a single process.

  • summarypy_low_memory (bool) – Enable summarypy -m (write .idx side-files for downstream seek-by-event consumers).

oasislmf.execution.bash.il(analysis_settings, max_process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None, summarypy_low_memory=False)[source]

Write all insured loss (IL) consumer commands.

Emits do_ord, do_tees, and do_summarycalcs commands for every process in the range.

Parameters:

  • analysis_settings (dict) – The full analysis settings dictionary.

  • max_process_id (int) – Upper bound of process IDs.

  • filename (str) – Path to the bash script being generated.

  • process_counter (dict) – Mutable counter dict tracking background PIDs.

  • fifo_dir (str) – Base directory for FIFOs.

  • work_dir (str) – Root working directory.

  • stderr_guard (bool) – If True, wrap commands in a sub-shell that redirects stderr to the log.

  • process_number (int or None) – If set, restrict to a single process.

  • summarypy_low_memory (bool) – Enable summarypy -m.

oasislmf.execution.bash.do_gul(analysis_settings, max_process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None, summarypy_low_memory=False)[source]

Write all ground-up loss (GUL) consumer commands.

Emits do_ord, do_tees, and do_summarycalcs commands for every process in the range.

Parameters:

  • analysis_settings (dict) – The full analysis settings dictionary.

  • max_process_id (int) – Upper bound of process IDs.

  • filename (str) – Path to the bash script being generated.

  • process_counter (dict) – Mutable counter dict tracking background PIDs.

  • fifo_dir (str) – Base directory for FIFOs.

  • work_dir (str) – Root working directory.

  • stderr_guard (bool) – If True, wrap commands in a sub-shell that redirects stderr to the log.

  • process_number (int or None) – If set, restrict to a single process.

  • summarypy_low_memory (bool) – Enable summarypy -m.

oasislmf.execution.bash.do_gul_full_correlation(analysis_settings, max_process_id, filename, process_counter, fifo_dir='fifo/full_correlation/', work_dir='work/full_correlation/', stderr_guard=None, process_number=None)[source]

Write GUL consumer commands for the full-correlation path.

Similar to do_gul() but operates on the full_correlation/ sub-directories and only emits do_tees (no do_ord or do_summarycalcs).

Parameters:

  • analysis_settings (dict) – The full analysis settings dictionary.

  • max_process_id (int) – Upper bound of process IDs.

  • filename (str) – Path to the bash script being generated.

  • process_counter (dict) – Mutable counter dict tracking background PIDs.

  • fifo_dir (str) – FIFO directory for full-correlation pipes.

  • work_dir (str) – Work directory for full-correlation outputs.

  • stderr_guard – Unused (kept for interface consistency).

  • process_number (int or None) – If set, restrict to a single process.

oasislmf.execution.bash.do_waits(wait_variable, wait_count, filename)[source]

Add waits to the script

Parameters:

  • wait_variable (str) – The type of wait

  • wait_count (int) – The number of processes to wait for

  • filename (str) – Script to add waits to

oasislmf.execution.bash.do_pwaits(filename, process_counter)[source]

Add pwaits to the script

oasislmf.execution.bash.do_awaits(filename, process_counter)[source]

Add awaits to the script

oasislmf.execution.bash.do_lwaits(filename, process_counter)[source]

Add lwaits to the script

oasislmf.execution.bash.do_kwaits(filename, process_counter)[source]

Add kwaits to the script

oasislmf.execution.bash.do_jwaits(filename, process_counter)[source]

Add jwaits to the script (join-summary-info processes)

oasislmf.execution.bash.get_getmodel_cmd(number_of_samples, gul_threshold, use_random_number_file, gul_alloc_rule, item_output, process_id, max_process_id, correlated_output, eve_shuffle_flag, modelpy_server=False, peril_filter=[], gulmc=False, gul_random_generator=2, gulmc_effective_damageability=False, gulmc_vuln_cache_size=200, model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False, **kwargs)[source]

Gets the GUL pipeline command (gulpy/gulmc) for a single process

Parameters:

  • number_of_samples (int) – The number of samples to run

  • gul_threshold (float) – The GUL threshold to use

  • use_random_number_file (bool) – flag to use the random number file

  • gul_alloc_rule (int) – back allocation rule for gulcalc

  • item_output (str) – The item output

  • process_id (int) – The number of this process, passed to evepy as the partition to run

  • max_process_id (int) – The total number of processes, passed to evepy as the partition count

  • correlated_output (str) – Redirection appended to the command when correlated output is requested, empty otherwise

  • eve_shuffle_flag (str) – The event shuffling rule

  • modelpy_server (bool) – if True, run against the ‘TCP’ ipc data server

  • peril_filter (list) – list of perils to include (all included if empty)

  • gulmc (bool) – if True, use the combined gulmc command, else ‘modelpy | gulpy’

  • gul_random_generator (int) – random number generator to use in gulmc or gulpy (0: Mersenne-Twister, 1: Latin Hypercube, 2: Latin Hypercube on Philox4x32-7)

  • gulmc_effective_damageability (bool) – if True, use the effective damageability to draw loss samples instead of the full Monte Carlo method

  • gulmc_vuln_cache_size (int) – size in MB of the cache for the vulnerability calculations

  • model_df_engine (str) – The engine to use when loading dataframes

  • dynamic_footprint (bool) – if True, enable dynamic footprint processing

  • **kwargs – additional keyword arguments, accepted and ignored so that callers can forward the whole bash_params dict

Returns:

The generated getmodel command

oasislmf.execution.bash.add_pid_to_shell_command(cmd, process_counter)[source]

Add a variable to the end of a command in order to track the ID of the process executing it. Each time this function is called, the counter process_counter is incremented.

Parameters:

  • cmd (str) – the command whose process ID is to be stored in a variable.

  • process_counter (Counter or dict) – the number of process IDs that are being tracked.

Returns:

the updated command string.

Return type:

cmd (str)

oasislmf.execution.bash.get_main_cmd_ri_stream(cmd, process_id, il_output, il_alloc_rule, ri_alloc_rule, num_reinsurance_iterations, fifo_dir='fifo/', stderr_guard=True, from_file=False, fmpy_low_memory=False, fmpy_sort_output=False, step_flag='', process_counter=None, ri_inuring_priorities=None, rl_inuring_priorities=None)[source]

Gets the fmpy command for the reinsurance stream

Parameters:

  • cmd (str) – either gulcalc command stream or correlated output file

  • process_id (int) – ID corresponding to thread

  • il_output (bool) – If insured loss outputs required

  • il_alloc_rule (int) – insured loss allocation rule for fmcalc

  • ri_alloc_rule (int) – reinsurance allocation rule for fmcalc

  • num_reinsurance_iterations (int) – number of reinsurance iterations

  • fifo_dir (str) – path to fifo directory

  • stderr_guard (bool) – send stderr output to log file

  • from_file (bool) – must be true if cmd is a file and false if it can be piped

  • fmpy_low_memory (bool) – use memory map instead of RAM to store the loss array

  • fmpy_sort_output (bool) – order fmpy output by item_id

  • step_flag (str) – step policy flag appended to the first fmpy call

  • process_counter (Counter or dict) – if given, the process ID of the command is tracked in it

  • ri_inuring_priorities (dict) – Inuring priorities where net output has been requested

  • rl_inuring_priorities (dict) – Inuring priorities where gross output has been requested

Returns:

the assembled reinsurance stream command

Return type:

str

oasislmf.execution.bash.get_main_cmd_il_stream(cmd, process_id, il_alloc_rule, fifo_dir='fifo/', stderr_guard=True, from_file=False, fmpy_low_memory=False, fmpy_sort_output=False, step_flag='', process_counter=None)[source]

Gets the fmpy command for the insured losses stream

Parameters:

  • cmd (str) – either gulcalc command stream or correlated output file

  • process_id (int) – ID corresponding to thread

  • il_alloc_rule (int) – insured loss allocation rule for fmcalc

  • fifo_dir (str) – path to fifo directory

  • stderr_guard (bool) – send stderr output to log file

  • from_file (bool) – must be true if cmd is a file and false if it can be piped

  • fmpy_low_memory (bool) – use memory map instead of RAM to store the loss array

  • fmpy_sort_output (bool) – order fmpy output by item_id

  • step_flag (str) – step policy flag appended to the fmpy call

  • process_counter (Counter or dict) – if given, the process ID of the command is tracked in it

Returns:

generated fmcalc command as str

oasislmf.execution.bash.get_main_cmd_gul_stream(cmd, process_id, fifo_dir='fifo/', stderr_guard=True, consumer='', process_counter=None)[source]

Gets the command to output ground up losses

Parameters:

  • cmd (str) – either gulcalc command stream or correlated output file

  • process_id (int) – ID corresponding to thread

  • fifo_dir (str) – path to fifo directory

  • stderr_guard (bool) – send stderr output to log file

  • consumer (str) – optional name of the consumer of the stream

  • process_counter (Counter or dict) – if given, the process ID of the command is tracked in it

Returns:

generated command as str

oasislmf.execution.bash.get_complex_model_cmd(custom_gulcalc_cmd, analysis_settings)[source]

Return a custom GUL command function for complex (third-party) models.

If custom_gulcalc_cmd is explicitly provided it must exist in PATH; otherwise the function infers a command name from <supplier>_<model>_gulcalc and checks for its presence.

Parameters:

  • custom_gulcalc_cmd (str or None) – Explicit custom GUL binary name, or None to attempt auto-detection.

  • analysis_settings (dict) – The full analysis settings dictionary (used to infer the binary name via model_supplier_id and model_name_id).

Returns:

A function with the same signature as get_getmodel_cmd() that builds the custom GUL command string, or None if no custom binary was found.

Return type:

callable or None

Raises:

OasisException – If custom_gulcalc_cmd is explicitly set but cannot be found on PATH.

oasislmf.execution.bash.do_computes(outputs)[source]

Execute a list of deferred compute operations.

Each entry in outputs is a dict with keys 'compute_fun', 'compute_args', and 'loss_type'. The function calls compute_fun(**compute_args) for every entry, preceded by a comment header identifying the loss type.

Parameters:

outputs (list[dict]) – Deferred compute descriptors built up by the caller (e.g. create_bash_analysis()).

oasislmf.execution.bash.get_main_cmd_lb(num_lb, num_in_per_lb, num_out_per_lb, get_input_stream_name, get_output_stream_name, stderr_guard)[source]

Yield load_balancer commands that redistribute streams across FIFOs.

Each load balancer reads from num_in_per_lb input FIFOs and writes to num_out_per_lb output FIFOs. Process IDs are assigned sequentially across all load balancers.

Parameters:

  • num_lb (int) – Number of load balancer processes.

  • num_in_per_lb (int) – Number of input FIFOs per load balancer.

  • num_out_per_lb (int) – Number of output FIFOs per load balancer.

  • get_input_stream_name (callable) – Function accepting producer_id and returning the input FIFO path.

  • get_output_stream_name (callable) – Function accepting producer_id and returning the output FIFO path.

  • stderr_guard (bool) – If True, wrap commands in a sub-shell that redirects stderr to the log.

Yields:

str – A shell command string for each load balancer process.

oasislmf.execution.bash.get_pla_cmd(pla, secondary_factor, uniform_factor)[source]

Determine whether Post Loss Amplification should be implemented and issue plapy command.

Parameters:

  • pla (bool) – flag to apply post loss amplification

  • secondary_factor (float) – secondary factor to apply to post loss amplification

  • uniform_factor (float) – uniform factor to apply across all losses

Returns:

post loss amplification command

Return type:

pla_cmd (str)

oasislmf.execution.bash.bash_params(analysis_settings, max_process_id=-1, number_of_processes=-1, num_reinsurance_iterations=0, model_storage_json=None, fifo_tmp_dir=True, gul_alloc_rule=None, il_alloc_rule=None, ri_alloc_rule=None, num_gul_per_lb=None, num_fm_per_lb=None, stderr_guard=True, bash_trace=False, filename='run_kools.sh', _get_getmodel_cmd=None, custom_gulcalc_cmd=None, custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None, custom_args={}, fmpy_low_memory=False, fmpy_sort_output=False, summarypy_low_memory=False, event_shuffle=None, gulmc=True, gul_random_generator=2, gulmc_effective_damageability=False, gulmc_vuln_cache_size=200, process_number=None, remove_working_files=True, model_run_dir='', model_py_server=False, join_summary_info=False, peril_filter=[], exposure_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False, log_level=None, **kwargs)[source]

Build the parameter dict consumed by create_bash_analysis() and create_bash_outputs().

Validates analysis settings (checking output/summary coherence, reinsurance iteration requirements), resolves default allocation and event-shuffle rules, sets up directory paths (FIFO, work, output), and determines the full-correlation and complex-model flags.

Parameters:

  • analysis_settings (dict) – The full analysis settings dictionary.

  • max_process_id (int) – Number of parallel processes (defaults to CPU count when <= 0).

  • number_of_processes (int) – Alias used elsewhere (defaults to CPU count when <= 0).

  • num_reinsurance_iterations (int) – Number of reinsurance iterations.

  • model_storage_json (dict or None) – Optional model storage metadata.

  • fifo_tmp_dir (bool) – If True, create FIFOs under /tmp/.

  • gul_alloc_rule (int or None) – GUL allocation rule override.

  • il_alloc_rule (int or None) – IL allocation rule override.

  • ri_alloc_rule (int or None) – RI allocation rule override.

  • num_gul_per_lb (int or None) – GUL streams per load balancer.

  • num_fm_per_lb (int or None) – FM streams per load balancer.

  • stderr_guard (bool) – Wrap commands with stderr redirection.

  • bash_trace (bool) – Enable bash -x tracing.

  • filename (str) – Output script filename.

  • _get_getmodel_cmd (callable or None) – Custom GUL command builder.

  • custom_gulcalc_cmd (str or None) – Explicit custom GUL binary name.

  • custom_gulcalc_log_start (str or None) – Custom log-start marker.

  • custom_gulcalc_log_finish (str or None) – Custom log-finish marker.

  • custom_args (dict) – Extra arguments forwarded to downstream functions.

  • fmpy_low_memory (bool) – Enable low-memory mode in fmpy.

  • fmpy_sort_output (bool) – Sort fmpy output.

  • summarypy_low_memory (bool) – Pass -m to summarypy so it writes a .idx side-file, consumed by the legacy ktools leccalc/aalcalc binaries.

  • event_shuffle (int or None) – Event shuffle rule override.

  • gulmc (bool) – Use gulmc (Monte Carlo sampler).

  • gul_random_generator (int) – Random number generator selector.

  • gulmc_effective_damageability (bool) – Enable effective damageability.

  • gulmc_vuln_cache_size (int) – Vulnerability cache size for gulmc.

  • process_number (int or None) – Single chunk number (distributed mode).

  • remove_working_files (bool) – Clean up work dirs after completion.

  • model_run_dir (str) – Root directory for the model run.

  • model_py_server (bool) – Launch a servedata server.

  • join_summary_info (bool) – Append summary metadata to outputs.

  • peril_filter (list) – Peril filter list.

  • exposure_df_engine (str) – DataFrame engine for exposure data.

  • model_df_engine (str) – DataFrame engine for model data.

  • dynamic_footprint (bool) – Enable dynamic footprint mode.

  • log_level (int or None) – Enable logging of pytools subprocesses.

  • **kwargs – additional keyword arguments, accepted and ignored so that callers can forward a whole computation-step parameter dict.

Returns:

Parameter dictionary ready for unpacking into create_bash_analysis() and create_bash_outputs().

Return type:

dict

Raises:

OasisException – If no valid output settings are found or an unknown event shuffle rule is specified.

oasislmf.execution.bash.bash_wrapper(filename, bash_trace, stderr_guard, log_sub_dir=None, process_number=None, custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None)[source]

Context manager that wraps the script body with header and footer boilerplate.

On entry, writes the bash shebang, shell options (set -euET), log directory setup, optional bash tracing, the error-trap function, and the completion-check function.

On exit (after the yield), writes the footer: either a check_complete call (single-script mode) or a chunk-validation block that verifies no output files are empty (distributed-chunk mode).

Parameters:

  • filename (str) – Path to the bash script being generated.

  • bash_trace (bool) – If True, enable bash -x tracing.

  • stderr_guard (bool) – If True, install the error trap and write the completion-check function.

  • log_sub_dir (str or None) – Sub-directory under log/ for chunk mode logging.

  • process_number (int or None) – Chunk number when running in distributed mode.

  • custom_gulcalc_log_start (str or None) – Custom log-start marker for the completion check.

  • custom_gulcalc_log_finish (str or None) – Custom log-finish marker for the completion check.

Yields:

None – Control is yielded to the caller to write the script body.

oasislmf.execution.bash.create_bash_analysis(process_counter, max_process_id, num_reinsurance_iterations, fifo_tmp_dir, bash_trace, filename, _get_getmodel_cmd, custom_args, fmpy_low_memory, fmpy_sort_output, summarypy_low_memory, process_number, remove_working_files, model_run_dir, fifo_queue_dir, fifo_full_correlation_dir, stderr_guard, work_dir, work_kat_dir, work_full_correlation_dir, work_full_correlation_kat_dir, output_dir, output_full_correlation_dir, gul_alloc_rule, il_alloc_rule, ri_alloc_rule, num_gul_per_lb, num_fm_per_lb, event_shuffle, eve_shuffle_flag, kat_sort_by_event, gul_threshold, number_of_samples, use_random_number_file, full_correlation, gul_output, il_output, ri_output, rl_output, need_summary_fifo_for_gul, analysis_settings, gulmc, gul_random_generator, gulmc_effective_damageability, gulmc_vuln_cache_size, model_py_server, peril_filter, model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False, log_level=None, **kwargs)[source]

Write the main analysis section of the bash script.

This is the core function that assembles the calculation pipeline. It:

  1. Sets up output, FIFO, and work directories.

  2. Creates FIFOs for every (process, summary, ORD-type) combination.

  3. Creates work-folder directories for post-wait aggregation.

  4. Writes consumer commands in reverse pipeline order (ORD consumers, tees, summarycalcs) so readers are ready before writers.

  5. Writes the main GUL pipeline commands (evepy | modelpy | gulpy/gulmc piped through fmpy for IL/RI), optionally with load balancers.

  6. Adds wait calls for all background processes.

  7. Cleans up FIFOs.

All parameters are typically supplied by unpacking the dict returned from bash_params().

oasislmf.execution.bash.create_bash_outputs(process_counter, fifo_tmp_dir, filename, remove_working_files, fifo_queue_dir, stderr_guard, work_dir, work_full_correlation_dir, output_dir, output_full_correlation_dir, full_correlation, gul_output, il_output, ri_output, rl_output, analysis_settings, num_reinsurance_iterations, num_gul_per_lb, num_fm_per_lb, max_process_id, work_kat_dir, kat_sort_by_event, work_full_correlation_kat_dir, join_summary_info, **kwargs)[source]

Write the output-aggregation and cleanup section of the bash script.

This function runs after the main analysis wait and handles:

  1. Kat concatenation – katpy merges per-process binary ORD files into single output files for each run type.

  2. Post-wait aggregation – aalpy (PALT/ALT) and lecpy (EPT/PSEPT) run over collected work-folder data.

  3. Cleanup – removes working directories and temporary FIFOs.

All parameters are typically supplied by unpacking the dict returned from bash_params().

oasislmf.execution.bash.genbash(max_process_id, analysis_settings, num_reinsurance_iterations=0, fifo_tmp_dir=True, gul_alloc_rule=None, il_alloc_rule=None, ri_alloc_rule=None, num_gul_per_lb=None, num_fm_per_lb=None, stderr_guard=True, bash_trace=False, filename='run_kools.sh', _get_getmodel_cmd=None, custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None, custom_args={}, fmpy_low_memory=False, fmpy_sort_output=False, summarypy_low_memory=False, event_shuffle=None, gulmc=True, gul_random_generator=2, gulmc_effective_damageability=False, gulmc_vuln_cache_size=200, model_py_server=False, peril_filter=[], join_summary_info=False, base_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', model_df_engine=None, dynamic_footprint=False, analysis_pk=None, socket_server_size=None, socket_server_port=None, log_level=None)[source]

Generates a bash script containing pytools calculation instructions for an Oasis model.

Parameters:

  • max_process_id (int) – The number of processes to create

  • analysis_settings (dict) – The analysis settings

  • num_reinsurance_iterations (int) – The number of reinsurance iterations

  • fifo_tmp_dir (bool) – When set to True, Create and use FIFO quese in /tmp/[A-Z,0-9]/fifo, if False run in ‘./fifo’

  • gul_alloc_rule (Int) – Allocation rule (None or 1) for gulcalc, if not set default to coverage stream

  • il_alloc_rule (Int) – Allocation rule (0, 1 or 2) for fmcalc

  • ri_alloc_rule (Int) – Allocation rule (0, 1 or 2) for fmcalc

  • num_gul_per_lb (int) – number of gul streams per load balancer

  • num_fm_per_lb (int) – number of fm streams per load balancer

  • stderr_guard (bool) – Wrap commands with stderr redirection.

  • bash_trace (bool) – Enable bash -x tracing.

  • filename (str) – The output file name

  • _get_getmodel_cmd (callable) – Method for getting the getmodel command, by default GenerateLossesCmd.get_getmodel_cmd is used.

  • custom_gulcalc_log_start (str or None) – Log message produced when the custom gulcalc binary process starts.

  • custom_gulcalc_log_finish (str or None) – Log message produced when the custom gulcalc binary process ends.

  • custom_args (dict) – Extra arguments forwarded to downstream functions.

  • fmpy_low_memory (bool) – use memory map instead of RAM to store the loss array.

  • fmpy_sort_output (bool) – order fmpy output by item_id.

  • summarypy_low_memory (bool) – Pass -m to summarypy so it writes a .idx side-file, consumed by the legacy ktools leccalc/aalcalc binaries.

  • event_shuffle (int or None) – Event shuffle rule override.

  • gulmc (bool) – use the full Monte Carlo gulcalc python version.

  • gul_random_generator (int) – random number generator to use in gulmc or gulpy (0: Mersenne-Twister, 1: Latin Hypercube, 2: Latin Hypercube on Philox4x32-7).

  • gulmc_effective_damageability (bool) – use the effective damageability to draw loss samples instead of the full Monte Carlo method.

  • gulmc_vuln_cache_size (int) – size in MB of the cache for the vulnerability calculations.

  • model_py_server (bool) – run the data server for modelpy.

  • peril_filter (list) – list of perils to include (all included if empty).

  • join_summary_info (bool) – join summary id information to outputcalc csvs.

  • base_df_engine (str) – The engine to use when loading dataframes.

  • model_df_engine (str) – The engine to use when loading model dataframes.

  • dynamic_footprint (bool) – Enable dynamic footprint mode.

  • analysis_pk (int or None) – analysis primary key reported with progress pings. When set, no local socket-server is started.

  • socket_server_size (int or None) – total number of events, passed to the local socket-server that reports run progress.

  • socket_server_port (int or None) – port the local socket-server listens on.

  • log_level (int or None) – Enable logging of pytools subprocesses.

oasislmf.execution.bash.add_server_call(call, analysis_pk=None, socket_server_port=None)[source]

Inject WebSocket or socket-server flags into a GUL command string.

If the environment variables OASIS_WEBSOCKET_URL and OASIS_WEBSOCKET_PORT are set and analysis_pk is provided, the gulmc/gulpy invocation within call is augmented with --socket-server and --analysis-pk flags. Otherwise, the --socket-server flag is set to IP:port when a port is known.

Parameters:

  • call (str) – The full pipeline command string.

  • analysis_pk (int or None) – Analysis primary key for WebSocket mode.

  • socket_server_port (int or None) – Pre-checked available port to use.

Returns:

The (possibly modified) command string.

Return type:

str