oasislmf.execution.bash ======================= .. py:module:: oasislmf.execution.bash .. autoapi-nested-parse:: Bash script generation for the pytools loss calculation pipeline. This module generates bash scripts that orchestrate the pytools loss calculation pipeline. The generated scripts coordinate multiple concurrent processes connected via named pipes (FIFOs) to compute ground-up losses (GUL), insured losses (IL), reinsurance losses (RI), reinsurance gross losses (RL), and optionally fully-correlated (FC) and load-balanced (LB) streams. The high-level flow of a generated bash script is: 1. **Initialisation** -- set shell options, logging, error traps, and the completion-check function. 2. **Directory and FIFO setup** -- create or clean work, output, and FIFO directories; create named pipes for every process/summary combination. 3. **Consumer commands** (written in reverse pipeline order so readers are ready before writers start): a. ``do_ord`` -- per-process ORD output consumers (eltpy, pltpy). b. ``do_tees`` -- ``tee`` commands that fan summary streams out to work folders and ORD FIFOs. c. ``do_summarycalcs`` -- ``summarypy`` commands that read per-runtype FIFOs and write per-summary FIFOs. 4. **Main GUL pipeline commands** -- ``evepy | modelpy | gulpy`` (or ``gulmc``) piped through ``fmpy`` for IL/RI, writing into the FIFOs consumed above. 5. **Waits** -- ``wait`` on all background PIDs from steps 3-4. 6. **Kat concatenation** -- ``katpy`` merges per-process binary ORD files into single output files. 7. **Post-wait aggregation** -- ``aalpy`` and ``lecpy`` run over the collected work-folder data to produce ALT and EPT/PSEPT outputs. 8. **Cleanup** -- remove working directories and temporary FIFOs. 9. **Completion check** -- verify every started process logged a successful finish. Run types: * ``gul`` -- ground-up loss * ``il`` -- insured (direct) loss * ``ri`` -- reinsurance net loss * ``rl`` -- reinsurance gross loss * ``fc`` -- fully correlated ground-up loss * ``lb`` -- load-balanced stream (intermediary between GUL and FM) Entry points: * :func:`genbash` -- legacy single-call interface that writes a complete bash script. * :func:`bash_params` -- builds a parameter dict consumed by :func:`create_bash_analysis` and :func:`create_bash_outputs`. Attributes ---------- .. autoapisummary:: oasislmf.execution.bash.logger oasislmf.execution.bash.INTERMEDIATE_INURING_PRIORITY_PREFIX oasislmf.execution.bash.WAIT_PROCESSING_SWITCHES oasislmf.execution.bash.ORD_EPT_OUTPUT_SWITCHES oasislmf.execution.bash.ORD_PSEPT_OUTPUT_SWITCHES oasislmf.execution.bash.ORD_LECCALC oasislmf.execution.bash.ORD_ALT_OUTPUT_SWITCHES oasislmf.execution.bash.ORD_ALT_MEANONLY_OUTPUT_SWITCHES oasislmf.execution.bash.ORD_PLT_OUTPUT_SWITCHES oasislmf.execution.bash.ORD_ELT_OUTPUT_SWITCHES oasislmf.execution.bash.ORD_SELT_OUTPUT_SWITCH oasislmf.execution.bash.OUTPUT_SWITCHES oasislmf.execution.bash.EVE_SHUFFLE_OPTIONS oasislmf.execution.bash.FIFO_CHECK_FUNC oasislmf.execution.bash.TRAP_FUNC oasislmf.execution.bash.BASH_TRACE oasislmf.execution.bash.WAIT_FUNC Functions --------- .. autoapisummary:: oasislmf.execution.bash.get_check_function oasislmf.execution.bash.process_range oasislmf.execution.bash.get_modelcmd oasislmf.execution.bash.get_gulcmd oasislmf.execution.bash.get_fmcmd oasislmf.execution.bash.print_command oasislmf.execution.bash.ord_enabled oasislmf.execution.bash.do_post_wait_processing oasislmf.execution.bash.get_fifo_name oasislmf.execution.bash.do_fifo_exec oasislmf.execution.bash.do_fifos_exec oasislmf.execution.bash.do_fifos_calc oasislmf.execution.bash.create_workfolders oasislmf.execution.bash.do_kats oasislmf.execution.bash.do_summarycalcs oasislmf.execution.bash.do_tees oasislmf.execution.bash.do_ord oasislmf.execution.bash.get_ri_inuring_priorities oasislmf.execution.bash.get_rl_inuring_priorities oasislmf.execution.bash.rl oasislmf.execution.bash.ri oasislmf.execution.bash.il oasislmf.execution.bash.do_gul oasislmf.execution.bash.do_waits oasislmf.execution.bash.do_pwaits oasislmf.execution.bash.do_awaits oasislmf.execution.bash.do_lwaits oasislmf.execution.bash.do_kwaits oasislmf.execution.bash.do_jwaits oasislmf.execution.bash.get_getmodel_cmd oasislmf.execution.bash.add_pid_to_shell_command oasislmf.execution.bash.get_main_cmd_ri_stream oasislmf.execution.bash.get_main_cmd_il_stream oasislmf.execution.bash.get_main_cmd_gul_stream oasislmf.execution.bash.get_complex_model_cmd oasislmf.execution.bash.do_computes oasislmf.execution.bash.get_main_cmd_lb oasislmf.execution.bash.get_pla_cmd oasislmf.execution.bash.bash_params oasislmf.execution.bash.bash_wrapper oasislmf.execution.bash.create_bash_analysis oasislmf.execution.bash.create_bash_outputs oasislmf.execution.bash.genbash oasislmf.execution.bash.add_server_call Module Contents --------------- .. py:data:: logger .. py:data:: INTERMEDIATE_INURING_PRIORITY_PREFIX :value: 'IP' .. py:data:: WAIT_PROCESSING_SWITCHES .. py:data:: ORD_EPT_OUTPUT_SWITCHES .. py:data:: ORD_PSEPT_OUTPUT_SWITCHES .. py:data:: ORD_LECCALC .. py:data:: ORD_ALT_OUTPUT_SWITCHES .. py:data:: ORD_ALT_MEANONLY_OUTPUT_SWITCHES .. py:data:: ORD_PLT_OUTPUT_SWITCHES .. py:data:: ORD_ELT_OUTPUT_SWITCHES .. py:data:: ORD_SELT_OUTPUT_SWITCH .. py:data:: OUTPUT_SWITCHES .. py:data:: EVE_SHUFFLE_OPTIONS .. py:data:: FIFO_CHECK_FUNC :value: Multiline-String .. raw:: html
Show Value .. code-block:: python """ check_fifos() { local has_error=0 for f in "$@"; do [ -e "$f" ] || { echo "[ERROR] Expected FIFO not found: $f"; has_error=1; continue; } [ -p "$f" ] || { echo "[ERROR] Not a FIFO: $f"; has_error=1; } done [ "$has_error" -eq 0 ] || false }""" .. raw:: html
.. py:data:: TRAP_FUNC :value: Multiline-String .. raw:: html
Show Value .. code-block:: python """ touch $LOG_DIR/stderror.err oasis_exec_monitor.sh $$ $LOG_DIR & pid0=$! exit_handler(){ exit_code=$? # disable handler trap - QUIT HUP INT KILL TERM ERR EXIT kill -9 $pid0 2> /dev/null [ -n "${spid:-}" ] && kill -9 "$spid" 2>/dev/null || true if [ "$exit_code" -gt 0 ]; then # Error - run process clean up echo 'Kernel execution error - exitcode='$exit_code set +x group_pid=$(ps -p $$ -o pgid --no-headers) sess_pid=$(ps -p $$ -o sess --no-headers) script_pid=$$ printf "Script PID:%d, GPID:%s, SPID:%d " $script_pid $group_pid $sess_pid >> $LOG_DIR/killout.txt ps -jf f -g $sess_pid > $LOG_DIR/subprocess_list PIDS_KILL=$(pgrep -a --pgroup $group_pid | awk 'BEGIN { FS = "[ \t\n]+" }{ if ($1 >= '$script_pid') print}' | grep -v celery | egrep -v \.log$ | egrep -v startup.sh$ | sort -n -r) echo "$PIDS_KILL" >> $LOG_DIR/killout.txt kill -9 $(echo "$PIDS_KILL" | awk 'BEGIN { FS = "[ \t\n]+" }{ print $1 }') 2>/dev/null exit $exit_code else # script successful exit 0 fi } trap exit_handler QUIT HUP INT KILL TERM ERR EXIT""" .. raw:: html
.. py:function:: get_check_function(custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None) Creates a bash function to check the logs to ensure same number of process started and finsished. :param custom_gulcalc_log_start: Custom message printed to the logs when a process starts. :type custom_gulcalc_log_start: str :param custom_gulcalc_log_finish: Custom message printed to the logs when a process ends. :type custom_gulcalc_log_finish: str .. py:data:: BASH_TRACE :value: Multiline-String .. raw:: html
Show Value .. code-block:: python """ # --- Redirect Bash trace to file --- bash_logging_supported(){ local BASH_VER_MAJOR=${BASH_VERSION:0:1} local BASH_VER_MINOR=${BASH_VERSION:2:1} if [[ "$BASH_VER_MAJOR" -gt 4 ]]; then echo 1; exit fi if [[ $BASH_VER_MAJOR -eq 4 ]] && [[ $BASH_VER_MINOR -gt 3 ]]; then echo 1; exit fi echo 0 } if [ $(bash_logging_supported) == 1 ]; then exec > >(tee -ia $LOG_DIR/bash.log) exec 2> >(tee -ia $LOG_DIR/bash.log >& 2) exec 19> $LOG_DIR/bash.log export BASH_XTRACEFD="19" set -x else echo "WARNING: logging disabled, bash version '$BASH_VERSION' is not supported, minimum requirement is bash v4.4" fi """ .. raw:: html
.. py:data:: WAIT_FUNC :value: Multiline-String .. raw:: html
Show Value .. code-block:: python """ exec_wait(){ local BASH_VER_MAJOR=${BASH_VERSION:0:1} local BASH_VER_MINOR=${BASH_VERSION:2:1} if [[ "$BASH_VER_MAJOR" -gt 5 ]] || { [[ "$BASH_VER_MAJOR" -eq 5 ]] && [[ "$BASH_VER_MINOR" -ge 1 ]]; }; then local pid_exitcode wait -p pid_exitcode "$@" else wait "$@" fi }""" .. raw:: html
.. py:function:: process_range(max_process_id, process_number=None) Creates an iterable for all the process ids, if process number is set then an iterable containing only that number is returned. This allows for the loss generation to be ran in different processes rather than accross multiple cores. :param max_process_id: The largest process number :param process_number: If set iterable only containing this number is returned :returns: iterable containing all the process numbers to process .. py:function:: get_modelcmd(server=False, peril_filter=[]) -> str Gets the construct model command line argument for the bash script. :param server: (bool) if set then enable 'TCP' ipc server/client mode :param peril_filter: (list) list of perils to include (all included if empty) .. py:function:: get_gulcmd(gulmc, gul_random_generator, gulmc_effective_damageability, gulmc_vuln_cache_size, modelpy_server, peril_filter, model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False) Get the ground-up loss calculation command. :param gulmc: if True, return the combined (model+ground up) command name, else use 'modelpy | gulpy' . :type gulmc: bool :param gul_random_generator: random number generator to use in gulmc or gulpy (0: Mersenne-Twister, 1: Latin Hypercube, 2: Latin Hypercube on Philox4x32-7) :type gul_random_generator: int :param gulmc_effective_damageability: if True, use the effective damageability to draw loss samples instead of the full Monte Carlo method :type gulmc_effective_damageability: bool :param gulmc_vuln_cache_size: size in MB of the cache for the vulnerability calculations :type gulmc_vuln_cache_size: int :param modelpy_server: if True, run gulmc against the 'TCP' ipc data server :type modelpy_server: bool :param peril_filter: list of perils to include (all included if empty) :type peril_filter: list :param model_df_engine: the dataframe reading engine to use when loading model files :type model_df_engine: str :param dynamic_footprint: if True, enable dynamic footprint processing :type dynamic_footprint: bool :returns: the ground-up loss calculation command :rtype: str .. py:function:: get_fmcmd(fmpy_low_memory=False, fmpy_sort_output=False) Build the fmpy (financial module) command string. :param fmpy_low_memory: If True, append the ``-l`` flag to enable low-memory mode. :type fmpy_low_memory: bool :param fmpy_sort_output: If True, append the ``--sort-output`` flag so that output records are sorted. :type fmpy_sort_output: bool :returns: The assembled ``fmpy`` command with any requested flags. :rtype: str .. py:function:: print_command(command_file, cmd) Writes the supplied command to the end of the generated script :param command_file: File to append command to. :param cmd: The command to append .. py:function:: ord_enabled(summary_options, ORD_SWITCHES) Checks if ORD leccalc is enabled in a summaries section Example: { "id": 1, "ord_output": { "ept_full_uncertainty_aep": true, "ept_full_uncertainty_oep": true, "ept_mean_sample_aep": true, "ept_mean_sample_oep": true, "ept_per_sample_mean_aep": true, "ept_per_sample_mean_oep": true, "psept_aep": true, "psept_oep": true, "return_period_file": true } } :param summary_options: Summaies section from an analysis_settings file :type summary_options: dict :param ORD_SWITCHES: Options from the analysis_settings 'Summaies' section to search :type ORD_SWITCHES: dict :returns: True is leccalc is enabled, False otherwise. .. py:function:: do_post_wait_processing(runtype, analysis_settings, filename, process_counter, work_sub_dir='', output_dir='output/', stderr_guard=True, inuring_priority=None, join_summary_info=False) Write post-wait aggregation commands (aalpy, lecpy) to the bash script. These commands run *after* all per-process ``wait`` calls have completed, operating on the binary work-folder data produced by ``tee`` during the main pipeline. Specifically this handles: * **PALT** (``aalpy``) -- Period Average Loss Table output. * **ALT mean-only** (``aalpy``) -- Average Loss Table (mean only) output. * **EPT / PSEPT** (``lecpy``) -- Exceedance Probability Table and Per-Sample Exceedance Probability Table outputs. :param runtype: The run type identifier (e.g. ``'gul'``, ``'il'``, ``'ri'``, ``'rl'``). :type runtype: str :param analysis_settings: The full analysis settings dictionary. :type analysis_settings: dict :param filename: Path to the bash script being generated. :type filename: str :param process_counter: Mutable counter dict tracking background PIDs (keys ``lpid_monitor_count``, etc.). :type process_counter: dict :param work_sub_dir: Relative path prefix for work sub-directories. :type work_sub_dir: str :param output_dir: Directory where final output files are written. :type output_dir: str :param stderr_guard: If True, wrap commands in a sub-shell that redirects stderr to the log. :type stderr_guard: bool :param inuring_priority: Inuring priority label to embed in file names, or None / empty for the final priority. :type inuring_priority: str or None :param join_summary_info: If True, append ``join-summary-info`` commands to enrich outputs with summary metadata. :type join_summary_info: bool .. py:function:: get_fifo_name(fifo_dir, producer, producer_id, consumer='') Build the standardised path for a named pipe (FIFO). The naming convention is ``__P`` when a consumer is specified, or ``_P`` otherwise. :param fifo_dir: Base directory for FIFOs (e.g. ``'fifo/'``). :type fifo_dir: str :param producer: Run type that produces the stream (e.g. ``'gul'``). :type producer: str :param producer_id: Process number used as a suffix (``P``). :type producer_id: int :param consumer: Optional consumer identifier appended between the producer and the process id. :type consumer: str :returns: The fully-qualified FIFO path. :rtype: str .. py:function:: do_fifo_exec(producer, producer_id, filename, fifo_dir, action='mkfifo', consumer='') -> str Write a single FIFO create or remove command to the bash script. :param producer: Run type that produces the stream. :type producer: str :param producer_id: Process number for the FIFO name. :type producer_id: int :param filename: Path to the bash script being generated. :type filename: str :param fifo_dir: Base directory for FIFOs. :type fifo_dir: str :param action: Shell command to execute (``'mkfifo'`` to create, ``'rm'`` to remove). :type action: str :param consumer: Optional consumer identifier for the FIFO name. :type consumer: str :returns: The FIFO path that was operated on. :rtype: str .. py:function:: do_fifos_exec(runtype, max_process_id, filename, fifo_dir, process_number=None, action='mkfifo', consumer='') -> list Write FIFO create/remove commands for every process in the range. Iterates over all process IDs and emits one ``mkfifo`` (or ``rm``) command per process for the given run type. :param runtype: The run type identifier (e.g. ``'gul'``, ``'il'``). :type runtype: str :param max_process_id: Upper bound of process IDs. :type max_process_id: int :param filename: Path to the bash script being generated. :type filename: str :param fifo_dir: Base directory for FIFOs. :type fifo_dir: str :param process_number: If set, only that single process ID is used instead of the full range. :type process_number: int or None :param action: Shell command (``'mkfifo'`` or ``'rm'``). :type action: str :param consumer: Optional consumer identifier for FIFO names. :type consumer: str :returns: The FIFO paths that were operated on. :rtype: list[str] .. py:function:: do_fifos_calc(runtype, analysis_settings, max_process_id, filename, fifo_dir='fifo/', process_number=None, consumer_prefix=None, action='mkfifo', summarypy_low_memory=False) Write FIFO create/remove commands for all summary and ORD output pipes. For every (process, summary) pair this creates: * A summary FIFO (and its ``.idx`` companion when leccalc or ALT output is enabled **and** ``summarypy_low_memory`` is True — only then will ``summarypy -m`` open the ``.idx`` writer that the companion consumer reads from). * One FIFO per active ORD output type (plt, elt, selt). :param runtype: The run type identifier. :type runtype: str :param analysis_settings: The full analysis settings dictionary. :type analysis_settings: dict :param max_process_id: Upper bound of process IDs. :type max_process_id: int :param filename: Path to the bash script being generated. :type filename: str :param fifo_dir: Base directory for FIFOs. :type fifo_dir: str :param process_number: If set, restrict to a single process. :type process_number: int or None :param consumer_prefix: Optional prefix prepended to consumer names (used for inuring priority labelling). :type consumer_prefix: str or None :param action: Shell command (``'mkfifo'`` or ``'rm'``). :type action: str :param summarypy_low_memory: Whether ``summarypy -m`` will be used. Gates creation of the ``.idx`` companion FIFO so we don't leave a tee consumer blocked on an unwritten pipe (only the legacy ktools ``leccalc``/``aalcalc`` binaries consume that pipe; ``lecpy``/ ``aalpy`` only read ``.bin`` work files). :type summarypy_low_memory: bool .. py:function:: create_workfolders(runtype, analysis_settings, filename, work_dir='work/', inuring_priority=None) Write ``mkdir -p`` commands for work sub-directories needed by summary consumers. Creates directories that ``tee`` will write binary summary data into for subsequent aggregation by ``lecpy`` (leccalc) and ``aalpy`` (aalcalc). :param runtype: The run type identifier (e.g. ``'gul'``, ``'il'``). :type runtype: str :param analysis_settings: The full analysis settings dictionary. :type analysis_settings: dict :param filename: Path to the bash script being generated. :type filename: str :param work_dir: Root working directory (e.g. ``'work/'``). :type work_dir: str :param inuring_priority: Inuring priority label to embed in directory names, or None / empty for the final priority. :type inuring_priority: str or None .. py:function:: do_kats(runtype, analysis_settings, max_process_id, filename, process_counter, work_dir='work/kat/', output_dir='output/', sort_by_event=False, process_number=None, inuring_priority=None, join_summary_info=False) Write ``katpy`` concatenation commands to merge per-process ORD files. After the main pipeline ``wait`` has completed, per-process binary output files sit in the kat work directory. ``katpy`` reads them all and produces a single sorted (or unsorted) output file per summary/ORD-table combination. :param runtype: The run type identifier. :type runtype: str :param analysis_settings: The full analysis settings dictionary. :type analysis_settings: dict :param max_process_id: Upper bound of process IDs. :type max_process_id: int :param filename: Path to the bash script being generated. :type filename: str :param process_counter: Mutable counter dict tracking background PIDs (key ``kpid_monitor_count``). :type process_counter: dict :param work_dir: Kat work directory containing per-process files. :type work_dir: str :param output_dir: Directory for final concatenated output files. :type output_dir: str :param sort_by_event: If True, ``katpy`` sorts output by event id. :type sort_by_event: bool :param process_number: If set, restrict to a single process. :type process_number: int or None :param inuring_priority: Inuring priority label for file names. :type inuring_priority: str or None :param join_summary_info: If True, append ``join-summary-info`` commands after each concatenation. :type join_summary_info: bool :returns: True if any ``katpy`` commands were emitted, False otherwise. :rtype: bool .. py:function:: do_summarycalcs(runtype, analysis_settings, process_id, filename, fifo_dir='fifo/', stderr_guard=True, num_reinsurance_iterations=0, gul_full_correlation=False, inuring_priority=None, summarypy_low_memory=False) Write a ``summarypy`` command for one process to the bash script. Reads the per-process run-type FIFO and fans the stream out to one summary FIFO per summary set defined in the analysis settings. :param runtype: The run type identifier (e.g. ``'gul'``, ``'il'``). :type runtype: str :param analysis_settings: The full analysis settings dictionary. :type analysis_settings: dict :param process_id: The process number for this command. :type process_id: int :param filename: Path to the bash script being generated. :type filename: str :param fifo_dir: Base directory for FIFOs. :type fifo_dir: str :param stderr_guard: If True, wrap the command in a sub-shell that redirects stderr to the log. :type stderr_guard: bool :param num_reinsurance_iterations: Total number of reinsurance iterations (used for directory switching). :type num_reinsurance_iterations: int :param gul_full_correlation: If True, read from the ``_sumcalc`` FIFO variant used by the full-correlation path. :type gul_full_correlation: bool :param inuring_priority: Inuring priority dict with keys ``'text'`` and ``'level'``, or None for non-reinsurance runs. :type inuring_priority: dict or None :param summarypy_low_memory: If True, append the ``-m`` flag to ``summarypy`` so it writes a ``.idx`` side-file alongside each summary stream. The ``.idx`` is consumed by the legacy ktools ``leccalc`` and ``aalcalc`` binaries via the parallel ``tee`` chain (see ``do_tees``). The Python aggregators (``lecpy``, ``aalpy``, ``eltpy``, ``pltpy``) glob ``*.bin`` work files and do not read ``.idx`` -- so they don't require this flag. :type summarypy_low_memory: bool .. py:function:: do_tees(runtype, analysis_settings, process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', inuring_priority=None, summarypy_low_memory=False) Write ``tee`` commands that fan each summary stream to ORD and work FIFOs. For each summary set, a ``tee`` reads from the summary FIFO and duplicates the stream to: * ORD output FIFOs (for ``eltpy``/``pltpy`` consumers). * Work-folder binary files (for ``aalpy``/``lecpy`` post-wait processing). When ``summarypy_low_memory`` is True and leccalc or ALT output is enabled, a companion ``.idx`` tee is also emitted. Only the legacy ktools ``leccalc``/``aalcalc`` binaries consume the ``.idx`` chain; ``lecpy`` and ``aalpy`` only glob ``*.bin`` work files and don't need it. :param runtype: The run type identifier. :type runtype: str :param analysis_settings: The full analysis settings dictionary. :type analysis_settings: dict :param process_id: The process number. :type process_id: int :param filename: Path to the bash script being generated. :type filename: str :param process_counter: Mutable counter dict tracking background PIDs (key ``pid_monitor_count``). :type process_counter: dict :param fifo_dir: Base directory for FIFOs. :type fifo_dir: str :param work_dir: Root working directory. :type work_dir: str :param inuring_priority: Inuring priority label for file and FIFO names, or None / empty for the final priority. :type inuring_priority: str or None :param summarypy_low_memory: Whether ``summarypy -m`` is being used. When False the ``.idx`` companion is skipped to avoid leaving a tee blocked on a pipe nothing writes to. :type summarypy_low_memory: bool .. py:function:: do_ord(runtype, analysis_settings, process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, inuring_priority=None) Write ORD output consumer commands (``eltpy``, ``pltpy``) for one process. For each summary set and each active ORD output type, reads from the corresponding FIFO and writes per-process binary output files into the kat work directory for later concatenation by ``katpy``. :param runtype: The run type identifier. :type runtype: str :param analysis_settings: The full analysis settings dictionary. :type analysis_settings: dict :param process_id: The process number. :type process_id: int :param filename: Path to the bash script being generated. :type filename: str :param process_counter: Mutable counter dict tracking background PIDs. :type process_counter: dict :param fifo_dir: Base directory for FIFOs. :type fifo_dir: str :param work_dir: Root working directory (kat sub-directory is used). :type work_dir: str :param stderr_guard: If True, wrap commands in a sub-shell that redirects stderr to the log. :type stderr_guard: bool :param inuring_priority: Inuring priority label for file and FIFO names. :type inuring_priority: str or None .. py:function:: get_ri_inuring_priorities(analysis_settings, num_reinsurance_iterations) Build the list of inuring priority dicts for reinsurance net loss. Returns intermediate inuring priorities specified in the analysis settings plus the final priority (the full reinsurance iteration count). :param analysis_settings: The full analysis settings dictionary, checked for the ``'ri_inuring_priorities'`` key. :type analysis_settings: dict :param num_reinsurance_iterations: Total number of reinsurance iterations. :type num_reinsurance_iterations: int :returns: Each dict has ``'text'`` (file-name prefix) and ``'level'`` (iteration number) keys. :rtype: list[dict] .. py:function:: get_rl_inuring_priorities(num_reinsurance_iterations) Build the list of inuring priority dicts for reinsurance gross loss. Unlike :func:`get_ri_inuring_priorities`, every iteration from 1 to ``num_reinsurance_iterations`` is included (there is no "final" entry). :param num_reinsurance_iterations: Total number of reinsurance iterations. :type num_reinsurance_iterations: int :returns: Each dict has ``'text'`` (file-name prefix) and ``'level'`` (iteration number) keys. :rtype: list[dict] .. py:function:: rl(analysis_settings, max_process_id, filename, process_counter, num_reinsurance_iterations, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None, summarypy_low_memory=False) Write all reinsurance gross loss (RL) consumer commands. Iterates over each RL inuring priority and emits ``do_ord``, ``do_tees``, and ``do_summarycalcs`` commands for every process. :param analysis_settings: The full analysis settings dictionary. :type analysis_settings: dict :param max_process_id: Upper bound of process IDs. :type max_process_id: int :param filename: Path to the bash script being generated. :type filename: str :param process_counter: Mutable counter dict tracking background PIDs. :type process_counter: dict :param num_reinsurance_iterations: Total number of reinsurance iterations. :type num_reinsurance_iterations: int :param fifo_dir: Base directory for FIFOs. :type fifo_dir: str :param work_dir: Root working directory. :type work_dir: str :param stderr_guard: If True, wrap commands in a sub-shell that redirects stderr to the log. :type stderr_guard: bool :param process_number: If set, restrict to a single process. :type process_number: int or None :param summarypy_low_memory: Enable summarypy ``-m`` (write ``.idx`` side-files for downstream seek-by-event consumers). :type summarypy_low_memory: bool .. py:function:: ri(analysis_settings, max_process_id, filename, process_counter, num_reinsurance_iterations, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None, summarypy_low_memory=False) Write all reinsurance net loss (RI) consumer commands. Iterates over each RI inuring priority and emits ``do_ord``, ``do_tees``, and ``do_summarycalcs`` commands for every process. :param analysis_settings: The full analysis settings dictionary. :type analysis_settings: dict :param max_process_id: Upper bound of process IDs. :type max_process_id: int :param filename: Path to the bash script being generated. :type filename: str :param process_counter: Mutable counter dict tracking background PIDs. :type process_counter: dict :param num_reinsurance_iterations: Total number of reinsurance iterations. :type num_reinsurance_iterations: int :param fifo_dir: Base directory for FIFOs. :type fifo_dir: str :param work_dir: Root working directory. :type work_dir: str :param stderr_guard: If True, wrap commands in a sub-shell that redirects stderr to the log. :type stderr_guard: bool :param process_number: If set, restrict to a single process. :type process_number: int or None :param summarypy_low_memory: Enable summarypy ``-m`` (write ``.idx`` side-files for downstream seek-by-event consumers). :type summarypy_low_memory: bool .. py:function:: il(analysis_settings, max_process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None, summarypy_low_memory=False) Write all insured loss (IL) consumer commands. Emits ``do_ord``, ``do_tees``, and ``do_summarycalcs`` commands for every process in the range. :param analysis_settings: The full analysis settings dictionary. :type analysis_settings: dict :param max_process_id: Upper bound of process IDs. :type max_process_id: int :param filename: Path to the bash script being generated. :type filename: str :param process_counter: Mutable counter dict tracking background PIDs. :type process_counter: dict :param fifo_dir: Base directory for FIFOs. :type fifo_dir: str :param work_dir: Root working directory. :type work_dir: str :param stderr_guard: If True, wrap commands in a sub-shell that redirects stderr to the log. :type stderr_guard: bool :param process_number: If set, restrict to a single process. :type process_number: int or None :param summarypy_low_memory: Enable summarypy ``-m``. :type summarypy_low_memory: bool .. py:function:: do_gul(analysis_settings, max_process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None, summarypy_low_memory=False) Write all ground-up loss (GUL) consumer commands. Emits ``do_ord``, ``do_tees``, and ``do_summarycalcs`` commands for every process in the range. :param analysis_settings: The full analysis settings dictionary. :type analysis_settings: dict :param max_process_id: Upper bound of process IDs. :type max_process_id: int :param filename: Path to the bash script being generated. :type filename: str :param process_counter: Mutable counter dict tracking background PIDs. :type process_counter: dict :param fifo_dir: Base directory for FIFOs. :type fifo_dir: str :param work_dir: Root working directory. :type work_dir: str :param stderr_guard: If True, wrap commands in a sub-shell that redirects stderr to the log. :type stderr_guard: bool :param process_number: If set, restrict to a single process. :type process_number: int or None :param summarypy_low_memory: Enable summarypy ``-m``. :type summarypy_low_memory: bool .. py:function:: do_waits(wait_variable, wait_count, filename) Add waits to the script :param wait_variable: The type of wait :type wait_variable: str :param wait_count: The number of processes to wait for :type wait_count: int :param filename: Script to add waits to :type filename: str .. py:function:: do_pwaits(filename, process_counter) Add pwaits to the script .. py:function:: do_awaits(filename, process_counter) Add awaits to the script .. py:function:: do_lwaits(filename, process_counter) Add lwaits to the script .. py:function:: do_kwaits(filename, process_counter) Add kwaits to the script .. py:function:: do_jwaits(filename, process_counter) Add jwaits to the script (join-summary-info processes) .. py:function:: get_getmodel_cmd(number_of_samples, gul_threshold, use_random_number_file, gul_alloc_rule, item_output, process_id, max_process_id, correlated_output, eve_shuffle_flag, modelpy_server=False, peril_filter=[], gulmc=False, gul_random_generator=2, gulmc_effective_damageability=False, gulmc_vuln_cache_size=200, model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False, **kwargs) Gets the GUL pipeline command (gulpy/gulmc) for a single process :param number_of_samples: The number of samples to run :type number_of_samples: int :param gul_threshold: The GUL threshold to use :type gul_threshold: float :param use_random_number_file: flag to use the random number file :type use_random_number_file: bool :param gul_alloc_rule: back allocation rule for gulcalc :type gul_alloc_rule: int :param item_output: The item output :type item_output: str :param process_id: The number of this process, passed to evepy as the partition to run :type process_id: int :param max_process_id: The total number of processes, passed to evepy as the partition count :type max_process_id: int :param correlated_output: Redirection appended to the command when correlated output is requested, empty otherwise :type correlated_output: str :param eve_shuffle_flag: The event shuffling rule :type eve_shuffle_flag: str :param modelpy_server: if True, run against the 'TCP' ipc data server :type modelpy_server: bool :param peril_filter: list of perils to include (all included if empty) :type peril_filter: list :param gulmc: if True, use the combined gulmc command, else 'modelpy | gulpy' :type gulmc: bool :param gul_random_generator: random number generator to use in gulmc or gulpy (0: Mersenne-Twister, 1: Latin Hypercube, 2: Latin Hypercube on Philox4x32-7) :type gul_random_generator: int :param gulmc_effective_damageability: if True, use the effective damageability to draw loss samples instead of the full Monte Carlo method :type gulmc_effective_damageability: bool :param gulmc_vuln_cache_size: size in MB of the cache for the vulnerability calculations :type gulmc_vuln_cache_size: int :param model_df_engine: The engine to use when loading dataframes :type model_df_engine: str :param dynamic_footprint: if True, enable dynamic footprint processing :type dynamic_footprint: bool :param \*\*kwargs: additional keyword arguments, accepted and ignored so that callers can forward the whole bash_params dict :returns: The generated getmodel command .. py:function:: add_pid_to_shell_command(cmd, process_counter) Add a variable to the end of a command in order to track the ID of the process executing it. Each time this function is called, the counter `process_counter` is incremented. :param cmd: the command whose process ID is to be stored in a variable. :type cmd: str :param process_counter: the number of process IDs that are being tracked. :type process_counter: Counter or dict :returns: the updated command string. :rtype: cmd (str) .. py:function:: get_main_cmd_ri_stream(cmd, process_id, il_output, il_alloc_rule, ri_alloc_rule, num_reinsurance_iterations, fifo_dir='fifo/', stderr_guard=True, from_file=False, fmpy_low_memory=False, fmpy_sort_output=False, step_flag='', process_counter=None, ri_inuring_priorities=None, rl_inuring_priorities=None) Gets the fmpy command for the reinsurance stream :param cmd: either gulcalc command stream or correlated output file :type cmd: str :param process_id: ID corresponding to thread :type process_id: int :param il_output: If insured loss outputs required :type il_output: bool :param il_alloc_rule: insured loss allocation rule for fmcalc :type il_alloc_rule: int :param ri_alloc_rule: reinsurance allocation rule for fmcalc :type ri_alloc_rule: int :param num_reinsurance_iterations: number of reinsurance iterations :type num_reinsurance_iterations: int :param fifo_dir: path to fifo directory :type fifo_dir: str :param stderr_guard: send stderr output to log file :type stderr_guard: bool :param from_file: must be true if cmd is a file and false if it can be piped :type from_file: bool :param fmpy_low_memory: use memory map instead of RAM to store the loss array :type fmpy_low_memory: bool :param fmpy_sort_output: order fmpy output by item_id :type fmpy_sort_output: bool :param step_flag: step policy flag appended to the first fmpy call :type step_flag: str :param process_counter: if given, the process ID of the command is tracked in it :type process_counter: Counter or dict :param ri_inuring_priorities: Inuring priorities where net output has been requested :type ri_inuring_priorities: dict :param rl_inuring_priorities: Inuring priorities where gross output has been requested :type rl_inuring_priorities: dict :returns: the assembled reinsurance stream command :rtype: str .. py:function:: get_main_cmd_il_stream(cmd, process_id, il_alloc_rule, fifo_dir='fifo/', stderr_guard=True, from_file=False, fmpy_low_memory=False, fmpy_sort_output=False, step_flag='', process_counter=None) Gets the fmpy command for the insured losses stream :param cmd: either gulcalc command stream or correlated output file :type cmd: str :param process_id: ID corresponding to thread :type process_id: int :param il_alloc_rule: insured loss allocation rule for fmcalc :type il_alloc_rule: int :param fifo_dir: path to fifo directory :type fifo_dir: str :param stderr_guard: send stderr output to log file :type stderr_guard: bool :param from_file: must be true if cmd is a file and false if it can be piped :type from_file: bool :param fmpy_low_memory: use memory map instead of RAM to store the loss array :type fmpy_low_memory: bool :param fmpy_sort_output: order fmpy output by item_id :type fmpy_sort_output: bool :param step_flag: step policy flag appended to the fmpy call :type step_flag: str :param process_counter: if given, the process ID of the command is tracked in it :type process_counter: Counter or dict :returns: generated fmcalc command as str .. py:function:: get_main_cmd_gul_stream(cmd, process_id, fifo_dir='fifo/', stderr_guard=True, consumer='', process_counter=None) Gets the command to output ground up losses :param cmd: either gulcalc command stream or correlated output file :type cmd: str :param process_id: ID corresponding to thread :type process_id: int :param fifo_dir: path to fifo directory :type fifo_dir: str :param stderr_guard: send stderr output to log file :type stderr_guard: bool :param consumer: optional name of the consumer of the stream :type consumer: str :param process_counter: if given, the process ID of the command is tracked in it :type process_counter: Counter or dict :returns: generated command as str .. py:function:: get_complex_model_cmd(custom_gulcalc_cmd, analysis_settings) Return a custom GUL command function for complex (third-party) models. If ``custom_gulcalc_cmd`` is explicitly provided it must exist in ``PATH``; otherwise the function infers a command name from ``__gulcalc`` and checks for its presence. :param custom_gulcalc_cmd: Explicit custom GUL binary name, or None to attempt auto-detection. :type custom_gulcalc_cmd: str or None :param analysis_settings: The full analysis settings dictionary (used to infer the binary name via ``model_supplier_id`` and ``model_name_id``). :type analysis_settings: dict :returns: A function with the same signature as :func:`get_getmodel_cmd` that builds the custom GUL command string, or None if no custom binary was found. :rtype: callable or None :raises OasisException: If ``custom_gulcalc_cmd`` is explicitly set but cannot be found on ``PATH``. .. py:function:: do_computes(outputs) Execute a list of deferred compute operations. Each entry in *outputs* is a dict with keys ``'compute_fun'``, ``'compute_args'``, and ``'loss_type'``. The function calls ``compute_fun(**compute_args)`` for every entry, preceded by a comment header identifying the loss type. :param outputs: Deferred compute descriptors built up by the caller (e.g. :func:`create_bash_analysis`). :type outputs: list[dict] .. py:function:: get_main_cmd_lb(num_lb, num_in_per_lb, num_out_per_lb, get_input_stream_name, get_output_stream_name, stderr_guard) Yield ``load_balancer`` commands that redistribute streams across FIFOs. Each load balancer reads from *num_in_per_lb* input FIFOs and writes to *num_out_per_lb* output FIFOs. Process IDs are assigned sequentially across all load balancers. :param num_lb: Number of load balancer processes. :type num_lb: int :param num_in_per_lb: Number of input FIFOs per load balancer. :type num_in_per_lb: int :param num_out_per_lb: Number of output FIFOs per load balancer. :type num_out_per_lb: int :param get_input_stream_name: Function accepting ``producer_id`` and returning the input FIFO path. :type get_input_stream_name: callable :param get_output_stream_name: Function accepting ``producer_id`` and returning the output FIFO path. :type get_output_stream_name: callable :param stderr_guard: If True, wrap commands in a sub-shell that redirects stderr to the log. :type stderr_guard: bool :Yields: *str* -- A shell command string for each load balancer process. .. py:function:: get_pla_cmd(pla, secondary_factor, uniform_factor) Determine whether Post Loss Amplification should be implemented and issue plapy command. :param pla: flag to apply post loss amplification :type pla: bool :param secondary_factor: secondary factor to apply to post loss amplification :type secondary_factor: float :param uniform_factor: uniform factor to apply across all losses :type uniform_factor: float :returns: post loss amplification command :rtype: pla_cmd (str) .. py:function:: bash_params(analysis_settings, max_process_id=-1, number_of_processes=-1, num_reinsurance_iterations=0, model_storage_json=None, fifo_tmp_dir=True, gul_alloc_rule=None, il_alloc_rule=None, ri_alloc_rule=None, num_gul_per_lb=None, num_fm_per_lb=None, stderr_guard=True, bash_trace=False, filename='run_kools.sh', _get_getmodel_cmd=None, custom_gulcalc_cmd=None, custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None, custom_args={}, fmpy_low_memory=False, fmpy_sort_output=False, summarypy_low_memory=False, event_shuffle=None, gulmc=True, gul_random_generator=2, gulmc_effective_damageability=False, gulmc_vuln_cache_size=200, process_number=None, remove_working_files=True, model_run_dir='', model_py_server=False, join_summary_info=False, peril_filter=[], exposure_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False, log_level=None, **kwargs) Build the parameter dict consumed by :func:`create_bash_analysis` and :func:`create_bash_outputs`. Validates analysis settings (checking output/summary coherence, reinsurance iteration requirements), resolves default allocation and event-shuffle rules, sets up directory paths (FIFO, work, output), and determines the full-correlation and complex-model flags. :param analysis_settings: The full analysis settings dictionary. :type analysis_settings: dict :param max_process_id: Number of parallel processes (defaults to CPU count when ``<= 0``). :type max_process_id: int :param number_of_processes: Alias used elsewhere (defaults to CPU count when ``<= 0``). :type number_of_processes: int :param num_reinsurance_iterations: Number of reinsurance iterations. :type num_reinsurance_iterations: int :param model_storage_json: Optional model storage metadata. :type model_storage_json: dict or None :param fifo_tmp_dir: If True, create FIFOs under ``/tmp/``. :type fifo_tmp_dir: bool :param gul_alloc_rule: GUL allocation rule override. :type gul_alloc_rule: int or None :param il_alloc_rule: IL allocation rule override. :type il_alloc_rule: int or None :param ri_alloc_rule: RI allocation rule override. :type ri_alloc_rule: int or None :param num_gul_per_lb: GUL streams per load balancer. :type num_gul_per_lb: int or None :param num_fm_per_lb: FM streams per load balancer. :type num_fm_per_lb: int or None :param stderr_guard: Wrap commands with stderr redirection. :type stderr_guard: bool :param bash_trace: Enable bash ``-x`` tracing. :type bash_trace: bool :param filename: Output script filename. :type filename: str :param _get_getmodel_cmd: Custom GUL command builder. :type _get_getmodel_cmd: callable or None :param custom_gulcalc_cmd: Explicit custom GUL binary name. :type custom_gulcalc_cmd: str or None :param custom_gulcalc_log_start: Custom log-start marker. :type custom_gulcalc_log_start: str or None :param custom_gulcalc_log_finish: Custom log-finish marker. :type custom_gulcalc_log_finish: str or None :param custom_args: Extra arguments forwarded to downstream functions. :type custom_args: dict :param fmpy_low_memory: Enable low-memory mode in ``fmpy``. :type fmpy_low_memory: bool :param fmpy_sort_output: Sort ``fmpy`` output. :type fmpy_sort_output: bool :param summarypy_low_memory: Pass ``-m`` to ``summarypy`` so it writes a ``.idx`` side-file, consumed by the legacy ktools leccalc/aalcalc binaries. :type summarypy_low_memory: bool :param event_shuffle: Event shuffle rule override. :type event_shuffle: int or None :param gulmc: Use ``gulmc`` (Monte Carlo sampler). :type gulmc: bool :param gul_random_generator: Random number generator selector. :type gul_random_generator: int :param gulmc_effective_damageability: Enable effective damageability. :type gulmc_effective_damageability: bool :param gulmc_vuln_cache_size: Vulnerability cache size for ``gulmc``. :type gulmc_vuln_cache_size: int :param process_number: Single chunk number (distributed mode). :type process_number: int or None :param remove_working_files: Clean up work dirs after completion. :type remove_working_files: bool :param model_run_dir: Root directory for the model run. :type model_run_dir: str :param model_py_server: Launch a ``servedata`` server. :type model_py_server: bool :param join_summary_info: Append summary metadata to outputs. :type join_summary_info: bool :param peril_filter: Peril filter list. :type peril_filter: list :param exposure_df_engine: DataFrame engine for exposure data. :type exposure_df_engine: str :param model_df_engine: DataFrame engine for model data. :type model_df_engine: str :param dynamic_footprint: Enable dynamic footprint mode. :type dynamic_footprint: bool :param log_level: Enable logging of pytools subprocesses. :type log_level: int or None :param \*\*kwargs: additional keyword arguments, accepted and ignored so that callers can forward a whole computation-step parameter dict. :returns: Parameter dictionary ready for unpacking into :func:`create_bash_analysis` and :func:`create_bash_outputs`. :rtype: dict :raises OasisException: If no valid output settings are found or an unknown event shuffle rule is specified. .. py:function:: bash_wrapper(filename, bash_trace, stderr_guard, log_sub_dir=None, process_number=None, custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None) Context manager that wraps the script body with header and footer boilerplate. On entry, writes the bash shebang, shell options (``set -euET``), log directory setup, optional bash tracing, the error-trap function, and the completion-check function. On exit (after the ``yield``), writes the footer: either a ``check_complete`` call (single-script mode) or a chunk-validation block that verifies no output files are empty (distributed-chunk mode). :param filename: Path to the bash script being generated. :type filename: str :param bash_trace: If True, enable bash ``-x`` tracing. :type bash_trace: bool :param stderr_guard: If True, install the error trap and write the completion-check function. :type stderr_guard: bool :param log_sub_dir: Sub-directory under ``log/`` for chunk mode logging. :type log_sub_dir: str or None :param process_number: Chunk number when running in distributed mode. :type process_number: int or None :param custom_gulcalc_log_start: Custom log-start marker for the completion check. :type custom_gulcalc_log_start: str or None :param custom_gulcalc_log_finish: Custom log-finish marker for the completion check. :type custom_gulcalc_log_finish: str or None :Yields: *None* -- Control is yielded to the caller to write the script body. .. py:function:: create_bash_analysis(process_counter, max_process_id, num_reinsurance_iterations, fifo_tmp_dir, bash_trace, filename, _get_getmodel_cmd, custom_args, fmpy_low_memory, fmpy_sort_output, summarypy_low_memory, process_number, remove_working_files, model_run_dir, fifo_queue_dir, fifo_full_correlation_dir, stderr_guard, work_dir, work_kat_dir, work_full_correlation_dir, work_full_correlation_kat_dir, output_dir, output_full_correlation_dir, gul_alloc_rule, il_alloc_rule, ri_alloc_rule, num_gul_per_lb, num_fm_per_lb, event_shuffle, eve_shuffle_flag, kat_sort_by_event, gul_threshold, number_of_samples, use_random_number_file, full_correlation, gul_output, il_output, ri_output, rl_output, need_summary_fifo_for_gul, analysis_settings, gulmc, gul_random_generator, gulmc_effective_damageability, gulmc_vuln_cache_size, model_py_server, peril_filter, model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False, log_level=None, **kwargs) Write the main analysis section of the bash script. This is the core function that assembles the calculation pipeline. It: 1. Sets up output, FIFO, and work directories. 2. Creates FIFOs for every (process, summary, ORD-type) combination. 3. Creates work-folder directories for post-wait aggregation. 4. Writes consumer commands in reverse pipeline order (ORD consumers, tees, summarycalcs) so readers are ready before writers. 5. Writes the main GUL pipeline commands (``evepy | modelpy | gulpy/gulmc`` piped through ``fmpy`` for IL/RI), optionally with load balancers. 6. Adds ``wait`` calls for all background processes. 7. Cleans up FIFOs. All parameters are typically supplied by unpacking the dict returned from :func:`bash_params`. .. py:function:: create_bash_outputs(process_counter, fifo_tmp_dir, filename, remove_working_files, fifo_queue_dir, stderr_guard, work_dir, work_full_correlation_dir, output_dir, output_full_correlation_dir, full_correlation, gul_output, il_output, ri_output, rl_output, analysis_settings, num_reinsurance_iterations, num_gul_per_lb, num_fm_per_lb, max_process_id, work_kat_dir, kat_sort_by_event, work_full_correlation_kat_dir, join_summary_info, **kwargs) Write the output-aggregation and cleanup section of the bash script. This function runs *after* the main analysis ``wait`` and handles: 1. **Kat concatenation** -- ``katpy`` merges per-process binary ORD files into single output files for each run type. 2. **Post-wait aggregation** -- ``aalpy`` (PALT/ALT) and ``lecpy`` (EPT/PSEPT) run over collected work-folder data. 3. **Cleanup** -- removes working directories and temporary FIFOs. All parameters are typically supplied by unpacking the dict returned from :func:`bash_params`. .. py:function:: genbash(max_process_id, analysis_settings, num_reinsurance_iterations=0, fifo_tmp_dir=True, gul_alloc_rule=None, il_alloc_rule=None, ri_alloc_rule=None, num_gul_per_lb=None, num_fm_per_lb=None, stderr_guard=True, bash_trace=False, filename='run_kools.sh', _get_getmodel_cmd=None, custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None, custom_args={}, fmpy_low_memory=False, fmpy_sort_output=False, summarypy_low_memory=False, event_shuffle=None, gulmc=True, gul_random_generator=2, gulmc_effective_damageability=False, gulmc_vuln_cache_size=200, model_py_server=False, peril_filter=[], join_summary_info=False, base_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', model_df_engine=None, dynamic_footprint=False, analysis_pk=None, socket_server_size=None, socket_server_port=None, log_level=None) Generates a bash script containing pytools calculation instructions for an Oasis model. :param max_process_id: The number of processes to create :type max_process_id: int :param analysis_settings: The analysis settings :type analysis_settings: dict :param num_reinsurance_iterations: The number of reinsurance iterations :type num_reinsurance_iterations: int :param fifo_tmp_dir: When set to True, Create and use FIFO quese in `/tmp/[A-Z,0-9]/fifo`, if False run in './fifo' :type fifo_tmp_dir: bool :param gul_alloc_rule: Allocation rule (None or 1) for gulcalc, if not set default to coverage stream :type gul_alloc_rule: Int :param il_alloc_rule: Allocation rule (0, 1 or 2) for fmcalc :type il_alloc_rule: Int :param ri_alloc_rule: Allocation rule (0, 1 or 2) for fmcalc :type ri_alloc_rule: Int :param num_gul_per_lb: number of gul streams per load balancer :type num_gul_per_lb: int :param num_fm_per_lb: number of fm streams per load balancer :type num_fm_per_lb: int :param stderr_guard: Wrap commands with stderr redirection. :type stderr_guard: bool :param bash_trace: Enable bash ``-x`` tracing. :type bash_trace: bool :param filename: The output file name :type filename: str :param _get_getmodel_cmd: Method for getting the getmodel command, by default ``GenerateLossesCmd.get_getmodel_cmd`` is used. :type _get_getmodel_cmd: callable :param custom_gulcalc_log_start: Log message produced when the custom gulcalc binary process starts. :type custom_gulcalc_log_start: str or None :param custom_gulcalc_log_finish: Log message produced when the custom gulcalc binary process ends. :type custom_gulcalc_log_finish: str or None :param custom_args: Extra arguments forwarded to downstream functions. :type custom_args: dict :param fmpy_low_memory: use memory map instead of RAM to store the loss array. :type fmpy_low_memory: bool :param fmpy_sort_output: order fmpy output by item_id. :type fmpy_sort_output: bool :param summarypy_low_memory: Pass ``-m`` to ``summarypy`` so it writes a ``.idx`` side-file, consumed by the legacy ktools leccalc/aalcalc binaries. :type summarypy_low_memory: bool :param event_shuffle: Event shuffle rule override. :type event_shuffle: int or None :param gulmc: use the full Monte Carlo gulcalc python version. :type gulmc: bool :param gul_random_generator: random number generator to use in gulmc or gulpy (0: Mersenne-Twister, 1: Latin Hypercube, 2: Latin Hypercube on Philox4x32-7). :type gul_random_generator: int :param gulmc_effective_damageability: use the effective damageability to draw loss samples instead of the full Monte Carlo method. :type gulmc_effective_damageability: bool :param gulmc_vuln_cache_size: size in MB of the cache for the vulnerability calculations. :type gulmc_vuln_cache_size: int :param model_py_server: run the data server for modelpy. :type model_py_server: bool :param peril_filter: list of perils to include (all included if empty). :type peril_filter: list :param join_summary_info: join summary id information to outputcalc csvs. :type join_summary_info: bool :param base_df_engine: The engine to use when loading dataframes. :type base_df_engine: str :param model_df_engine: The engine to use when loading model dataframes. :type model_df_engine: str :param dynamic_footprint: Enable dynamic footprint mode. :type dynamic_footprint: bool :param analysis_pk: analysis primary key reported with progress pings. When set, no local socket-server is started. :type analysis_pk: int or None :param socket_server_size: total number of events, passed to the local socket-server that reports run progress. :type socket_server_size: int or None :param socket_server_port: port the local socket-server listens on. :type socket_server_port: int or None :param log_level: Enable logging of pytools subprocesses. :type log_level: int or None .. py:function:: add_server_call(call, analysis_pk=None, socket_server_port=None) Inject WebSocket/HTTP or socket-server flags into a GUL command string. If ``analysis_pk`` is provided and either the environment variables ``OASIS_WEBSOCKET_URL``/``OASIS_WEBSOCKET_PORT`` (websocket mode) or ``OASIS_ANALYSIS_STATUS_URL`` (HTTP mode) are set, the ``gulmc``/``gulpy`` invocation within *call* is augmented with ``--socket-server`` and ``--analysis-pk`` flags. Otherwise, the ``--socket-server`` flag is set to ``IP:port`` when a port is known. :param call: The full pipeline command string. :type call: str :param analysis_pk: Analysis primary key for WebSocket mode. :type analysis_pk: int or None :param socket_server_port: Pre-checked available port to use. :type socket_server_port: int or None :returns: The (possibly modified) command string. :rtype: str