oasislmf.execution.bash¶
Attributes¶
Functions¶
|
Creates a bash function to check the logs to ensure same number of process started and finsished. |
|
Creates an iterable for all the process ids, if process number is set |
|
Gets the construct model command line argument for the bash script. |
|
Get the ground-up loss calculation command. |
|
|
|
Writes the supplied command to the end of the generated script |
|
Checks if leccalc is enabled in a summaries section |
|
Checks if ORD leccalc is enabled in a summaries section |
|
|
|
Standard name for FIFO |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Add waits to the script |
|
Add pwaits to the script |
|
Add awaits to the script |
|
Add lwaits to the script |
|
Add kwaits to the script |
|
Gets the getmodel ktools command (3.1.0+) Gulcalc item stream |
|
Gets the getmodel ktools command (version < 3.0.8) gulcalc coverage stream |
|
Add a variable to the end of a command in order to track the ID of the process executing it. |
|
Gets the fmcalc ktools command reinsurance stream |
|
Gets the fmcalc ktools command insured losses stream |
|
Gets the command to output ground up losses |
|
|
|
|
|
|
|
Determine whether Post Loss Amplification should be implemented and issue |
|
|
|
|
|
|
|
|
|
Generates a bash script containing ktools calculation instructions for an |
Module Contents¶
- oasislmf.execution.bash.TRAP_FUNC = Multiline-String[source]¶
Show Value
""" touch $LOG_DIR/stderror.err ktools_monitor.sh $$ $LOG_DIR & pid0=$! exit_handler(){ exit_code=$? # disable handler trap - QUIT HUP INT KILL TERM ERR EXIT kill -9 $pid0 2> /dev/null if [ "$exit_code" -gt 0 ]; then # Error - run process clean up echo 'Ktools Run Error - exitcode='$exit_code set +x group_pid=$(ps -p $$ -o pgid --no-headers) sess_pid=$(ps -p $$ -o sess --no-headers) script_pid=$$ printf "Script PID:%d, GPID:%s, SPID:%d " $script_pid $group_pid $sess_pid >> $LOG_DIR/killout.txt ps -jf f -g $sess_pid > $LOG_DIR/subprocess_list PIDS_KILL=$(pgrep -a --pgroup $group_pid | awk 'BEGIN { FS = "[ \t\n]+" }{ if ($1 >= '$script_pid') print}' | grep -v celery | egrep -v *\\.log$ | egrep -v *startup.sh$ | sort -n -r) echo "$PIDS_KILL" >> $LOG_DIR/killout.txt kill -9 $(echo "$PIDS_KILL" | awk 'BEGIN { FS = "[ \t\n]+" }{ print $1 }') 2>/dev/null exit $exit_code else # script successful exit 0 fi } trap exit_handler QUIT HUP INT KILL TERM ERR EXIT"""
- oasislmf.execution.bash.get_check_function(custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None)[source]¶
Creates a bash function to check the logs to ensure same number of process started and finsished.
- Args:
custom_gulcalc_log_start (str): Custom message printed to the logs when a process starts. custom_gulcalc_log_finish (str): Custom message printed to the logs when a process ends.
- oasislmf.execution.bash.BASH_TRACE = Multiline-String[source]¶
Show Value
""" # --- Redirect Bash trace to file --- bash_logging_supported(){ local BASH_VER_MAJOR=${BASH_VERSION:0:1} local BASH_VER_MINOR=${BASH_VERSION:2:1} if [[ "$BASH_VER_MAJOR" -gt 4 ]]; then echo 1; exit fi if [[ $BASH_VER_MAJOR -eq 4 ]] && [[ $BASH_VER_MINOR -gt 3 ]]; then echo 1; exit fi echo 0 } if [ $(bash_logging_supported) == 1 ]; then exec > >(tee -ia $LOG_DIR/bash.log) exec 2> >(tee -ia $LOG_DIR/bash.log >& 2) exec 19> $LOG_DIR/bash.log export BASH_XTRACEFD="19" set -x else echo "WARNING: logging disabled, bash version '$BASH_VERSION' is not supported, minimum requirement is bash v4.4" fi """
- oasislmf.execution.bash.process_range(max_process_id, process_number=None)[source]¶
Creates an iterable for all the process ids, if process number is set then an iterable containing only that number is returned.
This allows for the loss generation to be ran in different processes rather than accross multiple cores.
- Parameters:
max_process_id – The largest process number
process_number – If set iterable only containing this number is returned
- Returns:
iterable containing all the process numbers to process
- oasislmf.execution.bash.get_modelcmd(modelpy: bool, server=False, peril_filter=[]) str [source]¶
Gets the construct model command line argument for the bash script.
- Args:
modelpy: (bool) if the getmodel Python setting is True or not server: (bool) if set then enable ‘TCP’ ipc server/client mode peril_filter: (list) list of perils to include (all included if empty)
Returns: C++ getmodel if modelpy is False, Python getmodel if the modelpy if False
- oasislmf.execution.bash.get_gulcmd(gulpy, gulpy_random_generator, gulmc, gulmc_random_generator, gulmc_effective_damageability, gulmc_vuln_cache_size, modelpy_server, peril_filter, model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False)[source]¶
Get the ground-up loss calculation command.
- Args:
gulpy (bool): if True, return the python command name, else the c++ one.
- Returns:
str: the ground-up loss calculation command
- oasislmf.execution.bash.print_command(command_file, cmd)[source]¶
Writes the supplied command to the end of the generated script
- Parameters:
command_file – File to append command to.
cmd – The command to append
- oasislmf.execution.bash.leccalc_enabled(summary_options)[source]¶
Checks if leccalc is enabled in a summaries section
- Parameters:
summary_options (dict) – Summaies section from an analysis_settings file
Example: {
“aalcalc”: true, “eltcalc”: true, “id”: 1, “lec_output”: true, “leccalc”: {
“full_uncertainty_aep”: true, “full_uncertainty_oep”: true, “return_period_file”: true
}
} :return: True is leccalc is enables, False otherwise.
- oasislmf.execution.bash.ord_enabled(summary_options, ORD_SWITCHES)[source]¶
Checks if ORD leccalc is enabled in a summaries section
- Parameters:
Example: {
“id”: 1, “ord_output”: {
“ept_full_uncertainty_aep”: true, “ept_full_uncertainty_oep”: true, “ept_mean_sample_aep”: true, “ept_mean_sample_oep”: true, “ept_per_sample_mean_aep”: true, “ept_per_sample_mean_oep”: true, “psept_aep”: true, “psept_oep”: true, “return_period_file”: true
}
}
- Returns:
True is leccalc is enables, False otherwise.
- oasislmf.execution.bash.do_post_wait_processing(runtype, analysis_settings, filename, process_counter, work_sub_dir='', output_dir='output/', stderr_guard=True, inuring_priority=None)[source]¶
- oasislmf.execution.bash.get_fifo_name(fifo_dir, producer, producer_id, consumer='')[source]¶
Standard name for FIFO
- oasislmf.execution.bash.do_fifo_exec(producer, producer_id, filename, fifo_dir, action='mkfifo', consumer='')[source]¶
- oasislmf.execution.bash.do_fifos_exec(runtype, max_process_id, filename, fifo_dir, process_number=None, action='mkfifo', consumer='')[source]¶
- oasislmf.execution.bash.do_fifos_exec_full_correlation(runtype, max_process_id, filename, fifo_dir, process_number=None, action='mkfifo')[source]¶
- oasislmf.execution.bash.do_fifos_calc(runtype, analysis_settings, max_process_id, filename, fifo_dir='fifo/', process_number=None, consumer_prefix=None, action='mkfifo')[source]¶
- oasislmf.execution.bash.create_workfolders(runtype, analysis_settings, filename, work_dir='work/', inuring_priority=None)[source]¶
- oasislmf.execution.bash.do_kats(runtype, analysis_settings, max_process_id, filename, process_counter, work_dir='work/kat/', output_dir='output/', sort_by_event=False, process_number=None, inuring_priority=None)[source]¶
- oasislmf.execution.bash.do_summarycalcs(runtype, analysis_settings, process_id, filename, summarypy, fifo_dir='fifo/', stderr_guard=True, num_reinsurance_iterations=0, gul_legacy_stream=None, gul_full_correlation=False, inuring_priority=None)[source]¶
- oasislmf.execution.bash.do_tees(runtype, analysis_settings, process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', inuring_priority=None)[source]¶
- oasislmf.execution.bash.do_tees_fc_sumcalc_fmcalc(process_id, filename, correlated_output_stems)[source]¶
- oasislmf.execution.bash.do_ord(runtype, analysis_settings, process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, inuring_priority=None)[source]¶
- oasislmf.execution.bash.do_any(runtype, analysis_settings, process_id, filename, process_counter, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, inuring_priority=None)[source]¶
- oasislmf.execution.bash.get_ri_inuring_priorities(analysis_settings, num_reinsurance_iterations)[source]¶
- oasislmf.execution.bash.rl(analysis_settings, max_process_id, filename, process_counter, num_reinsurance_iterations, summarypy, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None)[source]¶
- oasislmf.execution.bash.ri(analysis_settings, max_process_id, filename, process_counter, num_reinsurance_iterations, summarypy, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None)[source]¶
- oasislmf.execution.bash.il(analysis_settings, max_process_id, filename, process_counter, summarypy, fifo_dir='fifo/', work_dir='work/', stderr_guard=True, process_number=None)[source]¶
- oasislmf.execution.bash.do_gul(analysis_settings, max_process_id, filename, process_counter, summarypy, fifo_dir='fifo/', work_dir='work/', gul_legacy_stream=None, stderr_guard=True, process_number=None)[source]¶
- oasislmf.execution.bash.do_gul_full_correlation(analysis_settings, max_process_id, filename, process_counter, fifo_dir='fifo/full_correlation/', work_dir='work/full_correlation/', gul_legacy_stream=None, stderr_guard=None, process_number=None)[source]¶
- oasislmf.execution.bash.do_waits(wait_variable, wait_count, filename)[source]¶
Add waits to the script
- oasislmf.execution.bash.get_getmodel_itm_cmd(number_of_samples, gul_threshold, use_random_number_file, gul_alloc_rule, item_output, process_id, max_process_id, correlated_output, eve_shuffle_flag, modelpy=False, modelpy_server=False, peril_filter=[], gulpy=False, gulpy_random_generator=1, gulmc=False, gulmc_random_generator=1, gulmc_effective_damageability=False, gulmc_vuln_cache_size=200, model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False, **kwargs)[source]¶
Gets the getmodel ktools command (3.1.0+) Gulcalc item stream :param number_of_samples: The number of samples to run :type number_of_samples: int :param gul_threshold: The GUL threshold to use :type gul_threshold: float :param use_random_number_file: flag to use the random number file :type use_random_number_file: bool :param gul_alloc_rule: back allocation rule for gulcalc :type gul_alloc_rule: int :param item_output: The item output :type item_output: str :param eve_shuffle_flag: The event shuffling rule :type eve_shuffle_flag: str :param model_df_engine: The engine to use when loading dataframes :type model_df_engine: str :return: The generated getmodel command
- oasislmf.execution.bash.get_getmodel_cov_cmd(number_of_samples, gul_threshold, use_random_number_file, coverage_output, item_output, process_id, max_process_id, eve_shuffle_flag, modelpy=False, modelpy_server=False, peril_filter=[], gulpy=False, gulpy_random_generator=1, gulmc=False, gulmc_random_generator=1, gulmc_effective_damageability=False, gulmc_vuln_cache_size=200, model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False, **kwargs) str [source]¶
Gets the getmodel ktools command (version < 3.0.8) gulcalc coverage stream :param number_of_samples: The number of samples to run :type number_of_samples: int :param gul_threshold: The GUL threshold to use :type gul_threshold: float :param use_random_number_file: flag to use the random number file :type use_random_number_file: bool :param coverage_output: The coverage output :type coverage_output: str :param item_output: The item output :type item_output: str :param eve_shuffle_flag: The event shuffling rule :type eve_shuffle_flag: str :param df_engine: The engine to use when loading dataframes :type df_engine: str :return: (str) The generated getmodel command
- oasislmf.execution.bash.add_pid_to_shell_command(cmd, process_counter)[source]¶
Add a variable to the end of a command in order to track the ID of the process executing it. Each time this function is called, the counter process_counter is incremented.
- Args:
cmd (str): the command whose process ID is to be stored in a variable. process_counter (Counter or dict): the number of process IDs that are being tracked.
- Returns:
cmd (str): the updated command string.
- oasislmf.execution.bash.get_main_cmd_ri_stream(cmd, process_id, il_output, il_alloc_rule, ri_alloc_rule, num_reinsurance_iterations, fifo_dir='fifo/', stderr_guard=True, from_file=False, fmpy=True, fmpy_low_memory=False, fmpy_sort_output=False, step_flag='', process_counter=None, ri_inuring_priorities=None, rl_inuring_priorities=None)[source]¶
Gets the fmcalc ktools command reinsurance stream :param cmd: either gulcalc command stream or correlated output file :type cmd: str :param process_id: ID corresponding to thread :type process_id: int :param il_output: If insured loss outputs required :type il_output: Boolean :param il_alloc_rule: insured loss allocation rule for fmcalc :type il_alloc_rule: int :param ri_alloc_rule: reinsurance allocation rule for fmcalc :type ri_alloc_rule: int :param num_reinsurance_iterations: number of reinsurance iterations :type num_reinsurance_iterations: int :param fifo_dir: path to fifo directory :type fifo_dir: str :param stderr_guard: send stderr output to log file :type stderr_guard: bool :param from_file: must be true if cmd is a file and false if it can be piped :type from_file: bool :param ri_inuring_priorities: Inuring priorities where net output has been requested :type ri_inuring_priorities: dict :param rl_inuring_priorities: Inuring priorities where gross output has been requested :type rl_inuring_priorities: dict
- oasislmf.execution.bash.get_main_cmd_il_stream(cmd, process_id, il_alloc_rule, fifo_dir='fifo/', stderr_guard=True, from_file=False, fmpy=True, fmpy_low_memory=False, fmpy_sort_output=False, step_flag='', process_counter=None)[source]¶
Gets the fmcalc ktools command insured losses stream :param cmd: either gulcalc command stream or correlated output file :type cmd: str :param process_id: ID corresponding to thread :type process_id: int :param il_alloc_rule: insured loss allocation rule for fmcalc :type il_alloc_rule: int :param fifo_dir: path to fifo directory :type fifo_dir: str :param stderr_guard: send stderr output to log file :type stderr_guard: bool :param from_file: must be true if cmd is a file and false if it can be piped :type from_file: bool :return: generated fmcalc command as str
- oasislmf.execution.bash.get_main_cmd_gul_stream(cmd, process_id, fifo_dir='fifo/', stderr_guard=True, consumer='', process_counter=None)[source]¶
Gets the command to output ground up losses :param cmd: either gulcalc command stream or correlated output file :type cmd: str :param process_id: ID corresponding to thread :type process_id: int :param fifo_dir: path to fifo directory :type fifo_dir: str :param stderr_guard: send stderr output to log file :type stderr_guard: bool :param consumer: optional name of the consumer of the stream :type consumer: string :return: generated command as str
- oasislmf.execution.bash.get_main_cmd_lb(num_lb, num_in_per_lb, num_out_per_lb, get_input_stream_name, get_output_stream_name, stderr_guard)[source]¶
- oasislmf.execution.bash.get_pla_cmd(pla, secondary_factor, uniform_factor)[source]¶
Determine whether Post Loss Amplification should be implemented and issue plapy command.
- Args:
pla (bool): flag to apply post loss amplification secondary_factor (float): secondary factor to apply to post loss
amplification
uniform_factor (float): uniform factor to apply across all losses
- Returns:
pla_cmd (str): post loss amplification command
- oasislmf.execution.bash.bash_params(analysis_settings, max_process_id=-1, number_of_processes=-1, num_reinsurance_iterations=0, model_storage_json=None, fifo_tmp_dir=True, gul_alloc_rule=None, il_alloc_rule=None, ri_alloc_rule=None, num_gul_per_lb=None, num_fm_per_lb=None, stderr_guard=True, gul_legacy_stream=False, bash_trace=False, filename='run_kools.sh', _get_getmodel_cmd=None, custom_gulcalc_cmd=None, custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None, custom_args={}, fmpy=True, fmpy_low_memory=False, fmpy_sort_output=False, event_shuffle=None, modelpy=False, gulpy=False, gulpy_random_generator=1, gulmc=False, gulmc_random_generator=1, gulmc_effective_damageability=False, gulmc_vuln_cache_size=200, process_number=None, remove_working_files=True, model_run_dir='', model_py_server=False, summarypy=False, peril_filter=[], exposure_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False, **kwargs)[source]¶
- oasislmf.execution.bash.bash_wrapper(filename, bash_trace, stderr_guard, log_sub_dir=None, process_number=None, custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None)[source]¶
- oasislmf.execution.bash.create_bash_analysis(process_counter, max_process_id, num_reinsurance_iterations, fifo_tmp_dir, bash_trace, filename, _get_getmodel_cmd, custom_args, fmpy, fmpy_low_memory, fmpy_sort_output, process_number, remove_working_files, model_run_dir, fifo_queue_dir, fifo_full_correlation_dir, stderr_guard, gul_item_stream, work_dir, work_kat_dir, work_full_correlation_dir, work_full_correlation_kat_dir, output_dir, output_full_correlation_dir, gul_alloc_rule, il_alloc_rule, ri_alloc_rule, num_gul_per_lb, num_fm_per_lb, event_shuffle, eve_shuffle_flag, kat_sort_by_event, gul_threshold, number_of_samples, use_random_number_file, full_correlation, gul_output, il_output, ri_output, rl_output, need_summary_fifo_for_gul, analysis_settings, modelpy, gulpy, gulpy_random_generator, gulmc, gulmc_random_generator, gulmc_effective_damageability, gulmc_vuln_cache_size, model_py_server, peril_filter, summarypy, gul_legacy_stream=False, model_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', dynamic_footprint=False, **kwargs)[source]¶
- oasislmf.execution.bash.create_bash_outputs(process_counter, fifo_tmp_dir, filename, remove_working_files, fifo_queue_dir, stderr_guard, work_dir, work_full_correlation_dir, output_dir, output_full_correlation_dir, full_correlation, gul_output, il_output, ri_output, rl_output, analysis_settings, num_reinsurance_iterations, num_gul_per_lb, num_fm_per_lb, max_process_id, work_kat_dir, kat_sort_by_event, gul_item_stream, work_full_correlation_kat_dir, **kwargs)[source]¶
- oasislmf.execution.bash.genbash(max_process_id, analysis_settings, num_reinsurance_iterations=0, fifo_tmp_dir=True, gul_alloc_rule=None, il_alloc_rule=None, ri_alloc_rule=None, num_gul_per_lb=None, num_fm_per_lb=None, stderr_guard=True, gul_legacy_stream=False, bash_trace=False, filename='run_kools.sh', _get_getmodel_cmd=None, custom_gulcalc_log_start=None, custom_gulcalc_log_finish=None, custom_args={}, fmpy=True, fmpy_low_memory=False, fmpy_sort_output=False, event_shuffle=None, modelpy=False, gulpy=False, gulpy_random_generator=1, gulmc=False, gulmc_random_generator=1, gulmc_effective_damageability=False, gulmc_vuln_cache_size=200, model_py_server=False, peril_filter=[], summarypy=False, base_df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', model_df_engine=None, dynamic_footprint=False)[source]¶
Generates a bash script containing ktools calculation instructions for an Oasis model.
- Parameters:
max_process_id (int) – The number of processes to create
analysis_settings (dict) – The analysis settings
filename (string) – The output file name
num_reinsurance_iterations (int) – The number of reinsurance iterations
fifo_tmp_dir (boolean) – When set to True, Create and use FIFO quese in /tmp/[A-Z,0-9]/fifo, if False run in ‘./fifo’
gul_alloc_rule (Int) – Allocation rule (None or 1) for gulcalc, if not set default to coverage stream
il_alloc_rule (Int) – Allocation rule (0, 1 or 2) for fmcalc
ri_alloc_rule (Int) – Allocation rule (0, 1 or 2) for fmcalc
num_gul_in_calc_block (Int) – number of gul in calc block
num_fm_in_calc_block (Int) – number of gul in calc block
get_getmodel_cmd (callable) – Method for getting the getmodel command, by default
GenerateLossesCmd.get_getmodel_cmd
is used.base_df_engine (str) – The engine to use when loading dataframes.
model_df_engine (str) – The engine to use when loading model dataframes.