oasislmf.pytools.getmodel.manager¶

This file is the entry point for the python get model command for the package

TODO: use selector and select for output

Attributes¶

`logger`
`buff_size`
`buff_int_size`
`areaperil_int_relative_size`
`oasis_float_relative_size`
`results_relative_size`
`VulnerabilityIndex_dtype`
`VulnerabilityIndex`
`VulnerabilityRow_dtype`
`VulnerabilityRow`
`vuln_offset`
`load_areaperil_id`

Functions¶

`load_areaperil_id_u4`(int32_mv, cursor, areaperil_id)
`load_areaperil_id_u8`(int32_mv, cursor, areaperil_id)
`load_items`(items)	Processes pre-sorted, pre-filtered items extracting vulnerability metadata.
`get_items`(input_path[, ignore_file_type, ...])	Loads the items from the items file.
`get_intensity_bin_dict`(input_path)	Loads the intensity bin dictionary file and creates arrays to map intensities to bins.
`encode_peril_id`(peril_id)	Encode a string to an integer.
`get_intensity_adjustment`(input_path)
`load_vuln_probability`(vuln_array, vuln, vuln_id)
`load_vulns_bin_idx`(vulns_bin, vulns_idx_bin, vuln_map, ...)	Loads the vulnerability binary index file.
`load_vulns_bin_idx_adjusted`(vulns_bin, vulns_idx_bin, ...)	Loads the vulnerability binary index file, prioritizing the data in the adjustments file over the data in the
`load_vulns_bin`(vulns_bin, vuln_map, vuln_map_keys, ...)	Loads the vulnerability data grouped by the intensity and damage bins.
`load_vulns_bin_adjusted`(vulns_bin, vuln_map, ...[, ...])	Loads the vulnerability data grouped by the intensity and damage bins, prioritizing the data
`update_vuln_array_with_adj_data`(vuln_array, vuln_map, ...)	Update the vulnerability array with adjustment data (used for parquet loading).
`get_vulns`(storage, run_dir, vuln_map, vuln_map_keys, ...)	Loads the vulnerabilities from the file.
`get_vulnerability_replacements`(run_dir, vuln_ids_set)	Loads the vulnerability adjustment file.
`get_mean_damage_bins`(storage[, ignore_file_type])	Loads the mean damage bins from the damage_bin_dict file, namely, the interpolation value for each bin.
`get_damage_bins`(storage[, ignore_file_type])	Loads the damage bins from the damage_bin_dict file.
`damage_bin_prob`(p, intensities_min, intensities_max, ...)	Calculate the probability of an event happening and then causing damage.
`do_result`(vulns_id, vuln_array, mean_damage_bins, ...)	Calculate the result concerning an event ID.
`doCdf`(event_id, num_intensity_bins, footprint, ...)	Calculates the cumulative distribution function (cdf) for an event ID.
`convert_vuln_id_to_index`(vuln_map, vuln_map_keys, ...)
`run`(run_dir, file_in, file_out, ignore_file_type, ...)	Runs the main process of the getmodel process.

Module Contents¶

oasislmf.pytools.getmodel.manager.logger[source]¶

oasislmf.pytools.getmodel.manager.buff_size = 65536[source]¶

oasislmf.pytools.getmodel.manager.buff_int_size[source]¶

oasislmf.pytools.getmodel.manager.areaperil_int_relative_size[source]¶

oasislmf.pytools.getmodel.manager.oasis_float_relative_size[source]¶

oasislmf.pytools.getmodel.manager.results_relative_size[source]¶

oasislmf.pytools.getmodel.manager.VulnerabilityIndex_dtype[source]¶

oasislmf.pytools.getmodel.manager.VulnerabilityIndex[source]¶

oasislmf.pytools.getmodel.manager.VulnerabilityRow_dtype[source]¶

oasislmf.pytools.getmodel.manager.VulnerabilityRow[source]¶

oasislmf.pytools.getmodel.manager.vuln_offset = 4[source]¶

oasislmf.pytools.getmodel.manager.load_areaperil_id_u4(int32_mv, cursor, areaperil_id)[source]¶

oasislmf.pytools.getmodel.manager.load_areaperil_id_u8(int32_mv, cursor, areaperil_id)[source]¶

oasislmf.pytools.getmodel.manager.load_areaperil_id[source]¶

oasislmf.pytools.getmodel.manager.load_items(items)[source]¶

Processes pre-sorted, pre-filtered items extracting vulnerability metadata.

Items must be sorted by (areaperil_id, vulnerability_id) before calling.

Parameters:

items (np.ndarray[items_dtype]) – sorted items array

Returns:

vuln_map (packed: hashmap), vuln_map_keys, unique_areaperil_ids (sorted), areaperil_to_vulns_idx_array, areaperil_to_vulns

Return type:

Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]

oasislmf.pytools.getmodel.manager.get_items(input_path, ignore_file_type=set(), valid_area_peril_id=None)[source]¶

Loads the items from the items file.

Parameters:

input_path (str) – the path pointing to the file
ignore_file_type (set(str)) – file extension to ignore when loading
valid_area_peril_id (np.ndarray) – array of area_peril_id to include (if None, all are included)

Returns:

vuln_map: (packed hashmap), vuln_map_keys, areaperil_id_ind (id_index), areaperil_to_vulns_idx_array, areaperil_to_vulns, unique_areaperil_ids

Return type:

Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]

oasislmf.pytools.getmodel.manager.get_intensity_bin_dict(input_path)[source]¶

Loads the intensity bin dictionary file and creates arrays to map intensities to bins. Used in the dynamic footprint generation as intensities can be adjusted for defences at runtime.

Parameters:

input_path (str) – the path pointing to the file

Returns:

intensity_bin_peril_ids, a 1-d array of: unique encoded peril_ids (length n_perils), and intensity_bins, a 2-d array of shape (n_perils, max_intensity + 1) mapping [peril_idx, intensity_value] -> intensity_bin_id. Slots not present in the CSV are pre-filled with the fallback bin for intensity=0 of that peril.

Return type:

Tuple[np.array[int32], np.array[int32, 2d]]

oasislmf.pytools.getmodel.manager.encode_peril_id(peril_id)[source]¶

Encode a string to an integer.

Parameters:: peril_id (str) – 3-digit Oasis peril code (also works with numeric codes).
Returns:: The encoded peril_id.
Return type:: int

oasislmf.pytools.getmodel.manager.get_intensity_adjustment(input_path)[source]¶

oasislmf.pytools.getmodel.manager.load_vuln_probability(vuln_array, vuln, vuln_id)[source]¶

oasislmf.pytools.getmodel.manager.load_vulns_bin_idx(vulns_bin, vulns_idx_bin, vuln_map, vuln_map_keys, num_damage_bins, num_intensity_bins, rowsize)[source]¶

Loads the vulnerability binary index file.

Parameters:

vulns_bin (List[VulnerabilityRow]) – vulnerability data from the vulnerability file
vulns_idx_bin (List[VulnerabilityIndex]) – vulnerability index data from the vulnerability idx file
vuln_map (np.ndarray[uint8]) – packed hashmap table mapping vuln_id to dense index
vuln_map_keys (np.ndarray[int32]) – array of unique vulnerability ids (hashmap keys)
num_damage_bins (int) – number of damage bins in the data
num_intensity_bins (int) – the number of intensity bins
rowsize (int) – size in bytes of one vulnerability row, used to turn the byte offsets held in the idx file into row indices

Returns:

vulnerability data grouped by intensity bin and damage bin,: and the vulnerability id held at each dense index

Return type:

Tuple[np.ndarray, np.ndarray]

oasislmf.pytools.getmodel.manager.load_vulns_bin_idx_adjusted(vulns_bin, vulns_idx_bin, vuln_map, vuln_map_keys, num_damage_bins, num_intensity_bins, rowsize, adj_vuln_data=None)[source]¶

Loads the vulnerability binary index file, prioritizing the data in the adjustments file over the data in the vulnerability file.

Parameters:

vulns_bin (List[VulnerabilityRow]) – vulnerability data from the vulnerability file
vulns_idx_bin (List[VulnerabilityIndex]) – vulnerability index data from the vulnerability idx file
vuln_map (np.ndarray[uint8]) – packed hashmap table mapping vuln_id to dense index
vuln_map_keys (np.ndarray[int32]) – array of unique vulnerability ids (hashmap keys)
num_damage_bins (int) – number of damage bins in the data
num_intensity_bins (int) – the number of intensity bins
rowsize (int) – size in bytes of one vulnerability row, used to turn the byte offsets held in the idx file into row indices
adj_vuln_data (List[vulnerability_dtype]) – vulnerability adjustment data, sorted by vuln_id

Returns:

vulnerability data grouped by intensity bin and damage bin,: and the vulnerability id held at each dense index

Return type:

Tuple[np.ndarray, np.ndarray]

oasislmf.pytools.getmodel.manager.load_vulns_bin(vulns_bin, vuln_map, vuln_map_keys, num_damage_bins, num_intensity_bins)[source]¶

Loads the vulnerability data grouped by the intensity and damage bins.

Parameters:

vulns_bin (List[Vulnerability]) – vulnerability data from the vulnerability file
vuln_map (np.ndarray[uint8]) – packed hashmap table mapping vuln_id to dense index
vuln_map_keys (np.ndarray[int32]) – array of unique vulnerability ids (hashmap keys)
num_damage_bins (int) – number of damage bins in the data
num_intensity_bins (int) – the number of intensity bins

Returns:

vulnerability data grouped by intensity bin and damage bin,: and the vulnerability id held at each dense index

Return type:

Tuple[np.ndarray, np.ndarray]

oasislmf.pytools.getmodel.manager.load_vulns_bin_adjusted(vulns_bin, vuln_map, vuln_map_keys, num_damage_bins, num_intensity_bins, adj_vuln_data=None)[source]¶

Loads the vulnerability data grouped by the intensity and damage bins, prioritizing the data in the adjustments file over the data in the vulnerability file.

Parameters:

vulns_bin (List[Vulnerability]) – vulnerability data from the vulnerability file
vuln_map (np.ndarray[uint8]) – packed hashmap table mapping vuln_id to dense index
vuln_map_keys (np.ndarray[int32]) – array of unique vulnerability ids (hashmap keys)
num_damage_bins (int) – number of damage bins in the data
num_intensity_bins (int) – the number of intensity bins
adj_vuln_data (List[vulnerability_dtype]) – vulnerability adjustment data, sorted by vuln_id

Returns:

vulnerability data grouped by intensity bin and damage bin,: and the vulnerability id held at each dense index

Return type:

Tuple[np.ndarray, np.ndarray]

oasislmf.pytools.getmodel.manager.update_vuln_array_with_adj_data(vuln_array, vuln_map, vuln_map_keys, adj_vuln_data)[source]¶

Update the vulnerability array with adjustment data (used for parquet loading).

Parameters:

vuln_array (3D array) – The vulnerability data array.
vuln_map (np.ndarray[uint8]) – packed hashmap table mapping vuln_id to dense index.
vuln_map_keys (np.ndarray[int32]) – array of unique vulnerability ids (hashmap keys).
adj_vuln_data (List[vulnerability_dtype]) – The vulnerability adjustment data.

Returns:

The updated vulnerability data array.

Return type:

3D array

oasislmf.pytools.getmodel.manager.get_vulns(storage: oasis_data_manager.filestore.backends.base.BaseStorage, run_dir, vuln_map, vuln_map_keys, num_intensity_bins, ignore_file_type=set(), df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader')[source]¶

Loads the vulnerabilities from the file.

Parameters:

storage (BaseStorage) – the storage manager for fetching model data
run_dir (str) – the path to the run folder (used to load the analysis settings)
vuln_map (np.ndarray[uint8]) – packed hashmap table mapping vuln_id to dense index
vuln_map_keys (np.ndarray[int32]) – array of unique vulnerability ids (hashmap keys)
num_intensity_bins (int) – the number of intensity bins
ignore_file_type (set(str)) – file extension to ignore when loading
df_engine (str) – the engine to use when loading dataframes

Returns:

vulnerability data, vulnerabilities id,: number of damage bins

Return type:

Tuple[List[List[float]], np.array[int], int]

oasislmf.pytools.getmodel.manager.get_vulnerability_replacements(run_dir, vuln_ids_set)[source]¶

Loads the vulnerability adjustment file.

Parameters:

run_dir (str) – the path pointing to the run directory
vuln_ids_set (set) – set of vulnerability IDs to filter by

Returns:

vulnerability replacement data, or None when the run has no: valid vulnerability_adjustments entry

Return type:

List[vulnerability_dtype]

oasislmf.pytools.getmodel.manager.get_mean_damage_bins(storage: oasis_data_manager.filestore.backends.base.BaseStorage, ignore_file_type=set())[source]¶

Loads the mean damage bins from the damage_bin_dict file, namely, the interpolation value for each bin.

Parameters:

storage (BaseStorage) – the storage connector for fetching the model data
ignore_file_type (set(str)) – file extension to ignore when loading

Returns:

the interpolation column of the damage_bin_dict file

Return type:

List[Union[damagebindictionary]]

oasislmf.pytools.getmodel.manager.get_damage_bins(storage: oasis_data_manager.filestore.backends.base.BaseStorage, ignore_file_type=set())[source]¶

Loads the damage bins from the damage_bin_dict file.

Parameters:

storage (BaseStorage) – the storage connector for fetching the model data
ignore_file_type (set(str)) – file extension to ignore when loading

Returns:

loaded data from the damage_bin_dict file

Return type:

List[Union[damagebindictionary]]

oasislmf.pytools.getmodel.manager.damage_bin_prob(p, intensities_min, intensities_max, vulns, intensities)[source]¶

Calculate the probability of an event happening and then causing damage. Note: vulns is a 1-d array containing 1 damage bin of the damage probability distribution as a function of hazard intensity.

Parameters:

p (float) – the probability to be updated
intensities_min (int) – minimum intensity bin id
intensities_max (int) – maximum intensity bin id
vulns (List[float]) – slice of damage probability distribution given hazard intensity
intensities (List[float]) – intensity probability distribution

Returns:

the updated probability

Return type:

float

oasislmf.pytools.getmodel.manager.do_result(vulns_id, vuln_array, mean_damage_bins, int32_mv, num_damage_bins, intensities_min, intensities_max, intensities, event_id, areaperil_id, vuln_i, cursor)[source]¶

Calculate the result concerning an event ID.

Writes one cdf record into the output buffer as event_id, areaperil_id, vulnerability_id, number of damage bins, then a (prob_to, bin_mean) pair per damage bin. Bins stop being written once the cumulative probability reaches 1.

Parameters:

vulns_id (List[int]) – list of vulnerability IDs
vuln_array (List[List[list]]) – list of vulnerabilities and their data
mean_damage_bins (List[float]) – the mean of each damage bin (len(mean_damage_bins) == num_damage_bins)
int32_mv (np.ndarray[int32]) – int32 view of the output buffer the record is written into
num_damage_bins (int) – number of damage bins in the data
intensities_min (int) – minimum intensity bin id
intensities_max (int) – maximum intensity bin id
intensities (List[float]) – intensity probability distribution
event_id (int) – the event ID that concerns the result being calculated
areaperil_id (List[int]) – the areaperil ID that concerns the result being calculated
vuln_i (int) – the index concerning the vulnerability inside the vuln_array
cursor (int) – index in int32_mv (in int32 words) where the record is written

Returns:

the cursor position (in int32 words) just past the record written

Return type:

int

oasislmf.pytools.getmodel.manager.doCdf(event_id, num_intensity_bins, footprint, areaperil_id_ind, areaperil_to_vulns_idx_array, areaperil_to_vulns, vuln_array, vulns_id, num_damage_bins, mean_damage_bins, int32_mv, max_result_relative_size)[source]¶

Calculates the cumulative distribution function (cdf) for an event ID.

Parameters:

event_id (int) – the event ID the the CDF is being calculated to.
num_intensity_bins (int) – the number of intensity bins for the CDF
footprint (List[Tuple[int, int, float]]) – information about the footprint with event_id, areaperil_id, probability
areaperil_id_ind (np.array) – id_index structure mapping areaperil_id to dense index
areaperil_to_vulns_idx_array (List[Tuple[int, int]]) – the index where the areaperil ID starts and finishes
areaperil_to_vulns (List[int]) – maps the areaperil ID to the vulnerability ID
vuln_array (np.ndarray[oasis_float]) – damage probabilities of shape (n_vulnerabilities, num_damage_bins, num_intensity_bins)
vulns_id (List[int]) – list of vulnerability IDs
num_damage_bins (int) – number of damage bins in the data
mean_damage_bins (List[float]) – the mean of each damage bin (len(mean_damage_bins) == num_damage_bins)
int32_mv (np.ndarray[int32]) – int32 view of the output buffer the cdf records are written into
max_result_relative_size (int) – the maximum result size

Yields:

int –

the number of bytes written into the buffer, emitted whenever the buffer is full and: once more at the end of the event (0 when the event produced no result)

oasislmf.pytools.getmodel.manager.convert_vuln_id_to_index(vuln_map, vuln_map_keys, areaperil_to_vulns)[source]¶

oasislmf.pytools.getmodel.manager.run(run_dir, file_in, file_out, ignore_file_type, data_server, peril_filter, df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', analysis_pk=None)[source]¶

Runs the main process of the getmodel process.

Parameters:

run_dir (str) – the directory of where the process is running
file_in (Optional[str]) – the path to the input directory
file_out (Optional[str]) – the path to the output directory
ignore_file_type (set(str)) – file extension to ignore when loading
data_server (bool) – if set to True runs the data server
peril_filter (list[int]) – list of perils to include in the computation (if None, all perils will be included).
df_engine (str) – The engine to use when loading dataframes
analysis_pk (int) – the analysis primary key reported alongside the completion ping