oasislmf.pytools.getmodel.manager ================================= .. py:module:: oasislmf.pytools.getmodel.manager .. autoapi-nested-parse:: This file is the entry point for the python get model command for the package TODO: use selector and select for output Attributes ---------- .. autoapisummary:: oasislmf.pytools.getmodel.manager.logger oasislmf.pytools.getmodel.manager.buff_size oasislmf.pytools.getmodel.manager.buff_int_size oasislmf.pytools.getmodel.manager.areaperil_int_relative_size oasislmf.pytools.getmodel.manager.oasis_float_relative_size oasislmf.pytools.getmodel.manager.results_relative_size oasislmf.pytools.getmodel.manager.VulnerabilityIndex_dtype oasislmf.pytools.getmodel.manager.VulnerabilityIndex oasislmf.pytools.getmodel.manager.VulnerabilityRow_dtype oasislmf.pytools.getmodel.manager.VulnerabilityRow oasislmf.pytools.getmodel.manager.vuln_offset oasislmf.pytools.getmodel.manager.load_areaperil_id Functions --------- .. autoapisummary:: oasislmf.pytools.getmodel.manager.load_areaperil_id_u4 oasislmf.pytools.getmodel.manager.load_areaperil_id_u8 oasislmf.pytools.getmodel.manager.load_items oasislmf.pytools.getmodel.manager.get_items oasislmf.pytools.getmodel.manager.get_intensity_bin_dict oasislmf.pytools.getmodel.manager.encode_peril_id oasislmf.pytools.getmodel.manager.get_intensity_adjustment oasislmf.pytools.getmodel.manager.load_vuln_probability oasislmf.pytools.getmodel.manager.load_vulns_bin_idx oasislmf.pytools.getmodel.manager.load_vulns_bin_idx_adjusted oasislmf.pytools.getmodel.manager.load_vulns_bin oasislmf.pytools.getmodel.manager.load_vulns_bin_adjusted oasislmf.pytools.getmodel.manager.update_vuln_array_with_adj_data oasislmf.pytools.getmodel.manager.get_vulns oasislmf.pytools.getmodel.manager.get_vulnerability_replacements oasislmf.pytools.getmodel.manager.get_mean_damage_bins oasislmf.pytools.getmodel.manager.get_damage_bins oasislmf.pytools.getmodel.manager.damage_bin_prob oasislmf.pytools.getmodel.manager.do_result oasislmf.pytools.getmodel.manager.doCdf oasislmf.pytools.getmodel.manager.convert_vuln_id_to_index oasislmf.pytools.getmodel.manager.run Module Contents --------------- .. py:data:: logger .. py:data:: buff_size :value: 65536 .. py:data:: buff_int_size .. py:data:: areaperil_int_relative_size .. py:data:: oasis_float_relative_size .. py:data:: results_relative_size .. py:data:: VulnerabilityIndex_dtype .. py:data:: VulnerabilityIndex .. py:data:: VulnerabilityRow_dtype .. py:data:: VulnerabilityRow .. py:data:: vuln_offset :value: 4 .. py:function:: load_areaperil_id_u4(int32_mv, cursor, areaperil_id) .. py:function:: load_areaperil_id_u8(int32_mv, cursor, areaperil_id) .. py:data:: load_areaperil_id .. py:function:: load_items(items) Processes pre-sorted, pre-filtered items extracting vulnerability metadata. Items must be sorted by (areaperil_id, vulnerability_id) before calling. Args: items: (np.ndarray[items_dtype]) sorted items array Returns: (Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]) vuln_map (packed hashmap), vuln_map_keys, unique_areaperil_ids (sorted), areaperil_to_vulns_idx_array, areaperil_to_vulns .. py:function:: get_items(input_path, ignore_file_type=set(), valid_area_peril_id=None) Loads the items from the items file. Args: input_path: (str) the path pointing to the file ignore_file_type: set(str) file extension to ignore when loading valid_area_peril_id: array of area_peril_id to include (if None, all are included) Returns: (Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]) vuln_map (packed hashmap), vuln_map_keys, areaperil_id_ind (id_index), areaperil_to_vulns_idx_array, areaperil_to_vulns, unique_areaperil_ids .. py:function:: get_intensity_bin_dict(input_path) Loads the intensity bin dictionary file and creates arrays to map intensities to bins. Used in the dynamic footprint generation as intensities can be adjusted for defences at runtime. Args: input_path: (str) the path pointing to the file Returns: (np.array[int32], np.array[int32, 2d]) intensity_bin_peril_ids: 1-d array of unique encoded peril_ids (length n_perils). intensity_bins: 2-d array of shape (n_perils, max_intensity + 1) mapping [peril_idx, intensity_value] -> intensity_bin_id. Slots not present in the CSV are pre-filled with the fallback bin for intensity=0 of that peril. .. py:function:: encode_peril_id(peril_id) Encode a string to an integer. Args: peril_id (str): 3-digit Oasis peril code (also works with numeric codes). Returns: int: The encoded peril_id. .. py:function:: get_intensity_adjustment(input_path) .. py:function:: load_vuln_probability(vuln_array, vuln, vuln_id) .. py:function:: load_vulns_bin_idx(vulns_bin, vulns_idx_bin, vuln_map, vuln_map_keys, num_damage_bins, num_intensity_bins, rowsize) Loads the vulnerability binary index file. Args: vulns_bin: (List[VulnerabilityRow]) vulnerability data from the vulnerability file vulns_idx_bin: (List[VulnerabilityIndex]) vulnerability index data from the vulnerability idx file vuln_map: (np.ndarray[uint8]) packed hashmap table mapping vuln_id to dense index vuln_map_keys: (np.ndarray[int32]) array of unique vulnerability ids (hashmap keys) num_damage_bins: (int) number of damage bins in the data num_intensity_bins: (int) the number of intensity bins Returns: (List[List[List[floats]]]) vulnerability data grouped by intensity bin and damage bin .. py:function:: load_vulns_bin_idx_adjusted(vulns_bin, vulns_idx_bin, vuln_map, vuln_map_keys, num_damage_bins, num_intensity_bins, rowsize, adj_vuln_data=None) Loads the vulnerability binary index file, prioritizing the data in the adjustments file over the data in the vulnerability file. Args: vulns_bin: (List[VulnerabilityRow]) vulnerability data from the vulnerability file vulns_idx_bin: (List[VulnerabilityIndex]) vulnerability index data from the vulnerability idx file vuln_map: (np.ndarray[uint8]) packed hashmap table mapping vuln_id to dense index vuln_map_keys: (np.ndarray[int32]) array of unique vulnerability ids (hashmap keys) num_damage_bins: (int) number of damage bins in the data num_intensity_bins: (int) the number of intensity bins adj_vuln_data: (List[vulnerability_dtype]) vulnerability adjustment data, sorted by vuln_id Returns: (List[List[List[floats]]]) vulnerability data grouped by intensity bin and damage bin .. py:function:: load_vulns_bin(vulns_bin, vuln_map, vuln_map_keys, num_damage_bins, num_intensity_bins) Loads the vulnerability data grouped by the intensity and damage bins. Args: vulns_bin: (List[Vulnerability]) vulnerability data from the vulnerability file vuln_map: (np.ndarray[uint8]) packed hashmap table mapping vuln_id to dense index vuln_map_keys: (np.ndarray[int32]) array of unique vulnerability ids (hashmap keys) num_damage_bins: (int) number of damage bins in the data num_intensity_bins: (int) the number of intensity bins Returns: (List[List[List[floats]]]) vulnerability data grouped by intensity bin and damage bin .. py:function:: load_vulns_bin_adjusted(vulns_bin, vuln_map, vuln_map_keys, num_damage_bins, num_intensity_bins, adj_vuln_data=None) Loads the vulnerability data grouped by the intensity and damage bins, prioritizing the data in the adjustments file over the data in the vulnerability file. Args: vulns_bin: (List[Vulnerability]) vulnerability data from the vulnerability file vuln_map: (np.ndarray[uint8]) packed hashmap table mapping vuln_id to dense index vuln_map_keys: (np.ndarray[int32]) array of unique vulnerability ids (hashmap keys) num_damage_bins: (int) number of damage bins in the data num_intensity_bins: (int) the number of intensity bins adj_vuln_data: (List[vulnerability_dtype]) vulnerability adjustment data, sorted by vuln_id Returns: (List[List[List[floats]]]) vulnerability data grouped by intensity bin and damage bin .. py:function:: update_vuln_array_with_adj_data(vuln_array, vuln_map, vuln_map_keys, adj_vuln_data) Update the vulnerability array with adjustment data (used for parquet loading). Args: vuln_array: (3D array) The vulnerability data array. vuln_map: (np.ndarray[uint8]) packed hashmap table mapping vuln_id to dense index. vuln_map_keys: (np.ndarray[int32]) array of unique vulnerability ids (hashmap keys). adj_vuln_data: (List[vulnerability_dtype]) The vulnerability adjustment data. Returns: (3D array) The updated vulnerability data array. .. py:function:: get_vulns(storage: oasis_data_manager.filestore.backends.base.BaseStorage, run_dir, vuln_map, vuln_map_keys, num_intensity_bins, ignore_file_type=set(), df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader') Loads the vulnerabilities from the file. Args: storage: (str) the storage manager for fetching model data run_dir: (str) the path to the run folder (used to load the analysis settings) vuln_map: (np.ndarray[uint8]) packed hashmap table mapping vuln_id to dense index vuln_map_keys: (np.ndarray[int32]) array of unique vulnerability ids (hashmap keys) num_intensity_bins: (int) the number of intensity bins ignore_file_type: set(str) file extension to ignore when loading Returns: (Tuple[List[List[float]], int, np.array[int]) vulnerability data, vulnerabilities id, number of damage bins .. py:function:: get_vulnerability_replacements(run_dir, vuln_ids_set) Loads the vulnerability adjustment file. Args: run_dir: (str) the path pointing to the run directory vuln_ids_set: (set) set of vulnerability IDs to filter by Returns: (List[vulnerability_dtype]) vulnerability replacement data .. py:function:: get_mean_damage_bins(storage: oasis_data_manager.filestore.backends.base.BaseStorage, ignore_file_type=set()) Loads the mean damage bins from the damage_bin_dict file, namely, the `interpolation` value for each bin. Args: storage: (BaseStorage) the storage connector for fetching the model data ignore_file_type: set(str) file extension to ignore when loading Returns: (List[Union[damagebindictionary]]) loaded data from the damage_bin_dict file .. py:function:: get_damage_bins(storage: oasis_data_manager.filestore.backends.base.BaseStorage, ignore_file_type=set()) Loads the damage bins from the damage_bin_dict file. Args: storage: (BaseStorage) the storage connector for fetching the model data ignore_file_type: set(str) file extension to ignore when loading Returns: (List[Union[damagebindictionary]]) loaded data from the damage_bin_dict file .. py:function:: damage_bin_prob(p, intensities_min, intensities_max, vulns, intensities) Calculate the probability of an event happening and then causing damage. Note: vulns is a 1-d array containing 1 damage bin of the damage probability distribution as a function of hazard intensity. Args: p: (float) the probability to be updated intensities_min: (int) minimum intensity bin id intensities_max: (int) maximum intensity bin id vulns: (List[float]) slice of damage probability distribution given hazard intensity intensities: (List[float]) intensity probability distribution Returns: (float) the updated probability .. py:function:: do_result(vulns_id, vuln_array, mean_damage_bins, int32_mv, num_damage_bins, intensities_min, intensities_max, intensities, event_id, areaperil_id, vuln_i, cursor) Calculate the result concerning an event ID. Args: vulns_id: (List[int]) list of vulnerability IDs vuln_array: (List[List[list]]) list of vulnerabilities and their data mean_damage_bins: (List[float]) the mean of each damage bin (len(mean_damage_bins) == num_damage_bins) int32_mv: (List[int]) FILL IN LATER num_damage_bins: (int) number of damage bins in the data intensities_min: (int) minimum intensity bin id intensities_max: (int) maximum intensity bin id intensities: (List[float]) intensity probability distribution event_id: (int) the event ID that concerns the result being calculated areaperil_id: (List[int]) the areaperil ID that concerns the result being calculated vuln_i: (int) the index concerning the vulnerability inside the vuln_array cursor: (int) PLEASE FILL IN Returns: (int) PLEASE FILL IN .. py:function:: doCdf(event_id, num_intensity_bins, footprint, areaperil_id_ind, areaperil_to_vulns_idx_array, areaperil_to_vulns, vuln_array, vulns_id, num_damage_bins, mean_damage_bins, int32_mv, max_result_relative_size) Calculates the cumulative distribution function (cdf) for an event ID. Args: event_id: (int) the event ID the the CDF is being calculated to. num_intensity_bins: (int) the number of intensity bins for the CDF footprint: (List[Tuple[int, int, float]]) information about the footprint with event_id, areaperil_id, probability areaperil_id_ind: (np.array) id_index structure mapping areaperil_id to dense index areaperil_to_vulns_idx_array: (List[Tuple[int, int]]) the index where the areaperil ID starts and finishes areaperil_to_vulns: (List[int]) maps the areaperil ID to the vulnerability ID vuln_array: (List[list]) FILL IN LATER vulns_id: (List[int]) list of vulnerability IDs num_damage_bins: (int) number of damage bins in the data mean_damage_bins: (List[float]) the mean of each damage bin (len(mean_damage_bins) == num_damage_bins) int32_mv: (List[int]) FILL IN LATER max_result_relative_size: (int) the maximum result size Returns: (int) .. py:function:: convert_vuln_id_to_index(vuln_map, vuln_map_keys, areaperil_to_vulns) .. py:function:: run(run_dir, file_in, file_out, ignore_file_type, data_server, peril_filter, df_engine='oasis_data_manager.df_reader.reader.OasisPandasReader', analysis_pk=None) Runs the main process of the getmodel process. Args: run_dir: (str) the directory of where the process is running file_in: (Optional[str]) the path to the input directory file_out: (Optional[str]) the path to the output directory ignore_file_type: set(str) file extension to ignore when loading data_server: (bool) if set to True runs the data server peril_filter (list[int]): list of perils to include in the computation (if None, all perils will be included). df_engine: (str) The engine to use when loading dataframes Returns: None