oasislmf.pytools.gul.io

This file contains the utilities for all the I/O necessary in gulpy.

Functions

gen_structs()	Generate some data structures needed for the whole computation.
gen_valid_area_peril(valid_area_peril_id)
read_getmodel_stream(stream_in, item_map, coverages, ...)	Read the getmodel output stream yielding data event by event.
stream_to_data(byte_mv, valid_buf, size_cdf_entry, ...)	Parse streamed data into data arrays.

Module Contents

oasislmf.pytools.gul.io.gen_structs()

Generate some data structures needed for the whole computation.

Returns:

Dict(int,int), List: map of group ids to random seeds,

list storing the index where a specific cdf record starts in the rec numpy array.

oasislmf.pytools.gul.io.gen_valid_area_peril(valid_area_peril_id)

oasislmf.pytools.gul.io.read_getmodel_stream(stream_in, item_map, coverages, compute, seeds, valid_area_peril_id=None, buff_size=PIPE_CAPACITY)

Read the getmodel output stream yielding data event by event.

Args:

stream_in (buffer-like): input stream, e.g. sys.stdin.buffer. item_map (Dict[ITEM_MAP_KEY_TYPE, ITEM_MAP_VALUE_TYPE]): dict storing

the mapping between areaperil_id, vulnerability_id to item.

coverages (numpy.ndarray[coverage_type]): array with coverage data. compute (numpy.array[int]): list of coverages to be computed. seeds (numpy.array[int]): the random seeds for each coverage_id. buff_size (int): size in bytes of the read buffer (see note). valid_area_peril_id (list[int]): list of valid areaperil_ids.

Raises:

ValueError: If the stream type is not 1.

Yields:

int, int, numpy.array[items_data_type], numpy.array[oasis_float], numpy.array[int], int:

event_id, index of the last coverage_id stored in compute, item-related data, cdf records, array with the indices of rec where each cdf record starts, number of unique random seeds computed so far.

Note:

It is advisable to set buff_size as 2x the maximum pipe limit (65536 bytes) to ensure that the stream is always read in the biggest possible chunks, which nominally is the largest between the pipe limit and the remaining memory to fill the memoryview.

oasislmf.pytools.gul.io.stream_to_data(byte_mv, valid_buf, size_cdf_entry, last_event_id, item_map, coverages, valid_area_peril_dict, compute_i, compute, items_data_i, items_data, seeds, rng_index, group_id_rng_index, damagecdf_i, rec_idx_ptr)

Parse streamed data into data arrays.

Args:

byte_mv (ndarray): byte view of the buffer valid_buf (int): number of bytes with valid data size_cdf_entry (int): size (in bytes) of a single record last_event_id (int): event_id of the last event that was completed item_map (Dict[ITEM_MAP_KEY_TYPE, ITEM_MAP_VALUE_TYPE]): dict storing

the mapping between areaperil_id, vulnerability_id to item.

coverages (numpy.ndarray[coverage_type]): array with coverage data. compute_i (int): index of the last coverage id stored in compute. compute (numpy.array[int]): list of coverage ids to be computed. items_data_i (int): index of the last items_data_i stored in items_data. items_data (numpy.array[items_data_type]): item-related data. seeds (numpy.array[int]): the random seeds for each coverage_id. rng_index (int): number of unique random seeds computed so far. group_id_rng_index (Dict([int,int])): map of group ids to random seeds. damagecdf_i (int): index of the last cdf record that has been read from stream and stored in rec. rec_idx_ptr (numpy.array[int]): array with the indices of rec where each cdf record starts.

Returns:

int, bool, int, numpy.array[ProbMean], numpy.array[int], int, int, int, numpy.array[items_data_type], int, Dict[ITEM_MAP_KEY_TYPE, ITEM_MAP_VALUE_TYPE]), int:

number of int numbers read from the int32_mv ndarray, whether the current event (id=`event_id`) has been fully read, cdf record, array with the indices of rec where each cdf record starts, last or current event id, index of the last coverage id stored in compute, index of the last items_data_i stored in items_data, item-related data, number of unique random seeds computed so far, map of group ids to random seeds, index of the last cdf record that has been read from stream and stored in `rec`W