oasislmf.pytools.gul.io

This file contains the utilities for all the I/O necessary in gulpy.

Functions

gen_structs()

Generate some data structures needed for the whole computation.

read_getmodel_stream(stream_in, items, item_map_hm, ...)

Read the getmodel output stream yielding data event by event.

stream_to_data(byte_mv, valid_buf, size_cdf_entry, ...)

Parse streamed data into data arrays.

Module Contents

oasislmf.pytools.gul.io.gen_structs()[source]

Generate some data structures needed for the whole computation.

Returns:

map of group ids to random seeds,

list storing the index where a specific cdf record starts in the rec numpy array.

Return type:

Dict(int,int), List

oasislmf.pytools.gul.io.read_getmodel_stream(stream_in, items, item_map_hm, item_map_hm_keys, item_map_ja_offsets, coverages, compute, seeds, buff_size=PIPE_CAPACITY)[source]

Read the getmodel output stream yielding data event by event.

Parameters:

  • stream_in (buffer-like) – input stream, e.g. sys.stdin.buffer.

  • items (numpy.ndarray) – sorted items array (already filtered by peril if applicable).

  • item_map_hm (np.array[uint8]) – packed hashmap (areaperil_id, vulnerability_id) → pair_index.

  • item_map_hm_keys (np.array[item_map_key_dtype]) – key storage for the hashmap.

  • item_map_ja_offsets (np.array[oasis_int]) – CSR offsets into items array (N_pairs + 1).

  • coverages (numpy.ndarray[coverage_type]) – array with coverage data.

  • compute (numpy.array[int]) – list of coverages to be computed.

  • seeds (numpy.array[int]) – the random seeds for each coverage_id.

  • buff_size (int) – size in bytes of the read buffer. Defaults to PIPE_CAPACITY.

Raises:

ValueError – If the stream type is not 1.

Yields:

int, int, numpy.array[items_data_type], numpy.array[oasis_float], numpy.array[int], int – event_id, index of the last coverage_id stored in compute, item-related data, cdf records, array with the indices of rec where each cdf record starts, number of unique random seeds computed so far.

Note

It is advisable to set buff_size as 2x the maximum pipe limit (65536 bytes) to ensure that the stream is always read in the biggest possible chunks, which nominally is the largest between the pipe limit and the remaining memory to fill the memoryview.

oasislmf.pytools.gul.io.stream_to_data(byte_mv, valid_buf, size_cdf_entry, last_event_id, items, item_map_hm, item_map_hm_keys, item_map_ja_offsets, coverages, compute_i, compute, items_data_i, items_data, seeds, rng_index, group_id_rng_index, damagecdf_i, rec_idx_ptr)[source]

Parse streamed data into data arrays.

Parameters:

  • byte_mv (ndarray) – byte view of the buffer

  • valid_buf (int) – number of bytes with valid data

  • size_cdf_entry (int) – size (in bytes) of a single record

  • last_event_id (int) – event_id of the last event that was completed

  • items (numpy.ndarray) – sorted items array.

  • item_map_hm (np.array[uint8]) – packed hashmap (areaperil_id, vulnerability_id) → pair_index.

  • item_map_hm_keys (np.array[item_map_key_dtype]) – key storage for the hashmap.

  • item_map_ja_offsets (np.array[oasis_int]) – CSR offsets into items array (N_pairs + 1).

  • coverages (numpy.ndarray[coverage_type]) – array with coverage data.

  • compute_i (int) – index of the last coverage id stored in compute.

  • compute (numpy.array[int]) – list of coverage ids to be computed.

  • items_data_i (int) – index of the last items_data_i stored in items_data.

  • items_data (numpy.array[items_data_type]) – item-related data.

  • seeds (numpy.array[int]) – the random seeds for each coverage_id.

  • rng_index (int) – number of unique random seeds computed so far.

  • group_id_rng_index (Dict([int,int])) – map of group ids to random seeds.

  • damagecdf_i (int) – index of the last cdf record that has been read from stream and stored in rec.

  • rec_idx_ptr (numpy.array[int]) – array with the indices of rec where each cdf record starts.

Returns:

Tuple[int, bool, int, numpy.array[damagecdfrec], numpy.array[ProbMean], List[int], int, int, int, numpy.array[items_data_type], int, Dict[int, int], int]:

  • cursor: number of bytes read from byte_mv

  • yield_event: whether the event being read (id=`last_event_id`) has been fully read

  • event_id: event_id of the record parsing stopped on. When yield_event is True this is the id of the next event, which starts at cursor

  • dmgcdfrec: areaperil_id and vulnerability_id of each cdf record read

  • rec: the cdf bins read

  • rec_idx_ptr: list with the indices of rec where each cdf record starts

  • last_event_id: event id currently being accumulated

  • compute_i: index of the last coverage id stored in compute

  • items_data_i: index of the last items_data_i stored in items_data

  • items_data: item-related data, a new larger array when items_data had to grow

  • rng_index: number of unique random seeds computed so far

  • group_id_rng_index: map of group ids to random seeds

  • damagecdf_i: index of the last cdf record that has been read from stream and stored in rec