oasislmf.pytools.elt.manager

Attributes

logger

Classes

ELTReader

Abstract class to read event stream

Functions

read_buffer(byte_mv, cursor, valid_buff, event_id, ...)
read_input_files(run_dir, compute_melt, compute_qelt, ...)	Reads all input files and returns a dict of relevant data
run(run_dir, files_in[, selt_output_file, ...])	Runs ELT calculations
main([run_dir, files_in, selt, melt, qelt, noheader, ext])

Module Contents

oasislmf.pytools.elt.manager.logger

class oasislmf.pytools.elt.manager.ELTReader(len_sample, compute_selt, compute_melt, compute_qelt, unique_event_ids, event_rates, intervals)

Bases: oasislmf.pytools.common.event_stream.EventReader

Abstract class to read event stream

This class provide a generic interface to read multiple event stream using: - selector : handle back pressure, the program is paused and don’t use resource if nothing is in the stream buffer - memoryview : read a chuck (PIPE_CAPACITY) of data at a time then work on it using a numpy byte view of this buffer

To use those methods need to be implemented: - __init__(self, …) the constructor with all data structure needed to read and store the event stream - read_buffer(self, byte_mv, cursor, valid_buff, event_id, item_id)

simply point to a local numba.jit function name read_buffer (a template is provided bellow) this function should implement the specific logic of where and how to store the event information.

Those to method may be overwritten - item_exit(self):

specific logic to do when an item is finished (only executed once the stream is finished but no 0,0 closure was present)

• event_read_log(self):

  what kpi to log when a full event is read

usage snippet:

with ExitStack() as stack:

streams_in, (stream_type, stream_agg_type, len_sample) = init_streams_in(files_in, stack) reader = CustomReader(<read relevant attributes>) for event_id in reader.read_streams(streams_in):

<event logic>

logger

selt_data

selt_idx

melt_data

melt_idx

qelt_data

qelt_idx

state

unique_event_ids

event_rates

intervals

curr_file_idx = None

get_data(out_type)

get_data_idx(out_type)

read_buffer(byte_mv, cursor, valid_buff, event_id, item_id, file_idx)

oasislmf.pytools.elt.manager.read_buffer(byte_mv, cursor, valid_buff, event_id, item_id, selt_data, selt_idx, state, melt_data, melt_idx, qelt_data, qelt_idx, intervals, unique_event_ids, event_rates)

oasislmf.pytools.elt.manager.read_input_files(run_dir, compute_melt, compute_qelt, sample_size)

Reads all input files and returns a dict of relevant data

Parameters:

• run_dir (str | os.PathLike) – Path to directory containing required files structure

• compute_melt (bool) – Compute MELT bool

• compute_qelt (bool) – Compute QELT bool

• sample_size (int) – Sample size

Returns:

A dict of relevent data extracted from files

Return type:

file_data (Dict[str, Any])

oasislmf.pytools.elt.manager.run(run_dir, files_in, selt_output_file=None, melt_output_file=None, qelt_output_file=None, noheader=False, output_format='csv')

Runs ELT calculations

Parameters:

• run_dir (str | os.PathLike) – Path to directory containing required files structure

• files_in (list[str]) – Path to summary binary input file

• selt_output_file (str, optional) – Path to SELT output file. Defaults to None.

• melt_output_file (str, optional) – Path to MELT output file. Defaults to None.

• qelt_output_file (str, optional) – Path to QELT output file. Defaults to None.

• noheader (bool) – Boolean value to skip header in output file. Defaults to False.

• output_format (str) – Output format extension. Defaults to “csv”.

oasislmf.pytools.elt.manager.main(run_dir='.', files_in=None, selt=None, melt=None, qelt=None, noheader=False, ext='csv', **kwargs)