oasislmf.pytools.elt.manager

Attributes

logger
SELT_output
MELT_output
QELT_output
SELT_headers
MELT_headers
QELT_headers
SELT_dtype
MELT_dtype
QELT_dtype
SELT_fmt
MELT_fmt
QELT_fmt

Classes

ELTReader

Abstract class to read event stream

Functions

read_buffer(byte_mv, cursor, valid_buff, event_id, ...)
read_input_files(run_dir, compute_melt, compute_qelt, ...)	Reads all input files and returns a dict of relevant data
run(run_dir, files_in[, selt_output_file, ...])	Runs ELT calculations
main([run_dir, files_in, selt, melt, qelt, noheader])

Module Contents

oasislmf.pytools.elt.manager.logger

oasislmf.pytools.elt.manager.SELT_output

oasislmf.pytools.elt.manager.MELT_output

oasislmf.pytools.elt.manager.QELT_output

oasislmf.pytools.elt.manager.SELT_headers

oasislmf.pytools.elt.manager.MELT_headers

oasislmf.pytools.elt.manager.QELT_headers

oasislmf.pytools.elt.manager.SELT_dtype

oasislmf.pytools.elt.manager.MELT_dtype

oasislmf.pytools.elt.manager.QELT_dtype

oasislmf.pytools.elt.manager.SELT_fmt = ''

oasislmf.pytools.elt.manager.MELT_fmt = ''

oasislmf.pytools.elt.manager.QELT_fmt = ''

class oasislmf.pytools.elt.manager.ELTReader(len_sample, compute_selt, compute_melt, compute_qelt, unique_event_ids, event_rates, intervals)

Bases: oasislmf.pytools.common.event_stream.EventReader

Abstract class to read event stream

This class provide a generic interface to read multiple event stream using: - selector : handle back pressure, the program is paused and don’t use resource if nothing is in the stream buffer - memoryview : read a chuck (PIPE_CAPACITY) of data at a time then work on it using a numpy byte view of this buffer

To use those methods need to be implemented: - __init__(self, …) the constructor with all data structure needed to read and store the event stream - read_buffer(self, byte_mv, cursor, valid_buff, event_id, item_id)

simply point to a local numba.jit function name read_buffer (a template is provided bellow) this function should implement the specific logic of where and how to store the event information.

Those to method may be overwritten - item_exit(self):

specific logic to do when an item is finished (only executed once the stream is finished but no 0,0 closure was present)

• event_read_log(self):

  what kpi to log when a full event is read

usage snippet:

with ExitStack() as stack:

streams_in, (stream_type, stream_agg_type, len_sample) = init_streams_in(files_in, stack) reader = CustomReader(<read relevant attributes>) for event_id in reader.read_streams(streams_in):

<event logic>

logger

selt_data

selt_idx

melt_data

melt_idx

qelt_data

qelt_idx

state

unique_event_ids

event_rates

intervals

curr_file_idx = None

read_buffer(byte_mv, cursor, valid_buff, event_id, item_id, file_idx)

oasislmf.pytools.elt.manager.read_buffer(byte_mv, cursor, valid_buff, event_id, item_id, selt_data, selt_idx, state, melt_data, melt_idx, qelt_data, qelt_idx, intervals, unique_event_ids, event_rates)

oasislmf.pytools.elt.manager.read_input_files(run_dir, compute_melt, compute_qelt, sample_size)

Reads all input files and returns a dict of relevant data Args:

run_dir (str | os.PathLike): Path to directory containing required files structure compute_melt (bool): Compute MELT bool compute_qelt (bool): Compute QELT bool sample_size (int): Sample size

Returns:

file_data (Dict[str, Any]): A dict of relevent data extracted from files

oasislmf.pytools.elt.manager.run(run_dir, files_in, selt_output_file=None, melt_output_file=None, qelt_output_file=None, noheader=False)

Runs ELT calculations Args:

run_dir (str | os.PathLike): Path to directory containing required files structure files_in (str | os.PathLike): Path to summary binary input file selt_output_file (str, optional): Path to SELT output file. Defaults to None. melt_output_file (str, optional): Path to MELT output file. Defaults to None. qelt_output_file (str, optional): Path to QELT output file. Defaults to None. noheader (bool): Boolean value to skip header in output file

oasislmf.pytools.elt.manager.main(run_dir='.', files_in=None, selt=None, melt=None, qelt=None, noheader=None, **kwargs)