oasislmf.pytools.converters.bintocsv.utils.gul

Attributes

logger

Classes

GulReader

Abstract class to read event stream

Functions

read_buffer(byte_mv, cursor, valid_buff, event_id, ...)
gul_tocsv(stack, file_in, file_out, file_type, noheader)

Module Contents

oasislmf.pytools.converters.bintocsv.utils.gul.logger

class oasislmf.pytools.converters.bintocsv.utils.gul.GulReader(len_sample, data_dtype)

Bases: oasislmf.pytools.common.event_stream.EventReader

Abstract class to read event stream

This class provide a generic interface to read multiple event stream using: - selector : handle back pressure, the program is paused and don’t use resource if nothing is in the stream buffer - memoryview : read a chuck (PIPE_CAPACITY) of data at a time then work on it using a numpy byte view of this buffer

To use those methods need to be implemented: - __init__(self, …) the constructor with all data structure needed to read and store the event stream - read_buffer(self, byte_mv, cursor, valid_buff, event_id, item_id)

simply point to a local numba.jit function name read_buffer (a template is provided bellow) this function should implement the specific logic of where and how to store the event information.

Those to method may be overwritten - item_exit(self):

specific logic to do when an item is finished (only executed once the stream is finished but no 0,0 closure was present)

• event_read_log(self):

  what kpi to log when a full event is read

usage snippet:

with ExitStack() as stack:

streams_in, (stream_type, stream_agg_type, len_sample) = init_streams_in(files_in, stack) reader = CustomReader(<read relevant attributes>) for event_id in reader.read_streams(streams_in):

<event logic>

logger

data

idx

state

read_buffer(byte_mv, cursor, valid_buff, event_id, item_id, **kwargs)

oasislmf.pytools.converters.bintocsv.utils.gul.read_buffer(byte_mv, cursor, valid_buff, event_id, item_id, data, idxs, state)

oasislmf.pytools.converters.bintocsv.utils.gul.gul_tocsv(stack, file_in, file_out, file_type, noheader)