oasislmf.pytools.fm.stream_sparse

Attributes

logger

remove_empty

SPECIAL_SIDX_COUNT

ITEM_HEADER_SIZE

SIDX_LOSS_WRITE_SIZE

Classes

FMReader

when reading the stream we store relenvant value into a slithly modified version of the CSR sparse matrix where

EventWriterSparse

EventWriterOrderedOutputSparse

Functions

reset_empty_items(compute_idx, sidx_indptr, sidx_val, ...)

add_new_loss(sidx, loss, compute_i, sidx_indptr, ...)

event_log_msg(event_id, sidx_indptr, len_array, node_count)

read_buffer(byte_mv, cursor, valid_buff, event_id, ...)

load_event(byte_mv, event_id, nodes_array, ...)

get_compute_end(computes, compute_idx)

Module Contents

oasislmf.pytools.fm.stream_sparse.logger[source]
oasislmf.pytools.fm.stream_sparse.remove_empty = False[source]
oasislmf.pytools.fm.stream_sparse.SPECIAL_SIDX_COUNT = 6[source]
oasislmf.pytools.fm.stream_sparse.ITEM_HEADER_SIZE[source]
oasislmf.pytools.fm.stream_sparse.SIDX_LOSS_WRITE_SIZE[source]
oasislmf.pytools.fm.stream_sparse.reset_empty_items(compute_idx, sidx_indptr, sidx_val, loss_val, computes)[source]
oasislmf.pytools.fm.stream_sparse.add_new_loss(sidx, loss, compute_i, sidx_indptr, sidx_val, loss_val)[source]
oasislmf.pytools.fm.stream_sparse.event_log_msg(event_id, sidx_indptr, len_array, node_count)[source]
oasislmf.pytools.fm.stream_sparse.read_buffer(byte_mv, cursor, valid_buff, event_id, item_id, nodes_array, sidx_indexes, sidx_indptr, sidx_val, loss_indptr, loss_val, pass_through, computes, compute_idx)[source]
class oasislmf.pytools.fm.stream_sparse.FMReader(nodes_array, sidx_indexes, sidx_indptr, sidx_val, loss_indptr, loss_val, pass_through, len_array, computes, compute_idx)[source]

Bases: oasislmf.pytools.common.event_stream.EventReader

when reading the stream we store relenvant value into a slithly modified version of the CSR sparse matrix where the column indices for row i are stored in indices[indptr[i]:indptr[i+1]] and their corresponding values are stored in data[indptr[i]:indptr[i+1]].

nodes_array: array containing all the static information on the nodes loss_indptr: array containing the indexes of the beginning and end of samples of an item loss_sidx: array containing the sidx of the samples loss_val: array containing the loss of the samples

nodes_array[source]
sidx_indexes[source]
sidx_indptr[source]
sidx_val[source]
loss_indptr[source]
loss_val[source]
pass_through[source]
len_array[source]
computes[source]
compute_idx[source]
logger[source]
read_buffer(byte_mv, cursor, valid_buff, event_id, item_id)[source]
item_exit()[source]
event_read_log(event_id)[source]
oasislmf.pytools.fm.stream_sparse.load_event(byte_mv, event_id, nodes_array, sidx_indexes, sidx_indptr, sidx_val, loss_indptr, loss_val, pass_through, computes, compute_idx, output_array, i_layer, i_index)[source]
class oasislmf.pytools.fm.stream_sparse.EventWriterSparse(files_out, nodes_array, output_array, sidx_indexes, sidx_indptr, sidx_val, loss_indptr, loss_val, pass_through, len_sample, computes)[source]
files_out[source]
nodes_array[source]
sidx_indexes[source]
sidx_indptr[source]
sidx_val[source]
loss_indptr[source]
loss_val[source]
pass_through[source]
len_sample[source]
computes[source]
output_array[source]
byte_mv[source]
write(event_id, compute_idx)[source]
oasislmf.pytools.fm.stream_sparse.get_compute_end(computes, compute_idx)[source]
class oasislmf.pytools.fm.stream_sparse.EventWriterOrderedOutputSparse(files_out, nodes_array, output_array, sidx_indexes, sidx_indptr, sidx_val, loss_indptr, loss_val, pass_through, len_sample, computes)[source]

Bases: EventWriterSparse

write(event_id, compute_idx)[source]