oasislmf.pytools.eve.manager¶

Attributes¶

`logger`
`DEFAULT_EVENTS_FILE`
`NUMPY_RANDOM_SEED`

Functions¶

`read_events`(input_file)	Read the event IDs from the binary events file.
`stream_events`(events, stream_out)	Stream the output events.
`calculate_events_per_process`(n_events, total_processes)	Calculate number of events per process.
`partition_events__no_shuffle`(events, process_number, ...)	Assign events in the order they are loaded to each process in turn. Only
`partition_events__random_builtin`(events, ...)	Shuffle the events randomly and allocate to each process using builtin
`partition_events__random`(events, process_number, ...)	Shuffle the events randomly and allocate to each process. Only output
`partition_events__round_robin`(events, process_number, ...)	Partition the events sequentially in a round robin style per process.
`run`(input_file, process_number, total_processes[, ...])	Generate event ID partitions as a binary data stream with shuffling. By
`main`([input_file, process_number, total_processes, ...])

Module Contents¶

oasislmf.pytools.eve.manager.logger[source]¶

oasislmf.pytools.eve.manager.DEFAULT_EVENTS_FILE[source]¶

oasislmf.pytools.eve.manager.NUMPY_RANDOM_SEED = 723706[source]¶

oasislmf.pytools.eve.manager.read_events(input_file)[source]¶

Read the event IDs from the binary events file.

Parameters:: input_file (str | os.PathLike) – Path to binary events file.
Returns:: the event IDs, in the order they are held in the file.
Return type:: np.array[oasis_int]

oasislmf.pytools.eve.manager.stream_events(events, stream_out)[source]¶

Stream the output events.

Parameters:

events (Iterable) – Iterable containing the events to stream.
stream_out (File object) – File object with write method for handling output.

oasislmf.pytools.eve.manager.calculate_events_per_process(n_events, total_processes)[source]¶: Calculate number of events per process.

oasislmf.pytools.eve.manager.partition_events__no_shuffle(events, process_number, total_processes)[source]¶

Assign events in the order they are loaded to each process in turn. Only output the event IDs allocated to the given process_number.

Parameters:

events (np.array) – Array of ordered event IDs.
process_number (int) – The process number to receive a partition of events.
total_processes (int) – Total number of processes to distribute the events over.

oasislmf.pytools.eve.manager.partition_events__random_builtin(events, process_number, total_processes)[source]¶

Shuffle the events randomly and allocate to each process using builtin shuffle. Only output the event IDs to the given process_number.

Note that this can be memory intensive. For len(events) > 10**5 recommend using partition_events__random.

Parameters:

events (np.array) – Array of ordered event IDs.
process_number (int) – The process number to receive a partition of events.
total_processes (int) – Total number of processes to distribute the events over.

oasislmf.pytools.eve.manager.partition_events__random(events, process_number, total_processes)[source]¶

Shuffle the events randomly and allocate to each process. Only output the event IDs to the given process_number. Generates an iterator.

Randomisation is implemented using the Fisher-Yates algorithm.

Parameters:

events (np.array) – Array of ordered event IDs.
process_number (int) – The process number to receive a partition of events.
total_processes (int) – Total number of processes to distribute the events over.

Yields:

int – each event ID allocated to process_number, in shuffled order

oasislmf.pytools.eve.manager.partition_events__round_robin(events, process_number, total_processes)[source]¶

Partition the events sequentially in a round robin style per process. Only output the events allocated to the given process_number.

Parameters:

events (np.array) – Array of ordered event IDs.
process_number (int) – The process number to receive a partition of events.
total_processes (int) – Total number of processes to distribute the events over.

oasislmf.pytools.eve.manager.run(input_file, process_number, total_processes, no_shuffle=False, randomise=False, randomise_builtin=False, output_file='-')[source]¶

Generate event ID partitions as a binary data stream with shuffling. By default the events are shuffled by assiging to processes one by one cyclically.

Parameters:

input_file (str | os.PathLike) – Path to binary events file. If None
DEFAULT_EVENTS_FILE. (then defaults to)
process_number (int) – The process number to receive a partition of events.
total_processes (int) – Total number of processes to distribute the events over.
no_shuffle (bool, optional) – Disable shuffling events. Events are split and distributed into blocks in the order they are input. Takes priority over randomise(_builtin).
randomise (bool, optional) – Shuffle events randomly in the blocks and stream events on the fly. If no_shuffle is True then it takes priority.
randomise_builtin (bool, optional) – Shuffle events randomly in the blocks using builtin shuffle. If no_shuffle or randomise is True then it they take priority.
output_file (str | os.PathLike) – Path to output file. If ‘-’ then outputs to stdout.

oasislmf.pytools.eve.manager.main(input_file=None, process_number=None, total_processes=None, no_shuffle=False, randomise=False, randomise_builtin=False, output_file='-', **kwargs)[source]¶