oasislmf.pytools.eve.manager

Attributes

logger
DEFAULT_EVENTS_FILE
NUMPY_RANDOM_SEED

Functions

read_events(input_file)	Read the event IDs from the binary events file.
stream_events(events, stream_out)	Stream the output events.
calculate_events_per_process(n_events, total_processes)	Calculate number of events per process.
partition_events__no_shuffle(events, process_number, ...)	Assign events in the order they are loaded to each process in turn. Only
partition_events__random_builtin(events, ...)	Shuffle the events randomly and allocate to each process using builtin
partition_events__random(events, process_number, ...)	Shuffle the events randomly and allocate to each process. Only output
partition_events__round_robin(events, process_number, ...)	Partition the events sequentially in a round robin style per process.
run(input_file, process_number, total_processes[, ...])	Generate event ID partitions as a binary data stream with shuffling. By
main([input_file, process_number, total_processes, ...])

Module Contents

oasislmf.pytools.eve.manager.logger

oasislmf.pytools.eve.manager.DEFAULT_EVENTS_FILE

oasislmf.pytools.eve.manager.NUMPY_RANDOM_SEED = 723706

oasislmf.pytools.eve.manager.read_events(input_file)

Read the event IDs from the binary events file.

Args:

input_file (str | os.PathLike): Path to binary events file.

oasislmf.pytools.eve.manager.stream_events(events, stream_out)

Stream the output events.

Args:

events (Iterable): Iterable containing the events to stream. stream_out (File object): File object with write method for handling output.

oasislmf.pytools.eve.manager.calculate_events_per_process(n_events, total_processes)

Calculate number of events per process.

oasislmf.pytools.eve.manager.partition_events__no_shuffle(events, process_number, total_processes)

Assign events in the order they are loaded to each process in turn. Only output the event IDs allocated to the given process_number.

Args:

events (np.array): Array of ordered event IDs. process_number (int): The process number to receive a partition of events. total_processes (int): Total number of processes to distribute the events over.

oasislmf.pytools.eve.manager.partition_events__random_builtin(events, process_number, total_processes)

Shuffle the events randomly and allocate to each process using builtin shuffle. Only output the event IDs to the given process_number.

Note that this can be memory intensive. For len(events) > 10**5 recommend using partition_events__random.

Args:

events (np.array): Array of ordered event IDs. process_number (int): The process number to receive a partition of events. total_processes (int): Total number of processes to distribute the events over.

oasislmf.pytools.eve.manager.partition_events__random(events, process_number, total_processes)

Shuffle the events randomly and allocate to each process. Only output the event IDs to the given process_number. Generates an iterator.

Randomisation is implemented using the Fisher-Yates algorithm.

Args:

events (np.array): Array of ordered event IDs. process_number (int): The process number to receive a partition of events. total_processes (int): Total number of processes to distribute the events over.

oasislmf.pytools.eve.manager.partition_events__round_robin(events, process_number, total_processes)

Partition the events sequentially in a round robin style per process. Only output the events allocated to the given process_number.

Args:

events (np.array): Array of ordered event IDs. process_number (int): The process number to receive a partition of events. total_processes (int): Total number of processes to distribute the events over.

oasislmf.pytools.eve.manager.run(input_file, process_number, total_processes, no_shuffle=False, randomise=False, randomise_builtin=False, output_file='-')

Generate event ID partitions as a binary data stream with shuffling. By default the events are shuffled by assiging to processes one by one cyclically.

Args:

input_file (str | os.PathLike): Path to binary events file. If None then defaults to DEFAULT_EVENTS_FILE. process_number (int): The process number to receive a partition of events. total_processes (int): Total number of processes to distribute the events over. no_shuffle (bool, optional): Disable shuffling events. Events are split

and distributed into blocks in the order they are input. Takes priority over randomise(_builtin).

randomise (bool, optional): Shuffle events randomly in the blocks and

stream events on the fly. If no_shuffle is True then it takes priority.

randomise_builtin (bool, optional): Shuffle events randomly in the blocks using

builtin shuffle. If no_shuffle or randomise is True then it they take priority.

output_file (str | os.PathLike): Path to output file. If ‘-’ then outputs to stdout.

oasislmf.pytools.eve.manager.main(input_file=None, process_number=None, total_processes=None, no_shuffle=False, randomise=False, randomise_builtin=False, output_file='-', **kwargs)