oasislmf.pytools.eve.manager
============================

.. py:module:: oasislmf.pytools.eve.manager


Attributes
----------

.. autoapisummary::

   oasislmf.pytools.eve.manager.logger
   oasislmf.pytools.eve.manager.DEFAULT_EVENTS_FILE
   oasislmf.pytools.eve.manager.NUMPY_RANDOM_SEED


Functions
---------

.. autoapisummary::

   oasislmf.pytools.eve.manager.read_events
   oasislmf.pytools.eve.manager.stream_events
   oasislmf.pytools.eve.manager.calculate_events_per_process
   oasislmf.pytools.eve.manager.partition_events__no_shuffle
   oasislmf.pytools.eve.manager.partition_events__random_builtin
   oasislmf.pytools.eve.manager.partition_events__random
   oasislmf.pytools.eve.manager.partition_events__round_robin
   oasislmf.pytools.eve.manager.run
   oasislmf.pytools.eve.manager.main


Module Contents
---------------

.. py:data:: logger

.. py:data:: DEFAULT_EVENTS_FILE

.. py:data:: NUMPY_RANDOM_SEED
   :value: 723706


.. py:function:: read_events(input_file)

   Read the event IDs from the binary events file.

   Args:
       input_file (str | os.PathLike): Path to binary events file.


.. py:function:: stream_events(events, stream_out)

   Stream the output events.

   Args:
       events (Iterable): Iterable containing the events to stream.
       stream_out (File object): File object with `write` method for handling output.


.. py:function:: calculate_events_per_process(n_events, total_processes)

   Calculate number of events per process.
       


.. py:function:: partition_events__no_shuffle(events, process_number, total_processes)

   Assign events in the order they are loaded to each process in turn. Only
   output the event IDs allocated to the given `process_number`.

   Args:
       events (np.array): Array of ordered event IDs.
       process_number (int): The process number to receive a partition of events.
       total_processes (int): Total number of processes to distribute the events over.


.. py:function:: partition_events__random_builtin(events, process_number, total_processes)

   Shuffle the events randomly and allocate to each process using builtin
   shuffle. Only output the event IDs to the given `process_number`.

   Note that this can be memory intensive. For `len(events) > 10**5` recommend
   using `partition_events__random`.

   Args:
       events (np.array): Array of ordered event IDs.
       process_number (int): The process number to receive a partition of events.
       total_processes (int): Total number of processes to distribute the events over.


.. py:function:: partition_events__random(events, process_number, total_processes)

   Shuffle the events randomly and allocate to each process. Only output
   the event IDs to the given `process_number`. Generates an iterator.

   Randomisation is implemented using the Fisher-Yates algorithm.

   Args:
       events (np.array): Array of ordered event IDs.
       process_number (int): The process number to receive a partition of events.
       total_processes (int): Total number of processes to distribute the events over.


.. py:function:: partition_events__round_robin(events, process_number, total_processes)

   Partition the events sequentially in a round robin style per process.
   Only output the events allocated to the given `process_number`.

   Args:
       events (np.array): Array of ordered event IDs.
       process_number (int): The process number to receive a partition of events.
       total_processes (int): Total number of processes to distribute the events over.


.. py:function:: run(input_file, process_number, total_processes, no_shuffle=False, randomise=False, randomise_builtin=False, output_file='-')

   Generate event ID partitions as a binary data stream with shuffling. By
   default the events are shuffled by assiging to processes one by one
   cyclically.

   Args:
       input_file (str | os.PathLike): Path to binary events file. If None
       then defaults to DEFAULT_EVENTS_FILE.
       process_number (int): The process number to receive a partition of events.
       total_processes (int): Total number of processes to distribute the events over.
       no_shuffle (bool, optional): Disable shuffling events. Events are split
           and distributed into blocks in the order they are input. Takes
           priority over `randomise(_builtin)`.
       randomise (bool, optional): Shuffle events randomly in the blocks and
           stream events on the fly. If `no_shuffle` is `True` then it takes
           priority.
       randomise_builtin (bool, optional): Shuffle events randomly in the blocks using
           builtin shuffle. If `no_shuffle` or `randomise` is `True` then it
           they take priority.
       output_file (str | os.PathLike): Path to output file. If '-' then outputs to stdout.


.. py:function:: main(input_file=None, process_number=None, total_processes=None, no_shuffle=False, randomise=False, randomise_builtin=False, output_file='-', **kwargs)