oasislmf.pytools.aal.manager

Attributes

logger
OASIS_AAL_MEMORY

Functions

process_bin_file(fbin, offset, occ_csr, ...)	Reads summary<n>.bin file event_ids and summary_ids to populate summaries_data
process_idx_file(fbin, idx_data, idx_cursor, occ_csr, ...)	Use a pre-built .idx file to populate summaries_data without scanning sample records.
sort_and_save_chunk(summaries_data, temp_file_path)	Sort a chunk of summaries data and save it to a temporary file.
merge_sorted_chunks(memmaps)	Merge sorted chunks using a k-way merge algorithm and yield next smallest row
get_summaries_data(path, files_handles, occ_csr, ...)	Gets the indexed summaries data, ordered with k-way merge if not enough memory.
summary_index(path, occ_csr, stack)	Index the summary binary outputs.
read_input_files(run_dir)	Reads all input files and returns a dict of relevant data
get_num_subsets(alct, sample_size, max_summary_id)	Gets the number of subsets required to generates the Sample AAL np map for subset sizes up to sample_size
get_weighted_means(vec_sample_sum_loss, weighting, ...)	Get sum of weighted mean and weighted mean_squared
do_calc_end(period_no, no_of_periods, period_weights, ...)	Updates Analytical and Sample AAL vectors from sample sum losses
read_losses(summary_fin, cursor, vec_sample_sum_loss)	Read losses from summary_fin starting at cursor, populate vec_sample_sum_loss
skip_losses(summary_fin, cursor)	Skip through losses in summary_fin starting at cursor
run_aal(memmaps, no_of_periods, period_weights, ...)	Run AAL calculation loop to populate vec data
calculate_mean_stddev(observable_sum, ...)	Compute the mean and standard deviation from the sum and squared sum of an observable
get_aal_data(vec_analytical_aal, vecs_sample_aal, ...)	Generate AAL csv data
get_aal_data_meanonly(vec_analytical_aal, ...)	Generate AAL csv data
calculate_confidence_interval(std_err, confidence_level)	Calculate the confidence interval based on standard error and confidence level.
get_alct_data(vecs_sample_aal, max_summary_id, ...)	Generate ALCT csv data
run(run_dir, subfolder[, aal_output_file, ...])	Runs AAL calculations
main([run_dir, subfolder, aal, alct, meanonly, ...])

Module Contents

oasislmf.pytools.aal.manager.logger

oasislmf.pytools.aal.manager.OASIS_AAL_MEMORY

oasislmf.pytools.aal.manager.process_bin_file(fbin, offset, occ_csr, summaries_data, summaries_idx, file_index)

Reads summary<n>.bin file event_ids and summary_ids to populate summaries_data Args:

fbin (np.memmap): summary binary memmap offset (int): file offset to read from occ_csr (OccurrenceCSR): id_index-backed CSR occurrence map summaries_data (ndarray[_SUMMARIES_DTYPE]): Index summary data (summaries.idx data) summaries_idx (int): current index reached in summaries_data file_index (int): Summary bin file index

Returns:

summaries_idx (int): current index reached in summaries_data resize_flag (bool): flag to indicate whether to resize summaries_data when full offset (int): file offset to read from

oasislmf.pytools.aal.manager.process_idx_file(fbin, idx_data, idx_cursor, occ_csr, summaries_data, summaries_idx, file_index)

Use a pre-built .idx file to populate summaries_data without scanning sample records.

Instead of reading the entire .bin sequentially (including all loss records just to advance the cursor), each .idx entry gives the byte offset of an event header directly. Only the 4-byte event_id is read from the .bin per entry; loss records are never touched here.

Args:

fbin (np.memmap): summary binary memmap (dtype u1) idx_data (np.memmap): index file memmap (dtype summary_stream_index_dtype) idx_cursor (int): current position in idx_data to resume from occ_csr (OccurrenceCSR): id_index-backed CSR occurrence map summaries_data (ndarray[_SUMMARIES_DTYPE]): output index buffer summaries_idx (int): next free slot in summaries_data file_index (int): which .bin file this is (used as file_idx in output)

Returns:

summaries_idx (int): updated cursor into summaries_data resize_flag (bool): True if summaries_data is full and must be flushed before resuming idx_cursor (int): position in idx_data to resume from after a flush

oasislmf.pytools.aal.manager.sort_and_save_chunk(summaries_data, temp_file_path)

Sort a chunk of summaries data and save it to a temporary file. Args:

summaries_data (ndarray[_SUMMARIES_DTYPE]): Indexed summary data temp_file_path (str | os.PathLike): Path to temporary file

oasislmf.pytools.aal.manager.merge_sorted_chunks(memmaps)

Merge sorted chunks using a k-way merge algorithm and yield next smallest row Args:

memmaps (List[np.memmap]): List of temporary file memmaps

Yields:

smallest_row (ndarray[_SUMMARIES_DTYPE]): yields the next smallest row from sorted summaries partial files

oasislmf.pytools.aal.manager.get_summaries_data(path, files_handles, occ_csr, aal_max_memory, idx_handles=None)

Gets the indexed summaries data, ordered with k-way merge if not enough memory.

When idx_handles[i] is not None (a memmap of summary_stream_index_dtype records), uses process_idx_file for that file — seeking directly to each event header via pre-computed offsets, never reading sample records during indexing. Falls back to process_bin_file (full sequential scan) for any file without a paired .idx.

Args:

path (os.PathLike): Path to the workspace folder containing summary binaries files_handles (List[np.memmap]): List of memmaps for summary files data occ_csr (OccurrenceCSR): id_index-backed CSR occurrence map aal_max_memory (float): OASIS_AAL_MEMORY value (has to be passed in as numba won’t update from environment variable) idx_handles (List[np.memmap | None] | None): Per-file .idx memmaps, or None to use sequential scan for all files

Returns:

memmaps (List[np.memmap]): List of temporary file memmaps max_summary_id (int): Max summary ID

oasislmf.pytools.aal.manager.summary_index(path, occ_csr, stack)

Index the summary binary outputs.

If a .idx file (summary_stream_index_dtype) exists alongside a .bin file, uses process_idx_file to build the index without scanning sample records. Falls back to process_bin_file (full sequential scan) for any .bin without a paired .idx.

Args:

path (os.PathLike): Path to the workspace folder containing summary binaries occ_csr (OccurrenceCSR): id_index-backed CSR occurrence map stack (ExitStack): Exit stack

Returns:

files_handles (List[np.memmap]): List of memmaps for summary files data sample_size (int): Sample size max_summary_id (int): Max summary ID memmaps (List[np.memmap]): List of temporary file memmaps

oasislmf.pytools.aal.manager.read_input_files(run_dir)

Reads all input files and returns a dict of relevant data Args:

run_dir (str | os.PathLike): Path to directory containing required files structure

Returns:

file_data (Dict[str, Any]): A dict of relevent data extracted from files

oasislmf.pytools.aal.manager.get_num_subsets(alct, sample_size, max_summary_id)

Gets the number of subsets required to generates the Sample AAL np map for subset sizes up to sample_size Example: sample_size[10], max_summary_id[2] generates following ndarray [

# subset_size, mean, mean_squared, mean_period [0, 0, 0], # subset_size = 1 , summary_id = 1 [0, 0, 0], # subset_size = 1 , summary_id = 2 [0, 0, 0], # subset_size = 2 , summary_id = 1 [0, 0, 0], # subset_size = 2 , summary_id = 2 [0, 0, 0], # subset_size = 4 , summary_id = 1 [0, 0, 0], # subset_size = 4 , summary_id = 2 [0, 0, 0], # subset_size = 10 , summary_id = 1, subset_size = sample_size [0, 0, 0], # subset_size = 10 , summary_id = 2, subset_size = sample_size

] Subset_size is implicit based on position in array, grouped by max_summary_id So first two arrays are subset_size 2^0 = 1 The next two arrays are subset_size 2^1 = 2 The next two arrays are subset_size 2^2 = 4 The last two arrays are subset_size = sample_size = 10 Doesn’t generate one with subset_size 8 as double that is larger than sample_size Therefore this function returns 4, and the sample aal array is 4 * 2 Args:

alct (bool): Boolean for ALCT output sample_size (int): Sample size max_summary_id (int): Max summary ID

Returns:

num_subsets (int): Number of subsets

oasislmf.pytools.aal.manager.get_weighted_means(vec_sample_sum_loss, weighting, sidx, end_sidx)

Get sum of weighted mean and weighted mean_squared Args:

vec_sample_sum_loss (ndarray[_AAL_REC_DTYPE]): Vector for sample sum losses weighting (float): Weighting value sidx (int): start index end_sidx (int): end index

Returns:

weighted_mean (float): Sum weighted mean weighted_mean_squared (float): Sum weighted mean squared

oasislmf.pytools.aal.manager.do_calc_end(period_no, no_of_periods, period_weights, sample_size, curr_summary_id, max_summary_id, vec_analytical_aal, vecs_sample_aal, vec_used_summary_id, vec_sample_sum_loss)

Updates Analytical and Sample AAL vectors from sample sum losses Args:

period_no (int): Period Number no_of_periods (int): Number of periods period_weights (ndarray[periods_dtype]): Period Weights sample_size (int): Sample Size curr_summary_id (int): Current summary_id max_summary_id (int): Max summary_id vec_analytical_aal (ndarray[_AAL_REC_DTYPE]): Vector for Analytical AAL vecs_sample_aal (ndarray[_AAL_REC_PERIODS_DTYPE]): Vector for Sample AAL vec_used_summary_id (ndarray[bool]): vector to store if summary_id is used vec_sample_sum_loss (ndarray[_AAL_REC_DTYPE]): Vector for sample sum losses

oasislmf.pytools.aal.manager.read_losses(summary_fin, cursor, vec_sample_sum_loss)

Read losses from summary_fin starting at cursor, populate vec_sample_sum_loss Args:

summary_fin (np.memmap): summary file memmap cursor (int): data offset for reading binary files (ndarray[_AAL_REC_DTYPE]): Vector for sample sum losses

Returns:

cursor (int): data offset for reading binary files

oasislmf.pytools.aal.manager.skip_losses(summary_fin, cursor)

Skip through losses in summary_fin starting at cursor Args:

summary_fin (np.memmap): summary file memmap cursor (int): data offset for reading binary files

Returns:

cursor (int): data offset for reading binary files

oasislmf.pytools.aal.manager.run_aal(memmaps, no_of_periods, period_weights, sample_size, max_summary_id, files_handles, vec_analytical_aal, vecs_sample_aal, vec_used_summary_id)

Run AAL calculation loop to populate vec data Args:

memmaps (List[np.memmap]): List of temporary file memmaps no_of_periods (int): Number of periods period_weights (ndarray[periods_dtype]): Period Weights sample_size (int): Sample Size max_summary_id (int): Max summary_id files_handles (List[np.memmap]): List of memmaps for summary files data vec_analytical_aal (ndarray[_AAL_REC_DTYPE]): Vector for Analytical AAL vecs_sample_aal (ndarray[_AAL_REC_PERIODS_DTYPE]): Vector for Sample AAL vec_used_summary_id (ndarray[bool]): vector to store if summary_id is used

oasislmf.pytools.aal.manager.calculate_mean_stddev(observable_sum, observable_squared_sum, number_of_observations)

Compute the mean and standard deviation from the sum and squared sum of an observable Args:

observable_sum (ndarray[oasis_float]): Observable sum observable_squared_sum (ndarray[oasis_float]): Observable squared sum number_of_observations (int | ndarray[int]): number of observations

Returns:

mean (ndarray[oasis_float]): Mean std (ndarray[oasis_float]): Standard Deviation

oasislmf.pytools.aal.manager.get_aal_data(vec_analytical_aal, vecs_sample_aal, vec_used_summary_id, sample_size, no_of_periods)

Generate AAL csv data Args:

vec_analytical_aal (ndarray[_AAL_REC_DTYPE]): Vector for Analytical AAL vecs_sample_aal (ndarray[_AAL_REC_PERIODS_DTYPE]): Vector for Sample AAL vec_used_summary_id (ndarray[bool]): vector to store if summary_id is used sample_size (int): Sample Size no_of_periods (int): Number of periods

Returns:

aal_data (List[Tuple]): AAL csv data

oasislmf.pytools.aal.manager.get_aal_data_meanonly(vec_analytical_aal, vecs_sample_aal, vec_used_summary_id, sample_size, no_of_periods)

Generate AAL csv data Args:

vec_analytical_aal (ndarray[_AAL_REC_DTYPE]): Vector for Analytical AAL vecs_sample_aal (ndarray[_AAL_REC_PERIODS_DTYPE]): Vector for Sample AAL vec_used_summary_id (ndarray[bool]): vector to store if summary_id is used sample_size (int): Sample Size no_of_periods (int): Number of periods

Returns:

aal_data (List[Tuple]): AAL csv data

oasislmf.pytools.aal.manager.calculate_confidence_interval(std_err, confidence_level)

Calculate the confidence interval based on standard error and confidence level. Args:

std_err (float): The standard error. confidence_level (float): The confidence level (e.g., 0.95 for 95%).

Returns:

confidence interval (float): The confidence interval.

oasislmf.pytools.aal.manager.get_alct_data(vecs_sample_aal, max_summary_id, sample_size, no_of_periods, confidence)

Generate ALCT csv data Args:

vecs_sample_aal (ndarray[_AAL_REC_PERIODS_DTYPE]): Vector for Sample AAL max_summary_id (int): Max summary_id sample_size (int): Sample Size no_of_periods (int): Number of periods confidence (float): Confidence level between 0 and 1, default 0.95

Returns:

alct_data (List[List]): ALCT csv data

oasislmf.pytools.aal.manager.run(run_dir, subfolder, aal_output_file=None, alct_output_file=None, meanonly=False, noheader=False, confidence=0.95, output_format='csv')

Runs AAL calculations Args:

run_dir (str | os.PathLike): Path to directory containing required files structure subfolder (str): Workspace subfolder inside <run_dir>/work/<subfolder> aal_output_file (str, optional): Path to AAL output file. Defaults to None alct_output_file (str, optional): Path to ALCT output file. Defaults to None meanonly (bool): Boolean value to output AAL with mean only noheader (bool): Boolean value to skip header in output file confidence (float): Confidence level between 0 and 1, default 0.95 output_format (str): Output format extension. Defaults to “csv”.

oasislmf.pytools.aal.manager.main(run_dir='.', subfolder=None, aal=None, alct=None, meanonly=False, noheader=False, confidence=0.95, ext='csv', **kwargs)