oasislmf.pytools.aal.manager

Attributes

logger

OASIS_AAL_MEMORY

Functions

process_bin_file(fbin, offset, occ_csr, ...)

Reads summary<n>.bin file event_ids and summary_ids to populate summaries_data

process_idx_file(fbin, idx_data, idx_cursor, occ_csr, ...)

Use a pre-built .idx file to populate summaries_data without scanning sample records.

sort_and_save_chunk(summaries_data, temp_file_path)

Sort a chunk of summaries data and save it to a temporary file.

merge_sorted_chunks(memmaps)

Merge sorted chunks using a k-way merge algorithm and yield next smallest row

get_summaries_data(path, files_handles, occ_csr, ...)

Gets the indexed summaries data, ordered with k-way merge if not enough memory.

summary_index(path, occ_csr, stack)

Index the summary binary outputs.

read_input_files(run_dir)

Reads all input files and returns a dict of relevant data

get_num_subsets(alct, sample_size, max_summary_id)

Gets the number of subsets required to generates the Sample AAL np map for subset sizes up to sample_size

get_weighted_means(vec_sample_sum_loss, weighting, ...)

Get sum of weighted mean and weighted mean_squared

do_calc_end(period_no, no_of_periods, period_weights, ...)

Updates Analytical and Sample AAL vectors from sample sum losses

read_losses(summary_fin, cursor, vec_sample_sum_loss)

Read losses from summary_fin starting at cursor, populate vec_sample_sum_loss

skip_losses(summary_fin, cursor)

Skip through losses in summary_fin starting at cursor

run_aal(memmaps, no_of_periods, period_weights, ...)

Run AAL calculation loop to populate vec data

calculate_mean_stddev(observable_sum, ...)

Compute the mean and standard deviation from the sum and squared sum of an observable

get_aal_data(vec_analytical_aal, vecs_sample_aal, ...)

Generate AAL csv data

get_aal_data_meanonly(vec_analytical_aal, ...)

Generate AAL csv data

calculate_confidence_interval(std_err, confidence_level)

Calculate the confidence interval based on standard error and confidence level.

get_alct_data(vecs_sample_aal, max_summary_id, ...)

Generate ALCT csv data

run(run_dir, subfolder[, aal_output_file, ...])

Runs AAL calculations

main([run_dir, subfolder, aal, alct, meanonly, ...])

Module Contents

oasislmf.pytools.aal.manager.logger[source]
oasislmf.pytools.aal.manager.OASIS_AAL_MEMORY[source]
oasislmf.pytools.aal.manager.process_bin_file(fbin, offset, occ_csr, summaries_data, summaries_idx, file_index)[source]

Reads summary<n>.bin file event_ids and summary_ids to populate summaries_data

Parameters:

  • fbin (np.memmap) – summary binary memmap

  • offset (int) – file offset to read from

  • occ_csr (OccurrenceCSR) – id_index-backed CSR occurrence map

  • summaries_data (ndarray[_SUMMARIES_DTYPE]) – Index summary data (summaries.idx data)

  • summaries_idx (int) – current index reached in summaries_data

  • file_index (int) – Summary bin file index

Returns:

current index reached in summaries_data resize_flag (bool): flag to indicate whether to resize summaries_data when full offset (int): file offset to read from

Return type:

summaries_idx (int)

oasislmf.pytools.aal.manager.process_idx_file(fbin, idx_data, idx_cursor, occ_csr, summaries_data, summaries_idx, file_index)[source]

Use a pre-built .idx file to populate summaries_data without scanning sample records.

Instead of reading the entire .bin sequentially (including all loss records just to advance the cursor), each .idx entry gives the byte offset of an event header directly. Only the 4-byte event_id is read from the .bin per entry; loss records are never touched here.

Parameters:

  • fbin (np.memmap) – summary binary memmap (dtype u1)

  • idx_data (np.memmap) – index file memmap (dtype summary_stream_index_dtype)

  • idx_cursor (int) – current position in idx_data to resume from

  • occ_csr (OccurrenceCSR) – id_index-backed CSR occurrence map

  • summaries_data (ndarray[_SUMMARIES_DTYPE]) – output index buffer

  • summaries_idx (int) – next free slot in summaries_data

  • file_index (int) – which .bin file this is (used as file_idx in output)

Returns:

updated cursor into summaries_data resize_flag (bool): True if summaries_data is full and must be flushed before resuming idx_cursor (int): position in idx_data to resume from after a flush

Return type:

summaries_idx (int)

oasislmf.pytools.aal.manager.sort_and_save_chunk(summaries_data, temp_file_path)[source]

Sort a chunk of summaries data and save it to a temporary file.

Parameters:

  • summaries_data (ndarray[_SUMMARIES_DTYPE]) – Indexed summary data

  • temp_file_path (str | os.PathLike) – Path to temporary file

oasislmf.pytools.aal.manager.merge_sorted_chunks(memmaps)[source]

Merge sorted chunks using a k-way merge algorithm and yield next smallest row

Parameters:

memmaps (List[np.memmap]) – List of temporary file memmaps

Yields:

smallest_row (ndarray[_SUMMARIES_DTYPE]) – yields the next smallest row from sorted summaries partial files

oasislmf.pytools.aal.manager.get_summaries_data(path, files_handles, occ_csr, aal_max_memory, idx_handles=None)[source]

Gets the indexed summaries data, ordered with k-way merge if not enough memory.

When idx_handles[i] is not None (a memmap of summary_stream_index_dtype records), uses process_idx_file for that file — seeking directly to each event header via pre-computed offsets, never reading sample records during indexing. Falls back to process_bin_file (full sequential scan) for any file without a paired .idx.

Parameters:

  • path (os.PathLike) – Path to the workspace folder containing summary binaries

  • files_handles (List[np.memmap]) – List of memmaps for summary files data

  • occ_csr (OccurrenceCSR) – id_index-backed CSR occurrence map

  • aal_max_memory (float) – OASIS_AAL_MEMORY value (has to be passed in as numba won’t update from environment variable)

  • idx_handles (List[np.memmap | None] | None) – Per-file .idx memmaps, or None to use sequential scan for all files

Returns:

List of temporary file memmaps max_summary_id (int): Max summary ID

Return type:

memmaps (List[np.memmap])

oasislmf.pytools.aal.manager.summary_index(path, occ_csr, stack)[source]

Index the summary binary outputs.

If a .idx file (summary_stream_index_dtype) exists alongside a .bin file, uses process_idx_file to build the index without scanning sample records. Falls back to process_bin_file (full sequential scan) for any .bin without a paired .idx.

Parameters:

  • path (os.PathLike) – Path to the workspace folder containing summary binaries

  • occ_csr (OccurrenceCSR) – id_index-backed CSR occurrence map

  • stack (ExitStack) – Exit stack

Returns:

List of memmaps for summary files data sample_size (int): Sample size max_summary_id (int): Max summary ID memmaps (List[np.memmap]): List of temporary file memmaps

Return type:

files_handles (List[np.memmap])

oasislmf.pytools.aal.manager.read_input_files(run_dir)[source]

Reads all input files and returns a dict of relevant data

Parameters:

run_dir (str | os.PathLike) – Path to directory containing required files structure

Returns:

A dict of relevent data extracted from files

Return type:

file_data (Dict[str, Any])

oasislmf.pytools.aal.manager.get_num_subsets(alct, sample_size, max_summary_id)[source]

Gets the number of subsets required to generates the Sample AAL np map for subset sizes up to sample_size Example: sample_size[10], max_summary_id[2] generates following ndarray [ # subset_size, mean, mean_squared, mean_period [0, 0, 0], # subset_size = 1 , summary_id = 1 [0, 0, 0], # subset_size = 1 , summary_id = 2 [0, 0, 0], # subset_size = 2 , summary_id = 1 [0, 0, 0], # subset_size = 2 , summary_id = 2 [0, 0, 0], # subset_size = 4 , summary_id = 1 [0, 0, 0], # subset_size = 4 , summary_id = 2 [0, 0, 0], # subset_size = 10 , summary_id = 1, subset_size = sample_size [0, 0, 0], # subset_size = 10 , summary_id = 2, subset_size = sample_size ] Subset_size is implicit based on position in array, grouped by max_summary_id So first two arrays are subset_size 2^0 = 1 The next two arrays are subset_size 2^1 = 2 The next two arrays are subset_size 2^2 = 4 The last two arrays are subset_size = sample_size = 10 Doesn’t generate one with subset_size 8 as double that is larger than sample_size Therefore this function returns 4, and the sample aal array is 4 * 2

Parameters:

  • alct (bool) – Boolean for ALCT output

  • sample_size (int) – Sample size

  • max_summary_id (int) – Max summary ID

Returns:

Number of subsets

Return type:

num_subsets (int)

oasislmf.pytools.aal.manager.get_weighted_means(vec_sample_sum_loss, weighting, sidx, end_sidx)[source]

Get sum of weighted mean and weighted mean_squared

Parameters:

  • vec_sample_sum_loss (ndarray[_AAL_REC_DTYPE]) – Vector for sample sum losses

  • weighting (float) – Weighting value

  • sidx (int) – start index

  • end_sidx (int) – end index

Returns:

Sum weighted mean weighted_mean_squared (float): Sum weighted mean squared

Return type:

weighted_mean (float)

oasislmf.pytools.aal.manager.do_calc_end(period_no, no_of_periods, period_weights, sample_size, curr_summary_id, max_summary_id, vec_analytical_aal, vecs_sample_aal, vec_used_summary_id, vec_sample_sum_loss)[source]

Updates Analytical and Sample AAL vectors from sample sum losses

Parameters:

  • period_no (int) – Period Number

  • no_of_periods (int) – Number of periods

  • period_weights (ndarray[periods_dtype]) – Period Weights

  • sample_size (int) – Sample Size

  • curr_summary_id (int) – Current summary_id

  • max_summary_id (int) – Max summary_id

  • vec_analytical_aal (ndarray[_AAL_REC_DTYPE]) – Vector for Analytical AAL

  • vecs_sample_aal (ndarray[_AAL_REC_PERIODS_DTYPE]) – Vector for Sample AAL

  • vec_used_summary_id (ndarray[bool]) – vector to store if summary_id is used

  • vec_sample_sum_loss (ndarray[_AAL_REC_DTYPE]) – Vector for sample sum losses

oasislmf.pytools.aal.manager.read_losses(summary_fin, cursor, vec_sample_sum_loss)[source]

Read losses from summary_fin starting at cursor, populate vec_sample_sum_loss

Parameters:

  • summary_fin (np.memmap) – summary file memmap

  • cursor (int) – data offset for reading binary files

  • vec_sample_sum_loss (ndarray[_AAL_REC_DTYPE]) – Vector for sample sum losses

Returns:

data offset for reading binary files

Return type:

cursor (int)

oasislmf.pytools.aal.manager.skip_losses(summary_fin, cursor)[source]

Skip through losses in summary_fin starting at cursor

Parameters:

  • summary_fin (np.memmap) – summary file memmap

  • cursor (int) – data offset for reading binary files

Returns:

data offset for reading binary files

Return type:

cursor (int)

oasislmf.pytools.aal.manager.run_aal(memmaps, no_of_periods, period_weights, sample_size, max_summary_id, files_handles, vec_analytical_aal, vecs_sample_aal, vec_used_summary_id)[source]

Run AAL calculation loop to populate vec data

Parameters:

  • memmaps (List[np.memmap]) – List of temporary file memmaps

  • no_of_periods (int) – Number of periods

  • period_weights (ndarray[periods_dtype]) – Period Weights

  • sample_size (int) – Sample Size

  • max_summary_id (int) – Max summary_id

  • files_handles (List[np.memmap]) – List of memmaps for summary files data

  • vec_analytical_aal (ndarray[_AAL_REC_DTYPE]) – Vector for Analytical AAL

  • vecs_sample_aal (ndarray[_AAL_REC_PERIODS_DTYPE]) – Vector for Sample AAL

  • vec_used_summary_id (ndarray[bool]) – vector to store if summary_id is used

oasislmf.pytools.aal.manager.calculate_mean_stddev(observable_sum, observable_squared_sum, number_of_observations)[source]

Compute the mean and standard deviation from the sum and squared sum of an observable

Parameters:

  • observable_sum (ndarray[oasis_float]) – Observable sum

  • observable_squared_sum (ndarray[oasis_float]) – Observable squared sum

  • number_of_observations (int | ndarray[int]) – number of observations

Returns:

Mean std (ndarray[oasis_float]): Standard Deviation

Return type:

mean (ndarray[oasis_float])

oasislmf.pytools.aal.manager.get_aal_data(vec_analytical_aal, vecs_sample_aal, vec_used_summary_id, sample_size, no_of_periods)[source]

Generate AAL csv data

Parameters:

  • vec_analytical_aal (ndarray[_AAL_REC_DTYPE]) – Vector for Analytical AAL

  • vecs_sample_aal (ndarray[_AAL_REC_PERIODS_DTYPE]) – Vector for Sample AAL

  • vec_used_summary_id (ndarray[bool]) – vector to store if summary_id is used

  • sample_size (int) – Sample Size

  • no_of_periods (int) – Number of periods

Returns:

AAL csv data

Return type:

aal_data (List[Tuple])

oasislmf.pytools.aal.manager.get_aal_data_meanonly(vec_analytical_aal, vecs_sample_aal, vec_used_summary_id, sample_size, no_of_periods)[source]

Generate AAL csv data

Parameters:

  • vec_analytical_aal (ndarray[_AAL_REC_DTYPE]) – Vector for Analytical AAL

  • vecs_sample_aal (ndarray[_AAL_REC_PERIODS_DTYPE]) – Vector for Sample AAL

  • vec_used_summary_id (ndarray[bool]) – vector to store if summary_id is used

  • sample_size (int) – Sample Size

  • no_of_periods (int) – Number of periods

Returns:

AAL csv data

Return type:

aal_data (List[Tuple])

oasislmf.pytools.aal.manager.calculate_confidence_interval(std_err, confidence_level)[source]

Calculate the confidence interval based on standard error and confidence level.

Parameters:

  • std_err (float) – The standard error.

  • confidence_level (float) – The confidence level (e.g., 0.95 for 95%).

Returns:

The confidence interval.

Return type:

confidence interval (float)

oasislmf.pytools.aal.manager.get_alct_data(vecs_sample_aal, max_summary_id, sample_size, no_of_periods, confidence)[source]

Generate ALCT csv data

Parameters:

  • vecs_sample_aal (ndarray[_AAL_REC_PERIODS_DTYPE]) – Vector for Sample AAL

  • max_summary_id (int) – Max summary_id

  • sample_size (int) – Sample Size

  • no_of_periods (int) – Number of periods

  • confidence (float) – Confidence level between 0 and 1, default 0.95

Returns:

ALCT csv data

Return type:

alct_data (List[List])

oasislmf.pytools.aal.manager.run(run_dir, subfolder, aal_output_file=None, alct_output_file=None, meanonly=False, noheader=False, confidence=0.95, output_format='csv')[source]

Runs AAL calculations

Parameters:

  • run_dir (str | os.PathLike) – Path to directory containing required files structure

  • subfolder (str) – Workspace subfolder inside <run_dir>/work/<subfolder>

  • aal_output_file (str, optional) – Path to AAL output file. Defaults to None

  • alct_output_file (str, optional) – Path to ALCT output file. Defaults to None

  • meanonly (bool) – Boolean value to output AAL with mean only

  • noheader (bool) – Boolean value to skip header in output file

  • confidence (float) – Confidence level between 0 and 1, default 0.95

  • output_format (str) – Output format extension. Defaults to “csv”.

oasislmf.pytools.aal.manager.main(run_dir='.', subfolder=None, aal=None, alct=None, meanonly=False, noheader=False, confidence=0.95, ext='csv', **kwargs)[source]