oasislmf.pytools.kat.manager

Attributes

logger

Functions

check_file_extensions(file_paths)	Check file path extensions are all identical
find_csv_with_header(stack, file_paths)	Find and check csv files for consistent and present headers
check_correct_headers(headers, file_type)	Checks headers found in csv file matches excpected headers for file type
get_header_idxs(headers, headers_to_search)	Search for index of headers_to_search in headers of csv file
csv_concat_unsorted(stack, file_paths, ...)	Concats CSV files in order they are passed in.
csv_concat_sort_by_headers(stack, file_paths, ...)	Concats CSV files in order determined by the header_idxs and sort_fn
bin_concat_unsorted(stack, file_paths, out_file)	Concats Binary files in order they are passed in.
merge_elt_data(memmaps)	Merge sorted chunks using a k-way merge algorithm
merge_plt_data(memmaps)	Merge sorted chunks using a k-way merge algorithm
bin_concat_sort_by_headers(stack, file_paths, ...)	Concats Binary files in order determined out_type and their respective merge functions
parquet_concat_unsorted(file_paths, out_file)	Concats Parquet files in order they are passed in.
parquet_kway_merge(file_paths, keys[, chunk_size])	Merge sorted chunks using a k-way merge algorithm
parquet_concat_sorted(file_paths, out_type, out_file)	Concats Parquet files in order determined out_type and their respective merge functions
run(out_file[, file_type, files_in, dir_in, ...])	Concatenate output files (optionally sorted)
main([out, file_type, files_in, dir_in, selt, melt, ...])

Module Contents

oasislmf.pytools.kat.manager.logger

oasislmf.pytools.kat.manager.check_file_extensions(file_paths)

Check file path extensions are all identical

Parameters:

file_paths (List[str | os.PathLike]) – List of csv file paths.

Returns:

file extension as a str

Return type:

ext (str)

oasislmf.pytools.kat.manager.find_csv_with_header(stack, file_paths)

Find and check csv files for consistent and present headers

Parameters:

• stack (ExitStack) – Exit Stack.

• file_paths (List[str | os.PathLike]) – List of csv file paths.

Returns:

Bool list of files with header present header (str): Header to write

Return type:

files_with_header (List[bool])

oasislmf.pytools.kat.manager.check_correct_headers(headers, file_type)

Checks headers found in csv file matches excpected headers for file type

Parameters:

• headers (List[str]) – Headers

• file_type (int) – File type int matching KAT_NAMES index

oasislmf.pytools.kat.manager.get_header_idxs(headers, headers_to_search)

Search for index of headers_to_search in headers of csv file

Parameters:

• headers (List[str]) – Headers

• headers_to_search (List[str]) – Headers to search

Returns:

Indexes of searched headers

Return type:

idxs (List[int])

oasislmf.pytools.kat.manager.csv_concat_unsorted(stack, file_paths, files_with_header, headers, out_file)

Concats CSV files in order they are passed in.

Parameters:

• stack (ExitStack) – Exit Stack.

• file_paths (List[str | os.PathLike]) – List of csv file paths.

• files_with_header (List[bool]) – Bool list of files with header present

• headers (List[str]) – Headers to write

• out_file (str | os.PathLike) – Output Concatenated CSV file.

oasislmf.pytools.kat.manager.csv_concat_sort_by_headers(stack, file_paths, files_with_header, headers, header_idxs, sort_fn, out_file, **sort_kwargs)

Concats CSV files in order determined by the header_idxs and sort_fn

Parameters:

• stack (ExitStack) – Exit Stack.

• file_paths (List[str | os.PathLike]) – List of csv file paths.

• files_with_header (List[bool]) – Bool list of files with header present

• headers (List[str]) – Headers to write

• header_idxs (List[int]) – Indices of headers to sort by

• sort_fn (Callable[[List[int]], Any]) – Sort function to apply to header_idxs

• out_file (str | os.PathLike) – Output Concatenated CSV file.

• **sort_kwargs – additional keyword arguments passed to sort_fn

oasislmf.pytools.kat.manager.bin_concat_unsorted(stack, file_paths, out_file)

Concats Binary files in order they are passed in.

Parameters:

• stack (ExitStack) – Exit Stack.

• file_paths (List[str | os.PathLike]) – List of bin file paths.

• out_file (str | os.PathLike) – Output Concatenated Binary file.

oasislmf.pytools.kat.manager.merge_elt_data(memmaps)

Merge sorted chunks using a k-way merge algorithm

Parameters:

memmaps (List[np.memmap]) – List of temporary file memmaps

Yields:

buffer (ndarray) – yields sorted buffer from memmaps

oasislmf.pytools.kat.manager.merge_plt_data(memmaps)

Merge sorted chunks using a k-way merge algorithm

Parameters:

memmaps (List[np.memmap]) – List of temporary file memmaps

Yields:

buffer (ndarray) – yields sorted buffer from memmaps

oasislmf.pytools.kat.manager.bin_concat_sort_by_headers(stack, file_paths, file_type, out_type, out_file)

Concats Binary files in order determined out_type and their respective merge functions

Parameters:

• stack (ExitStack) – Exit Stack.

• file_paths (List[str | os.PathLike]) – List of bin file paths.

• file_type (int) – File type int matching KAT_NAMES index

• out_type (str) – Out type str between “elt” and “plt”

• out_file (str | os.PathLike) – Output Concatenated Binary file.

oasislmf.pytools.kat.manager.parquet_concat_unsorted(file_paths, out_file)

Concats Parquet files in order they are passed in.

Parameters:

• file_paths (List[str | os.PathLike]) – List of parquet file paths.

• out_file (str | os.PathLike) – Output Concatenated Parquet file.

oasislmf.pytools.kat.manager.parquet_kway_merge(file_paths, keys, chunk_size=100000)

Merge sorted chunks using a k-way merge algorithm

Parameters:

• file_paths (List[str | os.PathLike]) – List of parquet file paths.

• keys (List[str]) – List of keys to sort by

• chunk_size (int) – Chunk size for reading parquet files. Defaults to 100000.

Yields:

buffer (pa.Table) – yields sorted pyarrow table from input files

oasislmf.pytools.kat.manager.parquet_concat_sorted(file_paths, out_type, out_file, chunk_size=100000)

Concats Parquet files in order determined out_type and their respective merge functions

Parameters:

• file_paths (List[str | os.PathLike]) – List of parquet file paths.

• out_type (str) – Out type str between “elt” and “plt”

• out_file (str | os.PathLike) – Output Concatenated Parquet file.

• chunk_size (int) – Chunk size for reading parquet files. Defaults to 100000.

oasislmf.pytools.kat.manager.run(out_file, file_type=None, files_in=None, dir_in=None, concatenate_selt=False, concatenate_melt=False, concatenate_qelt=False, concatenate_splt=False, concatenate_mplt=False, concatenate_qplt=False, unsorted=False)

Concatenate output files (optionally sorted)

Parameters:

• out_file (str | os.PathLike) – Output Concatenated file.

• file_type (str, optional) – Input file type suffix, if not discernible from input files. Defaults to None.

• files_in (List[str | os.PathLike], optional) – Individual file paths to concatenate. Defaults to None.

• dir_in (str | os.PathLike, optional) – Path to the directory containing files for concatenation. Defaults to None.

• concatenate_selt (bool, optional) – Concatenate SELT file. Defaults to False.

• concatenate_melt (bool, optional) – Concatenate MELT file. Defaults to False.

• concatenate_qelt (bool, optional) – Concatenate QELT file. Defaults to False.

• concatenate_splt (bool, optional) – Concatenate SPLT file. Defaults to False.

• concatenate_mplt (bool, optional) – Concatenate MPLT file. Defaults to False.

• concatenate_qplt (bool, optional) – Concatenate QPLT file. Defaults to False.

• unsorted (bool, optional) – Do not sort by event/period ID. Defaults to False.

oasislmf.pytools.kat.manager.main(out=None, file_type=None, files_in=None, dir_in=None, selt=False, melt=False, qelt=False, splt=False, mplt=False, qplt=False, unsorted=False, **kwargs)