oasislmf.pytools.kat.manager¶
Attributes¶
Functions¶
|
Check file path extensions are all identical |
|
Find and check csv files for consistent and present headers |
|
Checks headers found in csv file matches excpected headers for file type |
|
Search for index of headers_to_search in headers of csv file |
|
Concats CSV files in order they are passed in. |
|
Concats CSV files in order determined by the header_idxs and sort_fn |
|
Concats Binary files in order they are passed in. |
|
Merge sorted chunks using a k-way merge algorithm |
|
Merge sorted chunks using a k-way merge algorithm |
|
Concats Binary files in order determined out_type and their respective merge functions |
|
Concatenate CSV files (optionally sorted) |
|
Module Contents¶
- oasislmf.pytools.kat.manager.check_file_extensions(file_paths)[source]¶
Check file path extensions are all identical Args:
file_paths (List[str | os.PathLike]): List of csv file paths.
- Returns:
ext (str): file extension as a str
- oasislmf.pytools.kat.manager.find_csv_with_header(stack, file_paths)[source]¶
Find and check csv files for consistent and present headers Args:
stack (ExitStack): Exit Stack. file_paths (List[str | os.PathLike]): List of csv file paths.
- Returns:
files_with_header (List[bool]): Bool list of files with header present header (str): Header to write
- oasislmf.pytools.kat.manager.check_correct_headers(headers, file_type)[source]¶
Checks headers found in csv file matches excpected headers for file type Args:
headers (List[str]): Headers file_type (int): File type int matching KAT_NAMES index
- oasislmf.pytools.kat.manager.get_header_idxs(headers, headers_to_search)[source]¶
Search for index of headers_to_search in headers of csv file Args:
headers (List[str]): Headers headers_to_search (List[str]): Headers to search
- Returns:
idxs (List[int]): Indexes of searched headers
- oasislmf.pytools.kat.manager.csv_concat_unsorted(stack, file_paths, files_with_header, headers, out_file)[source]¶
Concats CSV files in order they are passed in. Args:
stack (ExitStack): Exit Stack. file_paths (List[str | os.PathLike]): List of csv file paths. files_with_header (List[bool]): Bool list of files with header present headers (List[str]): Headers to write out_file (str | os.PathLike): Output Concatenated CSV file.
- oasislmf.pytools.kat.manager.csv_concat_sort_by_headers(stack, file_paths, files_with_header, headers, header_idxs, sort_fn, out_file, **sort_kwargs)[source]¶
Concats CSV files in order determined by the header_idxs and sort_fn Args:
stack (ExitStack): Exit Stack. file_paths (List[str | os.PathLike]): List of csv file paths. files_with_header (List[bool]): Bool list of files with header present headers (List[str]): Headers to write header_idxs (List[int]): Indices of headers to sort by sort_fn (Callable[[List[int]], Any]): Sort function to apply to header_idxs out_file (str | os.PathLike): Output Concatenated CSV file.
- oasislmf.pytools.kat.manager.bin_concat_unsorted(stack, file_paths, out_file)[source]¶
Concats Binary files in order they are passed in. Args:
stack (ExitStack): Exit Stack. file_paths (List[str | os.PathLike]): List of bin file paths. out_file (str | os.PathLike): Output Concatenated Binary file.
- oasislmf.pytools.kat.manager.merge_elt_data(memmaps)[source]¶
Merge sorted chunks using a k-way merge algorithm Args:
memmaps (List[np.memmap]): List of temporary file memmaps
- Yields:
buffer (ndarray): yields sorted buffer from memmaps
- oasislmf.pytools.kat.manager.merge_plt_data(memmaps)[source]¶
Merge sorted chunks using a k-way merge algorithm Args:
memmaps (List[np.memmap]): List of temporary file memmaps
- Yields:
buffer (ndarray): yields sorted buffer from memmaps
- oasislmf.pytools.kat.manager.bin_concat_sort_by_headers(stack, file_paths, file_type, out_type, out_file)[source]¶
Concats Binary files in order determined out_type and their respective merge functions Args:
stack (ExitStack): Exit Stack. file_paths (List[str | os.PathLike]): List of csv file paths. file_type (int): File type int matching KAT_NAMES index out_type (str): Out type str between “elt” and “plt” out_file (str | os.PathLike): Output Concatenated CSV file.
- oasislmf.pytools.kat.manager.run(out_file, file_type=None, files_in=None, dir_in=None, concatenate_selt=False, concatenate_melt=False, concatenate_qelt=False, concatenate_splt=False, concatenate_mplt=False, concatenate_qplt=False, unsorted=False)[source]¶
Concatenate CSV files (optionally sorted) Args:
out_file (str | os.PathLike): Output Concatenated CSV file. file_type (str, optional): Input file type suffix, if not discernible from input files. Defaults to None. files_in (List[str | os.PathLike], optional): Individual CSV file paths to concatenate. Defaults to None. dir_in (str | os.PathLike, optional): Path to the directory containing files for concatenation. Defaults to None. concatenate_selt (bool, optional): Concatenate SELT CSV file. Defaults to False. concatenate_melt (bool, optional): Concatenate MELT CSV file. Defaults to False. concatenate_qelt (bool, optional): Concatenate QELT CSV file. Defaults to False. concatenate_splt (bool, optional): Concatenate SPLT CSV file. Defaults to False. concatenate_mplt (bool, optional): Concatenate MPLT CSV file. Defaults to False. concatenate_qplt (bool, optional): Concatenate QPLT CSV file. Defaults to False. unsorted (bool, optional): Do not sort by event/period ID. Defaults to False.