oasislmf.pytools.kat.manager ============================ .. py:module:: oasislmf.pytools.kat.manager Attributes ---------- .. autoapisummary:: oasislmf.pytools.kat.manager.logger oasislmf.pytools.kat.manager.KAT_SELT oasislmf.pytools.kat.manager.KAT_MELT oasislmf.pytools.kat.manager.KAT_QELT oasislmf.pytools.kat.manager.KAT_SPLT oasislmf.pytools.kat.manager.KAT_MPLT oasislmf.pytools.kat.manager.KAT_QPLT oasislmf.pytools.kat.manager.KAT_MAP Functions --------- .. autoapisummary:: oasislmf.pytools.kat.manager.check_file_extensions oasislmf.pytools.kat.manager.find_csv_with_header oasislmf.pytools.kat.manager.check_correct_headers oasislmf.pytools.kat.manager.get_header_idxs oasislmf.pytools.kat.manager.csv_concat_unsorted oasislmf.pytools.kat.manager.csv_concat_sort_by_headers oasislmf.pytools.kat.manager.bin_concat_unsorted oasislmf.pytools.kat.manager.merge_elt_data oasislmf.pytools.kat.manager.merge_plt_data oasislmf.pytools.kat.manager.bin_concat_sort_by_headers oasislmf.pytools.kat.manager.run oasislmf.pytools.kat.manager.main Module Contents --------------- .. py:data:: logger .. py:data:: KAT_SELT :value: 0 .. py:data:: KAT_MELT :value: 1 .. py:data:: KAT_QELT :value: 2 .. py:data:: KAT_SPLT :value: 3 .. py:data:: KAT_MPLT :value: 4 .. py:data:: KAT_QPLT :value: 5 .. py:data:: KAT_MAP .. py:function:: check_file_extensions(file_paths) Check file path extensions are all identical Args: file_paths (List[str | os.PathLike]): List of csv file paths. Returns: ext (str): file extension as a str .. py:function:: find_csv_with_header(stack, file_paths) Find and check csv files for consistent and present headers Args: stack (ExitStack): Exit Stack. file_paths (List[str | os.PathLike]): List of csv file paths. Returns: files_with_header (List[bool]): Bool list of files with header present header (str): Header to write .. py:function:: check_correct_headers(headers, file_type) Checks headers found in csv file matches excpected headers for file type Args: headers (List[str]): Headers file_type (int): File type int matching KAT_NAMES index .. py:function:: get_header_idxs(headers, headers_to_search) Search for index of headers_to_search in headers of csv file Args: headers (List[str]): Headers headers_to_search (List[str]): Headers to search Returns: idxs (List[int]): Indexes of searched headers .. py:function:: csv_concat_unsorted(stack, file_paths, files_with_header, headers, out_file) Concats CSV files in order they are passed in. Args: stack (ExitStack): Exit Stack. file_paths (List[str | os.PathLike]): List of csv file paths. files_with_header (List[bool]): Bool list of files with header present headers (List[str]): Headers to write out_file (str | os.PathLike): Output Concatenated CSV file. .. py:function:: csv_concat_sort_by_headers(stack, file_paths, files_with_header, headers, header_idxs, sort_fn, out_file, **sort_kwargs) Concats CSV files in order determined by the header_idxs and sort_fn Args: stack (ExitStack): Exit Stack. file_paths (List[str | os.PathLike]): List of csv file paths. files_with_header (List[bool]): Bool list of files with header present headers (List[str]): Headers to write header_idxs (List[int]): Indices of headers to sort by sort_fn (Callable[[List[int]], Any]): Sort function to apply to header_idxs out_file (str | os.PathLike): Output Concatenated CSV file. .. py:function:: bin_concat_unsorted(stack, file_paths, out_file) Concats Binary files in order they are passed in. Args: stack (ExitStack): Exit Stack. file_paths (List[str | os.PathLike]): List of bin file paths. out_file (str | os.PathLike): Output Concatenated Binary file. .. py:function:: merge_elt_data(memmaps) Merge sorted chunks using a k-way merge algorithm Args: memmaps (List[np.memmap]): List of temporary file memmaps Yields: buffer (ndarray): yields sorted buffer from memmaps .. py:function:: merge_plt_data(memmaps) Merge sorted chunks using a k-way merge algorithm Args: memmaps (List[np.memmap]): List of temporary file memmaps Yields: buffer (ndarray): yields sorted buffer from memmaps .. py:function:: bin_concat_sort_by_headers(stack, file_paths, file_type, out_type, out_file) Concats Binary files in order determined out_type and their respective merge functions Args: stack (ExitStack): Exit Stack. file_paths (List[str | os.PathLike]): List of csv file paths. file_type (int): File type int matching KAT_NAMES index out_type (str): Out type str between "elt" and "plt" out_file (str | os.PathLike): Output Concatenated CSV file. .. py:function:: run(out_file, file_type=None, files_in=None, dir_in=None, concatenate_selt=False, concatenate_melt=False, concatenate_qelt=False, concatenate_splt=False, concatenate_mplt=False, concatenate_qplt=False, unsorted=False) Concatenate CSV files (optionally sorted) Args: out_file (str | os.PathLike): Output Concatenated CSV file. file_type (str, optional): Input file type suffix, if not discernible from input files. Defaults to None. files_in (List[str | os.PathLike], optional): Individual CSV file paths to concatenate. Defaults to None. dir_in (str | os.PathLike, optional): Path to the directory containing files for concatenation. Defaults to None. concatenate_selt (bool, optional): Concatenate SELT CSV file. Defaults to False. concatenate_melt (bool, optional): Concatenate MELT CSV file. Defaults to False. concatenate_qelt (bool, optional): Concatenate QELT CSV file. Defaults to False. concatenate_splt (bool, optional): Concatenate SPLT CSV file. Defaults to False. concatenate_mplt (bool, optional): Concatenate MPLT CSV file. Defaults to False. concatenate_qplt (bool, optional): Concatenate QPLT CSV file. Defaults to False. unsorted (bool, optional): Do not sort by event/period ID. Defaults to False. .. py:function:: main(out=None, file_type=None, files_in=None, dir_in=None, selt=False, melt=False, qelt=False, splt=False, mplt=False, qplt=False, unsorted=False, **kwargs)