oasislmf.pytools.kat.manager ============================ .. py:module:: oasislmf.pytools.kat.manager Attributes ---------- .. autoapisummary:: oasislmf.pytools.kat.manager.logger Functions --------- .. autoapisummary:: oasislmf.pytools.kat.manager.check_file_extensions oasislmf.pytools.kat.manager.find_csv_with_header oasislmf.pytools.kat.manager.check_correct_headers oasislmf.pytools.kat.manager.get_header_idxs oasislmf.pytools.kat.manager.csv_concat_unsorted oasislmf.pytools.kat.manager.csv_concat_sort_by_headers oasislmf.pytools.kat.manager.bin_concat_unsorted oasislmf.pytools.kat.manager.merge_elt_data oasislmf.pytools.kat.manager.merge_plt_data oasislmf.pytools.kat.manager.bin_concat_sort_by_headers oasislmf.pytools.kat.manager.parquet_concat_unsorted oasislmf.pytools.kat.manager.parquet_kway_merge oasislmf.pytools.kat.manager.parquet_concat_sorted oasislmf.pytools.kat.manager.run oasislmf.pytools.kat.manager.main Module Contents --------------- .. py:data:: logger .. py:function:: check_file_extensions(file_paths) Check file path extensions are all identical :param file_paths: List of csv file paths. :type file_paths: List[str | os.PathLike] :returns: file extension as a str :rtype: ext (str) .. py:function:: find_csv_with_header(stack, file_paths) Find and check csv files for consistent and present headers :param stack: Exit Stack. :type stack: ExitStack :param file_paths: List of csv file paths. :type file_paths: List[str | os.PathLike] :returns: Bool list of files with header present header (str): Header to write :rtype: files_with_header (List[bool]) .. py:function:: check_correct_headers(headers, file_type) Checks headers found in csv file matches excpected headers for file type :param headers: Headers :type headers: List[str] :param file_type: File type int matching KAT_NAMES index :type file_type: int .. py:function:: get_header_idxs(headers, headers_to_search) Search for index of headers_to_search in headers of csv file :param headers: Headers :type headers: List[str] :param headers_to_search: Headers to search :type headers_to_search: List[str] :returns: Indexes of searched headers :rtype: idxs (List[int]) .. py:function:: csv_concat_unsorted(stack, file_paths, files_with_header, headers, out_file) Concats CSV files in order they are passed in. :param stack: Exit Stack. :type stack: ExitStack :param file_paths: List of csv file paths. :type file_paths: List[str | os.PathLike] :param files_with_header: Bool list of files with header present :type files_with_header: List[bool] :param headers: Headers to write :type headers: List[str] :param out_file: Output Concatenated CSV file. :type out_file: str | os.PathLike .. py:function:: csv_concat_sort_by_headers(stack, file_paths, files_with_header, headers, header_idxs, sort_fn, out_file, **sort_kwargs) Concats CSV files in order determined by the header_idxs and sort_fn :param stack: Exit Stack. :type stack: ExitStack :param file_paths: List of csv file paths. :type file_paths: List[str | os.PathLike] :param files_with_header: Bool list of files with header present :type files_with_header: List[bool] :param headers: Headers to write :type headers: List[str] :param header_idxs: Indices of headers to sort by :type header_idxs: List[int] :param sort_fn: Sort function to apply to header_idxs :type sort_fn: Callable[[List[int]], Any] :param out_file: Output Concatenated CSV file. :type out_file: str | os.PathLike :param \*\*sort_kwargs: additional keyword arguments passed to sort_fn .. py:function:: bin_concat_unsorted(stack, file_paths, out_file) Concats Binary files in order they are passed in. :param stack: Exit Stack. :type stack: ExitStack :param file_paths: List of bin file paths. :type file_paths: List[str | os.PathLike] :param out_file: Output Concatenated Binary file. :type out_file: str | os.PathLike .. py:function:: merge_elt_data(memmaps) Merge sorted chunks using a k-way merge algorithm :param memmaps: List of temporary file memmaps :type memmaps: List[np.memmap] :Yields: *buffer (ndarray)* -- yields sorted buffer from memmaps .. py:function:: merge_plt_data(memmaps) Merge sorted chunks using a k-way merge algorithm :param memmaps: List of temporary file memmaps :type memmaps: List[np.memmap] :Yields: *buffer (ndarray)* -- yields sorted buffer from memmaps .. py:function:: bin_concat_sort_by_headers(stack, file_paths, file_type, out_type, out_file) Concats Binary files in order determined out_type and their respective merge functions :param stack: Exit Stack. :type stack: ExitStack :param file_paths: List of bin file paths. :type file_paths: List[str | os.PathLike] :param file_type: File type int matching KAT_NAMES index :type file_type: int :param out_type: Out type str between "elt" and "plt" :type out_type: str :param out_file: Output Concatenated Binary file. :type out_file: str | os.PathLike .. py:function:: parquet_concat_unsorted(file_paths, out_file) Concats Parquet files in order they are passed in. :param file_paths: List of parquet file paths. :type file_paths: List[str | os.PathLike] :param out_file: Output Concatenated Parquet file. :type out_file: str | os.PathLike .. py:function:: parquet_kway_merge(file_paths, keys, chunk_size=100000) Merge sorted chunks using a k-way merge algorithm :param file_paths: List of parquet file paths. :type file_paths: List[str | os.PathLike] :param keys: List of keys to sort by :type keys: List[str] :param chunk_size: Chunk size for reading parquet files. Defaults to 100000. :type chunk_size: int :Yields: *buffer (pa.Table)* -- yields sorted pyarrow table from input files .. py:function:: parquet_concat_sorted(file_paths, out_type, out_file, chunk_size=100000) Concats Parquet files in order determined out_type and their respective merge functions :param file_paths: List of parquet file paths. :type file_paths: List[str | os.PathLike] :param out_type: Out type str between "elt" and "plt" :type out_type: str :param out_file: Output Concatenated Parquet file. :type out_file: str | os.PathLike :param chunk_size: Chunk size for reading parquet files. Defaults to 100000. :type chunk_size: int .. py:function:: run(out_file, file_type=None, files_in=None, dir_in=None, concatenate_selt=False, concatenate_melt=False, concatenate_qelt=False, concatenate_splt=False, concatenate_mplt=False, concatenate_qplt=False, unsorted=False) Concatenate output files (optionally sorted) :param out_file: Output Concatenated file. :type out_file: str | os.PathLike :param file_type: Input file type suffix, if not discernible from input files. Defaults to None. :type file_type: str, optional :param files_in: Individual file paths to concatenate. Defaults to None. :type files_in: List[str | os.PathLike], optional :param dir_in: Path to the directory containing files for concatenation. Defaults to None. :type dir_in: str | os.PathLike, optional :param concatenate_selt: Concatenate SELT file. Defaults to False. :type concatenate_selt: bool, optional :param concatenate_melt: Concatenate MELT file. Defaults to False. :type concatenate_melt: bool, optional :param concatenate_qelt: Concatenate QELT file. Defaults to False. :type concatenate_qelt: bool, optional :param concatenate_splt: Concatenate SPLT file. Defaults to False. :type concatenate_splt: bool, optional :param concatenate_mplt: Concatenate MPLT file. Defaults to False. :type concatenate_mplt: bool, optional :param concatenate_qplt: Concatenate QPLT file. Defaults to False. :type concatenate_qplt: bool, optional :param unsorted: Do not sort by event/period ID. Defaults to False. :type unsorted: bool, optional .. py:function:: main(out=None, file_type=None, files_in=None, dir_in=None, selt=False, melt=False, qelt=False, splt=False, mplt=False, qplt=False, unsorted=False, **kwargs)