converter.runner.base

Module Contents

Classes

Converters	dict() -> new empty dictionary
BaseRunner	Runs the transformations on the extracted data and writes
BaseAsyncRunner	Runs the transformations on the extracted data and writes it to the data

Functions

build_converter(t) → Callable[[Any, bool, List], Any]

Attributes

RowType

converter.runner.base.RowType

class converter.runner.base.Converters

Bases: TypedDict

dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object’s

(key, value) pairs

dict(iterable) -> new dictionary initialized as if via:

d = {} for k, v in iterable:

d[k] = v

dict(**kwargs) -> new dictionary initialized with the name=value pairs

in the keyword argument list. For example: dict(one=1, two=2)

int :Callable[[Any, bool, List], Union[int, None]]

float :Callable[[Any, bool, List], Union[float, None]]

string :Callable[[Any, bool, List], Union[str, None]]

converter.runner.base.build_converter(t) → Callable[[Any, bool, List], Any]

class converter.runner.base.BaseRunner(config: converter.config.Config, **options)

Bases: _BaseRunner

Runs the transformations on the extracted data and writes it to the data loader

Parameters

config – The global config for the system

run(self, extractor: converter.connector.base.BaseConnector, mapping: converter.mapping.base.BaseMapping, loader: converter.connector.base.BaseConnector)

Runs the transformation process and swnds the data to the data loader

Parameters

• extractor – The data connection to extract data from

• mapping – Mapping object describing the transformations to apply

• loader – The data connection to load data to

abstract transform(self, extractor: converter.connector.base.BaseConnector, mapping: converter.mapping.base.BaseMapping) → Iterable[Dict[str, Any]]

Performs the transformation

Parameters

• extractor – The data connection to extract data from

• mapping – Mapping object describing the transformations to apply

Returns

An iterable containing the transformed data

class converter.runner.base.BaseAsyncRunner(config: converter.config.Config, **options)

Bases: _BaseRunner

Runs the transformations on the extracted data and writes it to the data loader. The connectors and transformation are all async objects allowing for async data providers such as websockets or polled apis to be used as a data connection.

The connectors and transformations should be done in an eager way so that each row is processed and passed to the loader as it’s received or cached for processing later. After each row if processed the next should be awaited so that new data can be extracted.

Parameters

config – The global config for the system

run(self, extractor: converter.connector.base.BaseConnector, mapping: converter.mapping.base.BaseMapping, loader: converter.connector.base.BaseConnector)

abstract async transform(self, extractor: converter.connector.base.BaseConnector, mapping: converter.mapping.base.BaseMapping) → AsyncIterable[Dict[str, Any]]

Performs the transformation

Parameters

• extractor – The data connection to extract data from

• mapping – Mapping object describing the transformations to apply

Returns

An iterable containing the transformed data