Executing a data transformation

A data transformation can be executed via the User Interface or the Command Line. Either method will allow to use flat file input data, or a database connection.

Either method will allow transformation of flat file data, or via SQL database connection.

In addition to this written user guide, a video tutorial on how to use the ODTF is also available.

Running ODTF via User Interface

The user interface is accessed by running a Windows-compatible executable file (.exe) which should be downloaded from the project GitHub pages.

  1. Download the file, unzip it, and run the file from its unzipped location. A terminal window will open while the UI window opens. The terminal windows remains open while the UI is running, and will display processing messages, including any errors.

  2. Load a configuration .yaml file, which will populate the fields in the UI. Alternatively, you can configure the transformation entirely in the UI. These settings can be saved to a text config file for reference.

To do this, enter on the Location, Account, and Reinsurance tabs:

  • Mapping: the source and destination mapping

  • Extractor: source data path or database connection

  • Loader: destination data path or database connection

  • Runner: pandas is used as the default

For further details on these inputs, please see section on config file inputs:

  1. Enter portfolio metadata on the Metadata tab. This information is optional, but will help to record the transformations being performed. It will be particularly useful in IDF/CRC testing procedures, to track what portfolio types and levels of complexity have been tested and how the framework performs for them.

  2. Execute the transformation on the Run tab. In the run tab you will see the log messages being written, with the duration of transformation given, when it is completed. This screen will also show the data values written according to the fields and groupings given in the validation input file.

The ODTF user interface

Running ODTF via Command Line

Running via the command line interface requires installation of the converter package from GitHub:

Pre-requisites:

To use the ODTF you will require:

  • Python version 3.8 or higher. To install Python, follow instructions at the Python For Beginners page.

  • To install the package via command line, you will require: the package pip.

It is recommended to install the package in a clean virtual environment.

Download and install:

The tool operates on a local installation only.

To install the package via command line (Mac OS, Linux, Windows):

  1. Download the Python package from the project GitHub pages at github.com/OasisLMF/OpenDataTransform.

  2. Install the Python package using pip install git+https://github.com/OasisLMF/OpenDataTransform.git

The most basic command to run a transformation is converter -c <path to config file> run.

Command line usage:

Usage: converter [OPTIONS] COMMAND [ARGS]...

  Initialises the cli grouping with default options.

Options:
  -o, --option TEXT...  Sets a configuration option, a path and value are
                        required eg -o extractor.options.foo.bar bash

  -c, --config TEXT     Path to the configuration file.
  -v, --verbose         Specifies the verbosity level, if used multiple times
                        the verbosity is increased further

  --no-color            Disables colorised output.
  --help                Show this message and exit.

Commands:
  run          Runs the data conversion
  show-config  Prints the resolved config to the console

Further information on configuration and command line use: