Building and Running Models =========================== The repository provides a Python toolkit for building, running and testing Oasis models end-to-end, including performing individual steps in this process. It includes: - a Python class framework for working with Oasis models and model resources as Python objects (the ``models`` subpackage) - a Python class framework for managing model exposures and resources, and also for generating Oasis files from these (the ``exposures`` subpackage) - a Python factory class for instantiating keys lookup services for models, and generating and saving keys outputs from these lookup services (the ``keys`` subpackage) - executable scripts, based on these class frameworks, for writing keys outputs from model lookup services (``generate_keys.py``), generating Oasis files from model source exposures and other resources (``generate_oasis_files.py``), and generating losses for models (``generate_losses.py``). This includes a "master" script that can perform all these steps to run the model end-to-end (``run_model.py``). Generating keys --------------- ``generate_keys.py`` is an executable script which can generate and write Oasis keys (area peril ID, vulnerability ID) for a model, given the following arguments (in no particular order) :: ./generate_keys.py -k /path/to/keys/data -v /path/to/model/version/csv/file -l /path/to/lookup/service/package -e /path/to/model/exposures/csv/file -o /path/to/output/file [-f ] When calling the script this way paths can be given relative to the script, in particular, file paths should include the filename and extension. The paths to the keys data, lookup service package (Python package containing the lookup source code), and model version file will usually be located in the model keys server (Git) repository. If the repository was created by or is managed by Oasis LMF then the lookup service package will usually be contained in the ``src/keys_server`` Python subpackage and can be given as the path to that subpackage (see the OasisPiWind repository as a reference for how to structure an Oasis keys server repository) It is also possible to run the script by defining these arguments in a JSON configuration file and calling the script with option ``-f`` and the (relative or absolute) path to the file. :: ./generate_keys.py -f /path/to/script/config/json/file The JSON file should contain the following keys (in no particular order) :: "keys_data_path" "model_version_file_path" "lookup_package_path" "model_exposures_file_path" "output_file_path" "output_format" and the values of the path-related keys should be string paths, given relative to the location of JSON file. The JSON file is usually placed in the model keys server repository. The ``"output_format"`` key is optional - by default the script will generate an Oasis keys file. Keys records returned by an Oasis keys lookup service (see the PiWind lookup service for reference) will be Python dicts with the following structure :: { "id": , "peril_id": , "coverage": , "area_peril_id": , "vulnerability_id": , "message": , "status": } The ``generate_keys.py`` script can generate keys records in this format, and write them to file. For model loss calculations however ktools requires a keys CSV file with the following format :: LocID,PerilID,CoverageID,AreaPerilID,VulnerabilityID .. .. where the headers correspond to the relevant Oasis keys record fields. The ``generate_keys.py`` script can also generate and write Oasis keys files. Generating Oasis files ---------------------- ``generate_oasis_files.py`` is an executable script which can generate Oasis files (items, coverages, GUL summary) for a model, given the following arguments (in no particular order) :: ./generate_oasis_files.py -k /path/to/keys/data -v /path/to/model/version/csv/file -l /path/to/lookup/service/package -p /path/to/canonical/exposures/profile/JSON/file -e /path/to/source/exposures/file -a /path/to/source/exposures/validation/file -b /path/to/source/to/canonical/exposures/transformation/file -c /path/to/canonical/exposures/validation/file -d /path/to/canonical/to/model/exposures/transformation/file -x /path/to/xtrans/executable [-o /path/to/oasis/files/directory] When calling the script this way paths can be given relative to the script, in particular, file paths should include the filename and extension. The paths to the keys data, lookup service package, model version file, canonical exposures profile JSON, source exposures file, transformation and validation files, will usually be located in the model keys server repository. The path to the Oasis files directory is optional - by default the script will create a timestamped folder in ``omdk/runs`` with the prefix ``OasisFiles``. It is also possible to run the script by defining these arguments in a JSON configuration file and calling the script using the path to this file using the option ``-f`` and the (relative or absolute) path to the file. :: ./generate_oasis_files.py -f /path/to/script/config/json/file The JSON file contain the following keys (in no particular order) :: "keys_data_path" "model_version_file_path" "lookup_package_path" "canonical_exposures_profile_json_path" "source_exposures_file_path" "source_exposures_validation_file_path" "source_to_canonical_exposures_transformation_file_path" "canonical_exposures_validation_file_path" "canonical_to_model_exposures_transformation_file_path" "xtrans_path" "oasis_files_path" and the values of these keys should be string paths, given relative to the location of the JSON file. The JSON file is usually placed in the model keys server repository. The ``"oasis_files_path"`` key is optional - by default the script will create a timestamped folder in ``omdk/runs`` with the prefix ``OasisFiles``. Generating losses ----------------- ``generate_losses.py`` is an executable script which, given Oasis files, model analysis settings JSON file, model data, and some other parameters, can generate losses using the installed ktools framework. The script can be called directly from the command line given the following arguments (in no particular order) :: ./generate_losses.py -o /path/to/oasis/files -j /path/to/analysis/settings/json/file -m /path/to/model/data [-r /path/to/model/run/directory] [-s ] [-n ] [--execute | --no-execute] When calling the script this way paths can be given relative to the script, in particular, file paths should include the filename and extension. The path to the model run directory is optional - by default the script will create a timestamped folder in ``omdk/runs`` with the prefix ``ProgOasis``. The ktools script name and number of calculation processes are optional - by default the script will create a ktools script named ``run_tools.sh`` and set the number of calculation processes to 2. By default executing ``generate_losses.py`` will automatically execute the ktools losses script it generates. If you don't want this provide the (optional) ``--no-execute`` argument. The default here is automatic execution. The script copies the analysis settings JSON file to the model run directory and sets up the following folder structure inside :: ├── analysis_settings.json ├── fifo/ ├── input/ ├── output/ ├── static/ └── work/ Depending on the OS type the model data is symlinked (Linux, Darwin) or copied (Cygwin, Windows) into the ``static`` subfolder. The input files are kept in the ``input`` subfolder and the losses are generated as CSV files in the ``output`` subfolder. It is also possible to run the script by defining these arguments in a JSON configuration file and calling the script using the path to this file using the option ``-f`` and the (relative or absolute) path to the file. :: ./generate_losses.py -f /path/to/script/config/json/file' The JSON file should contain the following keys (in no particular order) :: "oasis_files_path" "analysis_settings_json_file_path" "model_data_path" "model_run_dir_path" "ktools_script_name" "ktools_num_processes" "execute" and the values of the path-related keys should be string paths, given relative to the location of the JSON file. The JSON file is usually placed in the model keys server repository. The ``"model_run_dir_path"`` key is optional - by default the script will create a timestamped folder in ``omdk/runs`` with the prefix ``ProgOasis``. The ``"ktools_script_name"`` and ``"ktools_num_processes"`` keys are optional - by default the script will create a ktools script named ``run_tools.sh`` and set the number of calculation processes to 2. The ``"execute"`` key is optional - if present it should be either ``true`` or ``false`` depending on whether you want the generated ktools losses scripts to be automatically executed or not. The default here is automatic execution. Running a model end-to-end -------------------------- ``run_model.py`` is an executable "master" script that can run models end-to-end, i.e. generate losses given model resources, including keys data, canonical exposure profiles, exposure transformation and validation files, model data, analysis settings etc., given the following arguments (in no particular order) :: ./run_model.py -k /path/to/keys/data/folder -v /path/to/model/version/file -l /path/to/lookup/service/package -p /path/to/canonical/exposures/profile/JSON/file -e /path/to/source/exposures/file -a /path/to/source/exposures/validation/file -b /path/to/source/to/canonical/exposures/transformation/file -c /path/to/canonical/exposures/validation/file -d /path/to/canonical/to/model/exposures/transformation/file -x /path/to/xtrans/executable -j /path/to/analysis/settings/json/file -m /path/to/model/data [-r /path/to/model/run/directory] [-s ] [-n ] When calling the script this way paths can be given relative to the script, in particular, file paths should include the filename and extension. The paths to the keys data, lookup service package, model version file, canonical exposures profile JSON, source exposures file, transformation and validation files, and analysis settings JSON file, will usually be located in the model keys server repository. The path to the model run directory is optional - by default the script will create a timestamped folder in ``omdk/runs`` with the prefix ``ProgOasis``. The ktools script name and number of calculation processes are also optional - by default the script will create a ktools script named ``run_tools.sh`` and set the number of calculation processes to 2. It is also possible to run the script by defining these arguments in a JSON configuration file and calling the script using the path to this file using the option ``-f`` and the (relative or absolute) path to the file. :: ./run_model.py -f /path/to/script/config/json/file' The JSON file should contain the following keys (in no particular order) :: "keys_data_path" "model_version_file_path" "lookup_package_path" "canonical_exposures_profile_json_path" "source_exposures_file_path" "source_exposures_validation_file_path" "source_to_canonical_exposures_transformation_file_path" "canonical_exposures_validation_file_path" "canonical_to_model_exposures_transformation_file_path" "xtrans_path" "analysis_settings_json_file_path" "model_data_path" "model_run_dir_path" "ktools_script_name" "ktools_num_processes" and the values of the path-related keys should be string paths, given relative to the location of the JSON file. The JSON file is usually placed in the model keys server repository. The ``"model_run_dir_path"`` key is optional - by default the script will create a timestamped folder in ``omdk/runs`` with the prefix ``ProgOasis``. The ``"ktools_script_name"`` and ``"ktools_num_processes"`` keys are optional - by default the script will create a ktools script named ``run_tools.sh`` and set the number of calculation processes to 2. You can define a separate JSON configuration file for each model, provided you have the model keys server repository and other required model resources available locally. **NOTE**: For a given model the JSON script configuration files for ``generate_oasis_files.py``, ``generate_losses.py`` and ``run_model.py`` should complement each other, except for ``generate_losses.py`` which requires the path to Oasis files, not required by ``run_model.py``. You can run any of these scripts against a single master script configuration file, provided that the path to an actual set of Oasis files is added in order to run ``generate_losses.py``. Running PiWind -------------- PiWind is a reference windstorm model developed by Oasis. The lookup source code, keys server, keys data and model data and all other model resources are in the model GitHub repository which is https://github.com/OasisLMF/OasisPiWind The repository also contains a `JSON configuration file `_ for the model which can be used to run it end-to-end with the MDK master script. :: { "keys_data_path": "keys_data/PiWind", "model_version_file_path": "keys_data/PiWind/ModelVersion.csv", "lookup_package_path": "src/keys_server", "canonical_exposures_profile_json_path": "oasislmf-piwind-canonical-profile.json", "source_exposures_file_path": "tests/data/SourceLocPiWind.csv", "source_exposures_validation_file_path": "flamingo/PiWind/Files/ValidationFiles/Generic_Windstorm_SourceLoc.xsd", "source_to_canonical_exposures_transformation_file_path": "flamingo/PiWind/Files/TransformationFiles/MappingMapToGeneric_Windstorm_CanLoc_A.xslt", "canonical_exposures_validation_file_path": "flamingo/PiWind/Files/ValidationFiles/Generic_Windstorm_CanLoc_B.xsd", "canonical_to_model_exposures_transformation_file_path": "flamingo/PiWind/Files/TransformationFiles/MappingMapTopiwind_modelloc.xslt", "xtrans_path": "../omdk/xtrans/xtrans.exe", "analysis_settings_json_file_path": "analysis_settings.json", "model_data_path": "model_data/PiWind" } **NOTE**: All the paths, except for the ``xtrans.exe`` executable, are given relative to the location of the PiWind repository. The ``xtrans.exe`` executable is not part of the MDK repository, and you need to build it for your platform by running the ``make-trans`` executable shell script - this will built it in the ``omdk/xtrans`` folder. For the ``xtrans.exe`` path to be found you should locate the MDK repository adjacent to the PiWind repository, i.e. :: ... |- OasisPiWind/ |- omdk/ ...