Inside a run: the kernel pipeline step by step

Run this tutorial yourself

This page is a Jupyter notebook, executed when the docs are built. Download pipeline-step-by-step.ipynb

Set up an environment and open it in Jupyter:

python -m venv venv && source venv/bin/activate
pip install oasislmf jupyterlab matplotlib
jupyter lab pipeline-step-by-step.ipynb

The example data ships in the OasisModels repository (under docs/source/tutorials/); tutorials that run a model need that model’s data and the loss engine — follow the prerequisites described on this page.

The high-level walkthrough runs a whole analysis with one command. This companion opens the hood: after preparing the inputs, oasislmf model run generates a kernel script (run_kernel.sh) that streams data through the pytools tools. Here we walk that pipeline one stage at a time and inspect the intermediary data.

The generated pipeline

The core of run_kernel.sh is, per partition, a single streamed chain:

evepy 1 8 | gulmc --random-generator=2 --vuln-cache-size 200 -S10 -L0 -a0 \
          | tee fifo/gul_P1 \
          | fmpy -a2 > fifo/il_P1
# then, off the tee'd streams:
summarypy -t gul -1 fifo/gul_S1_summary_P1 < fifo/gul_P1
summarypy -t il  -1 fifo/il_S1_summary_P1  < fifo/il_P1
eltpy -E bin -s work/kat/gul_S1_elt_sample_P1 < fifo/gul_S1_selt_ord_P1

The real script runs this across 8 partitions in parallel, connected by named pipes (fifo/...), with modelpy serving model data and kat concatenating the partitions at the end. Below we run the logical single-stream version to files so we can look at what flows between the tools.

Note

Runnable cells below load committed samples produced by running each pytools tool once (a single event); the engine is not run at docs-build time. The bash blocks show the actual commands. To reproduce, run them yourself in a run directory.

from pathlib import Path
import pandas as pd

_c = [Path("data/pipeline"), Path("tutorials/data/pipeline"),
      Path("docs/source/tutorials/data/pipeline")]
DATA = next((c for c in _c if c.exists()), None)
assert DATA is not None, "pipeline sample data not found"

Stage 1 — events (evepy)

evepy emits a partition of event ids to process (evepy <p> <N> = partition p of N). It’s the entry point of the stream.

evepy 1 1 -o events.bin        # all events, single partition

pd.read_csv(DATA / "events.csv").head()

	event_id
0	1

Stage 2 — ground-up loss (gulmc)

gulmc (ground-up Monte-Carlo) reads the model data (footprint, vulnerability, …) from the run directory and, for each item and event, samples S ground-up losses.

gulmc --run-dir . -S10 -a0 -i events.bin -o gul.bin

The GUL stream is item-level, keyed by event_id, item_id, sidx, loss. Negative sidx values are special statistics, positive ones are the actual loss samples (1..S); loss-free samples are dropped (-L0 threshold):

sidx	meaning
-1	numerical mean
-2	standard deviation
-3	impacted exposure
-4	chance of loss
-5	max loss
≥ 1	sample number

gul = pd.read_csv(DATA / "gul_stream_sample.csv")   # one item's rows
gul

	event_id	item_id	sidx	loss
0	1	30288	-5	200000.00
1	1	30288	-4	0.36
2	1	30288	-3	400000.00
3	1	30288	-2	37784.74
4	1	30288	-1	20560.00
5	1	30288	2	16650.79
6	1	30288	7	100639.45
7	1	30288	8	46570.86

Stage 3 — insured loss (fmpy)

fmpy (the Financial Module) applies the policy terms — the financial structure built into the run’s input/ — to the ground-up stream, producing insured losses.

fmpy -a2 -i gul.bin -o il.bin        # back-allocation rule 2

The stream keeps the same shape but is now keyed by output_id, and the losses are reduced by deductibles/limits. Compare the mean (sidx = -1) with the ground-up value above:

il = pd.read_csv(DATA / "il_stream_sample.csv")
il

	event_id	output_id	sidx	loss
0	1	60575	-5	334.44
1	1	60575	-3	332.20
2	1	60575	-1	312.39
3	1	60575	2	250.31
4	1	60575	7	1462.30
5	1	60575	8	713.89

Stage 4 — summary & ORD outputs (summarypy → eltpy / pltpy / lecpy / aalpy)

The loss streams are aggregated to the reporting summary level by summarypy, then turned into ORD result tables by the output tools:

summarypy -t gul -1 gul_summary.bin < gul.bin      # aggregate to summary level
eltpy  -E bin -s gul_S1_elt_sample  < gul_S1_selt_ord   # event loss table
# pltpy / lecpy / aalpy produce PLT / EPT / ALT similarly

The resulting SELT / EPT / ALT tables are exactly the outputs analysed in the high-level walkthrough.

Inspecting the streams (bintocsv)

The binary streams above were turned into the CSVs shown here with the bintocsv converter (one sub-command per stream type):

bintocsv eve -i events.bin -o events.csv
bintocsv gul -i gul.bin    -o gul.csv
bintocsv fm  -i il.bin     -o il.csv

Where next

• The kernel component and stream-format reference in the OasisLMF docs (reference/kernel — CoreComponents, Specification) documents each tool and the binary stream layouts in full.

• The high-level walkthrough shows the ORD outputs this pipeline produces.