oasislmf.pytools.fm.compute_sparse ================================== .. py:module:: oasislmf.pytools.fm.compute_sparse .. autoapi-nested-parse:: Financial Module (FM) Sparse Computation Engine ================================================ This module implements the core loss computation for the Oasis Financial Module using sparse array storage. It processes insurance/reinsurance losses through a hierarchical node structure, applying financial terms (deductibles, limits, etc.) at each level. Architecture Overview --------------------- The FM computation uses a bottom-up traversal of a tree structure where: - Leaf nodes (items) contain ground-up losses from the GUL stream - Internal nodes aggregate losses from children and apply financial profiles - The root represents the final insured/reinsured loss output Storage Model ------------- Data is stored using a CSR (Compressed Sparse Row) inspired format: - sidx_indptr[i:i+1] gives the range of sample indices for node i - sidx_val contains the actual sample index values (sidx) - loss_indptr[i:i+1] gives the range of loss values for loss pointer i - loss_val contains the actual loss values aligned with sidx_val - extras_indptr/extras_val similarly store deductible/overlimit/underlimit values Computation Flow ---------------- For each event: 1. Read losses from input stream into sparse arrays 2. For each level (bottom to top): a. Aggregate children losses into parent nodes b. Apply financial profiles (calc rules) to compute insured loss c. Back-allocate results to base children (for allocation rules 1 & 2) d. Queue parent nodes for the next level 3. Output final losses to stream Key Concepts ------------ - profile_len: Number of profiles for a node (may differ from layer_len for step policies) - layer_len: Number of layers in output - cross_layer_profile: When True, one profile applies to merged loss across all layers - base_children: The leaf-level descendants of a node (used for back allocation) - allocation_rule: 0=no allocation, 1=proportional to input, 2=proportional to output Attributes ---------- .. autoapisummary:: oasislmf.pytools.fm.compute_sparse.logger Functions --------- .. autoapisummary:: oasislmf.pytools.fm.compute_sparse.get_base_children oasislmf.pytools.fm.compute_sparse.first_time_layer oasislmf.pytools.fm.compute_sparse.first_time_layer_extra oasislmf.pytools.fm.compute_sparse.aggregate_children_extras oasislmf.pytools.fm.compute_sparse.aggregate_children oasislmf.pytools.fm.compute_sparse.set_parent_next_compute oasislmf.pytools.fm.compute_sparse.load_net_value oasislmf.pytools.fm.compute_sparse.compute_event oasislmf.pytools.fm.compute_sparse.init_variable oasislmf.pytools.fm.compute_sparse.reset_variable Module Contents --------------- .. py:data:: logger .. py:function:: get_base_children(node, children, nodes_array, temp_children_queue) Find all leaf-level (base) descendants of a node using breadth-first traversal. Base children are the nodes at the lowest level that have no children themselves. These are needed for back allocation - when we apply financial terms at a higher level, we need to distribute the resulting losses back to the original items. Algorithm: 1. Start with the direct children of the node 2. For each child, check if it has its own children 3. If yes, add those grandchildren to the queue and continue 4. If no, this is a base child - store it at the front of the array 5. Continue until all descendants are processed The result is temp_children_queue[0:base_child_i] containing all base children. :param node: The parent node whose base children we want to find :param children: Array tracking children for each node (children[node['children']] = count, followed by child node IDs) :param nodes_array: Array of all node information :param temp_children_queue: Working array to store the traversal queue and final results :returns: Number of base children found :rtype: int .. py:function:: first_time_layer(profile_count, base_children_count, temp_children_queue, compute_idx, nodes_array, sidx_indptr, sidx_indexes, loss_indptr, loss_val) Initialize multi-layer loss storage for base children when first encountering layered computation. When a node has multiple layers but its children were computed with only one layer, we need to create separate loss arrays for each layer. This copies the layer 0 loss to layers 1..N-1 so that back allocation can work independently on each layer. This is a lazy initialization - we only create the extra layer storage when we actually need it, which saves memory for nodes that never reach multi-layer parents. :param profile_count: Number of profiles/layers to create :param base_children_count: Number of base children to process :param temp_children_queue: Array containing base children node IDs :param compute_idx: Computation state tracking pointers :param nodes_array: Array of node information :param sidx_indptr: Sample index pointers :param sidx_indexes: Node to sample index mapping :param loss_indptr: Loss value pointers :param loss_val: Loss values array .. py:function:: first_time_layer_extra(profile_count, base_children_count, temp_children_queue, compute_idx, nodes_array, sidx_indptr, sidx_indexes, loss_indptr, loss_val, extras_indptr, extras_val) Initialize multi-layer loss AND extras storage for base children. Same as first_time_layer but also handles the extras array (deductible, overlimit, underlimit). For aggregation cases (single base child), extras are copied from layer 0. For back allocation cases (multiple base children), extras are zeroed for new layers since each layer will compute its own extras through back allocation. :param profile_count: Number of profiles/layers to create :param base_children_count: Number of base children to process :param temp_children_queue: Array containing base children node IDs :param compute_idx: Computation state tracking pointers :param nodes_array: Array of node information :param sidx_indptr: Sample index pointers :param sidx_indexes: Node to sample index mapping :param loss_indptr: Loss value pointers :param loss_val: Loss values array :param extras_indptr: Extras value pointers :param extras_val: Extras values array (deductible, overlimit, underlimit) .. py:function:: aggregate_children_extras(node, children_count, nodes_array, children, temp_children_queue, compute_idx, temp_node_sidx, sidx_indexes, sidx_indptr, sidx_val, all_sidx, temp_node_loss, loss_indptr, loss_val, temp_node_extras, extras_indptr, extras_val) Aggregate losses AND extras from multiple children into a parent node. Similar to aggregate_children but also tracks the "extras" - deductible amount, overlimit, and underlimit values that are needed for back allocation when financial terms modify losses. The extras represent: - DEDUCTIBLE: Amount deducted from loss (policy deductible applied) - OVERLIMIT: Amount exceeding the policy limit - UNDERLIMIT: Remaining capacity under the limit (limit - loss) These extras must be aggregated alongside losses so that when back allocation occurs, we know how to distribute the deductible/limit effects back to individual items proportionally. :param node: Parent node being computed :param children_count: Number of direct children :param nodes_array: Array of all node information :param children: Children tracking array :param temp_children_queue: Working array for base children lookup :param compute_idx: Computation state pointers :param temp_node_sidx: Dense boolean array marking active sidx values :param sidx_indexes: Maps node_id to sidx array position :param sidx_indptr: Pointers into sidx_val :param sidx_val: Sample index values :param all_sidx: All possible sidx values :param temp_node_loss: Dense array for accumulating losses :param loss_indptr: Pointers into loss_val :param loss_val: Loss values :param temp_node_extras: Dense array [profile, sidx, 3] for extras accumulation :param extras_indptr: Pointers into extras_val :param extras_val: Extras values [deductible, overlimit, underlimit] :returns: Number of sidx values for this node :rtype: int .. py:function:: aggregate_children(node, children_count, nodes_array, children, temp_children_queue, compute_idx, temp_node_sidx, sidx_indexes, sidx_indptr, sidx_val, all_sidx, temp_node_loss, loss_indptr, loss_val) Aggregate losses from multiple children into a parent node (without extras tracking). This function sums the losses from all children for each sample index (sidx). It handles the sparse-to-dense-to-sparse conversion needed for aggregation: 1. For each child, read its sparse loss values 2. Accumulate into a dense temporary array (temp_node_loss) indexed by sidx 3. Convert back to sparse storage for the parent node Multi-layer handling: - If the parent has multiple profiles but children only have one layer, triggers first_time_layer to create layer storage for children - Each profile/layer is processed separately The function also creates the sidx array for the parent node (union of all children's sidx values) on the first profile iteration. :param node: Parent node being computed :param children_count: Number of direct children :param nodes_array: Array of all node information :param children: Children tracking array (count + child IDs per node) :param temp_children_queue: Working array for base children lookup :param compute_idx: Computation state pointers (sidx_i, sidx_ptr_i, loss_ptr_i, etc.) :param temp_node_sidx: Dense boolean array marking which sidx values have data :param sidx_indexes: Maps node_id to its sidx array position :param sidx_indptr: Pointers into sidx_val for each node :param sidx_val: Sample index values :param all_sidx: Ordered array of all possible sidx values for iteration :param temp_node_loss: Dense array [profile, sidx] for accumulating child losses :param loss_indptr: Pointers into loss_val :param loss_val: Loss values aligned with sidx_val :returns: Number of sidx values (node_val_count) for this node :rtype: int .. py:function:: set_parent_next_compute(parent_id, child_id, nodes_array, children, computes, compute_idx) Register a parent node for computation at the next level. As we process nodes at the current level, we track which parent nodes will need to be computed next. This function: 1. Adds the child to the parent's children list 2. If this is the first child seen for this parent, adds the parent to the compute queue for the next level The children array uses a count-then-values format: - children[parent['children']] = count of children - children[parent['children'] + 1..count] = child node IDs :param parent_id: Node ID of the parent to queue :param child_id: Node ID of the child being linked :param nodes_array: Array of all node information :param children: Children tracking array :param computes: Queue of nodes to compute (current level followed by next level) :param compute_idx: Contains next_compute_i pointing to end of queue .. py:function:: load_net_value(computes, compute_idx, nodes_array, sidx_indptr, sidx_indexes, loss_indptr, loss_val) Convert gross losses to net losses for output streaming. Net loss = input loss - insured loss (what remains after insurance pays) For multi-layer policies, net loss at layer i = net loss at layer i-1 minus the insured loss paid by layer i. This creates a waterfall where each layer pays from what remains after previous layers. This function iterates through the output nodes and replaces the gross loss values with net loss values in-place, then resets the compute index so the stream writer outputs from the beginning. Called when net_loss output is requested instead of or in addition to gross loss. .. py:function:: compute_event(compute_info, keep_input_loss, nodes_array, node_parents_array, node_profiles_array, len_array, max_sidx_val, sidx_indexes, sidx_indptr, sidx_val, loss_indptr, loss_val, extras_indptr, extras_val, children, computes, compute_idx, item_parent_i, fm_profile, stepped) Compute insured losses for a single event through the entire financial structure. This is the main computation function that processes one event's losses through all levels of the insurance/reinsurance hierarchy. Results are stored in-place in loss_val. Algorithm Overview ------------------ The computation proceeds bottom-up through the financial structure levels: For each level (starting from items, going up to final output): For each node to compute at this level: 1. AGGREGATE: Sum losses from children nodes - Multiple children: aggregate into temp arrays, create parent sidx - Single child: reuse child's storage (optimization) - No children (item level): use input losses directly 2. APPLY PROFILE: Apply financial terms to the aggregated loss - For each profile (may be 1 per layer or 1 cross-layer): - Run calc/calc_extra with the profile's calc rules - Handles deductibles, limits, shares, etc. 3. BACK ALLOCATE: Distribute results back to base children - Rule 0: No allocation (output at aggregate level) - Rule 1: Proportional to original input loss - Rule 2: Proportional to computed loss (pro-rata) 4. QUEUE PARENTS: Register parent nodes for next level computation Cross-Layer Profiles -------------------- When cross_layer_profile=True, losses from all layers are first merged, the profile is applied to the total, then results are back-allocated proportionally to each layer. :param compute_info: Computation metadata (levels, allocation rule, etc.) :param keep_input_loss: If True, preserve input loss for net loss calculation :param nodes_array: Static node information (agg_id, level_id, pointers) :param node_parents_array: Parent node IDs for each node :param node_profiles_array: Profile metadata (i_start, i_end into fm_profile) :param len_array: Size for dense temporary arrays :param max_sidx_val: Maximum sample index value :param sidx_indexes: Node to sidx array position mapping :param sidx_indptr: CSR-style pointers into sidx_val :param sidx_val: Sample index values :param loss_indptr: CSR-style pointers into loss_val :param loss_val: Loss values (modified in place) :param extras_indptr: CSR-style pointers into extras_val :param extras_val: Extras [deductible, overlimit, underlimit] :param children: Dynamic children tracking per node :param computes: Queue of nodes to process :param compute_idx: Computation state (current position, pointers) :param item_parent_i: Tracks which parent index for multi-parent items :param fm_profile: Array of financial profile terms :param stepped: True/None flag for stepped policies (None for JIT compatibility) .. py:function:: init_variable(compute_info, max_sidx_val, temp_dir, low_memory) Initialize all arrays needed for FM computation. Creates the sparse storage arrays for sample indices, losses, and extras. These use a CSR-like format where: - *_indptr arrays point to the start of each node's data - *_val arrays contain the actual values The loss and extras arrays share the same indexing as sidx - each node's loss[i] corresponds to sidx[i]. This allows using sidx_indexes to track the length of values for any node. :param compute_info: Metadata with array size requirements :param max_sidx_val: Maximum sample index (determines array sizing) :param temp_dir: Directory for memory-mapped files (low_memory mode) :param low_memory: If True, use memory-mapped files instead of RAM :returns: Tuple of all initialized arrays needed by compute_event .. py:function:: reset_variable(children, compute_idx, computes) Reset the per event array :param children: array of all the children with loss value for each node :param compute_idx: single element named array containing all the pointer needed to tract the computation (compute_idx_dtype) :param computes: array of node to compute