Output Schemas

Purpose

This spec defines the Parquet schemas for all output files produced by Cobre during simulation (policy evaluation) and training (policy construction). It covers the output directory layout, column-level schemas for every entity type, and the dictionary/metadata files that make outputs self-documenting.

For output infrastructure (manifests, MPI partitioning, configuration, scale reference, validation), see Output Infrastructure.

1. Directory Structure

output/
├── simulation/                              # Hive-partitioned simulation results
│   ├── _manifest.json                       # Checksums, row counts, partitions
│   ├── _SUCCESS                             # Marker on successful completion
│   ├── costs/
│   │   └── scenario_id=XXXX/data.parquet
│   ├── hydros/
│   │   └── scenario_id=XXXX/data.parquet
│   ├── thermals/
│   │   └── scenario_id=XXXX/data.parquet
│   ├── exchanges/
│   │   └── scenario_id=XXXX/data.parquet
│   ├── buses/
│   │   └── scenario_id=XXXX/data.parquet
│   ├── pumping_stations/                    # Optional: only if entity exists
│   │   └── scenario_id=XXXX/data.parquet
│   ├── contracts/                           # Optional: only if entity exists
│   │   └── scenario_id=XXXX/data.parquet
│   ├── non_controllables/                   # Optional: only if entity exists
│   │   └── scenario_id=XXXX/data.parquet
│   ├── batteries/                           # DEFERRED (forward-compatible placeholder)
│   │   └── scenario_id=XXXX/data.parquet
│   ├── inflow_lags/                         # Optional: only if AR order > 0
│   │   └── scenario_id=XXXX/data.parquet
│   └── violations/
│       └── generic/
│           └── scenario_id=XXXX/data.parquet
│
├── stochastic/                              # Stochastic model fitting artifacts
│   ├── noise_openings.parquet               # Noise term realizations per opening
│   ├── inflow_seasonal_stats.parquet        # Monthly inflow statistics
│   ├── inflow_ar_coefficients.parquet       # Autoregressive model coefficients
│   ├── correlation.json                     # Cross-correlation structure
│   ├── load_seasonal_stats.parquet          # Monthly load statistics
│   └── fitting_report.json                  # Stochastic model fitting diagnostics
│
├── hydro_models/                            # Computed FPHA hyperplanes (optional)
│   └── fpha_hyperplanes.parquet             # Written when any hydro uses computed FPHA
│
└── training/                                # Training phase outputs
    ├── _manifest.json
    ├── _SUCCESS
    ├── convergence.parquet                  # Iteration-level convergence
    ├── scaling_report.json                  # LP scaling diagnostics
    ├── model_provenance.json                # Model provenance metadata
    ├── timing/
    │   ├── iterations.parquet               # Per-iteration timing breakdown
    │   └── mpi_ranks.parquet                # Per-rank timing statistics
    ├── solver/
    │   ├── iterations.parquet               # Per-iteration solver statistics
    │   └── retry_histogram.parquet          # Per-iteration retry level histogram
    ├── cut_selection/
    │   └── iterations.parquet               # Per-iteration cut selection stats
    ├── dictionaries/
    │   ├── codes.json                       # Categorical code mappings
    │   ├── bounds.parquet                   # Entity bounds by stage/block
    │   ├── state_dictionary.json            # State space definition
    │   ├── variables.csv                    # Variable metadata
    │   └── entities.csv                     # Entity metadata
    └── metadata.json                        # Run configuration and system info

Optional files: Simulation entity directories are only written if the corresponding entities exist in the input model. The hydro_models/ directory is only written when at least one hydro uses the computed-source FPHA model.

2. Design Principles

2.1 Hive Partitioning by Scenario

Simulation outputs use Hive-style partitioning by scenario_id:

• Parallel writes: Each MPI rank writes exclusively to its assigned scenario partitions
• Partition pruning: Queries filtering by scenario read only relevant files
• Incremental updates: Individual scenarios can be recomputed without rewriting all data

Partition naming: {entity_type}/scenario_id={scenario_id:04d}/data.parquet

The scenario_id column is NOT stored in Parquet data — it is derived from the partition path.

2.2 Categorical Encoding

All categorical columns use integer codes with mappings in dictionaries/codes.json:

• Reduces storage (i8 vs variable-length strings)
• Enables efficient filtering and grouping
• Convention: categorical columns end with _code suffix (e.g., operative_state_code)

2.3 Constraint Violation Handling

Two mechanisms:

1. Slack columns in entity files: Physical bound violations (e.g., turbined_slack_m3s) appear as dedicated columns, value 0 when no violation
2. Generic violations file: User-defined generic constraint violations in violations/generic/

2.4 File Naming Conventions

3. Categorical Codes (dictionaries/codes.json)

{
  "version": "1.0",
  "generated_at": "2026-01-18T12:00:00Z",
  "operative_state": {
    "0": "deactivated",
    "1": "maintenance",
    "2": "operating",
    "3": "saturated"
  },
  "storage_binding": {
    "0": "none",
    "1": "below_minimum",
    "2": "above_maximum",
    "3": "both"
  },
  "contract_type": {
    "0": "import",
    "1": "export"
  },
  "entity_type": {
    "0": "hydro",
    "1": "thermal",
    "2": "bus",
    "3": "line",
    "4": "pumping_station",
    "5": "contract",
    "7": "non_controllable"
  },
  "bound_type": {
    "0": "storage_min",
    "1": "storage_max",
    "2": "turbined_min",
    "3": "turbined_max",
    "4": "outflow_min",
    "5": "outflow_max",
    "6": "generation_min",
    "7": "generation_max",
    "8": "flow_min",
    "9": "flow_max"
  }
}

Forward-compatible codes: entity_type code 6 (battery) is reserved for future use. See Deferred Features for implementation timeline.

Usage in analysis:

import json, polars as pl

with open("training/dictionaries/codes.json") as f:
    codes = json.load(f)

df = pl.read_parquet("simulation/hydros/")
df = df.with_columns(
    pl.col("operative_state_code")
      .map_dict({int(k): v for k, v in codes["operative_state"].items()})
      .alias("operative_state")
)

4. Dictionary Files

4.1 Bounds Dictionary (dictionaries/bounds.parquet)

Centralizes all entity bounds by stage and block, eliminating redundant bound columns from entity output files.

Bounds are stored only when they differ from default/infinite values.

4.2 State Dictionary (dictionaries/state_dictionary.json)

Documents the state space structure for the SDDP policy. See Input Constraints §4 for full state schema.

4.3 Variables Metadata (dictionaries/variables.csv)

4.4 Entities Metadata (dictionaries/entities.csv)

5. Simulation Output Schemas

5.1 Costs (simulation/costs/)

Stage and block-level cost breakdown for economic analysis. Cost columns are organized by the three penalty categories defined in Penalty System §2.

Rows per scenario: num_stages × (1 + num_blocks) (stage-level + block-level rows)

Cost relationships:

total_cost = immediate_cost + discount_factor_applied_to_future * future_cost
immediate_cost = thermal_cost + contract_cost
               + deficit_cost + excess_cost
               + storage_violation_cost + filling_target_cost + hydro_violation_cost
               + inflow_penalty_cost + generic_violation_cost
               + spillage_cost + fpha_turbined_cost + curtailment_cost
               + exchange_cost + pumping_cost

hydro_violation_cost = outflow_violation_below_cost + outflow_violation_above_cost
                     + turbined_violation_cost + generation_violation_cost
                     + evaporation_violation_cost + withdrawal_violation_cost

Note on pumping_cost: Pumping stations have no explicit cost parameter in the LP. The pumping_cost column reports the imputed cost: marginal price at the connected bus × energy consumed. This is derived from dual variables after the solve, not from a direct LP cost term.

Note on storage_violation_cost and filling_target_cost: These penalties apply to end-of-stage storage (hm³) and appear outside the ${\tau }_k$ block summation in the LP objective. They are NOT per-block costs. In the costs output, they appear only in stage-level rows (where block_id is null).

5.2 Hydros (simulation/hydros/)

Rows per scenario: num_stages × num_blocks × num_hydros

Water balance: storage_final = storage_initial + (inflow - outflow - evaporation + diverted_inflow - diverted_outflow) × duration

Slack interpretation: value > 0 means the corresponding constraint was relaxed. Storage and filling target slacks are stage-level (not per-block). See Penalty System §4 for the full violation catalogue.

5.3 Thermals (simulation/thermals/)

Rows per scenario: num_stages × num_blocks × num_thermals

GNL notes: gnl_committed_mw is capacity committed previously and available now; gnl_decision_mw is the decision made now for future availability. GNL decisions are state variables coupling stages.

5.4 Exchanges (simulation/exchanges/)

Rows per scenario: num_stages × num_blocks × num_lines

Sign convention: net_flow_mw positive = bus_from → bus_to; negative = bus_to → bus_from.

Note: The LP decision variables are direct_flow_mw ($f^{+}$) and reverse_flow_mw ($f^{-}$), both non-negative. net_flow_mw and losses are derived columns for analysis convenience. See System Elements §4 for the exchange model.

5.5 Buses (simulation/buses/)

Rows per scenario: num_stages × num_blocks × num_buses

Load balance: generation_total + imports − exports + deficit − excess = load

To compute generation by source, join with hydros, thermals, etc. using bus_id from entities.csv.

5.6 Pumping Stations (simulation/pumping_stations/) — Optional

Rows per scenario: num_stages × num_blocks × num_pumping_stations

Note on pumping_cost: Pumping stations have no explicit cost parameter in the LP objective. This column reports the imputed cost: marginal price at the connected bus (dual of load balance constraint, $/MWh) × energy consumed (MWh). It is computed after the solve from dual variables, not from a direct LP cost term. See Penalty System §8.

5.7 Contracts (simulation/contracts/) — Optional

Rows per scenario: num_stages × num_blocks × num_contracts

5.8 Non-Controllable Sources (simulation/non_controllables/) — Optional

Rows per scenario: num_stages × num_blocks × num_non_controllable_sources

Non-controllable sources are fully defined in Input System Entities §7. Curtailment cost is a Category 3 regularization penalty — see Penalty System §2.

5.9 Batteries (simulation/batteries/) — DEFERRED

Forward-compatible placeholder: The schema and entity_type code 6 are reserved. Exceptional validation rejects battery entities at input loading until the implementation is ready. See Deferred Features for implementation timeline.

5.10 Inflow Lags (simulation/inflow_lags/) — Optional

Rows per scenario: num_stages × num_hydros × max_ar_order

lag_index uses 1-based indexing. Maximum lag index equals the AR model order. These values are state variables affecting inflow sampling.

5.11 Generic Violations (simulation/violations/generic/)

Rows per scenario: num_stages × num_blocks × num_generic_constraints (only non-zero violations may be stored).

6. Training Output Schemas

6.1 Convergence Log (training/convergence.parquet)

Rows: num_iterations

Upper bound mechanisms: Two distinct mechanisms populate the upper bound columns:

1. Simulation-based (Monte Carlo): Runs the SDDP policy on sampled scenarios and averages costs. Provides upper_bound_mean and upper_bound_std. Valid only for risk-neutral problems — see Stopping Rules.
2. SIDP deterministic (inner approximation): Vertex-based upper bound via Lipschitz interpolation. Provides a deterministic upper_bound_mean (no std). Valid for both risk-neutral and risk-averse problems — see Upper Bound Evaluation.

Which mechanism is active depends on configuration. Both may run simultaneously (simulation-based for reporting, SIDP for convergence).

Gap computation: gap_percent = (upper_bound_mean − lower_bound) / |lower_bound| × 100. If UB evaluation is disabled, gap_percent is null. If only LB is available, the gap is computed from the LB change between iterations (see Stopping Rules §3).

Risk-averse interpretation: Under risk-averse (CVaR) settings, lower_bound is a convergence indicator, not a valid lower bound on the true risk-averse cost. It represents the first-stage objective value using the current cut approximation. For risk-averse convergence verification, the SIDP deterministic upper bound is required. gap_percent from the simulation-based UB is not meaningful for risk-averse problems. See Risk Measures §10.

6.2 Iteration Timing (training/timing/iterations.parquet)

Rows: num_iterations

6.3 MPI Rank Timing (training/timing/mpi_ranks.parquet)

Rows: num_iterations × num_mpi_ranks

Use idle_time_ms to identify load imbalance. Sum of scenarios_processed per iteration equals forward_passes. Communication patterns reveal MPI bottlenecks.

6.4 Solver Statistics (training/solver/iterations.parquet)

Per-iteration, per-phase, per-stage solver statistics for diagnosing LP conditioning and retry behavior. One row per (iteration, phase, stage) triple.

Rows: num_iterations × num_phases × num_stages

Phase values: The phase column identifies the SDDP algorithmic phase that produced the LP solves. "forward" and "backward" are the standard SDDP passes. "lower_bound" appears when a dedicated lower-bound evaluation solve is performed (separate from the forward pass). "simulation" appears when simulation-based upper bound evaluation is enabled (see Upper Bound Evaluation).

Retry-level histograms are stored separately in retry_histogram.parquet (SS6.6). See Solver Interface for retry strategy details.

6.5 Cut Selection Statistics (training/cut_selection/iterations.parquet)

Per-iteration, per-stage cut selection statistics. One row per (iteration, stage) pair.

Rows: num_iterations × num_stages (only iterations where cut selection runs)

6.6 Retry Histogram (training/solver/retry_histogram.parquet)

Per-iteration, per-phase, per-stage histogram of retry escalation levels. One row per (iteration, phase, stage, retry_level) tuple where the count is non-zero.

Rows: sparse — only (iteration, phase, stage, retry_level) tuples with count > 0 are stored.

The retry ladder applies increasingly aggressive solver reconfiguration at each level (e.g., scaling, presolve toggling, algorithm switching). See Solver Interface for the full retry strategy. This file complements the aggregate retry statistics in iterations.parquet (SS6.4) by providing per-level granularity.

7. Structured Output vs Parquet Schemas

The Cobre output system produces two distinct categories of schemas with different purposes and access patterns:

Key distinction: Parquet schemas define the on-disk storage format for simulation results and training diagnostics. Structured CLI output defines the presentation format for CLI responses. They serve different audiences and use different data paths.

No JSON equivalents for simulation Parquet schemas: The 11 simulation entity schemas (§5.1-5.11) and 2 of the 3 training diagnostic schemas (§6.2-6.3) remain Parquet-only. Agents access them via Arrow libraries (available in Python, R, Julia, and Rust). Converting hundreds of MB of tabular data to JSON would be impractical and unnecessary.

Convergence log has a JSON equivalent: The convergence log schema (§6.1) has a JSON equivalent via the JSON-lines streaming protocol. The progress events emitted during training (see Convergence Monitoring §4.1) contain the same fields as the convergence.parquet columns. The Parquet file is the permanent record; JSON-lines is the ephemeral real-time stream.

MCP resource serialization: When the MCP server exposes Parquet data as resources (see MCP Server), it converts Parquet columns to JSON types using the following mapping:

This conversion is performed on demand by the MCP server’s resource handlers, not stored as separate JSON files.

8. Preprocessing Output Schemas

The following files are written during the Initialization phase (before training), not during simulation or training proper. They document the result of preprocessing steps so that users can inspect and, if needed, re-use computed artifacts.

8.1 Computed FPHA Hyperplanes (output/hydro_models/fpha_hyperplanes.parquet)

Written automatically when one or more hydros use the source: "computed" FPHA fitting path. The file uses the same 11-column schema as the input file system/fpha_hyperplanes.parquet and is round-trip compatible with the parser in cobre-io.

Implementation note (v0.1.4): This file is written by prepare_hydro_models in cobre-sddp on MPI rank 0 only, immediately after the fitting pipeline completes. It is not present when all FPHA hydros use source: "precomputed" or when no hydros use FPHA.

The validity range columns (valid_v_min_hm3, valid_v_max_hm3, valid_q_max_m3s) are always null for computed planes. They are populated only when hyperplanes are provided externally via system/fpha_hyperplanes.parquet and the supplier chooses to include them. See Input Hydro Extensions §3 for the complete schema description and validation rules.

Round-trip use: The exported file can be copied to system/fpha_hyperplanes.parquet in a subsequent run and referenced with source: "precomputed" to skip re-fitting. This is useful for large systems where fitting cost is non-trivial.

Cross-References

• Output Infrastructure — Manifests, MPI partitioning, config, validation
• Input System Entities — Entity registries defining entity IDs
• Penalty System — Three-category penalty taxonomy, cost columns alignment
• Input Constraints — Generic constraints whose violations appear in output
• LP Formulation — Mathematical definitions of output variables
• System Elements — Exchange model (direct/reverse flow), NCS, Variable Units Convention
• Risk Measures — Lower bound validity under risk aversion (§10)
• Upper Bound Evaluation — SIDP deterministic upper bounds
• Stopping Rules — Simulation-based stopping and convergence criteria
• Discount Rate — Cumulative discount factor formula
• Inflow Non-Negativity — Inflow penalty in costs
• Deferred Features — Batteries (deferred)
• Structured Output — CLI response envelope and JSON-lines streaming protocol
• MCP Server — MCP resource handlers serving Parquet data as JSON
• Convergence Monitoring — JSON-lines streaming schema (§4.1) as the JSON equivalent of convergence.parquet