Penalty System

Purpose

This spec defines the unified penalty system that ensures LP feasibility across all scenarios while correctly pricing operational costs and constraint violations. The penalty system uses a three-tier cascade resolution: global defaults → entity overrides → stage overrides.

This is a key area expected to evolve. See LP Formulation for how penalties enter the objective function.

1. Design Rationale

The LP must always be feasible. Several physical and operational constraints may be impossible to satisfy in extreme scenarios (droughts, equipment failures, etc.). The penalty system provides slack variables with graduated costs to maintain feasibility while signaling the severity of violations.

Override Resolution Behavior

The penalty system supports three levels of specificity. The effective penalty for a given entity, stage, and penalty type is determined by the most specific value available:

1. Stage-level override — If a value is defined for this specific (entity, stage, penalty_type) tuple, it takes precedence.
2. Entity-level override — If no stage override exists, a per-entity default (defined in the entity registry file) is used.
3. Global default — If neither stage nor entity overrides exist, the global default from penalties.json applies.

Implementation note: The resolution behavior above describes the semantics of the cascade, not the algorithm. In practice, the implementation should pre-resolve penalties during input loading (e.g., start from global defaults, apply entity overrides while parsing registries, then batch-apply stage overrides) rather than querying files at runtime. The exact resolution strategy is an implementation concern.

Format Rationale

2. Penalty Categories

Penalties serve three distinct purposes in the LP formulation. Understanding these categories is important for setting appropriate cost magnitudes and interpreting results.

Category 1: Recourse Slacks (LP Feasibility)

These penalties ensure that the SDDP algorithm has relatively complete recourse — every subproblem must be feasible regardless of the scenario realization. Without these slacks, the LP would be infeasible when generation cannot meet demand or when excess uncontrollable generation cannot be absorbed.

Deficit and excess are conceptually slack variables on the load balance constraint, but they have special names because of their importance in the hydrothermal dispatch application. Deficit represents the value of lost load; excess is a regularization-level cost to eliminate spurious slack generation.

Note on excess_cost range: The wide typical range (0.001–100 $/MWh) reflects two distinct use cases. At the low end (0.001–0.1), the cost acts as a pure regularization term that breaks solver degeneracy without distorting the policy. At the high end (1–100), the cost serves a policy-shaping role, actively discouraging excess generation to keep dispatch solutions physically realistic. The global default example ("excess_cost": 100.0) uses the upper end of this range.

Category 2: Constraint Violation Penalties (Policy Shaping)

These penalties provide slack for physical or operational constraints that may be impossible to satisfy under extreme conditions (e.g., drought, environmental directives from the system operator). Their cost must be high enough to affect the value function in earlier stages, signaling that the system should avoid states that lead to these violations.

These penalties create an artificial cost in the objective function that propagates backward through the value function, telling earlier stages to store more water (or dispatch differently) to avoid reaching states where violations are necessary.

Category 3: Regularization Costs (Solution Guidance)

These are small costs inserted into the objective function to guide the solver toward physically preferred solutions when the LP would otherwise be indifferent. They do not represent real costs and should be orders of magnitude smaller than any economic cost to avoid distorting the optimal policy.

Penalty Priority Ordering

When setting penalty magnitudes, the following ordering must be maintained:

$$\text{Filling target}>\text{Storage violation}>\text{Deficit}>\text{Constraint violations}>\text{Resource costs}>\text{Regularization}$$

The filling target violation cost must be the highest penalty in the system — filling the dead volume is prioritized above all other objectives. The storage violation below cost must exceed deficit cost — keeping reservoirs above dead volume is more critical than serving load (operating below dead volume risks dam safety and equipment damage).

FPHA validation rule: For each hydro using the fpha production model, fpha_turbined_cost > spillage_cost must hold. The concave FPHA geometry allows interior LP solutions where generation is less than the physical production function would yield — the turbined flow penalty prevents the solver from “spilling through the turbines.” See Internal Structures §3 for the full explanation.

Penalty Ordering Validation

The qualitative ordering above is enforced by five adjacent-pair validation checks. Each check compares a higher-priority penalty against the next lower-priority penalty in the hierarchy. All five checks produce warnings, not errors — violating the ordering degrades policy quality but does not break algorithmic correctness. The FPHA validation rule (fpha_turbined_cost > spillage_cost) is a separate error because it affects LP solution correctness (interior FPHA solutions), not merely policy quality.

Validation runs on post-resolution penalty values — after the three-tier cascade (global defaults, entity overrides, stage overrides) has been applied. Each (entity, stage) pair is checked independently, so a stage override that inverts the ordering for a specific entity will trigger a warning for that entity at that stage.

Constraint violation costs (used in checks 3 and 4): the set {turbined_violation_below_cost, outflow_violation_below_cost, outflow_violation_above_cost, generation_violation_below_cost, evaporation_violation_cost, water_withdrawal_violation_cost} for the hydro being checked.

Resource costs (used in checks 4 and 5): thermal generation costs (cost_per_mwh) for thermals connected to the bus being checked.

Regularization costs (used in check 5): the set {spillage_cost, fpha_turbined_cost, diversion_cost, curtailment_cost, exchange_cost} across all entities in the system.

Warning aggregation: To avoid excessive output when many (entity, stage) pairs violate the same check, warnings are aggregated: one warning per violated check across all entities and stages. The warning message reports the total violation count and the most extreme example (the pair with the largest inversion magnitude).

3. Global Penalty Defaults (penalties.json)

This file defines default penalty values for all entities. It is required and must be present in the case directory root.

{
  "$schema": "https://cobre.dev/schemas/v2/penalties.schema.json",
  "version": "1.0",
  "bus": {
    "deficit_segments": [
      { "depth_mw": 500, "cost": 1000.0 },
      { "depth_mw": 1000, "cost": 3000.0 },
      { "depth_mw": null, "cost": 5000.0 }
    ],
    "excess_cost": 100.0
  },
  "line": {
    "exchange_cost": 2.0
  },
  "hydro": {
    "spillage_cost": 0.01,
    "fpha_turbined_cost": 0.05,
    "diversion_cost": 0.1,
    "storage_violation_below_cost": 10000.0,
    "filling_target_violation_cost": 50000.0,
    "turbined_violation_below_cost": 500.0,
    "outflow_violation_below_cost": 500.0,
    "outflow_violation_above_cost": 500.0,
    "generation_violation_below_cost": 1000.0,
    "evaporation_violation_cost": 5000.0,
    "water_withdrawal_violation_cost": 1000.0,
    "water_withdrawal_violation_pos_cost": null,
    "water_withdrawal_violation_neg_cost": null,
    "evaporation_violation_pos_cost": null,
    "evaporation_violation_neg_cost": null,
    "inflow_nonnegativity_cost": null
  },
  "non_controllable_source": {
    "curtailment_cost": 0.005
  }
}

Directional Penalty Overrides

Five additional hydro penalty fields support directional and inflow-specific penalties. All are optional and default to the symmetric value when null:

When null, the directional withdrawal and evaporation costs fall back to the symmetric water_withdrawal_violation_cost and evaporation_violation_cost respectively. The inflow_nonnegativity_cost defaults to 1000.0 when absent. See Inflow Non-Negativity for details on the penalty method.

Piecewise Deficit

Deficit is modeled as piecewise linear segments. Each segment specifies a depth (MW of unmet demand) and cost. Segments are cumulative: first depth_mw MW at first cost, next at second cost, etc. The last segment MUST have depth_mw: null to ensure LP feasibility (unbounded extension).

Deficit segments can be overridden per bus in buses.json (see Input System Entities §1), but cannot be stage-varying (piecewise structure is too complex for per-stage override).

4. Constraint Violation Categories

This section enumerates all constraints in the LP that use slack variables, organized by the system element they belong to. For the full LP constraint formulations, see LP Formulation.

System-Level (Bus)

Hydro — Flow and Generation Constraints

Hydro — Storage Bounds

Minimum storage (min_storage_hm3) uses a slack variable (storage_violation_below) with a very high penalty cost — above deficit cost in the penalty hierarchy. Operating below dead volume risks dam safety and equipment damage, so this is treated as a near-physical violation. The slack ensures LP feasibility in extreme scenarios (severe drought, or the transition from filling to operating when the reservoir did not fully reach min_storage_hm3).

Maximum storage (max_storage_hm3) is a hard physical limit (reservoir capacity). If storage would exceed the maximum, emergency spillage handles the excess. No slack variable is used.

Filling target constraint: At the last filling stage (entry_stage_id - 1), a terminal constraint enforces v_h >= min_storage_hm3. This uses the filling_target_violation slack with the highest penalty in the system (filling_target_violation_cost), ensuring the solver prioritizes filling above all other objectives including load serving. The slack only activates when there is physically insufficient water — see Input System Entities §3 for the filling model description and Input Constraints §2 for the filling inflow sufficiency validation.

Lines

Exchange bounds are hard variable bounds on the direct and reverse flow variables. No slack variables. The exchange_cost is a regularization term on the flow variables themselves, not a violation penalty.

Thermals

Thermal bounds (min_generation, max_generation) are hard constraints. No slack variables. Thermal dispatch is directly controllable (unlike hydro, which depends on exogenous inflows), so if bounds cannot be met, this indicates a data error.

Generic Constraints

User-defined generic constraints (see Input Constraints) can optionally have slack variables with user-specified penalty costs. These are typically used for physical or operational directives from the system operator, and fall into the constraint violation category.

Non-Controllable Sources

Non-controllable generation bounds: Generation is bounded by [0, available_generation] where available_generation comes from the scenario pipeline. Both bounds are hard constraints — no slack variables. The curtailment_cost is a regularization term (Category 3) applied to the difference between available and dispatched generation, penalizing curtailment to prioritize using “free” non-controllable energy. Analogous to hydro spillage_cost — curtailment discards available energy. See Input System Entities §7.

5. Stage-Varying Penalty Overrides

Stage-varying overrides allow penalty values to change at specific stages for specific entities. Only entries that differ from the entity or global defaults need to be specified (sparse storage). Each entity type has its own Parquet file under constraints/.

Bus Penalty Overrides (constraints/penalty_overrides_bus.parquet) — Optional

Line Penalty Overrides (constraints/penalty_overrides_line.parquet) — Optional

Hydro Penalty Overrides (constraints/penalty_overrides_hydro.parquet) — Optional

Non-Controllable Source Penalty Overrides (constraints/penalty_overrides_ncs.parquet) — Optional

Sparse storage: Only include entries where values differ from defaults to minimize file size and I/O.

6. Negative Evaporation (Condensation) Handling

While evaporation is typically positive (water loss), the evaporation coefficient can be negative:

The evaporation constraint uses bidirectional slack variables:

Q_evaporated - evap_slack_positive + evap_slack_negative = EvapCoef x Area(V_avg)

where:
  evap_slack_positive >= 0  (actual evap > computed evap)
  evap_slack_negative >= 0  (actual evap < computed evap, including negative target)

Each slack variable receives its own penalty: evaporation_violation_pos_cost for over-evaporation and evaporation_violation_neg_cost for under-evaporation. When the directional costs are not specified in penalties.json (i.e., null), both default to the symmetric evaporation_violation_cost value.

7. Hydro Variables and Bounds Summary

Generation bounds: The user explicitly sets min_generation_mw and max_generation_mw. These are not derived from turbined flow bounds, because the production function is not always constant productivity. When a complete hydro model is available, the installed capacity provides a natural hard upper bound. The lower bound always requires a slack to maintain feasibility.

Relationship: outflow = turbined_flow + spillage, generation = f(turbined_flow, storage) (depends on production model)

Dead-Volume Filling Specifics

During the filling period ([start_stage_id, entry_stage_id)), the hydro is in commissioning state. The reservoir is being filled to min_storage_hm3 (dead volume). This model is based on CEPEL’s dead-volume filling approach (enchimento de volume morto).

Operational constraints during filling:

• No generation: turbined_flow = 0, generation = 0 (hard constraint — turbines not installed/operational)
• Outflow via spillage only: During filling, turbines are not operational. Outflow is limited to spillage once water reaches the spillway crest. Bottom discharge outlets are a simulation-only feature (see Input System Entities §3).
• Min outflow requirement: Environmental flow must be met via spillage. If spillage is not physically possible (reservoir level below spillway crest), the outflow_violation_below slack absorbs the shortfall.
• Filling retention: filling_inflow_m3s (from Input Constraints §2) is removed from available inflow before the water balance — it goes directly to storage.
• Storage bounds relaxed: min_storage_hm3 does not apply during filling — storage can be anywhere in [0, max_storage_hm3].

Terminal filling constraint: At the last filling stage (entry_stage_id - 1), a constraint enforces:

v_h >= min_storage_hm3 - filling_target_violation

where filling_target_violation >= 0 has filling_target_violation_cost — the highest penalty in the system. This ensures the solver prioritizes filling above all other objectives. The slack only activates when nature physically did not provide enough water.

Transition to operating: At entry_stage_id, the hydro becomes operational. The storage at the end of the last filling stage becomes the initial storage for the first operating stage. If filling_target_violation > 0 (filling fell short), the operating stage’s own storage_violation_below slack handles the shortfall — both LPs remain feasible.

Penalties during filling: The same outflow_violation_cost from penalties.json applies. Spillage during filling also incurs spillage_cost. The storage_violation_below slack is not active during filling (storage is allowed below min_storage_hm3 by design).

8. LP Objective Function Impact

For each stage t, block b, scenario s:

minimize:
  // Resource costs
  + Sigma_thermal (generation x cost_per_mwh)
  + Sigma_contract (import x import_price - export x export_price)

  // Recourse slacks (LP feasibility)
  + Sigma_bus Sigma_segment (deficit_segment x segment_cost)
  + Sigma_bus (excess x excess_cost)

  // Constraint violation penalties (policy shaping)
  + Sigma_hydro (storage_violation_below x storage_violation_below_cost)
  + Sigma_hydro_filling (filling_target_violation x filling_target_violation_cost)
  + Sigma_hydro (turbined_violation_below x turbined_violation_below_cost)
  + Sigma_hydro (outflow_violation_below x outflow_violation_below_cost)
  + Sigma_hydro (outflow_violation_above x outflow_violation_above_cost)
  + Sigma_hydro (generation_violation_below x generation_violation_below_cost)
  + Sigma_hydro (evaporation_violation_pos x evaporation_violation_pos_cost)
  + Sigma_hydro (evaporation_violation_neg x evaporation_violation_neg_cost)
  + Sigma_hydro (water_withdrawal_violation_pos x water_withdrawal_violation_pos_cost)
  + Sigma_hydro (water_withdrawal_violation_neg x water_withdrawal_violation_neg_cost)

  // Regularization costs (solution guidance)
  + Sigma_hydro (spillage x spillage_cost)
  + Sigma_hydro_fpha (turbined_flow x fpha_turbined_cost)
  + Sigma_hydro (diversion x diversion_cost)
  + Sigma_ncs (curtailment x curtailment_cost)
  + Sigma_line (direct_flow x exchange_cost + reverse_flow x exchange_cost)

  // Future cost function
  + theta  // Cut approximation (future cost)

Note on pumping: Pumping stations do not have an explicit cost in the objective. The cost of pumping is implicitly captured through the energy consumed at the connected bus (appears as negative demand in the load balance). The marginal cost at that bus determines the effective pumping cost.

Cross-References

• Input System Entities — Entity registries with optional penalty overrides
• Input Constraints — Time-varying entity bounds; generic constraint slack penalties
• Input Hydro Extensions — Hydro geometry for evaporation calculation
• Internal Structures — Pre-resolved penalty tables, FPHA turbined cost rationale
• LP Formulation — Cost taxonomy (§1) and penalty terms (§9)
• Configuration Reference — Penalty-related config settings
• Design Principles — General design approach