Production Scale Reference

Purpose

This spec defines the production-scale dimensions of the Cobre SDDP solver: system sizes, LP variable and constraint counts, state dimension formulas, and performance expectations by scale. It serves as the reference for capacity planning, memory budgeting, and performance regression detection.

1. Production Scale Dimensions

Based on the target production scenario:

Decision DEC-009 (active): 60 stages is the production-scale reference baseline for all capacity planning, memory budgets, and performance targets across the corpus. Some specs reference 120 stages as a theoretical maximum or worst-case bound (e.g., FLOP cost calculations, temporal flattening memory estimates, hypothetical sizing). Such references are explicitly labeled as “worst-case” or “hypothetical maximum” to distinguish from the production baseline.

2. State Dimension Estimates

2.1 State Variables and Dimension

The state dimension determines the size of Benders cuts. State variables include:

State Dimension Formula:

$$N_{state}=N_{hydro}+\sum _{h=1}^{N_{hydro}}{P}_h+N_{battery}+\sum _{gnl}{L}_{gnl}$$

For the current implementation (batteries and GNL deferred, both zero):

$$N_{state}=N_{hydro}+\sum _{h=1}^{N_{hydro}}{P}_h$$

For production scale (160 hydros, AR order up to 12):

• Storage: 160
• AR lags: $160\times 12=1,920$ (all hydros at max order)
• Total: 2,080 (production assumption)

Note: The production assumption is $D_{state}=2,080$, corresponding to all hydros using AR(12). This is the worst-case ceiling and the value used for all capacity planning and performance budgeting in this spec. If most hydros use AR(6), the dimension reduces to $160+160\times 6=1,120$, which is the optimistic case.

3. Variable and Constraint Counts

3.1 Variable Count per Subproblem

3.2 Constraint Count per Subproblem

Note: Among active constraints, FPHA hyperplane constraints dominate (~18,750 of ~24,750). The total constraint count including pre-allocated Benders cut slots reaches ~35,000-40,000, but during early iterations most cut constraints are inactive (bounds set to $[-\infty ,+\infty ]$). See Solver Abstraction §5 for pre-allocation design.

3.3 Counting Formulas (Exact)

For precise sizing, use the following formulas where parameters come from the system configuration. The production-scale values in §3.1 and §3.2 are obtained by substituting the parameters from §1.

Variables:

N_VAR = 1                                                      # theta
      + N_BUS × N_BLOCK × (AVG_DEF_SEGMENTS + 1)              # deficit + excess
      + 2 × N_LINE × N_BLOCK                                   # exchange
      + N_HYDRO                                                 # storage
      + N_HYDRO × AR_ORDER                                      # incremental inflow model
      + N_HYDRO × N_BLOCK × 4                                   # q, s, g, inflow
      + N_HYDRO_DIV × N_BLOCK                                   # diversion
      + N_HYDRO_EVAP × N_BLOCK                                  # evaporation
      + N_HYDRO_WITHDRAWAL × N_BLOCK                            # withdrawal
      + N_HYDRO × N_BLOCK × N_SLACK_TYPES                       # slack vars
      + N_THERMAL × N_BLOCK × AVG_COST_SEGMENTS                 # thermal
      + (N_CONTRACT_IMP + N_CONTRACT_EXP) × N_BLOCK             # contracts
      + N_PUMP × N_BLOCK × 2                                    # pump flow + power

Constraints:

N_CON = N_BUS × N_BLOCK                                        # load balance
      + N_HYDRO                                                 # water balance
      + N_HYDRO                                                 # inflow AR dynamics
      + N_HYDRO × AR_ORDER                                      # lagged inflow fixing
      + (N_HYDRO - N_HYDRO_FPHA) × N_BLOCK                       # generation (constant)
      + N_HYDRO_FPHA × N_BLOCK × AVG_FPHA_PLANES                # FPHA (additional)
      + N_HYDRO × N_BLOCK                                       # outflow definition
      + N_HYDRO × N_BLOCK × 2                                   # outflow bounds
      + N_HYDRO × N_BLOCK                                       # turbined min
      + N_HYDRO × N_BLOCK                                       # generation min
      + N_HYDRO_EVAP × N_BLOCK                                  # evaporation
      + N_HYDRO_WITHDRAWAL × N_BLOCK                            # withdrawal
      + N_GENERIC                                                # generic constraints
      + N_CUT_CAPACITY                                           # Benders cuts

State Dimension:

N_STATE = N_HYDRO                                               # storage
        + SUM(AR_ORDER[h] for h in HYDROS)                      # AR lags
        # + N_BATTERY  (deferred C.2)
        # + SUM(GNL_LAG[t] for t in GNL_THERMALS)  (deferred C.1)

Production-scale parameter values: AVG_DEF_SEGMENTS = 1, AVG_COST_SEGMENTS = 1, AVG_FPHA_PLANES = 125, N_HYDRO_EVAP = N_HYDRO = 160, N_HYDRO_WITHDRAWAL = N_HYDRO = 160, AR_ORDER = 12 (worst case). All other parameters as in §1.

4. Performance Expectations by Scale

Note: The timing targets in §4.2 are engineering targets for each test system scale, serving as upper bounds for performance regression testing. Section §4.6 provides a separate first-principles wall-clock model for the Production configuration using LP solve time KPIs (§4.3) and parallelism parameters. The complete derivation is in the timing model analysis audit artifact. All estimates assume fully warm-started steady-state iterations and do not account for I/O, checkpointing, or convergence check overhead.

Purpose: This table provides expected timing targets for different problem scales, enabling performance validation and regression detection. Timings are per-iteration unless otherwise noted.

4.1 Hardware Assumptions

4.2 Test Systems

System Definition

Hardware Configuration and Performance Targets

Note: All timing values are engineering targets based on domain experience. The Production row model-derived estimates from §4.6 are substantially lower than the engineering targets, indicating headroom for I/O, cold-starts, and solver variability. Memory/Rank values are to be calculated using the sizing tool; approximate sizing can be derived from the formulas in §3 and Memory Architecture §2.1.

4.3 Key Performance Indicators

4.4 Scaling Expectations

Thread utilization at 4 ranks: The forward and backward passes have different utilization characteristics. Forward pass: With $M=192$ trajectories distributed across $R=4$ ranks, each rank receives $192/4=48$ trajectories. With $T_h=48$ threads per rank, each thread handles exactly 1 trajectory — 100% thread utilization. Backward pass: The $M=192$ trial states (gathered from the forward pass) are distributed across 4 ranks (48 per rank) and then across 48 threads (1 trial state per thread). Each thread evaluates all $N_{open}=10$ openings sequentially for its trial state, maintaining warm-start benefit. Per stage: $48\times 10=480$ LP solves per rank, 10 sequential solves per thread — 100% thread utilization, with sequential depth $N_{open}\times {\tau }_{LP}$. All 4 ranks are active (each rank independently generates cuts from its assigned trial states). The per-iteration compute time is determined by the sequential depth per stage: $(T-1)\times N_{open}\times {\tau }_{LP}$ for the backward pass. See the timing model analysis §8 for the complete utilization comparison.

4.5 Convergence Reference

Note: Iteration counts assume reasonable initial policy (warm-start from previous study). Cold-start may require 2-3x more iterations. The Production system (60 stages, 160 hydros, AR(12)) falls in the “Complex” category.

4.6 Wall-Clock Time Budget

The following per-iteration time budget is derived from a first-principles model at ${\tau }_{LP}=25$ ms, $R=4$ ranks, $T_h=48$ threads/rank, $D_{state}=2,080$ (worst-case AR(12)), InfiniBand HDR. The complete derivation with all intermediate steps is in the timing model analysis.

Forward pass: $T\times \lceil M/(R\times T_h)\rceil \times {\tau }_{LP}=60\times 1\times 25=1,500$ ms. Each rank receives $192/4=48$ trajectories, fully utilizing 48 threads.

Backward pass: $(T-1)\times N_{open}\times {\tau }_{LP}=59\times 10\times 25=14,750$ ms. Each rank sequentially solves all 10 openings per stage for warm-start benefit.

50-iteration projection:

Sensitivity: Critical LP Solve Time

The model is linear in ${\tau }_{LP}$. LP solves per iteration: $60+59\times 10=650$. Including barrier overhead (2% of backward compute), the LP-equivalent coefficient is $650+11.8=661.8$. The total 50-iteration compute time is:

$$T_{total}^{full}\approx 50\times 661.8\times {\tau }_{LP}=33,090\times {\tau }_{LP}\text{ seconds}$$

Setting $T_{total}^{full}=7,200$ s and solving for the critical LP solve time:

$${\tau }_{LP}^{crit}=\frac{7,200}{33,090}\approx 217.6\text{ ms}$$

If the effective LP solve time exceeds approximately 218 ms, the 50-iteration budget is violated. With 60 stages, 10 openings, and 4 ranks, the solver has substantial headroom — the critical threshold is nearly 9x the baseline assumption.

Sensitivity: Backward Pass Warm-Start Rate

The backward pass constitutes ~89% of compute. If the backward pass warm-start hit rate ${\eta }_{ws}^{bwd}$ drops below 100% (the design target from Work Distribution §2.3), LP solve times blend between warm-start (${\tau }_{ws}=25$ ms) and cold-start (${\tau }_{cs}=250$ ms):

With 10 openings and 60 stages, warm-start remains important for performance but is not an existential threat to the 2-hour budget. Even at 70% hit rate, the solver uses ~40% of the budget, leaving 60% headroom for other overhead. The sequential opening evaluation design (Work Distribution §2.3) still targets near-100% warm-start for optimal LP throughput.

4.7 Model Assumptions

Every timing model estimate in §4.6 depends on the assumptions below. The table distinguishes spec-defined KPIs (which are targets to be met by the implementation), spec-defined architecture parameters (which are design decisions), and engineering estimates (which need validation).

High-sensitivity parameters: LP solve time (${\tau }_{ws}=25$ ms) is the primary assumption with material impact on the time budget, though the critical threshold (~218 ms) provides nearly 9x headroom above the baseline. With 10 openings, backward pass warm-start hit rate (${\eta }_{ws}^{bwd}$) is a moderate concern — at 70% hit rate, the budget fraction reaches ~40%, still well under the limit but with less headroom than the baseline. Early solver benchmarking should prioritize measuring LP solve time under warm-start conditions with production-scale subproblems (~8,400 variables, ~25,000 active constraints).

Cross-References

• Design Principles — Format selection criteria and design goals
• Notation Conventions — Mathematical symbols used in formulas above
• LP Formulation — Complete LP subproblem that these dimensions describe
• SDDP Algorithm — Forward/backward pass structure driving performance targets
• Solver Abstraction §5 — Cut pool pre-allocation and capacity management
• Solver Workspaces §1.2 — Per-thread workspace sizing (~36 MB HiGHS, ~21 MB CLP)
• Solver Abstraction SS11.4 — StageLpCache design and sizing (~22.3 GB SharedRegion)
• Memory Architecture §2 — Node memory budget (~27.7 GB), two-tier model
• Hybrid Parallelism — MPI+OpenMP architecture for achieving these scaling targets
• Communication Patterns §3 — Communication volume analysis, pure transfer < 0.1%
• Work Distribution §2 — Forward/backward pass distribution, thread-trajectory affinity
• Training Loop §6 — Backward pass stage loop, cut synchronization per stage
• SLURM Deployment — Job scripts for the test system scales above