Structured Output Protocol

Status: Not Implemented. This spec describes a planned design that has not yet been implemented.

Purpose

This spec defines the Cobre structured output protocol: the JSON response envelope schema, the structured error schema with the complete error kind registry, the JSON-lines streaming protocol for long-running operations, per-subcommand output specifications, output format negotiation via CLI flags, and the schema versioning strategy. Together, these define the machine-parseable interface that agents, CI/CD pipelines, and programmatic tools use to interact with the Cobre CLI. The protocol implements the four agent-readability design rules from Design Principles SS6.2 and the four design decisions resolved in the structured CLI impact assessment (findings-019 SS2.1–SS2.4): human-first default, JSON-lines streaming via structured stdout with envelope, hybrid subcommand detection, and the generalized validation report as error schema standard.

1. Response Envelope Schema

Every CLI subcommand produces a response conforming to this envelope when invoked with --output-format json. The envelope generalizes the existing validation report format (Validation Architecture SS5) into a universal response schema.

1.1 Top-Level Fields

1.2 Invariants

Invariant 1: When success is false, errors is non-empty and exit_code is non-zero.

Invariant 2: When success is true, errors is an empty array and exit_code is 0.

Invariant 3: warnings may be non-empty regardless of the value of success.

1.3 Complete Example

{
  "$schema": "urn:cobre:response:v1",
  "command": "validate",
  "success": false,
  "exit_code": 3,
  "cobre_version": "2.0.0",
  "errors": [
    {
      "kind": "MissingReference",
      "message": "bus_id 'BUS_99' not found in buses.json",
      "context": {
        "file": "system/hydros.json",
        "entity_id": "ITAIPU",
        "field": "bus_id",
        "referenced_value": "BUS_99",
        "referenced_registry": "buses.json"
      },
      "suggestion": "Check that bus_id 'BUS_99' exists in system/buses.json. Available bus IDs can be listed from that file."
    }
  ],
  "warnings": [
    {
      "kind": "PhysicalConstraint",
      "message": "Thermal 'THERMAL_OLD' has max_generation=0 for all stages",
      "context": {
        "file": "system/thermals.json",
        "entity_id": "THERMAL_OLD"
      },
      "suggestion": "This thermal is effectively inactive. If this is intentional, no action is needed."
    }
  ],
  "data": null,
  "summary": {
    "files_checked": 24,
    "entities_validated": 456,
    "error_count": 1,
    "warning_count": 1
  }
}

2. Error Schema

2.1 Error Record Fields

Every error and warning record contains the following four fields, implementing Design Principles SS6.2 Rule 2.

2.2 Design Principles for Error Records

Error records follow the agent-friendly conventions from architecture-021 SS4.4:

• Specific: Identify the file, field, entity, stage, or iteration where the problem occurred
• Actionable: Tell the consumer what to check or inspect, not just what went wrong
• Contextual: Include identifiers (entity IDs, stage numbers, iteration counts) needed to locate the problem
• Non-prescriptive: Suggest what to investigate, not what the fix must be

2.3 Error Kind Registry

The error kind registry is the single authoritative list of all structured error kinds emitted by Cobre. Error kinds are stable identifiers governed by semantic versioning (SS6): additions are non-breaking; removals or renames are breaking changes.

The registry is organized into two categories: validation kinds (emitted during input validation, corresponding to Validation Architecture SS4) and runtime kinds (emitted during execution or post-hoc operations).

2.3.1 Validation Error Kinds

These 14 kinds map to the validation layers defined in Validation Architecture SS2.

Note on severity: PhysicalConstraint is the only validation kind that may appear as either an error or a warning. For example, an unused entity (thermal with max_generation = 0 for all stages) is a PhysicalConstraint warning, while an acyclic cascade violation is a PhysicalConstraint error. All other validation kinds are always errors.

Mapping to existing catalog: These 14 kinds refine and generalize the error kinds from Validation Architecture SS4. The mapping is: FileNotFound becomes MissingFile; InvalidReference becomes MissingReference; InvalidValue becomes OutOfRange; CycleDetected, BusinessRuleViolation, UnusedEntity, and ModelQuality are subsumed by PhysicalConstraint, IncompatibleSettings, CapacityViolation, or PenaltyConsistency as appropriate; DimensionMismatch splits into CoverageMismatch and StageMismatch; WarmStartIncompatible, ResumeIncompatible, and NotImplemented are expressed as IncompatibleSettings.

Example – validation error:

{
  "kind": "OutOfRange",
  "message": "Hydro 'SOBRADINHO' has storage_min (5000) greater than storage_max (4500)",
  "context": {
    "file": "system/hydros.json",
    "entity_id": "SOBRADINHO",
    "field": "storage_min",
    "value": 5000,
    "min": 0,
    "max": 4500
  },
  "suggestion": "Check that storage_min <= storage_max in the hydro definition. Current values: storage_min=5000, storage_max=4500."
}

Example – validation warning:

{
  "kind": "PhysicalConstraint",
  "message": "PAR seasonal polynomial for hydro 'TUCURUI' at season 3 has root inside the unit circle",
  "context": {
    "file": "scenarios/inflow_ar_coefficients.parquet",
    "entity_id": "TUCURUI",
    "rule": "par_seasonal_stability",
    "detail": "Characteristic polynomial root magnitude: 0.87"
  },
  "suggestion": "The PAR model for this hydro-season may be unstable. Review the AR coefficients for season 3. This is a warning and does not prevent execution."
}

2.3.2 Runtime Error Kinds

These 6 kinds cover errors that occur during execution or post-hoc operations, outside the scope of input validation.

Example – runtime error:

{
  "kind": "SolverFailure",
  "message": "LP infeasible at stage 5, iteration 23, opening 7",
  "context": {
    "stage": 5,
    "iteration": 23,
    "opening": 7,
    "solver_status": "infeasible",
    "active_penalty_slacks": ["deficit_SE", "deficit_S"]
  },
  "suggestion": "Check penalty costs in penalties.json; large deficits may indicate missing generation capacity or overly tight constraints."
}

Example – post-hoc error:

{
  "kind": "IncompatibleRuns",
  "message": "Cannot compare runs: stage count differs (120 vs 60)",
  "context": {
    "field": "stages",
    "run_a_value": 120,
    "run_b_value": 60,
    "detail": "Both runs must have the same number of stages for a meaningful comparison"
  },
  "suggestion": "Verify that both output directories correspond to the same case configuration or adjust comparison parameters."
}

3. JSON-Lines Streaming Protocol

Long-running operations (training, simulation) emit a stream of self-describing JSON objects to stdout when --output-format json-lines is specified. This implements Design Principles SS6.2 Rule 3 and the streaming transport decision from findings-019 SS2.2.

3.1 Line Format Rules

1. One JSON object per line: Each line is a complete, independently valid JSON object terminated by a newline character (\n, U+000A).
2. No pretty-printing: JSON objects are serialized on a single line with no embedded newlines.
3. UTF-8 encoding: All output is UTF-8 encoded.
4. Type-discriminated: Every JSON object contains a type field as the first key, enabling consumers to dispatch by envelope type without parsing the full object.
5. Ordered emission: Lines are emitted in the order defined below (SS3.2). Consumers may rely on this ordering.

3.2 Envelope Types

The JSON-lines stream uses four envelope types, emitted in a fixed lifecycle order:

started  -->  progress (repeated)  -->  terminated  -->  result

3.2.1 started Payload

Emitted once when training or simulation begins. Corresponds to the TrainingStarted lifecycle event from architecture-021 SS3.2.

{
  "type": "started",
  "case": "/data/case_001",
  "stages": 120,
  "hydros": 164,
  "thermals": 82,
  "ranks": 8,
  "threads_per_rank": 16,
  "timestamp": "2026-02-25T10:00:00Z",
  "cobre_version": "2.0.0"
}

3.2.2 progress Payload

Emitted once per training iteration after convergence evaluation. The payload matches the ConvergenceUpdate event struct from architecture-021 SS3.3 and the per-iteration output record from Convergence Monitoring SS2.4.

{
  "type": "progress",
  "iteration": 42,
  "lower_bound": 72105.4,
  "upper_bound": 73211.8,
  "upper_bound_std": 1842.3,
  "ci_95": 361.1,
  "gap": 0.0151,
  "wall_time_ms": 523400,
  "iteration_time_ms": 12400
}

Canonical source: The 8 fields of the progress payload are exactly the 8 fields of the per-iteration output record defined in Convergence Monitoring SS2.4. Implementations must keep these in sync. The ConvergenceUpdate Rust struct (architecture-021 SS3.3) is the shared type definition.

For simulation operations, a simulation-specific progress event is emitted instead:

{
  "type": "progress",
  "phase": "simulation",
  "scenarios_complete": 500,
  "scenarios_total": 2000,
  "elapsed_ms": 34200
}

3.2.3 terminated Payload

Emitted once when the iterative loop ends. Corresponds to the TrainingFinished lifecycle event from architecture-021 SS3.2.

{
  "type": "terminated",
  "reason": "bound_stalling",
  "iterations": 87,
  "final_lb": 72105.4,
  "final_ub": 73211.8,
  "total_time_ms": 1082400,
  "total_cuts": 10440
}

3.2.4 result Payload

Emitted as the final line of the stream. Contains the complete response envelope (SS1) with the data field populated by the subcommand-specific result (SS4).

{
  "type": "result",
  "$schema": "urn:cobre:response:v1",
  "command": "run",
  "success": true,
  "exit_code": 0,
  "cobre_version": "2.0.0",
  "errors": [],
  "warnings": [],
  "data": {
    "output_directory": "/data/case_001/output",
    "training": {
      "iterations": 87,
      "final_lower_bound": 72105.4,
      "final_upper_bound": 73211.8,
      "final_gap": 0.0151,
      "termination_reason": "bound_stalling",
      "total_cuts": 10440,
      "wall_time_ms": 1082400
    },
    "simulation": {
      "scenarios": 2000,
      "output_files": ["simulation/costs.parquet", "simulation/hydros.parquet"],
      "wall_time_ms": 245000
    }
  },
  "summary": {
    "status": "completed",
    "total_time_ms": 1327400
  }
}

3.3 Partial Output Semantics

Crash recovery: Each line in the JSON-lines stream is independently valid JSON. If the process crashes or is killed mid-stream, a consumer can parse all completed lines without error. The absence of a result line indicates abnormal termination. The absence of a terminated line indicates the iterative loop did not complete.

Consumer guidance: Consumers should not assume the result line will be present. A robust consumer treats the last successfully parsed progress line as the best-known state if no terminated or result line appears.

3.4 Relationship to Event Stream Architecture

The JSON-lines streaming protocol is one consumer of the shared event stream defined in architecture-021 SS3. The JSON-lines writer subscribes to the broadcast channel alongside other consumers (text logger, TUI renderer, Parquet writer). The progress payload is the JSON serialization of the ConvergenceUpdate event type. The started and terminated payloads serialize the TrainingStarted and TrainingFinished lifecycle events, respectively. See architecture-021 SS3.4 for the consumer registration pattern.

4. Subcommand Output Specifications

Each subcommand defines the shape of the data field in the response envelope (SS1) and its behavior in both json and json-lines output modes. Subcommand definitions are derived from the inventory in findings-019 SS2.3 and the phase mapping in CLI and Lifecycle SS5.

4.1 run

Executes the full lifecycle (validation, training, simulation) or a subset thereof.

--output-format json: Emits a single response envelope after the operation completes. No streaming progress is emitted. This is the appropriate mode when the consumer only needs the final result (e.g., CI/CD pipelines, batch post-processing).

--output-format json-lines: Emits the full streaming protocol (SS3): started, progress (per iteration), terminated, and result. This is the appropriate mode for real-time monitoring.

data shape:

{
  "output_directory": "/data/case_001/output",
  "training": {
    "iterations": 87,
    "final_lower_bound": 72105.4,
    "final_upper_bound": 73211.8,
    "final_gap": 0.0151,
    "termination_reason": "bound_stalling",
    "total_cuts": 10440,
    "wall_time_ms": 1082400
  },
  "simulation": {
    "scenarios": 2000,
    "output_files": [
      "simulation/costs.parquet",
      "simulation/hydros.parquet",
      "simulation/thermals.parquet",
      "simulation/exchanges.parquet"
    ],
    "wall_time_ms": 245000
  }
}

Note: If training is disabled (training.enabled = false), the training field is null. If simulation is disabled, the simulation field is null. At least one must be non-null for a run to succeed.

4.2 validate

Validates all input files in a case directory without executing the solver.

--output-format json: Emits a single response envelope. This is the primary use case for validate.

--output-format json-lines: Behaves identically to json mode. Validation is not a streaming operation; there is no iterative progress to report. A single JSON object is emitted on one line.

data shape:

{
  "case_directory": "/data/case_001",
  "timestamp": "2026-02-25T10:05:00Z",
  "layers_completed": 5,
  "files_checked": 24,
  "entities_validated": 456
}

summary shape (same for both modes):

{
  "files_checked": 24,
  "entities_validated": 456,
  "error_count": 2,
  "warning_count": 1
}

4.3 report

Summarizes a completed run’s outputs by reading manifests and metadata from an output directory.

--output-format json: Emits a single response envelope with the run summary.

--output-format json-lines: Behaves identically to json mode. Report generation is not a streaming operation.

data shape:

{
  "output_directory": "/data/case_001/output",
  "case_name": "case_001",
  "status": "completed",
  "cobre_version": "2.0.0",
  "training": {
    "iterations": 87,
    "final_lower_bound": 72105.4,
    "final_upper_bound": 73211.8,
    "final_gap": 0.0151,
    "termination_reason": "bound_stalling",
    "total_cuts": 10440,
    "wall_time_ms": 1082400,
    "ranks": 8,
    "threads_per_rank": 16
  },
  "simulation": {
    "scenarios": 2000,
    "wall_time_ms": 245000,
    "output_files": ["simulation/costs.parquet", "simulation/hydros.parquet"]
  },
  "metadata": {
    "config_hash": "sha256:a1b2c3...",
    "system_hash": "sha256:d4e5f6...",
    "started_at": "2026-02-25T10:00:00Z",
    "finished_at": "2026-02-25T10:22:07Z"
  }
}

4.4 compare

Compares two completed run output directories.

--output-format json: Emits a single response envelope with comparison results.

--output-format json-lines: Behaves identically to json mode. Comparison is not a streaming operation.

data shape:

{
  "run_a": "/data/case_001/output_v1",
  "run_b": "/data/case_001/output_v2",
  "compatible": true,
  "training": {
    "iterations_a": 87,
    "iterations_b": 92,
    "final_lb_a": 72105.4,
    "final_lb_b": 72201.1,
    "final_lb_delta": 95.7,
    "final_lb_delta_pct": 0.0013,
    "final_ub_a": 73211.8,
    "final_ub_b": 73150.2,
    "final_ub_delta": -61.6,
    "final_ub_delta_pct": -0.0008,
    "final_gap_a": 0.0151,
    "final_gap_b": 0.013
  },
  "simulation": {
    "mean_cost_a": 72580.3,
    "mean_cost_b": 72615.1,
    "mean_cost_delta": 34.8,
    "mean_cost_delta_pct": 0.0005,
    "std_a": 1842.3,
    "std_b": 1790.1
  }
}

4.5 version

Prints version and build information.

--output-format json: Emits a single response envelope with version data.

--output-format json-lines: Behaves identically to json mode.

data shape:

{
  "version": "2.0.0",
  "git_commit": "a1b2c3d",
  "build_date": "2026-02-20",
  "rustc_version": "1.85.0",
  "solver_backend": "highs",
  "solver_version": "1.9.0",
  "mpi_available": true,
  "features": ["mpi", "highs"]
}

4.6 Summary of Output Mode Behavior

5. Output Format Negotiation

5.1 The --output-format Flag

The global CLI flag --output-format <format> controls the presentation layer. It does not affect computation, solver behavior, or output files written to disk.

Design decision reference: The human-first default follows findings-019 SS2.1. Agents must explicitly opt in to structured output via --output-format json or --output-format json-lines.

5.2 Interaction with --quiet and --no-progress

Rationale for --quiet: Some CI/CD pipelines only inspect the exit code and the output files on disk. --quiet eliminates all stdout, reducing log noise. This applies uniformly across all output modes.

Rationale for --no-progress: In json-lines mode, suppressing progress lines reduces output volume for consumers that only need the lifecycle events and final result. The started and terminated envelopes are retained because they carry metadata not available in the result envelope.

5.3 TTY Detection

Cobre does not auto-detect whether stdout is a TTY to switch output formats. The output format is always determined by the explicit --output-format flag (defaulting to human).

Rationale: Auto-detection creates unpredictable behavior for agents that redirect stdout through pipes, MPI launchers, or container runtimes where TTY status is ambiguous. Explicit is better than implicit (findings-019 SS2.1).

TTY status is used for one purpose only: when --output-format human is active and stdout is a TTY, the text logger may use ANSI color codes for emphasis. When stdout is not a TTY, color codes are suppressed. This is purely cosmetic and does not affect content.

5.4 MPI Rank-0 Rule

Rule: In MPI execution, only rank 0 produces structured output to stdout. Non-rank-0 processes suppress all stdout and stderr output. This is consistent with existing behavior documented in CLI and Lifecycle SS5.2 and the MPI note in findings-019 SS2.3.

This rule applies to all output formats (human, json, json-lines). The structured output protocol is a rank-0-only concern. Output files written to disk (Parquet, manifests, metadata) follow the existing distributed I/O patterns documented in Design Principles SS2 Goal 5.

6. Schema Versioning

6.1 $schema URN Format

Every JSON response envelope includes a $schema field with a URN that identifies the schema version:

urn:cobre:response:v1

URN structure:

The version component is a major version only. Minor/patch updates to the schema that are additive-only do not increment the version number.

6.2 Additive-Only Evolution Rules

Schema evolution within a major version follows strict additive-only rules:

6.3 Breaking Change Policy

When a breaking change is required:

1. The $schema version is incremented (e.g., urn:cobre:response:v2).
2. The previous version continues to be supported for at least one major Cobre release cycle.
3. Consumers detect the version via the $schema field and can dispatch to version-appropriate parsing logic.
4. The structured output spec is updated with both the old and new schema documented during the overlap period.

6.4 Version Compatibility Check

Consumers should validate the $schema field before parsing:

if response["$schema"] starts with "urn:cobre:response:v1":
    parse with v1 schema
else:
    report unsupported schema version

The cobre_version field (SemVer) provides additional granularity for consumers that need to detect specific additive changes within a schema major version.

Cross-References

• Design Principles – Agent-readability rules (SS6.2 Rules 1-4), format selection criteria (SS1), declaration order invariance (SS3)
• Validation Architecture – Validation layers (SS2), error type catalog (SS4), validation report format (SS5) that this spec generalizes
• Convergence Monitoring – Per-iteration output record (SS2.4) that defines the progress payload fields; training log format (SS4) that is the human mode equivalent
• Training Loop – Iteration lifecycle (SS2.1) defining the 7 steps at which events are emitted; forward/backward pass structure
• CLI and Lifecycle – Invocation pattern (SS2), command-line interface (SS3), exit codes (SS4), execution phases (SS5)
• Output Infrastructure (planned update) – Manifests and metadata accessed by the report subcommand
• Configuration Reference – Stopping rules and mode selection that determine training outcomes reported in data
• MCP Server – MCP tool definitions that consume this protocol’s error schema and event types
• Python Bindings – Python exception hierarchy mapped from this protocol’s error kinds
• Terminal UI – TUI rendering that consumes the same event stream as the JSON-lines writer