PROD-SR-002 — Validate and measure recommendation effectiveness
| Field | Value |
|---|---|
| Status | Open; source-verified; effectiveness not demonstrated |
| Severity | High |
| Components | sparqd/spark-recommendation, sparqd/workflow-backend, sparqd/oqullus, AI gateway |
| Associated issue | Not opened |
Current behavior and evidence
The model returns three free-text sections. Parsing checks only that the sections exist. It does not validate Spark keys, values, mutability, Spark version, image compatibility, Cogrion task schema, node policy, resource limits, or contradictions between related settings. The prompt is the only guardrail.
No record captures whether advice was reviewed, accepted, rejected, applied, rolled back, or improved a comparable later run. Generated token cost is stored, but recommendation benefit is not.
Expected condition
The model returns a versioned structured schema. Deterministic code validates each suggestion against Spark metadata, workflow-backend rules, environment policy, and mutually dependent settings. The UI exposes evidence and limitations. Comparable subsequent runs measure performance, reliability, resource use, cost, and correctness; generation alone never means “effective.”
Alternatives considered
- Recommended: deterministic candidate rules and validation, with the LLM used for bounded explanation and optional ranking.
- Structured LLM output followed by strict validation. Acceptable as an incremental step but still requires outcome measurement.
- Prompt-only constraints. Rejected because they cannot enforce correctness.
Reproduction and verification
Resolution conditions
Close when invalid or unsupported suggestions cannot reach users, advice states its evidence and confidence, and controlled comparable-run results establish quality and regression boundaries for each supported recommendation class.