AI Agents

Eval-gated dev → prod promotion (spec 0025)

Prod promotion is a PR that bumps a pinned image tag. Before opening it, the candidate should have passed the per-role golden evals (and the integration suite) against the live dev environment.

Flow

release → dev auto-rolls (Argo CD Image Updater on the aia-dev Application)
   → pre-promote-evals.yml runs the golden evals against dev,
     judged by the local Qwen model (no external API)
   → green → a `dev-evals-green` commit status is recorded on the
     apps.ai-agents commit
   → promotion PR in runtime.k8s.bor.gitops bumps the pinned tag in
     infrastructure/argocd/app-aia-prod.yaml (gitops-image-promotion skill)
   → merge → Argo CD rolls prod

dev stays loose (auto-rolls every release); only prod waits for a green eval.

Enforcement

dev-evals-green is advisory: it is recorded on the apps.ai-agents commit, and no ruleset on runtime.k8s.bor.gitops requires it. Check it on the release commit before opening the promotion PR. The eval workflow needs the AI_AGENTS_API_KEY secret (a dev bearer key with no admin scope; a machine key for GitHub Actions only — to run the evals locally, see evals/promptfooconfig.yaml).

Adding a golden case

Every production incident should become a permanent regression case. Add a row to the matching role in evals/promptfooconfig.yaml with an llm-rubric assertion describing the correct behavior. Keep the grader on the local model so evals stay zero-cost.