Eval-gated dev → prod promotion (spec 0025)
Prod promotion is a PR that bumps a pinned image tag. Before opening it, the candidate should have passed the per-role golden evals (and the integration suite) against the live dev environment.
Flow
release → dev auto-rolls (Argo CD Image Updater on the aia-dev Application)
→ pre-promote-evals.yml runs the golden evals against dev,
judged by the local Qwen model (no external API)
→ green → a `dev-evals-green` commit status is recorded on the
apps.ai-agents commit
→ promotion PR in runtime.k8s.bor.gitops bumps the pinned tag in
infrastructure/argocd/app-aia-prod.yaml (gitops-image-promotion skill)
→ merge → Argo CD rolls prod
dev stays loose (auto-rolls every release); only prod waits for a green eval.
Enforcement
dev-evals-green is advisory: it is recorded on the apps.ai-agents commit, and
no ruleset on runtime.k8s.bor.gitops requires it. Check it on the release
commit before opening the promotion PR. The eval workflow needs the
AI_AGENTS_API_KEY secret (a dev bearer key with no admin scope; a machine key for
GitHub Actions only — to run the evals locally, see evals/promptfooconfig.yaml).
Adding a golden case
Every production incident should become a permanent regression case. Add a row
to the matching role in evals/promptfooconfig.yaml with an llm-rubric
assertion describing the correct behavior. Keep the grader on the local model so
evals stay zero-cost.