Architecture

Internals of the ai-agents execution engine. The engine uses Kubernetes CRDs as the sole state store — no Redis, no BullMQ, no database.

For the top-level system view (context, building blocks, deployment, RAD conformance) see the Software Architecture Document. This page is the engine-internals deep-dive.

Deployable Processes

One TypeScript codebase (npm workspace) produces two server entrypoints, a Next.js frontend, and a set of executor images:

The operator controller is conditional on the API process via ENABLE_OPERATOR (default true). Webhook pods run the same API image in MODE=webhook: when a WebhookSource CR is created, the API server auto-provisions a Deployment + Service + Ingress for it (src/k8s/webhook-sources.ts), each cleaned up via a finalizer on CR deletion. There are no separate processor/webhook images or charts — everything ships from the single charts/ai-agents-main chart.

GitHub event ──► Webhook pod ──► InferenceRequest CR
POST /api/jobs/submit ─────────►        │
POST /api/events/ingest ───────►        ▼
            ┌──────────────────────────────────────────┐
            │  API server (operator reconcile loop)     │
            │  watches IR CRs, creates k8s Jobs         │
            └──────────────────────────────────────────┘
                                         │
                                         ▼
                              ephemeral k8s Job
                              (executor container)
                                         │ inference + MCP tool callbacks
                                         ▼
                              Inference Proxy ──► Anthropic / OpenAI / GX10 vLLM / Ollama

CRDs (group: labrats.work, version: v1alpha1)

Prompts are not stored inline in IR specs (CRD size limit ≈1 MiB). Each IR holds a promptRef pointing to a per-job ConfigMap ({irName}-prompt).

InferenceRequest (full shape, src/k8s/inference-request-types.ts)

spec: {
  source: string                    // "github", "manual", "dashboard", "github-actions", ...
  type: string                      // "implement-issue", "code-review", "document-extraction", ...
  agentRole?: string                // matches AiAgent.metadata.name or trigger
  priority?: number                 // default 2; lower = higher priority
  promptRef: { configMapName, key } // ConfigMap holding the prompt
  workspaceSetup?: {
    gitRepo, gitRef, gitDepth, githubToken
  }
  workspaceFiles?:   [{ name, data }]      // base64-encoded
  workspaceUploads?: [{ id, name }]        // RWX uploads PVC references
  config?: { model, effort, fast, provider, timeoutMs, maxTurns, executorImage, reactEnabled }
  callbackUrl?: string
  pipeline?: { type: "pdf-sections" }
  chain?: {
    chainId, chainDepth, parentRequestName, chainPath[]
    nextAgentTrigger, nextAgentCondition, nextFailurePolicy
  }
  retryPolicy?: { maxAttempts, backoffMs }
  metadata?: Record<string, unknown>
  errors?: [{ context, message, stack, timestamp }]
}

status: {
  phase: "Pending"|"Running"|"Succeeded"|"Failed"|"Cancelled"
  attempts, accountName, k8sJobName, executorPodName
  startedAt, completedAt
  result: { exitCode, timedOut, budgetExceeded, budgetReason, usage{...}, provider, model, effort, fast }
  failedReason, logs[], conditions[]
}

Operator (src/operator/)

Controller (controller.ts)

• Informer-driven plus a 5-second poll fallback (DEFAULT_POLL_INTERVAL_MS = 5_000).
• Default concurrency is 5; override with WORKER_CONCURRENCY.
• Tracks in-flight reconciles in an internal Set of IR names (processing).
• Pending IRs are sorted by priority ascending, then by creationTimestamp.
• Running IRs are reconciled before Pending IRs so status polling stays fresh.
• New events (informer add/update) are debounced via setImmediate into the next reconcile pass.
• A slow retention GC sweep (gc.ts, default every 10 min via IR_GC_INTERVAL_MINUTES) hard-deletes terminal IRs past their per-phase TTL — IR_TTL_SUCCEEDED (24h), IR_TTL_FAILED (168h), IR_TTL_CANCELLED (24h); 0 disables a phase. (spec 0016)

Reconciler (reconciler.ts)

On terminal status the reconciler also calls maybeChainNextAgent() to optionally enqueue a child IR.

Prompt Builder (prompt-builder.ts)

Resolution at job start:

config.model    : IR.spec.config.model
                  → AiAgent.spec.defaultModel
                  → AiConfig.spec.providers[provider].model
                  → AiConfig.spec.model
                  → CLI default (no flag passed)

config.effort   : IR.spec.config.effort
                  → AiAgent.spec.defaultEffort
                  → AiConfig.spec.providers[provider].effort
                  → AiConfig.spec.effort
                  → "medium"

config.provider : IR.spec.config.provider
                  → AiAgent.spec.provider
                  → "claude"

Instruction injection (instructionWatcher.resolveForAgent()):

1. Collect all global AiInstructions sorted by priority ascending.
2. Append local instructions listed in the agent's spec.instructions[].
3. Prepend each as an <instruction name="…" scope="…">…</instruction> XML block to the prompt.

Pipeline jobs (currently only pdf-sections) skip instruction injection.

Chain (chain.ts)

After job completion, maybeChainNextAgent():

1. Returns immediately if spec.chain.nextAgentTrigger is unset.
2. Loop guard — refuses if nextAgentTrigger already appears in chainPath.
3. Depth guard — MAX_CHAIN_DEPTH = 10.
4. Condition — evaluates nextAgentCondition (always | on-success | on-failure) against exitCode === 0.
5. Failure policy — stop | skip | notify (only blocks chaining when policy says so).
6. Captures up to 4,000 chars of parent stdout, wraps it in <chain-context>…</chain-context>, and prepends it to the parent prompt to form the child prompt.
7. Creates a child IR with chainDepth + 1, extended chainPath, and the next agent's own next config (resolved from its eventTriggers).

See Agent Chains for configuration details.

Recursive sub-agent delegation (spec 0013). Separate from event chains, the bundled MCP server's delegate_subtask tool spawns a focused sub-agent — a child IR with its own tools and a small turn budget — capped at depth 3 with a cycle guard via the same spec.chain lineage. To stay deadlock-free the parent parks while waiting: POST /api/jobs/:id/park releases its AiModel slot and unpark re-acquires before it resumes, so leaf sub-agents free slots and the tree drains bottom-up (status.parked tracks it).

Job Executor (src/k8s/job-executor.ts)

Each IR transition to Running creates one ephemeral k8s batch/v1 Job.

Mounts on the executor pod:

Defaults:

• Image: per-provider executor image (default ghcr.io/labrats-work/apps.ai-agents/executor-claude:<app-version>, or the EXECUTOR_IMAGE env). Resolved per provider from AiExecutor CRs / AiConfig.spec.executors, and overridable per-job via the IR's spec.config.executorImage.
• Resources: requests cpu: 100m, memory: 512Mi; limits cpu: 2, memory: 4Gi (uniform default for all providers). Requests are intentionally a low floor — executors are I/O-bound on remote inference; the limits carry burst headroom (#1261). Override per provider via the AiExecutor CR's spec.resources.
• activeDeadlineSeconds = ceil(timeoutMs / 1000) + 60 (default timeoutMs = 3 300 000 ms / 55 min).
• Labels: app.kubernetes.io/managed-by=ai-agents, app.kubernetes.io/component=executor, ai-agents.labrats.work/job-id=<truncated jobId>.
• ServiceAccount: from EXECUTOR_SERVICE_ACCOUNT env. Must allow PATCHing the credentials Secret (used by the executor entrypoint to persist OAuth refresh on exit).

The operator polls the Job on each reconcile pass and, on a terminal Job condition, patches the IR status (phase, usage) and deletes the Job. The executor never writes IR status — it only PATCHes its own credentials Secret on exit.

Executor Container (executor/)

One image per CLI provider, built from executor/Dockerfile.{claude,codex,qwen-code,gemini} (Node Alpine base, CLIs pre-installed). All run as user executor (uid 1001). The qwen-code image bundles Qwen Code (a Gemini-CLI fork) pointed at an OpenAI-compatible endpoint — the in-cluster Inference Proxy, which fronts the GX10 vLLM backend.

vllm and ollama are not executor images: they are HTTP inference backends. When an AiModel has a spec.providers[].url, the reconciler routes inference via HttpInferenceClient (src/k8s/http-inference-client.ts) to that URL instead of creating a Job (falling back to a Job if the endpoint is unhealthy). mock is a test provider.

gemini is a fully supported, built provider (spec 0027) — it runs from its own entrypoint-gemini.sh / Dockerfile.gemini. The legacy gemma / bonsai / qwen (llama-cli + GGUF) branches were removed from entrypoint.sh (no image ever built them); the live providers are claude / codex / qwen-code / mock (+ gemini).

Entrypoint flow (executor/entrypoint.sh):

1. Raise file descriptor limit (claude-code spawns many fsnotify watchers).
2. Credentials:
   • PROVIDER=claude → copy /credentials/token into $HOME/.claude/.credentials.json (mode 0600). On EXIT, PATCH the Secret to persist refreshed tokens.
   • PROVIDER=codex → read /credentials/token into CODEX_API_KEY.
   • PROVIDER=qwen-code → render ~/.qwen/settings.json from the template, pointing at OPENAI_BASE_URL (the proxy); the caller's key is presented as the API key.
3. Git clone: read repo URL from /git-config/repo-url (or GIT_REPO_URL env); shallow clone with depth from GIT_DEPTH (default 1).
4. Workspace uploads: if WORKSPACE_UPLOADS env is set (JSON [{id,name}]), copy /uploads/{id}/{name} into the workspace.
5. MCP servers & skills: translate the MCP_SERVERS env into ~/.claude.json / ~/.qwen/settings.json, and install any mounted /skills/<name> into the CLI's skills dir for auto-discovery.
6. Prompt: read from /prompt/prompt.txt (or PROMPT_TEXT env).
7. CLI invocation (wrapped in timeout ${TIMEOUT_SEC}s):
   • claude — claude --print --verbose --dangerously-skip-permissions --output-format stream-json --no-session-persistence [--model M] [--effort E] [--fast] [--max-turns N]
   • codex — codex exec --json --full-auto --skip-git-repo-check [--model M] [--config model_reasoning_effort=E]
   • qwen-code — the Qwen Code CLI against the configured OpenAI-compatible endpoint.
   • mock — log metadata, exit 0 (used in tests).
8. Exit code is propagated; 124 means timeout.

Model Slots & Credentials (src/k8s/models.ts)

There is no AiAccount pool — it was removed in spec 0003; credential and concurrency tracking moved onto AiModel. A watcher-driven informer keeps a local cache of AiModel CRs.

• Status states (AiModelCR.status.state): idle, busy, unavailable, error. spec.paused: true keeps a model out of rotation.
• Concurrency: spec.maxConcurrency caps simultaneous jobs per model (0 = unlimited); status.activeJobs / status.activeJobIds[] track in-flight IRs.
• Acquisition (acquire()) uses an optimistic JSON Patch with a test op on status.activeJobs to avoid races; on conflict the next candidate is tried.
• Selection filters out paused / error / unavailable models and those at max concurrency, optionally filters by model name, then sorts by load ratio ascending, then least-recently-used. A provider URL is chosen from spec.providers[] (skipping unavailable ones).
• Crash recovery: a periodic in-operation slot audit (auditSlots, every 60s, also once at startup) reclaims only slots whose activeJobIds reference IRs that are missing or terminal. Startup deliberately does not zero busy models — in-flight executor Jobs survive an operator restart (#1525).

Credentials

• Caller identity: every IR carries spec.apiKeyRef (required) → executor AI_AGENTS_API_KEY for the MCP callback. Sourced from the request's Bearer token (submit/events/ingest), WebhookSource.spec.apiKeyRef (webhook events), AiAgent.apiKeyRef (triggers), or inherited (chains). No long-lived executor service credential. (spec 0007)
• Backend token: optional AiModel.spec.credentialsRef → /credentials/token, consumed only by the claude executor. qwen-code/codex use the caller's key, so discovered models need none.
• GitHub: short-lived App installation tokens minted per webhook event; webhooks HMAC-verified with the App webhookSecret. No PAT.

Webhook Ingest

GitHub webhooks land at POST /api/webhooks/github on the webhook pod (or the API pod for the unified deployment).

src/sources/github/index.ts flow:

1. Look up the WebhookSource CR by spec.type === "github".
2. Read webhookSecret, appId, privateKey, installationId from the credentials Secret referenced by spec.credentialsRef.
3. Verify HMAC-SHA-256 (x-hub-signature-256 header vs raw body).
4. Normalize x-github-event + payload.action into a canonical event name.
5. Public-repo filter: if payload.repository.private === false, deny unless allowPublicRepos: true or repository.full_name is in allowedPublicRepos. Returns 200 with matched: 0 on deny (not 4xx).
6. Match agents via agentWatcher.findByEvent(source, event) plus mention parsing (@ai-{role}, @agents-assemble).
7. Create one InferenceRequest per matched agent via irClient.create(...).

Workspace and Uploads

The shared RWX uploads PVC is mounted read-write by the API pod and read-only by executor Jobs. Files referenced via workspaceUploads are copied from /uploads/{id}/{name} into the workspace before the CLI runs. The PVC name comes from UPLOADS_PVC_NAME.

Per-job ConfigMaps and git Secrets are deleted after Job termination via owner references and TTL.

Metrics and Logs

• GET /metrics exposes Prometheus metrics (registry from src/metrics.ts). The dashboard timeseries page reads from PROMETHEUS_URL.
• GET /api/jobs/:id/logs is a Server-Sent Events stream that:
  1. Streams executor pod logs live (follow: true) when an executorPodName is recorded.
  2. Falls back to tailing the persisted log file in the uploads PVC (used by pipeline jobs).
  3. Closes with {done:true,status:…} once the IR reaches a terminal phase.

Config Defaults Resolution

Job submission spec.config
        │
        ▼ (per-field fallback)
AiAgent.spec.defaultModel / defaultEffort / provider
        │
        ▼
AiConfig.spec.providers[provider].model / effort
        │
        ▼
AiConfig.spec.model / effort
        │
        ▼
Hard defaults: provider="claude", effort="medium"

AiConfig.spec.executorImage and executorResources apply uniformly to all jobs unless an IR explicitly overrides them.