How a TaskRun works¶

A TaskRun represents a single attempt to complete a task. When Osmia picks up a labelled issue, it creates a TaskRun that tracks the work from start to finish, including retries, timeouts and human intervention.

State machine¶

Every TaskRun moves through a set of states. The controller enforces the transitions and rejects invalid ones.

stateDiagram-v2
    [*] --> Queued
    Queued --> Running
    Queued --> NeedsHuman
    Running --> Succeeded
    Running --> Failed
    Running --> TimedOut
    Running --> NeedsHuman
    NeedsHuman --> Running
    Failed --> Retrying
    Retrying --> Running
    Succeeded --> [*]
    TimedOut --> [*]

States explained¶

State	What is happening
Queued	The TaskRun has been created but no agent container has been launched yet. This is the initial state.
Running	An AI agent container is actively working on the task: cloning the repo, reading code, making changes, running tests.
NeedsHuman	The agent has asked a question that requires human input. Execution is paused until someone responds.
Succeeded	The agent finished successfully and produced a result (typically a pull request). This is a terminal state.
Failed	The agent failed. If retries are available, the TaskRun moves to `Retrying`. If retries are exhausted, this is terminal.
Retrying	A retry has been scheduled. The controller will create a new agent container and transition back to `Running`.
TimedOut	The agent exceeded its deadline (`max_job_duration_minutes`). This is a terminal state.

Lifecycle walkthrough¶

The full lifecycle of a TaskRun, from issue discovery to pull request:

sequenceDiagram
    participant Poller as Ticketing Poller
    participant Ctrl as Controller
    participant GR as Guard Rails
    participant Engine as Engine
    participant JB as JobBuilder
    participant K8s as Kubernetes
    participant WD as Watchdog
    participant SCM as SCM Backend

    Poller->>Ctrl: New ticket found
    Ctrl->>Ctrl: Generate idempotency key
    Ctrl->>GR: Validate ticket
    GR-->>Ctrl: Pass
    Ctrl->>Engine: BuildExecutionSpec(task)
    Engine-->>Ctrl: ExecutionSpec
    Ctrl->>JB: Build K8s Job from spec
    JB-->>Ctrl: batch/v1.Job
    Ctrl->>K8s: Create Job
    Note over Ctrl: TaskRun → Running
    loop Every 60s
        WD->>K8s: Check heartbeats
        WD->>WD: Evaluate detection rules
    end
    K8s-->>Ctrl: Job completed
    Ctrl->>Ctrl: Read TaskResult
    Ctrl->>SCM: Create pull request
    Note over Ctrl: TaskRun → Succeeded

Step by step¶

Ticket discovered: the ticketing poller finds a labelled issue and hands it to the controller.
Idempotency check: the controller generates a key from the ticket ID and attempt number. If a non-terminal TaskRun already exists for this key, the ticket is skipped. This prevents duplicate work.
Guard rail validation: the ticket is checked against controller-level rules (allowed repos, allowed task types, concurrent job limit, blocked file patterns). If it fails, the ticket is rejected.
Engine selection: the controller picks an engine (from config or ticket labels) and calls BuildExecutionSpec to produce a container specification.
Job creation: the JobBuilder translates the spec into a Kubernetes Job with security contexts, resource limits, tolerations, and environment variables. The TaskRun transitions to Running.
Agent execution: the container starts, clones the repository, and works on the task. It sends heartbeats so the watchdog knows it is alive.
Watchdog monitoring: a background loop checks all running TaskRuns for anomalies (loops, stalls, cost overruns). See Guard Rails Overview.
Completion: when the agent finishes, it writes a result.json with the outcome. The controller reads it and transitions the TaskRun to Succeeded or Failed.
Knowledge extraction: if episodic memory is enabled, the controller extracts knowledge from the completed task (success patterns, failure insights, engine notes) into the persistent knowledge graph. This runs in a background goroutine and does not block the TaskRun.
PR creation: on success, the SCM backend creates a pull request from the agent's branch.
Notification: the ticketing backend is updated and notifications are sent.

Retries¶

If a job fails and retries remain (default max_retries: 1), the controller transitions through Failed → Retrying → Running, creating a new agent container. The retry count increments each time. Once retries are exhausted, the TaskRun enters a terminal Failed state.

Retry helps with transient failures (API timeouts, rate limits). Persistent failures (invalid credentials, unsupported task) will fail again, so clear error messages help humans decide whether to retry manually.

Continuation strategies¶

When an agent hits --max-turns, Osmia has to decide what happens next. There are three strategies:

Strategy	Default?	Engine support	How it works
Git-based	Yes	All engines	The prior run pushes its branch. The retry prompt includes `## Continuation` with instructions to clone that branch and read `git log --oneline -20`
Session persistence	Opt-in	Claude Code only	The `~/.claude/` directory and workspace are stored on a PVC. The retry pod resumes with `--resume <session-id>`, restoring the full conversation, with no git clone and no wasted turns
User-prompted continuation	Opt-in	Claude Code only	When turns are exhausted, the controller asks the user via the approval backend (e.g. Slack) whether to continue or stop. On approval, a new pod resumes the session. Requires session persistence.

Session persistence removes the main downside of git-based continuation: the retry agent has to infer context from git history alone, spending turns re-reading code it already understood. With --resume, the agent continues as if the pod never restarted.

User-prompted continuation builds on session persistence to give humans control over long-running tasks. Instead of retrying indefinitely, the controller pauses and presents a summary of progress, and the operator decides whether the remaining work is worth another batch of turns.

Note: Session persistence and user-prompted continuation currently require the Claude Code engine. Other engines (Codex, Aider, OpenCode) use git-based continuation only.

See Session Persistence and Continuation Prompts in the Claude Code engine docs for configuration details.

Causal diagnosis¶

With diagnosis.enabled: true, Osmia replaces blind retries with corrective action (internal/diagnosis/). When a task fails, a diagnosis pipeline analyses the watchdog reason, the result data and the run's most recent stream events (up to the last 500) to classify the failure. Diagnosis runs before a fallback engine is chosen, and only when a fallback or retry is still possible:

Failure mode	Example	Corrective action
`WrongApproach`	Agent edited wrong files	"Focus specifically on [scope]. Do not modify files outside [area]."
`DependencyMissing`	`module not found` errors	"Before starting, ensure [dependency] is available."
`TestMisunderstanding`	Agent modified test files	"Do not modify test files. The existing tests define expected behaviour."
`ScopeCreep`	Too many files changed	"Limit changes to at most [N] files."
`PermissionBlocked`	`EACCES` errors	"Try [alternative approach] that doesn't require elevated permissions."
`ModelConfusion`	Repeated undo/redo cycles	"Take a step-by-step approach."
`InfraFailure`	OOMKilled, timeout	No retry and no fallback engine. The task is not run again, since the same infrastructure would fail the same way

The diagnosis is injected into the retry prompt alongside the original task. With diagnosis.enable_engine_switch, an engine-specific failure also moves the retry to an engine the task has not tried; see Causal diagnosis.

Heartbeats and staleness¶

Each TaskRun carries a HeartbeatAt timestamp. The agent container is expected to push heartbeats at regular intervals (via the PostToolUse hook). If the time since the last heartbeat exceeds heartbeat_ttl_seconds (default: 300), the TaskRun is considered stale and the watchdog may intervene.

Idempotency¶

Every TaskRun is keyed by <ticket_id>-<attempt>. The controller checks this key before processing any ticket. This ensures:

A ticket is never processed twice in the same poll cycle.
Restarted controllers do not duplicate in-flight work.
Retries get a new attempt number and are tracked separately.

Durable TaskRun state¶

By default, TaskRun state lives only in memory (MemoryStore) and is lost when the controller pod restarts. Set taskrun_store.backend to sqlite to persist TaskRun records to a SQLite database instead:

taskrun_store:
  backend: sqlite
  sqlite:
    path: /data/taskruns.db  # defaults to /data/taskruns.db if omitted

This requires a mounted, writable path. The Helm chart's persistence.enabled: true mounts a PVC at /data for this purpose (see charts/osmia/values.yaml).

Today this only persists state for post-mortem inspection and debugging (querying the SQLite file directly after a crash). The controller does not yet read the store back on startup, so in-flight TaskRuns are still forgotten across a restart. Recovering and rehydrating them is separate follow-up work. See ADR 0001 for the reasoning behind choosing SQLite over a TaskRun CRD.

Real-time monitoring¶

While a TaskRun is in the Running state, two monitoring systems run in parallel:

Progress watchdog: detects loops, stalls, thrashing and cost overruns using heartbeat telemetry.
PRM (Process Reward Model): when enabled, scores agent productivity at each tool call and intervenes with guidance before problems escalate.

Both work on observable telemetry. Neither modifies the agent's behaviour directly.