Osmia Feature Roadmap¶

This document tracks what to work on next, in priority order. Completed work is archived at the bottom.

Status Legend¶

[ ] Not started
[x] Complete
In Progress — marked in the item description when actively being worked on

1. Live-Backend E2E Validation — Remaining Gaps¶

The fake-agent E2E workflow test suite (Phase 5) covers all subsystem interactions end-to-end against a real kind cluster. Two live tests (make e2e-live-test) now run against the real Shortcut + GitLab + Claude Code stack. The items below require extended live workloads (50+ tasks, 3-engine comparison) and will be validated once a staging environment is available.

Live Test Suite ✅ (initial coverage)¶

[x] Happy path — story created → Claude Code picks up → MR opened → story done (TestLiveHappyPath)
[x] Graceful clone failure — invalid repo URL → Claude Code handles error → story done with failure description (TestLiveGracefulCloneFailure)

Key finding from initial live testing: Claude Code handles clone failures at the application level (exits 0, writes error description). MarkFailed / osmia-failed label is triggered by K8s-level failures only (watchdog termination, OOM kill, etc.).

Run with: make e2e-live-test (requires make live-up + valid secrets)

End-to-End Tests (Live Backend) — Extended Coverage¶

[ ] PRM with live Claude Code agent — verify scoring and interventions fire at correct thresholds
[ ] Memory across 50+ tasks — verify accumulation, decay, and prompt injection resistance
[ ] Adaptive watchdog across 15+ tasks — verify calibration reduces false positives
[ ] Diagnosis on intentionally failing tasks — correct classification, enriched retry
[ ] Routing across 20+ tasks on 3 engines — verify convergence to optimal engine selection
[ ] Cost estimator — validate predictions against actuals within 2×
[ ] 3-engine tournament on a real GitHub issue — verify judge selects best solution
[ ] All features enabled simultaneously — no conflicts or race conditions

2. Unimplemented — Claimed in Docs¶

These items have config schema and infrastructure code in place but are not wired into the live execution path. They were previously documented as complete or as behaving differently from the actual implementation.

Guardrails.md Prompt Injection¶

Priority: Medium Scope: Small

The promptbuilder package exists but the controller builds engine.Task directly from ticket fields without calling through the promptbuilder. A guardrails.md ConfigMap is not currently mounted or injected.

[ ] Wire promptbuilder.BuildPrompt (or equivalent) into ProcessTicket
[ ] Mount operator-provided guardrails.md ConfigMap into agent pods
[ ] Append content to every agent prompt before BuildExecutionSpec

Task-Profile File Pattern Enforcement¶

Priority: Medium Scope: Medium

TaskProfileConfig (allowed_file_patterns, blocked_file_patterns) is parsed and stored but not enforced. The controller reads AllowedTaskTypes for early rejection but does not translate profile constraints into agent pod configuration.

[ ] Translate blocked_file_patterns into hook environment variables or engine config
[ ] Apply per-profile cost and duration overrides at job creation time

Orphaned TaskRun Recovery¶

Priority: High Scope: Medium Dependencies: None Needed for: Application Build Pilot (long runs are the most exposed to controller restarts)

When a controller restarts mid-job (e.g. due to a pod eviction or ArgoCD rollout), in-memory task run state is lost. If the job's worker pod also failed (e.g. unschedulable, OOMKilled), the task run remains in Running state in the store with a job reference that no longer exists. The controller skips the ticket on every subsequent poll ("task run already exists") and never recovers — requiring manual intervention to reset the Shortcut story state and restart the controller.

Recovery should be automatic:

[ ] On startup, load all Running task runs from the store and reconcile them against live K8s Jobs — transition to Failed any task run whose job no longer exists
[ ] Add a periodic background reconciliation loop (e.g. every 5 minutes) that detects and recovers task runs whose jobs have disappeared
[ ] When a task run transitions to Failed due to a missing job, reset the ticketing backend state (move the story back to the trigger state) so it will be retried automatically

Leader Election (Controller HA)¶

Priority: Low Scope: Medium Dependencies: controller-runtime Lease support (already in the dependency tree)

The controller currently runs as a single replica. Leader election via Kubernetes Lease objects would allow multiple replicas with automatic failover.

[ ] Add --leader-elect flag and Lease RBAC to the Helm chart
[ ] Wire leaderElect option in main.go controller startup

Built-in AWS Secrets Manager Backend¶

Priority: Medium Scope: Medium Dependencies: Secrets plugin interface (complete), IRSA documentation

A native aws-sm backend using the AWS SDK for Go v2 that resolves aws-sm://secret-name#key URIs directly against AWS Secrets Manager. On EKS with IRSA, authentication is automatic (no static credentials). This removes the need for a separate gRPC plugin binary or External Secrets Operator for teams that want direct AWS SM integration.

[x] pkg/plugin/secrets/awssm/ — built-in backend implementation
[ ] IRSA service account annotation in Helm chart (opt-in)
[x] Secret caching with configurable TTL (avoid per-request API calls)
[x] Cross-account assume-role support for multi-tenant deployments
[ ] Integration tests with localstack
[x] Documentation in docs/plugins/secrets.md

Namespace-Per-Tenant Isolation¶

Priority: Low Scope: Large

TenancyConfig and TenantConfig are defined in the config schema but have no runtime effect. Namespace-per-tenant isolation (dedicated RBAC, secrets, quotas per tenant) is not implemented.

[ ] Implement tenant namespace routing in the reconciler
[ ] Per-tenant ticketing and secrets config dispatch
[ ] RBAC generation in the Helm chart

3. High-Priority Upcoming¶

10. Agent Dashboard (Web UI)¶

Priority: High Scope: Large (new service) Dependencies: Real-time streaming (done), TaskRun store (done); item 30 for viewing transcripts after a run ends Needed for: Application Build Pilot (long builds need a way to see where a run is and why it failed without reading controller logs)

A web dashboard for real-time agent observability and control.

Apart from webhook receivers and the optional local development UI, the controller serves only /metrics, /healthz and /readyz. Seeing a run's state, why it failed, or whether a PR is still being watched means reading controller logs and querying the TaskRun store by hand. The local development UI (internal/localui) has a ticket requeue action that may be a useful starting point for the control endpoints.

Approach options: - Grafana-based — Prometheus metrics + Loki logs + custom panels. Lowest effort, read-only. Good for internal ops teams. - Custom UI — Go backend + React/Next.js. Interactive approve/cancel/retry, live streaming via SSE/WebSocket. More effort but purpose-built. - Hybrid — Grafana for metrics + thin custom UI for interactive actions.

Minimum viable features: - [ ] Real-time TaskRun status view (queued, running, succeeded, failed, needs-human) - [ ] Live streaming progress per agent (tool calls, token usage, cost) - [ ] Task run history with filtering by engine, ticket, status, date range - [ ] Cost tracking dashboard (per-task, per-engine, daily/weekly aggregates) - [ ] Manual controls: approve, cancel, retry - [ ] Engine health overview

Investigating failures: - [ ] Per-run detail page: failure reason, engine attempts, Kubernetes Job and pod events, exit code, cost and duration - [ ] Transcript viewer for live and finished runs (finished runs need item 30) - [ ] Link from a run to its ticket, its PR/MR, and any follow-up runs it spawned

Review follow-ups: - [ ] List the PRs/MRs the review poller is watching, with follow-up count and when each was last checked - [ ] "Check now" action: poll one PR/MR for new comments immediately instead of waiting for the next poll interval - [ ] Re-watch action: start watching a PR/MR again after it dropped out (for example past the restore window), from its originating TaskRun

Stuck and orphaned runs: - [ ] Flag TaskRuns in Running whose Job no longer exists, and offer to mark them failed and release the ticket. A manual bridge until Orphaned TaskRun Recovery is automatic

Long-running builds: - [ ] Show the current stage of a multi-stage run and whether it is waiting on a person

Access control: - [ ] The dashboard can start jobs and shows ticket and transcript content, so it needs authentication (OIDC) and at least a read-only and an operator role. It must not be exposed unauthenticated, even on an internal network - [ ] Every control action is audit-logged with who did it

Interactive actions (check now, re-watch, cancel, retry) rule out a Grafana-only dashboard; they need at least the hybrid approach.

If custom UI: - [ ] cmd/osmia-dashboard/ — separate binary or embedded in controller - [ ] /api/v1/ REST endpoints: GET /taskruns, POST /taskruns/:id/approve, GET /taskruns/:id/stream (SSE) - [ ] Frontend: React + Tailwind or equivalent

If Grafana-based: - [ ] Dashboard JSON provisioning in charts/osmia/dashboards/ - [ ] Loki log aggregation for structured slog output - [ ] Alert rules for cost velocity, stalled agents, failed tasks

26. React to Failed CI Pipelines¶

Priority: High Scope: Medium Dependencies: Item 20 (PR/MR Comment Response, complete)

Item 20 made Osmia follow up on the merge requests it opens, but only in response to review comments. It never looks at whether the MR's pipeline passed. When a lint or test stage fails, nothing happens until a human pastes the failing output into a comment containing a trigger word such as "fix". The agent that opened the MR is the one best placed to fix its own red pipeline, and today it is never told.

Found in October 2026: an MR Osmia opened was raised cleanly, CI then failed on the lint stage, and the only way to get Osmia to fix it was a hand-written comment.

Current shape, for reference: internal/reviewpoller polls each tracked MR every poll_interval_minutes, lists its comments, classifies them with the keyword-based RuleBasedClassifier, and batches actionable ones into a FollowUpRequest that pushes to the original branch. scm.PullRequest carries only State (open/closed/merged); no SCM backend exposes pipeline status.

[ ] SCM interface: a pipeline-status method returning the latest pipeline's state and its failed jobs, with protobuf definition to match
[ ] GitLab (/merge_requests/:iid/pipelines, job traces) and GitHub (check runs) implementations
[ ] Poller: on a newly failed pipeline for a tracked MR, emit a follow-up carrying the failing job names and a bounded tail of each job's log
[ ] Count CI-triggered follow-ups against the same max_follow_up_jobs budget as comment-triggered ones, so a pipeline that fails for reasons the agent cannot fix (missing credentials, external access, flaky runners) does not loop
[ ] Do not react to a pipeline that is still running, or one already superseded by a newer commit
[ ] Config switch, off by default, so existing deployments do not start spawning jobs on upgrade

Open questions: - Failures outside the agent's control, like the external-access failure on the same MR, look identical to a lint failure from the pipeline API. Is the follow-up budget enough of a guard, or does it need a way to classify a failure as not fixable before spending a job on it? - Tracked MRs live in memory and are lost on controller restart; this inherits that limitation, and makes it more costly.

Application Build Pilot¶

Items 27 to 30 below, plus items 10 and 26 and Orphaned TaskRun Recovery, are what Osmia needs to run a multi-stage application build: a skill set that takes a written process description, builds the workflow code in stages, tests it against a development environment, and opens a draft MR for a person to review. That work is longer-running, needs more tools, and produces more output than a ticket fix, which is where the gaps below come from.

The pilot is deliberately limited to builds with no data residency requirement. Builds whose data must stay in a given region need a regional deployment of the whole of Osmia, not only regional model routing, because both the agent pod and the controller (via the agent stream and transcripts) handle data pulled from the systems being built against. That is tracked as item 31.

The pilot is deliberately a single build stage. Passing work from one stage to the next is a later piece of work, so it is not part of these criteria.

Pilot is done when: - One build stage runs from trigger to a draft MR with a passing pipeline, using a per-task plugin bundle, with its transcript and outputs kept in object storage - The run survives a controller restart while its Job is still running, and finishes normally - A run whose Job disappears during a controller restart is moved to Failed and its ticket released, rather than left Running (Orphaned TaskRun Recovery) - A failed pipeline on the MR starts a follow-up that fixes it, without a person writing a comment (item 26) - The run can be followed and investigated from the dashboard (item 10) without reading controller logs

From item 10 the pilot needs: the run list with live status, the per-run detail page and transcript viewer, the review follow-up actions, cancel and retry, and access control. Approve is needed only if the pilot uses an approval gate. Live per-agent streaming of tool calls and token usage, the cost views and engine health are deferred.

27. Per-Task Plugin Bundles¶

Priority: High Scope: Medium Dependencies: None

Skills and sub-agents (item 23) are configured once on the engine, so every task gets the same set. Only incident triage overrides them per flow. TaskProfileConfig.MCPServers is parsed but nothing merges it into the workspace MCP config. A build needs a different set of skills, sub-agents, MCP servers and hooks per task, and per stage within a task.

[ ] Define a plugin bundle: skills, sub-agents, MCP servers and hooks packaged together, in the layout Claude Code's --plugin-dir expects
[ ] Select bundles per task, from the ticket (label or field) or the task profile, merged on top of the engine defaults
[ ] Deliver bundles to the pod from a ConfigMap or object storage, not as environment variables, so size is not bounded by the env var limit
[ ] Wire TaskProfileConfig.MCPServers through the same path
[ ] Validate bundle contents at load time and fail the launch on an unknown bundle name, rather than starting the agent without it

28. Build-Tools Engine Image¶

Priority: High Scope: Small Dependencies: None

The Claude Code engine image has git, glab, gh and Claude Code. Build work also needs Python tooling (uv), a headless browser (Playwright and Chromium), ffmpeg, copier, kubectl and a workflow-engine CLI. Adding all of these to the default image would grow it and its attack surface for every task.

[ ] Publish a separate engine-claude-code-build image variant built on the default one, with its own weekly rebuild and Renovate coverage
[ ] Select the image per task profile, so only build tasks pull it
[ ] Confirm Chromium runs under the existing securityContext (non-root, read-only root filesystem, all capabilities dropped), with a writable emptyDir for its profile and cache
[ ] Document that Docker inside the agent pod is not supported; builds test against a development cluster instead

29. Workload Identity for Agent Pods¶

Priority: High Scope: Medium Dependencies: Task-Scoped Secret References (complete)

Build tasks call cloud and deployment APIs. Today the only way to give an agent that access is a long-lived static credential in a secret. Short-lived, narrowly scoped identities are safer and match the workload identity preference in CLAUDE.md.

[ ] Per-task-profile Kubernetes ServiceAccount, so IRSA (AWS) or Workload Identity Federation (GCP) can grant each kind of task its own role
[ ] Projected service account tokens for OIDC exchange with other internal services
[ ] Default agent ServiceAccount keeps no cloud permissions; a task gets access only through a profile that names an identity
[ ] Record which identity each TaskRun ran as, for audit

30. Remote Transcript and Artifact Storage¶

Priority: High Scope: Medium Dependencies: Item 21 (Transcript Storage, complete)

Item 21 added the TranscriptSink interface, but the only sink writes to the controller's local filesystem, and the S3 session store is a stub. A build's transcript and outputs (generated files, test reports, screenshots) are lost when the pod ends or the controller moves.

[ ] S3 TranscriptSink, with bucket and region set in config
[ ] Artifact upload from the agent pod: anything written to an outputs directory is uploaded at the end of the run and listed in result.json
[ ] Implement the S3 session store, so a resumed task can start on a different node
[ ] Define the tenant that storage is partitioned by. TaskRun.TenantID today is the use case, used to scope episodic memory, not a customer; storage needs a customer-level identity (see item 32)
[ ] Object keys scoped by that tenant and TaskRun ID, with access enforced by IAM policy at the storage boundary, not only by key prefix; lifecycle rules for expiry documented in the Helm chart
[ ] Read path for finished runs: the controller looks up a TaskRun's objects by ID and serves them to the dashboard (item 10) through the same authentication and roles as the rest of the dashboard; the agent pod has no read access to other runs' objects
[ ] Use workload identity (item 29) for the controller and pod, not static keys

31. Regional Deployments and Data Residency Routing¶

Priority: High (after the pilot) Scope: Large Dependencies: Items 29 and 30

Some work must keep its data, and the processing of it, inside a given region. Routing only the model calls to a regional endpoint is not enough: the agent pod reads data from the systems it works on, and the controller receives that data through the agent stream, transcripts and episodic memory. The whole of Osmia has to run in the region, so the plan is one independent deployment per region, with no central coordinator. Instances share only the ticketing and SCM systems.

Routing a ticket to the right instance and keeping it out of the wrong one are separate concerns, and need separate mechanisms. A per-ticket tag or owner is convenient for routing but is set by a person and can be forgotten; on its own it is not a safe boundary.

Routing (which instance picks a ticket up): - [ ] Document one ticketing identity per instance (for example a Shortcut user per region) using the existing owner filter, so assignment chooses the instance - [ ] Route webhooks the same way, so every instance can receive all events and ignore those that are not its own

Boundary (which instance may work on a repository): - [ ] Fail closed: an instance refuses any ticket whose repository it cannot show to be in its region, and comments on the ticket saying so, before any job, clone or model call - [ ] Add blocked_repos alongside allowed_repos, so a default-region instance can exclude regional repositories without listing every other one - [ ] Optionally resolve a repository's region from a file in the repository itself (read through the SCM API, without cloning), so the record and the configuration cannot drift apart - [ ] Apply the same check to every path that launches work: new tickets, repository URLs supplied later, review follow-ups and retries

Inside the region: - [ ] Model calls through the region's provider endpoint: wire the existing but unused auth.method: bedrock and bedrock_region settings - [ ] Transcripts, artifacts, session data and the TaskRun store in regional storage - [ ] Keep task content out of anything that leaves the region: central logs, metrics labels and notifications carry identifiers and status only - [ ] Per-instance NetworkPolicy for agent egress, limited to the region's systems and endpoints

Open questions: - Ticket text itself lives in the ticketing system's own region. Tickets for residency-bound work must not contain the data that needs protecting; this is a process rule Osmia cannot enforce. - Should a cross-region dashboard exist, showing status only, or is one dashboard per region enough?

34. Ticket Readiness Check¶

Priority: High Scope: Small to Medium Dependencies: None (uses internal/llm)

Osmia starts an agent on any ticket that passes the guard rails. A vague ticket still gets a full run: the agent guesses at the intent, spends the budget, and either opens a PR nobody wanted or fails. The cost estimator (internal/estimator) scores how big a task looks, not whether it is clear enough to do.

A readiness check runs before launch and asks three questions of the ticket and repository:

Is the scope bounded: one change, in a known place?
Is there enough context: the repository, the affected area, and what is wrong or wanted?
Is success observable: a test, a command, or a stated result a reviewer can check?

The result is a score with a short reason for each question. Above the threshold, the task launches as today. Below it, Osmia holds the ticket and comments on it with what is missing, instead of starting a job.

[ ] Readiness signature in internal/llm, returning a structured score and reasons; one call on a small model per ticket
[ ] Run it in ProcessTicket after guard rails and before cost estimation, for new tickets only (not review follow-ups or retries)
[ ] Config: readiness.enabled (off by default), readiness.threshold, and readiness.mode of advisory (comment and launch anyway) or gate (comment and hold)
[ ] When holding, move the ticket to a "needs info" state or label and re-check it when the ticket is edited
[ ] Record the score and reasons on the TaskRun, so they can be compared with outcomes and the threshold tuned from real data
[ ] Treat the ticket text as untrusted input: the check can only return a score and reasons, and its output never becomes part of the agent's instructions
[ ] Show the score on the run detail page (item 10)

Open questions: - Should the check see repository content (for example, whether tests exist for the affected area), or only the ticket? Repository access makes it more accurate but slower and costlier. - For build stages (Application Build Pilot), is the input a ticket at all, or a stage specification that needs its own checks?

3. Design-First — ADR Required Before Implementation¶

These items have a clear problem and rough direction but need a design document or ADR agreed before writing code.

24. Non-Standard Task Types (Analysis, Reporting, Review)¶

Priority: Medium Scope: Large (controller, prompt builder, execution spec) Dependencies: Task profiles (partially implemented), prompt builder

Tasks like "review open MRs and report which need approval" do not fit the standard clone-fix-push-MR flow. They need read-only execution and a ticket comment + notification as output rather than a merge request.

See Use-Case Abstraction (Design) for a design doc covering these questions, grounded against the current incident-triage pipeline as the second consumer shape. Not yet implemented.

Design questions before implementation:

Execution mode taxonomy: clone_push_mr (today) | read_only (no git clone) | api_read (no workspace, just SCM API access)
Result handler taxonomy: open_mr (today) | comment_and_notify (post summary as ticket comment + notify channels)
Profile dispatch: label-based (osmia:analysis) or story-type-based?
Prompt design: what system prompt makes a read-only analysis task produce a well-structured summary?

Rough sketch (validate design first): - Extend TaskProfileConfig with ExecutionMode and ResultHandler fields - Update BuildExecutionSpec in all engines to skip git clone for read_only mode - Add result_handler dispatch in handleJobComplete - Update prompt builder to inject different system prompt per execution mode

25. Supervisor Agent (or: PRM V2 with Strategic Oversight)¶

Priority: Medium Scope: Medium–Large Dependencies: PRM (internal/prm/), agentstream Forwarder, internal/llm/

The watchdog detects quantitative failure modes and terminates. The PRM scores each tool call and nudges. Neither can reason about whether the agent is pursuing the right approach.

A supervisor adds LLM-based qualitative oversight: "you're correctly implementing a cache but the ticket asked for pagination."

Key design question — standalone package vs PRM V2:

The PRM already has StreamEventProcessor, sliding window, hint file writer, and intervention logic. The cleanest path is probably to extend the PRM Evaluator with an optional LLM scoring backend (PRM V2) rather than a parallel internal/supervisor/ package.

Resolve before implementation: 1. PRM V2 extension vs standalone internal/supervisor/ — pick one 2. Full task description + codebase context, or only recent event window? 3. Can it trigger NeedsHuman, or only write hints? 4. Anti-thrashing: how to avoid over-correcting an on-track agent? 5. What constitutes "severely off-track" vs what PRM escalation already handles?

32. Customer Context and Repository Lookup¶

Priority: High Scope: Medium (Osmia side); the context store itself is outside Osmia Dependencies: Item 27 (plugin bundles), item 31 (regional deployments)

Builds happen in slices: one slice is built, the customer signs it off, then the next slice starts. Fixes come later, after the build is live. Each of these runs needs to know what earlier runs decided, and today each starts with only the ticket text and the repository.

Two kinds of context are involved:

In the repository: the CLAUDE.md, decision records and known issues that each run updates in its own MR. These are reviewed and signed off with the code, so the next run gets them for free. Osmia only needs to make updating them part of the task.
Outside the repository: sign-off notes, what the customer rejected and why, the current process description, call notes and open questions. These belong in a per-customer record kept by another system, not in Osmia. Osmia instances are per region, their storage is not a system of record, and the code is public.

The same record answers two other questions. Given a ticket that names a customer rather than a repository, which repository should the agent work on? And which region does that customer's work belong to (item 31)?

Direction: - [ ] Osmia reads context; it does not store it. Add a context provider plugin interface, in line with the rule that all external integrations go through defined interfaces, which takes a ticket and returns the customer's repositories, region, residency level, the shared libraries it depends on, and links to context documents - [ ] Repository lookup: when a ticket names a customer (by field, label or epic) and has no repository, ask the provider. If the customer has more than one candidate repository and the ticket does not say which, ask a person rather than guess - [ ] Context loading: put the provider's documents in the agent's prompt or workspace at start, inside the region - Episodic memory stays as it is: knowledge about how Osmia does tasks, not a customer record

Questions for the design: 1. Which system holds the customer record, and who keeps it current? Can the tool that creates a customer repository register it there at the same time? 2. How does a ticket name its customer reliably enough to route on? 3. What does a run write back to the record when it finishes (for example a slice's sign-off state), and through what interface? 4. How much context is loaded up front, and how much is left for the agent to fetch through a tool when it needs it?

33. Changes Spanning a Shared Library and a Customer Repository¶

Priority: Medium Scope: Medium Dependencies: Item 32 (to know which libraries a customer depends on)

Every task today works on exactly one repository. A fix reported against one customer's workflow sometimes belongs in a shared library that many customers use. Doing that well takes two linked changes: an MR to the library and a release, then an MR bumping the customer repository to the new version.

A library change carries more risk than a customer change. It affects every consumer, needs different reviewers, and must not carry anything specific to the customer whose ticket started it. For residency-bound customers this matters more, since a shared library is not tied to a region.

Direction: - [ ] Let a task declare extra repositories, cloned read-only by default, so the agent can read the library code it is debugging - [ ] When the agent concludes the fix belongs in a library, it opens a linked library ticket describing the change in general terms, rather than editing the library itself - [ ] Library tickets go through an approval gate before any job starts - [ ] After the library change is released, a follow-up task bumps the dependency in the customer repository, linked to the original ticket

Questions for the design: 1. Who approves a library change raised by an agent, and where? 2. Should the agent ever write to a library directly, or always via a new ticket? 3. How does the customer task wait for the library release: block, or end and be resumed by the release?

4. Infrastructure¶

9. Plugin SDKs (Python, Go, TypeScript)¶

Priority: Medium Scope: Large (separate repositories) Dependencies: Protobuf definitions (complete in proto/)

[ ] Configure buf.gen.yaml for multi-language stub generation
[ ] Python SDK (unitaryai/osmia-plugin-sdk-python) — gRPC stubs, base classes, scaffold/serve/test CLI, example plugins
[ ] Go SDK (unitaryai/osmia-plugin-sdk-go) — gRPC stubs (separate module), hashicorp/go-plugin boilerplate, example plugins
[ ] TypeScript SDK (unitaryai/osmia-plugin-sdk-ts) — gRPC stubs, grpc-js wrapper, example plugins
[ ] Publish SDK documentation to docs/plugins/

11. Documentation Site¶

Priority: High Framework: MkDocs Material

[x] Landing page, getting started, architecture overview, configuration reference
[x] Engine guides, plugin development guide, security model, deployment guides
[x] Search + dark mode
[x] CI/CD pipeline for automatic deployment on merge to main
[x] make docs-serve / make docs-build targets
[ ] API reference (webhook endpoints, protobuf service definitions)
[ ] Changelog and migration guides
[ ] Custom domain (docs.osmia.dev or similar)

5. Completed¶

Everything below is implemented and merged.

Task-Scoped Secret References ✅ (2026-08-27, v0.5.0)¶

Previously listed above as "not wired". The secretresolver package and its config had existed for some time, but nothing called Resolve, so every ticket-level secret declaration was silently ignored.

Implemented:

Ticket secret declarations (osmia:secrets comment block and osmia:secret:ENV=URI labels) parsed and resolved in the shared launch pipeline, so both ticketing and incident triage get them (#66)
Resolved refs injected via SecretKeyRefs; values from non-Kubernetes backends staged in an ephemeral Secret owned by the agent Job, so plaintext never appears in the Job manifest (#66)
Fail-closed policy enforcement, with the raw-ref check applied before alias expansion and scheme/env-name checks after it (#66)
Helm RBAC for the new verbs, an orphan sweep for Secrets stranded by a controller crash, and bounds on declaration size and count (#69)

Structured per-task audit logging, the third item originally listed here, is still not done.

Item 20: PR/MR Comment Response ✅ (2026-03-04)¶

Osmia now monitors open pull/merge requests it creates and spawns follow-up jobs to address actionable review feedback.

Implemented:

pkg/plugin/scm — ReviewComment type; ListReviewComments, ReplyToComment, ResolveThread added to the Backend interface
pkg/plugin/scm/github — GitHub REST implementation of the three new methods (ListReviewComments merges review + issue comment endpoints; ReplyToComment attempts review reply then falls back to issue comment; ResolveThread is a no-op since GitHub REST does not support thread resolution)
pkg/plugin/scm/gitlab — GitLab REST implementation (notes endpoint; discussion-aware reply; PUT resolved: true for thread resolution)
internal/reviewpoller — new package:
types.go — Classification, ClassifiedComment, TrackedPR, FollowUpRequest
classifier.go — RuleBasedClassifier (keyword + bot author heuristics) and LLMClassifier (ChainOfThought with rule-based fallback)
poller.go — background Poller with Register, DrainFollowUps, Start
internal/config — ReviewResponseConfig struct + validation
internal/taskrun — ParentTicketID, ReviewCommentID, ReviewThreadID, ReviewPRURL fields on TaskRun
internal/controller — reviewPoller field, WithReviewPoller option, handleFollowUpComplete, processFollowUpTask, scmFor helper; drain in reconcileOnce; register in handleJobComplete
cmd/osmia/main.go — review response subsystem wiring
tests/integration/review_response_test.go — 9 integration tests

Configuration (osmia-config.yaml):

review_response:
  enabled: true
  poll_interval_minutes: 5
  min_severity: warning        # info | warning | error
  max_follow_up_jobs: 3        # per PR
  reply_to_comments: true
  resolve_threads: false       # GitLab only; no-op on GitHub REST
  llm_classifier: false        # set true to enable LLM-backed classification

Active Integration — Phase 5 ✅¶

E2E workflow pipeline tests with fake-agent binary:

Test	Validates	Status
`TestWorkflowHappyPath`	Ticket → K8s Job → NDJSON stream → `StateSucceeded` + result	✅
`TestWorkflowJobFailure`	Non-zero exit → retry exhaustion → `StateFailed` + `MarkFailed`	✅
`TestWorkflowEngineChainFallback`	Primary engine fails → fallback engine completes → `StateSucceeded`	✅
`TestWorkflowPRMHintDelivery`	Looping agent triggers PRM nudge intervention and logs	✅
`TestWorkflowWatchdogTermination`	Cost-thrashing agent terminated by watchdog → `StateFailed`	✅
`TestWorkflowSequentialTasksMemory`	Memory extracted after task 1; injected into task 2 prompt	✅
`TestWorkflowTournamentEndToEnd`	2 candidates + 1 judge → judge selects winner → `MarkComplete`	✅

Infrastructure added: - hack/fake-agent/ — standalone Go module; Dockerfile (scratch, UID 10000) - make fake-agent-image / make fake-agent-load / make e2e-workflow-test / make e2e-workflow-test-verbose

Test suite hardening (post-merge fixes): - Job list queries filtered by LabelTaskRunID label to prevent stale-job count failures across runs - workflowFakeEngine.BuildExecutionSpec now calls BuildPrompt (mirrors real engine pattern) so MemoryContext is correctly captured in TestWorkflowSequentialTasksMemory - make e2e-workflow-test switched to quiet mode (failures only) + -count=1 to prevent cached results

Active Integration — Phase 4 ✅¶

SQLite persistence, security hardening, LLM V2 upgrades, and integration tests:

Subsystem	Status
SQLite persistence — routing (`SQLiteFingerprintStore`)	✅ WAL mode, upserts, persistence tests
SQLite persistence — estimator (`SQLiteEstimatorStore`)	✅ kNN similarity in Go, persistence tests
SQLite persistence — watchdog (`SQLiteProfileStore`)	✅ composite PK `(repo_pattern, engine, task_type)`
Memory graph tenant isolation	✅ `ListNodes`/`DeleteNode` tenant params; cross-tenant `SaveEdge` rejected; adversarial tests
Diagnosis prompt injection defence	✅ `sanitiseForPrompt`; XML delimiters in retry builder
Tournament judge prompt injection defence	✅ `CANDIDATE-DIFF-BEGIN/END` comment markers
Config validation	✅ `validate.go`; `Config.Validate()` called in `Load()`
Rate-limited LLM client	✅ `RateLimitedClient` with configurable RPS
PRM LLM scorer V2	✅ `LLMScorer` with `ChainOfThought` + V1 fallback
Memory LLM extractor V2	✅ `LLMExtractor` merging LLM + V1 results
Diagnosis LLM analyser V2	✅ `LLMAnalyser` with failure-mode validation + V1 fallback
Integration tests — 8 subsystem scenarios	✅ `tests/integration/subsystems_test.go`
Integration tests — all-features smoke test	✅ `tests/integration/all_features_test.go`

Active Integration — Phase 3 ✅¶

Tournament coordinator, PRM hint file writer:

Subsystem	Status
Tournament coordinator wiring (item 18)	✅ `launchTournament` / `handleCandidateComplete` / `launchJudge` / `handleJudgeComplete`
PRM hint file writer	✅ `writeHintFile` via K8s exec; `cleanupHintFile`; `validateHintPath`

Near-Term Items — All Complete ✅¶

#	Feature	Notes
21	Transcript Storage & Audit Log	`TranscriptSink` interface; local filesystem sink; wired into controller agentstream
22	Multi-SCM Backend Routing	`internal/scmrouter` package; host-pattern routing; backward-compatible config
23	Skills, Subagents & Per-Task MCP Plugins	`Skill` + `SkillEnvVars`; base64 env var delivery via `setup-claude.sh`

Active Integration — Phase 2 ✅¶

Diagnosis, calibration, routing, estimator, SCM router, and transcript all wired into the live controller and main.go:

Subsystem	Status
Causal diagnosis (item 14)	✅ Wired — `WithDiagnosis`; enriched retry prompts in `handleJobFailed`
Adaptive watchdog calibration (item 15)	✅ Wired — `WithWatchdog` + `WithWatchdogCalibration`; `ConsumeStreamEvent` in stream reader
Intelligent routing (item 16)	✅ Wired — `WithIntelligentSelector`; `RecordOutcome` after every terminal run
Predictive cost estimation (item 17)	✅ Wired — `WithEstimator`; auto-reject in `ProcessTicket`; `RecordOutcome` after terminal run
Multi-SCM routing (item 22)	✅ Wired — `WithSCMRouter`; `cfg.SCM.Backends` array in `main.go`
Transcript storage (item 21)	✅ Wired — `WithTranscriptSink`; `Append` in stream reader; `Flush` on completion
Bug fix: `launchRetryJob`	✅ Same-engine retries were transitioning to `StateRetrying` with no new job created

Active Integration — Phase 1 ✅¶

Subsystem	Notes
Phase 1 agent log filtering	`LoggingEventProcessor` in `agentstream/`
Multi-workflow Shortcut (item 4 from post-testing plan)	`workflows:` array; per-story state resolution
Code review config (item 7)	`code_review.enabled` guard in controller
PRM — Real-Time Agent Coaching (item 12)	Integrated; V2 (LLM scoring) pending
Episodic Memory (item 13)	Integrated; V2 (LLM extraction) pending

Original 13-Item Improvements Plan — 11/13 complete¶

#	Feature	Status
1	Webhook Receiver & Event-Driven Ingestion	✅
2	Agent Sandbox Integration (gVisor / Warm Pools)	✅
3	OpenCode Execution Engine	✅
4	Cline CLI Execution Engine	✅
5	Shortcut.com Ticketing Backend	✅
6	Telegram Notification Channel	✅
7	Linear Ticketing Backend	✅
8	Discord Notification Channel	✅
9	HashiCorp Vault Secrets Backend	✅
10	Task-Scoped Secret Resolution	✅ Complete in v0.5.0 — see Task-Scoped Secret References under Completed
11	NetworkPolicy & Security Hardening	✅ (opt-in; disabled by default)
12	Plugin SDKs (Python, Go, TypeScript)	🚧 Helper libraries checked in; generated stubs require `make sdk-gen`
13	Local Development Mode (Docker Compose)	✅

Summary Table¶

#	Feature	Priority	Status
10	Agent Dashboard	High	Not started
26	React to Failed CI Pipelines	High	Not started
27	Per-Task Plugin Bundles	High	Not started
28	Build-Tools Engine Image	High	Not started
29	Workload Identity for Agent Pods	High	Not started
30	Remote Transcript and Artifact Storage	High	Not started
31	Regional Deployments and Data Residency Routing	High	Not started
34	Ticket Readiness Check	High	Not started
24	Non-Standard Task Types	Medium	Design doc available
25	Supervisor Agent / PRM V2	Medium	Design doc required
32	Customer Context and Repository Lookup	High	Design doc required
33	Changes Spanning a Shared Library and a Customer Repository	Medium	Design doc required
9	Plugin SDKs	Medium	🚧 In progress
11	Documentation Site	High	Partially complete
—	E2E live-backend extended coverage (50+ tasks)	High	In progress
—	Guardrails.md Prompt Injection	Medium	🚧 Not wired
—	Task-Profile File Pattern Enforcement	Medium	🚧 Not wired
—	Orphaned TaskRun Recovery	High	🚧 Not started
—	Leader Election (Controller HA)	Low	🚧 Not started
—	Built-in AWS Secrets Manager Backend	Medium	🚧 Core complete; IRSA Helm annotation + localstack tests remaining
—	Namespace-Per-Tenant Isolation	Low	🚧 Not started
20	PR/MR Comment Response	High	✅ Complete
—	Task-Scoped Secret References (per-task)	Medium	✅ Complete (v0.5.0); per-task audit logging remaining
—	Cost & token usage in notifications	Low	✅ Complete
—	E2E workflow suite (fake-agent, 7 tests)	High	✅ Complete
—	E2E live-backend validation (initial 2 tests)	High	✅ Complete
—	Tournament coordinator wiring	High	✅ Complete
—	PRM hint file writer	High	✅ Complete
—	SQLite persistence (routing, estimator, calibrator)	Medium	✅ Complete
—	LLM V2 upgrades (PRM, memory, diagnosis, judge)	Medium	✅ Complete
—	Security hardening	High	✅ Complete
21	Transcript Storage & Audit Log	High	✅ Complete
22	Multi-SCM Backend Routing	High	✅ Complete
23	Skills, Subagents & Per-Task MCP Plugins	Medium	✅ Complete
14	Causal Diagnosis (Self-Healing Retry)	High	✅ Complete
15	Adaptive Watchdog Calibration	High	✅ Complete
16	Engine Fingerprinting + Routing	Medium	✅ Complete
17	Predictive Cost Estimation	Medium	✅ Complete
18	Competitive Execution (Tournament)	Medium	✅ Complete
19	Shortcut webhook state filtering	Low	✅ Complete