Investigate a failed run

A failing run tells you where the path broke: events never observed, the detection never ran, it ran and matched nothing, it fired but could not be attributed. It does not tell you why. The reason is almost always outside Tracemill — in the transport, a pipeline such as Cribl, or Splunk itself — and finding it means someone with credentials to every hop walking the path by hand.

This flow hands that walk to your agent. Tracemill exports everything it knows about the failure as a dossier: the run's timeline, the verdict, what was delivered where, what was and was not observed, and a probe chain — a sequence of ready-to-run, read-only Splunk searches, REST reads, and CLI listings, rendered with this run's values, each with a stated expectation and a next step. Your agent runs them from your side and reports a root cause.

Nothing is changed, by design. The flow ends at a diagnosis and a suggested fix.

What stays on your side

Tracemill never holds your Splunk, pipeline, or cloud credentials, and the TA-Tracemill add-on never runs diagnostic searches on your Splunk. Every read of your systems is performed by your own agent with the credentials you have configured locally: the Splunk MCP server, a read-only token used with curl, the aws CLI with a local profile.

The probes are generated from Tracemill's knowledge of your target and pipeline configuration. They are the only searches your agent should run: every value in them — a saved-search title, an index, a correlation value — was encoded for the context it appears in, and the prompt tells the agent never to assemble a search of its own from dossier values.

Start the flow

Connect the MCP server, then ask your agent for the investigate prompt with a reference to the failure. Most clients show prompts in a slash-command or template menu; in any client, this works as a plain message:

Investigate why this Tracemill run failed. Use the Tracemill MCP server's
`investigate` prompt with reference tm:run:<id>.
Read from my Splunk (and Cribl or AWS if the dossier points there) using the
credentials I have configured locally. Do not change anything.

The reference names what to investigate:

The run id is the last segment of the run's URL in the console; a test id is the last segment of its page URL. References are opaque and carry no workspace: they resolve inside the workspace of the token your agent is using, or not at all.

What the agent does

Fetches the dossier and reads the timeline. Every instant Tracemill recorded — when generation started, the window the events' own timestamps fall in, the validation deadline, when the verdict was stamped — plus the interval the verdict was judged over. That interval is labelled approximate: it opens at generation start rather than at the add-on's own claim time, which Tracemill never receives, so it is always a superset of what was actually judged.

Stops early when there is nothing to walk. A run still in progress, a non-Splunk target, or a run that failed on the Tracemill side before any workload could be judged all end the flow with a clear statement rather than a walk.

Scopes an execution. A suite execution adds a pattern no single run can show. Failed workloads are grouped by reason, event type, and pipeline; one group holding every failure points at a shared hop and is investigated once, while several groups are several hypotheses, each walked through its own representative run unless the first finding plainly explains the rest. A member that failed before any workload was judged belongs to no group and is reported from its own verdict. One failure among many passes is detection-specific and descends into that run. The counts are explicit, so one failure in a large execution is never read as an outage.

Walks the chain, in order, and stops where the evidence settles it. Each step says what it proves, what a healthy result looks like, what a failing one means, and where to go next on either outcome. Most contradictions end the walk at that hop, but not all: an empty correlation search leads on to an arrival-time search and then to a literal search for one of the actual events, which is how "never arrived" is told apart from "arrived with the wrong timestamp" and from "arrived but the correlation field was never extracted".

Asks you one question, when it needs to. Tracemill knows the destination it delivered to, not whether something sits between it and Splunk. When transport looks healthy and ingest does not, the agent asks whether a pipeline, relay, or forwarder tier is in the path, and inspects its routes if so. That is the only step in the chain that asks rather than runs.

Tells late from never. Scheduler and arrival-time probes come in pairs. The first is bounded by the judged interval and carries the expectation; the second covers the time since the deadline and exists so a late dispatch or a slow ingest is reported as latency at that hop, not as absence.

What you get back

A report under fixed headings:

## Summary        one paragraph
## Root cause     hop, one sentence, confidence high / medium / low
## Evidence       per probe walked: what was run, result excerpt, verdict
## Suggested fix  recommendation only, not applied
## Unresolved     what the chain could not settle

The root cause names a hop — transport, pipeline, Splunk ingest, Splunk scheduler, Splunk detection, or the TA-Tracemill add-on — or one of four other outcomes the prompt is explicit about:

• Every hop checked out. On the chains whose probes test the verdict — events not observed, detection did not run, no alert — and only when the results themselves show the path worked (a scheduler row that is an error state passes the row-count check but is a scheduler failure), this means the evidence contradicts it. The agent reports a suspected Tracemill defect and asks you to file it with the dossier attached, rather than inventing a cause elsewhere. The other chains confirm the verdict rather than test it: an unexpected alert whose scheduler row shows results and whose search is enabled is exactly what the failure predicts, and the finding there is what the matched events show, not a defect.
• Timing near the window edge. The only contradiction was a scheduler hit inside the approximate interval that Tracemill's own evidence does not show. Reported as "verify the add-on's claim time", never as a defect.
• Tracemill-side failure. The run failed before anything reached your systems, and the verdict block is the whole report.
• Blocked. A hop needed access the agent did not have. A permission error is never taken as evidence the hop is healthy; the report says what would unblock it.

Evidence quotes the minimum excerpt that proves each verdict. The results are your data, read by your agent, and they are not sent to Tracemill by this flow.

What each failure reason points at

Attach the report

When the investigation is done, the agent offers to attach its report to the run, test, or execution it investigated. It never uploads unasked. Say yes and the report is stored in your workspace, where anyone who can read runs can list it under the subject it was attached to, over the API or through list_investigations.

What is stored is the report itself: the one-paragraph summary, the full markdown under the fixed headings, and three structured fields the agent passes alongside — the root-cause hop, its confidence, and the probes it walked with each verdict. Those fields are never parsed out of the prose; they exist so the page can show a hop at a glance and so causes can be aggregated across your fleet later.

Reports are immutable. A second investigation is a second report, newest first, and each subject holds at most fifty. The summary is capped at 1 KB, the body at 256 KB, and the whole request at 300 KB; an oversized report is refused with a validation error, distinct from the capability refusal a read-only token gets and the cap refusal a full subject gets. Attaching needs a role that can write content (owner, admin, or editor); deleting a report is an admin action. The body may quote what your Splunk returned, so the prompt tells the agent to quote the minimum excerpt that proves each verdict.

Limits

• Splunk targets only. Other SIEM types are refused at the dossier call.
• Diagnosis only. Nothing in the flow enables, disables, edits, or re-runs anything; applying the fix and re-validating are yours to do.
• The dossier reports pipeline configuration as it is now. If a destination changed after the run was delivered, transport probes rendered from it describe a route the run did not take, and the agent is told to say so.