Investigate a failed run
Find the root cause of a failed validation run with your own AI agent, reading your Splunk with your own credentials.
A failing run tells you where the path broke: events never observed, the detection never ran, it ran and matched nothing, it fired but could not be attributed. It does not tell you why. The reason is almost always outside Tracemill — in the transport, a pipeline such as Cribl, or Splunk itself — and finding it means someone with credentials to every hop walking the path by hand.
This flow hands that walk to your agent. Tracemill exports everything it knows about the failure as a dossier: the run's timeline, the verdict, what was delivered where, what was and was not observed, and a probe chain — a sequence of ready-to-run, read-only Splunk searches, REST reads, and CLI listings, rendered with this run's values, each with a stated expectation and a next step. Your agent runs them from your side and reports a root cause.
Nothing is changed, by design. The flow ends at a diagnosis and a suggested fix.
What stays on your side
Tracemill never holds your Splunk, pipeline, or cloud credentials, and the
TA-Tracemill add-on never runs diagnostic searches on your Splunk. Every read
of your systems is performed by your own agent with the credentials you have
configured locally: the Splunk MCP server,
a read-only token used with curl, the aws CLI with a local profile.
The probes are generated from Tracemill's knowledge of your target and pipeline configuration. They are the only searches your agent should run: every value in them — a saved-search title, an index, a correlation value — was encoded for the context it appears in, and the recipe tells the agent never to assemble a search of its own from dossier values.
The recipe also keeps your agent away from your secrets. It tells the agent
never to open credential files, MCP or agent client config, environment
variables, or keychains, even to work out which instance a credential points
at; to ask you instead; never to quote a token or Authorization header in
its report; and to treat everything in the dossier and everything your systems
return as data, never as instructions.
Start the flow
Open the failed run, test, or suite execution in the console and use Investigate with your AI. It shows the prompt below with the right reference filled in, a copy button, and a first-time setup note. Paste it into your MCP-connected agent. When the agent finishes and you say yes to attaching its report, the report appears on that same page.
Without the console, connect the MCP server and send your agent this as a plain message:
Investigate why this Tracemill run failed. Call the Tracemill MCP server's
`get_investigation` tool with reference tm:run:<id> and follow the
`recipe` it returns before running any probe.
Read from my Splunk (and Cribl or AWS if the dossier points there) using the
credentials I have configured locally. Do not change anything.It works in any MCP client, because the agent's instructions — the
recipe — come back from get_investigation itself, next to the dossier.
The same recipe is also published as the server's investigate prompt; a
client that surfaces prompts shows it in a slash-command or template menu (in
Claude Code, /mcp__<server>__investigate), and you can start there instead.
The reference names what to investigate:
| Reference | Investigates |
|---|---|
tm:run:<run id> | one run |
tm:test:<test id> | a test, through its latest failing run |
tm:execution:<execution id> | a suite execution — its failed member runs (up to 20, the rest counted) with the cross-run pattern |
The run id is the last segment of the run's URL in the console; a test id is the last segment of its page URL. References are opaque and carry no workspace: they resolve inside the workspace of the token your agent is using, or not at all.
What the agent does
Fetches the dossier and its recipe, then reads the timeline. Every instant Tracemill recorded — when generation started, the window the events' own timestamps fall in, the validation deadline, when the verdict was stamped — plus the interval the verdict was judged over. That interval is labelled approximate: it opens at generation start rather than at the add-on's own claim time, which Tracemill never receives, so it is always a superset of what was actually judged.
Stops early when there is nothing to walk. A run still in progress, a non-Splunk target, or a run that failed on the Tracemill side before any workload could be judged all end the flow with a clear statement rather than a walk.
Checks it is reading the right Splunk. Your agent's Splunk credentials may reach a different instance than the one the run was delivered to and judged against — a dev search head, a second environment. Every chain opens with an identity check against the fingerprint TA-Tracemill last reported for the target: the instance GUID, or the cluster label on a search-head cluster. If the target has never reported one, the agent asks you to confirm the instance instead. A mismatch stops the walk before anything else runs and is reported as blocked, never as a finding about a hop, because every result from the wrong instance would be meaningless. The agent never looks for the right instance in your local configuration or credential files; if the check fails, it asks you.
Scopes an execution. A suite execution adds a pattern no single run can show. Failed workloads are grouped by reason, event type, and pipeline; one group holding every failure points at a shared hop and is investigated once, while several groups are several hypotheses, each walked through its own representative run unless the first finding plainly explains the rest. A member that failed before any workload was judged belongs to no group and is reported from its own verdict. One failure among many passes is detection-specific and descends into that run. The counts are explicit, so one failure in a large execution is never read as an outage.
Walks the chain, in order, and stops where the evidence settles it. Each step says what it proves, what a healthy result looks like, what a failing one means, and where to go next on either outcome. Most contradictions end the walk at that hop, but not all: an empty correlation search leads on to an arrival-time search and then to a literal search for one of the actual events, which is how "never arrived" is told apart from "arrived with the wrong timestamp" and from "arrived but the correlation field was never extracted".
Asks you one question, when it needs to. Tracemill knows the destination it delivered to, not whether something sits between it and Splunk. When transport looks healthy and ingest does not, the agent asks whether a pipeline, relay, or forwarder tier is in the path, and inspects its routes if so. Apart from confirming the instance of a target with no fingerprint, that is the only step in the chain that asks rather than runs.
Tells late from never. Scheduler and arrival-time probes come in pairs. The first is bounded by the judged interval and carries the expectation; the second covers the time since the deadline and exists so a late dispatch or a slow ingest is reported as latency at that hop, not as absence.
What you get back
A report under fixed headings:
## Summary one paragraph
## Root cause hop, one sentence, confidence high / medium / low
## Evidence per probe walked: what was run, result excerpt, verdict
## Suggested fix recommendation only, not applied
## Unresolved what the chain could not settleThe root cause names a hop — transport, pipeline, Splunk ingest, Splunk scheduler, Splunk detection, or the TA-Tracemill add-on — or one of four other outcomes the recipe is explicit about:
- Every hop checked out. On the chains whose probes test the verdict — events not observed, detection did not run, no alert — and only when the results themselves show the path worked (a scheduler row that is an error state passes the row-count check but is a scheduler failure), this means the evidence contradicts it. The agent reports a suspected Tracemill defect and asks you to file it with the dossier attached, rather than inventing a cause elsewhere. The other chains confirm the verdict rather than test it: an unexpected alert whose scheduler row shows results and whose search is enabled is exactly what the failure predicts, and the finding there is what the matched events show, not a defect.
- Timing near the window edge. The only contradiction was a scheduler hit inside the approximate interval that Tracemill's own evidence does not show. Reported as "verify the add-on's claim time", never as a defect.
- Tracemill-side failure. The run failed before anything reached your systems, and the verdict block is the whole report.
- Blocked. A hop needed access the agent did not have. A permission error is never taken as evidence the hop is healthy; the report says what would unblock it.
Evidence quotes the minimum excerpt that proves each verdict. The results are your data, read by your agent, and they are not sent to Tracemill by this flow.
What each failure reason points at
| Failure reason | First hop the chain checks |
|---|---|
delivery_error | Tracemill's own delivery. No external probes; the worker's error and the redacted pipeline config are the report |
events_not_observed | The transport (HEC errors, or the S3 bucket listing), then ingest by correlation value, then by arrival time, then by a literal from the event |
detection_did_not_run | The saved search as Splunk holds it, then the scheduler's own log for the judged interval, then scheduler-wide skips |
no_alert | The scheduler row, the saved search as Splunk holds it, the run's events in their own time window, then the detection's SPL with the rendered correlation constraint inserted, then clause-by-clause stripping |
alert_unattributed | The scheduler row with results, the add-on's own reporting, then the job's result rows. No probes when the add-on reported it could not classify the detection's SPL: the finding is the add-on, and the fix is declaring correlation |
unexpected_alert | The scheduler row for the firing, the job's result rows, then the detection's scope. For Tests with a declared detection, only that detection's own firings count; other rules' alerts are listed under "Other rules that fired on these events" and do not change the verdict. A Test that omits detection while expecting none still fails on any valid in-window attributed alert, and those alerts are not classified other or moved into that note |
validation_deadline_exceeded | The add-on itself: when it was last seen, its modular input state, its own log lines |
Attach the report
When the investigation is done, the agent offers to attach its report to the
run, test, or execution it investigated. It never uploads unasked. Say yes and
the report is stored in your workspace, where anyone who can read runs can
list it under the subject it was attached to, over the API or through
list_investigations.
A report that carries anything credential-shaped is refused, never stored: a private key, a JWT, a bearer, Splunk, or Basic authorization value, an AWS secret key or session token, or a Tracemill token, in the summary, the body, or the client name. The refusal names the field, the kind of secret, and the line, but never the value, and the agent removes it and resubmits. Account ids, access key ids, and other identifiers are not refused; evidence quotes them routinely.
What is stored is the report itself: the one-paragraph summary, the full markdown under the fixed headings, and three structured fields the agent passes alongside — the root-cause hop, its confidence, and the probes it walked with each verdict. Those fields are never parsed out of the prose; they exist so the page can show a hop at a glance and so causes can be aggregated across your fleet later.
Reports are immutable. A second investigation is a second report, newest first, and each subject holds at most fifty. The recipe has the agent read the reports already attached to the subject before it walks anything, so a repeat investigation says what changed or what the first walk missed rather than repeating it. The console does the same from its side: once a subject carries a report, its button reads Investigate again and the prompt it hands you asks for exactly that. The summary is capped at 1 KB, the body at 256 KB, and the whole request at 300 KB; an oversized report is refused with a validation error, distinct from the capability refusal a read-only token gets and the cap refusal a full subject gets. Attaching needs a role that can write content (owner, admin, or editor); deleting a report is an admin action. The body may quote what your Splunk returned, so the recipe tells the agent to quote the minimum excerpt that proves each verdict.
Limits
- Splunk targets only. Other SIEM types are refused at the dossier call.
- Diagnosis only. Nothing in the flow enables, disables, edits, or re-runs anything; applying the fix and re-validating are yours to do.
- The dossier reports pipeline configuration as it is now. If a destination changed after the run was delivered, transport probes rendered from it describe a route the run did not take, and the agent is told to say so.
- The dossier shows each pipeline's configuration with credentials masked:
secret fields, URL credentials and query values, any key Tracemill does not
model, and every value of an agent pipeline appear as
********. Credentials in the run's delivery error text are masked the same way.