Runs

How a run executes, how to read its result, and why a Test case fails.

A run is one execution of a Test against a target. It delivers the Test's events, waits for your SIEM to report on them, and records a verdict for each Test case.

Starting a run

  • On a schedule — a Test suite starts a run for each of its Tests.
  • Run now — on a Test's page, for a single Test.
  • Fire a Test suite — from the target's home page, under New run.
  • From an AI agent — the MCP server can start a run of content that is not yet a Test, which is how a new detection's content is validated before it is enrolled.

Starting runs needs the editor role or above. A run cannot be cancelled once started.

What a run does

  1. Generate — the engine renders the Test's events.
  2. Deliver — the events are sent through the Test's pipeline.
  3. Await validation — the SIEM connection looks for the delivered events in the SIEM, and for alerts from the Detection on those events.
  4. Finalize — a run with a failed Test case is final at once. Otherwise the run stays open until the target's validation window and finalization grace have passed, and each Test case still undecided is settled then: an expected alert that never arrived fails, and a no-alert expectation that saw no alert passes.

A run is shown as pending while it is in flight, with its current phase, and then as passed or failed. A run whose Test cases have all passed shows passed together with the time it finalizes. The verdict is already known; the run stays open so that alerts arriving later in the window are still recorded against it.

A run that fails before validation says where: it never started, events could not be generated, the SIEM connection did not pick the run up, or event delivery failed.

The Runs page

Runs in the sidebar lists the workspace's runs; each target has its own Runs page. Filter by status (In flight, Passed, Failed), by source (Test suite, Run now, or content submitted directly), by target, and by time.

Reading a run

The run page shows the run's Test cases, event counts, target, who or what triggered it, and its duration. The Test cases table gives each case's scenario, status, and events.

Other rules that fired on these events lists alerts from rules other than the Test's Detection. They are informational: they neither pass nor fail the Test.

Why a Test case failed

ReasonMeaning
delivery errorThe events could not be delivered through the pipeline.
events not observedThe events were delivered but were not found in the SIEM. The detection never had a chance to fire; check the pipeline, index, and sourcetype.
no qualifying alert observedThe events were indexed, an alert was expected, and the Detection did not alert on them.
unexpected alertNo alert was expected, and the Detection alerted.
detection did not runThe Detection's scheduled search was skipped or errored during the window.
alert not attributedThe Detection fired, but the alert could not be tied to this Test case's events. See Unclassifiable SPL.
deadline exceededThe SIEM connection did not report a result before the validation window closed.

Investigating a failure

Investigate with your AI on a failed run opens a prompt for an AI agent connected to the Tracemill MCP server. The agent walks the path from Tracemill to your SIEM with your own credentials and attaches its findings to the run. See Investigate a failed run.

The validation window

How long a run waits is set on the target: the Validation window and Finalization grace, under Settings › Configuration. See Targets.