Job Schema

Complete field reference for job configuration files. For conceptual overview, see job concepts.

IDE IntegrationCopy Anchor Link

Add this directive to the top of your YAML file for autocomplete and validation:

# yaml-language-server: $schema=https://tracemill.dev/schemas/job.schema.json

PropertiesCopy Anchor Link

stateCopy Anchor Link

A map of variable names to ExprStr values, evaluated once before workloads start.

Unlike scenario state, pool.* expressions are allowed here. Job state is available to workload bindings via ref.*, but does not flow to scenarios implicitly — use explicit bindings (e.g. account_id: ref.account_id).

Object/array state: values are not opaque blobs: string leaves nested at any depth evaluate as ExprStr once at job start, matching top-level state semantics. pool.* is allowed at any depth (not just at the top level), and nested ref.* leaves participate in the same cycle / undefined-ref checks as top-level refs.

state:
  account_id: "000000000000"
  region: us-east-1
  api_endpoint: "https://api.example.com:8443"

State values may be overridden at invocation via the CLI's --set key=value flag (repeatable). Override keys must be declared in this state block — unknown keys are rejected before any workload runs. Override values are YAML-decoded into the same type set as the YAML state block, and a literal override must have the same type as the declared literal (an integer satisfies a declared number; the CLI keeps --set text as a string for a key declared as a string). Overrides are type-checked, never converted. String overrides follow the ExprStr grammar (so --set region=gen.aws_region() resolves at runtime). Nested string leaves inside an override map or list value follow the same ExprStr rules as declared nested job state.

poolsCopy Anchor Link

An array of pool entries. Each entry is either an inline pool definition or a reference to an external pool file. See pool schema for the full pool configuration.

Inline poolCopy Anchor Link

Defines the pool directly in the job file. The type field is optional for inline pools.

pools:
  - id: threat-ips
    sampling:
      mode: random
      seed: 42
    ip_range:
      cidrs:
        - 198.51.100.0/24

External pool referenceCopy Anchor Link

Points to a standalone pool file by path or content ID. The job entry's id is used regardless of what id the file contains.

pools:
  - id: users
    path: ./pools/users.yaml

workloadCopy Anchor Link

Each workload runs a scenario with optional concurrency, looping, and bindings.

ExampleCopy Anchor Link

Reference counts use the merged job state, including overrides. Quoted numbers and floats such as "10" or 10.0 are rejected; use the integer 10. Zero still runs once—it does not disable a workload. The count applies to each concurrent instance, and bindings are evaluated afresh on every iteration. There is no additional workload-count cap beyond representable integers and normal cancellation; scenario foreach limits are separate.

Run-workload mutes are scoped to the reference key and its effective state expression, including overrides. Retuning attack_count: 3 to 5 changes that identity, so a mute for the old value does not match. An unchanged generator expression keeps the same identity across different draws. Literal loop counts remain outside this mute identity.

state:
  attack_count: 10

workloads:
  - scenario: ./scenarios/brute-force.yaml
    concurrency: 2
    bindings:
      source_ip: pool.threat-ips
      username: pool.target-usernames
      account_id: ref.account_id

  - scenario: ./scenarios/api-calls.yaml
    loop: ref.attack_count
    eps: 50
    start_after: 5s
    fields:
      environment: production

expectationCopy Anchor Link

Declares what a workload asserts about alert behavior. A workload without an expectation validates on delivery alone.

workloads:
  - scenario: scenarios/aws/iam/delete-policy
    expectation:
      expected: alert
      summary: "IAM DeletePolicy removes a managed policy"

Unclassifiable SPLCopy Anchor Link

Without correlation, the Splunk add-on attributes a firing to a workload by tracing the detection's result rows back to the events that produced them. It has to classify the detection's SPL to do that. When it cannot, for example a tstats search it cannot rewrite or a pipeline that overwrites its group-by fields, it skips attribution for that firing.

When the workload fails with alert not attributed, Tracemill Cloud explains that the detection fired but the add-on never attributed the firing to this workload. Passing workloads carry no such note, and alerts from other rules never pass a workload.

The investigation for such a workload stops at that finding rather than probing the SIEM. The verdict itself does not change.

To fix it, declare correlation with the result fields the detection outputs, so the add-on attributes by matching result rows instead:

workloads:
  - scenario: scenarios/azure/audit/enable-account-and-reset-password
    expectation:
      expected: alert
      correlation:
        user: ref.attack_target_upn
        initiatedBy: ref.admin_upn

detectionCopy Anchor Link

Optionally records which Detection a Splunk job validates, linking the Test to its detective control. It appears once at the job root and applies to every workload with an expectation. A job declares at most one Detection; workloads cannot select their own.

Resolution. Every workload that has an expectation inherits the optional job-level detection. A Test may omit detection while retaining expected: alert | none; that is not observation-only. In this form, any valid in-window alert attributed to the Test case satisfies alert and violates none. The manifest carries no detection_rule_name, so there is no scheduler did-not-run watch, and it carries no native_key, so pass-crediting evidence is not gated by Detection identity. If no alert arrives, alert resolves to no_alert at the deadline, subject to event-observation failure precedence.

Other rules that firedCopy Anchor Link

With a declared detection, the verdict is about that Detection only. Each alert attributed to the workload is classified as declared or other:

• Splunk: the first tier that applies decides, with no fallback to a later tier. If the Test's detection declares a native key, the alert's saved search must match that key. Otherwise, if both the detection and the alert carry an ESCU detection_id, they must match (case-insensitive); a mismatch means other, not a title comparison. Only when either id is missing are the titles compared (case-insensitive).
• CrowdStrike NG-SIEM: the correlation_rule_template_id matches (case-insensitive).

An other alert never satisfies expected: alert and never violates expected: none, so a benign control does not fail because an unrelated rule fired on its events. Workloads with no declared detection keep the broad behaviour above: any correlated alert counts.

The run detail in Tracemill Cloud lists other rules' alerts in a separate, neutral note, "Other rules that fired on these events", with the rule, its app or catalog link, the time, and the matched event count. The API exposes the classification as WorkloadAlert.rule_match (declared or other); it is absent for older alerts and for workloads with no declared detection.

An observation-only workload is different: it has no expectation, so the platform tracks event observation but does not correlate alerts.

# Broad alert assertion: any valid alert attributed to this Test case qualifies.
# There is intentionally no job-level detection block.
workloads:
  - scenario: scenarios/custom/probe
    expectation:
      expected: alert

# One job-level detection, inherited by every workload with an expectation
detection:
  source: escu
  name: "ESCU - AWS IAM Delete Policy - Rule"
  id: ec3a9362-92fe-11eb-99d0-acde48001122

workloads:
  - scenario: scenarios/aws/iam/delete-policy
    expectation:
      expected: alert

# CrowdStrike NG-SIEM: the detection names the template, not the deployed rule
detection:
  source: ngsiem_template
  name: "AWS - CloudTrail - Log Trail Deleted"
  id: 0f2a102d-bc64-484d-a3de-89063987235a

workloads:
  - scenario: scenarios/aws/cloudtrail/delete-trail
    expectation:
      expected: alert