Job Schema

Complete field reference for job configuration files. For conceptual overview, see job concepts.

IDE Integration

Add this directive to the top of your YAML file for autocomplete and validation:

# yaml-language-server: $schema=https://tracemill.dev/schemas/job.schema.json

Properties

state

A map of variable names to ExprStr values, evaluated once before workloads start.

Unlike scenario state, pool.* expressions are allowed here. Job state is available to workload bindings via ref.*, but does not flow to scenarios implicitly — use explicit bindings (e.g. account_id: ref.account_id).

Object/array state: values are not opaque blobs: string leaves nested at any depth evaluate as ExprStr once at job start, matching top-level state semantics. pool.* is allowed at any depth (not just at the top level), and nested ref.* leaves participate in the same cycle / undefined-ref checks as top-level refs.

state:
  account_id: "000000000000"
  region: us-east-1
  api_endpoint: "https://api.example.com:8443"

State values may be overridden at invocation via the CLI's --set key=value flag (repeatable). Override keys must be declared in this state block — unknown keys are rejected before any workload runs. Override values are YAML-decoded into the same type set as the YAML state block, and string overrides follow the ExprStr grammar (so --set region=gen.aws_region() resolves at runtime). Nested string leaves inside an override map or list value follow the same ExprStr rules as declared nested job state.

pools

An array of pool entries. Each entry is either an inline pool definition or a reference to an external pool file. See pool schema for the full pool configuration.

Inline pool

Defines the pool directly in the job file. The type field is optional for inline pools.

pools:
  - id: threat-ips
    sampling:
      mode: random
      seed: 42
    ip_range:
      cidrs:
        - 198.51.100.0/24

External pool reference

Points to a standalone pool file by path or content ID. The job entry's id is used regardless of what id the file contains.

pools:
  - id: users
    path: ./pools/users.yaml

workload

Each workload runs a scenario with optional concurrency, looping, and bindings.

Example

Reference counts use the merged job state, including overrides. Quoted numbers and floats such as "10" or 10.0 are rejected; use the integer 10. Zero still runs once—it does not disable a workload. The count applies to each concurrent instance, and bindings are evaluated afresh on every iteration. There is no additional workload-count cap beyond representable integers and normal cancellation; scenario foreach limits are separate.

Run-workload mutes are scoped to the reference key and its effective state expression, including overrides. Retuning attack_count: 3 to 5 changes that identity, so a mute for the old value does not match. An unchanged generator expression keeps the same identity across different draws. Literal loop counts remain outside this mute identity.

state:
  attack_count: 10

workloads:
  - scenario: ./scenarios/brute-force.yaml
    concurrency: 2
    bindings:
      source_ip: pool.threat-ips
      username: pool.target-usernames
      account_id: ref.account_id

  - scenario: ./scenarios/api-calls.yaml
    loop: ref.attack_count
    eps: 50
    start_after: 5s
    fields:
      environment: production

expectation

Declares what a workload asserts about alert behavior. A workload without an expectation validates on delivery alone.

workloads:
  - scenario: scenarios/aws/iam/delete-policy
    expectation:
      expected: alert
      summary: "IAM DeletePolicy removes a managed policy"

detection

Optionally records which Detection a Splunk job validates, linking the Test to its detective control. It appears once at the job root and applies to every workload with an expectation. A job declares at most one Detection; workloads cannot select their own.

Resolution. Every workload that has an expectation inherits the optional job-level detection. A Test may omit detection while retaining expected: alert | none; that is not observation-only. In this form, any valid in-window alert attributed to the Test case satisfies alert and violates none. The manifest carries no detection_rule_name, so there is no scheduler did-not-run watch, and it carries no native_key, so pass-crediting evidence is not gated by Detection identity. If no alert arrives, alert resolves to no_alert at the deadline, subject to event-observation failure precedence.

An observation-only workload is different: it has no expectation, so the platform tracks event observation but does not correlate alerts.

# Broad alert assertion: any valid alert attributed to this Test case qualifies.
# There is intentionally no job-level detection block.
workloads:
  - scenario: scenarios/custom/probe
    expectation:
      expected: alert

# One job-level detection, inherited by every workload with an expectation
detection:
  source: escu
  name: "ESCU - AWS IAM Delete Policy - Rule"
  id: ec3a9362-92fe-11eb-99d0-acde48001122

workloads:
  - scenario: scenarios/aws/iam/delete-policy
    expectation:
      expected: alert