Connect Splunk (validator)

To validate runs against a Splunk target, install the TA-Tracemill add-on in your Splunk and authenticate it to the target with a target key. Tracemill Cloud walks you through this on the target's Settings › Connection page, and the add-on's Tracemill Status dashboard does the same from the Splunk side; this page is the reference for each step.

RequirementsCopy Anchor Link

• Splunk Enterprise or Splunk Cloud Platform 9.3 or later, including 10.x. Install on search heads (standalone or a search head cluster, via the deployer).
• Outbound HTTPS from the search head to Tracemill, directly or through an HTTP proxy.
• To configure the add-on and view its Tracemill Status dashboard, the admin role (Splunk Enterprise) or sc_admin (Splunk Cloud Platform). The poller itself runs as the Splunk system user, so no service account is needed.

1. Issue a target keyCopy Anchor Link

In Tracemill Cloud, open the target, go to Settings › Connection, and choose Issue key under Target keys. The token (tmv_…) is shown once — copy it immediately. We store only a hash and cannot show it again; if you lose it, issue a new key. A target can hold up to two active keys at a time for make-before-break rotation.

2. Install the add-onCopy Anchor Link

Download the add-on package and install it in Splunk Web:

• Download the package above, or the newest one from dl.tracemill.io/ta-tracemill/latest.spl. (A Splunkbase listing is coming.)
• In Splunk Web → Apps → Manage Apps → Install app from file, upload the .spl.

The add-on is licensed under the Tracemill Add-on for Splunk EULA.

Cloud vs. self-managedCopy Anchor Link

The same .spl package works everywhere; how you install it depends on your deployment:

• Self-managed Splunk and Splunk Cloud (Victoria experience): upload the .spl directly via Install app from file (above).
• Splunk Cloud (Classic experience): self-service file upload is restricted. Open a Splunk support ticket to have the add-on installed on your stack.
• Automating installs: Splunk Cloud admins can install the add-on as a private app through the Admin Config Service (ACS); on self-managed Splunk you can deploy it with your usual app-management tooling (deployment server, Splunk CLI, or configuration management).

3. Configure the connectionCopy Anchor Link

Open the Tracemill Add-on. On a fresh install its Tracemill Status dashboard shows the setup steps below, with links to each page.

1. Under Configuration → Connection, add a connection with:
   • API URL — your Tracemill platform origin (for example https://api.tracemill.io). Only https:// URLs are accepted.
   • Validator Token — the tmv_… token from step 1.
   • Custom CA Certificate (PEM) — optional; see Proxies and TLS inspection. When you save, the add-on checks the key with Tracemill. If Tracemill rejects it, the save fails with a message to issue a new key. If Tracemill can't be reached yet (for example, the proxy isn't configured), the connection is saved anyway and the dashboard reports any problem once polling starts.
2. Turn on the poller: choose Create poller on the Tracemill Status dashboard, which uses the defaults (poll every 60 seconds, detections from DA-ESS-ContentUpdate), or create it yourself under Inputs → Create New Input → Tracemill Task Poller to set the Polling Interval (5–600 seconds) and Curated Apps.
3. The add-on begins polling for validation tasks.

Proxies and TLS inspectionCopy Anchor Link

If Splunk reaches the internet through an HTTP proxy, enable it under Configuration → Proxy (host, port, and optional username/password; the password is stored encrypted). HTTPS to Tracemill is tunnelled through the proxy with CONNECT; the add-on's calls to the local Splunk management port never use it. If the proxy settings can't be read, or the proxy is enabled without a host and port, the add-on sends nothing rather than connecting directly.

If the proxy inspects TLS with a private CA, paste that CA certificate (PEM, one or more -----BEGIN CERTIFICATE----- blocks) into the connection's Custom CA Certificate field. It is stored encrypted and added to the system trust store; certificate and hostname verification stay on. An invalid certificate is rejected when you save the connection.

4. Confirm it's connectedCopy Anchor Link

Once the add-on makes its first call, the target's setup checklist in Tracemill Cloud shows Connected and when it was last seen. Until then it reads Waiting for first poll, and so does the add-on's Tracemill Status dashboard.

Supported add-on versionsCopy Anchor Link

Every call the add-on makes carries its version. Tracemill can mark a version deprecated or no longer supported, and the add-on's Tracemill Status dashboard shows a banner when it does:

• Deprecated: everything keeps working; upgrade soon.
• No longer supported: the add-on keeps polling, so the target stays connected in Tracemill Cloud, but it pauses validation work until you upgrade. After the upgrade it resumes on its next poll, without a Splunk restart.

Data sent to TracemillCopy Anchor Link

All communication is outbound HTTPS from the add-on to your configured API URL, authenticated with the target-scoped tmv_ token.

The control-plane calls (checking the key when you save a connection, polling for validation tasks, acknowledging them, reading task state) carry the input's own settings and no Splunk data: the task poll reports its poll interval, its detection-inventory interval, and whether detection-inventory reporting is enabled, so Tracemill can judge how current a catalog should be and tell a quiet add-on from a deliberately disabled one. Beyond those settings, the only data that leaves your Splunk is these three flows:

1. Detection inventory (periodic, can be disabled). For each scheduled, shared saved search in the configured Curated Apps: app, owner, title, enabled/scheduled state, and classification signals (action.correlationsearch.enabled/.metadata/.annotations, presence of ESCU keys) — plus a scan digest, scanned count, partial-visibility flag, the add-on version, and an instance fingerprint from /services/server/info (GUID, server name, Splunk version, cluster label). This powers the detection catalog and target resolution in Tracemill Cloud.
2. Event observations: the Tracemill-generated event ID and the timestamp it was observed in Splunk.
3. Alert correlation: the fired detection's name, fire time, workload index, and a source reference (search job SID / savedsearch id). When the alert is correlated to a specific generated event, the matched Tracemill event ID is included; aggregate/result-row correlations carry no event ID.

Never sent: raw events or search results, SPL search strings, index names, Splunk session keys, or any customer log data. Validation only ever searches for events Tracemill itself generated.

To stop the periodic version of flow 1, disable Detection Inventory Reporting on the input (flows 2–3 are the validation function itself). While disabled, the add-on records a detection_inventory_disabled event in _internal instead of reporting on its own schedule.

An inventory refresh you request explicitly from Tracemill Cloud still runs while reporting is disabled, so an operator on your account can always obtain a current catalog on demand. Disabling therefore stops the add-on's own schedule, not every possible inventory read. To stop all of it, disable or remove the input itself — clearing Curated Apps does not do it, since an empty selection falls back to the default DA-ESS-ContentUpdate scope.

TroubleshootingCopy Anchor Link

• Still "waiting for first poll": confirm the input is enabled, the API URL is reachable from Splunk, and the token hasn't been revoked.
• "Tracemill rejected the target key": the Tracemill Status dashboard shows this, and the connection tile reads key rejected, when the latest poll got HTTP 401 or 403. The key was mistyped or revoked, or the API URL points to a different Tracemill. Issue a new key and paste it into the connection; the poller picks it up on its next poll.
• TLS verification failures: the add-on always verifies the Tracemill endpoint's certificate. If your egress passes through a TLS-inspection proxy with a private CA, paste that CA into the connection's Custom CA Certificate field (or exempt the Tracemill API host from inspection) — verification cannot be disabled.
• Proxy errors: failures reaching the proxy are logged as via proxy <host>: … in _internal (sourcetype ta_tracemill_poller).
• Where to look: the add-on's log is index=_internal sourcetype=ta_tracemill_poller (raise the level under Configuration → Logging); the poller events behind the dashboard are index=_internal sourcetype=tracemill_poller.
• Support bundle: splunk diag --collect app:TA-tracemill collects the add-on's logs, its configuration with secrets redacted, and KV Store row counts. Attach it when you contact support.