Pipelines

Applies to: Platform. This page covers adding delivery pipelines to a target in the Tracemill console. The console's Add pipeline drawer and the S3 IAM role guide link here.

A pipeline is a delivery destination attached to a target — where a run's generated events are sent. (The CLI configures the same destinations per invocation as sinks; on the platform they are saved, named pipelines on a target.) v1 supports two kinds:

• HEC — Splunk HTTP Event Collector.
• S3 — an Amazon S3 bucket.

Add one from the target's setup checklist (step 3 · Add pipeline) or the target's Pipelines section → Add pipeline. Pick a kind, then fill the form — required fields are shown first, with the rest under Advanced. A target can hold many pipelines.

Names and editing

• Each pipeline has a name, unique among a target's active pipelines. It is a human label; runs bind to a pipeline by its internal id, not its name.
• Pipelines are immutable. To change one, delete it and create a new one. Deleting soft-archives the pipeline, so in-flight and historical runs keep resolving the exact version they used; only active pipelines accept new runs.
• Delete from the pipeline's card — it is type-to-confirm.

HEC

Required

• Name — a label for this pipeline.
• HEC URL — your Splunk HTTP Event Collector endpoint (for example https://http-inputs.<host>:443 or https://splunk:8088).
• HEC token — the Splunk HEC token.

Optional

• Index — target index for delivered events.
• Sourcetype — sourcetype assigned to delivered events.

Advanced

• Source / Host — override the source / host fields on delivered events.
• Request timeout — per-request timeout as a Go duration (for example 30s).
• Batch — flush controls: max events, max bytes, and max wait per batch.
• Indexer acknowledgement — enable HEC ack and set its poll interval and timeout.
• Insecure TLS — skip certificate verification (self-signed development certs only).

The HEC token is not your validator key. The HEC token authenticates to Splunk's HTTP Event Collector and is created in Splunk (Settings → Data inputs → HTTP Event Collector). It is not the tmv_… validator key you issue in Tracemill for the TA-Tracemill add-on — see Connect Splunk.

S3

Required

• Name — a label for this pipeline.
• Bucket — the destination S3 bucket name.
• Region — the bucket's AWS region (for example us-east-1). Region is required — Tracemill never infers it, and a region-less pipeline silently fails on its first write. Commercial AWS regions only; GovCloud, China, and ISO partitions are not supported.
• Role ARN — the cross-account IAM role Tracemill assumes to write to your bucket (see Grant S3 access). Format arn:aws:iam::<account-id>:role/<name>.

Optional

• Prefix — key prefix events are written under (for example tracemill/). Leave blank to write at the bucket root. The IAM wildcards * and ? are not allowed.
• Format — jsonl (default) or cloudtrail. The CloudTrail format only accepts aws.cloudtrail events and writes the CloudTrail object layout.

Advanced

• CloudTrail org ID — organization id for the CloudTrail org-trail path layout (only used with the cloudtrail format).
• Batch — flush controls: max events and max wait per batch.

Grant S3 access

S3 delivery uses a cross-account IAM role: Tracemill's delivery worker calls sts:AssumeRole on a role in your AWS account, with the ExternalId set to your target's id and RoleSessionName = tracemill-<target id>. You never share long-lived credentials. Trust is scoped to that single External ID, so a leaked principal alone cannot assume the role on another target's behalf — this is confused-deputy protection.

The console's S3 form shows the two values you need: the delivery principal ARN (the Tracemill worker allowed to assume the role) and your target's External ID (the target id). Use the exact values shown there.

Create the role one of two ways.

CloudFormation (recommended)

1. In the S3 form's Grant access section, click Launch CloudFormation stack. This opens the AWS console quick-create with the template and parameters — delivery principal, External ID, bucket, and prefix — pre-filled. Leave the principal and External ID unchanged.
2. Acknowledge the IAM capability and create the stack.
3. Copy the stack's RoleArn output and paste it into the form's Role ARN field.

Manual (IAM console / Terraform)

Create a role with the trust policy below, substituting the delivery principal ARN and your External ID from the console:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "DELIVERY_PRINCIPAL_ARN" },
      "Action": "sts:AssumeRole",
      "Condition": { "StringEquals": { "sts:ExternalId": "YOUR_EXTERNAL_ID" } }
    }
  ]
}

Attach this permission policy — delivery only writes objects (it never lists, reads, or deletes), scoped to your bucket and prefix:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteEvents",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::YOUR_BUCKET/YOUR_PREFIX*"
    }
  ]
}

With no prefix, use arn:aws:s3:::YOUR_BUCKET/*. The trailing * covers every key under the prefix — don't put * or ? in the prefix itself.

Then paste the role's ARN into the form's Role ARN field.

The IAM role is global. The pipeline's Region is your bucket's region; it also selects which regional CloudFormation console the launch button opens.