Pipelines

A pipeline is a delivery destination attached to a target — where a run's generated events are sent. (The CLI configures the same destinations per invocation as sinks; on the platform they are saved, named pipelines on a target.) v1 supports three kinds:

• HEC — Splunk HTTP Event Collector.
• S3 — an Amazon S3 bucket.
• Google Cloud Pub/Sub — publish to a Pub/Sub topic (GCP Cloud Audit Logs).

Add one from the target's setup checklist (step 3 · Add pipeline) or the target's Pipelines section → Add pipeline. Pick a kind, then fill the form — required fields are shown first, with the rest under Advanced. A target can hold many pipelines.

Names and editing

• Each pipeline has a name, unique among a target's active pipelines. It is a human label; runs bind to a pipeline by its internal id, not its name.
• Pipelines are immutable. To change one, delete it and create a new one. Deleting soft-archives the pipeline, so in-flight and historical runs keep resolving the exact version they used; only active pipelines accept new runs.
• Delete from the pipeline's card — it is type-to-confirm.

HEC

Required

• Name — a label for this pipeline.
• HEC URL — your Splunk HTTP Event Collector endpoint (for example https://http-inputs.<host>:443 or https://splunk:8088).
• HEC token — the Splunk HEC token.

Optional

• Index — target index for delivered events.
• Sourcetype — sourcetype assigned to delivered events.

Advanced

• Source / Host — override the source / host fields on delivered events.
• Request timeout — per-request timeout as a Go duration (for example 30s).
• Batch — flush controls: max events, max bytes, and max wait per batch.
• Indexer acknowledgement — enable HEC ack and set its poll interval and timeout.
• Insecure TLS — skip certificate verification (self-signed development certs only).

The HEC token is not your validator key. The HEC token authenticates to Splunk's HTTP Event Collector and is created in Splunk (Settings → Data inputs → HTTP Event Collector). It is not the tmv_… validator key you issue in Tracemill for the TA-Tracemill add-on — see Connect Splunk.

S3

Required

• Name — a label for this pipeline.
• Bucket — the destination S3 bucket name.
• Region — the bucket's AWS region (for example us-east-1). Region is required — Tracemill never infers it, and a region-less pipeline silently fails on its first write. Commercial AWS regions only; GovCloud, China, and ISO partitions are not supported.
• Role ARN — the cross-account IAM role Tracemill assumes to write to your bucket (see Grant S3 access). Format arn:aws:iam::<account-id>:role/<name>.

Optional

• Prefix — key prefix events are written under (for example tracemill/). Leave blank to write at the bucket root. The IAM wildcards * and ? are not allowed.
• Format — jsonl (default), cloudtrail, or asl. The cloudtrail format only accepts aws.cloudtrail events and writes the CloudTrail object layout; the asl format only accepts aws.asl events and writes OCSF parquet in Amazon Security Lake object layout (consumed by the Splunk Add-on for AWS SQS-based-S3 / Security Lake input).

Advanced

• CloudTrail org ID — organization id for the CloudTrail org-trail path layout (only used with the cloudtrail format).
• Batch — flush controls: max events and max wait per batch.

Grant S3 access

S3 delivery uses a cross-account IAM role: Tracemill's delivery worker calls sts:AssumeRole on a role in your AWS account, with the ExternalId set to your target's id and RoleSessionName = tracemill-<target id>. You never share long-lived credentials. Trust is scoped to that single External ID, so a leaked principal alone cannot assume the role on another target's behalf — this is confused-deputy protection.

The Cloud UI's S3 form shows the two values you need: the delivery principal ARN (the Tracemill worker allowed to assume the role) and your target's External ID (the target id). Use the exact values shown there.

Create the role one of two ways.

CloudFormation (recommended)

1. In the S3 form's Grant access section, click Launch CloudFormation stack. This opens the AWS console quick-create with the template and parameters — delivery principal, External ID, bucket, and prefix — pre-filled. Leave the principal and External ID unchanged.
2. Acknowledge the IAM capability and create the stack.
3. Copy the stack's RoleArn output and paste it into the form's Role ARN field.

Manual (IAM console / Terraform)

Create a role with the trust policy below, substituting the delivery principal ARN and your External ID from Tracemill Cloud:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "DELIVERY_PRINCIPAL_ARN" },
      "Action": "sts:AssumeRole",
      "Condition": { "StringEquals": { "sts:ExternalId": "YOUR_EXTERNAL_ID" } }
    }
  ]
}

Attach this permission policy — delivery only writes objects (it never lists, reads, or deletes), scoped to your bucket and prefix:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteEvents",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::YOUR_BUCKET/YOUR_PREFIX*"
    }
  ]
}

With no prefix, use arn:aws:s3:::YOUR_BUCKET/*. The trailing * covers every key under the prefix — don't put * or ? in the prefix itself.

Then paste the role's ARN into the form's Role ARN field.

The IAM role is global. The pipeline's Region is your bucket's region; it also selects which regional CloudFormation console the launch button opens.

Google Cloud Pub/Sub

A gcp_pubsub pipeline publishes synthetic Google Cloud Audit Logs directly to a Pub/Sub topic — the real entry point of the GCP → SIEM ingestion pipeline (Cloud Logging → Log Router sink → Pub/Sub → Splunk Add-on / Dataflow). This is a higher-fidelity option than HEC for the GCP surface: Tracemill publishes the record at the pipeline's start and lets your own Log Router leg add the envelope and routing, exercising your real ingestion path end to end.

Required

• Name — a label for this pipeline.
• Project ID — the GCP project that owns the topic.
• Topic — the Pub/Sub topic id to publish to.

Advanced

• Batch — flush controls: max events, max bytes, and max wait per batch.

This pipeline accepts only gcp.audit events; it publishes the bare Cloud Audit LogEntry as the message body — exactly what a Cloud Logging Log Router sink writes to the topic. Your consumer leg adds its own envelope.

The published shape matches your consumer leg. The Splunk Add-on for GCP (pull) wraps the record under a top-level data object, so its detections read data.protoPayload.*. The Dataflow "Pub/Sub to Splunk" template (push) delivers the bare LogEntry, so its detections read protoPayload.*. A detection matches only the leg it was written for. The HEC pipeline reproduces the Add-on (pull) shape; this Pub/Sub pipeline delivers the bare record.

Grant publish access

Publish access uses Workload Identity Federation — Tracemill's delivery worker federates into GCP from AWS, so there are no service-account keys to share (the GCP analog of the S3 cross-account role). Access is scoped per target: you grant the Pub/Sub Publisher (roles/pubsub.publisher) role on your topic to a member that is unique to this target, so a leaked topic name alone can't let another target publish to it — the same confused-deputy protection as the S3 External ID.

The Add pipeline → Google Cloud Pub/Sub flow surfaces the exact member and a copy-paste grant for your project and topic:

gcloud pubsub topics add-iam-policy-binding TOPIC \
  --project=PROJECT_ID \
  --member='principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL/attribute.target_id/TARGET_ID' \
  --role='roles/pubsub.publisher'

The same grant is available as a google_pubsub_topic_iam_member Terraform resource, an "Open in Cloud Shell" link, and a console click-through. The attribute.target_id value is this target's id; the rest of the member is a Tracemill platform constant, shown in the guide.

Synthetic events land in your production Pub/Sub topic and index. Scope the topic and plan cleanup accordingly.