One add-on upgrade, one blind detection: current Splunk AWS TA fragments AssumedRole recon

TL;DR: How to lose your recon detection by doing a TA upgrade: about a year ago the Splunk Add-on for AWS started attributing 39 read-only CloudTrail events to the assumed-role session instead of the role. That splits one role's reconnaissance across its sessions - each below AWS Excessive Security Scanning's per-user threshold - and the detection quietly stops firing.

How we tripped over it

When we designed Tracemill, we initially aimed mostly for statefulness and state coherency across events in a single scenario. Another property sort of emerged once we started building the supporting parameter generators: variability. Change one parameter, get a different attack profile.

Take AWS identities for example: the same API call can arrive from a root user, a long-lived IAM user, an assumed role - which is what every SSO/Identity Center sign-in becomes - or a few other federation shapes. How each of those lands in CloudTrail is well-spec'd and inferable from the real captures our generators are modeled on.

That's where we hit an interesting asymmetry: mounting an attack under another identity against real AWS means provisioning the principal for real - the user or role, trust policies, federation config, credentials to plumb - while switching a Tracemill scenario to a different identity type is a one-liner:

 state:
   aws_region:  us-east-1
   account_id:  "111111111111"
-  actor:       gen.aws_identity(type=IAMUser, accountId=ref.account_id)
+  actor:       gen.aws_identity(type=AssumedRole, accountId=ref.account_id)
   source_ip:   gen.ipv4()

The obvious thing to do with cheap variants is to point them at the detection corpus: same attack pattern, adjacent identity - does the detection still hold?

Surprisingly often (shocker, right?) it does not. The identity axis alone turned up a family of stock ESCU detections that filter on user_type=IAMUser and never fire for any other principal type, but that's a story for another time.

Today we'll talk about a detection that used to work, and stopped after an add-on update.

Meet the actors

The detection is Splunk's AWS Excessive Security Scanning and the moving part is the Splunk Add-on for AWS (internally Splunk_TA_aws, the "AWS TA") - the layer that turns raw CloudTrail JSON into the CIM fields (user, user_name, src, user_type) that detections group and threshold on.

The breakage surfaced when we added an AssumedRole variant to the reconnaissance scenario that was authored back in May. A follow-up investigation showed that the offender is the AWS TA, so we traced it to the exact version that introduced it, and root-caused the mechanism.

Watch it break

Here's our setup: Tracemill S3/CloudTrail sink -> S3 -> Splunk_TA_aws (SQS-based-S3 input) -> Splunk 9.3, with TA-Tracemill validating each run. The only variable in the setup is the AWS TA version: 7.6.0, 7.9.1, 7.11.0 and 8.1.2.

The AWS Excessive Security Scanning detection counts distinct read-only API calls per "user" and has a threshold of 50. The relevant SPL snippet is this:

`cloudtrail` eventName=Describe* OR eventName=List* OR eventName=Get*
  | fillnull
  | rename user_name as user
  | stats dc(signature) as dc_events ... BY user
  | where dc_events > 50

Both scenarios model Describe/Get calls spanning many AWS services - 53 distinct actions for the IAM user, 60 for the assumed role - each comfortably above the threshold. Both live in the open-source content library, so you can inject them through your own CloudTrail path with the CLI:

brew install tracemill/tap/tracemill
tracemill update                       # fetch the open-source content library (once)
cd ~/.tracemill/library

# the same reconnaissance sweep under two identities
tracemill run jobs/splunk/aws/iam/aws-excessive-security-scanning.yaml \
  --s3-bucket <your-cloudtrail-bucket> --s3-region us-east-1 --s3-format cloudtrail

tracemill run jobs/splunk/aws/iam/aws-excessive-security-scanning-assumed-role.yaml \
  --s3-bucket <your-cloudtrail-bucket> --s3-region us-east-1 --s3-format cloudtrail

The cloudtrail sink writes to S3 with your ambient AWS credentials, so you'll need an active session - env vars, a named profile, or SSO - authorized to s3:PutObject on that bucket.

The CLI just emits the events into your pipeline; it can't tell you whether the detection fired. For that end-to-end verdict we used Tracemill Cloud, enrolling both variants as a test suite:

For demo purposes, we also added a few other scenarios that exercise ESCU detections across the same CloudTrail surface - a spread of event names, identity types, and aggregation styles:

First run against AWS TA 7.6 to establish the baseline:

Everything is green as expected. Now, upgrade the AWS TA to 8.1.2 and re-run:

Ok, got a failed test as expected. Digging into it, we see that the attack telemetry landed and the test failed with the "alert not attributed" cause:

What does this mean? We ran both scenarios (IAMUser and AssumedRole) simultaneously, so the alert actually did get triggered - by the IAMUser one. However, since Tracemill tracks the end-to-end causality between the input (scenario events) and the outcome (the fired alert), it could tell that the AssumedRole variant went uncaught. In other words, if the AssumedRole were the only signal, it would go undetected.

Now, we asked an LLM agent to finish the due diligence and determine where the breakage occurred. It worked through the intermediate AWS TA releases and their release notes - one version is no longer on Splunkbase (more on that below) - pinned the change to a specific version, and produced this summary:

You can see the split in the detection's own aggregation. On 7.6/7.9.1, one actor, over the threshold:

On 7.11 and later, six buckets, every one under it:

The one line that does it

The whole effect is a single added branch in the add-on's [aws:cloudtrail] field extractions. user_name is built by a case() chain; 7.10 inserted a clause mapping a specific set of read-only event names to the last segment of the ARN:

# Splunk_TA_aws, props.conf [aws:cloudtrail], EVAL-user_name
# added in 7.10.0 (absent in 7.9.1):
eventName IN ("DescribeSecurityGroups","DescribeSnapshots","DescribeInstances", ... 39 names ...),
  mvindex(split(mvindex(split('userIdentity.arn',":"), -1), "/"), -1)

For an AssumedRole ARN, arn:aws:sts::<acct>:assumed-role/<role>/<session>, the last /-segment is the session name - so each session becomes its own user_name. For an IAM-user ARN, arn:aws:iam::<acct>:user/<name>, the same expression returns <name> unchanged - which is why the IAM-user path is untouched, and why this is specifically an AssumedRole aggregation change.

That split does not need multiple sessions to bite. For any single assumed-role principal, the in-set reads now attribute to the session while the out-of-set reads still attribute to the role - so one actor's one sweep is cut across two identities. The recon-role -> 25 bucket and the five session buckets above are all the same principal, divided only by which events landed on the remapped list. Extra sessions fragment it further; they are not required to push it under the threshold.

Two facts worth pinning down, because they bound the blast radius:

• The flip is partial, not total. Only 35 of our scenario's 60 calls are in the changed name set; the other 25 stay role-level (that is the recon-role -> 25 bucket above). Any claim of the form "AssumedRole reads now attribute to session" is true only for the enumerated names. Author your reasoning against the list, not the vibe.
• The same names also joined tag=change. 7.10 added the identical set to the aws_cloudtrail_change eventtype, so these read-only events now flow into the CIM Change data model. We confirmed this at the config level; whether it perturbs any Change-based detection depends on that detection's own selection, and in our set the one Change-model detection we validated (Cloud Security Groups Modifications by User) filters on object_category=security_group and is not touched.

It has been this way for about a year

Apparently, this is not new. Splunk's own release notes attribute it to AWS TA 7.10.0 - "Enhanced CIM support for the aws:cloudtrail sourcetype", then a list of CloudTrail event names whose CIM fields (including user) were remapped. What they don't mention is the effect: that the user remap re-attributes assumed-role reads from role to session, and takes a threshold detection down with it. You don't need the pulled 7.10.0 package to confirm the breakpoint - the clause is absent in 7.6.0 and present in 7.11.0, both still on Splunkbase, and the notes place the change in 7.10.0, between them. 7.10.0 itself is gone from Splunkbase (the version list jumps 7.6.0 → 7.11.0), so a defender can't pull the exact package from the vendor - though the vendor still documents the change.

7.11.0 went GA on September 5, 2025, and the behavior persists unchanged through the current default release, 8.2.2 - roughly a year.

How we know it is this, and only this

Is it really the remap, and not some coincidence? The rest of the suite rules that out: a spread of neighboring detections, each differing from the failing case in exactly one variable - event in the changed set or not, role or user, thresholded or not - each with an outcome we predicted before running:

Every prediction held. Exactly one row moved across the version boundary - the one the mechanism says should. The NACL row is the sharp one: it performs the identical rename user_name as user that the failing detection does, and it stays green - proving the failure is about which event names got remapped, not about the rename or the identity type in general.

One footnote: Password Policy Changes reads GetAccountPasswordPolicy - in the changed set but un-thresholded, so it keeps firing; what silently changes on 7.11 is who the alert names - the risk object for an assumed-role actor becomes the session, not the role.

Check your environment

If you run AWS detections and have upgraded the AWS TA past the 7.6/7.9 line - or plan to - three checks tell you where you stand.

Are read-only AssumedRole events attributing to sessions? Pick a busy role and look at how many distinct user_name values its read calls carry:

index=<cloudtrail_index> sourcetype=aws:cloudtrail userIdentity.type=AssumedRole
    (eventName=Describe* OR eventName=List* OR eventName=Get*)
| stats dc(user_name) as distinct_users values(user_name) as users by userIdentity.sessionContext.sessionIssuer.userName

One user_name per role means role-level (old behavior); many - one per session - means the sweep is being fragmented before your threshold detections ever see it.

Which of your enabled detections threshold distinct actions per user over read events? Those are the ones this change can silence. AWS Excessive Security Scanning is the clearest case, but the pattern - stats dc(...) ... BY user | where ... > N over Describe/List/Get - is the thing to grep your enabled content for.

Did read-only events start landing in your Change data model? If you rely on datamodel=Change for AWS, confirm whether the read-event addition changed its volume or your notables:

| tstats count from datamodel=Change where All_Changes.vendor_product=AWS
    by All_Changes.action

You can run all three against your own indexed data - none require digging into the add-on's config to find the clause that caused this.

Does it matter?

This class of silent failures is a tricky one because it's not something you'd expect. You upgraded a vendor-authored add-on within the supported version matrix, so what can possibly go wrong, right?

Is this a big deal? You decide. For one thing, role-assuming is not a niche: every SSO/Identity Center sign-in is an AssumedRole session, so in most modern environments (SOC2 anyone?) the identity our "attack variant" models is arguably the prevalent auth mode for human operators too, not just your k8s cluster roles and stuff.

Here nobody on the defending side added anything, changed any config, or saw any error. Also, you obviously need to keep your environment up-to-date and the new AWS TA versions in question here bring improvements too, not just hard-to-catch regressions. The point is narrower and, we think, more useful:

Any add-on upgrade is a detection-engineering event, whether or not anyone treats it as one. Re-mapping one field rewrites the input to every detection that reads it - a fleet-wide re-tune nobody on your side chose, reviewed, or saw. Your documented coverage didn't change. What it actually covers did.

The only reliable way to know what an add-on upgrade did to your coverage is the boring way we did it here: replay the attacks through the real path on the old version and the new one, and count what still fires. A version bump is exactly the kind of change that looks like nothing and moves everything.

Notes and reproducibility

• Detection: AWS Excessive Security Scanning (1fdd164a-def8-4762-83a9-9ffe24e74d5a). Corpus pinned at security-content commit 0e6ea0a.
• Scenario and job are open source: assumed-role-describe-scanning and aws-excessive-security-scanning-assumed-role.
• Breakpoint: absent in Splunk Add-on for AWS 7.9.1, present in 7.10.0 (documented as "Enhanced CIM support for the aws:cloudtrail sourcetype") and every release since, through 8.2.2. 7.11.0 GA September 5, 2025.
• All arms ran on one Splunk 9.3 instance; only the add-on version varied. Each comparison re-queried the identical indexed events, so field differences are the add-on's, not the data's.