Running security for my startup, I’ve found it hard to "get to a known state and stay there" in SecOps.
You've set up your favorite SIEM/EDR/XDR/what-have-you, plugged in your data source, and got some detection rules armed. So far so good. Now: does it work? How do you know it works?
If your rules keep firing on harmless activity, you’ve got false positives to trim down. That can be a lot of work, but at least you can see what needs fixing. False negatives are trickier: when a search keeps running successfully but returns nothing, how do you tell whether there was nothing to detect or something in your setup stopped working?
Detection testing is, naturally, not new. Anton Chuvakin and his co-authors make the case for repeated detection testing in their detection engineering series, including injecting logs when running the actual activity is too risky. The practical question for me is how to make that part of everyday security operations.
Looking at what’s available, I’d roughly group the approaches into three buckets:
Config analysis: inspect rules, log sources, and fields to find broken dependencies without executing an attack. It can flag missing prerequisites, but doesn’t show that real data, after filtering and transformation in your production pipeline, still triggers the expected alert.
Testing against recorded data: feed recorded attack data into a test SIEM while developing detections or checking them in CI. A pass proves the rule against that dataset in that environment; it doesn’t establish that your production pipeline still delivers the data the rule needs.
Attack execution: run an attack technique on an endpoint or in a cloud account, then check for its telemetry and the expected alert. This exercises real event generation and collection, but requires permissions, test resources, and cleanup. Disruptive techniques are better suited to a sandbox (Stratus Red Team recommends one), which brings its own setup and maintenance work.
This is all good, but what I actually want is to keep checking that my detections work through my production pipeline, without having to execute an actual attack every time.
This is what we're building Tracemill for. Tracemill sends synthetic security events through the production detection pipeline, then checks whether they reached your SIEM and triggered the expected alert, end to end.
Why synthetic events? We do use real data to model Tracemill scenarios, but we don’t drive real activity every time, for every rule, because it's rather prohibitively complex. Synthetic events leave a much smaller footprint, making it practical to run the tests as often as needed:
- no actual detonations in prod
- no credentials with attack permissions to manage
- no attack infra to provision
- no detonation side effects to clean up
Send the events through the same filtering and transformation as your real data, and you can check that (a) the specific events you sent became searchable; and (b) the expected detection fired on those events.
Another advantage of synthetic telemetry is that you can vary the test scenarios without recreating them in a live environment. For example:
- Does the same activity still get detected when it comes from an assumed role instead of an IAM user?
- What about repeated failed MFA attempts?
- How do you test AWS organization-level or root-user activity?
You can test these without setting up the corresponding identities and authentication flows.
Here’s what that looks like in practice. Select the detections you want to check, and Tracemill finds the matching tests. In this batch of 52 detections, one has two variants worth keeping: reconnaissance by an IAM user and by an assumed role.
In this run, the IAM-user variant passed, but the assumed-role variant failed—even though all 60 of its events arrived. The detection fired on the IAM-user test, but no alert was attributed to the assumed-role test. We traced the difference to a change in the Splunk AWS add-on.
So, your detection ran. Did it work? The answer should be a known part of your security posture every day, not something you learn from an annual drill or IR forensics.