Disclaimer: This article is provided strictly for educational purposes and authorized security testing. Only run these techniques against systems you own or have explicit written permission to assess. Unauthorized access to computer systems is illegal in virtually every jurisdiction and can carry severe penalties.
Introduction
The Elastic Stack — Elasticsearch, Elastic Agent/Beats for ingestion, Kibana, and the Elastic Security detection engine built on top — is one of the most widely deployed open-core SIEM/XDR platforms, and building a usable detection pipeline on it is a matter of getting three layers right in order: normalized ingestion into the Elastic Common Schema (ECS), a detection engine configured with both prebuilt and custom rules mapped to ATT&CK, and a feedback loop (tuning, exceptions, response actions) that keeps the rule set healthy over time rather than accumulating noisy, abandoned rules.
Getting this right matters because the Elastic Security detection engine supports several genuinely different rule mechanics — simple query rules, multi-event EQL sequence rules, threshold rules, indicator-match rules against threat-intel feeds, and machine-learning-based anomaly rules — and picking the wrong mechanic for a given detection either misses correlated activity a simple query can’t express, or wastes compute on a rule type more complex than the pattern requires.
Attack Prerequisites
A working Elastic detection pipeline needs the following pieces in place:
- A deployed Elastic Stack — Elasticsearch and Kibana with the Elastic Security app enabled, either self-managed or on Elastic Cloud.
- Data shippers — Elastic Agent enrolled to Fleet with the relevant integrations (the “Windows” integration for Sysmon/Security log parsing, “Elastic Defend” for endpoint telemetry, or Filebeat/Winlogbeat modules for other sources), all normalizing into ECS fields.
- A license tier appropriate to the rule types needed — basic query and EQL rules are broadly available; some rule types and response actions are gated to higher subscription tiers.
- Query-language fluency — KQL/Lucene for simple rules, EQL for sequence/correlation rules, since rule authoring happens directly in one of these.
- A rule-versioning strategy — either Kibana’s own rule export/import (
.ndjson) or the publicelastic/detection-rulesrepository’s TOML-based format if pursuing detection-as-code.
How It Works
Ingestion: Elastic Agent, managed through Fleet, runs integrations that both collect and parse source data into ECS — the “Windows” integration, for example, ingests Sysmon and Security event logs and maps them to fields like event.code, process.name, process.parent.name, and process.command_line, so a detection rule written against ECS fields works regardless of whether the underlying log came from Sysmon, the Security log, or a different OS’s equivalent telemetry, as long as the integration maps it consistently.
The detection engine runs each enabled rule on its configured schedule against the specified index pattern. Query rules use KQL or Lucene for single-event matches. EQL rules support sequence by <field> with maxspan=<duration> syntax to correlate multiple related events — for example a process-creation event followed by a network connection from the same process within a time window — which a flat query rule cannot express. Threshold rules fire when an aggregation (e.g. distinct destination hosts per source in an hour) crosses a value. Indicator-match rules join incoming events against a threat-intel index of known-bad indicators. Every match becomes a signal written to the .alerts-security.alerts-* index, independent of the source index, and can be escalated into a Case for investigation tracking.
Rule management as code: the public elastic/detection-rules GitHub repository maintains hundreds of Elastic-authored and community-contributed rules as version-controlled TOML files, each carrying ATT&CK tags, and ships a Python CLI (python -m detection_rules) for schema validation, unit testing rule logic against sample data, and generating the .ndjson bundle Kibana imports — the same detection-as-code discipline commonly applied to Sigma, adapted to Elastic’s native rule format.
Practical Example / Configuration
A real Elastic EQL sequence rule detecting the WMI lateral-movement pattern — a WmiPrvSE.exe process followed by a suspicious child process on the same host within a short window, correlated on the parent’s entity ID:
sequence by host.id with maxspan=30s
[process where event.type == "start" and
process.name : "WmiPrvSE.exe"] by process.entity_id
[process where event.type == "start" and
process.parent.name : "WmiPrvSE.exe" and
process.name : ("cmd.exe", "powershell.exe")] by process.parent.entity_id
The rule metadata that wraps it when defined via the detection-rules repo’s TOML format (trimmed):
[rule]
name = "Suspicious WMI Process Spawn (Possible Lateral Movement)"
description = """
Detects a shell process spawned by WmiPrvSE.exe shortly after it starts,
consistent with remote WMI process execution (T1047).
"""
risk_score = 73
severity = "high"
type = "eql"
language = "eql"
index = ["logs-windows.*", "winlogbeat-*"]
[[rule.threat]]
framework = "MITRE ATT&CK"
[rule.threat.tactic]
id = "TA0008"
name = "Lateral Movement"
[[rule.threat.technique]]
id = "T1047"
name = "Windows Management Instrumentation"
Walkthrough / Exploitation
Standing up the pipeline from an empty Elastic Stack deployment:
1. Install Elastic Agent on endpoints and enroll to a Fleet policy that
includes the 'Windows' integration (or 'Elastic Defend' for full EDR
telemetry) so Sysmon/Security data lands normalized into ECS.
2. Verify field mapping in Kibana Discover against the target index
pattern (logs-windows.*) before writing any rule -- a rule against
the wrong field name silently never matches.
3. Load Elastic's prebuilt detection rules (Security > Rules > Add Elastic
rules) relevant to your environment as a fast baseline; these already
carry ATT&CK tags and ship pre-tested.
4. Author custom rules (Security > Rules > Create) for gaps the prebuilt
set doesn't cover, e.g. the EQL sequence rule above, setting the index
pattern, schedule, risk score, severity, and ATT&CK mapping fields.
5. Configure rule actions -- webhook/email/SOAR integration -- so a fired
rule reaches a human or an automated playbook, not just the alerts
index.
6. Tune false positives via the rule's Exceptions list rather than editing
rule logic ad hoc, keeping the base rule stable and reusable.
7. If pursuing detection-as-code, migrate custom rules into the
detection-rules repo TOML format and validate with
`python -m detection_rules test` in CI before import.
Note: Alerts live in a separate
.alerts-security.alerts-*index from the source data, which trips up new users troubleshooting a rule that “isn’t firing” — always confirm the underlying query matches data in the *source* index pattern first (via Discover or the rule’s own preview) before assuming the detection logic itself is wrong.
Opsec: Machine-learning anomaly rules need a meaningful baseline period (typically weeks) of representative data before they produce reliable results; enabling one against a freshly onboarded, low-history index produces noisy or meaningless output rather than useful anomaly detection.
Detection and Defense
Operational practices that keep an Elastic detection pipeline healthy rather than degrading into noise:
- Bootstrap from prebuilt rules — Elastic’s maintained rule set already carries ATT&CK coverage and is a faster path to baseline visibility than authoring everything from scratch.
- Monitor rule execution health — the Rule Monitoring view surfaces execution errors and gaps (missed scheduled runs), which silently defeat a rule just as effectively as a wrong query.
- Version-control exceptions too — false-positive suppressions drift out of sync with the rules they apply to if managed only through the UI.
- Pair detections with response actions — Elastic Defend supports automated host isolation and other response actions triggerable from high-severity alerts, shortening time-to-containment for confirmed lateral-movement or credential-access findings.
- Track ATT&CK coverage from rule tags — since both prebuilt and custom rules carry
threat.technique.idmetadata, this feeds directly into a coverage map without separate manual tagging.
Real-World Impact
Elastic Security is widely deployed as an open-core SIEM and endpoint security alternative to proprietary platforms, and the public elastic/detection-rules repository — with hundreds of community and Elastic-authored, ATT&CK-tagged, version-controlled rules — is used both directly by Elastic Security customers and as a reference rule set studied by teams building detections for other platforms. Elastic’s security research team regularly publishes detection content tied to newly reported ransomware and intrusion TTPs, which is one of the more visible public examples of a vendor operating its own rule set with the same detection-as-code discipline described for Sigma.
Conclusion
A durable Elastic detection pipeline is less about any single clever rule and more about getting the layers right in order: normalized ECS ingestion so rules are portable across log sources, the right rule mechanic (query, EQL sequence, threshold, indicator-match) for each pattern, and an operational loop — health monitoring, versioned exceptions, response actions — that keeps the rule set trustworthy as the environment and rule count grow rather than decaying into an unmaintained pile of stale detections.
You Might Also Like
If you found this useful, these related deep-dives cover adjacent techniques and their defenses:
Comments