Advanced SIEM: Architecture, Detection and Optimization
Advanced SIEM: Architecture, Detection and Optimization
This guide presents advanced concepts of Security Information and Event Management (SIEM): from modern architecture to sophisticated detection techniques, including optimization and best practices for a high-performing SOC.
Level: Advanced | Reading time: 15 minutes
📖 What is a SIEM?
SIEM = Security Information and Event Management
A SIEM is a centralized platform that collects, analyzes and correlates all security events from your infrastructure to detect threats in real-time.
The 3 Pillars of SIEM
┌─────────────────────────────────────────────────────────────────────┐
│ THE 3 PILLARS OF SIEM │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ SECURITY │ │ INFORMATION │ │ EVENT │ │
│ │ │ │ │ │ MANAGEMENT │ │
│ │ • Protection │ │ • Centralization│ │ • Collection │ │
│ │ • Detection │ │ • Context │ │ • Correlation │ │
│ │ • Response │ │ • History │ │ • Alerts │ │
│ └───────┬─────────┘ └───────┬─────────┘ └───────┬─────────┘ │
│ │ │ │ │
│ └──────────────────────┼──────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────┐ │
│ │ UNIFIED PLATFORM │ │
│ │ │ │
│ │ • Global Visibility │ │
│ │ • Informed Decisions │ │
│ │ • Automated Response │ │
│ └─────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
Simple Definition
A SIEM is like a "DVR" for your IT security:It records everything that happens 🔴It allows you to go back in case of incident 📼It alerts in real-time in case of threat 🚨It centralizes all cameras (logs) in one place 📺
A SIEM (Security Information and Event Management) is a cybersecurity solution that:
- Collects security logs from the entire infrastructure
- Aggregates and normalizes events in real-time
- Correlates data to detect threats
- Provides unified and centralized visibility
- Enables incident detection and response
🤔 Why Do You Need a SIEM?
The Problem: Modern Complexity
┌────────────────────────────────────────────────────────────────────┐
│ WITHOUT SIEM: CHAOS │
├────────────────────────────────────────────────────────────────────┤
│ │
│ 🏢 Your infrastructure: │
│ ┌──────────┬──────────┬──────────┬──────────┬──────────┐ │
│ │ Cloud │ Servers │ Network │ Users │ IoT │ │
│ │ (AWS) │ (Linux) │(Cisco) │(Windows) │ (Camera) │ │
│ └────┬─────┴────┬─────┴────┬─────┴────┬─────┴────┬─────┘ │
│ │ │ │ │ │ │
│ ▼ ▼ ▼ ▼ ▼ │
│ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ │
│ │ Logs │ │ Logs │ │ Logs │ │ Logs │ │ Logs │ │
│ │disperse│ │disperse│ │disperse│ │disperse│ │disperse│ │
│ └───────┘ └───────┘ └───────┘ └───────┘ └───────┘ │
│ │
│ ❌ Impossible to see the "big picture" │
│ ❌ Attacker slips through the cracks │
│ ❌ Investigation = Days of manual work │
│ ❌ Alert fatigue (noise) │
└────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ WITH SIEM: VISIBILITY │
├────────────────────────────────────────────────────────────────────┤
│ │
│ 🏢 Your infrastructure: │
│ ┌──────────┬──────────┬──────────┬──────────┬──────────┐ │
│ │ Cloud │ Servers │ Network │ Users │ IoT │ │
│ └────┬─────┴────┬─────┴────┬─────┴────┬─────┴────┬─────┘ │
│ │ │ │ │ │ │
│ └──────────┴──────────┼──────────┴──────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ SIEM 🔍 │ │
│ │ Centralized │ │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ SOC Analyst 👨💻 │ │
│ │ │ │
│ │ • Unified view │ │
│ │ • AI Detection │ │
│ │ • Investigation│ │
│ │ • Auto response│ │
│ └─────────────────┘ │
│ │
│ ✅ Everything centralized and correlated │
│ ✅ Detection of complex attacks │
│ ✅ Investigation in minutes, not days │
│ ✅ Contextual and relevant alerts │
└────────────────────────────────────────────────────────────────────┘
Concrete Benefits
| Metric | Without SIEM | With SIEM | Improvement |
|---|---|---|---|
| Time to detect | Days/weeks | Minutes/hours | -90% |
| Investigation time | 8-40 hours | 1-4 hours | -80% |
| Detection rate | ~20% | ~95% | +375% |
| False positives | 80%+ | <20% | -75% |
| Audit compliance | Difficult | Automatic | +100% |
Why is a SIEM Essential?
In a modern environment with hundreds of cloud services, containers, and microservices, security can no longer be done manually. The SIEM becomes the nerve center of the Security Operations Center (SOC).
📅 When to Deploy a SIEM?
Warning Signs You Need One
┌─────────────────────────────────────────────────────────────────────┐
│ ⚠️ WARNING SIGNS: YOU NEED A SIEM │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ 🔴 Recent incident not detected in time │
│ → An attack happened, you discovered it by chance │
│ │
│ 🔴 Investigation takes hours/days │
│ → You search manually through logs on each server │
│ │
│ 🔴 Auditor asks for logs you can't find │
│ → "Give us administrator access logs from March" │
│ → Panic because logs were overwritten │
│ │
│ 🔴 You suspect intrusion but can't prove it │
│ → "Did someone access customer data?" │
│ → "I don't know, impossible to verify" │
│ │
│ 🔴 Email alerts without context │
│ → 500 emails/day from your antivirus │
│ → Your team ignores everything (alert fatigue) │
│ │
│ 🔴 Regulatory compliance (GDPR, PCI-DSS, ISO 27001) │
│ → Auditor: "Show us your consolidated logs" │
│ → You: *cold sweat* │
│ │
│ 🔴 Complex multi-cloud infrastructure │
│ → AWS + Azure + On-premise + Kubernetes │
│ → No global visibility │
└─────────────────────────────────────────────────────────────────────┘
Natural Evolution Toward SIEM
┌─────────────────────────────────────────────────────────────────────┐
│ SECURITY MATURITY CURVE │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Maturity │
│ ▲ │
│ │ 🎯 SIEM + SOAR │
│ 10 │ ┌─────────────┐ │
│ │ ┌────▶│ Advanced │ │
│ 8 │ ┌────┴───┐│ Maturity │ │
│ │ ┌────▶│ Full │└─────────────┘ │
│ 6 │ ┌──────┴───┐ │ SIEM │ │
│ │ ┌────▶│ Basic │ └─────────┘ │
│ 4 │ ┌────┴───┐ │ SIEM │ │
│ │ │ Central│ └──────────┘ │
│ 2 │ ┌──┴────────┘ │
│ │ │ Logs │ │
│ 0 ├─┴────────────────────────────────────────────────────────▶ │
│ 1 5 10 20 50 100 500 1000 │
│ Employees / Infrastructure │
│ │
│ 🔵 Phase 1 : Central logs simple (< 50 employees) │
│ 🟢 Phase 2 : Basic SIEM (50-200 employees) │
│ 🟡 Phase 3 : Full SIEM (200-1000 employees) │
│ 🟠 Phase 4 : SIEM + SOAR (> 1000 employees) │
└─────────────────────────────────────────────────────────────────────┘
Recommended Deployment Timeline
┌─────────────────────────────────────────────────────────────────────┐
│ 📅 IDEAL TIMELINE FOR SIEM DEPLOYMENT │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌────────────┬────────────┬────────────┬────────────┐ │
│ │ Phase 1 │ Phase 2 │ Phase 3 │ Phase 4 │ │
│ │ Day 1 │ Month 1-3 │ Month 4-6 │ Month 7-12│ │
│ ├────────────┼────────────┼────────────┼────────────┤ │
│ │ 📋 Audit │ 🔧 Install │ 🎯 Custom │ 🤖 SOAR │ │
│ │ logs │ & config │ rules │ Playbooks │ │
│ └────────────┴────────────┴────────────┴────────────┘ │
│ │
│ Phase 1 : Identify critical sources │
│ → Cloud infrastructure, Active Directory, Firewalls │
│ │
│ Phase 2 : Progressive deployment │
│ → Start with most critical logs │
│ → Avoid "big bang" │
│ │
│ Phase 3 : Business-specific detection │
│ → Rules specific to your industry │
│ → Ex: Finance = fraud detection │
│ Ex: Healthcare = patient data access │
│ │
│ Phase 4 : Automation (SOAR) │
│ → Automated response to common threats │
│ → Investigation playbooks │
└─────────────────────────────────────────────────────────────────────┘
ROI: SIEM Pays for Itself Quickly
┌─────────────────────────────────────────────────────────────────────┐
│ 💰 ROI CALCULATION (Return on Investment) │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ INVESTMENT (SIEM Cost) │
│ ┌────────────────────────────────────────┐ │
│ │ • SIEM License : $50K - $300K/year │ │
│ │ • Infrastructure : $20K - $100K │ │
│ │ • Team Training : $10K - $30K │ │
│ │ • Integration : $15K - $50K │ │
│ └────────────────────────────────────────┘ │
│ Total : ~$100K - $500K first year │
│ │
│ SAVINGS (Benefits) │
│ ┌────────────────────────────────────────┐ │
│ │ • Analyst time saved : +$200K/year │ │
│ │ • GDPR fine avoided : $2M+ │ │
│ │ • Data breach avoided : Incalculable │ │
│ │ value (reputation) │ │
│ │ • Cyber insurance : -30% premium │ │
│ │ • SOC productivity : +300% │ │
│ └────────────────────────────────────────┘ │
│ │
│ 🎯 ROI = Typically < 6-12 months │
│ │
│ Real example : │
│ ─────────────── │
│ Average data breach cost = ~$3.86M (IBM Cost of Breach) │
│ A SIEM detecting and stopping an attack = Instant ROI │
└─────────────────────────────────────────────────────────────────────┘
Modern SIEM Architecture
An effective SIEM architecture relies on 5 key components:
┌─────────────────────────────────────────────────────────────┐
│ LOG SOURCES │
│ (Cloud, On-premise, Network, Endpoints, Applications) │
└────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ COLLECTORS │
│ • Kafka • Fluentd • Logstash │
│ • Beats • Vector • Syslog-ng │
└────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ INGESTION & PROCESSING │
│ • Buffering • Parsing • Enrichment │
│ • Normalization • Filtering • Tagging │
└────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ STORAGE │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Hot │ │ Warm │ │ Cold │ │
│ │ (1-7 days) │ │ (1-3 months)│ │ (1+ year) │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ Elasticsearch S3/Glacier Archive │
└────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ CORRELATION & DETECTION │
│ • SIEM Rules • Machine Learning • UEBA │
│ • Threat Intel • ATT&CK Mapping • Anomalies │
└────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ INTERFACE & RESPONSE (SOAR) │
│ • Dashboards • Investigations • Alerts │
│ • Playbooks • Automation • Cases │
└─────────────────────────────────────────────────────────────┘
The 5 Essential Components
1. Collectors (Forwarders)
Collection agents retrieve logs at the source:
| Agent | Use Case |
|---|---|
| Filebeat | Linux/Windows file logs |
| Winlogbeat | Windows events |
| Metricbeat | System metrics |
| Fluentd | Containers, Kubernetes |
| Vector | High performance, Rust |
| Syslog-ng | Network, legacy devices |
Optimal configuration:
# Filebeat example
filebeat.inputs:
- type: log
paths:
- /var/log/nginx/*.log
fields:
service: nginx
environment: production
fields_under_root: true
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after
output.kafka:
hosts: ["kafka1:9092", "kafka2:9092"]
topic: 'logs-nginx'
compression: gzip
2. Ingestion Processors
Real-time processing includes:
- Parsing: Field extraction (JSON, CSV, regex)
- Normalization: Mapping to common schema (ECS, CIM)
- Enrichment: Adding context (IP geolocation, threat intel)
- Filtering: Noise and duplicate removal
Enrichment example:
{
"source_ip": "192.168.1.100",
"enrichment": {
"geo": {
"country": "United States",
"city": "New York",
"asn": "AS7922"
},
"threat_intel": {
"reputation": "clean",
"tor_exit_node": false,
"known_bot": false
}
}
}
3. Storage and Indexing
Tiering strategy:
| Tier | Duration | Storage | Usage |
|---|---|---|---|
| Hot | 1-7 days | SSD/NVMe | Real-time search, alerts |
| Warm | 7-90 days | HDD | Investigation, dashboards |
| Cold | 3-12 months | Object Storage | Compliance, forensics |
| Frozen | 1-7 years | Archive | Legal audit, retention |
4. Correlation Engine
The heart of SIEM for detection:
- Complex correlation rules
- Behavioral detection (UEBA)
- Machine Learning for anomalies
- MITRE ATT&CK mapping
5. Interface and SOAR
Key features:
- Real-time dashboards
- Advanced forensic search
- Case investigation management
- Automated response (SOAR)
Advanced Correlation Rules
Types of Correlation
1. Temporal Correlation
Detection based on thresholds within a time window:
# Example: Brute force detection
detection:
selection:
EventID: 4625 # Windows login failure
timeframe: 5m
condition: selection | count() by SourceIP > 10
alert:
title: "Brute Force Detection"
severity: high
description: "More than 10 login failures in 5 minutes"
2. Sequential Correlation
Detection of multi-stage attack patterns:
# Example: Lateral Movement
detection:
step1:
EventID: 4624 # Successful login
LogonType: 3 # Network logon
step2:
EventID: 4688 # Process creation
CommandLine|contains:
- 'powershell'
- 'cmd.exe'
step3:
EventID: 5145 # Network share access
condition: step1 followed by step2 within 1m followed by step3 within 5m
3. Statistical Correlation
Anomaly detection compared to baseline:
# ML pseudo-code
baseline = calculate_baseline(user, "login_count", window="30d")
current = count_events(user, "login", window="1h")
if current > baseline.mean + (3 * baseline.std):
alert("Anomalous login volume detected")
4. Chain Correlation
Complex attack detection (MITRE ATT&CK):
Step 1: Initial Access (Phishing)
↓
Step 2: Execution (Malicious macro)
↓
Step 3: Persistence (Registry run key)
↓
Step 4: Privilege Escalation (Token impersonation)
↓
Step 5: Lateral Movement (PSExec)
↓
Step 6: Exfiltration (Data compression)
Advanced Rule Examples
Credential Stuffing:
IF:
> 5 login failures IN 60 seconds
AND
Source IP not in whitelist
AND
Valid login SUCCESS after failures
THEN:
Priority: CRITICAL
Type: Credential Stuffing suspected
Action: Alert + Temporary IP block
Data Exfiltration:
IF:
Outbound volume > 10x user baseline
AND
Connection to non-whitelisted IP
AND
Unusual time (weekend/night)
THEN:
Priority: HIGH
Type: Potential exfiltration
Action: SOC Alert + Quarantine
Advanced Detection Techniques
1. UEBA (User and Entity Behavior Analytics)
Behavioral analysis of users and entities:
Established profiles:
- Usual connection hours
- Geographic locations
- Data access volume
- Used applications
- Peers (colleagues with similar patterns)
Anomaly detection:
| Anomaly | Example |
|---|---|
| Unusual connection | Login at 3 AM from a foreign country |
| Abnormal volume | Downloading 10GB when baseline = 100MB |
| Privileged access | First use of sudo/root |
| Deviant behavior | Accessing never-before-consulted files |
| Insider threat | Searching for sensitive keywords |
2. ATT&CK Mapping
Mapping SIEM rules to MITRE ATT&CK techniques:
# Mapping example
detection:
name: "Suspicious PowerShell Download"
mitre:
technique: T1059.001 # Command and Scripting Interpreter: PowerShell
tactic: TA0002 # Execution
sub_technique: T1105 # Ingress Tool Transfer
query: >
ProcessName: "powershell.exe"
AND CommandLine: "*Invoke-WebRequest*"
AND CommandLine: "*http*"
false_positives:
- "Legitimate admin scripts"
- "Automated updates"
level: high
ATT&CK Coverage:
- Identify uncovered techniques
- Prioritize detections by criticality
- Measure SOC maturity
3. Threat Intelligence Integration
Enrichment with IOCs (Indicators of Compromise):
CTI Sources:
- MISP (Malware Information Sharing Platform)
- AlienVault OTX
- IBM X-Force
- VirusTotal
- Commercial feeds (Recorded Future, CrowdStrike)
IOC Types:
- Malicious IPs
- C2 domains
- File hashes
- Suspicious User-Agents
- Malicious SSL certificates
Implementation:
# Automatic lookup
def enrich_ip(ip_address):
reputation = threat_intel_lookup(ip_address)
return {
"ip": ip_address,
"reputation": reputation.score,
"categories": reputation.categories,
"first_seen": reputation.first_seen,
"sources": reputation.feeds
}
4. Machine Learning for Detection
Algorithms used:
| Algorithm | Usage | Example |
|---|---|---|
| Clustering (K-means) | Grouping similar events | Attack campaign detection |
| Isolation Forest | Anomaly detection | Rare activity detection |
| LSTM | Temporal sequences | Access pattern prediction |
| Random Forest | Classification | Risk scoring |
| Graph Analysis | Entity relationships | Lateral movement detection |
ML Use Cases:
- DGA (Domain Generation Algorithm) detection
- Unknown log classification
- User risk scoring
- SOC ticket prediction
Optimization and Performance
Noise Reduction (Alert Fatigue)
Problem: 10,000+ alerts/day → SOC team overwhelmed
Solutions:
- Deduplication:
# Group similar alerts
alert_group = {
"signature": "Brute Force",
"source_ip": "192.168.1.100",
"target": "[email protected]",
"count": 47,
"first_seen": "2026-02-15T10:00:00Z",
"last_seen": "2026-02-15T10:05:23Z"
}
- Temporal aggregation:
- Group events by time window
- Threshold before alert creation
- Context enrichment:
- Check if IP is internal/external
- Check working hours
- Check user history
- Smart filtering:
- Whitelist known IPs
- Blacklist known false positives
- Correlate with planned changes
Storage Optimization
Cost reduction strategies:
| Technique | Reduction | Impact |
|---|---|---|
| Compression | 50-70% | CPU + search time |
| Upfront filtering | 30-50% | Potential event loss |
| Sampling | 90%+ | Partial investigation |
| Tiering | 60-80% | Cold storage latency |
Optimized indexing:
- Index only searched fields
- Use dynamic mappings with constraints
- Archive old indexes
Quality Assurance
Continuous validation:
- Rule testing:
- Test environment before production
- Known attack datasets
- Scenario simulation
- Purple Teaming:
- Red team attacks
- Blue team detects
- Collaborative improvement
- Detection metrics:
- MTTD (Mean Time To Detect)
- MTTR (Mean Time To Respond)
- False positive/negative rates
- ATT&CK coverage
Popular SIEM Platforms
Solution Comparison
| SIEM | Strengths | Weaknesses | Price |
|---|---|---|---|
| Splunk | Powerful search, ecosystem, SPL | High cost, complexity | $$$$ |
| Elastic SIEM | Open source, performant, scalable | Complex configuration | $$ |
| Microsoft Sentinel | Azure integration, built-in ML | Microsoft lock-in, data costs | $$-$$$ |
| IBM QRadar | Advanced correlation, stability | Dated interface, cost | $$$ |
| Wazuh | Open source, full stack, lightweight | Less enterprise features | $ |
| Google Chronicle | Unlimited scale, data pricing | Less mature, cloud only | $$ |
| Sumo Logic | Cloud native, DevOps friendly | Latency, ingestion costs | $$ |
Choice by Context
Startup / Limited Budget:
- Wazuh (open source)
- Elastic SIEM (self-hosted)
Cloud-first Enterprise:
- Microsoft Sentinel (Azure)
- Google Chronicle (GCP)
- Sumo Logic (Multi-cloud)
On-premise Enterprise:
- Splunk Enterprise
- IBM QRadar
- Elastic SIEM
KPIs and SOC Metrics
Essential Performance Indicators
1. MTTD (Mean Time To Detect)
- Time between attack and detection
- Target: < 24h (ideally < 1h)
2. MTTR (Mean Time To Respond)
- Time between detection and containment
- Target: < 4h (ideally < 1h)
3. False Positives
- Percentage of non-relevant alerts
- Target: < 20% of total alerts
4. ATT&CK Coverage
- Percentage of covered techniques
- Target: > 80% of critical techniques
5. Alert Volume
- Alerts/day per analyst
- Target: 10-20 alerts/analyst/day
6. Dwell Time
- Attacker presence time
- Target: Constant reduction
Best Practices
10 SIEM Commandments
- Plan before deploying
- Define priority use cases
- Understand log sources
- Size correctly
- Collect maximum
- Raw logs before filtering
- Better too much than too little
- Archive intelligently
- Normalize formats
- Adopt a standard (ECS, CIM)
- Document schemas
- Consistent mappings
- Document exhaustively
- Detection rules
- Response playbooks
- ATT&CK mappings
- Investigation procedures
- Test regularly
- Purple team exercises
- Red team assessments
- Tabletop exercises
- Automate response
- SOAR for repetitive tasks
- Automated playbooks
- Automatic quarantine
- Train continuously
- SOC analyst training
- Technology monitoring
- Certifications (GCIH, GCIA)
- Optimize costs
- Intelligent tiering
- Selective filtering
- License review
- Measure and improve
- KPI dashboards
- Regular rule review
- Threshold adjustment
- Stay pragmatic
- Avoid "security theater"
- Focus on real risks
- Iterate and improve
Conclusion
A modern SIEM is not just a log collection tool, it's a threat detection and response platform that requires:
- A well-designed architecture (scalable and cost-effective)
- Relevant and maintained correlation rules
- Integration with threat intelligence
- Advanced techniques (UEBA, ML)
- A trained and equipped SOC team
- A continuous improvement process
Investment in a well-configured SIEM is measured in shorter detection and response times, and ability to detect advanced threats that would evade traditional controls.
References
- MITRE ATT&CK Framework
- Sigma Rules
- Elastic Common Schema (ECS)
- Splunk Common Information Model (CIM)
- MISP Project
Article written on February 15, 2026 - based on industry best practices