Data Sources and Telemetry

The Visibility Gap

Most organizations log something. Almost none log everything they need for effective hunting. And even organizations that collect the right sources often have quality problems that silently break queries without producing obvious errors. A query that returns zero results could mean "no threat found" or it could mean "this log source is broken." Without a data quality baseline, you cannot tell the difference.

Three distinct types of gaps affect hunt capability:

Adversaries know about visibility gaps. They operate in them deliberately. An attacker who understands that most organizations do not monitor WMI activity will use WMI for lateral movement exclusively. An actor who knows DNS logs are rarely parsed will use DNS tunneling for C2. Closing gaps eliminates their advantage in those specific areas.

The practical goal of this chapter: build a complete map of what you have, what you are missing, and what each gap costs in ATT&CK technique coverage. That map becomes the foundation for both your hunt prioritization and your telemetry improvement roadmap.

Endpoint Telemetry

Endpoint telemetry is the highest-value data source category for threat hunting. Adversaries spend most of their time on endpoints: executing code, reading credentials, modifying registry keys, moving files, making network connections. Everything they do leaves a trace in endpoint logs.

Windows Event Logs: The Minimum Viable Set

Windows Event Logs are available on every Windows system without additional software. They require specific audit policy configuration to be useful. The default audit policy is insufficient for hunting.

Sysmon: The Threat Hunter's Primary Source

System Monitor (Sysmon) is a Windows system service and device driver that logs detailed process and network activity to the Windows Event Log. It provides data that Windows native logging cannot: full process hash, parent command line in process creation events, DLL load events, process access events (critical for LSASS detection), and DNS query logs from individual processes.

Sysmon must be deployed with a thoughtful configuration. The default minimal config captures everything and produces enormous volumes. The hunt-optimized config captures signal-rich events while filtering known-good noise.

A minimal Sysmon configuration optimized for hunting credential theft, execution, network C2, and persistence:

<!-- Minimal Sysmon hunting configuration -->
<!-- Focus: credential theft, execution, network C2, persistence -->
<Sysmon schemaversion="4.90">
  <EventFiltering>

    <!-- Event 1: Process Create - capture all, exclude common noise -->
    <RuleGroup name="" groupRelation="or">
      <ProcessCreate onmatch="exclude">
        <Image condition="is">C:\Windows\System32\conhost.exe</Image>
      </ProcessCreate>
    </RuleGroup>

    <!-- Event 3: Network connections - exclude high-volume known-good -->
    <RuleGroup name="" groupRelation="or">
      <NetworkConnect onmatch="exclude">
        <!-- Exclude svchost to port 443 (Windows Update noise) -->
        <Image condition="is">C:\Windows\System32\svchost.exe</Image>
      </NetworkConnect>
    </RuleGroup>

    <!-- Event 7: Image Load - only capture unsigned or suspicious -->
    <RuleGroup name="" groupRelation="or">
      <ImageLoad onmatch="include">
        <Signed condition="is">false</Signed>
      </ImageLoad>
    </RuleGroup>

    <!-- Event 10: Process Access - capture LSASS reads (critical) -->
    <RuleGroup name="" groupRelation="or">
      <ProcessAccess onmatch="include">
        <TargetImage condition="end with">lsass.exe</TargetImage>
      </ProcessAccess>
    </RuleGroup>

    <!-- Event 11: File Create - capture drops in suspicious locations -->
    <RuleGroup name="" groupRelation="or">
      <FileCreate onmatch="include">
        <TargetFilename condition="contains">\Temp\</TargetFilename>
        <TargetFilename condition="contains">\AppData\Roaming\</TargetFilename>
        <TargetFilename condition="end with">.exe</TargetFilename>
        <TargetFilename condition="end with">.dll</TargetFilename>
        <TargetFilename condition="end with">.ps1</TargetFilename>
      </FileCreate>
    </RuleGroup>

    <!-- Event 13: Registry Value Set - capture persistence-relevant keys -->
    <RuleGroup name="" groupRelation="or">
      <RegistryEvent onmatch="include">
        <TargetObject condition="contains">Run</TargetObject>
        <TargetObject condition="contains">RunOnce</TargetObject>
        <TargetObject condition="contains">CurrentVersion\Image File Execution</TargetObject>
      </RegistryEvent>
    </RuleGroup>

    <!-- Event 22: DNS Query - capture all (high value, low volume) -->
    <RuleGroup name="" groupRelation="or">
      <DnsQuery onmatch="exclude">
        <!-- Optionally exclude known-good Windows domains -->
        <QueryName condition="end with">.microsoft.com</QueryName>
      </DnsQuery>
    </RuleGroup>

  </EventFiltering>
</Sysmon>

Network Telemetry

Network telemetry is the second pillar of hunt visibility. It covers what endpoint logs miss: traffic between hosts that have no EDR agent, east-west movement within segments, and command-and-control communication patterns that are invisible in process logs but obvious in flow data.

DNS Logs: Highest Return on Investment

DNS logs are the most underused hunt data source in most environments. Every C2 beacon performs a DNS resolution. Every DGA domain gets queried. DNS tunneling encodes data directly in query names. And yet most SIEMs have DNS logs disabled or unforwarded.

// Hunt for DNS tunneling: abnormally long query labels (T1071.004)
// DNS tunneling encodes data in subdomains, producing long and often
// encoded label strings. Normal DNS labels rarely exceed 30 characters.
DnsEvents
| where SubType == "LookupQuery"
| extend LabelLength = strlen(extract(@"^([^.]+)", 1, Name))
| where LabelLength > 45
| summarize
    TotalQueries = count(),
    UniqueLabels = dcount(Name),
    SampleLabels = make_set(Name, 5)
  by Computer, bin(TimeGenerated, 1h)
| where TotalQueries > 20 or UniqueLabels > 10
| order by TotalQueries desc

Web Proxy Logs

Proxy logs capture HTTP metadata: method, full URL, user agent, response code, request and response bytes, referrer. This data is essential for detecting staged payload downloads, suspicious user agents, large outbound POST requests (exfil), and C2 traffic patterns.

• Staged downloads: GET requests from processes that should not browse the web (regsvr32.exe, certutil.exe, bitsadmin.exe) to external hosts. URL often ends in .dat, .txt, .gif (disguised payload).
• Suspicious user agents: curl, python-requests, empty string, randomized or missing user agents indicate scripted access, not browser traffic.
• Exfil indicator: Large POST requests (megabytes) to external hosts not matching known SaaS destinations. Especially from servers that do not normally initiate outbound web traffic.
• HTTP C2 beaconing: Fixed-interval GET requests with small response sizes to the same URL. Jitter is common (small random variation in interval). Look at time deltas between consecutive requests from the same process to the same host.

NetFlow and IPFIX

NetFlow captures connection metadata without payload: source IP, destination IP, source port, destination port, protocol, byte count, packet count, start time, end time. It does not require proxy deployment and covers all traffic, not just HTTP.

Identity and Authentication Telemetry

Active Directory is the primary target in the vast majority of enterprise intrusions. Compromising AD means compromising everything it controls: access to file servers, databases, cloud workloads, remote management systems, and backup infrastructure. Identity telemetry is where the attack chain is most visible, yet it is consistently underutilized because the relevant event IDs require specific configuration and the volume is high.

// Hunt for Kerberoasting: multiple RC4-encrypted TGS requests (T1558.003)
// Attacker requests service tickets with RC4 encryption type for offline cracking.
// Legitimate Kerberos environments prefer AES (EncType 0x12 or 0x11).
// RC4 (EncType 0x17) is a strong indicator of Kerberoasting activity.
SecurityEvent
| where EventID == 4769
| where TicketOptions == "0x40810000"
| where TicketEncryptionType == "0x17"      // RC4-HMAC: weak, chosen by attackers
| where ServiceName !endswith "$"            // Exclude machine account service tickets
| where AccountName !endswith "$"            // Exclude machine accounts as requestors
| summarize
    RequestCount = count(),
    TargetAccounts = make_set(ServiceName, 20),
    SourceHosts = make_set(IpAddress, 5)
  by AccountName, bin(TimeGenerated, 30m)
| where RequestCount >= 3
| order by RequestCount desc

Azure AD / Entra ID adds additional identity telemetry for hybrid and cloud environments:

• Sign-in Logs: Risky sign-in flags (Microsoft's own risk engine), MFA status, conditional access result, location and IP. Essential for detecting impossible travel, token theft, and legacy protocol abuse.
• Audit Logs: Role assignment changes, group membership changes, conditional access policy modifications. An attacker who weakens a Conditional Access policy has changed your security posture without touching an endpoint.
• Identity Protection: Microsoft's ML-flagged risk events. Even if you do not block on risk, use these events as hunt seeds for the highest-risk accounts.

Cloud Telemetry

Cloud fundamentally changes the threat model. There is no network perimeter. Every action is an API call. Resources appear and disappear in minutes. Adversaries who compromise cloud environments operate entirely differently from on-premises attackers: they do not pivot across endpoints, they pivot across IAM roles and service accounts.

AWS CloudTrail

CloudTrail logs every API call made in an AWS account: who made it, from where, at what time, and with what parameters. For a threat hunter, this is the equivalent of Sysmon for the cloud. Every action an attacker takes produces a CloudTrail event.

Key patterns to hunt for:

• AssumeRole chains: One compromised role used to assume another, building toward a privileged role. Map the chain of AssumeRole events from a single external IP.
• IAM user creation + admin policy attachment: CreateUser followed by AttachUserPolicy with an admin ARN in the same short window. Classic persistence move.
• API calls from unusual regions: If your organization only operates in us-east-1 and eu-west-1, an API call from ap-southeast-2 warrants investigation.
• Mass S3 GetObject: Hundreds or thousands of GetObject calls in a short window from a single identity. Data exfiltration pattern.

Azure Activity Log and Entra ID

Mail forwarding rules are among the highest-yield BEC (Business Email Compromise) indicators. An attacker who gains mailbox access creates a forwarding rule to an external address. Every email the victim receives is now silently copied to the attacker. This can persist for months. The ExchangeTransportConfig event in the M365 Audit Log captures rule creation. It should be monitored for any rule that forwards to an external domain.

// Hunt for impossible travel in Microsoft 365 / Entra ID (T1078.004)
// Successful logins from two geographically distant locations within
// a timeframe that makes physical travel impossible (less than 2 hours).
// This is a strong indicator of credential theft or session token reuse.
SigninLogs
| where ResultType == "0"           // Successful sign-in only
| where UserType == "Member"        // Exclude guest accounts and service principals
| summarize
    SigninCount = count(),
    Locations = make_set(Location, 10),
    IPAddresses = make_set(IPAddress, 10),
    FirstSeen = min(TimeGenerated),
    LastSeen = max(TimeGenerated)
  by UserPrincipalName, bin(TimeGenerated, 2h)
| where array_length(Locations) > 1    // Multiple locations in 2-hour window
| extend DurationMinutes = datetime_diff('minute', LastSeen, FirstSeen)
| where DurationMinutes < 120          // Two locations within 120 minutes
| project
    TimeGenerated,
    UserPrincipalName,
    Locations,
    IPAddresses,
    SigninCount,
    DurationMinutes
| order by DurationMinutes asc

MITRE ATT&CK Data Sources Mapping

ATT&CK version 14 introduced a structured Data Sources taxonomy. Each technique now explicitly lists which Data Sources and Data Components are needed to detect it. This taxonomy is the bridge between "I want to hunt for T1003.001" and "which log source do I need?"

The hierarchy: Data Source contains Data Components. A Data Component represents a specific observable within a source. Each Data Component has a detection relationship with specific ATT&CK techniques.

The "big three" data sources cover approximately 60% of all ATT&CK techniques: Process Creation, Network Connection Creation, and DNS Query. These three are the non-negotiable minimum. An organization with reliable data flowing in all three can hunt the majority of the ATT&CK technique catalog.

Gap Analysis Process

1. List every confirmed log source you collect and forward to your SIEM.
2. Map each log source to its ATT&CK Data Source and Data Component (reference the ATT&CK Data Sources page).
3. In ATT&CK Navigator, create a layer marking covered Data Components in cyan and uncovered ones unmarked.
4. The uncovered techniques are your gap list. Prioritize them by technique frequency (Red Canary top 10 from Chapter 2) and business impact.
5. For each gap, identify the log source needed and the implementation cost. Escalate as a security control deficiency.

Data Quality Assessment

Collecting logs is necessary. Quality makes them huntable. A log source with critical fields missing or truncated provides false confidence: you believe you can detect a technique because you "have the logs," but the missing field means your query will never return a result even if the attacker is active.

Data quality checklist to run before scheduling any hunt:

TELEMETRY QUALITY CHECKLIST
Run before scheduling any hunt sprint.

Endpoint - Windows:
[ ] Event 4688 includes full CommandLine field on target hosts?
    Test: query recent 4688 events and check CommandLine is populated.
[ ] Sysmon Event 1 includes ParentCommandLine field?
[ ] Sysmon Event 10 (ProcessAccess) enabled with lsass.exe rules?
    Test: confirm recent events for EventID 10 targeting lsass.exe exist.
[ ] PowerShell Script Block Logging (Event 4104) enabled via GPO?
    Test: run a test PowerShell command, verify 4104 event appears in SIEM.
[ ] Sysmon Event 22 (DNS Query) deployed on endpoint set?

Network:
[ ] DNS events include full query name (not truncated beyond 45 chars)?
[ ] Proxy logs include full URL, User-Agent, and response bytes?
[ ] Kerberos events (4768, 4769) collected from ALL Domain Controllers?
    Verify count of DCs forwarding events matches total DC count.

Identity / Cloud:
[ ] Azure AD Sign-in logs flowing to SIEM (if hybrid/cloud)?
[ ] M365 Unified Audit Log enabled and forwarding?
[ ] CloudTrail logs covering all regions and accounts (if AWS)?

Retention:
[ ] Log retention minimum 30 days in SIEM for all sources?
[ ] High-value sources (Security events, Sysmon) retained 90 days?

Quality Validation:
[ ] Run quality test query (see below) on 4688 CommandLine coverage.
[ ] Spot-check 5 random Sysmon Event 1 records: all fields populated?

A KQL query to validate that Event 4688 is actually including the CommandLine field across your endpoints:

// Validate Event 4688 CommandLine field coverage per host
// If CoveragePct is 0: "Include command line in process creation events" GPO
// setting is not enabled on that host.
// If CoveragePct is low (under 90%): partial GPO application or mixed configs.
SecurityEvent
| where EventID == 4688
| where TimeGenerated > ago(1h)
| summarize
    TotalEvents = count(),
    WithCommandLine = countif(isnotempty(CommandLine)),
    WithoutCommandLine = countif(isempty(CommandLine))
  by Computer
| extend CoveragePct = round(100.0 * WithCommandLine / TotalEvents, 1)
| where CoveragePct < 90   // Flag any host below 90% CommandLine coverage
| order by CoveragePct asc
// Output: hosts with incomplete CommandLine auditing.
// Remediation: apply GPO to the OU containing these computers.