Evidence Quality and Scoring

Why Evidence Quality Matters

Not all evidence is equal. A hash match from a single threat intel feed is not the same confidence level as an analyst-verified behavior chain in EDR spanning three hosts and two accounts. Treating them identically leads to one of two failure modes.

Over-confidence: treating weak evidence as confirmed. The result is wasted IR resources, analyst credibility loss, and alert fatigue. Under-confidence: dismissing strong evidence because it seems "too simple." The result is a dwell time measured in months while an analyst waits for "more evidence."

Evidence quality affects every downstream decision: whether to escalate to IR, what to report to management, how much analyst time to allocate for follow-up, and what priority to assign detection rule creation. Without consistent quality scoring, each analyst makes these decisions differently. The team's output becomes inconsistent and unreproducible.

The Admiralty System

The Admiralty System originates from NATO AJP-2.1, with roots in World War II British Royal Navy intelligence evaluation. It is now used by Unit 42, SANS, and mature CTI and hunt teams for rating the quality of intelligence and evidence. The system uses two independent dimensions rated separately.

Source Reliability (A through F)

Rates the track record and credibility of where the information came from, not the information itself.

Information Credibility (1 through 6)

Rates the information itself, independent of where it came from.

Critical rule: rate source reliability and information credibility independently. A completely reliable source (A) can provide doubtful information (4). An unreliable source (E) occasionally provides confirmed information (1) that you independently verify. The ratings are not correlated. Do not let a trusted source's reputation pull up your credibility rating for information that has not been verified.

The Diamond Model

Developed in 2013 by Caltagirone, Pendergast, and Betz, the Diamond Model is a foundational framework for structuring adversary analysis and hunt pivots. Unit 42 uses it as the backbone of their attribution methodology.

Four vertices define every intrusion event:

• Adversary: The threat actor. Named group, suspected nation-state, financially motivated criminal, or insider. May be unknown at hunt time.
• Capability: What the adversary used. PowerShell, Cobalt Strike, CVE exploitation, LOLBins, custom malware. This is the "how."
• Infrastructure: The systems used to conduct the attack. C2 servers, compromised relay hosts, domains, IP addresses. This is the "from where."
• Victim: The target. The organization, specific systems, specific users, or specific data.

The model's hunting value: knowing any vertex, you can hypothesize the others and pivot toward them using available telemetry.

The Pyramid of Pain

Created by David Bianco in 2013, the Pyramid of Pain describes the cost to an adversary when each indicator type is detected and blocked. The higher the level, the more difficult it is for the adversary to adapt. The implication for hunters: the higher you hunt on the pyramid, the more durable and effective your hunt becomes.

          PYRAMID OF PAIN
  ┌─────────────────────────────────┐
  │             TTPs                │  <-- Highest adversary pain
  ├─────────────────────────────────┤
  │             Tools               │
  ├─────────────────────────────────┤
  │      Network/Host Artifacts     │
  ├─────────────────────────────────┤
  │          Domain Names           │
  ├─────────────────────────────────┤
  │           IP Addresses          │
  ├─────────────────────────────────┤
  │           Hash Values           │  <-- Lowest adversary pain
  └─────────────────────────────────┘

Hunting implication: hunts based on hash values are the most brittle. The adversary rotates before your hunt finds anything. Hunts based on TTPs are the most durable. The adversary cannot easily change how they move laterally or how they dump credentials without retraining their entire operation. HYPOS at h3ad-sec.github.io/HYPOS/ generates TTP-level hypotheses specifically for this reason.

Combining Pyramid of Pain with Admiralty System: a hash-based finding is inherently low-confidence and low-durability. Hashes are easily faked or reused across completely unrelated malware families. A behavioral TTP finding confirmed in EDR with multiple corroborating data points is high-confidence and high-durability. Score them differently and allocate analyst time accordingly.

Confidence and Severity Scoring

The combined scoring approach used by commercial hunt teams at Red Canary, Elastic, and Tidal Cyber separates two distinct dimensions that are often conflated: how certain you are that the finding is malicious (confidence), and how bad it would be if it is real (severity).

Confidence (0-100): How certain are you the finding is malicious? Driven by evidence quality (Admiralty ratings), corroboration from multiple independent data sources, and consistency with known actor TTPs. A finding from a single OSINT feed with no corroboration scores low. A finding confirmed in EDR memory analysis, corroborated by network logs and authentication events, scores high.

Severity: If this is real, how bad is it? Driven by technique impact (credential dumping on a domain controller is worse than a single file creation on a standard workstation), affected assets, and blast radius (1 host vs. lateral movement across 20 hosts).

Five evidence factors that raise confidence:

• Multiple independent data sources confirm the same behavior (endpoint, network, identity all agree).
• Behavior matches a known actor TTP profile documented in recent threat reports.
• Anomalous timing or pattern with no legitimate business explanation identified after checking with asset owners.
• Behavior observed across multiple hosts, not isolated to a single endpoint.
• Host artifact or network artifact confirmed in EDR memory analysis, not just log-level evidence.

Five evidence factors that lower confidence:

• Single data source only, no corroborating evidence from other log types.
• The behavior matches a known admin tool or software management agent pattern.
• Activity matches a documented false positive pattern in your environment's baseline.
• Activity is confined to a single host with no lateral or network indicators.
• The finding comes from an F-rated source with no additional corroboration.

False Positive Management

False positives are inevitable. A hunt query broad enough to catch novel threats will also catch legitimate activity. The goal is not to eliminate false positives. The goal is to handle them efficiently so they do not consume analyst time repeatedly.

Three Categories of False Positives

Known-good baselines: IT admin tools, software management agents (SCCM, Intune, Puppet), backup software, security scanners. Document them once. Filter them consistently in every hunt query that would otherwise surface them. These should never consume analyst review time twice.

Environment-specific behavior: An application that does something technically suspicious but is legitimate in your environment. A manufacturing ERP system that creates scheduled tasks via schtasks.exe. A backup agent that reads the LSASS process. These require context from asset owners and must be documented with that context so the next analyst does not re-investigate from scratch.

True negatives misclassified: The behavior is genuinely suspicious, but investigation confirms it is legitimate. A developer running Mimikatz in an authorized penetration test on their own machine. Update the knowledge base with the outcome. The next instance of that hash will be immediately recognized.

FP Triage Checklist

Before classifying any result as a false positive, work through five checks:

1. Check the process hash against your known-good software inventory. Is this a recognized binary from a managed application?
2. Check the parent process. Does the parent process make sense for this action, or is it an anomalous launcher?
3. Check the user account. Is this account expected to perform this action on this host at this time?
4. Check timing. Is this during business hours, a known maintenance window, or an anomalous time (3 AM on a standard workstation)?
5. Check asset context. Is this a server, a developer machine, a privileged admin workstation, or a standard user endpoint? The same behavior means different things on different assets.

Suppression vs. filtering: suppress known-good baselines in your hunt query using exclusion filters (faster, less noise in results). Apply post-result filtering during the review phase for edge cases where suppression would be too broad. Never suppress without documenting exactly why the suppression was added and which analyst added it.