USE CASE WHITEPAPER

How to Build
an Effective Threat Hunting Program
By Deploying a Local Threat Intelligence Infrastructure

Introduction
A common complaint among threat intelligence analysts is the near impossibility of searching global
threat intelligence feeds to find the specific threat and vulnerability information that matters to their
organization. This complaint is just the tip of the iceberg. The larger, underlying problem facing security
teams is the general lack of visibility and context across all internal files and objects that enter and then
move about, stay resident, and or leave their corporate environments.

Having more visibility and greater understanding of these files and objects would add critical
decision-making context to internal risks, vulnerabilities, and threats. It would also make global threat
intelligence more actionable. This lack of context presents itself to SOC teams as an absence of
in-depth knowledge about incidents, and to hunting teams as too few clues to build-out useful hunting
hypotheses. This paper will discuss in detail a path to closing the visibility gap through a unique, new
local threat intelligence infrastructure.

The Undetected Malware Visibility Gap

At its core, the problem is the lack of visibility into and understanding of the millions of objects that
move into and around an organization’s network every day. The problem extends to the insufficient
visibility of the transactions associated with those objects. These files, executables, code, photos, flash
videos, embedded scripts and other types of objects permeate a company’s network by the thousands.
A security team’s lack of knowledge about them makes it nearly impossible to identify relevant external

Expected Detection Rate Malware Detection Gap

Real Detection Rate

TIME
 threat intelligence that they can then use to identify new threats against their organization,
address undetected malware, and develop effective defenses against the new threats. No
matter how good a team’s cybersecurity detection and defenses are, some malware is
What if information
could be captured in a going to slip into their environment unseen. No organization can hit the magic 100%
rich and in-depth detection rate. Adversaries know this and know modified (polymorphic), or zero-day
manner covering all attacks will get through sooner or later. For this reason, hunt teams must continuously try
objects that an to find and contain undetected malware that has bypassed their defenses. However, the
organization is seeing? lack of knowledge about objects inside an organization’s network makes finding unknown
malware tremendously difficult. For attacks that have occurred, critical information
relating to the malicious objects was likely overlooked, dropped, or misclassified.
Attempting to find these objects after the fact, is ‘a bridge too far’ for even the most
advanced hunt and response teams. Security teams have instrumented many different
systems to gather information about what is happening in their environments. These
systems gather information from endpoints, networks, email servers, etc. and send it to
SIEMs or data lakes for correlation, alerts, and response. The problem is that the
information collected lacks depth and provides only a cursory snapshot of a select few
observables, which are limited by the various collecting technologies. For example, an
endpoint solution is limited to a specific set of executables, usually identifying only objects
that may have executed. Other objects would be passed over, neither noted nor analyzed.
Attempting to gain visibility at the object level, security teams often turn to dynamic
analysis (sandboxing), and global threat intelligence feeds. When capturing information
from sandboxes, however, the information is limited to objects that can be detonated (such
as Windows files), and that exhibit behaviors deemed worthy of collection. Evasive objects,
unsupported platforms, or statically embedded content is not captured. Also, dynamic
analysis can take minutes to execute a file so there is a practical limit as to how many files
can be analyzed in a given period.

Let’s not forget the Threat Intelligence and Threat Hunting Missions

Collect the Create an

Right Data Early Warning System

globally and internally to enable early warning that can match global intelligence with local evidence

RECONNAISSANCE WEAPONIZATION DELIVERY EXPLOIT INSTALLATION COMMAND ACTIONS

Global threat intelligence offers visibility of threats that exist beyond an organizations’ four
walls and provides ample context for thousands of threats. However, determining which
ones are relevant to a specific organization at any point in time involves guesswork. For
example, what if global threat intelligence reports information on a newly discovered
attack that may have already entered an organization’s network days or weeks earlier?
That team would have no reliable method for retrospectively determining the potential
relevance of that threat, and if relevant, finding and containing it. It is often difficult to
determine what is relevant and what is noise.
 Why Local Threat Intelligence Matters
If a security team could implement an infrastructure to automatically analyze, classify, and store
metadata for each of the objects that move in and around their network, it would close the file/object
visibility gap. The body of information created would provide a searchable knowledgebase of the objects
within an organization. That capability would be invaluable for recognizing relevant external threat
intelligence, supporting incident response, and combatting malware that evades defenses. The threat
hunting team would have comprehensive knowledge of local files and objects to compare against
global intelligence to help define which global data is relevant for their organization.

The SOC team would have the ability to understand the files and malware that underlie event
information being collected in the SIEM to tie together attack events and respond more effectively. Hunt
teams would have better visibility to identify and correlate previously undetected internal threats. This
information could be shared locally or with industry peers to enable collaboration and accelerate
decision making. If metadata and classification information could be captured and indexed for all
objects within an organization’s network, this information could be used to retrospectively hunt, identify
and correlate previously undetected malware. Analysts and threat hunters could then analyze all files
using a standard methodology, regardless of operating system or file type. Analysts would be able to
search using relevant attributes (e.g., hashes, strings, behavior attributes, similarity, etc.) to identify
unknown, unwanted content. From there, because all local file context is known, analysts could assess
the causes and the true extent of any given campaign. Intel, SOC and hunt teams would thus have highly
valuable and relevant locally derived knowledge to work with.

This locally collected information (Local Threat Intelligence) would give security teams a precise way to
match global threat intelligence and suspicious activity to unknown malware lurking inside their
organizations. It also would enable security operations teams to connect incidents,
analytics and objects to paint a complete picture of an emerging attack,
improving the speed, accuracy, and overall effectiveness
of the entire threat detection and ANALYTICS PLATFORM
response process.

Analysis of All
Detonation of Files & Objects
User Behavior Select Object
Data (Who?
IDS Logs When? What?)
(Protocol
Server Logs information)
(Logins, etc.)
SIEM
Endpoint Data
(Logins, etc.)
Local Threat Information
Netflow Data
(Network Traffic/DNS) Collection Maturity Model
 Components of a Local Threat Infrastructure
Data Inputs – To create a complete body of local threat intelligence information, the infrastructure
should collect from as many sources as possible, including the network, endpoints, email, storage, and
file transfers. Most companies will start with a limited data set and extend to new sources over time.
Initial data sources feeding into the infrastructure can be based on where the company perceives the
greatest risks or threats. High Volume Object Analysis – The data inputs are fed into a high-volume
automated static analysis engine. Static analysis has the advantage of reverse engineering and
analyzing objects in less than a second, providing the performance and scalability required to cover an
extremely large body of moving files. The output of this initial analysis is to gather hash and metadata
on every file and to classify each file as known good, known bad or unknown. These factors enable a
calculation of the risk level of the file based on its internal attributes

SOC
Object Analysis
Web Private Data Lake Analytics
[Splunk] Hunting
Email Scalable, high volume [Hadoop] Malware Analysis
analysis infrastructure [Elastic Search] Visualization
Endpoint for files, docments,
transactions
Storage+CASB

SAP Private File Lake YARA Rules

2 Years + Sharing Portal
SWIFT S3 Object Store
Cloudera

Local Threat Intelligence Infrastructure

Data Lake – A file’s hash, context, and detailed metadata collected in the object analysis phase is
stored in a central data store, either the SIEM or a security data lake. Because object analysis indexes
the context, classification and metadata on all the collected files, the Data Lake becomes the key for all
hunting activities in the File Lake.

File Lake – A big data storage source where all bad and unknown files can be safely kept for deeper
analysis or retro-search. For companies struggling to find value in big data deployments, the creation of
a data lake focused on collecting all “files of interest” offers an immediate and high value use. This data
lake becomes the local file intelligence repository for all analysis, threat hunting, and retrospective
hunting.

SIEM/Analytics – As the central clearinghouse for gathering events and information from security
tools and other logs, the SIEM is an integral part of the local threat intelligence infrastructure. In such
infrastructures, the SIEM’s role remains the same. However, analysts working with the SIEM will have a
much richer event, context and file metadata to use while making risk and response decisions. More
importantly, analysts will have the tools and processes they need to match suspect files and objects to
high-risk events and know what to do when an unknown and potentially damaging file is surfaced by an
event. But be aware of the limitations of many older SIEM platforms. 20 character description limited on
events is still the norm of many deployed SIEM environments and SOC teams limited by these
environments will need to look at modern and complimentary analytic tools.
 Malware Analysis Workbench – This component of the infrastructure is the primary investigative and
hunting tool for malware and is made possible by new automated static analysis technology. The addition
of this deep, static-analysis stage is a change from existing security processes which would send these
files to a sandbox environment for dynamic analysis. With their speed and comprehensiveness,
next-generation static analysis engines increase the accuracy and efficiency of unknown and high-risk file
definition, malware detection and response. This capability enables level one analysts to add value in the
investigation stage because new static analysis technology automatically identifies threat indicators. And
by quickly identifying false positives, already known bad malware, and capturing all information needed to
respond to unknown malware, the analysis workbench greatly reduces the number of files needing
dynamic analysis. Once malware is identified, contextual information for it is then sent to orchestration
and response tools for containment. If analysts want a deeper understanding of the malware, it can be
sent on to the sandbox. Doing so provides an added benefit, as static analysis will surface the deception
techniques malware used to evade dynamic analysis so that sandbox environments can be designed for
full dynamic capture.

Hunting in the Malware Analysis Workbench – The second use of the malware analysis workbench is
for hunting. Threat hunters can utilize the workbench for multi-conditional queries using logical
expressions and YARA rules to search through data stores and uncover hidden malware. Over time, with a
local threat intelligence infrastructure in place, threat hunters can utilize YARA rules to traverse large
historical data sets, greatly enhancing detection and reducing impact from breaches and newly identified
targeted attacks.

Global Reputation – A final critical piece of a local threat infrastructure is integration with a file
reputation service. An integrated file reputation service ensures that known good and known bad files are
quickly identified to reduce false positives and improve process efficiency. Services with larger datasets
that are curated regularly to ensure the most up-to-date threat data is included should be prioritized.

With
All threats have attack THE MISSING
components using LINK FOR COMPLETE MIGRATION TO CLOUD
EMBEDDED THREAT INTELLIGENCE threat intelligence is about
FILES AND OBJECTS
FILES & OBJECTS Global file reputation attached to
every file and object to identify Enable binary search of
whether on network, endpoint unknown threats & sensitive locally relevant and globally
or storage content unknown threats
 ReversingLabs Local Threat
Intelligence Infrastructure
ReversingLabs delivers powerful stand-alone solutions that individually provide significant
value across file reputation services, malware analysis, and malware hunting use cases.
However, when these solutions are used in concert, they form the foundation for a
comprehensive local threat intelligence infrastructure that easily extends and integrates
with an organization’s current security products.

“In today’s threat
environment it is critical to
deploy an internal
infrastructure that can
find, monitor, examine and
contain all files, objects,
and transactions that are
relevant to the enterprise’s
In building a local threat intelligence infrastructure, an organization would implement
ReversingLabs solutions to generate and store detailed information on files in data and file
lakes, making it accessible for search and analytics tools. The infrastructure would
integrate with SIEM solutions for real-time alerting, investigations and response. Each
standalone solution delivers a critical piece of the local threat intelligence infrastructure
and empowers organizations to accelerate and expand their ability to use external threat
intelligence to identify undetected malware and response to incidents. These solutions

well-being.” include:

• Enterprise-scale file analysis - TitaniumScale

• Malware analysis and hunting – A1000

• File reputation and threat intelligence - TitaniumCloud

File
Intelligence
Service

Enterprise
PRIVATE
Scale File DATA
Analysis LAKE

PRIVATE
FILE
LAKE

Analyze all Monitor Search &

Malware
Unknown Disposition Retrospective Analysis
Files Change Hunting
& Hunting

TitaniumCore
ReversingLabs
Automated Static
Analysis Engine
A cornerstone to ReversingLabs
solutions is Automate Static Analysis,
which is the only technology that can
process millions of files daily to
provide in-depth data for local threat
intelligence. Automated static
analysis technology recursively
unpacks and extracts detailed
metadata and hundreds of internal
threat indicators. It then calculates
the threat level of the object and adds
global reputational context to support
real-time or high-volume processing.
Because automated static analysis
does not require the file to be
executed, a detailed analysis is
performed in milliseconds to enable
processing of millions of files daily.
TitaniumCore’s rules engine
calculates threat levels based on
rules provided by ReversingLabs, and
YARA rules supplied by the customer.
No other product (e.g., sandboxes or
scanners) exposes the breadth and
depth of threat indicators extracted
by TitaniumCore. The TitaniumCore
engine also includes ReversingLabs’
Functional Hashing Algorithm (RHA)
to calculate functional similarity to
known malware.
Enterprise Scale File Analysis
Enterprise-scale file analysis is delivered through the company’s TitaniumScale solution.
TitaniumScale enables an organization to profile and classify large volumes of objects in
near real-time to create relevant data for use by advanced analytic platforms for threat
correlations, hunting, and response efforts.
TitaniumScale helps enterprises perform a comprehensive assessment of millions of files
from web traffic, email, file transfers, endpoints, and storage. It acquires objects by
integrating with existing security infrastructure, including email gateways, intrusion
detections systems, firewalls, and other devices. The results feed into SIEM, orchestration
and analytics platforms. The metadata created about each object is stored in a data lake,
and many deployments can include a dedicated file lake to store all files classified as high
risk or unknown.
MALWARE ANALYSIS & HUNTING
The ReversingLabs A1000 appliance provides unparalleled malware analysis and hunting.
It supports advanced hunting and investigations through high-speed automated static
analysis driven by the unique ReversingLabs TitaniumCore engine. The A1000 supports
advanced threat visualizations, APIs for integration with SIEMS, data lakes, and
automated workflows, a dedicated database for malware storage and search, global and
local YARA Rules, as well as integration with third-party sandbox tools. The A1000
accelerates analysis for users at all levels, from first-line helpdesk staff to senior team
members on an analyst workbench. It assesses malware and status changes as malware
families morph over time via obfuscation and other techniques.
Fully integrated with ReversingLabs file reputation services, it provides forensic analysts
and hunters with rich, in-depth context and threat classification on over seven billion files
of all types and platforms. This integration means that all analysis work is done entirely
onsite. No sensitive documents ever leave the organization, yet analysts have access to
the world’s largest authoritative source for file reputation to speed their investigations.
The implementation of a file lake with the A1000 covers a broad set of use cases that
includes detection of new or changing malware that has successfully entered a network,
and expanded historic search and analysis of past file-based attacks.
FILE REPUTATION & THREAT INTELLIGENCE
ReversingLabs’ TitaniumCloud Reputation Services are powerful threat intelligence
solutions with up-to-date threat classification and rich context on over seven billion
goodware and malware files. ReversingLabs does not depend on crowd-sourced collection
but instead curates the harvesting of files from software vendors and diverse malware
sources. All files are processed using the advanced capabilities of TitaniumCore’s static
analysis, combined with dynamic detection and AV engine analysis, to provide industry
reputation consensus. TitaniumCloud supports a robust set of API query and feed
functions that deliver targeted file and malware intelligence for threat identification,
analysis, intelligence development, and hunting.
 Fitting the Pieces in Place

SECURITY INFRASTRUCTURE
One of the most significant challenges facing any security team is the integration of deployed security
software and appliances to detect, manage, respond and prevent cyber-attacks. In building a local file
intelligence infrastructure, integration with existing systems is critical. ReversingLabs offers extensive
pre-built integrations and a complete set of APIs to connect even the most sophisticated security
infrastructures. These integrations allow for analysis of objects flowing into the ReversingLabs platform
and enhance existing detection, SIEM and SOAR tools with enriched file and malware context.

MANAGED ENDPOINT
NETWORK SECURITY
DETECTION DETECTION & SIEM
DETECTION ANALYTICS
& RESPONSE RESPONSE

SECURITY THREAT
IDENTITY & EMAIL / WEB /
ORCHESTRATION SANDBOX INTELLIGENCE
ACCESS CLOUD
& RESPONSE

FILE REPUTATION ENTERPRISE SCALE MALWARE

& MALWARE FILE ANALYSIS & ANALYSIS &
INTELLIGENCE CLASSIFICATION HUNTING

REVERSINGLABS SOLUTIONS

Local Threat Intelligence & Advanced Malware Hunting Integration Categories

DATA INFRASTRUCTURE
Security data lake deployments are increasingly popular due to the comprehensive visibility they provide
for investigation, hunting, and forensic analysis. They store massive amounts of security event and
file-related data from different security system logs and outputs. A major challenge of operationalizing a
security data lake is collecting, parsing and enriching the data so that it is relevant for search and hunt
projects. This is especially true when security teams are working with logs and data from many
different types of security tools. By utilizing TitaniumScale to analyze large volumes of inbound content,
the resulting object identification, metadata, and classification information for each file is indexed and
parsed into the data lake. When investigators and hunters utilize the advanced search or YARA rules
interface, they can accurately query, and surface target files and malware based on the analysis work
completed by TitaniumScale. Over time, a data lake will collect a rich historic set of local threat
intelligence. The A1000 includes tailor-made retro-search capabilities to perform a historic search of
files for newly uncovered malware.
 Conclusion: A Vision of the Future
Envision a security organization with a local threat intelligence infrastructure in place. In this organization,
when law enforcement or regulators request searches for malware with specific characteristics, an
analyst or threat hunter can write a rule, quickly search through a data lake and file lake, and find specific
suspect files. When other security tools provide hints about high-risk files, rules can be created that, in
turn, enable a speedy search of the entire local data lake or file lake to reveal all locally discovered files
that the rule describes.

The same rules can be used to bolster defenses in near real-time to detect that specific malware or
functionally similar malware as it enters the organization’s network. Further, imagine a spike in
ransomware attacks that utilize a new backdoor variant delivered through certain PEs on Win OS, and an
organization’s threat intelligence feeds have samples or additional information on it. In this organization,
analysts can quickly query the local threat intelligence data to see if they have exposure to this attack. If
there is exposure, analysts can write YARA rules based on discovered samples to search out and find all
variants of the malware to isolate and contain it. The technology makes subsequent investigations far
more focused, efficient and productive.

A successful local threat intelligence program provides critical information on events and objects that are
touching your organization. Being aware of what happens elsewhere is helpful, but nowhere near as
relevant as what is taking place inside your organization.

Effective Threat Hunting Components

Local
Threat
Intelligence Global
COLLECTION FILE REPUTATION
INFRASTRUCTURE

Advanced Binary
Search Search
THROUGH ALL LOCAL INFRASTRUCTURE
AND GLOBAL FILE THROUGH ALL LOCAL
REPUTATION DATA AND GLOBAL
CONTENT
 Local file intelligence, created by automated static analysis, stored in a data lake and a file lake, and
indexed so that it can be utilized for threat hunting makes global intelligence feeds truly actionable.
It also creates visibility to file-level. It also creates visibility to file-level threats that no other system
is capable of, and it acts as a force-multiplier for all other systems in your security infrastructure.

Future proof discovery Having all relevant

information Effective Sharing

LOCAL THREAT
LOCAL+GLOBAL YARA SEARCH AND
INTELLIGENCE
TEXT/BINARY RETRO-SEARCH
WITH STATIC FILE
SEARCH
ANALYSIS

Missing data prevents Match external indicators Regulators request that you
identification of threats in and rules against content look for malware with
EARLY STAGES of the kill seen locally. specific characteristics.
chain.
Missing data prevents
Identify whether threats Write a rule, search through
CHANGE ALERTING when
apply to you. Monitor a data lake and find the
zero day threats get
threat progression suspect evidence.
identified.
globally.

© Copyright 2018. ReversingLabs. All rights reserved. ReversingLabs is the registered trademark
of ReversingLabs US Inc. All other product and company names mentioned are trademarks or +1.617.250.7518
registered trademarks of their respective owners. 2018 December.
sales@reversinglabs.com

WHITE PAPER ReversingLabs