Name: Stephen Hinck
Years hunting: 5
Favorite datasets: network logs (proxy, Bro, DNS, etc), process execution, and AV logs
Favorite hunting techniques: Stacking, kill chain analysis
Favorite tools: Command line utilities (grep, sed, awk), ELK stack, ELSA, FireEye TAP
Who are you?
My name is Stephen Hinck. I have over ten years’ experience in information technology and security, including incident response, security operations and threat hunting.
Why do you hunt and what is your experience hunting?
I started hunting about five years ago, before “Threat Hunting” became the buzzword it is today. At that time, it was more of an exercise in relaxation after a long week of alert review, incident response and strategic programs, than a structured program for finding malicious or risky activity ideally found in security operations today.
We knew that existing monitoring systems would not have the ability to find all malicious activity occurring in the network. Using information sourced from various appliances, we would filter and stack what data we had to identify anomalies to investigate further. Nothing about it was really structured, tracked, or optimized, but it laid the groundwork for my understanding of hunting and its importance in a mature security organization.
How would you define Threat Hunting?
Threat hunting is the act of finding malicious or risky activities that have not triggered an alert.
What hunting techniques, tools, and datasets do you use most frequently?
I am a big fan of the Sqrrl Hunting Loop. The formation of a hypothesis before beginning the investigation ensures a hunt analyst is focused on one specific issue or risk. Additionally, mapping these hypotheses to a phase (or phases) of the kill chain enables tracking of metrics to justify the hunt program and ensure detection coverage across the different phases.
The datasets used during a hunt depend on the hypothesis and kill chain phase involved. Hypotheses mapped to the command and control phase generally focus more on network logs, while the installation or actions on objectives phases may focus more on endpoint logs. It is important to ensure that the event data flowing into your hunt tool provides hunt value. The dataset should ideally provide the ability to identify specific activities in one or more phases of the kill chain and the hunt platform should enable pivoting between disparate log sources through similar fields.
Tooling is a huge topic and can range from built-in utilities (grep, sed, awk), open source utilities (ELK stack, ELSA), to enterprise tools such as FireEye’s Threat Analytics Platform (TAP), Sqrrl, or Splunk. I personally have the most experience with TAP. However, each organization’s unique requirements will impact the options during the tool selection process. For best results, ensure that your hunt platform of choice can return search results quickly and provides both the ability to filter out data based on parsed fields and the ability to visually stack results by parsed field to identify patterns.
What types of friendly intelligence are most useful for a hunter to have in an investigation?
There's an industry focus on threat intelligence, but friendly intelligence is frequently overlooked and forgotten, despite the critical role it plays in the IR process. Integrating organizational context provides incredible power by enabling hunt analysts to review all applicable information within a single platform.
For example, by tagging all servers based on their roles, a hunter can hypothesize that “all database-tier servers should only be accessed by application-tier servers” and begin the hunt directly from the hunt-platform, instead of looking to an asset database to pull a list of all database and application servers.
What general advice do you have for new Threat Hunters?
Understanding and baselining your environment is critical to the long-term success of your threat hunting program. Spend time both in your log management/hunt tool as well as working alongside your IT and security operations personnel to understand what “normal” looks like, where risks have been “accepted” (potentially enabling malicious activity) and asking questions about things you identify that appear out of place.
What hunting procedure would you reccomend for a new threat hunter?
Pick a protocol that you have a solid familiarity with, including what information exists within that protocol and should be logged. For example, in HTTP logs, we know there should be a domain, URI, user-agent, method, and other important fields. Start by stacking those fields to look for oddities, for example, only a small set of certain terms should appear in the HTTP method field – any deviation from that list may be cause for further investigation.
Analyst understanding the available dataset in conjunction with the original activity and systems that generated that data is critical to success in your hunts.
What parts of a hunt could you see as being most successfully automated or assisted by a machine?
When it comes to the actual identification of malicious or risky behaviors, automation falls more into the realm of signature development than hunting. Additionally, automation can provide great starting points for your hunts where the outputs of the automation don’t have the accuracy to go directly to the alert monitoring team (e.g. geoinfeasibility). That said, automation of information collection can save large amounts of time for the hunter. For example, if your hunt tool can automatically tag your events or components of your events with contextual metadata, such as GeoIP, WHOIS, or asset information upon ingestion, the immediate availability of this data for the hunter within the tool will greatly increase the speed at which they can hunt.
What would you like to see Threat Hunting develop into across the industry in the future?
I am a fan of information sharing. I believe that by sharing our knowledge and experience with others, we can improve as a whole. For that reason, I am very excited by projects such as threathunting.net, designed as places to share with others just getting started in the space. I would like to see this continue in the security industry.
Additionally, I think that Threat Hunting has become too much of a marketing buzzword, and with that, some of its meaning as a methodology is lost. You cannot buy a silver bullet to do “Threat Hunting” for you – it is a practice carried out by analysts to identify that which tools cannot. I would like to see the industry progress past the buzzword phase and towards the implementation of hunting programs as a base requirement of a typical security program. The excitement around Threat Hunting is similar to the buzz several years ago around “malware detonation,” and its subsequent transition to become a standardized practice as a part of an organization’s risk mitigation strategy.