Name: Jason Smith
Years hunting: 6
Favorite datasets: Flow data, Bro logs (http, dns, etc.), Windows event logs
Favorite hunting techniques: Pivoting from statistical anomalies, behavioral deviations for local assets
Favorite tools: SiLK, FlowBAT, Bro, Security Onion, Wireshark, Bash
Who are you?
My name is Jason Smith and I currently work for Cisco from my home in Nashville, TN.
I have a background in physics and have built everything from particle accelerators to explosive neutralizing robots used by the military. I’ve worked in multiple US Department of Defense SOCs and was the lead security monitoring architect for the Commonwealth of Kentucky. I co-wrote Applied Network Security Monitoring and maintain the open source project FlowBAT, a graphical flow data analysis tool.
Why do you hunt and what is your experience hunting?
I've been threat hunting since working in my very first SOC position many years ago. Different people come from different experiences, and I was lucky enough to have real data to cut my teeth on in my early days of doing NSM. Not everyone is so lucky. More importantly, I had excellent mentors that brought all analysts up on the idea that hunting is essentially the job itself.
There is a negative mentality that some people have about SOC work, but one that is completely understandable if the only experience they have is making tickets based on alerts with no context. The SOCs that I’ve worked in and helped to build have been created around the idea that context and additional data should be accessible in a fluid nature. That kind of environment is conducive to successful hunting.
How would you define Threat Hunting?
"Threat Hunting” is human driven data analysis where an alert isn’t the impetus and where the primary focus is to find bad guys. This implies that if you only have alerts coming to a dashboard and you have no access to other data, you probably can’t be very successful in your hunting endeavors. Hunting was the first human activity in general, and the same can be said for the beginnings of NSM. Intrusion detection didn’t originate with premade tools and alerts, but instead it was an individual making sense of what data they had. Over time we made an automated means of discovering baddies based on indicators that we generated signatures out of. However, no one stopped there since it is that discovery that leads to being able to further automate. Threat hunting isn’t a thing that only the elite do, but instead it is the thing that any decent analyst does between handling alerts as long as he/she has the right data and means to search through it.
What projects and organizations are you involved with right now?
I maintain FlowBAT and FlowPlotter, two open-source tools that help analysts make sense of flow data. Simply put, FlowBAT is a graphical front end to SiLK. However, FlowBAT enables the analyst to parse through flow data in a much more streamlined and fluid way while also seriously killing the learning curve involved with typical flow tools. The best thing is that the entire solution can be automatically deployed in 10 minutes. Check it out at http://www.flowbat.com.
Which of the hunts you’ve carried out was the most interesting or challenging?
Hunts come in a 3 categories for me. Hunts can be boring where you have insufficient data, or the data is simply not exciting and leads to nothing. Hunts can be disappointing in that you go hunting down a rabbit hole only to discover that the baddie you are trying to discover is actually nothing at all. Hunts can be exhilarating in that when you are successful it almost certainly means you’ve either discovered a new indicator that you can add to your automation, or you have discovered a baddie and feel like Sherlock Holmes.
What hunting techniques, tools, and datasets do you use most frequently?
Although I consider hunting to be a human driven endeavor, sometimes the catalyst might be an alert or advisory that results in a tangential path to discovery. Some of the best detection tools are those that allow you to be notified of “weirdness”, and half of the successful hunting exercises end in additional “weirdness detection” automation.
For me, any tool that allows for fluid movement around data is a great tool for hunting. I think most hunters really focus on a number of data types. For instance, the reason I like flows so much is because they are an incredibly lightweight means of discovering where and when I should pivot between data types. Flows by themselves are hard to peruse for the average user, so I use something like SiLK (for automating searches) or FlowBAT (for manual searches) to discover where and when a host might have talked.
Once the flows give me the where and when, I can drill through some bro logs to get additional context around the “what” of the exercise. Knowing what is happening requires actual context, and that means that your tool of choice needs to always have enough context available. Bro is infinitely capable as far as automating goes for the skilled practitioner, but even out of the box it is extremely useful from a log perspective and usually has the context I need.
On occasion both flows and bro tell me that something weird is going on, but I still don’t have the full picture. In that case I might drill down further on who, what, when, and where with raw pcap analysis. Raw pcap alone is too cumbersome to parse through unless you have direction. In this case, flows provided a time, bro provided context into what might be happening, and the pcap (that is now easily selected) can provide the full picture with some additional work.
What types of friendly intelligence are most useful for a hunter to have in an investigation?
There are too few people that understand what is happening in their network. Statistical data can lead to friendly intelligence that is mostly unmatched by other data types. Statistical analysis of unsampled flows is ideal for gaining an understanding of what is happening on the network, and many free flow tools (FlowBAT/SiLK, Nfsen/Nfdump, Argus) allow stat collection. Using these results within other tools (especially those that allow for asset tagging) will greatly help during the next hunt. Again, this all comes down to the amount of data you have and the fluidity by which it allows you to move around it.
What general advice do you have for new Threat Hunters?
Don’t think that hunting is some sort of activity that only the NSM “elite” do. If you have accessible raw data, jump in it. Check for user agent weirdness, check for bottom-n stats on domains/users/etc, check for bizarre things like posts with no referrer. Most importantly, keep a good notebook. Once you fill up that notebook, get another one. There is nothing more important than keeping track of your steps, and also keeping track of your methods. Also, if you discover a new method, don’t assume that everyone else already uses it. Share it!
What hunting procedure would you reccomend for a new threat hunter?
Get to know the data you have, and understand where you can expand visibility efficiently. The best way to do this is to train with data where you already know the outcome. A quick way to do this is to set up something like Security Onion and churn some PCAPs through it with tcpreplay. These PCAPs don't need to necessarily be malicious, but instead they can be as simple as a capture of 1-2 minutes of regular browsing to your favorite sites. Start out by understanding some of the tools of the trades and the logs that they create, then you can move on to more challenging analysis. Once you are comfortable in your environment, you will be comfortable (and more successful) in your hunting exercises.