Name: James Bower
Organization: Quantum Security
Years hunting: 10
Favorite datasets: Bro logs, DNS, HTTP proxy logs, Sysmon and OSSEC
Favorite hunting techniques: Outbound traffic analysis, Stacking (outlier detection), process and registry change analysis, temporal baselining
Favorite tools: Bro, Unix commands (grep, sed, awk), TShark, Splunk
Who are you?
My name is James Bower, and I’m a threat researcher and hunter for Quantum Security here in Atlanta. I currently advise and consult on threat hunting and red/blue team services for a handful of companies in metro Atlanta.
I’ve been consulting in InfoSec for over a decade now and have been hunting in some form or another throughout my career. I’ve been fortunate enough to be able to spend time on both sides of the InfoSec coin from NSM to leading Red Teams engagements.
I also try to give back to the community and share the things that I’ve learned at https://www.jamesbower.com.
Why do you hunt and what is your experience hunting?
I hunt because it scratches that curiosity itch in my brain like few things can. For me, threat hunting isn’t a red team or blue team exercise, rather a perfect “grey team” line between the two that allows me to use defensive tools while learning and developing offensive techniques. Having worked on both sides it’s really the perfect blend for me. From a business prospective, Threat Hunting allows me to provide clients with actionable intelligence and creates the highest ROI in maturing their security posture in the shortest time possible.
My first real “hunt” was a total fluke in the early 2000’s and began because I’m a big data nerd. At the time, I had access to a large network and decided to spend a Friday night plotting the bandwidth utilization of multiple T1’s to find the best route to download _________. I was also single at the time if you can believe that. It was pretty standard stuff until I compared the daily utilization going back several months which showed a huge bandwidth spike from around 10pm Friday – 6am Sunday for the last several weekends. This led me to discovering my first botnet.
After consuming enough Mountain Dew to kill a small pony, I grabbed the only tools I had access to on the network which were TCPDUMP and the Windows command line along with a notebook. Fast forward to Sunday morning I had learned that the botnet was being used for all sorts things including pirated movies and PlayStation2 games. I found the registry keys along with the .bat config file for the C&C that contained the hardcoded IRC channel and login creds.
The channel had over 9K bots which, at the time, was considered to be really large. Of course, not knowing the naming scheme I was immediately greeted and booted by the op. I kept logging back in over the next few hours and eventually made peace with the op. He had been a “scanner” who had been recently promoted to “operator” and went on to school me on the complete workings and hierarchy of a Russian organized cybercrime gang, circa early 2000’s. I’ve been hunting ever since.
How do you define Threat Hunting?
For me, threat hunting is a proactive exercise looking for malicious activity as opposed to a responsive approach based off an alert or event. The end result also needs to be actionable, whether that’s changes to techniques, policies, or procedures, and should increase the efficiency and effectiveness of your detection.
What projects and organizations are you involved with?
The project that I’m most excited about right now is “Engaged Threat”. The idea is to create a threat analytics application that focuses on correlating and comparing attacker user behavior in honeypots as opposed to the static indicators you generally see. The end goal is to increasingly engage attackers to obtain higher quality threat intelligence. In a nut shell, I’ve taken the same concepts, such as funneling, that ecommerce websites use to drive sales and direct customer purchasing and begun applying it to attackers in honeypots.
I’m also the founder of the Threat Intelligence & Threat Hunting Meetup here in Atlanta. So far we’ve got about 150 members and I’m really excited for some of our upcoming meetings taking place this summer.
Which of the hunts you've carried out was the most interesting or challenging?
It’s hard for me to pick out a favorite hunt, besides my first I described before, as I tend to learn something new every time, which is what I really love. A memorable one was a compromised ad server being used as a command and control which hid the malicious HTTP requests with legitimate requests really well. That hunt taught me the importance of trusting my intuition when I all I had was a feeling that something was off with those requests.
What hunting techniques, tools, and datasets do you use most frequently?
Because of my Red Teaming and consulting background, I’m meticulous when it comes to tracking the different aspects of my hunting.
Tools: For hunting tools, Bro is my go to because of how quickly I can get a birds eye view and start gaining intelligence with it. I also use basic Unix commands like grep, sed, and awk and tools like Tshark because I tend to deal with a lot of PCAPs. Splunk is another great tool that I spend a lot of time in and have been incorporating more and more.
I tend to try and keep my toolset as simple as possible and instead focus on increasing my efficiency by coming up with better questions. I’m like a toddler in the sense that I just keep asking “Why?” and “What?”. Why is that talking to that? Why did this behave this way? Why did that guy log into this machine at 3:30am? What would exfil look like in this network protocol?
Datasets: Because I’m hunting for different clients, the datasets tend to be different. My favorite data would be Bro logs with DNS coming in as a close second. Then I would say HTTP proxy logs and flows. I typically spend more time on the network side but for users and endpoints I really appreciate Sysmon and OSSEC because of the wealth of information they can provide me.
Techniques: I keep my techniques pretty simple as well. I start with a simple hunting methodology that’s broken up into Network, Endpoint, or User focuses. Then I decide on one of those depending on the data sets available to me from the client. From their I tend to look for anomalies and outliers and see where the data leads me, while keeping track of my time to prevent me from following any one rabbit hole for too long.
- For Network, I tend to break down the traffic into protocols and look for anything I would consider “weirdness on the wire”. For example, in DNS traffic I like to look for outbound traffic to DNS servers I don’t recognize and in which countries those servers are located, and comparing request volume and packet sizes by each host. This can be a clue to identify C2 or exfil activity. Basically, anything that looks weird or I simply don’t understand.
- For Endpoints, I’m typically concerned with processes and registry changes that aren’t typical, along with OS and software versions that might be out of date and vulnerable.
- For Users, I want to see behavioral anomalies by establishing baselines. Most of my baselines are centered around time. A typical anomaly I look for is unusual login times. This can be seen with compromised accounts and also insider threats. It’s not uncommon in corporate espionage cases for data theft to occur late at night or on the weekends.
What value do you actively see come out of your hunting activities?
As a consultant, it’s important for me to remember that providing value to clients is ultimately the purpose behind any hunt. I don’t believe there is such a thing as a failed hunt. Whether value is derived from uncovering something sexy like APT activity or something seemingly boring like misconfigured VLANs, there’s always value to be gained. I recall a hunt where I uncovered hundreds of thousand credit card numbers being sent over a network in clear text. Needless to say the client considered this very valuable, which then allowed me engage with them to help mature their security posture.
For the personal value I receive from hunting, I would say it comes from how effective my hunting process was during an engagement. I love having the chance to test new ideas and techniques and track the effectiveness of them, then take that data, make changes, and then put it to the test during my next hunt, rinse and repeat.
What types of friendly intelligence are most useful for a hunter to have in an investigation?
Things like asset management and user roles is really helpful information because it can save a huge amount of time narrowing the hunting radius down to critical systems. I also like looking at technical use cases in a business to see how data flows.
What general advice do you have for new Threat Hunters?
Watch your assumptions and let the data tell the story, else you begin twisting facts to fit your theories, instead of theories to fit your facts. It’s important to get to know your data sets and what kind of intelligence can be gained from them before making recommendations.
Keep track of everything, including time spent. It’s easy for even the most senior guys to waste time chasing a rabbit down a hole that doesn’t lead anywhere or never ends. I’m a big Evernote user and have several years’ worth of queries and searches for different data sets which remind me the intelligence I can gain from them. I continually tweak my notes and info to become more efficient and faster in my hunting.
What hunting procedure would you recommend for a new to Threat Hunter?
For new hunters, I would say to keep it simple and try and be methodical in your thinking. Choose a particular stage in the kill chain and think about how it fits into your data set. This is a really easy way to start a hunt if you have limited data available to you. It’s also important to start with what you know and where your strong. Whether it be particular networking protocols or the normal behavior of an application. Use that knowledge as a baseline and start looking for abnormalities and outliers and continue to grow from there.
Check out our full Hunter Profile series for different takes, tips and tricks on threat hunting from the experts!