Sqrrl Blog

Apr 20, 2016 10:47:00 AM

Cyber Threat Hunting (3): Hunting in the Perimeter

Originally posted by Samuel Alonso, KPMG Global Security Operations Center threat hunter at http://cyber-ir.com/2016/03/01/cyber-threat-hunting-3-hunting-in-the-perimeter/ 

In this third post, we will learn what we need to look at when hunting and detecting adversaries in the perimeter. We are also going to look at some of the firewall technologies and their log formats in order to detect anomalies in the inbound and outbound traffic in your network.




Most traffic in modern organizations goes through different choke points in order to inbound or outbound your organization. This choke points are your perimeter firewalls where your organization is monitoring the network traffic and normalizing this traffic so it adheres to the company security policy. Many of the techniques you use to hunt in the perimeter can be used in the internal firewalls to add segmentation and security levels to different parts of your network; however, I am only going to focus on detection in the external firewalls.

If you have not figured this out yet, the perimeter is probably one of the most difficult places to hunt as literally all types of traffic go through the external firewalls. With such a large amount of traffic and protocols, it is very easy to get hopeless when looking at huge amounts of data in this part of your network. That’s why it is important to understand your company security policy and environment.

Assuming you company firewalls were setup with your security company policy in mind, you should be able to understand some basics such as:

  • Firewall security requirements for your organization
  • Permitted communications
  • Enforcement points
  • Allowed transaction flows in your environment
  • Connections with your business partners and guest access networks
  • How resources, applications and services need to be protected by your firewall
  • Traffic baseline for your firewall

This list covers a few of the basics; the more you understand the purpose of your firewall in regards to what your company policy enforces, the faster you will be able to spot anomalies in your firewalls.

Now that we have a good understanding of our perimeter firewalls, let’s start applying some techniques to spot potential adversaries in your perimeter. There are many techniques out there; I am only going to cover some of them.

1. Denied inbound traffic

Many people argue here and they end up fooling themselves. "If the firewall is stopping traffic, why would I look into it? It is being stopped and that’s all there is to it." It is a valid opinion, but from the security point of view has little benefit to offer. Why is that traffic being stopped at your doors? Is it a misconfiguration or is it a potential adversary probing or scanning your firewall? Where is that IP coming from? Is it a company-owned IP or an IP in the addressing space of an ISP? Is the IP blacklisted, does it have a history of known bad reputation?What traffic and ports are being blocked? Are there any services running behind the firewalls on these ports? Are they vulnerable? Is it always the same port being blocked, or is there evidence of the same IP (or group of IPs) probing other ports too? Maybe a vertical scan? Or horizontal? Could it be residual traffic from a malware infection? Is there evidence of connections to these blocked destinations?

The amount of questions you can think of will guide your investigation, and it is very important to extract as much information from these events as you can. Once you have all the information, you may want to pass this to other teams to collaborate in your investigation. This information can be used to establish correlation with internal information systems, start an open source intelligence investigation to discover additional intelligence on these events, or even just as threat intelligence.

Looking at this type of activity can only benefit your organization and it can be used as an early warning for your teams of what it is coming towards you in the near future.

2. Denied outbound traffic

You did not find any important anomalous inbound traffic being denied in your firewall? Give yourself one more chance. At some point your attacker went in, he also needs to go out and guess what…he has to use your choke points to do so. To have denied outbound traffic is concerning, so ruling out misconfiguration issues in your infrastructure, you should wonder what is happening inside your network with the traffic denied in the firewall. Do you have a considerable amount of IPs being denied to an external destination? What does that traffic look like? What’s the IP or set of IPs? What’s the protocol used? How often is happening? To which entity does that IP or set of IPs belong to? What sort of traffic is it? Could it be data exfiltration – an insider threat? Could it be an external attacker or malware trying to leave your organization?

All these questions and more should trigger an internal investigation to understand the behavior inside your network. If the traffic is being denied, it is because your security policy thought of that scenario beforehand, or because that communication it is not supposed to happen.

3. Permitted Inbound and Outbound traffic

This one is a difficult one, as the amount of traffic to analyze makes it a daunting task. One of the fastest ways to approach the problem is to use the rinse and repeat technique. Look at the traffic logged, define what is normal (whitelist), remove it from your log investigation and start the process again. At the end of the exercise you end up with some new information about your network or you end up with a set of logs where your investigation should begin.


In my previous post, I mentioned that it would be desirable to have a scripting language. Here is one of the use cases for that scripting language: hunting through a log file with 5,000 to 10,000 lines will take a long time without a scripting language. If you do hunting for entertainment though, you can get along without it; however, if you have to respond to an incident, time is critical.

What other indicators can we monitor in the firewalls?

The important indicators were already discussed above; however, there are still other indicators that can be used to spot anomalies in your perimeter traffic such as user activity, bandwith usage, protocol usage and amount of bytes transferred. I will not get into details as they are self-explanatory, but I will do a brief stop on protocol usage. I have to clarify first that this case is only applicable to cases in which you are doing mainly packet capture, as you may be able to observe the real traffic traversing the wire.

4. Port Independent Protocol Identification

PIPI technique allows the identification of the protocol used regardless of the port in use. PIPI makes it possible to discover covert channels, back doors and in general, any policy-violating channels.

The technique is easy to implement if you have an understanding and you can recognize different application layer protocols. You also have some tools out there that will allow you to implement the technique without a deep understanding of the protocols in used.

Being able to identify application layer protocols without relying on the TCP or UDP port number is crucial when analyzing malicious traffic such as malware, Command & Control (C2) communication, covert backdoors, and channels, since such communication often uses services on standard ports to blend the malicious traffic with the legitimate traffic.

Some examples:

  • Some botnets C2 protocols communicate port TCP 443, but using a proprietary protocol rather than HTTP over SSL
  • Backdoors on hacked endpoints and networks typically run on standard services such as SSH on port 22, in order to blend with normal traffic and hide
  • Some backdoors use port knocking to run a proprietary C2 protocol on a standard port, such as port 80

What does all of this mean?

By analyzing network traffic for port-protocol anomalies, like an outgoing TCP connection to TCP port 443 that is not SSL, you can effectively detect intrusions.

Remember this is only useful when having packet capture, as it is the only way to analyze the traffic that goes through the wire.

Now that we have an understanding of where to start hunting in our perimeter firewalls, let’s look at the log format that many firewalls out there in the market are throwing at us.

Needless to say that if you do not master the logging format of your perimeter firewalls before going out there on a hunting trip, you are wasting your time.

The logs we are covering are for the following firewalls: Cisco, Palo Alto and Juniper. This last section is not very long, it's just a matter of understanding what type of firewall each one is and how the logs look before starting a hunting trip. Why specifically these vendors? They are Enterprise Grade Firewalls.

Cisco firewalls

If Cisco is good at one thing in their firewalls, it is detecting potential network attacks or anomalies in the network layer. These firewalls are not the most advanced in the market; however, they do their job well. The log format is pretty easy to understand.

Cisco ASA logs have different levels:


These levels are very useful when you are searching in your log files. As you can see the different levels have different criticality, and they inform of different conditions in the traffic routed through the firewall.

Below you can find a subset of the most common logs used to detect incidents in the network layer.


If you look within your Cisco logs and you search for the "Mnemonic field," you can easily start seeing incidents being flagged by your firewalls. The section above does not cover all of them, but certainly the most important ones.

Here are some additional resources for you to study in detail:

Palo Alto firewalls

These firewalls are top shelf; this brand was introduced in the market as the so-called NGF (Next Generation Firewalls). These firewalls are capable of inspecting traffic in the upper layer protocols. They do packet filtering, they are protocol aware, and they inspect the packet payload to detect signatures of attacks. To put it simplly: they are hybrids between packet filtering firewalls and Intrusion Detection Systems.

The firewall can function as packet filtering device or IDS, you can see the difference in the way the log is tagged -- traffic logs when working in layer 4, and threats when working in layer 7 as IDS. Beyond this , all that's left for you is to study these logs in detail. Please find further documentation below.


Juniper is not a firewall, but rather a whole network operating system. You can find out more about it by doing a little research. Juniper’s origins are in the ISPs and core network markets; however, they have also evolved to cover the edge routers market. The complexity of their hardware can be seen in their system log reference for security devices.

When we look a Juniper firewalls, we find the SRX’s series firewall which is the evolution of their legacy ScreenOS solutions.


These are some of the firewalls you can find in the market with Enterprise Grade Capabilities. I suggest you research the documentation of your current firewalls and start understanding the log format and capabilities in order to protect your network. Of course this is not an easy task, and the size and complexity of your networks will also influence your results; however, some technologies such as signature, visualization, and automation can assist the hunter in his hunting trips to pinpoint which area to cover.

I haven’t mentioned it before, but it makes sense to document all actions taken during your hunt trip in a hunting log book. This will help you provide feedback to both management and your team, and it will avoid getting into loopholes on your next hunting trip.


Topics: Cyber Hunting, Incident Response, Threat Hunting, Cyber Threat Hunting