Sorry, the language specified is not available for this page
Sales: +1 866 714-7209
Support: +1 800 441-1181
This summer I had the opportunity to attend AWS’s re:Inforce security conference. There, I was able to catch up with old acquaintances, learn some new tricks, and generally geek out on cloud security.
Like most of AWS’s conferences, they like to announce all kinds of new services. My interest was piqued when AWS revealed their new EC2 Traffic Mirroring service - the cloud was now going to give us back the ability to look deeper into our network data.
Recently, I had a chance to dig deeper into this service to see how it works and understand any limitations or “gotchas.” Here is what I learned:
The AWS Traffic Mirroring service allows you to span, or copy, network traffic from an EC2 Elastic Network Interface and send it to a supported target. Essentially, we can now “sniff” our cloud network data going in and out of our EC2 instances. This copied traffic can be sent to a security or monitoring solution for content inspection, threat monitoring, and troubleshooting.
Supported targets include another EC2 instance or a fleet of EC2 instances behind a Network Load Balancer. Filters are used to select the traffic of interest to be copied.
Additional details on this service can be found in AWS’s Online Documentation: https://docs.aws.amazon.com/vpc/latest/mirroring/what-is-traffic-mirroring.html
Traffic Mirroring is another tool for monitoring network traffic. From a security perspective, we can now deploy out-of-band intrusion detection and analysis tools to look for malicious activity in the network traffic. For operations, we can use traffic analysis tools and packet inspection tools for troubleshooting and traffic modeling.
Prior to this service, we had no real way to look at our network traffic. Our options were to route traffic through an instance or deploy local collection agents on instances. VPC Flow Logs can be used for basic flow analysis but lacks full packet-level information and inspection.
When enabled, AWS’s GuardDuty Service performs some basic network threat and anomaly detection using the VPC flow log data, but it is somewhat limited in the threats that it can detect. GuardDuty is still an important tool in our security tool chest by providing anomaly detection on AWS API calls through CloudTrail analysis.
Traffic Mirroring works by copying inbound and outbound traffic, encapsulating the original packet in a VXLAN packet, and forwarding the new encapsulated packet to a UDP listener on the target. Figure 1 shows how this works with an AWS EC2 instance.
For larger scale or high-availability monitoring, a Network Load Balancer can be selected as the mirror target. The Network Load Balancer can then forward the traffic to one or more EC2 instance targets.
Once I got the load balancer architecture set up and fired inbound traffic to my simulated application on the Traffic Mirror Source, I noticed that the traffic was alternately routed to the different target instances. After a little digging, I confirmed this behavior through AWS’S Network Load Balancer documentation:
For UDP traffic, the load balancer selects a target using a flow hash algorithm based on the protocol, source IP address, source port, destination IP address, and destination port. A UDP flow has the same source and destination, so it is consistently routed to a single target throughout its lifetime. Different UDP flows have different source IP addresses and ports, so they can be routed to different targets.
As previously discussed, the copied packet is encapsulated in UDP VXLAN Packet and forwarded to the target. The data portion of the UDP packet contains the VXLAN ID and the portion of the original network packet.
The traffic mirror targets are the destinations for mirrored traffic. An Elastic Network Interface or Network Load Balancer can serve as mirror targets. A mirror target can reside in the same AWS account or routed to another AWS account.
The traffic is forwarded to UDP port 4789 on the target. The inbound traffic must be allowed in the target’s Security Group.
AWS’s Traffic Mirroring documentation has additional details on requirements for target connectivity.
Traffic mirror filters are the rules that define what inbound and outbound traffic is copied from the traffic mirror source. There are separate Inbound and outbound rules, and both must be set to capture bi-directional traffic.
The following traffic mirror filter rule parameters are available:
A traffic mirror session establishes a relationship between a traffic mirror source, a traffic mirror target, and the network traffic that is copied. A traffic mirror session is defined by:
Here are some items to consider and limitations of the service that have been summarized from AWS’s Documentation.
Additional limitations can be found here: https://docs.aws.amazon.com/vpc/latest/mirroring/traffic-mirroring-considerations.html
VXLAN traffic is essentially just a UDP packet with encapsulated data. Any normal network capture tool, such as tcpdump, can capture and view the data. TCP dump will extract and display the inner packet, but tcpdump’s capture filters only work on the UDP header unless you use special packet level content filters.
Below is a sample of a VXLAN packet captured via tcpdump on our Linux target. If you a familiar with tcpdump output, you can see the IP address information of the outer UDP VXLAN packet at the begging of the output. The outbound HTTP response from a web server is then seen in the output:
18:24:29.045912 IP 10.0.0.60.65413 > 10.0.1.215.4789: VXLAN, flags [I] (0x08), vni 4321
IP 10.0.0.60.80 > XXX.XXX.XXX.XXX.58953: Flags [P.], seq 1:1041, ack 609, win 236, options [nop,nop,TS val 2388734652 ecr 3585683716], length 1040: HTTP: HTTP/1.1 200 OK
0x0000: 4500 0476 0000 0000 fe11 a264 0a00 003c E..v.......d...<
0x0010: 0a00 01d7 ff85 12b5 0462 0000 0800 0000 .........b......
0x0020: 0010 e100 06ad d081 25c4 06ba cd46 0cfc ........%....F..
0x0030: 0800 4500 0444 40d9 4000 fe06 4ec9 0a00 ..E..D@.@...N...
0x0040: 003c d8cb 060a 0050 e649 3aa9 2a4d 95f2 .<.....P.I:.*M..
0x0050: 63a3 8018 00ec 91f6 0000 0101 080a 8e61 c..............a
0x0060: 32bc d5b9 3104 4854 5450 2f31 2e31 2032 2...1.HTTP/1.1.2
0x0070: 3030 204f 4b0d 0a44 6174 653a 2046 7269 00.OK..Date:.Fri
0x0080: 2c20 3233 2041 7567 2032 3031 3920 3138 ,.23.Aug.2019.18
0x0090: 3a32 343a 3239 2047 4d54 0d0a 5365 7276 :24:29.GMT..Serv
0x00a0: 6572 3a20 4170 6163 6865 2f32 2e34 2e32 er:.Apache/2.4.2
0x00b0: 3520 2844 6562 6961 6e29 0d0a 4578 7069 5.(Debian)..Expi
0x00c0: 7265 733a 2054 7565 2c20 3233 204a 756e res:.Tue,.23.Jun
0x00d0: 2032 3030 3920 3132 3a30 303a 3030 2047 .2009.12:00:00.G
0x00e0: 4d54 0d0a 4361 6368 652d 436f 6e74 726f MT..Cache-Contro
0x00f0: 6c3a 206e 6f2d 6361 6368 652c 206d 7573 l:.no-cache,.mus
So, any network security or monitoring tool we want to use to consume the mirrored traffic must be able to extract or process the encapsulated packets within the VXLAN packets. Refer to your favorite vendor’s documentation for their tools’ capabilities and configuration.
While I was testing, I wanted to be able to just work with the inner network packet. After doing a little bit of research, I found another little trick to only work with the original network packet. This technique uses a virtual VXLAN network interface in Linux to extract the inner packet.
The commands below show an example on how this can be accomplished:
sudo ip link add vxlan0 type vxlan id 4321 dev eth0 local 10.0.0.207 dstport 4789
sudo ip link set vxlan0 up
The first command creates a virtual network interface with the label “vxlan0” that extracts traffic with VXLAN ID “4321” sent to local interface “eth0” that has a defined IP address of “10.0.0.207”. The traffic is sent to the destination port of “4789”. The second command enables and turns-up the new interface.
These parameters would be changed to match the specific implementation and traffic mirroring session.
Now we can use tcpdump and its capture filters on the original packets. Tools that do not support VXLAN encapsulated packets can also process the mirrored packets using this method.
The AWS Traffic Mirroring fills a gap that organizations have with monitoring network level traffic within their workloads. This service provides us with a capability to copy network packets and forward them to security or network analysis tools to help us better understand what is happening at the network layer.
However, like any tool or service, you must understand the requirements and limitations of the service to obtain the most value from it. As you look to implement this service, you need to understand what you are trying to accomplish with your traffic monitoring, what is required to make it work in your solution, and if your tools can support the service’s output format.