Traditional network security includes protection against layer2 and layer3 spoofing attacks. Many security teams don’t realize it, but these threats are still relevant when running applications on a Kubernetes cluster in the cloud. You might be using a complex container network, but that doesn’t mean that simple spoofing attacks between pods aren’t possible. This matters because it dramatically increases the blast radius of compromised pods.
In Kubernetes, by default, all pods have CAP_NET_RAW permissions. This means that all pods can open raw sockets and inject malicious packets into the Kubernetes container network. The typical threat scenario here is that an attacker has managed to take over one pod (e.g. via an application vulnerability) and wants to move laterally in the cluster to other pods. Alternatively, the attacker may want to remain on the same pod but escalate their privileges to cluster-wide permissions via attacks directly against the host. The ability to inject raw packets into the network - which is allowed by default - is a useful and sometimes sufficient tool to accomplish such goals.
In Kubernetes, the impact of raw packet injection depends on the CNI in use. Some Kubernetes CNIs filter out certain spoofed packets between pods, limiting the scope of what an attacker can accomplish using packet injection. Unfortunately, this isn’t the case for the default CNIs on most cloud providers. To demonstrate this, we ran some checks for the popular cloud providers using the default CNI for each one. For each cloud provider we tested common layer2 and layer3 spoofing techniques as well as port scanning reconnaissance to help identify targets. Here are the results:
As you can see, the spoofing attacks all succeed on GKE and AKS, but they fail on EKS due to the way the VPC CNI routes traffic between pods. On the other hand, port-scanning is allowed on all cloud providers including EKS. This makes sense because port-scanning isn’t a spoofing attack and each individual packet in a port-scan is a legitimate packet in and of itself. You need a stateful firewall to detect such attacks - specifically one which was built for Kubernetes and can monitor traffic inside the container network itself.
There are two different strategies for dealing with spoofing attacks on Kubernetes. A good security strategy should implement both:
- Limit pod permissions: in this case, that means dropping CAP_NET_RAW permissions. You should use a Kubernetes admission controller to ensure that developers can’t accidentally deploy pods with CAP_NET_RAW. Traditionally, PodSecurityPolicies were used for this, but they are slated for deprecation in Kubernetes 1.22 despite the fact that no replacement currently exists in Kubernetes itself. Alternatives include OPA Gatekeeper and Alcide Advisor.
- Monitor and restrict traffic between pods: This can be done at a basic level using certain CNIs or at a more advanced level using microservice firewalls. As we saw in the above comparison, many CNIs aren’t focused on security. Even when they do block spoofing attacks, they don’t implement heuristics for detecting attacks involving “legitimate traffic” like port-scanning. To detect those attacks you need a stateful firewall focused on security. Therefore many companies choose to implement network security using both a hardened CNI and a microservices firewall.
Appendix 1: Securing your Cluster with Alcide
Alcide’s security platform protects your Kubernetes network in four important ways:
- The Alcide Runtime includes a security engine that identifies network attacks between pods. It contains a behavioural anomaly engine that detects spoofing attacks, cluster scans, and data exfiltration techniques like dns tunneling. It also includes an eBPF-based firewall for segmenting the container network.
- The Alcide Advisor scans your Kubernetes cluster for non-hardened pods and compliance violations. It checks for non-dropped capabilities like CAP_NET_RAW permissions and for missing PodSecurityPolicies. It also looks for other common threats like hostPath mounts and the use of host Linux namespaces like hostNetwork, hostIPC, and hostPID.
- The Alcide Admission Control module includes prebuilt policies that can not only detect the above issues but actively prevent their deployment into the cluster according to pre-configured policies. Like the Alcide Advisor, it comes with pre-built policies which build on Alcide’s experience hardening Kubernetes clusters.
- The Alcide kAudit module scans Kubernetes audit logs to detect anomalous behaviour. This doesn’t directly detect the packet spoofing itself, but it can easily detect the other suspicious behavior that happens during a breach. For example, if a pod manages to steal Kubernetes credentials from another pod then kAudit can detect the sudden change in pod permissions.
Appendix 2: Details for Reproducing Results
We tested the attacks discussed here using two pods running Kali Linux images on the same node. The pod YAMLs were standard with no options set other than the name, container image, and command options. One of the pods functioned as a compromised attacker pod and the other pod functioned as a target pod which the attacker wanted to send spoofed packets to.
In the below table you can find the commands we ran on both the attacker and target pod to check each attack. First, a few notes:
- The table includes several environment variables:
- $TARGET was set to the IP of the target pod
- $SPOOF was set to a random non-existent MAC address we wished to spoof
- $DEFAULTGW was set to the default gateway of the target pod
- For the IP/MAC spoofing attacks we used tcpdump to observe that the spoofed packets arrived at the target pod.
- For the ARP poisoning attack we implemented a simple DOS attack instead of a man-in-the-middle attack. This works by convincing the target pod that the attacker is it’s default gateway. Unlike a regular man-in-the-middle attack using ARP poisoning, the attacker doesn’t pass on traffic that it receives from the target - instead it drops all of the target’s packets. Therefore the target’s communication with the external world breaks because all packets are sent to the attacker which drops them.