Introduction

This blog post will walk through common Kubernetes network security risks and how we can secure our Cluster’s outbound traffic using tools like Cilium and Network Policies. Some basic knowledge of K8s is assumed.

What problem are we trying to solve?

By default, our workloads running in Pods within our Kubernetes cluster can talk to anything inside the cluster (East-West) and on the internet too (North-South).

This can raise some pretty serious security concerns and risks.

Some of these are:

Data Exfiltration - Our sensitive data could be sent to unapproved destinations, which could even include countries such as North Korea and Iran.
Malware Command & Control - Any compromised workloads can establish connections to servers that are controlled by attackers.

Without implementing network security controls you have limited ability to stop any of the aforementioned risks from materialising. If your Kubernetes cluster is running in a cloud like AWS, you can use Security Groups and Network Access Control Lists to lock down your outbound network traffic. Although both of these only work at L3 and L4 of the OSI model and therefore don’t provide any L7 protection. Outbound network traffic should follow a deny by default pattern with allow lists implemented for any workloads that need to talk to anything on the internet. Think GitHub, SaaS services or other package managers.

Traditionally these L7 network security controls were implemented using a URL filter via a Stateful Firewall from a vendor like Palo Alto, Fortinet or even AWS if your environment is completely Cloud Native. Whilst this does the job, Network and Cloud Platform teams can run into some issues like:

Runaway excessive costs due to the pay-as-you-go per-GB cost nature of Cloud Native Firewalls. I’ve personally seen this run into >$200k a month.
Impact on latency sensitive workloads due to traffic needing to hairpin through a firewall.
Increased overhead of self-hosted firewall management including patching, upgrading & HA.
Packet fragmentation issues due to components like AWS GWLB adding overhead causing legitimate traffic to be blocked by the Firewall.

Whilst traditional dedicated firewalls gave us the required protection as well as some extra features like TLS inspection & IDS/IPS, we can mitigate many of the above risks using a native Kubernetes feature called Network Policies. This also allows us to drop the traffic before it even leaves the cluster, as opposed to traversing the network racking up cloud data transfer costs.