You’re managing a complicated app on Kubernetes and want to see which microservices communicate with one another. Here are four different approaches you can take:
Istio and Kiali
Istio is the first solution that typically comes up and it’s usually overkill. Istio is hard to configure and has a high performance overhead. Istio’s own benchmarks show a 5.6x to 20.1x increase in http latency when adding Istio’s sidecars.
Whether that overhead is meaningful or not depends on what your microservices do. If you have a microservice which serves content from memory to another microservice in the same cluster, then the overall overhead that Istio adds probably has a significant impact on the request time. On the other hand, if you’re running an old school webapp which runs a db query and then generates an http page for each request from an internet client, then the overhead is likely negligible.
Despite the overhead, Istio is a great piece of technology. By deploying Istio you might spend more on server bills yet save on development costs by increasing developer efficiency. However, if all you need from Istio is a diagram of which microservices are speaking to one another then there are simpler and more lightweight solutions than Istio.
Cilium and Hubble
Cilium is a great CNI powered by eBPF just like the Alcide Runtime. If you are already using Cilium as your CNI then it really is a no-brainer to install Hubble and visualize traffic that way. Like Istio, Cilium can provide layer7 visibility, although this requires enabling an Envoy proxy which will have an impact on latency.
There are only two disadvantages to using Hubble and Cilium. First of all, switching CNIs can be a big undertaking with broad effects if all you want to do is visualize traffic. Second of all, by using Hubble you’re locked into the Cilium CNI.
Weave Scope is an open source offering from Weave Works which allows visualizing Kubernetes Clusters without making deep changes to your cluster’s infrastructure. Weave Scope uses eBPF probes to gather information about pods without man-in-the-middle traffic interception. This has many advantages from both a performance perspective and operational perspective. One major advantage of Weave Scope is that it works with multiple CNIs and supports writing plugins to show custom data.
Application and Namespaces view on Alcide's platform
All the above tools focus on providing a visual image of which microservices speak to one another and they do an excellent job at that. If all you care about is visualizing small clusters, then you can stop reading here.
In large Kubernetes deployments, even the simplest network visualization can blow up into a massively complicated mesh of microservices speaking to one another. For these types of deployments, it is important to focus on the specific questions that you want to answer and to separate the signal from the noise. Often the questions that need to be answered are security questions like the following:
- Which microservices speak with both the external world and other internal microservices?
- Is a malicious actor exfiltrating sensitive data from my cluster? (Simply visualizing the network doesn’t necessarily answer this question because a sophisticated attacker can use DNS tunneling or other techniques of masking malicious traffic as legitimate traffic.)
- Has my cluster started speaking to any new external domains in the past week?
- Is there traffic from within my cluster to a known bitcoin-related domain? (i.e. is there a bitcoin miner in my cluster)
- What 3rd party domains receive the most traffic from my cluster?
- Do microservices in a specific hardened Kubernetes namespace speak to microservices in a different non-hardened K8s namespace?
- Are any pods sending anomalous traffic? (e.g. Kubernetes layer2 attacks, DNS tunnelling, etc.)
- Are any pods abusing Kubernetes vulnerabilities (either zero-days or known attacks)?
For security questions like these, it isn’t enough to just visualize your cluster. For large clusters you need automatic detection of security issues to separate the signal from the noise, preferably with almost no configuration and low overhead. One such solution is the Alcide Runtime which is a commercial solution that focuses on Kubernetes network and process security. It mixes these security features with a rich network visualization that lets you see both how microservices communicate with one another. We recently added a namespace visualization feature which combines a cross-namespace traffic visualization with answers to security questions in one view.
Like Cilium and Weave Scope, the Alcide monitoring solution is based on eBPF probes.
Three of the four solutions mentioned above use eBPF to monitor traffic - only Istio intercepts all traffic with a man-in-the-middle proxy instead of using eBPF and it pays a higher performance price for doing so. Kubernetes networking is definitely moving towards eBPF based solutions.
If I forgot a visualization tool worth mentioning then please leave a comment below. I would love to hear from you about the solution you chose and why.
About the author
Natan Yellin is a senior software engineer at Alcide.io. (Psst, we’re hiring!)
He also blogs about low-level Linux at http://natanyellin.com.