A security issue was discovered in the kube-apiserver that could enable a privilege escalation from a compromised node.
If an attacker is able to intercept certain requests to the Kubelet, they can send a redirect response that may be followed by a client using the credentials from the original request. This can lead to compromise of other nodes.
If multiple clusters share the same certificate authority trusted by the client, and the same authentication credentials, this vulnerability may allow an attacker to redirect the client to another cluster. In this configuration, this vulnerability should be considered High severity.
This vulnerability requires an attacker to first compromise a node through separate means. Thus, it is not trivial to actually exploit this vulnerability.
However, if this precondition is met, the vulnerability lets the attacker direct specific commands (e.g. remotely execute local shell commands inside a Pod) to any Pod in the cluster. This significantly escalates and broadens the scope of the initial compromise of a single node.
Furthermore, if a cluster is vulnerable and also shares certificate authorities and authentication credentials with other Kubernetes clusters, this vulnerability enables an attacker to redirect commands to resources on the other clusters, breaking the trust boundary of a single cluster.
Are You Vulnerable?
- kube-apiserver v1.18.0-1.18.5
- kube-apiserver v1.17.0-1.17.8
- kube-apiserver v1.16.0-1.16.12
- all kube-apiserver versions prior to v1.16.0
Alcide Advisor , our tailor-made Kubernetes security configuration and hygiene scanner, has the ability to scan your Kubernetes cluster(s) for Kubernetes CVEs and identify the clusters that are vulnerable and require an update.
Image source: Alcide Advisor
Remediation & Mitigation
To mitigate this vulnerability you must upgrade the kube-apiserver to a patched version.
- kube-apiserver v1.18.6
- kube-apiserver v1.17.9
- kube-apiserver v1.16.13
Technical Break Down of the Exploit
A precondition of exploiting the vulnerability is that a node (let’s call it N1) in the Kubernetes cluster is compromised in a way that lets the attacker intercept and respond to network traffic to it, and specifically traffic to the kubelet component that runs on it.
In such scenario:
- The cluster’s API server receives from a client (kubectl for example) one of the following remote access commands: exec, attach or port-forward to a pod, or proxy to any resource. After the usual authentication and authorization processing, in order to fulfil the client’s request, the API server needs to open a WebSocket connection on his behalf to kubelet which will fulfil the command (exec/port-forward/attach/proxy).
- Kubernetes API Server identifies on which node the resource is deployed and sends a request to the kubelet at this node to open a WebSocket to the resource. Technically it is an HTTPS Upgrade request.
- If the target node is the compromised node N1, this request is intercepted by the attacker and a malicious response is returned to the Kubernetes API server to redirect the request to a different k8s API endpoint in the same cluster ,or in another k8s cluster that uses the same certificate authorities (CA) and authentication credentials. Let’s assume this is a Pod running in a different node N2.
- The Kubernetes API server mistakenly accepts the redirect response (HTTP 3xx code) and re-sends the remote access request to the attacker-selected target resource with the privileged credentials of the original request, even though this is not the original request that has passed the authorization checks.
The following diagrams show an example of an attack. The first shows the normal progress of connections while processing a remote command sent to a pod. The second shows how the attacker intercepts and redirects the command to a different node using the initial compromise and the vulnerability.
Diagram showing how the attacker intercepts and redirects the command to a different node using the initial compromise and the vulnerability
The mitigation of the vulnerability in the patched Kubernetes code is to fix the API server in a way that disables the redirection (step 4) in the described flow, so the attacker cannot control the target of the remote access and bypass the authorization.
Automatic Detection of Attack on the Vulnerability with Alcide kAudit
Attempts to exploit the vulnerability are detectable by the redirect response (HTTP code in the range 300-399) to the k8s remote access commands (exec, attach, portforward, proxy).
- Alcide kAudit automatically detects the anomalous response status codes (in the 3xx range) associated with the principal's requests.
- The user can also add an explicit policy rule to Alcide kAudit to find audit entries in the log with a response code in the 300 range associated with one of the remote access commands.
Image source: Alcide kAudit
Kubernetes, like any software, has bugs and vulnerabilities. The recent disclosures signal an increase in awareness and an increase in the number of security researchers digging deeper into Kubernetes software components.
While there is a consensus behind the use of Kubernetes as the de-facto cloud native application operating system - and consensus around the viability that running Kubernetes based infrastructure serves. It takes an effort from Kubernetes operators to monitor and secure all the moving parts, whether these are the application workloads or the platform and infrastructure components.
Alcide kAudit is uniquely positioned with it’s machine-learning based behavioral profiling to detect known and unknown weaknesses in Kubernetes API, just like this CVE and others in addition to a wider set security and forensics analysis capabilities. Start your 14-day trial or request a demo of Alcide kAudit.