However, Kubernetes Secrets are not as secure as they could be. Kubernetes’ base64 encoded secrets can be easily decoded making Kubernetes Secrets a less-than-ideal place to store sensitive credentials. If you run Kubernetes on the AWS cloud, here are some things you should consider.
Avoid node-level AWS IAM roles
IAM (Identity and Access Management) is used to manage access to the AWS cloud within an organization. With the help of an efficient IAM framework, access and privileges can be regulated and monitored. IAM roles enable you to set permissions and policies that decide what a particular person or application can or cannot do in AWS. These permissions can also be created for a particular role. These roles don’t have long-term passwords. Users are provided temporary credentials that are dynamically created if and when they assume a certain role.
In a Kubernetes cluster, IAM roles can be assigned to various nodes based on their functional requirements. A cluster is made up of various nodes and these nodes are composed of a number of pods. If you apply a role to a Kubernetes node, all the pods in that node inherit the permissions given to the node. This is the simplest use of IAM roles. However, it is not effective from a security point of view. The problem is that all the pods can now access the services they shouldn’t be accessing. This can be especially dangerous if you are using various third-party pods in your cluster. You need to be extremely wary of these pods as they could leave your AWS account exposed.
Principle of least privilege for Kubernetes on AWS
When it comes to IAM, the principle of least privilege must be followed. According to this principle, every pod in a cluster should have the least possible privileges making sure there are no unnecessary permissions that can lead to attacks. A pod should only have access it needs to perform its particular task. Assigning a global IAM role to a node violates this principle. Additionally, there is always the risk of developers making mistakes or adding bad code to the cluster that can cause trouble if there are pods with unnecessary access.
Alternatively, you can use IAM users instead of IAM roles. You can make each pod an IAM user with minimal and necessary permissions and pass the credentials to them using environment variables. This doesn’t violate the principle of least privilege, as each pod gets limited access.
These credentials can then be stored in Kubernetes Secrets. However, as mentioned above, these secrets can be easily decoded, which can lead to random pods gaining access to the AWS resources. Also, static configuration of AWS credentials will not allow for rotating keys, which is done automatically by the IAM roles.
There are also various restrictions imposed by organizations on IAM users. These restrictions can limit IAM users from accessing keys or requiring two-factor authentication, which complicates things further. Pods having access to keys also poses the risk of some developers accidentally logging them or hard coding these keys in order to avoid managing secrets, leaving them exposed.
Both of these methods have their fair share of disadvantages, which make them unsuitable for production. However, there are two open source solutions - Kube2iam and Kiam - that help manage IAM access for the pods while ensuring the principle of least privilege is not violated. Kube2iam was the first solution to come out and has been used in various deployments since. This solution deploys DaemonSets called Kube2iam pods in every node in the cluster. Each Kube2iam pod runs on the host network and has privileged access to intercept calls being made to the EC2 metadata API by various pods in that node. However, instead of authenticating these requests directly, it assumes the annotated pod role and returns temporary credentials that can be used by the requesting pods.
Kiam is built upon Kube2iam and includes a lot of improvements. The biggest difference, however, is that instead of having the DaemonSet directly communicating with IAM, it has an agent pod deployed in every node that communicates with a server pod in the master node that contains prefetched temporary credentials. This reduces latency and restricts IAM access further. With the help of separate server and agent nodes, there is increased security as every node doesn’t have the ability to assume any node they want. In order to successfully deploy your application using Kiam, you must deploy it in a namespace that allows the assumption of roles through Kiam.
Namespaces should be annotated with iam.amazonaws.com/permitted. For example:
AWS IAM Controller For Kubernetes
An interesting approach from Zalando, that takes a different approach implements a Kubernetes controller if you will, for distributing AWS IAM role credentials to pods via Kubernetes Secret resources, mounted as files to pods, and wire AWS SDK become aware of this file by pointing AWS_SHARED_CREDENTIALS_FILE environment variable, to the secret mount point.
The EKS approach for fine grained IAM roles for Pods
AWS recently delivered fine-grained IAM roles in EKS , which basically made Pods as first class citizens in AWS IAM. Rather than intercepting the requests to the EC2 metadata API to perform a call to the STS API to retrieve temporary credentials, which is how kiam and kube2iam works, AWS made changes in the AWS identity APIs to recognize Kubernetes pods. By combining an OpenID Connect (OIDC) identity provider and Kubernetes service account annotations, you can now use IAM roles at the pod level. Under the hood, the credentials wiring leverage a mutating admission controller, that based on the configured service account, will wire all the necessary configuration to the pod. A step by step example of this new EKS feature from AWS can be found here.
With the growing complexity of cloud-based deployments, there is a need for end-to-end security in a Kubernetes cluster. Alcide Advisor does just that. It’s a Kubernetes multi-level vulnerability scanner that covers Kubernetes’ best practices and compliance checks. It’s an agent-less, pure API-based command line tool that audits your cluster, nodes, and pods.
Alcide advisor hunts for misplaced secrets in ConfigMaps and pod environments that could be used by attackers. It also provides a comprehensive list of all the security issues sorted by criticality. It also performs IAM role pod delegation checks for AWS clusters helping users ensure stringent IAM access.
If you run Kubernetes on AWS, secure your clusters adequately using a capable end-to-end security solution that helps you enforce the principle of least privilege. Try Alcide Kubernetes Advisor now.
About Twain Taylor
Twain is a Fixate IO Contributor and began his career at Google, where, among other things, he was involved in technical support for the AdWords team. His work involved reviewing stack traces, and resolving issues affecting both customers and the Support team, and handling escalations. Later, he built branded social media applications, and automation scripts to help startups better manage their marketing operations. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications.Vince Power is an Enterprise Architect at Medavie Blue Cross. His focus is on cloud adoption and technology planning in key areas like core computing (IaaS), identity and access management, application platforms (PaaS), and continuous delivery.