Kubernetes V1.18-alpha.2 is live! The new version introduces an alpha stage field for both Secret and ConfigMap objects to mark their content as immutable.
How Does it Work?
Marking a ConfigMap or a Secret as immutable will deny any further change to the content of the object. However the object metadata can still be modified.
Suppose we have a video game and we define some configuration parameters:
We will place the ui and game in two files in a directory and package the directory into a ConfigMap
We will then provide this ConfigMap to the game Pod via a volume mount:
What is it good for?
The most popular and the most convenient way of consuming Secrets and ConfigMaps by Pods is consuming it as a file. However, any update to a Secret or ConfigMap object is quickly (roughly within a minute) reflected in updates of the file mounted for all Pods consuming them. That means that a bad update (push) of Secret and/or ConfigMap can very quickly break the entire application.
Methods for tracking changes in ConfigMap
When a ConfigMap content is changed, Kubernetes automatically propagates the change to all of the Pods connected to the ConfigMap. However unless a specific action is taken, the change will not be noticed by the software. For example, if a specific configuration parameter for a container is set in the ConfigMap, and the container reads the value during the software initialisation phase, the new value that will be applied to the ConfigMap after the change will have no effect on the software.
There can be several approaches to track a change in ConfigMap, and apply that change to the software. I will focus on two main methods here:
Method #1: Custom Code that Tracks the ConfigMap Content
Using this approach involves adding code in the application that will periodically sample the content of the ConfigMap and in case of a change, update the software internal data structure. This requires that the software has a mechanism that identifies the change by inspecting the ConfigMap content.
- A very generic and portable approach
- You need to develop a custom code and a specific solution to track ConfigMap change by inspecting the content
- You need to add an additional polling thread to the software
Method #2: Use Kubernetes Watch Mechanism
Kubernetes API server provides a built-in mechanism that allows the client to track changes performed to the resources residing in Kubernetes configuration database. This mechanism is called watch and allows the client to define a callback that is invoked when a change occurs.
Using this approach the software can set a watch and track the changes done in the configmap.
- No need to develop a solution for identifying ConfigMap change by inspecting the content
- You are dependent on the Kubernetes watch mechanism
- It requires simulating this mechanism during software unit test
- It increased the load on Kubernetes API server
Hey, let's pause for a minute…
While Methods #1 and #2 are very generic, they may present an over engineering approach. In many cases the ConfigMap contains values that are loaded to the software module upon initialisation only, and changing these values requires restart of the software module. Take for example the case where the ConfigMap includes a parameter that determines the size of a data structure that is created when the software is initialised. Re-allocation of the data structure will probably require restart of the software module going through the initialisation process again.
Hence we need a simpler approach that will cause an automated restart of the Pod upon the ConfigMap change. Say hello to sidecar!
Method #3: Sidecar
Using this approach a sidecar is created. The developer will create another container to be placed in the Pod. The container code will register to the Kubernetes API server via the watch mechanism and define the proper callback function. When a change to the ConfigMap is identified, the callback function will be invoked and the function can restart the Pod. The best practice for restarting the Pod will be by activating Kubernetes liveness probe (see Kubernetes documentation regarding setting liveness probes)
- The application software is not affected
- Easy to perform unit test outside Kubernetes
- You need to develop a sidecar container
- This can result in an increased load on Kubernetes API server due to the use of the watch mechanism
While methods 1-3 require developing some application code, there should be a better way that does not require application code development.
In fact, we can use a Kubernetes deployment procedure that will cause a Pod restart when a ConfigMap is changed.
Method #4: Versioning
In real-life Kubernetes systems the configuration files are created using some templating software such as Helm.
Using this approach, the ConfigMap name will include a version name. When a modification is required, a new ConfigMap is created with a new name, the application of the new name will result in creating a different deployment file content that, when applied, will cause a restart of the Pod.
- No need to develop a code
- No increased load on Kubernetes API server
- In this method the software does not monitor changes in the ConfigMap so, in case someone inadvertently changes an existing ConfigMap, the change to the content of the existing ConfigMap will not be noticed the the software and not reflected in the behavior of the software
Kubernetes Immutable ConfigMaps and Secrets to the Rescue!
This is exactly where the new Kubernetes feature supporting Immutable ConfigMaps and Secrets come into play.
Configuring the ConfigMaps and Secrets as immutable will protect against inadvertent change, thus help keeping the system consistent!
Applying Standards to Kubernetes Configuration
While using Immutable ConfigMaps and Secrets present a good practice, it is still an operational challenge to make sure that is it used consistently across all of the Kubernetes configurations.
Ensuring such consistency requires the use of a tool that is cable to check and verify that all configuration comply with a chosen standard.
Alcide Kubernetes Advisor is one option that is available for you in applying configuration standards and as well as protecting against security breaches caused by misconfigurations.
Current Feature Status
The implementation is hidden behind feature gate ImmutableEphemeralVolumes (currently in Alpha stage).
The details of the Kubernetes Enhancement Proposal (KEP) can be found here: https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20191117-immutable-secrets-configmaps.md