Turn configuration files on disk into structured Kubernetes ConfigMaps
At Tumblr, we have many sources for logical configuration data that defines how deployed services operate. Some of these sources are rapidly changing, some are updated automatically, and some are changed by humans. Because we always strive to limit complexity in our stack, we wanted to come up with a way that applications can be made aware of these changes as they happen, without becoming tightly coupled to our internal processes. Additionally, because there are many configuration sources (git repos, bots, cronjobs, tools like Collins, etc), and we wanted to write tools to manipulate this configuration data after being generated, we generate intermediary representations in a config git repo. However, because each application has very unique needs for what data it requires to run (eg TLS certs, comma separated lists of host names, env vars, specifically formatted JSON or YAML), we needed a method to transform the common configuration files into formats each application could consume or further process, without hardcoding configuration into our images, or requiring a rebuild/redeploy of the app each time configuration data changes. To accomplish this task, we built the k8s-config-projector.
The k8s-config-projector is a program to project config files from disk into Kubernetes ConfigMaps through Continuous Integration (CI). The ConfigMaps can be applied to a k8s cluster with kubectl apply -f <filename> (or create+replace) during Continuous Delivery (CD). These ConfigMaps can then be consumed by applications in Kubernetes, by mounting them as volumes or environment variables inside pods.
This is quite useful to inform applications in Kubernetes about changing configuration. For example, this can be used to discover the nodes participating in a memcached ring, configure an application with feature flags, etc.
To use this tool, it is expected that:
- You have some configuration data (YAML/JSON/raw files) you want to get into your pods without baking them into the Docker images
- That configuration data comes from some files on disk (we call this the "config repo").
- The files are either binary blobs, or YAML/JSON structured data
- You have some CI/CD mechanism you want to hook up to, so ConfigMaps are automatically generated and updated when you merge changes to your
configrepo
If you want to use this to manage secrets, you should check out k8s-secret-projector instead!
+--------------------------+
| |
| Projection Mappings Repo +-------------+ YAML Projection
| (--manifests) | | Manifests
| | |
+--------------------------+ |
|
|
+-----------------+ +---v--------------------+ +------------------+
| | config sources | | | |
| Config Repo +------------------> | generates configmaps | Output Directory |
| (--config-repo) | | k8s-config-projector +-----------------------> (--output) |
| | | | | |
+-----------------+ | | +---------+--------+
+------------------------+ |
|
|
| creates ConfigMaps
+----------------------+ | in Kubernetes
| | |
| kubectl create -f <----------------+
| |
+----------------------+
The projector can:
- Take raw files and stuff them into a ConfigMap
- Glob files in your config repo, and stuff ALL of them in your configmap
- Extract fields from your structured data (yaml/json)
- Create new structured outputs from a subset of a yaml/json source by pulling out some fields and dropping others
- Translate back and forth between JSON and YAML (convert a YAML source to a JSON output, etc)
- Support for extracting complex fields like objects+arrays from sources, and not just scalars!
You should get started by checking out some example projection manifests.
- Projection Manifests documentation: projection_manifests.md
DataSourceschema documentation: datasource.md
Check out hacking.md for how to build this!
- Have your
--config-repochecked out, and location in${CONFIG_REPO} - Have your path to your projection mappings (
--manifests) in${MANIFESTS_REPO} - Create an
${OUTPUT_DIR}to keep the generated manifests the tool spits out
$ docker run -it --rm \
-v "${OUTPUT_DIR}:/output" \
-v "${MANIFESTS_REPO}:/manifests:ro" \
-v "${CONFIG_REPO}:/config:ro" \
tumblr/k8s-config-projector:latestCollect your generated ConfigMaps in ${OUTPUT_DIR}!
$ make && ./bin/k8s-config-projector --debug --manifests=${MANIFESTS_REPO} --config-repo=${CONFIG_REPO} --output=${OUTPUT_DIR}
...Collect your generated ConfigMaps in ${OUTPUT_DIR}!
Example config map:
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: null
name: test-config-1
namespace: foo-bar
data:
test_config.txt: |
this is a config file with a whole bunch of stuff
to show that configs can be any form
blah
blah
blah
Example pod config:
apiVersion: v1
kind: Pod
metadata:
name: test-pod1
namespace: mynamespace
spec:
containers:
- name: busybox-test
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "cat /etc/config/test-config.txt" ]
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
name: test-config-1
items:
- key: test_config.txt
path: test-config.txt
restartPolicy: NeverThere are a few things to note when using the Config Projector. They are actually limits of Kubernetes/etcd, but it still is useful to be aware of them.
- ConfigMaps cannot be over 1M. Because ConfigMaps are stored in Kubernetes API, the ConfigMaps are backed by
etcd. This has a 1M limit of each object stored. - Internally,
kubectl applycreates annotations, which has a size limit of 256K. This translates to a ConfigMap that can't be over ~512K. If you want to avoid this, usekubectl createinstead.
For reference, these are size limits discussions:
Copyright 2019, Tumblr, Inc.