Secure Prometheus Configuration with Hashicorp Vault on K8s

This post documents a very interesting option to keep passwords out of the prometheus configuration files following the official documentation. I copied the prometheus definition delivered with the istio full installation and added alertmanager and MS Teams proxy server into a simple kustomize configuration. Please checkout the whole configuration in my Github repo.

Problem Statement

Many components uses configuration files that contain sensitive information like passwords, api keys or tokens. These files are usually checked in in Git hence you have to problem, that you have to restrict access to Git more strictly in order to keep this information secret. If you are working with a password vault, you can separate code and secrets completely but you need to have some preprocessing to create a working configuration file. Yes, this file will again contain the valid credentials but it is only available if you have access to the container itself and read the file from its filesystem directly. It is easier to restrict this kind of access. In this example I’m working with HashiCorp Vault to externalize secrets.

Vault Annotations

In the official documentation you can see some use cases how the control is passed to the vault init containers or sidecars. Prometheus config files are stored in a k8s configmap mounted as a volume. So I followed the sample for a configmap.

apiVersion: apps/v1
kind: Deployment
  name: prometheus
  namespace: istio-system
      annotations: "true" "true" "true" "true" "prometheus-config" "vault-agent-injector-tls" "/vault/tls/ca.crt" "prometheus"

This is the file controlling the patch that enables HashiCorp Vault to take control with an init container. The most important line is the reference to the configmap. It contains the vault configuration file config-init.hcl and the files referenced by it.

"auto_auth" = {
  "method" = {
    "config" = {
      "role" = "prometheus"
    "mount_path" = "auth/<yourpath>"
    "type" = "kubernetes"
  "sink" ={
    "config" = {
      "path" = "/home/vault/.token"
    "type" = "file"
"exit_after_auth" = true
"pid_file" = "/home/vault/.pid"
"template" = {
  "source" = "/vault/configs/prometheus.yml"
  "destination" = "/vault/secrets/prometheus.yml"

"vault" = {
  "address" = "http://vault.vault.svc:8200"
  "ca_cert" = "/vault/tls/ca.crt"
  "client_cert" = "/vault/tls/tls.crt"
  "client_key" = "/vault/tls/tls.key"

So the template does the trick: here the source file is referenced (contained in the configmap) and transformed to the destination file within the container. You can repeat the template structure for each file you want to preprocess to replace the vault tags with the actual secrets. Let’s have a look at this source file.

Prometheus Configuration



  - job_name: 'sample configuration for hashicorp vault'
    scheme: https
      username: 'admin'
      password: '{{- with secret "<yourpath>/admin" -}}{{ }}{{- end }}'
      - targets: ['dev-m-tower-1.yourdomain:9100']
      insecure_skip_verify: true


On line 9 you can see how you can pass a consul templating statement that will evaluate to the needed password. <yourpath> has to contain a valid path to a secret in your vault installation, in this sample to a secret called “admin” with a property called “password”.

After all these consul statements are replaced by the result of its evaluation, this file will be stored under /vault/secrets/prometheus.yml within the container. All you have to do now is to point the container to this file as its main configuration source.


      - args:
        - --storage.tsdb.retention=6h
        - --config.file=/vault/secrets/prometheus.yml
        image: prom/prometheus:v2.12.0


Kustomize Configmap

The configmap itself gets created by kustomize here:




  - name: prometheus-config
      - prometheus/config/config-init.hcl
      - prometheus/config/prometheus.yml
      - prometheus/config/rules.yml


With this the configmap prometheus-config gets created. The files are stored under a key that corresponds to its filename (without path).

Final Thoughts

HashiCorp Vault is a valuable option to separate passwords from the code and supports a lot of interesting use cases. The caveat at the moment is, however, that you might encounter issues using the operators that are popping up more and more. If such an operator does not give you the possiblity to patch the created instances with the needed annotations, you will not be able to combine both approaches.

1 Trackback / Pingback

  1. Secure Logstash Configuration with Hashicorp Vault on K8s - vDan's Blogs

Leave a Reply

Your email address will not be published.