Using Vault with Kubernetes: A Comprehensive Guide

Amaresh Pelleti

Introduction

HashiCorp Vault is a powerful tool for managing secrets and protecting sensitive data. When used in conjunction with Kubernetes, it provides a secure and efficient way to manage secrets for your containerized applications. This guide will walk you through the process of integrating Vault with Kubernetes, with a focus on using annotations to configure the Vault Agent Injector.

Prerequisites

  • A running Kubernetes cluster
  • Vault server installed and configured
  • Vault Helm chart installed in your Kubernetes cluster

Vault Agent Injector

The Vault Agent Injector is a Kubernetes mutation webhook controller that configures pods to leverage Vault Agent containers for managing secrets. It uses annotations on Kubernetes pods to determine how to inject secrets.

Key Annotations

Let’s break down the key annotations used for configuring Vault with Kubernetes:

annotations:
  vault.hashicorp.com/agent-init-first: "true"
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/agent-inject-status: "update"
  vault.hashicorp.com/preserve-secret-case: "true"
  vault.hashicorp.com/agent-pre-populate-only: "true"
  vault.hashicorp.com/role: {{ .Values.vault.role | quote }}
  vault.hashicorp.com/tls-skip-verify: "false"
  vault.hashicorp.com/agent-inject-secret-creds: "{{ $.Values.vault.secretPath }}"
  vault.hashicorp.com/agent-inject-template-creds: |
    {{ with secret "{{ $.Values.vault.secretPath }}" -}}
    export USERNAME="{{ .Data.data.username }}"
    export PASSWORD="{{ .Data.data.password }}"
    {{- end }}

Annotation Explanations

  1. vault.hashicorp.com/agent-init-first: "true"
  • Ensures that the Vault Agent init container runs before any other init containers.
  1. vault.hashicorp.com/agent-inject: "true"
  • Enables Vault Agent injection for the pod.
  1. vault.hashicorp.com/agent-inject-status: "update"
  • Configures the injector to update secrets when they change in Vault.
  1. vault.hashicorp.com/preserve-secret-case: "true"
  • Preserves the case of secret names when injecting them.
  1. vault.hashicorp.com/agent-pre-populate-only: "true"
  • Configures the Vault Agent to only pre-populate secrets and not run as a long-lived process.
  1. vault.hashicorp.com/role: {{ .Values.vault.role | quote }}
  • Specifies the Vault role to use for authentication. This is templated to use a value from your Helm chart.
  1. vault.hashicorp.com/tls-skip-verify: "false"
  • Ensures that TLS verification is not skipped when communicating with Vault.
  1. vault.hashicorp.com/agent-inject-secret-creds: "{{ $.Values.vault.secretPath }}"
  • Specifies the path in Vault where the database credentials are stored. This is templated to use a value from your Helm chart.
  1. vault.hashicorp.com/agent-inject-template-creds: |
  • Defines a template for how the injected secret should be formatted. In this case, it’s setting environment variables for the database username and password.

Example Usage

Here’s an example of how you might use these annotations in a Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        vault.hashicorp.com/agent-init-first: "true"
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/preserve-secret-case: "true"
        vault.hashicorp.com/agent-pre-populate-only: "true"
        vault.hashicorp.com/role: "myapp-role"
        vault.hashicorp.com/tls-skip-verify: "false"
        vault.hashicorp.com/agent-inject-secret-creds: "secret/data/myapp/db"
        vault.hashicorp.com/agent-inject-template-creds: |
          {{ with secret "secret/data/myapp/db" -}}
          export USERNAME="{{ .Data.data.username }}"
          export PASSWORD="{{ .Data.data.password }}"
          {{- end }}
    spec:
      containers:
      - name: myapp
        image: myapp:latest
        ports:
        - containerPort: 8080
        command: ["/bin/sh"]
        args: 
        - -c
        - |
          source /vault/secrets/creds
          echo "Database Username: $USERNAME"
          # Your application start command here
          ./start-app.sh

In this updated example:

  1. We’ve added a command and args section to the container specification.
  2. The container now runs a shell script that:
    a. Sources the injected secrets file using source /vault/secrets/creds
    b. Echoes the values of the environment variables (for demonstration purposes)
    c. Starts the application using a hypothetical start-app.sh script

Note: In a production environment, you should avoid echoing sensitive information. This is just for demonstration purposes.

Using Injected Secrets in Your Application

Once the secrets are injected and sourced, your application can use them as regular environment variables. Here’s a simple example in Python:

import os

username = os.environ.get('USERNAME')
password = os.environ.get('PASSWORD')

print(f"Connecting to database as {username}")
# Use these credentials to connect to your database

Remember to never log or display these secrets in a production environment.

Verifying Secret Injection

To verify that the secrets are correctly injected and accessible:

  1. Exec into your pod:
   kubectl exec -it <pod-name> -- /bin/sh
  1. Once inside the container, you can check the contents of the secrets file:
   cat /vault/secrets/creds
  1. You can also check if the environment variables are set:
   env | grep DB_

This should display your USERNAME and PASSWORD variables.

Best Practices

  1. Use Specific Roles: Create specific Vault roles for each application to limit access to only the secrets they need.
  2. Regularly Rotate Secrets: Use Vault’s dynamic secrets feature to automatically rotate credentials.
  3. Enable Audit Logging: Enable audit logging in Vault to track secret access and changes.
  4. Use TLS: Always use TLS for communication between Vault and Kubernetes. Avoid using tls-skip-verify: "true" in production environments.
  5. Limit Secret Access: Use Kubernetes RBAC in conjunction with Vault policies to limit which pods can access which secrets.

Conclusion

Integrating Vault with Kubernetes provides a powerful way to manage secrets in your containerized applications. By using the Vault Agent Injector and the appropriate annotations, you can securely inject secrets into your pods and keep your sensitive data protected.

Remember to always follow security best practices and regularly update both Vault and your Kubernetes cluster to ensure you have the latest security patches and features.