Secrets Management#

Best practices and tools for managing sensitive data in Kubernetes.

Overview#

Kubernetes secrets are used to store sensitive information such as:

Passwords and API tokens
TLS certificates
SSH keys
OAuth credentials
Database connection strings

Creating Secrets#

From Literal Values#

kubectl create secret generic my-secret \
  --from-literal=username=admin \
  --from-literal=password=secret123 \
  -n <namespace>

From Files#

# Single file
kubectl create secret generic my-secret \
  --from-file=config.json \
  -n <namespace>

# Multiple files
kubectl create secret generic my-secret \
  --from-file=ssh-privatekey=~/.ssh/id_rsa \
  --from-file=ssh-publickey=~/.ssh/id_rsa.pub \
  -n <namespace>

From Environment File#

# Create .env file
cat > .env <<EOF
USERNAME=admin
PASSWORD=secret123
API_KEY=abc123xyz
EOF

# Create secret
kubectl create secret generic my-secret \
  --from-env-file=.env \
  -n <namespace>

# Clean up
rm .env

TLS Secrets#

kubectl create secret tls my-tls-secret \
  --cert=path/to/cert.crt \
  --key=path/to/key.key \
  -n <namespace>

Docker Registry Secrets#

kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=user \
  --docker-password=password \
  --docker-email=user@example.com \
  -n <namespace>

Using Secrets in Pods#

As Environment Variables#

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: app
    image: myapp
    env:
    - name: USERNAME
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: username
    - name: PASSWORD
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: password

All Secret Keys as Environment Variables#

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: app
    image: myapp
    envFrom:
    - secretRef:
        name: my-secret

As Mounted Files#

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: app
    image: myapp
    volumeMounts:
    - name: secret-volume
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: secret-volume
    secret:
      secretName: my-secret

This creates files in /etc/secrets/ with filenames as keys and contents as values.

Specific Keys as Files#

volumes:
- name: secret-volume
  secret:
    secretName: my-secret
    items:
    - key: username
      path: credentials/username.txt
    - key: password
      path: credentials/password.txt

Managing Secrets#

View Secrets#

# List secrets
kubectl get secrets -n <namespace>

# View secret metadata (not values)
kubectl describe secret my-secret -n <namespace>

# View secret YAML (values are base64 encoded)
kubectl get secret my-secret -n <namespace> -o yaml

# Decode a specific key
kubectl get secret my-secret -n <namespace> -o jsonpath='{.data.password}' | base64 -d

Update Secrets#

# Edit directly
kubectl edit secret my-secret -n <namespace>

# Or recreate
kubectl delete secret my-secret -n <namespace>
kubectl create secret generic my-secret \
  --from-literal=password=newpassword \
  -n <namespace>

# Or patch
kubectl patch secret my-secret -n <namespace> \
  -p '{"data":{"password":"'$(echo -n newpassword | base64)'"}}'

Delete Secrets#

kubectl delete secret my-secret -n <namespace>

Best Practices#

1. Never Commit Secrets to Git#

Add to .gitignore:

# Secrets
secrets.sh
*.env
*-secret.yaml
kubeconfig*
*.key
*.pem

2. Use Separate Files for Secrets#

Create template files:

# secrets.sh.example
export DB_PASSWORD="CHANGE_ME"
export API_KEY="CHANGE_ME"

Users copy and customize:

cp secrets.sh.example secrets.sh
# Edit secrets.sh with actual values

3. Limit Secret Access#

Use RBAC to restrict secret access:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: default
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["specific-secret"]
  verbs: ["get"]

4. Rotate Secrets Regularly#

# Example rotation script
NEW_PASSWORD=$(openssl rand -base64 32)

kubectl patch secret my-secret -n <namespace> \
  -p '{"data":{"password":"'$(echo -n $NEW_PASSWORD | base64)'"}}'

# Update applications to use new password
kubectl rollout restart deployment/my-app -n <namespace>

5. Use Namespaces for Isolation#

Create secrets in appropriate namespaces:

kubectl create secret generic db-secret \
  --from-literal=password=secret123 \
  -n production

kubectl create secret generic db-secret \
  --from-literal=password=devpass \
  -n development

6. Encrypt Secrets at Rest#

Enable encryption at rest in K3s by creating an encryption configuration.

7. Use External Secret Managers#

Consider using external secret managers for production:

HashiCorp Vault
AWS Secrets Manager
Azure Key Vault
External Secrets Operator

Secret Management Scripts#

Helper Script for Setting Secrets#

#!/bin/bash
# set-secret.sh

NAMESPACE=${1:-default}
SECRET_NAME=${2:-my-secret}

# Prompt for values
read -p "Enter username: " USERNAME
read -sp "Enter password: " PASSWORD
echo

# Create secret
kubectl create secret generic $SECRET_NAME \
  --from-literal=username=$USERNAME \
  --from-literal=password=$PASSWORD \
  -n $NAMESPACE \
  --dry-run=client -o yaml | kubectl apply -f -

echo "Secret $SECRET_NAME created/updated in namespace $NAMESPACE"

Usage:

chmod +x set-secret.sh
./set-secret.sh production db-credentials

Backup Secrets#

#!/bin/bash
# backup-secrets.sh

NAMESPACE=${1:-default}
OUTPUT_DIR="secrets-backup-$(date +%Y%m%d)"

mkdir -p $OUTPUT_DIR

kubectl get secrets -n $NAMESPACE -o json | \
  jq -r '.items[] | select(.type=="Opaque") | .metadata.name' | \
  while read secret; do
    kubectl get secret $secret -n $NAMESPACE -o yaml > "$OUTPUT_DIR/${secret}.yaml"
  done

echo "Secrets backed up to $OUTPUT_DIR"

Restore Secrets#

#!/bin/bash
# restore-secrets.sh

BACKUP_DIR=$1
NAMESPACE=${2:-default}

if [ -z "$BACKUP_DIR" ]; then
  echo "Usage: $0 <backup-directory> [namespace]"
  exit 1
fi

for file in $BACKUP_DIR/*.yaml; do
  kubectl apply -f $file -n $NAMESPACE
done

echo "Secrets restored to namespace $NAMESPACE"

Common Secret Patterns#

Database Credentials#

kubectl create secret generic postgres-secret \
  --from-literal=postgres-user=dbuser \
  --from-literal=postgres-password=dbpass123 \
  --from-literal=postgres-db=mydb \
  -n default