Secrets in microservices¶

The microservices running in kubernetes are supposed to use secrets to store all sensitive variables. Service's pod references the secret in a manifest and its content is populated as environment variables.

In general, there are two kinds of resources that require secrets:

terraform generated resources, for example credentials to dbs, redis, kafka topics, etc.
resources external to new infra, for example credentials to dbs in legacy infrastructure

Terraform generated secrets¶

When a resource gets generated by terraform, its sensitive data are stored in a kubernetes secret called \<service-name>-extra-envs. Example:

-> kubectl describe secret tbl-extra-envs
Name:         tbl-extra-envs
Namespace:    tbl-api-dev
.
Type:  Opaque
.
Data
====
REDIS_DSN:       114 bytes
DB_DSN:          67 bytes
KAFKA_PASSWORD:  64 bytes

This "terraform knows what terraform knows" approach eliminates manual copying of sensitive data.

External secrets¶

The external secrets are stored with Sealed Secrets that are encrypted and safe to store in git repositories. The SealedSecret can be decrypted only by the controller running in the target cluster and target namespace. Nobody else is able to obtain the original Secret from the SealedSecret.

Prerequisites for SealedSecrets operations¶

kubeseal tool installed
Readonly access to the namespace (for creating SealedSecret locally).
(Optional) Read/Write access to application namespace, for applying the SealedSecret from laptop

Create a sealed secret¶

# naming convention <service-name>-secrets
SECRET_NAME="kotlin-example-secrets"
SECRET_NS="kotlin-spring-example"

# switch to target cluster and namespace
kubectx dev
kubens ${SECRET_NS}

# generate the secret and sealed-secret
kubectl create secret generic ${SECRET_NAME} --namespace ${SECRET_NS} --dry-run=client --from-literal=PASSWORD="12345" -o yaml | \
kubeseal --controller-namespace infra --controller-name sealedsecrets-sealed-secrets -o yaml > ${SECRET_NAME}-sealed.yaml

# ${SECRET_NAME}-sealed.yaml can be stored in git and/or applied
# This works only if you have R/W access to namespace of application. In prodoction, call this in Gitlab CI/CD
kubectl apply -f ${SECRET_NAME}-sealed.yaml

kubectl get sealedsecret

NAME                     AGE
kotlin-example-secrets   5s

# sealed secret is unsealed by controller and regular secret is created
# This might not work for you in production
kubectl get secret

NAME                                           TYPE                                  DATA   AGE
kotlin-example-secrets                         Opaque                                1      16s

Once your SealedSecret is deployed you can deploy your application and reference it in your deployment manifest.

Deploying SealedSecret via Gitlab CI/CD¶

As mentioned above, SealedSecret objects are safe to store in git. To keep the secret updated, make sure you deploy the SealedSecret together with your application code. kubectl apply is idempotent, so it's safe to run in multiple time even when no changes happened. Modify your deployment pipeline to something like this:

# I assume that SealedSecret file is in your app repostory in path: .gitlab/secrets/my-secret-prod.yaml
.deploy:
  stage: deploy
  image:
    name: alpine/k8s:1.28.9 # needed for Helm + kubectl tools in image
    entrypoint: [ "/bin/sh", "-c" ]
  when: manual
  before_script:
    - helm repo add --username cicd_token --password $HELM_REGISTRY_PASSWORD ftmo-helm-charts $CI_API_V4_URL/projects/56/packages/helm/stable

deploy-job:
  extends: .deploy
  environment:
    name: prod
    url: https://gw2.ftmo.com
    kubernetes:
      namespace: my-app-namespace-prod
  script:
    - kubectl apply -f .gitlab/secrets/my-secret-prod.yaml
    - |
      helm upgrade --install ... \ # rest of Helm command to be deployed

Add value-key to an existing sealed secret¶

Use the --merge-into command option to update an existing sealed secret:

# naming convention <service-name>-secrets
SECRET_NAME="kotlin-example-secrets"
SECRET_NS="kotlin-spring-example"

# switch to target cluster and namespace
kubectx dev
kubens ${SECRET_NS}

# create tmp secret with new key-value
kubectl create secret generic ${SECRET_NAME} --namespace ${SECRET_NS} --dry-run=client --from-literal=API_KEY="abcde" -o yaml | \
# merge it into existing file
kubeseal --controller-namespace infra --controller-name sealedsecrets-sealed-secrets --merge-into ${SECRET_NAME}-sealed.yaml -o yaml

# apply it
kubectl apply -f ${SECRET_NAME}-sealed.yaml

Remove value-key from an existing sealed secret¶

Simply edit the sealed secret and remove desired value-key pair line from it. Then apply.