Tanzu Application Platform (GitOps / SOPS) - create custom secrets

Published by Bill on (Updated: )

Assumed Audience: platform operators of the Tanzu Application Platform following the GitOps installation method

Tanzu Application Platform 1.5 includes beta support for GitOps installation. I’ve been using SOPS for secrets management with my installation. I quickly ran into the need to include custom secrets as part of the installation.

In this post, I outline the general process for including additional secrets with SOPS and then show how I used this to configure LetsEncrypt to issue a valid TLS certificate for the TAP GUI.

Including Additional Secrets with SOPS

At a high level, the steps for including additional encrypted secrets in your GitOps install of TAP are as follows.

  1. Create your unencrypted secret in a folder outside your git root
  2. Use SOPS to encrypt the file
  3. Place the encrypted secret in ./clusters/<<CLUSTER_NAME>>/cluster-config/config/ folder
  4. Commit and push your repo
  5. Validate that your secrets have been deployed

The detailed example below shows how I created an encrypted Secret for my LetsEncrypt ClusterIssuer. The same steps could be followed for any Secret.

Create your unencrypted secret. For example aws-route53-creds.yaml. Be sure to do this outside of your GitOps repo.

apiVersion: v1
data:
  aws-secret-key: BASE64-ENCODED-SECRET-VALUE
kind: Secret
metadata:
  name: aws-route53-creds
  namespace: cert-manager
type: Opaque

Use SOPS to encrypt the file and store it within the GitOps repo. Note it is important to give your encrypted secret a file name that ends .sops.yaml otherwise it won’t be decrypted on sync.

sops --encrypt aws-route53-creds.yaml > aws-route53-creds.sops.yaml
mv aws-route53-creds.sops.yaml <<GITOPS_ROOT>>/clusters/<<CLUSTER_NAME>>/cluster-config/config/aws-route53-creds.sops.yaml

Commit these changes to Git and push them to origin. You’ll need to wait a couple of minutes for things to sync. And then validate that your secrets have been deployed. In the example above, I’m looking for the aws-route53-creds certificate in the cert-manager namespace.

Debugging

If things don’t work as expected. Check for error messages on the tanzu-sync app.

k get app -n tanzu-sync sync -o=jsonpath='{.status.usefulErrorMessage}'

I came across both the following error messages when setting this up. Both errors were an indication that the secrets were not being decrypted when created in the cluster. In my case there were two causes of this behaviour:

  • I wasn’t storing encrypted yaml in the right location in the hierarchy: ./clusters/<<CLUSTER_NAME>>/cluster-config/config/
  • I was using .yaml as the file extension instead of .sops.yaml for my encrypted files.

This first error was returned when failing to decrypt secrets where only the data field was encrypted.

kapp: Error: create secret/aws-route53-creds (v1) namespace: cert-manager:
  Creating resource secret/aws-route53-creds (v1) namespace: cert-manager:
    API server says:
      Secret in version "v1" cannot be handled as a Secret: illegal base64 data at input byte 3 (reason: BadRequest)

This second error was returned when failing to decrypt secrets where the whole resource spec was encrypted.

kapp: Error: Expected to find kind 'ENC[AES256_GCM,data:yC8=,iv:Yb6GaRgSUXPSa+Fmr2IOtuLDg=,tag:qnigrwwGGc1UdGA==,type:str]/ENC[AES256_GCM,data:Co6P1wJc,iv:ULTB1g5JmRfQ=,tag:Z1XmxIN5/mzmQnA==,type:str]', but did not:
- Kubernetes API server did not have matching apiVersion + kind
- No matching CRD was found in given configuration

Deploy TAP 1.5 with GitOps (SOPS) and LetsEncrypt

At a high level the steps to use LetsEncrypt with TAP are as follows. For further information on using the Shared ingress issuer in TAP, please see the product documentation. This documentation outlines the supported process for using a Shared ingress issuer. The steps below outline what I did to get this working with the GitOps installation method which is currently in beta.

  • Create a ClusterIssuer
  • Create required secrets for ClusterIssuer
  • Update TAP to use ClusterIssuer
  • Commit, Push, and Wait

In the example below, I provide my configuration for using the dns01 challenge with AWS Route53. I recommend creating two ClusterIssuer resources, one for staging and one for production. Get things working with staging before switching to the production configuration.

Create your cluster issuer: ./clusters/<<CLUSTER_NAME>>/cluster-config/config/cluster-issuers.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-test
spec:
  acme:
    email: your-email@your-domain.tld
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: test-issuer-account-key
    solvers:
    - selector:
        dnsZones:
          - "<<DNS-ZONE>>"
      dns01:
        route53:
          region: us-east-1
          accessKeyID: <<AWS_USER_ACCESS_KEY>>
          secretAccessKeySecretRef:
            name: aws-route53-creds
            key: aws-secret-key
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: your-email@your-domain.tld
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: prod-issuer-account-key
    solvers:
    - selector:
        dnsZones:
          - "<<DNS-ZONE>>"
      dns01:
        route53:
          region: us-east-1
          accessKeyID: <<AWS_USER_ACCESS_KEY>>
          secretAccessKeySecretRef:
            name: aws-route53-creds
            key: aws-secret-key

Create your secret as defined above: ./clusters/<<CLUSTER_NAME>>/cluster-config/config/aws-route53-creds.sops.yaml. Note that by default the Secret is expected to be in the cert-manager namespace.

Update your tap-non-sensitive-values.yaml with a reference to your ClusterIssuer. Here I’m using the staging issuer.

shared:
  //...
  ingress_issuer: letsencrypt-test

Commit and push your changes to your GitOps origin repository. It will take a couple of minutes to sync your changes and provision the certificates.