Integrating AWS Secrets Manager to EKS using External Secrets

Managing kubernetes secrets in large scale is a challenging task for many companies in a world runned by kubernetes, gitops and automation everywhere. With the advent of GitOps, engineers are encouraged to store and manage infrastructure as code (e.g. control planes, kubernetes yaml’s) in a Git repository as single source of truth but for secrets management, the single source of truth usually are secret stores such as AWS SecretsManager, Azure Keyvaults and HashiCorp Vault. So in today’s post, we are going to learn how to sync your secrets from AWS Secrets Manager to the EKS cluster using External Secrets.

The External Secrets Operator

The External Secrets Operator is an open-source Kubernetes Operator started by GoDaddy that allows you to integrate your external secret store with Kubernetes through two main resources, the SecretStore and the ExternalSecrets. In addition to loading the secrets into the cluster, the ESO also provides the ability to transform the name of the secret to suit any standard expected by the application/operator.

Creating a secret test on AWS secrets manager

As first step of the tutorial, we will use terraform to create a key/value secret and illustrate database secrets stored on AWS Secrets Manager:

## Create the secret 
resource "aws_secretsmanager_secret" "secret" {
  name = "containscloud-lab-secret"
}

## Add secret values in the key/value format
resource "aws_secretsmanager_secret_version" "secret_verison" {
  secret_id     = aws_secretsmanager_secret.secret.id
  secret_string = jsonencode({
    sqldb_host = "db01.database.com"
    sqldb_user = "app01_operator"
    sqldb_password = "@helloworld"
    oracledb_host = "db01.oracle.com"
  })
}

Configuring the IAM role and policy

Now, we’ll prepare the AWS IAM resources that will be used by the operator to authenticate and obtain the secrets. For this lab, we will create two IAM roles and a policy and associate them with the EKS cluster using EKS Pod Identity Agent.

The first role will be associated with the external secret service account and has no policy associated with it while the second role is configured in the SecretStore resource and will be assumed exclusively in this context, thus following the concept of least privilegies.

## Create the base eso-role for the operator service account
resource "aws_iam_role" "eso_role" {
  name = "eso-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
      Effect = "Allow"
      Principal = {
        Service = [
					"pods.eks.amazonaws.com"
				]
      }
    }]
  })
}

## Create the eso-role for the operator
resource "aws_iam_role" "workload01_eso_role" {
  name = "workload01-eso-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
      Effect = "Allow"
      Principal = {
        AWS = aws_iam_role.eso_role.arn
      }
    }]
  })
}

## Create the policy document with the required actions over the secret created in the last step
## Ref: https://external-secrets.io/v0.9.13/provider/aws-secrets-manager/
data "aws_iam_policy_document" "eso_policy" {
    statement {
      sid = "eso"
      effect = "Allow"
      actions = [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds",
        "secretsmanager:ListSecrets"
      ]
      resources = [
        aws_secretsmanager_secret.secret.arn
      ]
    }

    statement {
      sid = "allowListSecrets"
      effect = "Allow"
      actions = [ 
        "secretsmanager:ListSecrets"
      ]
      resources = [
        "*"
      ]
    }
}

## Create the IAM policy with the document created above
resource "aws_iam_policy" "eso_policy" {
  name        = "eso-policy"
  description = "Policy for ExternalSecretsOperator read content from SecretsManager of workload01"
  policy      = data.aws_iam_policy_document.eso_policy.json
}

## Associate the policy with the workload-eso-role
resource "aws_iam_role_policy_attachment" "attach" {
  role       = aws_iam_role.workload01_eso_role.name
  policy_arn = aws_iam_policy.eso_policy.arn
}

## Associate the eso-role with the eks cluster with our namespace and service account
## Ref: https://github.com/external-secrets/external-secrets/issues/2951#issuecomment-1866943943
resource "aws_eks_pod_identity_association" "role_association" {
  cluster_name    = "eks-lab-cluster"
  namespace       = "external-secrets"
  service_account = "external-secrets"
  role_arn        = aws_iam_role.eso_role.arn
}

Preparing the EKS cluster

Now that we have all the roles created, we are going to install the External Secrets Operator and create a default namespace named workload01:

## Add the ESO helm chart repository to your list and install it 
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

## Create the namespace and service account
kubectl create namespace workload01

The EKS cluster terraform used during the lab is available on: https://github.com/diego7marques/eks-lab-cluster

Creating the SecretStore and External Secrets

The SecretStore resource isolates the configuration related to the secret store and how the operator must authenticate against it. The SecretStore configuration is namespaced and can be used by multiple external secrets. If you want the secret store not to be namespaced, use ClusterSecretStore instead.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: containscloud-lab-store
  namespace: workload01
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      # Role that will be assumed by the opertaor to access the secret
      role: arn:aws:iam::<AWS_ACCOUNT_ID>:role/workload01-eso-role

If it works successfully, the result must be like:

The ExternalSecrets resource allows you to specify workload-related logic on it, such as rewriting the secret names or changing the format of the key/value. In the following example, we will remove the prefix sqldb_ from the secret name:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: "workload01-database-secrets"
  namespace: workload01
spec:
  secretStoreRef:
    name: containscloud-lab-store
    kind: SecretStore
  ## Get the secrets from AWS Secrets Manager
  dataFrom:
  - extract:
      key: containscloud-lab-secret
      conversionStrategy: Default
      decodingStrategy: Auto
    ## Rewrite the secret name to remove the prefix sqldb-
    rewrite:
    - regexp:
        source: ^sqldb_
        target: ""
  ## Interval to refresh the secret
  refreshInterval: 60s
  target:
    name: workload01-database
    creationPolicy: Owner
    deletionPolicy: Retain

And it’s done! After apply the last file, the secrets will be synchronized and is ready to be used 🙂

Conclusion

As we can see, the External Secrets operator offers a streamlined approach to securely import secrets to Kubernetes that simplifies the whole secret management process and increase security and maintainability. Plus, its adaptability means you’re not stuck in one secret store. So, why not simplify your life including this tool in your toolbox? 😛

The code used in this post is available on: https://github.com/diego7marques/external-secrets-aws-secrets-manager

CONTAINS(CLOUD)