Managing kubernetes secrets in large scale is a challenging task for many companies in a world runned by kubernetes, gitops and automation everywhere. With the advent of GitOps, engineers are encouraged to store and manage infrastructure as code (e.g. control planes, kubernetes yaml’s) in a Git repository as single source of truth but for secrets management, the single source of truth usually are secret stores such as AWS SecretsManager, Azure Keyvaults and HashiCorp Vault. So in today’s post, we are going to learn how to sync your secrets from AWS Secrets Manager to the EKS cluster using External Secrets.
The External Secrets Operator
The External Secrets Operator is an open-source Kubernetes Operator started by GoDaddy that allows you to integrate your external secret store with Kubernetes through two main resources, the SecretStore and the ExternalSecrets. In addition to loading the secrets into the cluster, the ESO also provides the ability to transform the name of the secret to suit any standard expected by the application/operator.
Creating a secret test on AWS secrets manager
As first step of the tutorial, we will use terraform to create a key/value secret and illustrate database secrets stored on AWS Secrets Manager:
## Create the secret
resource "aws_secretsmanager_secret" "secret" {
name = "containscloud-lab-secret"
}
## Add secret values in the key/value format
resource "aws_secretsmanager_secret_version" "secret_verison" {
secret_id = aws_secretsmanager_secret.secret.id
secret_string = jsonencode({
sqldb_host = "db01.database.com"
sqldb_user = "app01_operator"
sqldb_password = "@helloworld"
oracledb_host = "db01.oracle.com"
})
}Configuring the IAM role and policy
Now, we’ll prepare the AWS IAM resources that will be used by the operator to authenticate and obtain the secrets. For this lab, we will create two IAM roles and a policy and associate them with the EKS cluster using EKS Pod Identity Agent.
The first role will be associated with the external secret service account and has no policy associated with it while the second role is configured in the SecretStore resource and will be assumed exclusively in this context, thus following the concept of least privilegies.
## Create the base eso-role for the operator service account
resource "aws_iam_role" "eso_role" {
name = "eso-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = [
"sts:AssumeRole",
"sts:TagSession"
]
Effect = "Allow"
Principal = {
Service = [
"pods.eks.amazonaws.com"
]
}
}]
})
}
## Create the workload01-eso-role for the SecretStore
resource "aws_iam_role" "workload01_eso_role" {
name = "workload01-eso-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = [
"sts:AssumeRole",
"sts:TagSession"
]
Effect = "Allow"
Principal = {
AWS = aws_iam_role.eso_role.arn
}
}]
})
}
## Create the policy document with the required actions over the secret created in the last step
## Ref: https://external-secrets.io/v0.9.13/provider/aws-secrets-manager/
data "aws_iam_policy_document" "eso_policy" {
statement {
sid = "eso"
effect = "Allow"
actions = [
"secretsmanager:GetResourcePolicy",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds",
"secretsmanager:ListSecrets"
]
resources = [
aws_secretsmanager_secret.secret.arn
]
}
statement {
sid = "allowListSecrets"
effect = "Allow"
actions = [
"secretsmanager:ListSecrets"
]
resources = [
"*"
]
}
}
## Create the IAM policy with the document created above
resource "aws_iam_policy" "eso_policy" {
name = "eso-policy"
description = "Policy for ExternalSecretsOperator read content from SecretsManager of workload01"
policy = data.aws_iam_policy_document.eso_policy.json
}
## Associate the policy with the workload-eso-role
resource "aws_iam_role_policy_attachment" "attach" {
role = aws_iam_role.workload01_eso_role.name
policy_arn = aws_iam_policy.eso_policy.arn
}
## Associate the eso-role with the eks cluster with our namespace and service account
## Ref: https://github.com/external-secrets/external-secrets/issues/2951#issuecomment-1866943943
resource "aws_eks_pod_identity_association" "role_association" {
cluster_name = "eks-lab-cluster"
namespace = "external-secrets"
service_account = "external-secrets"
role_arn = aws_iam_role.eso_role.arn
}Installing the operator
Now that we have all the roles created, we are going to install the External Secrets Operator and create a default namespace named workload01:
## Add the ESO helm chart repository to your list and install it
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace
## Create the namespace
kubectl create namespace workload01The EKS cluster terraform used during the lab is available on: https://github.com/diego7marques/eks-lab-cluster
Creating the SecretStore and External Secrets
The SecretStore resource isolates the configuration related to the secret store and how the operator must authenticate against it. The SecretStore configuration is namespaced and can be used by multiple external secrets. If you want the secret store not to be namespaced, use ClusterSecretStore instead.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: containscloud-lab-store
namespace: workload01
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
# Role that will be assumed by the opertaor to access the secret
role: arn:aws:iam::<AWS_ACCOUNT_ID>:role/workload01-eso-roleIf it works successfully, the result must be like:
The ExternalSecrets resource allows you to specify workload-related logic on it, such as rewriting the secret names or changing the format of the key/value. In the following example, we will remove the prefix sqldb_ from the secret name:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: "workload01-database-secrets"
namespace: workload01
spec:
secretStoreRef:
name: containscloud-lab-store
kind: SecretStore
## Get the secrets from AWS Secrets Manager
dataFrom:
- extract:
key: containscloud-lab-secret
conversionStrategy: Default
decodingStrategy: Auto
## Rewrite the secret name to remove the prefix sqldb-
rewrite:
- regexp:
source: ^sqldb_
target: ""
## Interval to refresh the secret
refreshInterval: 60s
target:
name: workload01-database
creationPolicy: Owner
deletionPolicy: RetainAnd it’s done! After apply the last file, the secrets will be synchronized and is ready to be used 🙂
Conclusion
As we can see, the External Secrets operator offers a streamlined approach to securely import secrets to Kubernetes that simplifies the whole secret management process and increase security and maintainability. Plus, its adaptability means you’re not stuck in one secret store. So, why not simplify your life including this tool in your toolbox? 😛
The code used in this post is available on: https://github.com/diego7marques/external-secrets-aws-secrets-manager
Discover more from contains(cloud)
Subscribe to get the latest posts sent to your email.
Be First to Comment