Skip to content

SKILL.md

Latest commit

 

History

History
467 lines (359 loc) · 16.2 KB

SKILL.md

File metadata and controls

467 lines (359 loc) · 16.2 KB
Raw
name atmos-stores
description Store backends: AWS SSM, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, Redis, Artifactory configuration, hooks integration, cross-component data sharing, atmos store CLI CRUD, type: store workflow step
metadata
copyright version
Copyright Cloud Posse, LLC 2026
1.0.0

Atmos External Stores

Stores are external key-value backends configured in atmos.yaml that enable components to share data outside of Terraform state. Atmos supports store providers including AWS SSM Parameter Store, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, Redis, and JFrog Artifactory.

When to Use Stores

Use stores when you need to:

  • Share data between components that is not managed by Terraform
  • Access configuration from external systems (SSM, Azure Key Vault, Redis)
  • Integrate with CI/CD pipelines that write to parameter stores
  • Store and retrieve Terraform outputs via hooks for faster cross-component reads
  • Share state across accounts, regions, or cloud providers

For Terraform-managed outputs, prefer !terraform.state (fastest) or !terraform.output. Use stores for external data or when you need a write-back mechanism via hooks.

Configuring Stores in atmos.yaml

All stores are declared under the top-level stores: key in atmos.yaml. Each store has a unique name, a kind, and provider-specific options. The legacy type field remains supported for compatibility, but kind is canonical:

# atmos.yaml
stores:
  prod/ssm:
    kind: aws/ssm
    options:
      region: us-east-1

  prod/asm:
    kind: aws/asm
    options:
      region: us-east-1

  prod/azure:
    kind: azure/keyvault
    options:
      vault_url: "https://my-keyvault.vault.azure.net/"

  prod/gcp:
    kind: gcp/secretmanager
    options:
      project_id: my-project

  cache:
    type: redis
    options:
      url: "redis://localhost:6379"

  artifacts:
    type: artifactory
    options:
      url: https://artifactory.example.com
      repo_name: my-repo

Store Naming Convention

Store names follow the pattern <environment>/<type> by convention:

  • prod/ssm -- Production SSM Parameter Store
  • dev/secrets -- Development secrets
  • shared/config -- Shared configuration store

These names are referenced in !store function calls and hook configurations.

Common Options (All Providers)

All store providers support these optional fields:

  • prefix -- String prepended to all keys (scopes the store namespace)
  • stack_delimiter -- Character used to split stack names into key path segments (defaults vary by provider)

Identity-Based Authentication

Stores that support identity-based authentication accept an identity field at the store level (not inside options). This connects the store to an Atmos auth identity for credential resolution:

stores:
  prod/ssm:
    kind: aws/ssm
    identity: prod-aws  # References an identity defined in the auth section
    options:
      region: us-east-1

Identity-based auth is supported by AWS SSM, AWS Secrets Manager, Azure Key Vault, and Google Secret Manager. It is not supported by Redis or Artifactory (a warning is logged if configured).

Store Provider Configuration

AWS SSM Parameter Store

stores:
  prod/ssm:
    kind: aws/ssm
    options:
      region: us-east-1           # Required
      prefix: myapp               # Optional: prepended to all key paths
      stack_delimiter: "/"         # Optional: default is "-"
      read_role_arn: arn:aws:iam::123456789012:role/SSMReader   # Optional: cross-account read
      write_role_arn: arn:aws:iam::123456789012:role/SSMWriter  # Optional: cross-account write

Authentication uses the AWS default credential chain (environment variables, shared credentials, instance profile). Use read_role_arn/write_role_arn for cross-account access via STS AssumeRole.

Key format: /<prefix>/<stack-parts>/<component-parts>/<key> (segments joined by /).

AWS Secrets Manager

stores:
  prod/asm:
    kind: aws/asm
    options:
      region: us-east-1
      prefix: myapp
      stack_delimiter: "/"

Use secret: true with kind: aws/asm for declared secrets managed through atmos secret and !secret.

Azure Key Vault

stores:
  prod/azure:
    kind: azure/keyvault
    options:
      vault_url: "https://my-keyvault.vault.azure.net/"  # Required
      prefix: myapp               # Optional
      stack_delimiter: "-"         # Optional: default is "-"

Authentication uses the Azure Default Credential chain (environment variables, managed identity, Azure CLI). Secret names are normalized to comply with Azure Key Vault restrictions: only alphanumeric characters and hyphens are allowed.

Key format: <prefix>-<stack-parts>-<component-parts>-<key> (segments joined by -, non-alphanumeric characters replaced with -).

Google Secret Manager

stores:
  prod/gcp:
    kind: gcp/secretmanager
    options:
      project_id: my-project     # Required
      prefix: myapp              # Optional
      stack_delimiter: "_"       # Optional: default is "-"
      credentials: '{"type":"service_account",...}'  # Optional: inline JSON credentials
      locations:                 # Optional: replication locations
        - us-east1
        - us-west1

Authentication uses the GCP default credential chain or the GOOGLE_APPLICATION_CREDENTIALS environment variable. Provide credentials inline for service account JSON. If locations is omitted, automatic replication is used.

Key format: <prefix>_<stack-parts>_<component-parts>_<key> (segments joined by _, slashes replaced with _).

Redis

stores:
  cache:
    type: redis
    options:
      url: "redis://localhost:6379"  # Required (or set ATMOS_REDIS_URL env var)
      prefix: myapp                  # Optional
      stack_delimiter: "/"           # Optional: default is "/"

The url supports Redis URL format including authentication: redis://:password@host:port/db. If url is not set, the ATMOS_REDIS_URL environment variable is used.

Key format: <prefix>/<stack-parts>/<component-parts>/<key> (segments joined by /). For !store.get, prefix is joined with : separator.

Artifactory

stores:
  artifacts:
    type: artifactory
    options:
      url: https://artifactory.example.com   # Required
      repo_name: my-repo                      # Required
      access_token: !env ARTIFACTORY_ACCESS_TOKEN  # Optional (see auth below)
      prefix: myapp                           # Optional
      stack_delimiter: "/"                    # Optional: default is "/"

Authentication uses access_token from options, or falls back to ARTIFACTORY_ACCESS_TOKEN or JFROG_ACCESS_TOKEN environment variables. Set token to "anonymous" for unauthenticated access.

Create a Generic repository type in JFrog Artifactory. Atmos stores data as JSON files, so no specific package type is required.

Key format: <repo_name>/<prefix>/<stack-parts>/<component-parts>/<key> (segments joined by /).

Reading from Stores with YAML Functions

!store -- Component-Aware Access

Reads values following the Atmos stack/component/key naming convention. The store constructs the full key path from the stack name, component name, and key:

vars:
  # Three-argument form: store + component + key (current stack implied)
  vpc_id: !store prod/ssm vpc vpc_id

  # Four-argument form: store + stack + component + key
  vpc_id: !store prod/ssm plat-ue2-prod vpc vpc_id

  # Dynamic stack reference using Go templates
  vpc_id: !store prod/ssm {{ .stack }} vpc vpc_id

  # With default value for cold-start scenarios
  api_key: !store prod/ssm config api_key | default "not-set"

  # With YQ query to extract nested data
  db_host: !store prod/ssm database config | query .host

  # Extract from list
  first_subnet: !store prod/ssm vpc subnet_ids | query .[0]

Dynamic stack construction using printf:

vars:
  # Cross-tenant reference
  vpc_id: !store prod/ssm {{ printf "net-%s-%s" .vars.environment .vars.stage }} vpc vpc_id

  # Full context-based stack name
  config: !store prod/ssm {{ printf "%s-%s-%s" .vars.tenant .vars.environment .vars.stage }} config settings

!store.get -- Arbitrary Key Access

Reads arbitrary keys directly from a store without the stack/component/key convention. Use this for values written by external systems or global configuration:

vars:
  # Direct key access
  db_password: !store.get prod/ssm /myapp/prod/db/password

  # With default value
  feature_flag: !store.get prod/ssm /features/new-feature | default "disabled"

  # With YQ query
  api_key: !store.get cache app-config | query .api.key

  # Dynamic key with templates
  config: !store.get cache "config-{{ .vars.region }}"

Key differences between !store and !store.get:

Feature !store !store.get
Key construction Builds from stack/component/key Uses exact key as provided
Use case Atmos-managed component outputs External systems, global config
Typical pattern prefix/stack/component/key Any format the store supports

atmos.Store -- Go Template Access

Read from stores within Go template expressions:

vars:
  vpc_id: '{{ atmos.Store "prod/ssm" .stack "vpc" "vpc_id" }}'
  config: !template '{{ (atmos.Store "cache" .stack "config" "config_map").defaults | toJSON }}'

Writing to Stores with Hooks

Hooks write Terraform outputs to stores after atmos terraform apply or atmos terraform deploy. Configure hooks at any level (global, terraform-level, component-level) and Atmos deep-merges them:

# Full hook definition on a component
components:
  terraform:
    vpc:
      hooks:
        store-outputs:
          events:
            - after-terraform-apply
          command: store
          name: prod/ssm
          outputs:
            vpc_id: .vpc_id
            private_subnet_ids: .private_subnet_ids
            public_subnet_ids: .public_subnet_ids

Output values starting with . reference Terraform output names. The hook retrieves these from the Terraform state and writes them to the configured store.

DRY Hook Configuration (Layered)

Split hook configuration across inheritance levels to avoid repetition:

# stacks/catalog/vpc/_defaults.yaml -- global level
hooks:
  store-outputs:
    events:
      - after-terraform-apply
    command: store

# stacks/orgs/acme/plat/prod/_defaults.yaml -- account level
terraform:
  hooks:
    store-outputs:
      name: prod/ssm

# stacks/orgs/acme/plat/prod/us-east-2.yaml -- component level
components:
  terraform:
    vpc:
      hooks:
        store-outputs:
          outputs:
            vpc_id: .vpc_id

Atmos merges these into a complete hook definition at resolution time.

Write to Stores with the CLI or a Workflow Step

For raw CRUD access to any configured store, use the atmos store CLI command family or the type: store workflow step. This access works for any store, not only Terraform outputs. Neither method requires a declaration. Both operate directly on any store configured under stores:, by name.

# CLI: set, get, delete, list -- scope to a stack and component, or omit for a global value
atmos store set app-metadata image_tag sha256:abc123 --stack=prod --component=ecs-service
atmos store get app-metadata image_tag --stack=prod --component=ecs-service
atmos store list
atmos store list app-metadata --stack=prod --component=ecs-service

Passing a store name to atmos store list lists the key/value pairs stored under a scope (instead of the configured backends themselves), for backends that support key enumeration. Most backends support it. 1Password and the default system keychain backend do not, because their underlying APIs do not support enumeration. Check the Listable column in a bare atmos store list before relying on it for a given store. Values are masked the same way atmos store get masks a single value.

# Workflow, custom-command, or hook step: write a value, for example an image tag from a build step
- name: record-tag
  type: store
  action: write
  with:
    store: app-metadata
    key: image_tag
    value: "{{ .steps.push.metadata.digest }}"
    stack: prod
    component: ecs-service

Atmos allows you to write to a secret: true store this way, for example to write a generated password. But this write skips the atmos secret declaration and scope system. When a value must be tracked as a formal secret, use secrets.vars and atmos secret set instead. See the atmos-secrets skill for that system. See the atmos-steps skill and the /workflows/steps/type/store docs for the step type.

Cross-Account and Cross-Region Access

AWS Cross-Account via Role Assumption

stores:
  prod/ssm:
    kind: aws/ssm
    options:
      region: us-east-1
      read_role_arn: arn:aws:iam::123456789012:role/SSMReader
      write_role_arn: arn:aws:iam::123456789012:role/SSMWriter

Atmos uses STS AssumeRole to obtain temporary credentials for the target account. Separate read and write roles allow least-privilege access.

Multi-Region Configuration

Define separate stores per region:

stores:
  prod-us/ssm:
    kind: aws/ssm
    options:
      region: us-east-1

  prod-eu/ssm:
    kind: aws/ssm
    options:
      region: eu-west-1

Reference the appropriate store in each stack's configuration.

End-to-End Example: VPC to EKS

  1. Configure the store in atmos.yaml:
stores:
  prod/ssm:
    kind: aws/ssm
    options:
      region: us-east-1
  1. Set up hooks on VPC to write outputs after apply:
# stacks/catalog/vpc/_defaults.yaml
hooks:
  store-outputs:
    events:
      - after-terraform-apply
    command: store
    name: prod/ssm
    outputs:
      vpc_id: .vpc_id
      private_subnet_ids: .private_subnet_ids
  1. Read stored values in EKS component:
# stacks/prod/us-east-1.yaml
components:
  terraform:
    eks:
      vars:
        vpc_id: !store prod/ssm vpc vpc_id
        subnet_ids: !store prod/ssm vpc private_subnet_ids

Security Best Practices

  • Secrets exposure: !store, !store.get, and atmos.Store read values in cleartext and reject stores marked secret: true. Use secret: true plus !secret and atmos secret for sensitive values.
  • Least privilege: Use read_role_arn/write_role_arn to separate read and write permissions. Grant only the permissions each operation needs.
  • Environment variables for tokens: Never hardcode access tokens. Use !env or environment variables (ARTIFACTORY_ACCESS_TOKEN, JFROG_ACCESS_TOKEN, ATMOS_REDIS_URL).
  • Cold-start handling: Always provide | default values for store lookups that may reference unprovisioned components.
  • DR implications: Be cautious with cross-region store references. If a region goes down, stores in that region become unavailable.
  • Permission scoping: When using atmos describe affected with !store references, Atmos needs read access to all referenced stores. Limited permissions (e.g., dev-only) will cause failures when referencing production stores.

Troubleshooting

Problem Cause Solution
store type not found Invalid kind/type in store config Use one of: aws/ssm, aws/asm, azure/keyvault, gcp/secretmanager, redis, artifactory
region is required Missing region for SSM store Add region to store options
vault_url is required Missing vault_url for Azure Add vault_url to store options
project_id is required Missing project_id for GCP Add project_id to store options
failed to parse redis url Invalid Redis URL format Use format redis://:password@host:port/db
access_token must be set Missing Artifactory token Set access_token in options or ARTIFACTORY_ACCESS_TOKEN env var
Key not found errors Component not yet provisioned Add a default fallback value to the !store call
Permission denied Insufficient IAM/RBAC permissions Check role ARNs, vault policies, or service account permissions
Identity warning logged Identity set on unsupported provider Remove identity from Redis and Artifactory stores

Reference

For detailed provider configuration, authentication patterns, and advanced hook integration, see references/store-providers.md.