Skip to main content

Security & Secrets

msh is designed with security-first principles, ensuring that sensitive credentials are never exposed in logs, command-line arguments, or version control.

Secret Injection​

msh relies entirely on Environment Variables for secret management. We follow the dlt convention for ingestion and standard dbt env vars for transformation.

How it Works​

When you define a credential in your .msh file using the ${VAR_NAME} syntax, msh does not substitute the value during compilation. Instead, it passes the reference to the execution engine.

# Safe
credentials: ${STRIPE_API_KEY}

The actual value is resolved only at runtime by the underlying engine (dlt or dbt), ensuring the static artifacts (compiled SQL/YAML) never contain secrets.

Subprocess Safety​

msh orchestrates tools like dbt and dlt by spawning subprocesses.

  • No CLI Arguments: Secrets are never passed as command-line arguments. This prevents them from appearing in process listings (ps aux) or shell history.
  • Environment Inheritance: Secrets are passed strictly via the process environment (os.environ).

Example: dbt Execution​

When msh runs dbt, it constructs the environment dictionary explicitly:

# Internal msh logic
env = os.environ.copy()
env['POSTGRES_PASSWORD'] = os.getenv('POSTGRES_PASSWORD')

subprocess.run(['dbt', 'run'], env=env)

Docker & CI/CD Best Practices​

When deploying msh in containerized environments (Kubernetes, GitHub Actions), follow these practices:

GitHub Actions​

Inject secrets into the step environment, not as build args.

- name: Run msh pipeline
  run: msh run --env prod
  env:
    # Source Credentials
    STRIPE_API_KEY: ${{ secrets.STRIPE_API_KEY }}
    
    # Destination Credentials
    DESTINATION__SNOWFLAKE__PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}

Docker​

Do not bake secrets into your Docker image.

BAD:

ENV STRIPE_KEY=sk_live_...

GOOD: Pass them at runtime:

docker run -e STRIPE_KEY=$STRIPE_KEY my-msh-image

Role-Based Access Control (RBAC)​

msh operates with the permissions of the user/role provided in the connection string.

  • Ingestion: Requires CREATE TABLE, INSERT on the raw schema.
  • Transformation: Requires CREATE VIEW, CREATE TABLE on the staging schema.
  • Swap: Requires DROP VIEW, CREATE VIEW on the public schema.

We recommend creating a dedicated service account (e.g., msh_user) with least-privilege access scoped to the specific schemas it manages.