Advanced Connectivity

msh supports complex integration patterns beyond simple REST APIs and SQL databases.

GraphQL Integration

Querying GraphQL endpoints requires constructing a specific query payload. msh handles this natively via the graphql source type.

Configuration

In your .msh file, use the query block to define your GraphQL query.

name: github_stargazers
source:
  type: dlt
  source_name: graphql
  endpoint: https://api.github.com/graphql
  credentials:
    token: ${GITHUB_TOKEN}

query: |
  query {
    repository(owner: "msh-tools", name: "msh") {
      stargazers(first: 100) {
        edges {
          node {
            login
            company
          }
        }
      }
    }
  }

sql: |
  -- The result is flattened automatically by dlt
  SELECT
    data__repository__stargazers__edges__node__login as username,
    data__repository__stargazers__edges__node__company as company
  FROM source('graphql', 'repository')

msh will automatically paginate through the results if the API supports standard pagination patterns (Cursor/Offset).

---

Reverse ETL Authentication

Publishing data back to operational systems (Reverse ETL) requires secure authentication. msh uses the same environment variable pattern for destinations as it does for sources.

msh publish Security

When you run msh publish, msh looks for credentials matching the DESTINATION__<NAME>__... pattern.

msh publish revenue_mart --to salesforce

Salesforce OAuth Setup

To publish to Salesforce, you need a Connected App.

1. Create Connected App in Salesforce Setup.
2. Enable OAuth Settings and select "Access and manage your data (api)".
3. Get Credentials: Consumer Key (Client ID) and Consumer Secret (Client Secret).

Environment Variables

Add these to your .env file (or CI/CD secrets):

# Salesforce Destination Credentials
DESTINATION__SALESFORCE__CREDENTIALS__CLIENT_ID=3MVG9...
DESTINATION__SALESFORCE__CREDENTIALS__CLIENT_SECRET=12345...
DESTINATION__SALESFORCE__CREDENTIALS__REFRESH_TOKEN=5Aep...
DESTINATION__SALESFORCE__CREDENTIALS__INSTANCE_URL=https://your-instance.salesforce.com

How It Works

1. msh reads the credentials from the environment.
2. It authenticates with Salesforce to get a fresh Access Token.
3. It streams the data from your warehouse to the Salesforce Bulk API.
4. It handles retries and error reporting automatically.