Skip to main content

msh ai tests

Use AI to generate or improve tests for an asset.

Usage​

msh ai tests <asset_path> [--apply] [--json]

Description​

Analyzes an asset and suggests appropriate tests:

  • Reviews asset schema
  • Identifies missing tests
  • Suggests test improvements
  • Generates JSON patch with test additions

Options​

  • --apply: Apply suggested test changes to the file
  • --json: Return suggested tests as JSON structure

Examples​

Suggest Tests (Preview Only)​

msh ai tests assets/revenue.msh

Example Output:

Asset: revenue

Suggested Tests:
1. Unique constraint on [customer_id, month]
   → Ensures no duplicate customer-month combinations

2. Not null check on customer_id
   → Ensures customer_id is always present

3. Not null check on monthly_revenue
   → Ensures revenue is always calculated

4. Assert: monthly_revenue > 0
   → Ensures revenue is positive

Patch Preview:
--- assets/revenue.msh
+++ assets/revenue.msh
@@ -15,6 +15,12 @@
   GROUP BY customer_id, DATE_TRUNC('month', order_date)
 
+tests:
+  - unique: [customer_id, month]
+  - not_null: [customer_id, monthly_revenue]
+  - assert: "monthly_revenue > 0"

Apply these tests? (y/n):

Apply Test Changes Automatically​

msh ai tests assets/revenue.msh --apply

Applies test changes automatically after showing preview.

Get Suggestions as JSON​

msh ai tests assets/revenue.msh --json

Example JSON Output:

{
  "suggested_tests": [
    {
      "type": "unique",
      "columns": ["customer_id", "month"]
    },
    {
      "type": "not_null",
      "columns": ["customer_id", "monthly_revenue"]
    },
    {
      "type": "assert",
      "sql": "monthly_revenue > 0"
    }
  ],
  "patch": {
    "patches": [{
      "file_path": "assets/revenue.msh",
      "operations": [
        {
          "op": "add",
          "path": "/tests",
          "value": [
            {"unique": ["customer_id", "month"]},
            {"not_null": ["customer_id", "monthly_revenue"]},
            {"assert": "monthly_revenue > 0"}
          ]
        }
      ]
    }]
  }
}

Test Types Generated​

AI suggests appropriate tests based on asset schema:

Unique Constraints​

  • Identifies natural keys
  • Suggests unique constraints on key columns

Not Null Checks​

  • Identifies required columns
  • Suggests not null checks

Assertions​

  • Suggests business logic assertions
  • Identifies data quality checks

Relationships​

  • Identifies foreign key relationships
  • Suggests relationship tests

Prerequisites​

  1. AI Configuration: Must configure AI provider first

    msh config ai --provider openai --model gpt-4 --api-key env:OPENAI_API_KEY

  2. Manifest: Generate manifest before using AI commands

    msh manifest

  3. Asset Schema: Asset must have a defined schema

Use Cases​

  • Missing Tests: Generate tests for assets without tests
  • Improve Coverage: Add tests to improve coverage
  • Best Practices: Follow testing best practices
  • Quality Assurance: Ensure data quality

Customizing Generated Tests​

After generation, you can:

  1. Review tests: Check suggested tests make sense
  2. Modify tests: Adjust tests to match your requirements
  3. Add more tests: Add additional tests manually
  4. Use test suites: Reference test suites from msh.yaml

Test Suites​

You can combine generated tests with test suites:

# msh.yaml
test_suites:
  staging:
    - assert: "id IS NOT NULL"
    - unique: id
# assets/revenue.msh
test_suites:
  - staging  # Use suite
tests:
  - assert: "monthly_revenue > 0"  # Add specific test