Skip to main content

Developer Workflow

msh provides a suite of tools designed to make the development lifecycle fast, consistent, and safe. This guide walks you through the standard workflow for creating and deploying assets.

1. Scaffolding (msh create)​

Start by creating a new asset using the built-in generator. This ensures you start with a production-ready template that follows best practices.

msh create asset revenue_mart

This creates models/revenue_mart.msh with:

  • Lifecycle Configuration: Blue/Green deployment enabled.
  • Incremental Logic: Pre-configured for append/merge strategies.
  • Data Quality Tests: Placeholder not_null and unique tests.
  • Expose Block: Example dashboard configuration.

2. Formatting (msh fmt)​

Keep your codebase clean and consistent. msh fmt automatically formats your YAML structure, SQL queries, and Python code blocks.

msh fmt

What it does:

  • Indents YAML blocks (2 spaces).
  • Uppercases SQL keywords (SELECT, FROM, WHERE).
  • Formats Python code using black.

3. Validation (msh validate)​

Before you run or commit your code, use the static analyzer to catch errors early. This is also perfect for CI/CD pipelines.

msh validate

Checks:

  • Syntax: Validates YAML structure.
  • Schema: Ensures required fields are present.
  • Safety: Checks for unclosed Jinja tags.
  • Security: Scans for hardcoded credentials.

4. Simulation (msh plan)​

Preview what will happen when you run your pipeline.

msh plan

This compiles your project and shows you which assets will be created, updated, or deleted, without actually touching your data.

5. Execution (msh run)​

Run your pipeline.

msh run

6. Data Sampling (msh sample)​

Quickly preview data from your assets for testing and debugging.

# Preview latest data
msh sample orders

# Check raw data quality
msh sample orders --source raw --limit 50

# Create test dataset
msh sample orders --size 1000

Use cases:

  • Verify data quality before deployment
  • Debug transformation issues
  • Create smaller test datasets
  • Inspect raw vs transformed data

See Sample Command for complete documentation.

7. Monitoring (msh status & msh versions)​

Track the state of your deployments.

Check Active Versions​

See what is currently live in production:

msh status

Output:

Project Status
┌──────────────┬─────────────┐
│ Asset        │ Active Hash │
├──────────────┼─────────────┤
│ revenue_mart │ a1b2        │
└──────────────┴─────────────┘

View History​

See the deployment history for a specific asset:

msh versions revenue_mart

8. Quality Monitoring​

After running tests, view quality metrics:

# Test results are automatically saved to .msh/test_results.json
# View in dashboard
msh ui

# Or check catalog
cat .msh/msh_catalog.json

Test results include:

  • Pass/fail status for each test
  • Execution times
  • Overall pass rate
  • Last run timestamp

See Using the Dashboard for more details.

9. Rollback​

If something goes wrong, instantly revert to a previous version:

msh rollback revenue_mart