Skip to main content

Using the Dashboard

msh comes with a built-in UI that gives you full visibility into your data pipeline. It's designed to help you debug issues, understand dependencies, and track the ROI of your data platform.

To start the dashboard:

msh ui

Then visit http://localhost:3000.

The Lineage Graph​

The center of the dashboard is the Lineage Graph. It visualizes the DAG (Directed Acyclic Graph) of your assets.

  • Nodes: Represent your assets (sources, models, exposures).
  • Edges: Represent data flow dependencies.
  • Colors:
    • Cyan: Ingestion sources.
    • Aqua: Transformation models.
    • Violet: Downstream exposures (dashboards).

Interaction​

  • Click: Select a node to view its details in the Side Drawer.
  • Drag: Rearrange nodes to better visualize clusters.
  • Zoom: Scroll to zoom in/out of complex graphs.

Smart Stats & ROI​

The top bar displays "Smart Stats" that quantify the value msh is providing.

  • Columns Saved: This metric tracks how many columns were excluded from ingestion thanks to Smart Ingest.
    • Why it matters: Every column you don't ingest saves compute, storage, and API bandwidth. High numbers here indicate a highly optimized pipeline.
  • Active Assets: The number of assets currently deployed and healthy.

Debugging with the Side Drawer​

When you click an asset, the Side Drawer opens with deep inspection tools.

1. Overview​

Shows the current Version Hash, Status (Deployed/Failed), and Last Run time.

2. Code View​

Toggle between Raw and Compiled views.

  • Raw: The code exactly as you wrote it in your .msh or .sql file.
  • Compiled: The final SQL that was executed against the warehouse, with all Jinja macros ({{ source }}) resolved to actual table names. This is critical for debugging SQL errors.

3. Lineage List​

A text-based list of Upstream (parents) and Downstream (children) dependencies, useful for quick navigation.

Command Palette​

Power users can navigate the dashboard entirely via keyboard using the Command Palette.

  • Trigger: Press Cmd+K (Mac) or Ctrl+K (Windows/Linux), or click the Search bar in the sidebar.
  • Features:
    • Search: Instantly find assets by name or description.
    • Navigate: Use ↑ and ↓ arrows to select results.
    • Select: Press Enter to jump directly to the asset details.

Asset Details & Execution​

Clicking any node in the graph or selecting it from the Command Palette opens the Asset Detail View.

Tabs​

  • Overview: High-level metadata, description, and Smart Ingest efficiency stats (visualizing columns saved vs. fetched).
  • Code: View the Schema (column types) and the Compiled SQL (debugging what actually ran).
  • Quality: See the results of data quality tests (unique, not_null, etc.) with pass/fail status and execution times.

Quality Metrics​

The dashboard provides comprehensive quality metrics and test results visibility.

Test Results​

After each pipeline run, test results are automatically captured and displayed:

  • Test Status: Pass/fail indicators for each test
  • Execution Time: How long each test took to run
  • Pass Rate: Overall percentage of tests passing
  • Last Run: Timestamp of the last test execution

Quality API​

The dashboard exposes a quality metrics API endpoint:

Endpoint: GET /api/quality.json

Response:

{
  "overall": {
    "total_tests": 25,
    "passed": 23,
    "failed": 2,
    "pass_rate": 0.92
  },
  "assets": [
    {
      "name": "orders",
      "tests": [...],
      "last_run": "2024-01-15T10:30:00Z",
      "summary": {
        "total": 5,
        "passed": 4,
        "failed": 1,
        "pass_rate": 0.8
      }
    }
  ]
}

Accessing Test Results​

Test results are available in multiple ways:

  1. Via Dashboard: View quality metrics in the asset detail view
  2. Via API: Query /api/quality.json when the UI is running
  3. Via Catalog: Test results are included in msh_catalog.json after each run
  4. Via File: Test results are saved to .msh/test_results.json after each test run

Test Results Structure​

Test results are stored in .msh/test_results.json:

{
  "orders": {
    "asset_name": "orders",
    "timestamp": "2024-01-15T10:30:00Z",
    "tests": [
      {
        "name": "test_orders_id_unique",
        "status": "pass",
        "execution_time": 0.5
      },
      {
        "name": "test_orders_amount_positive",
        "status": "fail",
        "execution_time": 0.3
      }
    ],
    "summary": {
      "total": 5,
      "passed": 4,
      "failed": 1
    }
  }
}

Running Assets​

You can trigger runs directly from the UI. The "Run" button offers three modes:

  1. Run Asset: Runs only the selected asset.
  2. Run w/ Upstreams (+asset): Runs the asset and all its parents (recursive). Useful when you've changed upstream logic.
  3. Run w/ Downstreams (asset+): Runs the asset and all its children. Useful to propagate a schema change.

Live Terminal​

Triggering a run opens the Terminal Modal, which streams live logs from the msh engine. You can watch the build process, test execution, and atomic swap in real-time.