AI Features Overview

OSS

msh includes powerful AI features that help you understand, review, and generate data assets using natural language. These features leverage AI providers to analyze your project structure, suggest improvements, and generate code.

Cloud vs OSS

This section documents OSS (Open Source) AI features. For Cloud Platform AI features, see Cloud Platform AI Features.

Quick Start

1. Configure AI provider: Set up your AI provider and model

   msh config ai --provider openai --model gpt-4 --api-key env:OPENAI_API_KEY

2. Generate manifest: Create project metadata cache

   msh manifest

3. Use AI commands: Start using AI-powered features

   msh ai explain assets/revenue.msh

Available Commands

Understanding Assets

• msh inspect - Parse and display structured asset metadata
• msh ai explain - Get natural language explanation of what an asset does
• msh ai review - Review asset for risks and issues

Generating Assets

• msh ai new - Generate new asset from natural language description
• msh ai fix - Suggest fixes for broken assets
• msh ai tests - Generate or improve tests for an asset

Metadata Management

• msh manifest - Generate project-level manifest
• msh ai context - Generate AI-ready context pack

Glossary Management

• msh glossary add-term - Create glossary terms
• msh glossary link-term - Link terms to assets
• msh glossary list - List all glossary terms
• msh glossary export - Export glossary as JSON

How It Works

msh builds context packs from your project metadata and sends them to AI providers to generate explanations, reviews, and code. All operations are validated by a safety layer to prevent dangerous operations.

Context Packs

Context packs are AI-ready bundles containing:

• Project information (name, warehouse, schema)
• Asset metadata (schemas, lineage, tests)
• Glossary terms and policies
• Lineage graphs

See Context Packs for more details.

AI Safety Layer

The safety layer blocks dangerous operations like:

• DROP TABLE
• TRUNCATE TABLE
• DELETE FROM
• ALTER TABLE ... DROP COLUMN

See AI Safety for more details.

Metadata Cache

msh maintains a metadata cache in .msh/ directory for fast AI operations:

• manifest.json - Compiled manifest of all assets
• lineage.json - Lineage graph
• schemas.json - Flattened schemas
• tests.json - Test definitions
• glossary.json - Cached glossary

See Metadata Cache for more details.

Supported AI Providers

msh supports multiple AI providers:

• OpenAI - GPT-4, GPT-3.5, etc.
• Anthropic - Claude 3 Opus, Sonnet, etc.
• Ollama - Local models (no API key needed)
• HuggingFace - Local models
• Azure OpenAI - Azure-hosted OpenAI models

See AI Configuration for setup instructions.

Example Workflow

# 1. Configure AI
msh config ai --provider openai --model gpt-4 --api-key env:OPENAI_API_KEY

# 2. Generate manifest
msh manifest

# 3. Explain an asset
msh ai explain assets/revenue.msh

# 4. Review for issues
msh ai review assets/revenue.msh

# 5. Generate fixes if needed
msh ai fix assets/revenue.msh --apply

# 6. Generate new asset
msh ai new --name customer_revenue --apply

Next Steps

• Setup AI Configuration - Configure your AI provider
• AI Commands - Learn about individual commands
• Context Packs - Understand how context works
• AI Safety - Learn about safety features
• Glossary System - Manage business glossary