Skip to main content

Snowflake Troubleshooting Guide

This guide helps you resolve common issues when using msh with Snowflake as your destination.

Common Errors and Solutions​

Warehouse Suspended​

Error Message:

[ERROR] Ingest: Snowflake warehouse is suspended. Resume it in the Snowflake UI.

Solution:

  1. Log into your Snowflake account
  2. Navigate to Warehouses in the left sidebar
  3. Find your warehouse and click Resume
  4. Wait for the warehouse to start (usually takes a few seconds)
  5. Retry your msh run command

Prevention:

  • Set up auto-resume for your warehouse in Snowflake settings
  • Use a warehouse that's configured to stay running during business hours

Connection Timeout​

Error Message:

[ERROR] Ingest: Snowflake connection timeout. Check network connectivity and warehouse status.

Solutions:

  1. Check Warehouse Status:

    • Ensure your warehouse is running (not suspended)
    • Verify the warehouse name in your environment variables matches exactly

  2. Check Network Connectivity:

    • Test your internet connection
    • If behind a firewall, ensure Snowflake endpoints are whitelisted
    • Check if your IP is allowed in Snowflake network policies

  3. Check Credentials:

    # Verify your Snowflake credentials are set correctly
    echo $SNOWFLAKE_ACCOUNT
    echo $SNOWFLAKE_USER
    echo $SNOWFLAKE_DATABASE
    echo $SNOWFLAKE_WAREHOUSE

  4. Increase Timeout (if needed):

    • The default connection timeout is 30 seconds
    • For slow networks, you may need to adjust this in msh-engine/src/msh_engine/snowflake_utils.py

Quota Exceeded​

Error Message:

[ERROR] Ingest: Snowflake quota exceeded. Check your account limits.

Solutions:

  1. Check Your Account Limits:

    • Log into Snowflake and check your account usage
    • Review storage limits, compute credits, and data transfer limits

  2. Reduce Data Volume:

    • Use incremental loading instead of full replacements
    • Filter data at the source before ingestion
    • Consider using a smaller warehouse for testing

  3. Upgrade Your Account:

    • Contact Snowflake support to increase your limits
    • Consider upgrading to a higher tier plan

Authentication Failed​

Error Message:

[ERROR] Ingest: Snowflake authentication failed. Check your credentials.

Solutions:

  1. Verify Environment Variables:

    # Check all required variables are set
    env | grep SNOWFLAKE

  2. Check Credential Format:

    • Account name should be just the account identifier (e.g., xyz123), not the full URL
    • Username and password should match exactly (case-sensitive)
    • Ensure there are no extra spaces or special characters

  3. Test Connection Manually:

    import snowflake.connector
    conn = snowflake.connector.connect(
        user=os.environ['SNOWFLAKE_USER'],
        password=os.environ['SNOWFLAKE_PASSWORD'],
        account=os.environ['SNOWFLAKE_ACCOUNT'],
        warehouse=os.environ['SNOWFLAKE_WAREHOUSE'],
        database=os.environ['SNOWFLAKE_DATABASE']
    )

  4. Check User Permissions:

    • Ensure your user has CREATE SCHEMA, CREATE TABLE, and CREATE VIEW permissions
    • Verify the role has access to the warehouse

Schema Name Issues​

Error Message:

Schema name 'xxx' exceeds Snowflake max length (255)

Solutions:

  1. Use Shorter Schema Names:

    • In msh.yaml, set a shorter target_schema: 
      target_schema: "ANALYTICS"  # Max 63 chars recommended

  2. Git-Aware Schemas:

    • In dev mode, git branch names are appended to schema names
    • Use shorter branch names to avoid exceeding limits
    • Schema names are automatically sanitized and truncated

  3. Check Schema Name Format:

    • Schema names are automatically converted to uppercase
    • Special characters are replaced with underscores
    • Names starting with numbers are prefixed with underscore

Transaction Errors​

Error Message:

Snowflake error: Transaction failed

Solutions:

  1. Check Transaction Isolation:

    • Snowflake uses READ COMMITTED isolation by default
    • Ensure your operations are compatible with this isolation level

  2. Verify Connection State:

    • Connections are automatically managed by msh
    • If you see transaction errors, try running msh run again

  3. Check for Lock Conflicts:

    • Multiple concurrent operations on the same table can cause locks
    • Wait for other operations to complete before retrying

Configuration Checklist​

Before running msh with Snowflake, ensure:

  • All required environment variables are set:

    • SNOWFLAKE_ACCOUNT
    • SNOWFLAKE_USER
    • SNOWFLAKE_PASSWORD
    • SNOWFLAKE_DATABASE
    • SNOWFLAKE_WAREHOUSE
    • SNOWFLAKE_ROLE (optional)

  • Warehouse is running (not suspended)

  • User has necessary permissions:

    • CREATE SCHEMA
    • CREATE TABLE
    • CREATE VIEW
    • USAGE on warehouse

  • Network connectivity to Snowflake is available

  • Account has sufficient quota/credits

Getting More Help​

If you continue to experience issues:

  1. Run with Debug Mode:

    msh run --debug

    This will show detailed error messages and connection information.

  2. Check Logs:

    • Review .msh/logs/msh.log for detailed error traces
    • Look for Snowflake-specific error codes

  3. Test Connection Directly:

    • Use Snowflake's web UI to verify credentials work
    • Test SQL queries manually in Snowflake

  4. Contact Support:

    • For msh-specific issues: Open an issue on GitHub
    • For Snowflake issues: Contact Snowflake support with error codes

Best Practices​

  1. Use Separate Warehouses:

    • Use a dedicated warehouse for msh operations
    • This prevents conflicts with other workloads

  2. Monitor Warehouse Usage:

    • Set up alerts for warehouse suspension
    • Monitor compute credit usage

  3. Use Incremental Loading:

    • Reduces data transfer and compute costs
    • Faster pipeline execution

  4. Test in Dev Environment First:

    • Use git-aware schemas for development
    • Test changes before deploying to production

  5. Keep Schema Names Short:

    • Use descriptive but concise names
    • Remember that git suffixes are appended in dev mode