Brainforge Knowledge

data-quality-monitoring-scripts-setup

1 min read

Data Quality Monitoring Scripts Setup

This guide covers setup for the observability monitoring table scripts in apps/platform/scripts/observability/.

Purpose

Create and sync Supabase monitoring tables used by the client observability pages:

  • monitoring config tables
  • historical layer/source/workflow/QA checks
  • current-state rollup tables

Script location

  • apps/platform/scripts/observability/client_config.template.json
  • apps/platform/scripts/observability/runtime.py
  • apps/platform/scripts/observability/run_ingestion_layer.py
  • apps/platform/scripts/observability/run_source_freshness_layer.py
  • apps/platform/scripts/observability/run_cicd_layer.py
  • apps/platform/scripts/observability/run_modeling_freshness_layer.py
  • apps/platform/scripts/observability/run_metrics_qa_layer.py

Required environment variables

  • NEXT_PUBLIC_SUPABASE_URL or BF_SUPABASE_URL
  • BF_SUPABASE_SERVICE_KEY
  • SUPABASE_DB_URL (required because each layer script runs CREATE TABLE IF NOT EXISTS)

Optional:

  • DQ_CLIENT_CONFIG_PATH (path to your client config JSON)
  • DQ_CHECKED_AT (fixed ISO timestamp for deterministic tests)

Run layers independently

From repo root:

python3 apps/platform/scripts/observability/run_ingestion_layer.py --dry-run
python3 apps/platform/scripts/observability/run_source_freshness_layer.py --dry-run
python3 apps/platform/scripts/observability/run_cicd_layer.py --dry-run
python3 apps/platform/scripts/observability/run_modeling_freshness_layer.py --dry-run
python3 apps/platform/scripts/observability/run_metrics_qa_layer.py --dry-run

Write mode (remove --dry-run):

python3 apps/platform/scripts/observability/run_ingestion_layer.py
python3 apps/platform/scripts/observability/run_source_freshness_layer.py
python3 apps/platform/scripts/observability/run_cicd_layer.py
python3 apps/platform/scripts/observability/run_modeling_freshness_layer.py
python3 apps/platform/scripts/observability/run_metrics_qa_layer.py

Completing templated collectors

Each layer script includes placeholder collector logic. Replace defaults with real providers:

  • ingestion checks (e.g. Polytomic/Fivetran/Snowflake load checks)
  • source freshness checks
  • GitHub Actions workflow checks
  • modeling freshness (dbt workflow state)
  • metrics and QA checks

Keep credentials in env vars or 1Password-backed runtime injection; do not hardcode secrets in repo files.

Graph View