Brainforge Knowledge

production-deployment-investigation-2026-03-07

4 min read

Production Deployment Investigation — 2026-03-07

Context: Pushed Rill code (dashboards, models) but changes not visible in production.

Summary

Rill dashboards and models do not auto-deploy. Pushing to GitHub does not update Rill Cloud. You must run rill project deploy manually after merge.

1. Rill Cloud Deployment (dashboards & models)

How it works

  • Source: rill/delivery/ → delivery-analytics (Rill Cloud)
  • Source: rill/finance/ → finance-analytics (Rill Cloud)

Deployment is manual

There is no GitHub Action or CI that deploys Rill. After merging Rill changes:

export PATH="$HOME/.rill:$PATH"
 
# Delivery
rill project deploy /workspace \
  --org brainforge \
  --project delivery-analytics \
  --subpath rill/delivery \
  --description "Delivery health dashboards" \
  --push-env=false --interactive=false \
  --api-token "$RILL_SERVICE_TOKEN"
 
# Finance (Pipeline & Backlog, etc.)
rill project deploy /workspace \
  --org brainforge \
  --project finance-analytics \
  --subpath rill/finance \
  --description "Revenue and cost/margin dashboards" \
  --push-env=false --interactive=false \
  --api-token "$RILL_SERVICE_TOKEN"

Reference: standards/03-knowledge/engineering/setup/rill-setup.md § 5

Known issues (from playbook)

  • delivery-analytics: May not exist yet; deploy with service token can fail with only users can list projects by fingerprint (PermissionDenied).
  • Workaround: Run deploy once with RILL_USER_TOKEN on a trusted machine, then use RILL_SERVICE_TOKEN for CI/automation.
  • PLT-886: “Deploy Pipeline & Backlog canvas to Rill Cloud” — explicitly documents that finance-analytics must be redeployed after merge.

Parse error: /finance/rill.yaml — “resource type not specified” (2026-03-12)

If delivery-analytics was deployed with subpath rill/ (whole rill folder) instead of rill/delivery/, Rill Cloud discovers finance/rill.yaml and tries to parse it as a resource. Project files don’t have a type field, so the parser fails.

Fix: Use the correct deploy subpath so only that project’s files are included: delivery-analytics → --subpath rill/delivery, finance-analytics → --subpath rill/finance. The deploy workflow already uses these. Do not deploy with subpath rill/ for a single project. (ignore_paths is not a documented Rill property and was removed per Bugbot.)

2. Platform (The Forge) — Railway

How it works

  • Platform deploys via Railway (Heroku is legacy; pull-env now uses Railway).
  • Railway deploys from the branch configured in Service Settings (usually main).

If you pushed to a feature branch

  • Current branch: cursor/production-deployment-issue-5668
  • Production typically deploys from main.
  • Action: Merge your changes to main so Railway picks them up (if GitHub is connected and production is set to main).

Verify deployment

  1. Internal deploy dashboard: https://<platform-url>/internal/deploy-status (requires RAILWAY_API_TOKEN in env).
  2. Railway dashboard: https://railway.app/dashboard — check build/deploy status and which branch is configured.
ItemAction
Rill dashboards/modelsAuto-deploy: .github/workflows/rill-deploy.yml runs on push to main when rill/** changes. Ensure repo secret RILL_SERVICE_TOKEN is set. Manual: see rill-setup.md § 5.5.
Platform (Railway)Ensure changes are merged to main (or the branch Railway production is configured for).
AutomationRill auto-deploy is in place (2026-03-12).

4. Rill Cloud errors — investigation

Rill Cloud projects (delivery-analytics, finance-analytics) fail to reconcile because source queries hit Snowflake objects that do not exist in the database/schema the Rill DSN uses (production: PROD_MARTS).

4.1 Root causes (from Rill project status)

  • delivery-analytics: Source delivery_time_effort_mart → Snowflake error: Object ‘MART_DELIVERY_TIME_EFFORT’ does not exist. The Rill source uses unqualified FROM MART_DELIVERY_TIME_EFFORT.
  • finance-analytics: Source mart_finance_summary → Snowflake error: Object ‘PROD_MARTS.MARTS.MART_FINANCE_SUM…’ does not exist. The Rill source uses FROM MARTS.MART_FINANCE_SUMMARY (resolved under PROD_MARTS).

4.2 Snowflake investigation (2026-03-12)

Queries run with Snowflake CLI (snow sql -c brainforge-internal) and ACCOUNTADMIN:

CheckResult
PROD_MARTS schemasBRAINFORGE_RILL, BRAINFORGE_RILL_MANUFACTURING, BRAINFORGE_RILL_SALES, MARTS, PUBLIC — no DELIVERY schema.
PROD_MARTS.MARTS tablesDIM_*, FCT_*, HOURS_SUMMARY — no MART_DELIVERY_TIME_EFFORT, no MART_FINANCE_SUMMARY.
DEV_MARTSSchema DELIVERY exists; DEV_MARTS.DELIVERY.MART_DELIVERY_TIME_EFFORT exists.
ROLE_REPORTHas ROLE_PROD_MARTS_READ; that role has USAGE on PROD_MARTS and SELECT on PROD_MARTS.MARTS.* (and selected BRAINFORGE_RILL_* tables). Permissions are not the issue — the objects are missing in PROD.

Conclusions:

  1. Delivery: MART_DELIVERY_TIME_EFFORT exists only in DEV (DEV_MARTS.DELIVERY). It was never created or promoted to PROD. Rill Cloud uses a DSN that targets PROD, so the unqualified table name resolves in PROD and fails.
  2. Finance: MART_FINANCE_SUMMARY does not exist in PROD_MARTS.MARTS. The Rill source comment references DP-188 (planned work); the table has not been built in production.

Recommended next steps:

  • Delivery: Create PROD_MARTS.DELIVERY (if desired) and build/promote MART_DELIVERY_TIME_EFFORT into it (e.g. from dbt or copy from DEV). Then either keep Rill source as FROM MART_DELIVERY_TIME_EFFORT with Rill DSN default schema set to DELIVERY, or use fully qualified FROM PROD_MARTS.DELIVERY.MART_DELIVERY_TIME_EFFORT in the Rill source.
  • Finance: Implement DP-188 to create and populate PROD_MARTS.MARTS.MART_FINANCE_SUMMARY; then grant SELECT to ROLE_PROD_MARTS_READ (so service_user_report / Rill can read it).

5. References

  • standards/03-knowledge/engineering/setup/rill-setup.md
  • standards/03-knowledge/engineering/setup/rill-production-service-user-report.md
  • standards/03-knowledge/engineering/setup/snowflake/cli-setup.md
  • standards/03-knowledge/engineering/setup/railway-cli-setup.md
  • PLT-886: Deploy Pipeline & Backlog canvas to Rill Cloud
  • PLT-835: Set up CI/CD (staging, QA, prod) — Backlog

Graph View