Data Platform Scaling Harnesses and Retro
Date: 2026-03-03
Status: Draft / Planning
Related: Delivery & Finance Analytics Plan, Agent-powered data environment (plan folder), PR #236
Context
Two converging streams:
- Delivery & Finance Analytics — Rill deployment (DP-195–DP-200), ingest, dashboards. Merged PRs: #247 (foundation), #255 (delivery mart), #258 (Rill dashboard), #250 (dbt init), #249 (Rill DSN).
- Agent-Powered Data Environment (PR #236) — Strategy for worker/meta/orchestrator model, dbt PR impact pipeline, self-learning agents. Branch:
cursor/agent-powered-data-environment-a357.
Current gaps: No CI for rill/, knowledge/engineering/data-platform/, or dbt. No Cursor rules, commands, or skills for data work. AGENTS.md omits data platform. No playbook prompts for Rill/dbt workflows.
Part 1: Retro — Extract Learnings from Logs and PRs
1.1 Log Sources to Mine
| Source | Location | What to Extract |
|---|---|---|
| Cursor agent transcripts | ~/.cursor/projects/.../agent-transcripts/*.jsonl | Failed flows, repeated corrections, context gaps, tool usage patterns |
| PR comments and reviews | GitHub API / gh pr view | Reviewer feedback, merge-blocking issues, “why did you…” patterns |
| GTM RUN_LOG pattern | RUN_LOG.md | Adapt for data runs: Run ID, context, input, output, decisions, outcome |
| Eden agentic learnings | learnings | “Ask don’t fill”, scope to input only, cross-contamination pitfalls |
| Vercel agent learnings | VERCEL_LEAD_AGENT_LEARNINGS.md | Buy-in > build, start small, shadow best people |
1.2 Retro Process
- Gather — Export agent transcripts for data-related sessions (filter by paths:
rill/,knowledge/plans/agent-powered-data-environment/,knowledge/engineering/data-platform/,standards/03-knowledge/engineering/setup/). Pull PR comments for #236, #247, #255, #258, #250, #249. - Categorize — Group by: setup/credentials, dbt/Rill workflow, CI/deployment, agent context gaps, reviewer friction.
- Synthesize — Produce a retro doc:
knowledge/engineering/data-platform/learnings/data-platform-agent-retro-2026-03.mdwith patterns, pain points, and proposed fixes. - Feed into harnesses — Use retro findings to prioritize docs, rules, CI, and skills.
Part 2: Doc Harnesses
2.1 Playbook Additions
| Doc | Purpose |
|---|---|
| standards/03-knowledge/engineering/setup/data-platform-quickstart.md | Single entry point: Snowflake + Rill + dbt + Cursor Cloud Agent. Links to existing setup docs. |
| standards/04-prompts/tickets/data-platform-ticket-agent.md | Prompt for creating Linear tickets for data work (DE, dbt, Rill, ingest). |
| standards/04-prompts/data/dbt-pr-impact-prompt.md | Prompt for “run dbt on changed models and summarize impact” (aligns with PR #236 dbt PR impact worker). |
| standards/04-prompts/data/rill-deployment-prompt.md | Prompt for “deploy Rill project” or “validate Rill sources” — checklist from rill-setup + governance. |
2.2 Vault Learnings
- knowledge/engineering/data-platform/learnings/ — New folder for data-specific learnings.
- data-platform-agent-retro-2026-03.md — Output of Part 1 retro.
- RUN_LOG — Extend GTM pattern: add
knowledge/engineering/data-platform/RUN_LOG.mdfor data agent runs (dbt, Rill, ETL).
Part 3: Cursor Agent Harnesses
3.1 Rules
| Rule | Purpose |
|---|---|
| .cursor/rules/data-platform-setup.mdc | When working in rill/, knowledge/engineering/data-platform/, or dbt: check playbook setup first; use 1Password for Snowflake; never commit secrets. |
| .cursor/rules/data-platform-linear.mdc | When creating data tickets: use Data Platform team; assign to “Delivery & Finance Analytics — Phase 1 Foundation” project. |
3.2 Commands
| Command | Purpose |
|---|---|
| .cursor/commands/data-platform-dbt-run.md | Run dbt run and dbt test on changed models (or full project); output summary. |
| .cursor/commands/data-platform-rill-validate.md | Validate Rill sources (syntax, connector config); optionally run rill start and check health. |
| .cursor/commands/data-platform-retro-log.md | Add a row to data platform RUN_LOG from current session context. |
3.3 AGENTS.md Update
Add a Data Platform section to AGENTS.md:
- Working directory for Rill:
rill/; for dbt:knowledge/engineering/data-platform/dbt/ - Env:
SNOWFLAKE_*,SNOWFLAKE_CLOUD_AGENT_*(see playbook) - Pre-push: platform build only when
apps/platformchanges; data changes skip platform build - Reference: playbook data setup, data-platform rules
Part 4: CI Harnesses
4.1 New Workflow: Data Platform CI
File: .github/workflows/data-platform-ci.yml
- Triggers: PRs touching
rill/**,knowledge/engineering/data-platform/**,standards/03-knowledge/engineering/setup/** - Jobs:
rill-validate(rill compile),dbt-compile(dbt compile) - Scope: Start with compile-only (no Snowflake credentials in CI initially). Add
dbt run/dbt testwhen a read-only Snowflake CI role exists.
4.2 Path Triggers
- rill/ — Rill validation
- knowledge/engineering/data-platform/dbt/ — dbt compile (and later run/test)
- standards/ — Optional: link checker or lint; low priority
Part 5: Skills
5.1 Data Platform Skill
File: .cursor/skills/data-platform/SKILL.md
- When to use: Rill, dbt, Snowflake, data platform setup, delivery/finance analytics
- Key paths:
rill/,knowledge/engineering/data-platform/, playbook setup docs - Env: 1Password for Snowflake;
SNOWFLAKE_CLOUD_AGENT_*for Cloud Agents - Workflow: setup → compile/validate → run → deploy (Rill Cloud)
- References: rill-setup, snowflake/cursor-cloud-agent-setup, dbt brainforge-internal-data-platform-setup
5.2 Optional: Linear + Data Skill
Extend meeting-prep or create a thin wrapper: when user asks “create data ticket” or “log data work”, route to Data Platform team and project.
Part 6: Platform Harnesses (Light Touch)
Platform (apps/platform) is Supabase-centric. For scaling data work, minimal additions:
| Change | Purpose |
|---|---|
| Link to Rill Cloud | Add a nav item or dashboard widget linking to brainforge Rill org (delivery-analytics, finance-analytics) when deployed. |
| Deck storage | PR #236 mentions analyst deck flow needs persistent storage. Defer to Phase 2 of agent-powered data; document as open item. |
No heavy platform integration in this phase.
Part 7: Integration with PR #236
Merge or align PR #236 with this plan:
- Canonical docs — Strategy and research for this initiative live under
knowledge/plans/agent-powered-data-environment/(same content scope as PR #236’s vault additions; prefer this path for links going forward). - Workers — dbt PR impact worker, Rill deployment worker: implement as Cursor commands + prompts first; automate later.
- Feedback loop — Use
RUN_LOG.mdfor data runs; periodic pattern review per FEEDBACK_LOOP_PROCESS.md. - CI — PR #236 calls for “dbt PR impact pipeline”; Part 4 CI is the first step.
Execution Order
- Retro first — Informs prioritization of docs, rules, CI.
- Docs — Quickstart, prompts. Enables agents to self-serve.
- Cursor — Rules and commands. Highest leverage for daily use.
- CI — Compile/validate gates. Catches errors before merge.
- Skills — Bundles context for “data platform” asks.
- Platform — Link to Rill; defer deck storage.
Deliverables Summary
| Deliverable | Location |
|---|---|
| Strategy bundle (PR #236) | knowledge/plans/agent-powered-data-environment/ |
| Retro doc | knowledge/engineering/data-platform/learnings/data-platform-agent-retro-2026-03.md |
| Data RUN_LOG | knowledge/engineering/data-platform/RUN_LOG.md |
| Quickstart | standards/03-knowledge/engineering/setup/data-platform-quickstart.md |
| Prompts | standards/04-prompts/data/ (dbt, Rill, ticket) |
| Rules | .cursor/rules/data-platform-*.mdc |
| Commands | .cursor/commands/data-platform-*.md |
| CI | .github/workflows/data-platform-ci.yml |
| Skill | .cursor/skills/data-platform/SKILL.md |
| AGENTS.md | Data Platform section |