dagster-migration-next-steps

Dagster monorepo migration – next steps (post–PR 233)

PR 233 merged the dagster-pipelines repo into the monorepo as a subtree under apps/dagster-pipelines/. Use this doc to align with the AI team, decide when to turn off the old repo, and track follow-up work in Linear.

---

1. What to ask the AI team to confirm

Send these to the AI team so you’re aligned before changing anything in production or turning off the old repo:

• Deployment source
  “We’re now running Dagster from the monorepo (apps/dagster-pipelines/). Can you confirm that all Dagster Cloud deployments (and any CI that deploys to Dagster Cloud) are pointed at the brainforge-platform repo and no deployments still use the old dagster-pipelines repo?”

• Schedules
  “Schedules are gated by DAGSTER_ENABLE_SCHEDULES. Have you enabled schedules in the monorepo deployment, or are we still intentionally leaving them off during the cutover? Which jobs/schedules are currently required in production?”

• Webhooks / triggers
  “Do any external systems (e.g. n8n, other services) trigger Dagster jobs by repo or by deployment? If yes, have those been switched to trigger the monorepo-based deployment only?”

• Secrets and env
  “Are DAGSTER_CLOUD_API_TOKEN, DAGSTER_CLOUD_ORGANIZATION, and any job-specific secrets configured in the monorepo’s Dagster Cloud deployment and working for all jobs you care about?”

• Sign-off to turn off old repo
  “Once we’ve confirmed the above, are you okay with us archiving/read-only’ing the standalone dagster-pipelines repo and treating the monorepo as the single source of truth?”

---

2. When you’re good to turn off the other repo

You’re good to turn the old repo off (archive or read-only) when all of these are true:

Check	Owner
Dagster Cloud is deploying only from brainforge-platform (no deploys from dagster-pipelines).	AI / platform
All required jobs run successfully from the monorepo deployment (run a few manually or via triggers).	AI team
No CI/CD or external systems still reference or push to the old dagster-pipelines repo.	You / AI team
AI team has explicitly confirmed they’re fine with the old repo being archived/read-only.	AI team

Until then, keep the old repo in read-only or minimal use rather than deleting it, so you can compare or roll back if needed.

---

3. Next steps as Linear tickets (same project)

Create these in your existing Linear project (e.g. the one you used for the Dagster merge work). Copy title + description as-is or adjust to your project’s labels/priority.

---

Ticket 1: AI team sign-off – Dagster running from monorepo only

Title: [Dagster] Confirm AI team: all deployments from monorepo, no reliance on old repo

Description:

• Get written confirmation from the AI team that:
  • All Dagster Cloud deployments use brainforge-platform (apps/dagster-pipelines).
  • No systems still trigger or deploy from the standalone dagster-pipelines repo.
• Document their answer (Slack thread or Linear comment) and then close or downscope any “migrate Dagster” epic.

Acceptance: AI team confirms in writing; ticket closed with link to confirmation.

---

Ticket 2: Update playbook – Dagster repo location

Title: [Dagster] Update playbook: point all docs to monorepo path

Description:

• In standards/03-knowledge/engineering/setup/dagster-cli-setup.md:
  • Replace references to the standalone repo path (e.g. /Users/.../dagster-pipelines/) with the monorepo path: apps/dagster-pipelines/ (or full path inside brainforge-platform).
  • Update “Repository Location” and any grep/CLI examples to use apps/dagster-pipelines/ or brainforge-platform/apps/dagster-pipelines/.
• In standards/03-knowledge/engineering/setup/slack-daily-summary-setup.md (and any other playbook doc that mentions “dagster-pipelines” repo):
  • Change to “in brainforge-platform monorepo, apps/dagster-pipelines/” or equivalent.

Acceptance: No playbook doc points to the old standalone repo path; all say monorepo.

---

Ticket 3: Schedules and env – confirm production config

Title: [Dagster] Confirm DAGSTER_ENABLE_SCHEDULES and env for production

Description:

• With the AI team, confirm:
  • Whether DAGSTER_ENABLE_SCHEDULES is enabled in the monorepo Dagster Cloud deployment.
  • Which schedules/jobs are required in production and that they’re registered and running from apps/dagster-pipelines/.
• Document the decision (enable/disable schedules, and list of critical jobs).

Acceptance: Decision and list of critical jobs documented; no surprise schedule gaps.

---

Ticket 4: Archive or read-only the old dagster-pipelines repo

Title: [Dagster] Archive or set read-only the standalone dagster-pipelines repo

Description:

• Prerequisite: Ticket “AI team sign-off – Dagster running from monorepo only” is done and AI team has confirmed.
• In GitHub (or wherever the repo lives):
  • Either archive the dagster-pipelines repo, or make it read-only and add a prominent README note: “Dagster code has moved to brainforge-platform monorepo under apps/dagster-pipelines/. Do not push here.”
• Optionally: add a short note in the monorepo README or AGENTS.md that Dagster lives in apps/dagster-pipelines/ and the old repo is deprecated.

Acceptance: Old repo is archived or read-only with clear deprecation notice; no new commits there.

---

Ticket 5: Migration table and runbook (optional)

Title: [Dagster] Update migration table and runbook for monorepo

Description:

• In standards/03-knowledge/engineering/setup/dagster-cli-setup.md:
  • Ensure the “Migration Status” table is up to date (which jobs are migrated to inline processing vs still on Dagster).
  • Add a short “Monorepo runbook” subsection: where code lives (apps/dagster-pipelines/), how to run/deploy from monorepo, and that the old repo is deprecated.

Acceptance: Migration table current; runbook mentions monorepo and deprecation of old repo.

---

4. Summary

Do this	Then
Send the “What to ask the AI team” section to the AI team and get answers.	You know if it’s safe to change deployments and turn off the old repo.
Use the “When you’re good to turn off the other repo” checklist.	You avoid turning off the repo before everything is cut over.
Create the 5 Linear tickets above in your existing project.	You have clear next steps and a single place to track sign-off, docs, config, and repo archival.

After Ticket 1 and 3 are done and the checklist is satisfied, you’re good to execute Ticket 4 (archive/read-only the old repo).