Brainforge Knowledge

Reminders Pipeline — Architecture and Operations

9 min read

Reminders Pipeline — Architecture and Operations

Overview

The reminders system gently nudges Slack users about unanswered @-mentions. It optimizes for minimizing false positives over catching every stalled thread — the bot should feel helpful, not naggy.

Each @-mention of a user is treated as its own open item. The pipeline filters out items the user has already engaged with, runs the survivors through an LLM that grades whether the item still needs a response, and DMs the target user with a 1–2 sentence summary plus a permalink. Suppression rules cap how often any one person can be pinged.

Slack history (per channel the bot is in)
  → extract every @-mention as an open item
  → drop items the target already reacted to or replied past (position-based ack)
  → drop items younger than threshold or older than max age
  → drop items already DM'd recently or where target hit daily cap
  → drop items where target is OOO (presence)
  → LLM eval (gpt-4.1-mini, structured output)
  → if has_open_question && confidence ≥ threshold:
       open IM → post {summary + permalink} → record in reminder_log
  → reaction on the DM = snooze (acknowledged_at set; never re-nudged)

Why this design

The first version of reminders scanned threads heuristically (no bot reply, no other human reply within 24h). It produced too many false positives — FYIs, social chat, and questions someone else had already answered all triggered nudges. Per-mention grading by an LLM cuts the false-positive rate at the cost of one classification call per surviving item, which is cheap (gpt-4.1-mini) and gated by cooldowns.

Position-based acknowledgment is deliberate: if a user reacts to any message at or after the mention, or replies after the mention, the item is acknowledged without needing a reaction_added event subscription or a reactions database. This means the entire pipeline runs from conversations.history and conversations.replies alone.

Pipeline phases

1. Channel discovery (reminders.ts:findOpenItems)

  • If REMINDER_CHANNELS is set: scan only those channel IDs/names.
  • If unset: call users.conversations for the bot account to enumerate every channel the bot is a member of (public + private).
  • Falls back to conversations.list filtered by is_member if users.conversations is unavailable.

The bot must be a member of the channel — otherwise conversations.history returns not_in_channel.

2. Open-item extraction

For each channel, fetch up to 200 messages older than now - REMINDER_MAX_AGE_HOURS. For each parent message:

  • Resolve the thread root (thread_ts || ts).
  • Fetch up to 200 replies via conversations.replies and sort by ts.
  • Walk every message in the thread. For each user mention <@U…> in text:
    • Skip if the target is the bot itself, the message author, or <!channel> / <!here> / <!subteam>.
    • Skip if the mention is younger than REMINDER_THRESHOLD_HOURS (default 24h).
    • Skip if the mention is older than REMINDER_MAX_AGE_HOURS (default 14 days).
    • Build itemId = ${channelId}:${threadTs}:${mentionTs}:${targetUserId}.

Each (thread, mention timestamp, target user) triple is one open item. The same target mentioned twice in the same thread produces two items; ack of one does not auto-resolve the other (although in practice acknowledgment is forward-looking and usually does).

3. Position-based acknowledgment

For each open item at thread index i, target user U:

  • Acknowledged if any message at index >= i has a reaction by U (any emoji; bot reactions filtered).
  • Acknowledged if U authored any non-bot reply at index >= i.
  • Otherwise eligible for the next phase.

The mention itself counts as index i, so if the target reacts to the mention message they are acknowledged immediately.

4. Suppression

Eligible items are filtered through, in order:

CheckSourceSkipped state
Already snoozed via DM reactionassistant_reminder_log.acknowledged_atsuppressedAcknowledged
Last DM for this item less than REMINDER_ITEM_COOLDOWN_DAYS agoassistant_reminder_log.sent_atsuppressedCooldown
Target already received REMINDER_MAX_DMS_PER_USER_PER_DAY reminders in 24hassistant_reminder_log countsuppressedDailyCap
Target presence is away (and REMINDER_RESPECT_PRESENCE=true)users.getPresencesuppressedAway

Channel mute detection is intentionally not implemented — Slack doesn’t expose mute state to bots reliably. Presence (away/active) is the proxy.

5. LLM evaluation (reminders.ts:evaluateOpenItem)

For each survivor, call gpt-4.1-mini (the classifier deployment) with generateObject and the schema:

{
  has_open_question: boolean,
  question_summary: string,   // 1–2 sentences for the DM
  confidence: number          // 0..1
}

System prompt directs the model to mark has_open_question = false for rhetorical questions, FYI mentions, threads someone else already answered on the target’s behalf, self-resolved questions, and social/jokey mentions.

Items where !has_open_question || confidence < REMINDER_CONFIDENCE_THRESHOLD (default 0.7) are dropped → counted as suppressedLowConfidence.

6. Delivery

  • Live mode (REMINDER_NOTIFY_CHANNEL_ID unset): open IM via conversations.open, post the summary + permalink as a DM. Slack message metadata carries event_type: 'reminder_dm' with the item_id so the snooze handler can find it.
  • Test mode (REMINDER_NOTIFY_CHANNEL_ID set): post the same content to that channel formatted as :test_tube: Reminder test — would DM <@U…>: <summary> <permalink>. Cooldowns and caps still apply, so test runs exercise the full suppression path.
  • Dry run (REMINDER_DRY_RUN=true): skip the post entirely; log [reminders:dry-run] line per item.

Each successful post records a row in assistant_reminder_log with reminder_dm_channel, reminder_dm_ts, and sent_at = now().

7. Snooze (assistant.ts:reaction_added handler)

When a reaction_added event lands on a message authored by the bot:

  1. Look up rows where reminder_dm_channel = item.channel AND reminder_dm_ts = item.ts AND acknowledged_at IS NULL.
  2. Set acknowledged_at = now() for each match.
  3. Future scans skip these items via the suppressedAcknowledged gate.

This works in both live mode (DM in IM channel) and test mode (post in test channel) — any reaction by anyone on a reminder post snoozes the underlying item.

Data model

assistant_reminder_log (Postgres, on the assistant’s DATABASE_URL)

ColumnTypePurpose
item_idtext PK${channel}:${thread}:${mention_ts}:${target_user}
channel_idtextsource channel of the mention
thread_tstextthread root
mention_tstextthe message that did the @-mention
target_user_idtextwho would be DM’d
reminder_dm_channeltextchannel of the post (IM channel id, or test channel)
reminder_dm_tstextts of the reminder post (for snooze lookup)
sent_attimestamptzlast time we posted for this item
acknowledged_attimestamptz | nullsnooze marker

Indexes: (target_user_id, sent_at) for the daily-cap query, (reminder_dm_channel, reminder_dm_ts) for snooze lookup.

The schema is created idempotently in state.ts:init(); no separate migration file.

Code map

FileWhat lives here
apps/slack-apps/brainforge-assistant/src/reminders.tsfindOpenItems, evaluateOpenItem, runReminderScan, all suppression logic
apps/slack-apps/brainforge-assistant/src/state.tsrecordReminderSent, getReminderSentAt, countReminderDmsSentToday, isItemAcknowledged, acknowledgeItemsByDm (Postgres + in-memory)
apps/slack-apps/brainforge-assistant/src/index.tsPOST /internal/reminders endpoint that calls runReminderScan and returns the stats JSON
apps/slack-apps/brainforge-assistant/src/assistant.tsreaction_added handler that calls acknowledgeItemsByDm for snooze
apps/slack-apps/brainforge-assistant/src/config.tsenv var → AssistantConfig mapping for all reminder tunables

Configuration

Env varDefaultEffect
REMINDER_CHANNELSempty (= all bot-member channels)Comma-separated channel IDs or names to watch
REMINDER_THRESHOLD_HOURS24Minimum age of a mention before it’s eligible
REMINDER_MAX_AGE_HOURS336 (14d)Ignore mentions older than this
REMINDER_CONFIDENCE_THRESHOLD0.7LLM confidence gate for sending
REMINDER_ITEM_COOLDOWN_DAYS7Don’t re-nudge the same item to the same user inside this window
REMINDER_MAX_DMS_PER_USER_PER_DAY3Per-user daily DM cap
REMINDER_RESPECT_PRESENCEtrueSkip targets whose Slack presence is away
REMINDER_DRY_RUNfalseLog-only mode; no posts at all
REMINDER_NOTIFY_CHANNEL_IDemptyIf set, redirect all reminders to this channel (test mode)

Required Slack scopes: channels:history, groups:history, im:history, mpim:history, reactions:read, chat:write, users:read, im:write, plus channels:read and groups:read so users.conversations enumerates the bot’s memberships.

Endpoint

POST /internal/reminders (no body required) returns:

{
  "ok": true,
  "scanned": 142,
  "eligible": 18,
  "classifiedOpen": 4,
  "sent": 4,
  "suppressedAcknowledged": 6,
  "suppressedCooldown": 12,
  "suppressedDailyCap": 0,
  "suppressedAway": 2,
  "suppressedLowConfidence": 14
}

Designed to be called on a cron. The current scheduler is the GitHub Actions workflow slack-assistant-reminders.yml (weekday mornings); rate-of-call must respect the per-item cooldown (calling more than once a day is fine — items will be suppressed by the cooldown, not re-DM’d).

Operations runbook

Test mode (current)

REMINDER_NOTIFY_CHANNEL_ID is set on Railway → all candidate reminders post to that channel as :test_tube: Reminder test — would DM …. Reviewers in that channel can sanity-check the bot’s judgment before flipping live.

To trigger a scan manually:

curl -X POST https://brainforge-assistant-production.up.railway.app/internal/reminders

To go live: unset the var.

railway variables --set "REMINDER_NOTIFY_CHANNEL_ID="
# then deploy or restart the service

Tuning

  • Too many false positives (test channel posts that shouldn’t have fired): raise REMINDER_CONFIDENCE_THRESHOLD (try 0.8 first), or shorten REMINDER_MAX_AGE_HOURS to ignore stale threads.
  • Too noisy for a specific user: lower REMINDER_MAX_DMS_PER_USER_PER_DAY or rely on snooze. Per-user excludes are not implemented; if needed, add a REMINDER_EXCLUDE_USERS allowlist before going live.
  • Missing legitimate items: lower REMINDER_THRESHOLD_HOURS (12h) or REMINDER_CONFIDENCE_THRESHOLD (0.6). Monitor for false-positive regressions.

Kill switch

Set REMINDER_DRY_RUN=true and redeploy — the endpoint still runs but posts nothing. The DB is untouched, so cooldowns persist if you flip back.

Inspecting state

-- recent reminder activity for a user
select item_id, channel_id, sent_at, acknowledged_at
from assistant_reminder_log
where target_user_id = 'Uxxxxxxxx'
order by sent_at desc
limit 20;
 
-- daily volume
select date_trunc('day', sent_at) as day, target_user_id, count(*)
from assistant_reminder_log
group by 1, 2
order by 1 desc, 3 desc;

Edge cases handled

  • Multiple @s of the same user in one thread → tracked as separate items keyed by mention_ts. Acknowledgment of a later mention does NOT auto-resolve earlier ones (although forward-looking acknowledgment usually catches both).
  • Target replied then was @-mentioned again → new item, clock resets.
  • Mention target = the bot itself → skipped at extraction.
  • Mention author = target (someone tagged themselves) → skipped at extraction.
  • @channel / @here / @subteam → not user mentions; the regex <@([UW][A-Z0-9]+)> ignores them.
  • Thread the bot can’t read (not_in_channel) → logged warning, channel skipped, scan continues.
  • Target user is deactivated / deleted → conversations.open will fail; the item is logged and skipped.

Known limitations

  • Channel mute is not detected. Presence (away) is the proxy. If a user mutes a channel but stays online, they can still receive a reminder for an @ in that channel.
  • One classifier call per surviving open item. Cost scales with eligible, not scanned. The cooldown gate is the main cost lever.
  • No per-channel sensitivity controls. All channels currently use the same threshold and confidence. If we need per-channel tuning (e.g., looser threshold for #help, stricter for #general), introduce a channel→config map.
  • Cooldown is per (item, user), not per (user, day) for re-asks. The daily cap REMINDER_MAX_DMS_PER_USER_PER_DAY is the only per-user limit beyond the cooldown.

Future work

  • Track DM CTRs (did the user click the permalink? did they reply within X hours?) to feed back into the confidence gate.
  • Optional escalation path: if an item is unacknowledged after N DMs, surface to a designated reviewer instead of the target.
  • Allow per-user opt-out via a /brainforge mute reminders slash command.
  • Replace the gpt-4.1-mini classifier with a fine-tuned model once we have enough labeled (thread, vote) data from the feedback loop.

Graph View

Backlinks