Feedback Loop Process: How Logs → Patterns → Agent Improvements
Purpose: The actual process for closing the loop from agent runs → learning → agent improvements. This is the discipline that makes the context graph work.
Related: CONTEXT_GRAPH_APPROACH.md, RUN_LOG.md, AGENT_FEEDBACK_LOOP.md (automated version), Agent Registry
Two Approaches
Option A: Automated (Recommended)
See AGENT_FEEDBACK_LOOP.md for the full automated system:
- Run agent → System prompts for feedback → Auto-generates logs → Analyzes patterns → Shows impact
- Uses “thinking to summary” approach from learn-extraction skill
- Status: To be implemented (agent wrapper + pattern analysis script)
Option B: Manual (Works Today)
This document describes the manual process you can use today while automation is built.
The problem: Logs don’t automatically improve agents
What we have: A run log (markdown table) where you manually record runs.
What’s missing: The process that turns logs into agent improvements.
Without a feedback loop: You log runs, but nothing learns from them.
With a feedback loop: Logs → patterns → PRD/taxonomy updates → agents get smarter.
The feedback loop (manual today, can automate later)
Step 1: Log every run (discipline)
When: Immediately after running an agent or following a process.
How: Add one row to RUN_LOG.md:
| Run ID | Campaign/Context | Archetype/Type | Input | Output | Decisions | Outcome | Date |
|---|---|---|---|---|---|---|---|
| design-ready-copy-2026-02-04-insurance-broker | insurance-broker-lead-intake | service_2pager | brief.md | 2pager.md | Single service → 2-pager | Used, no edits | 2026-02-04 |
Time: 30 seconds per run.
Why it matters: Without logs, you can’t see patterns. With logs, you can.
Step 2: Review patterns (periodic)
When: After 5–10 runs of the same agent, or monthly.
How: Look at RUN_LOG.md and ask:
- Archetype patterns: “Which archetypes get used most? Why?”
- Outcome patterns: “Which runs succeed? Which fail? What’s different?”
- Deviation patterns: “When do we deviate from the expected path? Why?”
Example review (after 10 design-ready copy runs):
| Pattern | Count | Insight |
|---|---|---|
| Insurance campaigns → Service 2-pager | 4/4 | Rule works |
| dbt campaigns → Sprint | 3/3 | Rule works |
| ”One-pager” brief → Service 2-pager | 2/2 | Brief says “one-pager” but we use 2-pager structure |
| Designer deleted “Trusted by” | 8/10 | No case studies match → section is noise |
Action: Document patterns in PATTERNS.md or RUN_LOG.md “Pattern notes” section.
Step 3: Update agents/PRDs (when pattern is clear)
When: Pattern is consistent (e.g. happens 80%+ of the time) and actionable.
How: Create a PR that updates the agent/PRD/taxonomy based on the pattern.
Example PR (from pattern review above):
Title: “Make ‘Trusted by’ section optional in design-ready copy (designers delete it 80% of the time)”
Changes:
- Update taxonomy: “Trusted by section: optional; hide if no matching case study”
- Update PRD: “If no case study matches ICP, skip Trusted by section (don’t show placeholder)”
- Update agent logic: “Check if case study exists; if not, skip Trusted by section”
Result: Next time someone runs the agent, it automatically skips the section designers always delete.
Step 4: Agents use updated rules (automatic)
When: PR is merged, agent PRD/taxonomy is updated.
How: Next agent run reads the updated PRD/taxonomy and follows the new rules.
Example: After the PR above, next design-ready copy run:
- Checks for matching case study
- If none → skips “Trusted by” section automatically
- Designer gets cleaner output (no section to delete)
Result: Agent is smarter because it learned from past runs.
Making it work: Discipline + process
Option A: Manual (works today, requires discipline)
- Log every run (30 sec) →
RUN_LOG.md - Review monthly (15 min) → Look for patterns
- PR when pattern is clear (30 min) → Update agent/PRD
- Agent improves → Next run uses new rules
Time investment: ~1 hour/month per agent.
Risk: Easy to skip logging → no patterns → no improvements.
Option B: Automated (v2, requires infrastructure)
What it would do:
- Auto-log runs: Agent wrapper captures run automatically (inputs, outputs, decisions, outcomes)
- Auto-detect patterns: Script analyzes run log, surfaces patterns (e.g. “Designers delete section X 80% of the time”)
- Auto-suggest PRs: Bot creates PR draft: “Update taxonomy: make section X optional”
- Auto-update agents: After PR merge, agent reads updated PRD automatically
Time investment: Setup once, then runs automatically.
Requires: Agent wrapper, pattern detection script, PR bot, or integration with agent execution layer.
Concrete example: Design-ready copy agent
Run 1–5 (baseline)
Logs:
- Insurance broker → Service 2-pager → Used
- dbt audit → Service 2-pager → Used
- AI Growth Sprint → Sprint → Used
- Holiday deal → Seasonal → Used
- Edge to Activation → Service 2-pager → Used
Pattern (after review): “Single service → Service 2-pager” rule works. No changes needed.
Run 6–10 (patterns emerge)
Logs:
- Insurance broker #2 → Service 2-pager → Designer deleted “Trusted by”
- CPG attribution → Service 2-pager → Designer deleted “Trusted by”
- Product analytics → Strategy guide → Designer deleted “Trusted by”
- Lifecycle sprint → Sprint → Designer deleted “Trusted by”
- Partner kit → Partner kit → Designer kept “Trusted by” (had case study)
Pattern (after review): “Designers delete ‘Trusted by’ 8/10 times when no case study matches.”
Action: PR to update taxonomy: “Make Trusted by optional; hide if no matching case study.”
Run 11+ (agent improved)
After PR merge: Agent checks for matching case study before including “Trusted by.”
Logs:
- Insurance broker #3 → Service 2-pager → No “Trusted by” (no case study) → Designer happy (no section to delete)
- dbt audit #2 → Service 2-pager → “Trusted by” included (had case study) → Designer kept it
Result: Agent is smarter because it learned from past runs.
How to start (today)
- Log the next 5 runs of any agent →
RUN_LOG.md - After 5 runs, review → Look for one pattern
- If pattern is clear → Create PR to update agent/PRD
- Repeat → Each cycle makes agents smarter
First pattern to look for: “What do designers/users actually do with the output?” (That’s where blind spots hide.)