Global Search - Documentation

Overview

The Global Search feature provides unified search capabilities across multiple data sources in the Brainforge Platform:

Meetings: Zoom meeting recordings and metadata
Slack Messages: Messages from client-specific Slack channels
HubSpot Deals: Deal information including status, contacts, and companies

This feature was implemented as part of Linear ticket AI-530.

Architecture

Data Flow

┌─────────────────┐
│  Data Sources   │
│  - Meetings     │
│  - Slack        │
│  - Deals        │
└────────┬────────┘
         │
         │ Indexing Script
         ▼
┌─────────────────┐
│  Turbopuffer    │
│   (global_      │
│    search)      │
└────────┬────────┘
         │
         │ Search Query
         ▼
┌─────────────────┐
│  Search API     │
│  /api/search/   │
│   global        │
└────────┬────────┘
         │
         │ Results
         ▼
┌─────────────────┐
│  Frontend UI    │
│  /search        │
└─────────────────┘

Key Components

Indexing Utilities (src/lib/turboPuffer/indexingUtils.ts)
- indexZoomMeetings() - Indexes meeting recordings
- indexSlackMessages() - Indexes Slack messages from all client tables
- indexHubSpotDeals() - Indexes HubSpot deals
- indexAllDataSources() - Convenience function to index everything
Search API (src/app/api/brainforge/search/global/route.ts)
- POST endpoint accepting search parameters
- Queries Turbopuffer’s global_search namespace
- Returns unified results with type indicators and metadata
Search Hook (src/hooks/useGlobalSearch.ts)
- React hook for managing search state
- Handles debouncing and abort controllers
- Type-safe interfaces for search results
UI Components
- GlobalSearchResults (src/components/GlobalSearchResults.tsx) - Displays search results with type badges
- GlobalSearchPage (src/app/(main)/search/page.tsx) - Main search interface
Indexing Script (scripts/index-global-search.ts)
- CLI tool for indexing data into Turbopuffer
- Supports selective indexing by type

Data Structure

Turbopuffer Namespace

Namespace: global_search
Region: aws-us-east-2

Indexed Fields

Meeting Records

{
  id: "meeting_{supabase_id}",
  type: "meeting",
  name: string,              // Meeting topic/folder name
  summary: string,           // AI-generated summary
  content: string,           // Transcript content
  participants: string[],    // List of participant names
  meeting_date: string,      // ISO date
  created_at: string,        // ISO date
  updated_at: string,        // ISO date
  supabase_id: string       // Reference to source record
}

Slack Message Records

{
  id: "slack_{client_id}_{message_ts}",
  type: "slack_message",
  content: string,           // Message text
  user_id: string,           // Slack user ID
  user_name: string,         // Display name
  channel_id: string,        // Slack channel ID
  channel_name: string,      // Channel name
  client_id: string,         // Associated client
  client_name: string,       // Client name
  message_ts: string,        // Message timestamp (Slack ID)
  thread_ts: string,         // Thread parent timestamp
  is_thread_reply: boolean,  // Whether this is a thread reply
  created_at: string,        // ISO date
  updated_at: string         // ISO date
}

Deal Records

{
  id: "deal_{deal_id}",
  type: "deal",
  deal_id: string,           // HubSpot deal ID
  deal_name: string,         // Deal name
  content: string,           // Searchable content (aggregated)
  status: string,            // Deal status (e.g., "Qualified: Onboarding")
  stage: string,             // Pipeline stage
  amount: number,            // Deal value
  owner_name: string,        // Deal owner
  company_names: string[],   // Associated companies
  contact_names: string[],   // Associated contacts
  created_at: string,        // ISO date
  updated_at: string         // ISO date
}

API Reference

POST /api/brainforge/search/global

Searches across all indexed data sources.

Request Body

{
  searchTerm: string;              // Search query (optional)
  types?: string[];                // Filter by types: ["meeting", "slack_message", "deal"]
  limit?: number;                  // Results per page (default: 50)
  offset?: number;                 // Pagination offset (default: 0)
  startDate?: string;              // Filter by date range (ISO)
  endDate?: string;                // Filter by date range (ISO)
  clientId?: string;               // Filter Slack messages by client
}

Response

{
  results: GlobalSearchResult[];   // Array of search results
  total: number;                   // Total results found
  hasMore: boolean;                // Whether more results exist
  source: "turbopuffer";           // Data source indicator
  breakdown: {                     // Results by type
    meetings: number;
    slack_messages: number;
    deals: number;
  }
}

GlobalSearchResult Interface

interface GlobalSearchResult {
  id: string;                      // Unique identifier
  type: "meeting" | "slack_message" | "deal";
  title: string;                   // Display title
  content: string;                 // Preview content
  metadata: Record<string, any>;   // Type-specific metadata
  matchScore: number;              // Relevance score
  matchedFields: string[];         // Which fields matched the query
  url?: string;                    // Link to source (if applicable)
}

Usage

Running the Indexing Script

Index all data sources:

npx tsx scripts/index-global-search.ts

Index specific types:

npx tsx scripts/index-global-search.ts --type=meetings
npx tsx scripts/index-global-search.ts --type=slack
npx tsx scripts/index-global-search.ts --type=deals

Using the Search Hook

import { useGlobalSearch } from "@/hooks/useGlobalSearch";
 
function MyComponent() {
  const { search, searchResults, isSearching } = useGlobalSearch({
    filters: {
      searchTerm: "customer feedback",
      types: ["meeting", "slack_message"]
    }
  });
 
  // Trigger search
  useEffect(() => {
    search({ searchTerm: "customer feedback" }, 50, 0);
  }, []);
 
  return (
    <div>
      {isSearching && <Spinner />}
      {searchResults.results.map(result => (
        <ResultCard key={result.id} result={result} />
      ))}
    </div>
  );
}

Accessing the Search Page

Navigate to /search in the application to use the global search interface.

Search Ranking Strategy

The search uses BM25 ranking across multiple fields:

For Meetings

content (transcript)
name (meeting topic)
summary (AI-generated summary)
participants

For Slack Messages

content (message text)
user_name
client_name

For Deals

content (aggregated searchable text)
deal_name
status
owner_name

Results are ranked by the sum of BM25 scores across all fields.

UI Features

Search Interface

Search Input
- Debounced search (500ms delay)
- Clear button for quick reset
Type Filters
- Toggle between Meetings, Slack Messages, and Deals
- Multiple types can be selected simultaneously
Results Display
- Color-coded type badges
  - Meetings: Blue (primary)
  - Slack Messages: Purple (secondary)
  - Deals: Green (success)
- Matched fields indicator showing which fields triggered the match
- Click to open result in new tab or navigate to detail page
Results Summary
- Total count across all types
- Breakdown by type (e.g., “3 Meetings, 5 Slack Messages, 2 Deals”)

Result Cards

Each result card displays:

Type badge with icon
Title (context-specific)
Content preview (first 3 lines)
Metadata chips:
- Meetings: Date, participant count, transcript indicator
- Slack: Client name, channel, thread indicator, date
- Deals: Stage, amount, owner, company
Matched fields (highlighted)
Open link (external icon)

Performance Considerations

Indexing

Batch size: 1000 records per type by default
Uses upsert for idempotent updates
Failed records are logged but don’t stop the batch

Search

Results cached in Turbopuffer for fast retrieval
Pagination supported via limit and offset
Abort controller prevents duplicate queries

Recommendations

Re-index daily via cron job
Use pagination for large result sets
Consider adding incremental indexing for real-time updates

Configuration

Environment Variables

Required:

TURBOPUFFER_API_KEY=your_api_key

The following are already configured in the codebase:

TURBOPUFFER_REGION="aws-us-east-2"
GLOBAL_SEARCH_NAMESPACE="global_search"

Limitations

Slack Messages
- Only indexes clients with slack_messages_table configured
- Limited to configured message limit (default: 1000 per client)
Real-time Updates
- Data is indexed in batches, not real-time
- New records won’t appear until next indexing run
Search Scope
- Searches are limited to indexed fields only
- Attachments and binary content not indexed

Future Enhancements

Potential improvements (not in current scope):

Incremental Indexing
- Index only new/updated records since last run
- Use webhooks for real-time indexing
Advanced Filters
- Filter by specific clients
- Date range filters
- Participant filters for meetings
Search Analytics
- Track popular search terms
- Measure search relevance
Additional Data Sources
- Linear tickets
- Documents
- GitHub issues
Semantic Search
- Upgrade from BM25 to vector embeddings
- Support natural language queries

Troubleshooting

Common Issues

Issue: No results returned

Check: Verify data has been indexed (run indexing script)
Check: Confirm TURBOPUFFER_API_KEY is set
Check: Check Turbopuffer dashboard for namespace data

Issue: Slack messages not appearing

Check: Verify clients have slack_messages_table configured
Check: Run indexing script with --type=slack
Check: Inspect Slack Supabase for message data

Issue: Search is slow

Check: Reduce result limit
Check: Add more specific filters
Check: Verify Turbopuffer region is aws-us-east-2

Debugging

Enable detailed logging:

// In indexing script
console.log("Indexing progress:", { meetings, slackMessages, deals });
 
// In search API
console.log("Search query:", { searchTerm, types, filters });
console.log("Turbopuffer response:", turbopufferResponse);

Testing

Manual Testing

Index Data
```
npx tsx scripts/index-global-search.ts
```
Navigate to Search
- Go to /search in the application
Test Scenarios
- Empty search (should show all recent items)
- Keyword search (e.g., “customer”)
- Filter by type (toggle each type)
- Pagination (scroll to load more)
- Click results (verify links work)

Automated Testing

See docs/testing.md for unit test examples.

References

Slack Assistant Integration

The Brainforge Slack assistant can leverage global search to answer questions about internal meetings, Slack threads, and HubSpot deals.

Configuration

In the Slack assistant service (Railway), set:

GLOBAL_SEARCH_API_URL=https://platform.brainforge.ai/api/brainforge/search/global
PLATFORM_API_SECRET=<same_secret_as_platform>
# Optional: BRAINFORGE_PLATFORM_ORIGIN=https://platform.brainforge.ai

Production API smoke (Railway)

The Forge web service on Railway has PLATFORM_API_SECRET. Run the repo smoke script with Railway injecting env (no secret printed):

cd apps/platform && railway run -- node ../../scripts/smoke-global-search.mjs

Expect HTTP 200 OK, source: turbopuffer, and a non-empty returnedCount when the index matches. For local-only testing, use .env.local and npm run smoke:global-search from the monorepo root, or set GLOBAL_SEARCH_SMOKE_URL=http://127.0.0.1:3000/api/brainforge/search/global with npm run dev in apps/platform.

How It Works

When a user asks a question in Slack:

The assistant analyzes the query to determine if internal content is likely relevant
If so, it queries the global search API in parallel with other context sources (repo, web)
Results (meetings, Slack messages, deals) are included in the prompt with source attribution
Citations appear as clickable links in the Slack response

Query Intelligence

The assistant uses shouldUseGlobalSearch() to decide when to query:

Searches for: Client names, meetings, transcripts, Granola, HubSpot deals, “what did we discuss”, SOW questions, project status
Skips for: Generic trivia (weather, definitions, “who invented X”) to reduce latency
Override: Set GLOBAL_SEARCH_ALWAYS=1 in the assistant environment to search on every query

Keeping the Index Fresh

The assistant’s answers are only as good as the indexed data. See .github/workflows/global-search-index.yml for automated daily reindexing, or run manually:

npx tsx scripts/index-global-search.ts

Support

For questions or issues:

Check this documentation
Review Linear ticket AI-530
Contact the development team

Last Updated: March 30, 2026 Version: 1.1 Status: Production-ready

Brainforge Knowledge

Explorer

global-search

Global Search - Documentation

Overview

Architecture

Data Flow

Key Components

Data Structure

Turbopuffer Namespace

Indexed Fields

Meeting Records

Slack Message Records

Deal Records

API Reference

POST /api/brainforge/search/global

Request Body

Response

GlobalSearchResult Interface

Usage

Running the Indexing Script

Using the Search Hook

Accessing the Search Page

Search Ranking Strategy

For Meetings

For Slack Messages

For Deals

UI Features

Search Interface

Result Cards

Performance Considerations

Indexing

Search

Recommendations

Configuration

Environment Variables

Limitations

Future Enhancements

Troubleshooting

Common Issues

Debugging

Testing

Manual Testing

Automated Testing

References

Slack Assistant Integration

Configuration

Production API smoke (Railway)

How It Works

Query Intelligence

Keeping the Index Fresh

Support

Graph View

Table of Contents