A multi-agent discovery pipeline across 96 sources
I needed a pipeline that could discover, evaluate and rank opportunities across nearly 100 data sources: API platforms, RSS feeds, career pages and community forums. Doing it by hand missed items that expired within hours, and judgment varied from day to day.
What I built
A production multi-agent system in TypeScript and Next.js.
- A discovery engine with 7 adapter types that searches 96 sources in parallel.
- Claude Haiku, called through a Portkey gateway, to extract structured data from unstructured text and to score each item from 0 to 100 with a written rationale.
- Deduplication by Jaccard similarity across all sources.
- A source quality tracker that disables a source after 5 consecutive failures.
- An 11-tab dashboard covering search, results, pipeline, documents, contacts, history, settings and analytics.
- Two layers of caching, in memory and in Upstash Redis.
- Langfuse tracing for cost per run, latency and token usage.
- A cron job that polls every 15 minutes during business hours.
What came of it
All 96 sources are monitored automatically, with quality tracked per source. Scoring costs about $0.12 a run and leaves a documented reason for every item. Time-sensitive items are caught within 15 minutes of posting. The pipeline is held up by 447 test files with 80% coverage thresholds, and Semgrep scans it every 12 hours.
