Teaching a newsroom pipeline to read first
Editors on several websites were searching dozens of sources by hand every day. Nothing was scored, so every item had to be read. RSS feeds, web pages and APIs all arrived in different formats. Each site covered different topics but needed the same steps: find content, assess quality, show it to editors, publish.
What I built
- Pluggable source adapters for RSS, web pages and APIs, each with its own parser and normalizer.
- Scoring with Claude Haiku against configurable criteria, from 0 to 100 with a rationale.
- Deduplication by exact URL and by title similarity.
- Per-site configuration layers, so one pipeline serves different topics and editorial standards.
- A contributor workflow where non-technical editors approve, edit or reject scored content, and their feedback is captured.
- Daily and weekly email digests of the top-scored content with links to add an item to a site.
- Langfuse tracing on every model call for cost, latency and token usage.
What came of it
Discovery runs automatically across dozens of sources per site. Editors start from scored content. The same pipeline serves multiple sites with different topic configurations. I can see cost per run, scoring trends and source quality.
