AI News Newsletter Pipeline

AI News Newsletter Pipeline

Project Overview

Built a production-grade automated newsletter system that processes RSS feeds from AI news sources, filters relevant content using AI agents, and delivers personalized daily digests to subscribers. The system demonstrates advanced cost optimization, intelligent caching, and modular architecture design.

Architecture & Design Philosophy

Modular Agent-Based Pipeline

The core architectural insight was breaking down newsletter generation into discrete, autonomous agents, each responsible for a single transformation step:

RSS Feeds → Fetcher Agent → Deduplication → Keyword Filter
→ Relevance Agent → Categorization Agent → Ranking Agent
→ Distribution → Email Delivery

Key Design Principles:

• Separation of Concerns: Each agent has one clear responsibility
• Testability: Individual agents can be tested in isolation
• Swappability: Models can be swapped per-agent without affecting other stages
• Debugging: When something goes wrong, you know exactly which agent failed
• Parallelization Potential: Agents could theoretically run in parallel

Technical Implementation

1. RSS Feed Ingestion (Fetcher Agent)

Key Features:

• Retry logic with exponential backoff (3 attempts with 1s, 2s, 3s delays)
• Robust date parsing for inconsistent RSS feed formats
• Time window filtering (configurable 24-hour window for daily digest)
• Rate limiting (0.5s sleep between feeds) to prevent overwhelming servers
• Graceful error handling (continues processing if individual feeds fail)

2. Relevance Filtering Agent

Model Selection Strategy:

• Started with GPT-4 for highest quality
• Optimization: Switched to GPT-3.5-turbo for binary relevance decisions
• Cost Impact: 80% reduction (~$1.50-3.00/day → ~$0.30-0.50/day)
• Quality Impact: Minimal - relevance filtering is simpler than generation

Caching Strategy:

• Two-layer caching system:
  1. LangChain's SQLiteCache (automatic, based on prompt + inputs)
  2. Custom article_relevance table (explicit cache for article-level decisions)
• Cache key: MD5 hash of title:summary for deterministic lookup

3. Categorization System Evolution

Phase 1: LLM-Based Categorization

• Used GPT-3.5-turbo for categorization
• Cost: ~$0.02-0.05/day
• Quality: High, but slow

Phase 2: Keyword-Based Categorization (Current)

• Key Insight: Categories are well-defined with clear keywords
• Cost: $0.00/day (100% reduction)
• Speed: Instant (no API calls)
• Quality: Excellent for well-defined categories

Scoring Algorithm:

CATEGORIES = {
  "TOOLS_AND_FRAMEWORKS": {
    "keywords": ["agent", "mcp", "framework", "sdk"],
    "url_patterns": ["langchain.dev", "mistral"]
  }
}
# Keywords: +1 point per match
# URL patterns: +2 points per match
# Returns category with highest score

4. Ranking Agent

Category-Aware Ranking:

• Optimization: Only rank categories with >5 articles
• Saves LLM calls for small categories (keeps all articles)
• Reduces API costs by 20-40% depending on article distribution
• Focuses on "innovation, utility, and strategic impact"

5. Distribution & Analytics

Email Distribution:

• Individual emails per recipient (maximum privacy)
• Personalized tracking pixels for open rate analytics
• Gmail SMTP with proper authentication
• Error handling per recipient

Google Sheets Integration:

• Subscriber management with service account authentication
• Columns: Email, Subscribed, Timestamp, Unsubscribed_at
• Validates email format and checks for duplicates

Analytics Tracking:

• Email opens via 1x1 transparent tracking pixel
• JSONL file storage (one event per line, easy to process)
• Daily analytics: emails sent, opened, open rate

Cost Optimization Results

Before Optimizations:

• Relevance (GPT-4): ~$1.50-3.00/day
• Categorization (LLM): ~$0.02-0.05/day
• Macro Summary: ~$0.03/day
• Total: ~$1.57-3.18/day

After Optimizations:

• Relevance (GPT-3.5-turbo): ~$0.30-0.50/day
• Categorization (Keyword): $0.00/day
• Macro Summary: ~$0.03/day
• Total: ~$0.33-0.53/day

Savings: ~$1.24-2.65/day (56-80% reduction)

Optimization Techniques

1. Model Downgrading: GPT-4 → GPT-3.5-turbo for simple tasks
2. Keyword-Based Categorization: Replaced LLM with keyword matching
3. Conditional Ranking: Only rank large categories (>5 articles)
4. RSS Summaries: Use publisher summaries instead of AI-generated
5. Feature Flags: Make expensive features optional via config

Caching Architecture

Multi-Layer Caching Strategy:

Layer 1: LangChain SQLiteCache

• Automatic caching of LLM responses
• 30-50% hit rate on repeat runs

Layer 2: Custom Article-Level Cache

• Explicit caching of article-level decisions
• MD5 hash-based cache keys
• 40-60% hit rate when articles repeat

Performance Impact:

• Cost Savings: 30-50% reduction on repeat runs
• Speed Improvements: 10-1000x speedup on cached articles
• Estimated savings: $0.10-0.25/day on cached runs

Automation & CI/CD

GitHub Actions Workflow:

• Daily automated runs at 10:37 AM EST
• Secrets management via GitHub Secrets
• Artifact retention (30 days)
• Manual trigger option for testing

Secrets Managed:

• OpenAI API key
• Gmail app password
• Google Sheets credentials
• Sheet ID

Code Organization

Package Boundary Design:

• Core Package (rss_feed_summarizer/): Business logic for article processing
• Supporting Packages:
  • cost_tracking/: Isolated cost tracking with stable API
  • distribution/: Email and subscriber management
  • analytics/: Raw data storage and viewing tools
  • scripts/: Thin wrappers for execution

Benefits:

• Prevents circular dependencies
• Makes testing easier (mock external packages at boundaries)
• Clear ownership of functionality
• Faster onboarding for new developers

Key Technical Learnings

Architecture & Design

• Start Simple, Optimize Later: Build the simplest thing that works, measure, then optimize based on data
• Separation of Concerns: Strict package boundaries prevent complexity and enable testing
• Design for Failure: Real-world systems have transient errors—handle them gracefully
• Configuration Over Code: Make behavior changes easy without code modifications

Cost Optimization

• Measure Everything: Track costs from day one—you can't optimize what you don't measure
• Right Tool for Job: Not every task needs the most powerful model
• Cache Aggressively: Even 30% cache hit rate saves significant money over time
• Skip Unnecessary Work: Only process what's needed (conditional ranking)

Production Readiness

• Retry Logic: Exponential backoff for transient failures
• Error Handling: Continue processing even if individual steps fail
• Monitoring: Track performance metrics and cost per agent
• Automation: GitHub Actions ensures consistency and prevents human error

System Metrics

Future Improvements

Cost Optimizations:

• Batch processing (5-10 articles per API call)
• Similarity-based caching
• Model selection per article complexity

Architecture:

• Parallel processing with asyncio
• Streaming processing (process as articles are fetched)
• Database backend (PostgreSQL) for better querying

Features:

• A/B testing framework for prompts and models
• User personalization (category preferences)
• Multi-language support
• Advanced analytics dashboards

Outcome

Delivered a production-ready, cost-effective AI newsletter automation system that:

• Processes hundreds of articles daily
• Delivers high-quality curated content to subscribers
• Operates at <$0.50/day with 80% cost optimization
• Demonstrates advanced AI engineering principles: modularity, caching, cost optimization, and production automation

This project showcases how thoughtful architecture, data-driven optimization, and proper tooling can create efficient, scalable AI systems that deliver real value.