AI-News-Ranker: Architecture Design and Implementation of an Intelligent News Aggregation System

AI-News-Ranker is a real-time AI news aggregation platform that integrates AI field information from over 50 trusted sources. It uses semantic deduplication, intelligent scoring, and topic clustering technologies to build a modern information distribution system. Key technologies include Next.js, Supabase, pgvector, Redis cache, and Claude, addressing core pain points like information overload, content repetition, uneven quality, and insufficient real-time performance.

In the information explosion era, efficiently obtaining high-quality AI information is a pain point for developers and researchers. Manual tracking of scattered AI content (papers, blogs, product releases) is time-consuming and error-prone. AI-News-Ranker solves four core issues: information overload, content repetition, uneven quality, and lack of real-time updates. Its core approach is cross-source corroboration—events reported by multiple trusted sources rank higher, resisting clickbait and robot manipulation.

The system uses a modern layered architecture:

• Frontend: Built with Next.js 16 (App Router, TypeScript) and Tailwind CSS v4 for a modern UI.
• Load Balancing: Nginx as reverse proxy and load balancer, supporting horizontal scaling via upstream node configuration.
• Data Layer: Three-tier structure—Supabase (PostgreSQL + pgvector) for core data, Redis for regional key-value caching, S3 for thumbnail storage.
• Workflow Layer: Independent worker containers handling tasks like content ingestion, enrichment (LLM summaries/scores), topic clustering, and notification pushes.

Key functions include:

1. Multi-source aggregation: 50+ verified sources (labs like OpenAI/DeepMind, infrastructure vendors, researcher blogs, newsletters, safety orgs, academic resources like arXiv). Each source undergoes strict validation (URL reachability, RSS parsing, content freshness).
2. Semantic deduplication: Uses Voyage AI's voyage-3 model to generate 1024-dimensional vectors, with pgvector for cosine similarity calculation to merge similar articles.
3. Intelligent scoring: Claude Haiku assigns 0-100 scores based on source weight sum, topic size, LLM-assessed importance, and time decay (excludes clicks/views).
4. Real-time push: High-priority content via Discord webhook; Supabase Realtime WebSocket for frontend updates without polling.

Notable technical features:

• Generic crawler adapter: Config-driven for non-RSS sources (CSS selectors for content extraction, no code changes needed).
• Source validation: scripts/verify-sources.mjs checks HTTP status, parsing correctness, article count, freshness (60-day window), and optional Claude relevance scoring.
• Data pipeline validation: scripts/verify-pipeline.sql ensures end-to-end data flow correctness (items.xml, region defaults, S3 thumbnails, deduplication effect).

Deployment:

• POC: Docker Compose (copy .env.example to .env.local, configure keys, run docker compose up --build).
• Horizontal scaling:

1. Copy app service to app2/app3.
2. Add upstream nodes in nginx.conf.
3. Optional least_conn scheduling.
4. Redeploy. Redis and Supabase act as shared state layers for new instances.

Engineering practices:

• Region-keyed cache with conservative invalidation to ensure freshness.
• Degradation paths: Fallback to database if Redis fails, placeholder for missing S3 thumbnails, skip summaries if LLM API is limited.
• Observability: topic_engagement table for user behavior analysis (not used for sorting). Applicable scenarios: AI community portal, enterprise tech intelligence, research literature tracking, media topic assistant. Future plans: Multi-language support, personalized recommendations, more notification channels (Slack/email), third-party API access.

AI-News-Ranker demonstrates a complete tech stack for modern news aggregation, with engineering thinking in every aspect (multi-source collection, semantic deduplication, smart scoring, real-time push). Its cross-source corroboration approach provides a sustainable solution for information quality assessment. For developers, it offers runnable code and clear architecture docs; for users, it helps understand aggregation tools better.