Hybrid Fanout Feed System is a Twitter-style news feed backend written by Sajal Halder in Java with Spring Boot. It chooses how to deliver each post based on how many followers the author has: accounts under 10,000 followers use fan-out on write, and accounts with 10,000 or more use fan-out on read.
It is a demonstration project for learning distributed systems architecture. The performance figures on this page are design targets from the project, not measurements from production traffic.
Why not use one strategy
Pushing every post into every follower's feed makes reads fast, but an account with millions of followers turns one post into millions of writes. Building every feed at read time avoids that write cost, but it makes each feed request expensive. Neither works well for every account, so the system picks a strategy per author.
Fan-out on write, for accounts under 10K followers
- The post is validated (280 characters maximum) and saved to the posts table in Cassandra.
- Follower IDs are read from PostgreSQL.
- The post ID is added asynchronously to each follower's feed in Redis, a sorted set scored by timestamp, in batches of 100.
- Feeds are pre-computed, so reading one is a fast cache lookup.
Fan-out on read, for accounts with 10K or more followers
- The post is saved to a celebrity posts table in Cassandra, partitioned by user ID and ordered by creation time.
- No fan-out happens at write time, which removes the write amplification.
- When a follower opens their feed, the newest posts from the celebrities they follow are queried and merged in.
Reading a feed
The feed service combines both paths. It reads cached post IDs from the user's Redis feed, then loads each post's metadata from Redis, falling back to Cassandra and re-caching on a miss. It then looks up which celebrities the user follows in PostgreSQL and fetches their recent posts from Cassandra.
A feed merger combines the two lists, sorts them newest first, applies the limit and returns a page with a has-more flag for infinite scrolling. Page size is capped at 100.
Why three databases
- PostgreSQLUsers and follow relationships, where ACID transactions and joins matter. Celebrity status is maintained automatically by database triggers.
- CassandraPosts and celebrity posts, chosen for high write throughput and time-series access, with tables partitioned for efficient queries.
- RedisPre-computed feeds and post metadata. Entries expire after one hour by default, and the cache uses LRU eviction with a 512 MB limit.
Performance targets
The design aims for feed generation under 100 ms at p95, feed retrieval under 150 ms at p95, post creation under 200 ms for regular accounts and under 50 ms for celebrity accounts. The supporting techniques are asynchronous fan-out, batched processing, multi-level caching and TTL-based expiry.