Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Show HN: StreamHouse – S3-native Kafka alternative written in Rust (github.com/gbram1)
10 points by gbram 16 days ago | hide | past | favorite | 8 comments
Hey HN,

I built StreamHouse, an open-source streaming platform that replaces Kafka's broker-managed storage with direct S3 writes. The goal: same semantics, fraction of the cost.

How it works: Producers batch and compress records, a stateless server manages partition routing and metadata (SQLite for dev, PostgreSQL for prod), and segments land directly in S3. Consumers read from S3 with a local segment cache. No broker disks to manage, no replication factor to tune — S3 gives you 11 nines of durability out of the box.

What's there today: - Producer API with batching, LZ4 compression, and offset tracking (62K records/sec) - Consumer API with consumer groups, auto-commit, and multi-partition fanout (30K+ records/sec) - Kafka-compatible protocol (works with existing Kafka clients) - REST API, gRPC API, CLI, and a web UI - Docker Compose setup for trying it locally in 5 minutes

The cost model is what motivated this. Kafka's storage costs scale with replication factor × retention × volume. With S3 at $0.023/GB/month, storing a TB of events costs ~$23/month instead of hundreds on broker EBS volumes.

Written in Rust, ~50K lines across 15 crates. Apache 2.0 licensed.

GitHub: https://github.com/gbram1/streamhouse

Happy to answer questions about the architecture, tradeoffs, or what I learned building this.



Running the quickstart snippet I get a bunch of errors and a warning that my anthropic key is not set. Smells vibe coded!


Fixed. quickstart works end to end now, no API key needed. Would appreciate a second look if you have a few minutes.

reply


Working on a few fixes now. Will update the quick start and let you know when it’s all set!


where are you using this in production?


For systems that can tolerate slightly higher latency, and prefer operational simplicity. A good example would be a ecommerce platform that is streaming order history into a data warehouse. They likely don’t need sub-millisecond latency that Kafka provides, and can use a simpler, significantly cheaper option to achieve the same result


I thought we were all ditching minio as it is no longer maintained


I still find it useful as a local blob storage and no big code changes to cut over to s3 since it’s the same APIs


Try rustfs

reply




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: