Optimizing Kafka is generally a function of the underlying hardware, not workload.
Something about Kafka being fast on SSDs. I was given to understand that Kafka’s sweet spot is on spinning disks because it favours high-throughput batch reads. Is this not true anymore?
Two new proposals in review
Tiered storage: store recent data in Kafka, and move cold data to another storage system. PoCs for HDFS and S3. This makes broker additions a lot faster because a lot less data needs to be copied to the new node.
Remove zookeeper: Use Kafka’s replicated log (via Raft, I think?) to store metadata instead of a separate system. This should allow scaling (theoretically) to tens of millions of partitions.