Kafka Partition Count Is a Floor, Not a Target
This calculator treats partition count as a minimum derived two independent ways, then takes whichever is larger: a throughput-based floor (ceil(targetThroughput / 10), using a conservative 10 MB/s-per-partition write ceiling) and a consumer-based floor (numConsumers, since Kafka only lets one consumer per group read a given partition at a time). Storage is computed separately from partition count: totalStorageMB = targetThroughput × retentionSeconds × replicationFactor. At 100 MB/s with 7-day retention (604,800 seconds) and RF=3, that alone works out to over 170 TB sitting on disk simultaneously, before any tiered storage or log compaction reclaims space.
Replication Multiplies Network IO, Not Just Disk
The chart's IO breakdown includes replicationThroughput = targetThroughput × (replicationFactor - 1) — every follower replica pulls the full producer stream from the leader over the broker network, independent of what consumers pull out. At RF=3, replication alone adds 2x the producer's throughput to broker-to-broker traffic, which is why it's usually the largest bar once RF climbs past 2.
Sizing Gotchas This Model Simplifies
- Partitions only go up: Kafka lets you increase a topic's partition count but never decrease it without recreating the topic and losing key-to-partition ordering guarantees.
- 10 MB/s is conservative: real clusters often push 10–50 MB/s per partition depending on message size, batching, and compression codec (
lz4/zstdvs uncompressed) — treat the calculator's figure as a safe floor, not a ceiling. - Over-partitioning has a cost: more partitions means more open file handles per broker and longer leader-election time during a broker failure, so padding "just in case" isn't free.