11. Apache Kafka Best Practices

Before going to production: set replication factors (>=3), set min.insync.replicas to align durability goals, configure retention and segment sizes, and ensure disk provisioning with headroom. Harden security (TLS, auth, ACLs), enable monitoring and alerting (under-replicated partitions, consumer lag, disk usage), and automate backups/exports of critical topics or rely on cloud snapshots for disaster recovery. Run load and chaos testing, define runbooks for common failures, and establish metrics-based SLOs.

Encrypt data in transit (TLS) and at rest where possible, authenticate clients (SASL, mTLS, or cloud IAM integrations), and apply fine-grained ACLs to topics and consumer groups. Use a Schema Registry with authentication to prevent unauthorized schema changes and maintain schema compatibility settings. Rotate credentials and keys regularly, audit access logs, and segment network access using VPCs and security groups.

Optimize partition counts per topic based on consumption parallelism needs; pick appropriate message sizes and batch settings for both producers and consumers; monitor and tune JVM settings for brokers, and prioritize I/O with fast, durable disks. Use compression (snappy, lz4, zstd) to reduce network and storage usage while balancing CPU cost. Avoid large numbers of tiny partitions; they increase controller load and metadata overhead.

Avoid under-partitioning which limits consumer parallelism, and over-partitioning which increases cluster load and recovery time. Don’t ignore monitoring  many production incidents start with slow-growing lag or disk pressure.

Avoid relying on default security settings in production; defaults are convenient but insecure at scale. Finally, treat topic and schema management as first-class: undocumented topic semantics or uncoordinated schema changes are frequent causes of breakage in event-driven systems.