6. Kafka Security Fundamentals

Kafka supports several authentication mechanisms, but the most common in production are SSL client authentication and SASL. SSL client authentication uses mutual TLS, where both client and broker verify each other’s certificates. This works well in environments that already have certificate management in place.

SASL is a framework that supports multiple mechanisms. SASL/PLAIN is simple but must always run over SSL, while SASL/SCRAM is safer because it uses challenge-response instead of sending passwords directly.

SASL/GSSAPI is Kerberos-based and common in enterprise environments, while SASL/OAUTHBEARER is useful in cloud-native setups with external identity providers.

Authentication answers who is connecting, while authorization answers what that identity is allowed to do. Kafka uses ACLs to define permissions on topics, consumer groups, and transactional IDs. Each ACL specifies a principal, a resource, an operation, and whether it is allowed or denied.

ACLs are managed through kafka-acls.sh or admin APIs. A producer typically needs Write permission, while a consumer needs Read on the topic and Describe on its group. In larger environments, teams often use wildcard patterns, naming conventions, and infrastructure-as-code tools to manage ACLs at scale.

SSL/TLS protects Kafka traffic from interception between clients and brokers. It encrypts client-to-broker and broker-to-broker communication using keystores and truststores. Brokers need SSL listener settings plus certificate paths and passwords, while clients need matching truststore settings.

SSL adds some CPU overhead, but the cost is usually modest on modern hardware. High-throughput systems may reduce overhead with session resumption or TLS offloading. In most production setups, per-broker SSL with regular certificate rotation is the preferred approach.