Simple way to secure Kafka / Zookeeper cluster on Kubernetes

After successful Kafka and Zookeeper deployment on Kubernetes following this post, it's time to make it more secure.

1- ZooKeeper DIGEST authentication

Account configuration

(one for Zookeeper nodes, another for Kafka broker) in config.secured.yaml file.

Client section will be used by zkCli.sh helper srcipt to test configuration. In integration environement, Client section must be removed.

apiVersion: v1
kind: Secret
metadata:
  name: zookeeper-jaas
type: Opaque
stringData:
  zookeeper-jaas.conf: |-
    QuorumServer {
          org.apache.zookeeper.server.auth.DigestLoginModule required
          user_zk="passcode";
    };
    QuorumLearner {
          org.apache.zookeeper.server.auth.DigestLoginModule required
          username="zk"
          password="passcode";
    }; 
    Server {
          org.apache.zookeeper.server.auth.DigestLoginModule required
          user_kafka="passcode"
          user_client="passcode";
    };
    Client {
          org.apache.zookeeper.server.auth.DigestLoginModule required
          username="client"
          password="passcode";
    };

Create Secret / ConfigMap and deploy StatefuSet:

kubectl apply -f zookeeper/config.secured.yaml 
kubectl apply -f zookeeper/statefulset.secured.yaml 
# Secret for kafka
kubectl apply -f kafka/config.secured.yaml

2- Kafka SSL encryption

If your want to disable the Host Name verification for some raison, you need to omit the extension parameter -ext and add empty environment variable to Kafka broker KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM="" or use Kafka helper script for every broker.

kafka-configs.sh --bootstrap-server [kafka-0].kafka-broker.kafka.svc.cluster.local:9092 --entity-type brokers --entity-name 0 --alter --add-config "listener.name.internal.ssl.endpoint.identification.algorithm="

Create CA (certificate authority)

openssl req -new -x509 -keyout ca-key -out ca-cert -days 365

Generate server keystore and client keystore

keytool -keystore kafka.server.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA 
keytool -keystore kafka.client.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA

Add generated CA to the trust store

keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert 
keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert

Sign the key store (with passcode and ssl.cnf)

You need to update alt_names section of ssl.cnf with list of your brokers hostname.

keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file 
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:passcode -extfile ssl.cnf -extensions req_ext 
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert 
keytool -keystore kafka.server.keystore.jks -alias localhost -import -file cert-signed

Sign the client keystore

keytool -keystore kafka.client.keystore.jks -alias localhost -certreq -file cert-file-client 
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file-client -out cert-signed-client -days 365 -CAcreateserial -passin pass:passcode -extfile ssl.cnf -extensions req_ext 
keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file ca-cert 
keytool -keystore kafka.client.keystore.jks -alias localhost -import -file cert-signed-client

Kafka SSL Kubernetes

Create kubernetes secret from kafka.keystore.jks and kafka.truststore.jks

kubectl create secret generic ssl --from-literal=keystore_password=passcode --from-file=kafka.keystore.jks=ssl/kafka.server.keystore.jks --from-literal=truststore_password=passcode --from-file=kafka.truststore.jks=ssl/kafka.server.truststore.jks

Update Kafka StatefulSet to mount ssl secret and broker, service configuration:

kubectl apply -f kafka/statefulset.ssl.yaml kubectl apply -f kafka/service.ssl.yaml

Testing

Use openssl to debug connectionto valid certificate data:

kubectl exec -ti zk-0 -- openssl s_client -debug -connect kafka-0.kafka-broker.kafka.svc.cluster.local:9093 -tls1

Create client configuration file (client.properties):

security.protocol=SSL
ssl.truststore.location=/opt/kafka/config/kafka.truststore.jks
ssl.truststore.password=passcode
ssl.keystore.location=/opt/kafka/config/client.keystore.jks
ssl.keystore.password=passcode ssl.key.password=passcode
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

Send to test pod:

mkdir -p config 
cp client.properties config/client.properties 
cp ssl/kafka.client.keystore.jks config/client.keystore.jks 
cp ssl/kafka.client.truststore.jks config/kafka.truststore.jks 
kubectl cp config kafka-1:/opt/kafka

Run test:

kubectl exec -ti kafka-1 -- kafka-console-producer.sh --bootstrap-server kafka-0.kafka-broker.kafka.svc.cluster.local:9093 --topic k8s --producer.config /opt/kafka/config/client.properties 
>> Hello with secured connection

kubectl exec -ti kafka-1 -- kafka-console-consumer.sh --bootstrap-server kafka-0.kafka-broker.kafka.svc.cluster.local:9093 --topic k8s -consumer.config /opt/kafka/config/client.properties --from-beginning 
<< Hello with secured connection

Prepare Secret for client:

kubectl create secret generic client-ssl --from-file=ca-certs.pem=ssl/ca-cert --from-file=cert.pem=ssl/cert-signed-client 
kubectl apply -f consumer.secured.yaml kubectl logs consumer-secured

Sources & Links

@GitHub source