Running Kafka on Kubernetes for local development with Storage class
In a recent post I showed a setup to run Kafka on Kubernetes locally using Persistent Volumes(PV) and Persistent Volume Claims(PVC), this post covers the setup using Kubernetes Storage Class(SC) and leveraging the default one provisioned automatically by Kind which is the Rancher local-path-provisioner.
This setup is simpler and uses less code than the previous one with the trade off of having a bit less control over the path of data externalized to the host machine while requiring some internal knowledge of Kind to set it up. In general I favor this approach for local development over the one from my previous post.
Strimzi is an awesome simpler alternative to achieve the same, check it out. My goal with this setup here is for learning and to have a more "realistic" Kubernetes setup on local development machine so I opted to not use Strimzi or Helm charts.
I have created and tested these approaches on a Linux Development machine. It should work for Mac and Windows with some minimal adjustments also but I have never tried it.
You can get the full source from Github repo where you will find the files and Quick Start for both aforementioned approaches. To clone the repo git clone git@github.com:mmaia/kafka-local-kubernetes.git.
Pre-reqs, install:
The setup using Storage class
If you checked out the repo described above the setup presented here is under storage-class-setup
folder. You will find multiple Kubernetes declarative files in this
folder, please notice that you could also combine all files in a single
one separating them with a line containing triple dashes ---, if combining them is your preference you can open a terminal and from the storage-class-setup folder run for each in ./kafka-k8s/*; do cat $each; echo "---"; done > local-kafka-combined.yaml this will concatenate all files in a single one called local-kafka-combined.yaml.
The main thing to notice in this setup below compared to the previous one is that you don't have any PV or PVC configuration files this time because we're leveraging the default Rancher local-path-provisioner provided by Kind automatically through it's default Storage class.
I keep them separate to explicitly separate each type in this case and because it's convenient as you can just run kubectl pointing to the directory as described below in the "Running it" section.
kind-config.yaml - This file configures Kind to expose
the kafka and schema-registry ports to the local machine host so you can
connect your application while developing from your IDE or command line
and connect with Kafka running on Kubernetes.
kind-config.yaml - This file configures Kind to expose
the kafka and schema-registry ports to the local machine host so you can
connect your application while developing from your IDE or command line
and connect with Kafka running on Kubernetes it also maps the default
path of the Rancher storage provisioner from Kind container to your
local host machine.
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
- role: control-plane
- role: worker
extraPortMappings:
- containerPort: 30092 # internal kafka nodeport
hostPort: 9092 # port exposed on "host" machine for kafka
- containerPort: 30081 # internal schema-registry nodeport
hostPort: 8081 # port exposed on "host" machine for schema-registry
extraMounts:
- hostPath: ./tmp
containerPath: /var/local-path-provisioner
readOnly: false
selinuxRelabel: false
propagation: Bidirectional
kafka-network-np.yaml - Sets up the internal Kubernetes network used by the setup.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: kafka-network
spec:
ingress:
- from:
- podSelector:
matchLabels:
network/kafka-network: "true"
podSelector:
matchLabels:
network/kafka-network: "true"
kafka-service.yaml - This file defines the mappings between the internal containers and ports that are exposed, called NodePorts by defaul in Kubernetes nodeports can be used in the range 30000 to 32767.
apiVersion: v1
kind: Service
metadata:
labels:
service: kafka
name: kafka
spec:
selector:
service: kafka
ports:
- name: internal
port: 29092
targetPort: 29092
- name: external
port: 30092
targetPort: 9092
nodePort: 30092
type: NodePort
kafka-ss.yaml - This is the definition of Kafka in this setup, this time we use a Stateful Set.
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
service: kafka
name: kafka
spec:
serviceName: kafka
replicas: 1
selector:
matchLabels:
service: kafka
template:
metadata:
labels:
network/kafka-network: "true"
service: kafka
spec:
enableServiceLinks: false
containers:
- name: kafka
imagePullPolicy: IfNotPresent
image: confluentinc/cp-kafka:7.0.1
ports:
- containerPort: 29092
- containerPort: 9092
env:
- name: CONFLUENT_SUPPORT_CUSTOMER_ID
value: "anonymous"
- name: KAFKA_ADVERTISED_LISTENERS
value: "INTERNAL://kafka:29092,LISTENER_EXTERNAL://kafka:9092"
- name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
value: "true"
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_DEFAULT_REPLICATION_FACTOR
value: "1"
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: "INTERNAL"
- name: KAFKA_LISTENERS
value: "INTERNAL://:29092,LISTENER_EXTERNAL://:9092"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT"
- name: KAFKA_NUM_PARTITIONS
value: "1"
- name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
value: "1"
- name: KAFKA_LOG_CLEANUP_POLICY
value: "compact"
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper:2181"
resources: {}
volumeMounts:
- mountPath: /var/lib/kafka/data
name: kafka-data
hostname: kafka
restartPolicy: Always
volumeClaimTemplates:
- metadata:
name: kafka-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
The remaining files are declarative Kubernetes configurations files to schema-registry and zookeeper.
schema-registry-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
service: schema-registry
name: schema-registry
spec:
replicas: 1
selector:
matchLabels:
service: schema-registry
strategy: {}
template:
metadata:
labels:
network/kafka-network: "true"
service: schema-registry
spec:
enableServiceLinks: false
containers:
- env:
- name: SCHEMA_REGISTRY_HOST_NAME
value: "schema-registry"
- name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
value: "kafka:29092"
- name: SCHEMA_REGISTRY_LISTENERS
value: "http://0.0.0.0:30081"
image: confluentinc/cp-schema-registry:7.0.1
name: schema-registry
ports:
- containerPort: 30081
resources: {}
hostname: schema-registry
restartPolicy: Always
schema-registry-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
service: schema-registry
name: schema-registry
spec:
ports:
- port: 30081
name: outport
targetPort: 30081
nodePort: 30081
type: NodePort
selector:
service: schema-registry
zookeeper-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
service: zookeeper
name: zookeeper
spec:
ports:
- name: "2181"
port: 2181
targetPort: 2181
selector:
service: zookeeper
zookeeper-ss.yaml - Again the main difference this time is the usage of Stateful Set.
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
service: zookeeper
name: zookeeper
spec:
serviceName: zookeeper
replicas: 1
selector:
matchLabels:
service: zookeeper
template:
metadata:
labels:
network/kafka-network: "true"
service: zookeeper
spec:
enableServiceLinks: false
containers:
- name: zookeeper
imagePullPolicy: IfNotPresent
image: confluentinc/cp-zookeeper:7.0.1
ports:
- containerPort: 2181
env:
- name: ZOOKEEPER_CLIENT_PORT
value: "2181"
- name: ZOOKEEPER_DATA_DIR
value: "/var/lib/zookeeper/data"
- name: ZOOKEEPER_LOG_DIR
value: "/var/lib/zookeeper/log"
- name: ZOOKEEPER_SERVER_ID
value: "1"
resources: {}
volumeMounts:
- mountPath: /var/lib/zookeeper/data
name: zookeeper-data
- mountPath: /var/lib/zookeeper/log
name: zookeeper-log
hostname: zookeeper
restartPolicy: Always
volumeClaimTemplates:
- metadata:
name: zookeeper-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 250Mi
- metadata:
name: zookeeper-log
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 250Mi
Running it
- Open a terminal and cd to the storage-class-setup folder.
- Create a folder called "tmp", this is where the storage will be automatically provisioned by the default Kind storage class.
- Run kind specifying configuration:
kind create cluster --config=kind-config.yml. This will start a Kind kubernetes control plane + worker. - Run kubernetes configuration for kafka
kubectl apply -f kafka-k8s - When done stop kubernetes objects:
kubectl delete -f kafka-k8sand then if you want also stop the kind cluster which will also delete the storage on the host machine:kind delete cluster.
After running the
kubectl applycommand(step 4 above) check your local tmp folder where you will find the automated storage mapped to your local host disk, notice that those folders will be deleted when you shutdown the Kind cluster but they will persist over pod restarts of Kafka and zookeeper.
Connecting a Kafka client
I decided to add this section because I commonly see developers struggling to connect their Kafka clients from the IDE or local dev machine to a local Kafka running on docker, docker-compose or kubernetes, mostly because the client gives an error that it can't find host kafka when running.
There are many ways to solve that I will explain a few ways here.
The simplest way to do it is to add kafka and schema-registry to your /etc/hosts file in the host machine, so when the broker configures
LISTENER_EXTERNAL://kafka:9092which will be advertised to the kafka client the client will resolvekafkato localhost and it will just work. In some configurations you do the same withschema-registry. Some people don't like to do this so there are options.Change the configuration on your compose file or kubernetes kafka config file to
LISTENER_EXTERNAL://localhost:9092, so now when in your local client you specify kafka broker address to localhost:9092 it will just work as it will match the advertised address the client received from the broker. The downside of doing this is that if you decide to package your application to run locally in a container inside the docker-compose or kubernetes network the call will fail as localhost will then point to the application container / pod instead of kafka.When using Kubernetes use a tool like Telepresence to expose and route kubernetes ports.
When using docker-compose you might leverage the docker internal address mappings as describe nicely in this article.
You can find a detailed explanation on why this happens in this Confluent blog post.
That's it, it's done, you have a functional local Kafka +
Schema Registry running on Kubernetes that you can reach from your application running on your developer machine or IDE.
