Are you struggling to deploy NLB and Kafka Controller in the same Availability Zone (AZ) using Bitnami Kafka Chart on AWS EKS? Look no further! In this article, we’ll take you through a step-by-step guide on how to achieve this deployment with ease.
Prerequisites
Before we dive into the deployment process, make sure you have the following prerequisites in place:
- AWS account with an active subscription
- AWS EKS cluster set up and running
- Bitnami Kafka Chart installed on your EKS cluster
- Kubernetes command-line tool (kubectl) installed on your machine
Understanding the Challenge
When deploying Kafka on AWS EKS using Bitnami Kafka Chart, one of the significant challenges is ensuring that the NLB (Network Load Balancer) and Kafka Controller are deployed in the same Availability Zone (AZ). This is crucial for high availability and performance reasons.
The default behavior of Bitnami Kafka Chart is to deploy the NLB and Kafka Controller in different AZs, which can lead to increased latency and reduced performance. To overcome this, we need to tweak the chart’s configuration to deploy both components in the same AZ.
Configuring the Bitnami Kafka Chart
To configure the Bitnami Kafka Chart to deploy NLB and Kafka Controller in the same AZ, follow these steps:
Step 1: Update the values.yaml file
Create a copy of the `values.yaml` file from the Bitnami Kafka Chart repository and update it with the following configuration:
controller:
replicas: 3
nodeSelector:
zone: us-west-2a
topologySpreadConstraints:
- maxSkew: 1
topologyKey: "kubernetes.io/hostname"
whenUnsatisfiable: DoNotSchedule
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: kafka
app.kubernetes.io/component: controller
topologyKey: "kubernetes.io/hostname"
This configuration updates the controller’s `nodeSelector` to deploy in the `us-west-2a` AZ, and adds a `topologySpreadConstraints` to ensure that the controller replicas are spread across different nodes in the same AZ.
Step 2: Update the Kafka Cluster Configuration
Update the Kafka cluster configuration to include the following settings:
kafka:
config:
"controller.quorum.voters": "zk:2181"
"controller.quorum.make_observers": "zk:2181"
"controller.quorum.learners": "zk:2181"
listeners:
- internal: PLAINTEXT://:9092
external: PLAINTEXT://localhost:9094
ssl:
enabled: false
This configuration updates the Kafka cluster to use the internal listener on port 9092 and the external listener on port 9094, without SSL encryption.
Deploying the NLB and Kafka Controller
Now that we have updated the configuration files, let’s deploy the NLB and Kafka Controller using the following command:
helm install my-kafka-chart --set external-access.enabled=true --set external-access.service.type=LoadBalancer --set external-access.service.loadBalancerIPs[0]="<YOUR_EXTERNAL_IP>" --set kafkadata.pvc.size=<SIZE> bitnami/kafka
Replace `` with the external IP address of your NLB, and `` with the desired size of the persistent volume claim (PVC).
Verifying the Deployment
Once the deployment is complete, verify that the NLB and Kafka Controller are running in the same AZ by using the following command:
kubectl get pods -o wide
This command will display the list of pods running in your cluster, including the Kafka Controller and NLB. Check the `NODE` column to ensure that both components are running on nodes in the same AZ.
Troubleshooting Common Issues
If you encounter any issues during the deployment process, here are some common troubleshooting steps to follow:
-
Error: “Invalid configuration: Controller topologySpreadConstraints maxSkew must be greater than 0”
Solution: Update the `maxSkew` value in the `topologySpreadConstraints` section of the `values.yaml` file to a value greater than 0.
-
Error: “Failed to create NLB: ‘invalid IP address’
Solution: Ensure that the external IP address specified in the `helm install` command is valid and reachable.
-
Error: “Kafka Controller not starting: ‘kafka.config’ not found”
Solution: Ensure that the Kafka configuration files are correctly mounted and accessible by the Kafka Controller pod.
Conclusion
In this article, we have walked you through the process of ensuring NLB and Kafka Controller deployment in the same AZ using Bitnami Kafka Chart on AWS EKS. By following these steps and configuring the chart correctly, you can achieve high availability and performance for your Kafka cluster.
| Component | AZ |
|---|---|
| NLB | us-west-2a |
| Kafka Controller | us-west-2a |
Remember to update the `values.yaml` file and Kafka cluster configuration to match your specific requirements. Happy deploying!
Frequently Asked Questions
Get your answers on deploying NLB and Kafka Controller in the same AZ using Bitnami Kafka Chart on AWS EKS!
Q: What is the recommended approach to ensure NLB and Kafka Controller deployment in the same AZ?
A: To ensure NLB and Kafka Controller deployment in the same AZ, you should create a separate Availability Zone (AZ) and specify it in the `kafka.yaml` file under `cluster`. This will allow you to deploy NLB and Kafka Controller in the same AZ, ensuring high availability and low latency.
Q: How do I configure the Bitnami Kafka Chart to deploy NLB and Kafka Controller in the same AZ?
A: You can configure the Bitnami Kafka Chart by adding the `zone` property to the `values.yaml` file. Set the `zone` property to the same AZ where you want to deploy NLB and Kafka Controller. This will ensure that the chart deploys the components in the specified AZ.
Q: What are the benefits of deploying NLB and Kafka Controller in the same AZ using Bitnami Kafka Chart on AWS EKS?
A: Deploying NLB and Kafka Controller in the same AZ using Bitnami Kafka Chart on AWS EKS provides several benefits, including improved latency, increased throughput, and better fault tolerance. This setup also allows for more efficient resource utilization and simplified management.
Q: How do I verify that NLB and Kafka Controller are deployed in the same AZ after using the Bitnami Kafka Chart?
A: You can verify the deployment by checking the AWS EKS dashboard, Kubernetes cluster, or using the `kubectl` command-line tool. Look for the `Availability Zone` label in the deployment YAML files or the EKS cluster console to confirm that NLB and Kafka Controller are running in the same AZ.
Q: Are there any additional considerations or prerequisites for deploying NLB and Kafka Controller in the same AZ using Bitnami Kafka Chart on AWS EKS?
A: Yes, it’s essential to ensure that you have a compatible Kubernetes version, sufficient resources (e.g., CPU, memory, and storage), and proper network configurations in place. You should also review the Bitnami Kafka Chart documentation and AWS EKS best practices for deploying NLB and Kafka Controller in the same AZ.
