[KAFKA-12179] Install Kafka on IBM Cloud

• Type: Improvement
• Status: Closed
• Priority: Trivial
• Resolution: Invalid
• Affects Version/s: 2.7.0
• Component/s: docs, documentation
• External issue URL: https://github.com/arifnafees/kafka
• Assignee: Unassigned
• Reporter: Muhammad Arif
• Created: 11/Jan/21 19:57
• Updated: 25/Jan/21 09:21
• Resolved: 25/Jan/21 09:20

Description

Installing Kafka on IBM Cloud

 

This document will describe how to install Kafka on IBM Cloud using Kubernetes Service.

 

Contents

1. Introduction
2. Provision Kubernetes Cluster
3. Deploy IBM Cloud Block-Storage Plugin
4. Deploy Kafka
5. Verifying the Kafka Installation

 Introduction

To complete this tutorial, you should have an IBM Cloud account, if you do not have one, please register/signup here.

For installing Kafka, we have used the Kubernetes cluster, and used the IBM Cloud Block-Storage plugin for our persistent volume. Upon the completion of this tutorial, you would have the Kafka up and running on the Kubernetes cluster.

1. Provision the Kubernetes cluster, if you have already setup one, skip to step 2.
2. Deploy the IBM Cloud Block-Storage Plugin to the created cluster, if you have already done this, skip to step 3.
3. Deploy the Kafka.

Provision Kubernetes Cluster

• Click on the Catalog button on top center. Open Catalog. 

• In search catalog box, search for Kubernetes Service and click on it.
• You are now at Create Kubernetes Cluster page, there you have the two plans to create the Kubernetes cluster, either using free plan or standard plan.

 

Option1- Using Free Plan:

• Select Pricing Plan as “Free”.
• Click on Create.
• Wait a few minutes, and then your Cloud would be ready.

 

Note: Please be careful when choosing free cluster, as your pods could be stuck at pending state due to insufficient compute and memory resources, if you face such kind of issue please increase your resource by creating them choosing the standard plan.

 

Option2- Using Standard Plan:

• Select Pricing Plan as “Standard” 
• Select your Kubernetes Version as latest available or desired one by application. In our example we have set it to be ‘1.18.13’.
• 
• Select Infrastructure as “Classic”
• Leave Resource Group to “Default”
• Select Geography as “Asia” or your desired one.
• Select Availability as “Single Zone”.

This option allows you to create the resources in either single or multi availability zones. Multi availability zone provides you the option to create the resources in more than one availability zones so in case of catastrophe it could sustain the disaster and continues to work.

• Select Worker Zone as Chennai 01.
•  In Worker Pool, input your desired number of nodes as “3”
• Leave the Encrypt Local Disk option to “On”
• Select Master Service Endpoint to “Both private and public endpoints”
• Give your cluster-name as “Kafka-Cluster”
• Provide tags to your cluster and click on Create.
• Wait a few minutes, and then your Cloud would be ready.

Deploy IBM Cloud Block-Storage Plugin

• Click on the Catalog button on top center.
• In search catalog box, search for IBM Cloud Block Storage Plug-in and click on it
• Select your cluster as "Kafka-Cluster"
• Provide Target Namespace as “kafka-storage”, leave name and resource group to default
• Click on Install

 

Deploy Kafka

• Again go to the Catalog and search for Kafka.
• Provide the details as below.
• Target: IBM Kubernetes Service
• Method: Helm chart
• Kubernetes cluster: Kafka-Cluster(jp-tok)
• Target namespace: kafka
• Workspace: kafka-01-07-2021
• Resource group: Default
• Click on Parameters with Default Values, you can set the deployment values or use the default ones, we have used the default ones in this example.
• Click on Install.

Verifying the Kafka Installation

• Go to the Resources List in the Left Navigation Menu and click on ** Kubernetes and then Clusters.    
• Click on your created Kafka-Cluster.
• A screen would come up for your created cluster, click on Actions, and then Web Terminal
• A warning will appear asking you to install the Web Terminal, click on Install

• When the terminal is installed, click on the action button again and click on web terminal and type the following command in below command window. It will show you the workspaces of your cluster, you can see kafka active.

$ kubectl get ns

$ kubectl get pod –n Namespace –o wide 

$ kubectl get service –n Namespace 

                               

The installation is done. Enjoy!