Kafka installation locally on Ubuntu 18

In this tutorial we are going to install Kafka on Ubuntu 18.04 TLS

Prerequisites:

  • Ubuntu OS
  • Java JDK 8
  • 4GB of RAM on the server.

1- First check the java version :

$ java –version

2- Go to Kafka website to download

selected “Scala 2.12 version”

create downlaod directory

$ mkdir ~/kafka && cd ~/kafka

and download the file into kafka directory

open tar file kafka_2.12-2,1,0.tgz

$ tar -xvf kafka_2.12-2.1.0.tgz

you should be able to see below files under your bin directory.

Test your installation using below command:

$ ./bin/kafka-topics.sh

Kafka Architecture:

Zookeeper : Which is used by Kafka to maintain state between the nodes of the cluster.

Kafka brokers : The “pipes” in our pipeline, which store and emit data.

Producers : That insert data into the cluster.

Consumers : That read data from the cluster.

First we need to start zookeeper: lets check zookeeper config file where we specify client port

Run the command

$ bin/zookeeper-server-start.sh ../config/zookeeper.properties

You should see a confirmation that the server has started.

$ bin/kafka-server-start.sh ../config/server.properties

Its time to create our first topic :

$ ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic firstTopic

Created topic “firstTopic”

We can list the topics which we created using below command:

$ /kafka-topics.sh –list –zookeeper localhost:2181

Now check how producer and consumer works briefly:

I started my consumer : (we already created the topic)

$ ./kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic firstTopic –from-beginning

I started my consumer:

$ ./kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic FirstTopic –from-beginning