kafka: cluster installation

Cluster Env Setup

1. remote ssh access using generated ssh key

a. ssh-keygen -t rsa

b. go to ~/.ssh/ and check for id_rsa and id_rsa.pub

c. ssh-copy-id -i ~/.ssh/id_rsa.pub <targetserver>

d. more ~/.ssh/authorized_keys

2. download the kafka and zookeeper to each nodes

3. for each purpose, use clush to push the kafka to all nodes.

clush -g nodeGroup -c <path of kafka>

4. setup zookeeper:

a. rename zoo_sample to zoo.cfg

b. add the following to zoo.cfg for cluster setup.

server.1 = <host1>:2888:3888

server.2 = <host2>:2888:3888

server.3 = <host3>:2888:3888

2888: is used between leader and follower.

3888: is used for leader election.

5. sync the zoo.cfg to all clusters node

6. assign the different ID to different node.

1. create unique id under /tmp/zookeeper

7. start zookeeper on each node

8. test if the all 3 zookeeper are synced.

1. login zookeeper conosle: ./zookeeper-shell.sh <nodename>:2181

2. create /hello-test test

3. login all of other zookeeper and all of them should see the created path: /hello-test

1. update the kafka.properties file to use zookeeper cluster for all node

2. update the broker id for each of node.

Here, we just set the broker id as 1, 2, 3 for each node.

3. start the kafka on each node: clush -g kafka {path}/kafka-server-start.sh ../config/server.properties --daemon

4. verify the kafka cluster works by creating topic.