Skip to main content

Posts

Showing posts from August, 2020

kafka: cluster installation

Cluster Env Setup 1. remote ssh access using generated ssh key    a. ssh-keygen -t rsa    b. go to ~/.ssh/ and check for id_rsa and id_rsa.pub     c. ssh-copy-id -i ~/.ssh/id_rsa.pub <targetserver>    d.  more ~/.ssh/authorized_keys 2. download the kafka and zookeeper to each nodes 3. for each purpose, use clush to push the kafka to all nodes.     clush -g nodeGroup -c <path of kafka> Install zookeeper in Cluster 4. setup zookeeper:    a. rename zoo_sample to zoo.cfg    b. add the following to zoo.cfg for cluster setup.         server.1 = <host1>:2888:3888           server.2 = <host2>:2888:3888           server.3 = <host3>:2888:3888      2888: is used between leader and follower.     3888: is used for leader election. 5. sync the zoo.cfg to all clusters node 6. assign the different ID to different node.     1. create unique id under /tmp/zookeeper 7. start zookeeper on each node 8.    test if the all 3 zookeeper are synced.     1. login zookeeper conosle: