Skip to main content

How to run Kafka from source code - Kafka Source Code

Get source code of Kafka, the version is 2.3

 git clone -b 2.3 https://github.com/apache/kafka.git

Generate Jar File

./gradlew jar

Start the zookeeper

bin/zookeeper-server-start.sh config/zookeeper.properties


Since we are mainly focus on Kafka source code, we won't spend some time to dig into how the zookeeper start by using jars directly.

Start the Kafka Server

Option 1: start with script

bin/kafka-server-start.sh config/server.properties


Option 2:

If we inspect the real command from kafka-run-class.sh which is called by kafka-start-server.sh, here is the final command to run:
exec /Library/Java/JavaVirtualMachines/jdk-11.0.1.jdk/Contents/Home/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Xlog:gc*:file=/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../logs/kafkaServer-gc.log:time,tags:filecount=10,filesize=102400 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../logs -Dlog4j.configuration=file:bin/../config/log4j.properties -cp /Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../core/build/dependant-libs-2.12.8/*:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../examples/build/libs/kafka-examples-2.3.1-SNAPSHOT-sources.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../examples/build/libs/kafka-examples-2.3.1-SNAPSHOT.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../clients/build/libs/kafka-clients-2.3.1-SNAPSHOT-sources.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../clients/build/libs/kafka-clients-2.3.1-SNAPSHOT.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../streams/build/libs/kafka-streams-2.3.1-SNAPSHOT-sources.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../streams/build/libs/kafka-streams-2.3.1-SNAPSHOT.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../streams/examples/build/libs/kafka-streams-examples-2.3.1-SNAPSHOT.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../streams/build/dependant-libs-2.12.8/rocksdbjni-5.18.3.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../tools/build/libs/kafka-tools-2.3.1-SNAPSHOT-sources.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../tools/build/libs/kafka-tools-2.3.1-SNAPSHOT.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../tools/build/dependant-libs-2.12.8/*:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../connect/api/build/libs/connect-api-2.3.1-SNAPSHOT-sources.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../connect/api/build/libs/connect-api-2.3.1-SNAPSHOT.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../connect/api/build/dependant-libs/*:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../connect/transforms/build/libs/connect-transforms-2.3.1-SNAPSHOT.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../connect/transforms/build/dependant-libs/*:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../connect/runtime/build/libs/connect-runtime-2.3.1-SNAPSHOT.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../connect/runtime/build/dependant-libs/*:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../connect/file/build/libs/connect-file-2.3.1-SNAPSHOT.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../connect/file/build/dependant-libs/*:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../connect/json/build/libs/connect-json-2.3.1-SNAPSHOT.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../connect/json/build/dependant-libs/*:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../connect/basic-auth-extension/build/libs/connect-basic-auth-extension-2.3.1-SNAPSHOT.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../connect/basic-auth-extension/build/dependant-libs/*:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../core/build/libs/kafka_2.12-2.3.1-SNAPSHOT-sources.jar:/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../core/build/libs/kafka_2.12-2.3.1-SNAPSHOT.jar kafka.Kafka config/server.properties

 With the same setting above,  we are able to run the Kafka directly from IntelliJ IDEA

Check the Server is running or not:


bin/kafka-topics.sh --list --bootstrap-server localhost:9092
Congratulation, now we have the run Kafka from scratch. 
Get the time to enjoy the adventure of Kafka.

 Compare the source code between Spring and Kafka, the Kafka's code is more modern way by using stream interfaces and other features introduced by JDK8. And it will be good start entry point for java developer to learn how to write good code.

 The another source code I will suggest is Netty which provide the deep inside about how to use JAVA NIO and good practice pattern to code at parrales.

Comments

Post a Comment

Popular posts from this blog

How to fix "ValueError when trying to compile python module with VC Express"

When I tried to compile the python, I always get compile issue as following: ------------ ... File "C:\Python26\lib\ distutils\msvc9compiler.py ", line 358, in initialize vc_env = query_vcvarsall(VERSION, plat_spec) File "C:\Python26\lib\ distutils\msvc9compiler.py ", line 274, in query_vcvarsall raise ValueError(str(list(result.keys()))) ValueError: [u'path'] --------------------- Python community discussed a lot but no solution: http://bugs.python.org/issue7511 The root cause is because the latest visual studio change the *.bat file a lot especially on 64bit env. The python 2.7 didn't update the path accordingly. Based on the assumption above, the following solution worked for me. To install Visual Studio 2008 Express Edition with all required components: 1. Install Microsoft Visual Studio 2008 Express Edition. The main Visual Studio 2008 Express installer is available from (the C++ installer name is vcsetup.exe): https://ww
21 comments
Read more

How to convert the ResultSet to Stream

Java 8 provided the Stream family and easy operation of it. The way of pipeline usage made the code clear and smart. However, ResultSet is still go with very legacy way to process. Per actual ResultSet usage, it is really helpful if converted as Stream. Here is the simple usage of above: StreamUtils.uncheckedConsumer is required to convert the the SQLException to runtimeException to make the Lamda clear.
Post a Comment
Read more

How to run odoo(openerp8) in IDE from source on windows

1. install python 2.7 (per openerp8's official doc, python 27 is required.) 2. download get-pip.py from https://bootstrap.pypa.io/get-pip.py , execute the command: python get-pip.py 3. get source of openerp8 from https://github.com/odoo/odoo.git 4. execute the command: pip install -r D:\source_code\odoo\openerp8/requirements.txt . (requirements.txt contains all dependencies. ) The pip will install the python module automatically. However, the real world always bring us the issues because our C++ compile environment is not setup correctly.  we will get the link error when pip try to install psycopg2 (driver to access postgresql db.). Go to  http://www.stickpeople.com/projects/python/win-psycopg/  and choose the compiled binary file directly. For Python-ldap, go to  http://www.lfd.uci.edu/~gohlke/pythonlibs/ 5. Finally, go to http://sourceforge.net/projects/pywin32/files/pywin32 and choose correct version for python-win32service. 6. If you are family with eclipse a lot,
6 comments
Read more