SpringFlex

start zookeeper: bin/zookeeper-server-start.sh config/zookeeper.properties start kafka server: bin/kafka-server-start.sh config/server.properties run the demo: a. For unlimited sync-producer-consumer run, `run bin/java-producer-consumer-demo.sh sync` b. For unlimited async-producer-consumer run, `run bin/java-producer-consumer-demo.sh`adf Let's take a look the source code java-producer-consumer-demo.sh,. the script is mainly use  kafka.examples.KafkaConsumerProducerDemo  as the demo class. the code created 2 topics(queues) for produce and consumer. the time to dig into code step by step. Firstly, take a look at source code of Producer Producer created messages and send to kafka server. on line 53, KafkaProducer send the message with callback :  ProducerRecord Per each message, the topic, messgeNo and messager body are defined. DemoCallBack provide asynchronous handling of request completion. here, DemoCallBack just print the m...

Get source code of Kafka, the version is 2.3  git clone -b 2.3 https://github.com/apache/kafka.git Generate Jar File ./gradlew jar Start the zookeeper bin/zookeeper-server-start.sh config/zookeeper.properties Since we are mainly focus on Kafka source code, we won't spend some time to dig into how the zookeeper start by using jars directly. Start the Kafka Server Option 1: start with script bin /kafka-server-start .sh config /server .properties Option 2: If we inspect the real command from kafka-run-class.sh which is called by kafka-start-server.sh, here is the final command to run: exec /Library/Java/JavaVirtualMachines/jdk-11.0.1.jdk/Contents/Home/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Xlog:gc*:file=/Users/leizheng/vcs/github/kafka/kafka-2.3/bin/../logs/kafkaServer-gc.log:time,tags:filecount=10,files...

Java 8 provided the Stream family and easy operation of it. The way of pipeline usage made the code clear and smart. However, ResultSet is still go with very legacy way to process. Per actual ResultSet usage, it is really helpful if converted as Stream. Here is the simple usage of above: StreamUtils.uncheckedConsumer is required to convert the the SQLException to runtimeException to make the Lamda clear.

figure 1 Let's do more adventure following yesterday's work. Highlight the rink to show the active play area as following: figure 2                                                                             The red highlight area is exactly the active play area from figure 2. In order to get it, firstly resize the rink image to match with play image. so we can use for pixel to pixel mapping. create black/white mask from resized rink image: figure 3 truncate the play image suing figure 3 and then create active red area figure 4 Finally weightly add figure 4 to the normalized to generate figure 1. Here is source code:

As the dad of hockey player, what can we do if we don't know how to skate?! yes, we can write the code. As the name of loving, do whatever we can do for kids. The final object is to create a video analysis system automatically which will use deep learning to analysis the video, tracing all player's position/movement and collect the athlete data. However,  a thousand miles begins with a single step.  The first step of today is to resolve the fundamental challenge - map the rink from video image to organized rink image. The image transformer is very straightforward as the first version. OpenCV api provided 2 different ways to resolve it: 1.  getPerspectiveTransform : support only 4 points mapping 2. findHomography: support at least 4 points mapping. I wrote a quick code to help mark the key points from original image: and we generated the image with key point marked the python code could be simple as well Make sure each of point is mapped to orga...

In the last post, we go through the whole process about how to register a regular <bean>.  Now, let's take a look at how to load <import> tag: < import resource = "services.xml" /> < import resource = "resources/messageSource.xml" /> < import resource = "/resources/themeSource.xml" /> </import> see the entry code here: importBeanDefinitionResource has all the implementation. the logic is as following: 1. get the value of attribute: resource 2. resolve the location string by replacing system properties: e.g. "${user.dir}" 3. check the location is starting with classpath or regular url. 4. Line 26 will use AbstractBeanDefinitionReader-> loadBeanDefinitions to parse the spring beans as mentioned in my previous blog.  g et the file resource list (if the location path contains "*" or "?") or return the file resource directly. parse the resource to spring be...

In the previous post, we just go through how code step to step and starting load spring beans. For different node type, the related loading code are different. BEAN_ELEMENT: ParseBeanDefinition Example of xml BeanDefinitionParserDelegate.parseBeanDefinitionElement The second parameter containingBean is used to parse a value, ref or collection sub-element of a property or constructor-arg element. Here we just parse the regular bean, just leave the value as null. Firstly, the attribute “id” and “name” are retrieved from <bean/>. The name is used for alias purpose. The value could be “name1,name2;name3” with dilimiters {,;} If there is Id assigned, the first name will used as bean id and the left be used as aliases. Once the id and name are done, parseBeanDefinitionElement A bean Definition is returned and add the bean name to bean definition holder. We will go over all the logic later to see how will generate the bean name automatically when bean name is empty from x...

The most simple usage of Spring is to load xml definitions beans and use it. We will go through the whole work flow based on spring unittest. Go and check XmlBeanDefinitionReaderTests from package org.springframework.beans.factory.xml, The method "withImport" show the simple workflow to load beans. Let’s jump inside loadBeanDefinitions. The overview workflow is compromised by 3 steps as following:   Validate Mode for resource. The first main part is validate mode for resource. As we know, XML had 2 type of validation: DTD VS XSD. The first step is check the file content to detect which mode we will parse the XML file. Load document   Convert the input stream to xml document. Register Bean Definition CreateDelegate: Return BeanDefinitaionParseDelegate class.   The actual implement class which parse the xml content and create Spring bean. Before parse the actual bean, checking the profile at first. The work...

The spring could be most popular java framework in this world. It had been born at 2002 which is even earlier than Adobe Flex (dead already). Spring source code provided a very elegant way about how to organize your code and define the Apis.  As java developer, all of us should spend time to read source code Spring and get deeply know how to write the solid and high quality code. I am trying to summarize what I got when I review the code and provide a quick guide to understand the code easily. Hope I can finish this giant project. First of all, if we fork the source code from  https://github.com/spring-projects/spring-framework  it is simple to figoure the modules dep from gradle file. For easy purpose, I draw the diagram to represent the dependencies.  The black line is for compile dep and yellow one is for optional dep (just ignore it at very first begging to keep everything simple). Add caption Let's go start from spring-core and spring-beans on the nex...

The model build tools like gradle/maven provided a pretty easy way to run unit test and generate test summary. Also, gradle support to run unit test concurrently which improved performance a lot. One of project is converting from ANT build to Gradle. However, one challenge of current project can't use gradle's multiple thread to run tests because some of test cases access the same database with conflict and can't run them at the same time.  The existing ant solution was trying to group all tests as 3 collection to avoid the test conflict. Unfortunately, gradle didn't support it by default because gradle will generate temp file with same name during test. So multiple test processes will lead to file access violation. For simple purpose, I created 3 classes and the related test case as following: and the test command is: It will simulate the real project case and provide an easy way to verify quickly if our solution workable or not, 1. Firstly, in order to f...

Hibernate is used widely for data access. It is great to release developer from tons of sql scripts. However, the performance is about 50% less than pure jdbc access. For core function, my current project is still using the pure jdbc to access. The pain part is the current code is all write down by stringbuilder which is easy to make a mistake in syntax and missing space between key sql words. My purpose is trying to create simple java class which help build sql prepared statement. The class should be working in the following feature as the first version: 1. Support select, Update, delete sql generation. 2.  Code usage should go with the flow feature 3. Support where, join, and, or key words. 4. support parameters for prepared statement. Let's start from simple simple select/delete statement, the unit test will be as following at first: The implementation will be simple as well: After that, we add support of "WHERE" "UPDATE" as well as al...

The JDK 9+ provided incubator to support non-blocking jdbc. The idea is trying to process data when reading or updating the database.  The code support most existing jdbc driver and just need minimal change of incubator code. With the non-blocking jdbc code, the performance increased significantly for updating operations. But just hit slightly change on selection. It make sense since updating will cost on database side and our code will wait longer time to database IO response. The chart clearly show performance comparison between block jdbc and non-block jdbc. Overview, all of them hit the similar query time by access 10000 rows simple data. When using just one thread, the non-block api just took half time of block api to update the database. However, multiple thread increase the performance block api a lot. It should be like it since multiple thread used more system resources to process services. See the table before for detail data: Although multiple thead...

Provide Non-blocking REST API: The benefit of non-blocking API will benefit a lot for big scale concurrency calls. We will add @EnableAsync to ResourceServerConfig @Configuration @EnableAsync @EnableResourceServer /*@EnableResourceServer enables a Spring Security filter that authenticates requests using an incoming OAuth2 token.*/ public class ResourceServerConfig extends ResourceServerConfigurerAdapter { and add new async method to demoController: @GetMapping ( "/async-hello" ) public DeferredResult<ResponseEntity<?>> helloAsync () { DeferredResult<ResponseEntity<?>> output = new DeferredResult<>() ; ForkJoinPool. commonPool ().submit(() -> { try { Thread. sleep ( 6000 ) ; } catch (InterruptedException e) { } output .setResult(ResponseEntity. ok ( HELLOWORLD )) ; }) ; return output ; } add refactor test case to support async call with mvcmock @Test public void c...

Do more with test: on the previous practice, we run the embed tomcat to test rest, and we don't need do that since spring provide mock mvc to simplify our test work. the older code: /** * The test case used general resttemplate to call api and compare the response. * the whole test is running with an actual tomcat server. */ package org.lz.boilerplate.springrest ; import org.junit.Assert ; import org.junit. Test ; import org.junit.runner. RunWith ; import org.springframework.beans.factory.annotation. Autowired ; import org.springframework.boot.test.context. SpringBootTest ; import org.springframework.boot.test.web.client.TestRestTemplate ; import org.springframework.boot.web.server. LocalServerPort ; import org.springframework.http.* ; import org.springframework.security.oauth2.client.OAuth2RestTemplate ; import org.springframework.security.oauth2.client.token.grant.password.ResourceOwnerPasswordResourceDetails ; import org.springframework.test.context.junit4.SpringRunner ...