Kafka_producer_consumer

Kafka Producer and consumer code to read csv file

Command to start ZooKeeper Server

bin\windows\zookeeper-server-start.bat config\zookeeper.properties

Command to start Kafka Server

bin\windows\kafka-server-start.bat config\server.properties

Command to list the Kafka-topics

bin\windows\kafka-topics.bat --list --bootstrap-server localhost:9092

Problem 1:

• Kafka producer that will read “SalesRecords.csv” file and convert each line to json and send the data to a Kafka topic(“test.topic.raw”)
• Kafka Consumer to check wheather the data was written to the topic or not.

Problem 2:

Kafka consumer that will read data from the topic named “test.topic.raw”. It should read the data from the beginning and do the below filtering of data:

• Select only those data whose “Total Cost” value is greater than 304017.56
• After selection, it should send the filtered data to another topic named “test.topic.threshold_costs”

Problem 3:

Write a Kafka consumer that will read data from multiple topics:

• The name of the topics is "test.topic.raw" and "test.topic.threshold_costs"
• Filter the data based on the country and send data to different topic depending on country value “test.topic.{country}"

Problem 4:

Write down 2 Kafka consumers, both reading data from the same topic “test.topic.raw”. Both the consumers should receive the same messages and do as described ahead.

Consumer1 should do the following:

• Get the “Sale Channel” from json data.
• Send data to a Kafka topic depending on the channel of the sale.
• List the topics, you should see all the topics created for each Sale Channel.

Consumer2 should do the following:

• Filter the data based on Unit Cost
• Send data to Kafka topic named “topic.costly_order” if ‘Unit Cost’ is more than “500”
• Start consumer console to verify data reaching to the given topic.