Skip to content
Ryan CrawCour edited this page Apr 23, 2020 · 19 revisions

Welcome to the kafka-connect-cosmosdb wiki!

Cosmos DB

Sample code

End-to-end sample (not finished yet but here is the repo, I'll tag a v3 and v4 version): https://github.com/Azure-Samples/azure-cosmos-java-sql-api-end-to-end-sample

Migration guide: https://github.com/Azure-Samples/azure-cosmos-java-sql-api-samples/blob/master/migration-guide.md

Reactor vs RxJava: https://github.com/Azure-Samples/azure-cosmos-java-sql-api-samples/blob/master/reactor-rxjava-guide.md

Reactor Pattern Guide: https://github.com/Azure-Samples/azure-cosmos-java-sql-api-samples/blob/master/reactor-pattern-guide.md

Basic samples: https://github.com/Azure-Samples/azure-cosmos-java-sql-api-samples

Kafka

VS Code dev containers

Watch for updates in .devcontainer for the required Docker files.

Setting up Kafka and Kafka Connect dev environments

use the cp-all-in-one-community docker compose file. This Docker Compose file launches only the community services in Confluent Platform, and runs them in containers in your local host.

You should end up with the following environment :

  • zookeeper : 2181
  • broker: 29092 | 9092 *see note on connectivity below
  • schema-registry : 8081
  • connect: 8083
  • ksqldb-server: 8088
  • rest-proxy: 8082

if you prefer a "more realistic" environment where zookeeper and kafka servers are distinct from each other then the docker compose files on kafka-stack-docker-compose are worth looking in to.

single zookeeper & single kafka should be sufficient.

NB: accessing Kafka from clients on other machines:

The config used here exposes port 9092 for external connections to the broker i.e. those from outside the docker network. This could be from the host machine running docker, or maybe further afield if you've got a more complicated setup. If the latter is true, you will need to change the value 'localhost' in KAFKA_ADVERTISED_LISTENERS to one that is resolvable to the docker host from those remote clients

For connections internal to the docker network, such as from other services and components, use kafka:29092.

See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details