Kafka Avro Python Gradle CI Template

each program has to then have a setup.py

A template of setup.py can be generated by generateSetupPy for example:

./gradlew :python-kafka-mirror:generateSetupPy.

After the initial setup.py is generated the setup() function needs to be modified with the correct entry point to your program.

setup.py then can be comitted to the repository and afterwards doesn't have to be regenerated again and it has already been generated for the provided example.

building python modules

./gradlew :python-kafka-mirror:build

running python moudles using generated venv

The application requires some environmental variables so they need to be exported first

export KAFKA_BOOTSTRAP_SERVERS="localhost:9092"
export SCHEMA_REGISTRY_URL="http://localhost:8082"
./python-kafka-mirror/build/venv/bin/example_mirror

Note on local pypi repository

The build phase includes a local repoistory generator which uses pygradle pivy converter. This script can be found in the root of the project.

IDE Setup

• create a Python SDK using system interpreter (this works only with gradle 4.0 due to that api used by pygradle plugin)
• under the interpreter's packages add: requests, confluent-kafka and avro-python3
• finally mark src as Sources and test as Tests under module structure > Sources
• use main.py to execute the avro_consumer from the IDE
  • adding KAFKA_BOOTSTRAP_SERVERS and SCHEMA_REGISTRY_URL environment variables
  • (these variables are automatically set both in docker and python environment when running gradle test)