Kafka configuration issues

[SharifMrCreed]

Hey there,

Thank you for the good \work on this repo. Unfortunately, I am unable to get the entire system to work. this is what the log looked like

zookeeper                                           | 2023-05-28 07:12:47,253 [myid:1] - INFO  [main:o.e.j.s.s.DefaultSessionIdManager@334] - DefaultSessionIdManager workerName=node0
zookeeper                                           | 2023-05-28 07:12:47,253 [myid:1] - INFO  [main:o.e.j.s.s.DefaultSessionIdManager@339] - No SessionScavenger set, using defaults
zookeeper                                           | 2023-05-28 07:12:47,261 [myid:1] - INFO  [main:o.e.j.s.s.HouseKeeper@132] - node0 Scavenging every 660000ms
zookeeper                                           | 2023-05-28 07:12:47,270 [myid:1] - WARN  [main:o.e.j.s.ConstraintSecurityHandler@759] - [email protected]@1ab06251{/,null,STARTING} has uncovered http methods for path: /*
kafka                                               | kafka 07:12:47.31 WARN  ==> KAFKA_CFG_LISTENERS must include a listener for CONTROLLER
zookeeper                                           | 2023-05-28 07:12:47,315 [myid:1] - INFO  [main:o.e.j.s.h.ContextHandler@921] - Started o.e.j.s.ServletContextHandler@1ab06251{/,null,AVAILABLE}
kafka                                               | kafka 07:12:47.32 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
kafka                                               | kafka 07:12:47.34 INFO  ==> Initializing Kafka...
zookeeper                                           | 2023-05-28 07:12:47,353 [myid:1] - INFO  [main:o.e.j.s.AbstractConnector@333] - Started ServerConnector@7cb502c{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
zookeeper                                           | 2023-05-28 07:12:47,354 [myid:1] - INFO  [main:o.e.j.s.Server@415] - Started @2329ms
zookeeper                                           | 2023-05-28 07:12:47,355 [myid:1] - INFO  [main:o.a.z.s.a.JettyAdminServer@196] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands
kafka                                               | kafka 07:12:47.36 INFO  ==> No injected configuration files found, creating default config files
zookeeper                                           | 2023-05-28 07:12:47,369 [myid:1] - INFO  [main:o.a.z.s.ServerCnxnFactory@169] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
zookeeper                                           | 2023-05-28 07:12:47,375 [myid:1] - WARN  [main:o.a.z.s.ServerCnxnFactory@309] - maxCnxns is not configured, using default value 0.
zookeeper                                           | 2023-05-28 07:12:47,380 [myid:1] - INFO  [main:o.a.z.s.NIOServerCnxnFactory@652] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
zookeeper                                           | 2023-05-28 07:12:47,383 [myid:1] - INFO  [main:o.a.z.s.NIOServerCnxnFactory@660] - binding to port 0.0.0.0/0.0.0.0:2181
zookeeper                                           | 2023-05-28 07:12:47,415 [myid:1] - INFO  [main:o.a.z.s.w.WatchManagerFactory@42] - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
zookeeper                                           | 2023-05-28 07:12:47,416 [myid:1] - INFO  [main:o.a.z.s.w.WatchManagerFactory@42] - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
zookeeper                                           | 2023-05-28 07:12:47,422 [myid:1] - INFO  [main:o.a.z.s.ZKDatabase@132] - zookeeper.snapshotSizeFactor = 0.33
zookeeper                                           | 2023-05-28 07:12:47,422 [myid:1] - INFO  [main:o.a.z.s.ZKDatabase@152] - zookeeper.commitLogCount=500
zookeeper                                           | 2023-05-28 07:12:47,431 [myid:1] - INFO  [main:o.a.z.s.p.SnapStream@61] - zookeeper.snapshot.compression.method = CHECKED
zookeeper                                           | 2023-05-28 07:12:47,433 [myid:1] - INFO  [main:o.a.z.s.p.FileSnap@85] - Reading snapshot /bitnami/zookeeper/data/version-2/snapshot.0
zookeeper                                           | 2023-05-28 07:12:47,440 [myid:1] - INFO  [main:o.a.z.s.DataTree@1712] - The digest value is empty in snapshot
zookeeper                                           | 2023-05-28 07:12:47,446 [myid:1] - INFO  [main:o.a.z.s.ZKDatabase@289] - Snapshot loaded in 24 ms, highest zxid is 0x0, digest is 1371985504
zookeeper                                           | 2023-05-28 07:12:47,449 [myid:1] - INFO  [main:o.a.z.s.p.FileTxnSnapLog@479] - Snapshotting: 0x0 to /bitnami/zookeeper/data/version-2/snapshot.0
zookeeper                                           | 2023-05-28 07:12:47,452 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@558] - Snapshot taken in 4 ms
zookeeper                                           | 2023-05-28 07:12:47,516 [myid:1] - INFO  [main:o.a.z.s.RequestThrottler@75] - zookeeper.request_throttler.shutdownTimeout = 10000 ms
zookeeper                                           | 2023-05-28 07:12:47,519 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::o.a.z.s.PrepRequestProcessor@138] - PrepRequestProcessor (sid:0) started, reconfigEnabled=false
zookeeper                                           | 2023-05-28 07:12:47,563 [myid:1] - INFO  [main:o.a.z.s.ContainerManager@84] - Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0
zookeeper                                           | 2023-05-28 07:12:47,566 [myid:1] - INFO  [main:o.a.z.a.ZKAuditProvider@42] - ZooKeeper audit is disabled.
kafka                                               | kafka 07:12:47.64 INFO  ==> Initializing KRaft...
kafka                                               | kafka 07:12:47.64 WARN  ==> KAFKA_KRAFT_CLUSTER_ID not set - If using multiple nodes then you must use the same Cluster ID for each one
': No such file or directory                        | env: can't execute 'node
central-ledger exited with code 127
kafka                                               | kafka 07:12:50.92 INFO  ==> Generated Kafka cluster ID '7zFapi__QEG7s87bjmR3NA'
kafka                                               | kafka 07:12:50.93 INFO  ==> Formatting storage directories to add metadata...
': No such file or directory                        | env: can't execute 'node
': No such file or directory                        | env: can't execute 'node
': No such file or directory                        | env: can't execute 'node
dependency failed to start: container account-lookup-service exited (127)

Looking deeply into it I have found that there is an issue with the Kafka configuration. something to do with ZooKeeper, its known issue with the docker.io/bitnami/kafka:3.2 container that you are using: bitnami/containers#31394

I hope the information I have provided could help in trying to solve this

Thank you in advance

[Khane1]

I'm facing the same problem and since the central ledger container depends on kafka the other containers are all failing to run too.

Profile Name	Description	Dependent Profiles
debug	Debug utilities (kowl)	kafka
central-ledger	Central Ledger service	kafka
ml-api-adapter	ML API Adapter service	central-ledger
quoting-service	Quoting service	central-ledger
account-lookup-service	Account lookup service	central-ledger

Is there a work around this?

[vijayg10]

@SharifMrCreed @Khane1 Thanks for reporting the issue with kafka. I think this is due to bitnami released a new sub version in 3.2. Since we referred the version in docker-compose file as 3.2, it is broken. I fixed the issue and created a PR. Please test the branch from following PR and confirm. Then we can merge it to master.
#6

[vijayg10]

@SharifMrCreed @Khane1 Some more fixes are pushed to the PR #6
Please test it on your side.

[mdebarros]

@SharifMrCreed, we have created a new release which includes fixes for your issue. --> https://github.com/mojaloop/ml-core-test-harness/releases/tag/v1.0.0

Please could you verify if the issue is resolved now.