TTK errors in Helm v15.0.0

[tdaly61]

When I clone Mojaloop v15.0.0 and deploy the vanilla charts and then run helm test I see 3 TTK failures.
this is using k3s version 1.24

Also at the time of filing this ticket the Mojaloop slack announcements channel is reporting TTK failures in the CI/CD pipleine for v15.0.0 yet a week or so ago this was running clean. Miguel suggests that the CI/CD pipeline deployment has exhausted some resource and this is likely as there are > 100 failures.

Along with the helm test error and the TTK test failures themselves what is not clear is why the current vanilla deployment fails yet a week or so ago the GP tests ran clean. I have searched for something I have inadvertently changed from default but running diff on helm/mojaloop/values.yaml from what I deploy and a fresh clone of v15.0.0 shows I made no changes to the default. It makes me wonder of there might be a difference between the v15.0/0 repo and the bundled helm charts as this could explain the difference (I will check this and update this ticket when I have).

To reproduce manually :
git clone --branch v15.0.0 https://github.com/mojaloop/helm.git
cd helm ; ./package.sh
helm install the example backend chart
helm install the mojaloop chart e.g. helm install ml --wait --timeout 2400s /home/ubuntu/helm/mojaloop
run the ttk tests : helm test ml --logs

(alternatively just use mini-loop v5.0 branch)

results :

ubuntu@miniloop-vm ~/mini-loop (v5.0) $ helm test ml --logs
NAME: ml
LAST DEPLOYED: Sat Apr 22 10:36:32 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: ml-ml-ttk-test-setup
Last Started: Sat Apr 22 11:14:41 2023
Last Completed: Sat Apr 22 11:17:12 2023
Phase: Succeeded
TEST SUITE: ml-ml-ttk-test-val-gp
Last Started: Sat Apr 22 11:17:12 2023
Last Completed: Sat Apr 22 11:21:56 2023
Phase: Failed
NOTES:
Thank you for installing mojaloop.

Your release is named: ml.

Mojaloop Official Website: https://mojaloop.io

To learn more about Mojaloop: https://docs.mojaloop.io
To learn more about Mojaloop deployments: https://docs.mojaloop.io/legacy/deployment-guide
To learn more about this release: https://github.com/mojaloop/helm/releases/tag/v15.0.0

This Helm chart lives @ http://github.com/mojaloop/helm

To learn more about this deployment, try:

$ helm status ml

Use the following command to execute Test cases:

$ helm -n default test ml

Use the following command to execute Test cases and print logs to console:

$ helm -n default test ml --logs

Use the following command to execute a specific Test case:

$ helm -n default test ml --filter "name=ml-<TEST_NAME_1>,name=ml-<TEST_NAME_2>"

Use the following command to ignore a specific Test case:

$ helm -n default test ml --filter "!name=ml-<TEST_NAME_1>"

View Test logs with the following commands:
$ kubectl -n default logs pod/ml-ml-ttk-test-setup
$ kubectl -n default logs pod/ml-ml-ttk-test-val-gp
$ kubectl -n default logs pod/ml-ml-ttk-test-cleanup

Copyright © 2020 Mojaloop Foundation

Error: unable to get pod logs for ml-ml-ttk-test-cleanup: pods "ml-ml-ttk-test-cleanup" not found

The report is attached.

report.zip

[tdaly61]

as further investigation and verification of the issues I ran the following and still got 3 TTK test failures
i.e. I ran installing completely from the repo.

cat ./deploy-from-repo.sh
#!/usr/bin/env bash

script to test vanilla deploy of Mojaloop from package repo

assumes correct version of kubernetes , helm, ingress etc already installed and configured.

April 2023

see install instructions in readme.md at https://github.com/mojaloop/helm and https://github.com/mojaloop/helm/blob/master/thirdparty/README.md

helm repo add stable https://charts.helm.sh/stable
helm repo add incubator https://charts.helm.sh/incubator
helm repo add kiwigrid https://kiwigrid.github.io
helm repo add kokuwa https://kokuwaio.github.io/helm-charts
helm repo add elastic https://helm.elastic.co
helm repo add codecentric https://codecentric.github.io/helm-charts
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add mojaloop-charts https://mojaloop.github.io/charts/repo
helm repo add redpanda-console https://packages.vectorized.io/public/console/helm/charts/
helm repo add mojaloop http://mojaloop.io/helm/repo/

helm repo update
helm repo list
helm delete ml
helm delete be

helm install be --wait --timeout 300s mojaloop/example-mojaloop-backend
helm install ml --wait --timeout 2400s mojaloop/mojaloop
--set account-lookup-service.account-lookup-service.config.featureEnableExtendedPartyIdType=true
--set account-lookup-service.account-lookup-service-admin.config.featureEnableExtendedPartyIdType=true
--set thirdparty.enabled=true
--set ml-ttk-test-setup-tp.tests.enabled=true
--set ml-ttk-test-val-tp.tests.enabled=true

ubuntu@miniloop-vm:~/mini-loop/util/test$ ./deploy-from-repo.sh
"stable" already exists with the same configuration, skipping
"incubator" already exists with the same configuration, skipping
"kiwigrid" already exists with the same configuration, skipping
"kokuwa" already exists with the same configuration, skipping
"elastic" already exists with the same configuration, skipping
"codecentric" already exists with the same configuration, skipping
"bitnami" already exists with the same configuration, skipping
"mojaloop-charts" already exists with the same configuration, skipping
"redpanda-console" already exists with the same configuration, skipping
"mojaloop" already exists with the same configuration, skipping
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kokuwa" chart repository
...Successfully got an update from the "codecentric" chart repository
...Successfully got an update from the "elastic" chart repository
...Successfully got an update from the "kiwigrid" chart repository
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "redpanda-console" chart repository
...Successfully got an update from the "mojaloop-charts" chart repository
...Successfully got an update from the "mojaloop" chart repository
...Successfully got an update from the "bitnami" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
NAME URL
kiwigrid https://kiwigrid.github.io
kokuwa https://kokuwaio.github.io/helm-charts
elastic https://helm.elastic.co
codecentric https://codecentric.github.io/helm-charts
bitnami https://charts.bitnami.com/bitnami
mojaloop http://mojaloop.io/helm/repo/
stable https://charts.helm.sh/stable
incubator https://charts.helm.sh/incubator
mojaloop-charts https://mojaloop.github.io/charts/repo
redpanda-console https://packages.vectorized.io/public/console/helm/charts/
Error: uninstall: Release not loaded: ml: release: not found
Error: uninstall: Release not loaded: be: release: not found
NAME: be
LAST DEPLOYED: Mon May 22 07:03:23 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
coalesce.go:220: warning: cannot overwrite table with non table for mojaloop.thirdparty.consent-oracle.env (map[])
coalesce.go:220: warning: cannot overwrite table with non table for mojaloop.thirdparty.consent-oracle.env (map[])
coalesce.go:220: warning: cannot overwrite table with non table for mojaloop.thirdparty.consent-oracle.env (map[])
NAME: ml
LAST DEPLOYED: Mon May 22 07:05:13 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing mojaloop.

Your release is named: ml.

Mojaloop Official Website: https://mojaloop.io

To learn more about Mojaloop: https://docs.mojaloop.io
To learn more about Mojaloop deployments: https://docs.mojaloop.io/legacy/deployment-guide
To learn more about this release: https://github.com/mojaloop/helm/releases/tag/v15.0.0

This Helm chart lives @ http://github.com/mojaloop/helm

To learn more about this deployment, try:

$ helm status ml

Use the following command to execute Test cases:

$ helm -n default test ml

Use the following command to execute Test cases and print logs to console:

$ helm -n default test ml --logs

Use the following command to execute a specific Test case:

$ helm -n default test ml --filter "name=ml-<TEST_NAME_1>,name=ml-<TEST_NAME_2>"

Use the following command to ignore a specific Test case:

$ helm -n default test ml --filter "!name=ml-<TEST_NAME_1>"

View Test logs with the following commands:
$ kubectl -n default logs pod/ml-ml-ttk-test-setup
$ kubectl -n default logs pod/ml-ml-ttk-test-val-gp
$ kubectl -n default logs pod/ml-ml-ttk-test-setup-tp
$ kubectl -n default logs pod/ml-ml-ttk-test-val-tp
$ kubectl -n default logs pod/ml-ml-ttk-test-cleanup

Copyright © 2020 Mojaloop Foundation
NAME: ml
LAST DEPLOYED: Mon May 22 07:05:13 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: ml-ml-ttk-test-setup
Last Started: Mon May 22 07:15:51 2023
Last Completed: Mon May 22 07:18:33 2023
Phase: Succeeded
TEST SUITE: ml-ml-ttk-test-val-gp
Last Started: Mon May 22 07:18:33 2023
Last Completed: Mon May 22 07:23:16 2023
Phase: Failed
NOTES:
Thank you for installing mojaloop.

Your release is named: ml.

Mojaloop Official Website: https://mojaloop.io

To learn more about Mojaloop: https://docs.mojaloop.io
To learn more about Mojaloop deployments: https://docs.mojaloop.io/legacy/deployment-guide
To learn more about this release: https://github.com/mojaloop/helm/releases/tag/v15.0.0

This Helm chart lives @ http://github.com/mojaloop/helm

To learn more about this deployment, try:

$ helm status ml

Use the following command to execute Test cases:

$ helm -n default test ml

Use the following command to execute Test cases and print logs to console:

$ helm -n default test ml --logs

Use the following command to execute a specific Test case:

$ helm -n default test ml --filter "name=ml-<TEST_NAME_1>,name=ml-<TEST_NAME_2>"

Use the following command to ignore a specific Test case:

$ helm -n default test ml --filter "!name=ml-<TEST_NAME_1>"

View Test logs with the following commands:
$ kubectl -n default logs pod/ml-ml-ttk-test-setup
$ kubectl -n default logs pod/ml-ml-ttk-test-val-gp
$ kubectl -n default logs pod/ml-ml-ttk-test-setup-tp
$ kubectl -n default logs pod/ml-ml-ttk-test-val-tp
$ kubectl -n default logs pod/ml-ml-ttk-test-cleanup

Copyright © 2020 Mojaloop Foundation

Error: unable to get pod logs for ml-ml-ttk-test-cleanup: pods "ml-ml-ttk-test-cleanup" not found
ubuntu@miniloop-vm:/mini-loop/util/test$ ps -eaf | grep -i mongosg
ubuntu 2449327 1722982 0 07:38 pts/0 00:00:00 grep --color=auto -i mongosg
ubuntu@miniloop-vm:/mini-loop/util/test$ ps -eaf | grep -i mongos
1001 2320131 2319367 0 07:09 ? 00:00:02 [mongosh mongodb]
1001 2321877 2319367 0 07:09 ? 00:00:02 [mongosh mongodb]
1001 2324111 2321458 0 07:09 ? 00:00:02 [mongosh mongodb]
1001 2324145 2319367 0 07:09 ? 00:00:02 [mongosh mongodb]
1001 2325221 2321458 0 07:10 ? 00:00:02 [mongosh mongodb]
1001 2325225 2319367 0 07:10 ? 00:00:02 [mongosh mongodb]
1001 2325338 2321458 0 07:10 ? 00:00:02 [mongosh mongodb]
1001 2325349 2319367 0 07:10 ? 00:00:02 [mongosh mongodb]
1001 2326418 2321458 0 07:10 ? 00:00:02 [mongosh mongodb]
1001 2326922 2319367 0 07:10 ? 00:00:02 [mongosh mongodb]
1001 2327167 2319367 0 07:10 ? 00:00:02 [mongosh mongodb]
1001 2327182 2321458 0 07:10 ? 00:00:02 [mongosh mongodb]
1001 2328223 2321458 0 07:10 ? 00:00:02 [mongosh mongodb]
1001 2328344 2319367 0 07:10 ? 00:00:02 [mongosh mongodb]
1001 2329060 2319367 0 07:10 ? 00:00:02 [mongosh mongodb]
1001 2329100 2321458 0 07:10 ? 00:00:02 [mongosh mongodb]
1001 2334210 2319367 0 07:11 ? 00:00:02 [mongosh mongodb]
1001 2334285 2321458 0 07:11 ? 00:00:02 [mongosh mongodb]
1001 2335176 2321458 0 07:11 ? 00:00:02 [mongosh mongodb]
1001 2335343 2319367 0 07:11 ? 00:00:02 [mongosh mongodb]
1001 2336174 2321458 0 07:12 ? 00:00:02 [mongosh mongodb]
1001 2336192 2319367 0 07:12 ? 00:00:02 [mongosh mongodb]
1001 2336287 2321458 0 07:12 ? 00:00:02 [mongosh mongodb]
1001 2336310 2319367 0 07:12 ? 00:00:02 [mongosh mongodb]
1001 2337259 2321458 0 07:12 ? 00:00:02 [mongosh mongodb]
1001 2337413 2319367 0 07:12 ? 00:00:02 [mongosh mongodb]
1001 2338209 2321458 0 07:12 ? 00:00:02 [mongosh mongodb]
1001 2338244 2319367 0 07:12 ? 00:00:02 [mongosh mongodb]
1001 2350516 2321458 0 07:15 ? 00:00:02 [mongosh mongodb]
1001 2350616 2319367 0 07:15 ? 00:00:02 [mongosh mongodb]
ubuntu 2449543 1722982 0 07:38 pts/0 00:00:00 grep --color=auto -i mongos