Investigate this failed move-contacts job

[kennsippell]

https://users-chis-ke.app.medicmobile.org/board/queue/MOVE_CONTACT_QUEUE?status=failed

Error: Job fe9d3d11-21ad-44b6-9682-b21247233ecd failed with the following error: Error: Move contact command exited with code 1. Last output: �[31mERROR undefined �[0m

    at /app/dist/worker/move-contact-worker.js:150:15
    at Generator.next (<anonymous>)
    at fulfilled (/app/dist/worker/move-contact-worker.js:5:58)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[paulpascal]

When double checking the worker logs using:

kubectl --context arn:aws:eks:eu-west-2:720541322708:cluster/prod-cht-eks  \
    --namespace users-chis-prod logs deploy/users-chis-ke-cht-user-management-worker \
    --since 8h

Its shows, that its cht-conf error:

cht-conf: �[32mINFO Staged changes to lineage information for X contact(s) and Y report(s). �[0m
�[32mINFO move-contacts complete. �[0m

cht-conf: �[32mINFO Starting action: upload-docs… �[0m

cht-conf: �[33mWARN This operation will permanently write Z docs.  Are you sure you want to continue? �[0m

cht-conf: 
0/Z docs []   0% ?:??
cht-conf: 
100/Z docs []  26% 0:07
cht-conf: �[32mINFO {
  "code": 500,
  "error": "Server error",
  "status": 500,
  "name": "Server error",
  "stack": "Error\n    at Object.generateErrorFromResponse (/app/node_modules/pouchdb-errors/lib/index.js:104:18)\n    at /app/node_modules/pouchdb-adapter-http/lib/index.js:254:33\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
} �[0m

cht-conf: �[31mERROR undefined �[0m

And its seems that an error happen when cht-conf was running from the 100/ Z (total docs) to be written.

I will investigate more on that and report here.

So far the main causes found are:

• Token Expiration (after many delayed)
• Timeout Handling (execution take more time than expected)
• Undefined errors in cht execution (cht exection exited)