docker-compose: incomplete ThingsBoard PE upgrade

[jernejcvek]

Problem:
When updating ThingsBoard PE cluster setup with Docker compose to a newer version using the procedure described in the documentation, it seems that only the tb-core microservice updates to a newer version, leaving some of the other microservices stuck at the previous version. This may cause errors due to incompatible versions, for example, problems with deserialization of messages used for communication between microservices.

Cause:
Current procedure:

$ ./docker-stop-services.sh
$ ./docker-upgrade-tb.sh --fromVersion=[FROM_VERSION]
$ ./docker-start-services.sh

• docker-stop-services.sh script stops all containers.
• docker-upgrade-tb.sh script pulls new version only for tb-core1 microservice, which is used to upgrade ThingsBoard.
• it seems docker-start-services.sh script just starts all stopped containers - and not recreates them - which causes that containers are using the old version, except for tb-core1 microservice's container which Docker image was explicitly updated in the previous step.

Proposed Solution:
To make sure all containers are properly updated to newer version, the procedure may be changed to:

• ./docker-stop-services.sh
• update environment variable TB_VERSION stored in .env to the newer version
• ./docker-upgrade-tb.sh --fromVersion=[FROM_VERSION]
• ./docker-update-service.sh
• reinstall license after rebooting; (1) delete license for affected TB instance at Thingsboard license portal, (2) remove TB license data file from tb-core Docker container(s) (location of this file is defined in the TB_LICENSE_INSTANCE_DATA_FILE environment variable stored in tb-node.env).

Page to Update:
https://thingsboard.io/docs/user-guide/install/pe/cluster/docker-compose-setup/#upgrading

[DenysDi]

Hi @jernejcvek

Thanks for your issue.
I can advise you next simple workaround:

1. $ ./docker-stop-services.sh
2. $ ./docker-remove-services.sh
3. edit .env file set "TB_VERSION" to target version (f.e. currently you on 3.2.0 so in this case you need to set 3.2.1)
4. $ ./docker-upgrade-tb.sh --fromVersion=[FROM_VERSION]
5. $ ./docker-update-service.sh [SERVICE...]
   Where:
   [SERVICE...] - list of services to update (defined in docker-compose configurations). If not specified all services will be updated.

Best regards,
Denys

[jernejcvek]

Hello @DenysDi ,

Thank you for your answer.

The workaround you described is similar to my proposed solution (please see above). However, I was afraid that usage of ./docker-remove-services.sh may have unwanted side effects since it contains docker-compose down -v command, which also removes named volumes. But there are probably no named volumes used in the ThingsBoard PE docker-compose setup, am I right? Therefore, it's safe to use ./docker-remove-services.sh.

Btw, I saw that documentation has already been updated. Thanks 👍 Maybe it's also good to mention that reactivation of TB license is needed after ThingsBoard reboot; otherwise, users may encounter 503 Service Unavailable errors due to ThingsBoard licence check failure.

With best regards,
Jernej

[DenysDi]

Hi @jernejcvek ,

docker-compose down -v - disconnect all the volumes from the docker but doesn't delete it from host so you can be sure that all your volumes would be safe.

Btw, I saw that documentation has already been updated. Thanks +1 Maybe it's also good to mention that reactivation of TB license is needed after ThingsBoard reboot; otherwise, users may encounter 503 Service Unavailable errors due to ThingsBoard licence check failure. - great idea! I'll push it on site as soon as possible.

Best regards,
Denys

[DenysDi]

Hi @jernejcvek

Do you have any additional questions or I can close this issue?

Best regards,
Denys

[jernejcvek]

Hello @DenysDi,

I think this is all I have to say related to this issue. I suggest that you close the issue after a note about needed license reactivation on ThingsBoard reboot is added to the documentation, so it won't be forgotten. ☺️ Thank you!

Best regards,

Jernej