Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DLPX-84565 telegraf.service restarts after upgrade when it should be masked #739

Merged
merged 1 commit into from
Oct 18, 2023
Merged

DLPX-84565 telegraf.service restarts after upgrade when it should be masked #739

merged 1 commit into from
Oct 18, 2023

Conversation

palash-gandhi
Copy link
Contributor

@palash-gandhi palash-gandhi commented Sep 19, 2023 •
edited
Loading

Problem

The telegraf package enables telegraf.service. The service ends up in a restart loop as noted in the Jira.
While the service is masked and disabled on a fresh install, there is nothing in the upgrade logic to mask this service.

Solution

Mask it like other services.

Testing Done

ab-pre-push: http://selfservice.jenkins.delphix.com/job/appliance-build-orchestrator-pre-push/7191/console

Manual test: Upgrade from 6.0.13.0 to 7.0 to 16.0 and verify telegraf.service is not running
http://selfservice.jenkins.delphix.com/job/blackbox-chained/5396/console

From the syslog:
During verify:

Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: //var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-
build-develop-pre-push-1584/common.sh:fix_and_migrate_services:448 systemctl is-enabled telegraf
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-b
uild-develop-pre-push-1584/common.sh:fix_and_migrate_services:448 [[ enabled == enabled ]]
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-develop-pre-push-1584/common.sh:fix_and_migrate_services:449 mask_service telegraf delphix.iyDDbhM
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-develop-pre-push-1584/common.sh:mask_service:374 local svc=telegraf
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-develop-pre-push-1584/common.sh:mask_service:375 local container=delphix.iyDDbhM
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-develop-pre-push-1584/common.sh:mask_service:380 [[ -n delphix.iyDDbhM ]]
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-develop-pre-push-1584/common.sh:mask_service:381 chroot /var/lib/machines/delphix.iyDDbhM systemctl mask telegraf
...
...
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-b
uild-develop-pre-push-1584/common.sh:fix_and_migrate_services:478 is_svc_new_or_masked_or_disabled telegraf.service
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-b
uild-develop-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:389 local svc=telegraf.service
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-develop-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:391 systemctl cat telegraf.service
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-develop-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:392 '[' 0 -eq 1 ']'
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: //var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-develop-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:396 systemctl is-enabled telegraf.service
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-develop-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:396 state=enabled
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-develop-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:397 [[ enabled == masked ]]
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-develop-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:397 [[ enabled == disabled ]]
Oct  5 08:11:05 ip-10-110-240-233 upgrade-scripts:upgrade-container[11483]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-develop-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:401 return 1

During apply:

Oct  5 08:39:50 ip-10-110-240-233 upgrade-scripts:execute[83847]: //var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-develop-pre-push-1584/common.sh:fix_and_migrate_services:448 systemctl is-enabled telegraf
Oct  5 08:39:50 ip-10-110-240-233 upgrade-scripts:execute[83847]: telegraf.service is not a native service, redirecting to systemd-sysv-install.
Oct  5 08:39:50 ip-10-110-240-233 upgrade-scripts:execute[83847]: Executing: /lib/systemd/systemd-sysv-install is-enabled telegraf
Oct  5 08:39:50 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:fix_and_migrate_services:448 [[ enabled == enabled ]]
Oct  5 08:39:50 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:fix_and_migrate_services:449 mask_service telegraf ''
Oct  5 08:39:50 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:mask_service:374 local svc=telegraf
Oct  5 08:39:50 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:mask_service:375 local container=
Oct  5 08:39:50 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:mask_service:380 [[ -n '' ]]
Oct  5 08:39:50 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:mask_service:384 systemctl mask --now telegraf
Oct  5 08:39:50 ip-10-110-240-233 upgrade-scripts:execute[83847]: Created symlink /etc/systemd/system/telegraf.service → /dev/null.
Oct  5 08:39:50 ip-10-110-240-233 systemd[1]: Reloading.
Oct  5 08:39:51 ip-10-110-240-233 systemd[1]: Stopping telegraf.service...
Oct  5 08:39:51 ip-10-110-240-233 systemd[1]: telegraf.service: Succeeded.
Oct  5 08:39:51 ip-10-110-240-233 systemd[1]: Stopped telegraf.service.
...
...
Oct  5 08:39:54 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:fix_and_migrate_services:478 is_svc_new_or_masked_or_disabled telegraf.service
Oct  5 08:39:54 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:389 local svc=telegraf.service
Oct  5 08:39:54 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:391 systemctl cat telegraf.service
Oct  5 08:39:54 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:392 '[' 0 -eq 1 ']'
Oct  5 08:39:54 ip-10-110-240-233 upgrade-scripts:execute[83847]: //var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-deve
lop-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:396 systemctl is-enabled telegraf.service
Oct  5 08:39:54 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:396 state=masked
Oct  5 08:39:54 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:397 [[ masked == masked ]]
Oct  5 08:39:54 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:is_svc_new_or_masked_or_disabled:398 return 0
Oct  5 08:39:54 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:fix_and_migrate_services:479 mask_service telegraf.service ''
Oct  5 08:39:54 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:mask_service:374 local svc=telegraf.service
Oct  5 08:39:54 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:mask_service:375 local container=
Oct  5 08:39:54 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:mask_service:380 [[ -n '' ]]
Oct  5 08:39:54 ip-10-110-240-233 upgrade-scripts:execute[83847]: /var/dlpx-update/16.0.0.0-snapshot.20231004163329604+jenkins-selfservice-appliance-build-devel
op-pre-push-1584/common.sh:mask_service:384 systemctl mask --now telegraf.service

@palash-gandhi palash-gandhi force-pushed the dlpx/pr/palash-delphix/81295727-ba5b-49c0-935b-d0c44d5d0ba6 branch from 1dfe4f2 to 4a20e2f Compare September 19, 2023 00:02
prakashsurya
prakashsurya approved these changes Sep 19, 2023
View reviewed changes
Copy link
Contributor

@prakashsurya prakashsurya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, pending some manual tests with the relevant versions to ensure it works as intended (e.g. upgrade from pre-6.0.15, to post-6.0.15).

@palash-gandhi palash-gandhi force-pushed the dlpx/pr/palash-delphix/81295727-ba5b-49c0-935b-d0c44d5d0ba6 branch 3 times, most recently from 2bc9468 to 0c989c7 Compare September 26, 2023 19:29
@palash-gandhi palash-gandhi force-pushed the dlpx/pr/palash-delphix/81295727-ba5b-49c0-935b-d0c44d5d0ba6 branch from 0c989c7 to e156319 Compare October 4, 2023 03:07
@palash-gandhi palash-gandhi force-pushed the dlpx/pr/palash-delphix/81295727-ba5b-49c0-935b-d0c44d5d0ba6 branch from e156319 to 2d6d0fb Compare October 4, 2023 16:26
@palash-gandhi palash-gandhi marked this pull request as ready for review October 5, 2023 16:32
@palash-gandhi palash-gandhi force-pushed the dlpx/pr/palash-delphix/81295727-ba5b-49c0-935b-d0c44d5d0ba6 branch 2 times, most recently from 0ee6761 to 432c681 Compare October 5, 2023 16:48
sebroy
sebroy approved these changes Oct 5, 2023
View reviewed changes
upgrade/upgrade-scripts/common.sh Show resolved Hide resolved
upgrade/upgrade-scripts/execute Show resolved Hide resolved
prakashsurya
prakashsurya approved these changes Oct 9, 2023
View reviewed changes
Copy link
Contributor

@prakashsurya prakashsurya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

annoying, but makes sense.. thank you.

@palash-gandhi palash-gandhi force-pushed the dlpx/pr/palash-delphix/81295727-ba5b-49c0-935b-d0c44d5d0ba6 branch 2 times, most recently from 1c8208c to 6a42a15 Compare October 16, 2023 23:47
@palash-gandhi palash-gandhi force-pushed the dlpx/pr/palash-delphix/81295727-ba5b-49c0-935b-d0c44d5d0ba6 branch from 6a42a15 to e13f3cb Compare October 17, 2023 15:02
@palash-gandhi palash-gandhi merged commit b71c3cd into develop Oct 18, 2023
9 checks passed
@palash-gandhi palash-gandhi deleted the dlpx/pr/palash-delphix/81295727-ba5b-49c0-935b-d0c44d5d0ba6 branch October 18, 2023 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sebroy sebroy sebroy approved these changes

@prakashsurya prakashsurya prakashsurya approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@palash-gandhi @prakashsurya @sebroy