Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Block storage: correct steps order #701

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

rajathere
Copy link
Contributor

To deploy block storage service i.e. cinder, we should first remove
the service entries from old deployments to avoid any issues in
the new RHOSO deployment.

For that we need to rewrite the steps to:

  1. Deploy cinder-api pod
  2. List all scheduler and backup services from old deployment
  3. Remove the old scheduler and backup services
  4. Deploy scheduler, volume and backup services
  5. Perform DB migration

This will ensure that our RHOSO deployment is fresh and does not
contain traces of errors from services of old deployment.

@rajathere rajathere force-pushed the block-storage-correct-steps-order branch from cfe2290 to d95a5a3 Compare November 4, 2024 15:45
Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/0fd7d44d3da2451095ad70e511dd46f8

adoption-standalone-to-crc-ceph FAILURE in 2h 12m 18s
adoption-standalone-to-crc-no-ceph RETRY_LIMIT in 55m 18s
✔️ adoption-docs-preview SUCCESS in 1m 15s

@rajathere rajathere force-pushed the block-storage-correct-steps-order branch 2 times, most recently from cb9171b to 43d9b3b Compare November 4, 2024 18:30
Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/38fef15ee8ed4f7497ece2ec21154f5b

adoption-standalone-to-crc-ceph FAILURE in 52m 35s
adoption-standalone-to-crc-no-ceph FAILURE in 55m 05s
✔️ adoption-docs-preview SUCCESS in 1m 12s

@rajathere
Copy link
Contributor Author

recheck unrelated error while deploying standalone

+ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i /home/zuul/.ssh/id_rsa [email protected] 'rm -f /tmp/repo-setup.sh'
kex_exchange_identification: Connection closed by remote host
Connection closed by 192.168.122.100 port 22

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/6256563d3d834913b7395d64660c4e80

adoption-standalone-to-crc-ceph FAILURE in 2h 03m 21s
adoption-standalone-to-crc-no-ceph FAILURE in 2h 00m 42s
✔️ adoption-docs-preview SUCCESS in 1m 12s

@rajathere rajathere force-pushed the block-storage-correct-steps-order branch from 43d9b3b to b3b85d0 Compare November 5, 2024 09:14
This is to support a followup patch where we want to clean up
the old scheduler and backup services before deploying the
cinder services.
To deploy block storage service i.e. cinder, we should first remove
the service entries from old deployments to avoid any issues in
the new RHOSO deployment.

For that we need to rewrite the steps to:

1. Deploy cinder-api pod
2. List all scheduler and backup services from old deployment
3. Remove the old scheduler and backup services
4. Deploy scheduler, volume and backup services
5. Perform DB migration

This will ensure that our RHOSO deployment is fresh and does not
contain traces of errors from services of old deployment.
@rajathere rajathere force-pushed the block-storage-correct-steps-order branch from b3b85d0 to 0201e42 Compare November 5, 2024 09:17
Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/f888d266e238491892ada48ac263a7e5

adoption-standalone-to-crc-ceph FAILURE in 1h 37m 50s
adoption-standalone-to-crc-no-ceph FAILURE in 1h 44m 43s
✔️ adoption-docs-preview SUCCESS in 1m 15s

delay: 2

- name: Get cinder-backup down services
ansible.builtin.shell: |
Copy link
Contributor

@katarimanojk katarimanojk Nov 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rajathere {{ shell_header }} is missing here and at L34
i see the ci [1] failed with this "ERROR! failed at splitting arguments, either an unbalanced jinja2 block or quotes: {{ oc_header }}"

[1] https://softwarefactory-project.io/zuul/t/rdoproject.org/build/ef8a4e5832b34336b15d2c047d1f84a9

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is intentionally removed.
If you see, shell_header provides the following command, set -euxo pipefail
The -o pipefail results in the whole command failing if any of the piped command fails, which we don't want here.
The comment on L#20 describes the scenario and we also do, set -eux avoiding the -o pipefail part

Strangely this playbook runs for me locally but failing here, will try to debug more

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants