-
Notifications
You must be signed in to change notification settings - Fork 898
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Re-insert MiqFileStorage
and MiqFTPLib
#19547
Re-insert MiqFileStorage
and MiqFTPLib
#19547
Conversation
Butts... Still looks like I have to re-work this a bit... but I think it is a bit closer than it has been. Mostly the first commit isn't bringing everything over that it should, and the links to the commits are wrong (which I know why, and probably can figure out a fix). |
5bcecc4
to
8deaef7
Compare
A bit better now... the links are at least correct. The first commit is still a little weird, since it has nothing to do with the code that I am injecting, so I am going to be revisiting the pruning code tomorrow. But tomorrow... my head hurts from all of this |
8deaef7
to
fe3c13e
Compare
Okay, quick update: This is mostly working as expected thanks to this change: juliancheal/code-extractor@0bbc2c1 However, I did have to update the :name: object_storage_and_ftp_lib
:reinsert: true
:target_name: "ManageIQ/manageiq"
:target_remote: "[email protected]:NickLaMuro/manageiq.git"
:target_base_branch: "master"
:upstream: "[email protected]:NickLaMuro/manageiq-gems-pending.git"
:upstream_name: "ManageIQ/manageiq-gems-pending"
:extractions:
- lib/gems/pending/util/mount/
- lib/gems/pending/util/object_storage/
- lib/gems/pending/util/miq_file_storage.rb
- lib/gems/pending/util/miq_ftp_lib.rb
- lib/gems/pending/util/miq_object_storage.rb
- spec/support/contexts/generated_tmp_files.rb
- spec/support/contexts/with_ftp_server.rb
- spec/support/custom_matchers/
- spec/util/mount/
- spec/util/object_storage/
- spec/util/miq_file_storage_spec.rb
- spec/util/miq_ftp_lib_spec.rb
- spec/util/miq_object_storage_spec.rb
:extra_cmds:
- mv lib/gems/pending/util/* lib/
- git mv spec/util/* spec/lib/
- mkdir -p spec/support/examples_group
- git mv spec/support/contexts/generated_tmp_files.rb spec/support/examples_group/
- git mv spec/support/contexts/with_ftp_server.rb spec/support/examples_group/
- rm -rf lib/gems spec/support/contexts
- git add lib
- git commit -m "Move lib/gems/pending/util/ to lib/"
:destination: "mount_extractions" Leaving the original in the PR description for historical reasons, but I ran through all of the PRs this had touched, so I "think" this should be working now. Hopefully there isn't any files that I missed... again... |
fe3c13e
to
de83ed7
Compare
Okay, tests were passing on the last build, and only I think it makes sense to possibly correct those in the future, but I would rather solve that in a different PR, so ignoring for the time being. For what it is worth, it isn't "adding" new vulnerabilities, since this code already existed previously, just in a different location. Let me know what you think. I am guessing there are a few things that are confusing, and possible some items that you don't like, so let me know if you want me to go over anything. |
MiqFileStorage
and MiqFTPLib
MiqFileStorage
and MiqFTPLib
@NickLaMuro A few questions
|
@@ -0,0 +1,102 @@ | |||
require 'util/miq_ftp_lib' | |||
require 'util/miq_object_storage' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These still say "util/". but that's not where these live now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had thought about fixing this, but forgot when I saw the tests were passing, at first, I was confused at why this was still working, but I think the answer is simple:
The corresponding PR for this in manageiq-gems-pending
is not merged, so without running this with manageiq-cross_repo_tests
, this still loads just fine and the specs pass.
Will look into fixing these next week. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you do a cross_repo test with the removal PR in the manageiq-gems-pending? Then this would prove that out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup, that was the plan. And now is done (sorry for the delay... other things...):
ManageIQ/manageiq-cross_repo-tests#39
Will use that PR going forward... though sure would be nice to have used ManageIQ/miq_bot#454 for this as a test case...
( ͡° ᴥ ͡°)
The only things I had to do was move some files around (and I think as you pointed out), make a few edits. Though, this question is answered more when added to the next...
The main question ("Is this ready to go?"), I think mostly falls on you (and/or the powers that be) that this is a okay route to go with over all, which is mostly what I was waiting on. The stuff that I did in juliancheal/code-extractor#11 was kind of a big "hey, here is how I got here" for this PR, since there was a lot of trial and error that I wanted documented in some fashion, as well as be reusable in some capacity with other items for gems pending that could use the same treatment. I think the commits have been moved over, and the process that I used is #goodEnough™, but I wouldn't mind a thorough look over of that if someone has time. Also, I can walk someone through the steps next week since I realize the process to get here is pretty confusing (even for me...).
This is a easy question to answer. The reason those dependencies were raw like that was because we couldn't include But those gems are already included indirectly via the provider gems, so now we just don't need to include them here since they already exist via that indirect dependency. That said, as we have discussed in the past, it would be ideal if the code for this would live with the providers, but since there isn't a plug-ability hook for database backup providers, this was the first step to getting there. |
So I answered this partially incorrectly. There were a decent amount of changes, and even new files, done in In fact, all the commits that exist here are commits that only exist in (Note: Those commits in that PR were techically "rewritten commits", but the SHAs ended up being different and the content changes were scoped to only the files we were moving back) The one "slight" exception of commits that were pulled over was the first commit:
Which is taking the last commit that existed in both This is one of the things that I really think needs to be critiqued heavily since I think this process would be reproducible for other parts of the "Keep in core" section in: ManageIQ/manageiq-gems-pending#231 Happy to take our time on this to get it right, just re-aligning expectations since I realize this is a very confusing subject matter. |
dcb00ae
to
b00ff96
Compare
Okay, had to rebase this because some of the recent ManageIQ/manageiq-cross_repo-tests#39 I also re-ran the extract in such a way so as to remove the fixup commit that used to exist (previous version of the branch can be found here), and that was fixed by a modification to the --- a/extractions.yml
+++ b/extractions.yml
@@ -21,8 +21,8 @@
- spec/util/miq_object_storage_spec.rb
:extra_cmds:
- mv lib/gems/pending/util/* lib/
+ - mkdir -p spec/support/examples_group spec/lib
- git mv spec/util/* spec/lib/
- - mkdir -p spec/support/examples_group
- git mv spec/support/contexts/generated_tmp_files.rb spec/support/examples_group/
- git mv spec/support/contexts/with_ftp_server.rb spec/support/examples_group/
- rm -rf lib/gems spec/support/contexts This allows the "Move lib/gems/pending/util/ to lib/" commit to do everything properly that was previously needing a Current last two commits:
Were https://github.com/NickLaMuro/manageiq/tree/reinject_mount_and_ftp_libs_old Which there, they are the |
lib/miq_object_storage.rb
Outdated
@@ -0,0 +1,62 @@ | |||
require 'net/protocol' | |||
require 'util/miq_file_storage' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
util
still here?
*** Original Commit message shown below *** Author: Oleg Barenboim [email protected] Tue May 3 23:53:27 2016 -0500 Committer: Oleg Barenboim [email protected] Tue May 3 23:53:27 2016 -0500 Merge pull request ManageIQ#8404 from Ladas/fix_event_monitor_example_script Fix event_monitor_example.rb script (transferred from ManageIQ/manageiq-gems-pending@c0e7a76)
(transferred from ManageIQ/manageiq-gems-pending@2331b93)
Add an Openstack Swift session object to allow backups to Swift. (transferred from ManageIQ/manageiq-gems-pending@ecf715f)
S3 was moved over to the object file session rather than this mount session subclass by a separate PR, so adding it back in via this PR is incorrect and causes Travis build failures. (transferred from ManageIQ/manageiq-gems-pending@5c3ebdc)
Initial commit to support DB backups to S3. Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1513520 (transferred from ManageIQ/manageiq-gems-pending@65ee189)
Removing the swift class for the Mount Session, and adding a new swift class for Object Storage instead. This works with the streaming db backups/dumps now implemented. (transferred from ManageIQ/manageiq-gems-pending@f3a3f05)
Based on PR review comments, use the existing MountSession "add" operation to upload the database file to S3. Correctly handle the situation where the bucket name requested exists but access is denied, indicating (probably) that is owned by another user. (The S3 namespace is Global - each user does not get their own a namespace). (transferred from ManageIQ/manageiq-gems-pending@67fd6ee)
The comment already mentioned that. (transferred from ManageIQ/manageiq-gems-pending@9966890)
Removes some unneeded lines, and unnecessary words in `it` lines. (transferred from ManageIQ/manageiq-gems-pending@354d08f)
Since we need a connection for any download/upload to work (sets @ftp on the instance), we need a slight override of `#magic_number_for` to wrap it with a connection. (transferred from ManageIQ/manageiq-gems-pending@f70c48d)
…o-swift-object-store Add MiqSwiftStorage#download_single (transferred from ManageIQ/manageiq-gems-pending@1525c2c)
If there is no trailing slash in the parsed out `path` from `URI.parse`, then setting of `@container_name` didn't match the regexp, and so it caused the path generated for the URL in the `#container` method to generate as: "/v1/AUTH_asdfglkjwearimsdf/%2Fcontainer_name" Instead of: "/v1/AUTH_asdfglkjwearimsdf/container_name" Where `"%2f"` is due to `fog-core`'s URI escaping code of the string passed into the `swift.directories.get(container_name)` when it includes a slash (so `"/container_name" => "%2Fcontainer_name"`). The fix makes sure the regexp is correct, and the removal of the extra `\/` check at the end was removed. Some extra tests were put in place to make sure everything still works as expected. (transferred from ManageIQ/manageiq-gems-pending@242bf6f)
…to-miq-file-storage [MiqFileStorage] Add #magic_number_for (transferred from ManageIQ/manageiq-gems-pending@b3f99e1)
(transferred from ManageIQ/manageiq-gems-pending@f0a5d5a)
…ngle Fixes to swift storage (transferred from ManageIQ/manageiq-gems-pending@d241557)
Does better fallback handling and nil checking for this method. This can fail as part of `MiqGenericMountSession#add`'s `rescue` statement when the `@source_input` is not defined, and that will mask the actual error that was raised. This fixes that bug so the proper error is shown in the logs. (transferred from ManageIQ/manageiq-gems-pending@7f8b69a)
update aws-sdk gem to v3 (transferred from ManageIQ/manageiq-gems-pending@f92379b)
…rors [MiqGenericMountSession] Fix .source_for_log (transferred from ManageIQ/manageiq-gems-pending@bb0b1d4)
b00ff96
to
2df5336
Compare
Since this has been moved from `manageiq-gems-pending`, this is no longer needed, and keeping it in causes the build to fail.
- Adds ignore cases for two items that have been there for a bit - Removes an outdated warning.
This is used by the `MiqFtpLib` specs
741c7b9
to
982b901
Compare
Some comments on commits NickLaMuro/manageiq@9bc80e4~...982b901 spec/support/examples_group/with_ftp_server.rb
|
Checked commits NickLaMuro/manageiq@9bc80e4~...982b901 with ruby 2.5.5, rubocop 0.69.0, haml-lint 0.20.0, and yamllint 1.10.0 lib/miq_file_storage.rb
lib/miq_ftp_lib.rb
lib/miq_object_storage.rb
lib/mount/miq_generic_mount_session.rb
lib/mount/miq_glusterfs_session.rb
lib/mount/miq_nfs_session.rb
lib/mount/miq_smb_session.rb
lib/object_storage/miq_ftp_storage.rb
lib/object_storage/miq_s3_storage.rb
lib/object_storage/miq_swift_storage.rb
spec/lib/miq_ftp_lib_spec.rb
spec/support/custom_matchers/exist_on_ftp_server.rb
spec/support/custom_matchers/have_size_on_ftp_server_of.rb
spec/support/examples_group/with_ftp_server.rb
|
This is a modified matrix that tests the changes in the following PRs: - ManageIQ/manageiq#19547 - ManageIQ/manageiq-gems-pending#454 but replaces ManageIQ/manageiq#19547 with the following branch: https://github.com/NickLaMuro/manageiq/tree/rebase_testing Which does a few extra steps to attempt to graft commits prior to rebasing. Unsure of the `git` results, but before open a separate PR, this should test to see if specs still pass.
This is a modified matrix that tests the changes in the following PRs: - ManageIQ/manageiq#19547 - ManageIQ/manageiq-gems-pending#454 but replaces ManageIQ/manageiq#19547 with the following branch: https://github.com/NickLaMuro/manageiq/tree/rebase_testing Which does a few extra steps to attempt to graft commits prior to rebasing. Unsure of the `git` results, but before open a separate PR, this should test to see if specs still pass.
This pull request is not mergeable. Please rebase and repush. |
Since #19742 is the replacement for this PR, and there is a high probability of that effort never being merged in favor of a different approach (see #19742 (comment) ), I am going to close this since it has just became unmergeable. |
EDIT @Fryguy: To be merged with ManageIQ/manageiq-gems-pending#454
It's working! (I think...)
This puts back the code for
MiqFileStorage
(mounting, object storage, etc.) and theMiqFTPLib
code frommanageiq-gems-pending
back into this repo. This code really has no business living in gems pending, and until we find a better place for it ("core" gem or what not), this will allow us to clean up the gems pending repo a bit more.Note: The
MiqFTPLib
technically never was a part of this repo in any form, but moving it here in one swoop with the rest of the code that uses it.How (seriously...)
The code to build these commits uses a [WIP] branch of the
code-extactor
project:juliancheal/code-extractor#11
That I have been using to handle and reproduce the steps needed to perform what I have so far (as of opening this PR). As of right now, that code most likely won't be merged into the master branch of that project, but it is a good reference for the steps I took and the trial and error that was required to get the branch to it's current point.
Configuration
The configuration that I used was the following:
(UPDATE: The above is out of date. See #19547 (comment) )
and simply used the executable that was added as part of that branch to run the "re-insert".
Once the whole process was completed, the following extra commands were run on this PR:
(just gave it a shorter branch name)
Links
Steps for Testing/QA (TODO)
I highly suspect that the code in it's current form will break the test suite. I will be adding some follow up commits to address that.