Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-insert MiqFileStorage and MiqFTPLib #19547

Closed

Conversation

NickLaMuro
Copy link
Member

@NickLaMuro NickLaMuro commented Nov 22, 2019

EDIT @Fryguy: To be merged with ManageIQ/manageiq-gems-pending#454

It's working! (I think...)

This puts back the code for MiqFileStorage (mounting, object storage, etc.) and the MiqFTPLib code from manageiq-gems-pending back into this repo. This code really has no business living in gems pending, and until we find a better place for it ("core" gem or what not), this will allow us to clean up the gems pending repo a bit more.

Note: The MiqFTPLib technically never was a part of this repo in any form, but moving it here in one swoop with the rest of the code that uses it.

How (seriously...)

The code to build these commits uses a [WIP] branch of the code-extactor project:

juliancheal/code-extractor#11

That I have been using to handle and reproduce the steps needed to perform what I have so far (as of opening this PR). As of right now, that code most likely won't be merged into the master branch of that project, but it is a good reference for the steps I took and the trial and error that was required to get the branch to it's current point.

Configuration

The configuration that I used was the following:

:name: object_storage_and_ftp_lib
:reinsert: true
:target_name: "ManageIQ/manageiq"
:target_remote: "[email protected]:NickLaMuro/manageiq.git"
:target_base_branch: "master"
:upstream: "[email protected]:NickLaMuro/manageiq-gems-pending.git"
:upstream_name: "ManageIQ/manageiq-gems-pending"
:extractions:
  - lib/gems/pending/util/mount/
  - lib/gems/pending/util/object_storage/
  - lib/gems/pending/util/miq_file_storage.rb
  - lib/gems/pending/util/miq_ftp_lib.rb
  - lib/gems/pending/util/miq_object_storage.rb
  - spec/util/mount/
  - spec/util/object_storage/
  - spec/util/miq_file_storage_spec.rb
  - spec/util/miq_ftp_lib_spec.rb
  - spec/util/miq_object_storage_spec.rb
:extra_cmds:
  - mv lib/gems/pending/util/* lib/
  - git mv spec/util/* spec/lib/
  - rm -rf lib/gems
  - git add lib
  - git commit -m "Move lib/gems/pending/util/ to lib/"
:destination: "mount_extractions"

(UPDATE: The above is out of date. See #19547 (comment) )

and simply used the executable that was added as part of that branch to run the "re-insert".

Once the whole process was completed, the following extra commands were run on this PR:

$ git branch -m reinject_mount_and_ftp_libs
$ git push -u code_extractor_target_for_object_storage_and_ftp_lib reinject_mount_and_ftp_libs

(just gave it a shorter branch name)

Links

Steps for Testing/QA (TODO)

I highly suspect that the code in it's current form will break the test suite. I will be adding some follow up commits to address that.

@miq-bot miq-bot added the wip label Nov 22, 2019
@NickLaMuro
Copy link
Member Author

Butts...

Still looks like I have to re-work this a bit... but I think it is a bit closer than it has been. Mostly the first commit isn't bringing everything over that it should, and the links to the commits are wrong (which I know why, and probably can figure out a fix).

@NickLaMuro NickLaMuro force-pushed the reinject_mount_and_ftp_libs branch from 5bcecc4 to 8deaef7 Compare November 22, 2019 03:05
@NickLaMuro
Copy link
Member Author

A bit better now... the links are at least correct.

The first commit is still a little weird, since it has nothing to do with the code that I am injecting, so I am going to be revisiting the pruning code tomorrow.

But tomorrow... my head hurts from all of this git-foo...

@NickLaMuro
Copy link
Member Author

Okay, quick update:

This is mostly working as expected thanks to this change:

juliancheal/code-extractor@0bbc2c1

However, I did have to update the extraction.yml to the following:

:name: object_storage_and_ftp_lib
:reinsert: true
:target_name: "ManageIQ/manageiq"
:target_remote: "[email protected]:NickLaMuro/manageiq.git"
:target_base_branch: "master"
:upstream: "[email protected]:NickLaMuro/manageiq-gems-pending.git"
:upstream_name: "ManageIQ/manageiq-gems-pending"
:extractions:
  - lib/gems/pending/util/mount/
  - lib/gems/pending/util/object_storage/
  - lib/gems/pending/util/miq_file_storage.rb
  - lib/gems/pending/util/miq_ftp_lib.rb
  - lib/gems/pending/util/miq_object_storage.rb
  - spec/support/contexts/generated_tmp_files.rb
  - spec/support/contexts/with_ftp_server.rb
  - spec/support/custom_matchers/
  - spec/util/mount/
  - spec/util/object_storage/
  - spec/util/miq_file_storage_spec.rb
  - spec/util/miq_ftp_lib_spec.rb
  - spec/util/miq_object_storage_spec.rb
:extra_cmds:
  - mv lib/gems/pending/util/* lib/
  - git mv spec/util/* spec/lib/
  - mkdir -p spec/support/examples_group
  - git mv spec/support/contexts/generated_tmp_files.rb spec/support/examples_group/
  - git mv spec/support/contexts/with_ftp_server.rb spec/support/examples_group/
  - rm -rf lib/gems spec/support/contexts
  - git add lib
  - git commit -m "Move lib/gems/pending/util/ to lib/"
:destination: "mount_extractions"

Leaving the original in the PR description for historical reasons, but I ran through all of the PRs this had touched, so I "think" this should be working now.

Hopefully there isn't any files that I missed... again...

@NickLaMuro NickLaMuro force-pushed the reinject_mount_and_ftp_libs branch from fe3c13e to de83ed7 Compare November 22, 2019 23:14
@NickLaMuro
Copy link
Member Author

Okay, tests were passing on the last build, and only brakeman was having issues.

I think it makes sense to possibly correct those in the future, but I would rather solve that in a different PR, so ignoring for the time being. For what it is worth, it isn't "adding" new vulnerabilities, since this code already existed previously, just in a different location.

@miq-bot assign @Fryguy

Let me know what you think. I am guessing there are a few things that are confusing, and possible some items that you don't like, so let me know if you want me to go over anything.

cc @carbonin @jerryk55

@NickLaMuro NickLaMuro changed the title [WIP] Re-insert MiqFileStorage and MiqFTPLib Re-insert MiqFileStorage and MiqFTPLib Nov 22, 2019
@miq-bot miq-bot removed the wip label Nov 22, 2019
@Fryguy
Copy link
Member

Fryguy commented Dec 11, 2019

@NickLaMuro A few questions

  • Is this a straight reinsert, or are there changes above the code that was transferred from manageiq-gems-pending?
  • Is this ready to go? I see it's unWIP, but then there is a [FIXUP] commit
  • Why do we remove dependencies from manageiq-gems-pending, but they are not added here (specifically fog-openstack...the others are more or less already here)

@@ -0,0 +1,102 @@
require 'util/miq_ftp_lib'
require 'util/miq_object_storage'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These still say "util/". but that's not where these live now.

Copy link
Member Author

@NickLaMuro NickLaMuro Dec 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had thought about fixing this, but forgot when I saw the tests were passing, at first, I was confused at why this was still working, but I think the answer is simple:

The corresponding PR for this in manageiq-gems-pending is not merged, so without running this with manageiq-cross_repo_tests, this still loads just fine and the specs pass.

Will look into fixing these next week. Thanks!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you do a cross_repo test with the removal PR in the manageiq-gems-pending? Then this would prove that out.

Copy link
Member Author

@NickLaMuro NickLaMuro Dec 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, that was the plan. And now is done (sorry for the delay... other things...):

ManageIQ/manageiq-cross_repo-tests#39

Will use that PR going forward... though sure would be nice to have used ManageIQ/miq_bot#454 for this as a test case...

( ͡° ᴥ ͡°)

@NickLaMuro
Copy link
Member Author

@Fryguy

  • Is this a straight reinsert, or are there changes above the code that was transferred from manageiq-gems-pending?

The only things I had to do was move some files around (and I think as you pointed out), make a few edits. Though, this question is answered more when added to the next...

  • Is this ready to go? I see it's unWIP, but then there is a [FIXUP] commit

The main question ("Is this ready to go?"), I think mostly falls on you (and/or the powers that be) that this is a okay route to go with over all, which is mostly what I was waiting on.

The stuff that I did in juliancheal/code-extractor#11 was kind of a big "hey, here is how I got here" for this PR, since there was a lot of trial and error that I wanted documented in some fashion, as well as be reusable in some capacity with other items for gems pending that could use the same treatment.

I think the commits have been moved over, and the process that I used is #goodEnough™, but I wouldn't mind a thorough look over of that if someone has time. Also, I can walk someone through the steps next week since I realize the process to get here is pretty confusing (even for me...).

  • Why do we remove dependencies from manageiq-gems-pending, but they are not added here (specifically fog-openstack...the others are more or less already here)

This is a easy question to answer. The reason those dependencies were raw like that was because we couldn't include manageiq-providers-openstack to them directly, because it was a git gem, and that isn't possible to add in the *.gemspec. So I opted with these to write the connectors directly with the raw API gems for the given providers (manageiq-providers-openstack and manageiq-providers-aws in this case).

But those gems are already included indirectly via the provider gems, so now we just don't need to include them here since they already exist via that indirect dependency.

That said, as we have discussed in the past, it would be ideal if the code for this would live with the providers, but since there isn't a plug-ability hook for database backup providers, this was the first step to getting there.

@NickLaMuro
Copy link
Member Author

NickLaMuro commented Dec 16, 2019

Is this a straight reinsert, or are there changes above the code that was transferred from manageiq-gems-pending?

So I answered this partially incorrectly.

There were a decent amount of changes, and even new files, done in manageiq-gems-pending, so this is not putting back he code that once was here.

In fact, all the commits that exist here are commits that only exist in manageiq-gems-pending, and not any that had existed prior in this repo. This is by design, as re-adding those previously existing commits would be confusing, and you even mentioned it being so in my past example of attempting this:

#15358

(Note: Those commits in that PR were techically "rewritten commits", but the SHAs ended up being different and the content changes were scoped to only the files we were moving back)

The one "slight" exception of commits that were pulled over was the first commit:

Re-insert extractions from ManageIQ/manageiq-gems-pending

*** Original Commit message shown below ***

Author:    Oleg Barenboim [email protected] Tue May 3 23:53:27 2016 -0500
Committer: Oleg Barenboim [email protected] Tue May 3 23:53:27 2016 -0500

Merge pull request #8404 from Ladas/fix_event_monitor_example_script

...

Which is taking the last commit that existed in both ManageIQ/manageiq, and using that as the base commit for this PR. That said, the content of that commit differs, as instead of the original content of the commit, it completely re-injects the state of the files that were meant to be moved over as they were at the state of the commit it is based off of.


This is one of the things that I really think needs to be critiqued heavily since I think this process would be reproducible for other parts of the "Keep in core" section in:

ManageIQ/manageiq-gems-pending#231

Happy to take our time on this to get it right, just re-aligning expectations since I realize this is a very confusing subject matter.

@NickLaMuro NickLaMuro force-pushed the reinject_mount_and_ftp_libs branch from dcb00ae to b00ff96 Compare December 17, 2019 02:31
@NickLaMuro
Copy link
Member Author

Okay, had to rebase this because some of the recent Gemfile changes causing the tests to fail in cross repo:

ManageIQ/manageiq-cross_repo-tests#39

I also re-ran the extract in such a way so as to remove the fixup commit that used to exist (previous version of the branch can be found here), and that was fixed by a modification to the extractions.yml:

--- a/extractions.yml
+++ b/extractions.yml
@@ -21,8 +21,8 @@
   - spec/util/miq_object_storage_spec.rb
 :extra_cmds:
   - mv lib/gems/pending/util/* lib/
+  - mkdir -p spec/support/examples_group spec/lib
   - git mv spec/util/* spec/lib/
-  - mkdir -p spec/support/examples_group
   - git mv spec/support/contexts/generated_tmp_files.rb spec/support/examples_group/
   - git mv spec/support/contexts/with_ftp_server.rb spec/support/examples_group/
   - rm -rf lib/gems spec/support/contexts

This allows the "Move lib/gems/pending/util/ to lib/" commit to do everything properly that was previously needing a [FIXUP] commit to move the offending files.

Current last two commits:

  • "[Gemfile] Add ftpd to test dependencies"
  • "Update brakeman.ignore for MiqObjectStorage"

Were git cherry-pick'd back into this branch. I pulled the from the reference branch:

https://github.com/NickLaMuro/manageiq/tree/reinject_mount_and_ftp_libs_old

Which there, they are the reinject_mount_and_ftp_libs_old^^[ref] and reinject_mount_and_ftp_libs_old^ [ref] commits respectively.

@@ -0,0 +1,62 @@
require 'net/protocol'
require 'util/miq_file_storage'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

util still here?

NickLaMuro and others added 8 commits December 24, 2019 15:35
*** Original Commit message shown below ***

Author:    Oleg Barenboim [email protected] Tue May 3 23:53:27 2016 -0500
Committer: Oleg Barenboim [email protected] Tue May 3 23:53:27 2016 -0500

Merge pull request ManageIQ#8404 from Ladas/fix_event_monitor_example_script

Fix event_monitor_example.rb script

(transferred from ManageIQ/manageiq-gems-pending@c0e7a76)
Add an Openstack Swift session object to allow backups to Swift.


(transferred from ManageIQ/manageiq-gems-pending@ecf715f)
S3 was moved over to the object file session rather than this mount session subclass by a separate PR,
so adding it back in via this PR is incorrect and causes Travis build failures.


(transferred from ManageIQ/manageiq-gems-pending@5c3ebdc)
Removing the swift class for the Mount Session, and adding a new swift class
for Object Storage instead.
This works with the streaming db backups/dumps now implemented.


(transferred from ManageIQ/manageiq-gems-pending@f3a3f05)
Based on PR review comments, use the existing MountSession "add" operation
to upload the database file to S3.

Correctly handle the situation where the bucket name requested exists but
access is denied, indicating (probably) that is owned by another user.
(The S3 namespace is Global - each user does not get their own a namespace).


(transferred from ManageIQ/manageiq-gems-pending@67fd6ee)
NickLaMuro and others added 11 commits December 24, 2019 15:35
Removes some unneeded lines, and unnecessary words in `it` lines.


(transferred from ManageIQ/manageiq-gems-pending@354d08f)
Since we need a connection for any download/upload to work (sets @ftp on
the instance), we need a slight override of `#magic_number_for` to wrap
it with a connection.


(transferred from ManageIQ/manageiq-gems-pending@f70c48d)
…o-swift-object-store

Add MiqSwiftStorage#download_single

(transferred from ManageIQ/manageiq-gems-pending@1525c2c)
If there is no trailing slash in the parsed out `path` from `URI.parse`,
then setting of `@container_name` didn't match the regexp, and so it
caused the path generated for the URL in the `#container` method to
generate as:

  "/v1/AUTH_asdfglkjwearimsdf/%2Fcontainer_name"

Instead of:

  "/v1/AUTH_asdfglkjwearimsdf/container_name"

Where `"%2f"` is due to `fog-core`'s URI escaping code of the string
passed into the `swift.directories.get(container_name)` when it includes
a slash (so `"/container_name" => "%2Fcontainer_name"`).

The fix makes sure the regexp is correct, and the removal of the extra
`\/` check at the end was removed.  Some extra tests were put in place
to make sure everything still works as expected.


(transferred from ManageIQ/manageiq-gems-pending@242bf6f)
…to-miq-file-storage

[MiqFileStorage] Add #magic_number_for

(transferred from ManageIQ/manageiq-gems-pending@b3f99e1)
Does better fallback handling and nil checking for this method.

This can fail as part of `MiqGenericMountSession#add`'s `rescue`
statement when the `@source_input` is not defined, and that will mask
the actual error that was raised.

This fixes that bug so the proper error is shown in the logs.


(transferred from ManageIQ/manageiq-gems-pending@7f8b69a)
…rors

[MiqGenericMountSession] Fix .source_for_log

(transferred from ManageIQ/manageiq-gems-pending@bb0b1d4)
@NickLaMuro NickLaMuro force-pushed the reinject_mount_and_ftp_libs branch from b00ff96 to 2df5336 Compare December 24, 2019 23:04
Since this has been moved from `manageiq-gems-pending`, this is no
longer needed, and keeping it in causes the build to fail.
- Adds ignore cases for two items that have been there for a bit
- Removes an outdated warning.
This is used by the `MiqFtpLib` specs
@NickLaMuro NickLaMuro force-pushed the reinject_mount_and_ftp_libs branch from 741c7b9 to 982b901 Compare January 3, 2020 18:51
@miq-bot
Copy link
Member

miq-bot commented Jan 3, 2020

Some comments on commits NickLaMuro/manageiq@9bc80e4~...982b901

spec/support/examples_group/with_ftp_server.rb

  • ⚠️ - 15 - Detected puts. Remove all debugging statements.
  • ⚠️ - 53 - Detected puts. Remove all debugging statements.

@miq-bot
Copy link
Member

miq-bot commented Jan 3, 2020

Checked commits NickLaMuro/manageiq@9bc80e4~...982b901 with ruby 2.5.5, rubocop 0.69.0, haml-lint 0.20.0, and yamllint 1.10.0
25 files checked, 114 offenses detected

lib/miq_file_storage.rb

lib/miq_ftp_lib.rb

lib/miq_object_storage.rb

lib/mount/miq_generic_mount_session.rb

lib/mount/miq_glusterfs_session.rb

  • ⚠️ - Line 16, Col 17 - Lint/UriEscapeUnescape - URI.encode method is obsolete and should not be used. Instead, use CGI.escape, URI.encode_www_form or URI.encode_www_form_component depending on your specific use case.
  • ❗ - Line 25, Col 116 - Layout/MultilineMethodCallBraceLayout - Closing method call brace must be on the line after the last argument when opening brace is on a separate line from the first argument.

lib/mount/miq_nfs_session.rb

  • ⚠️ - Line 15, Col 13 - Lint/UselessAssignment - Useless assignment to variable - userinfo. Use _ or _userinfo as a variable name to indicate that it won't be used.
  • ⚠️ - Line 15, Col 30 - Lint/UselessAssignment - Useless assignment to variable - port. Use _ or _port as a variable name to indicate that it won't be used.
  • ⚠️ - Line 15, Col 36 - Lint/UselessAssignment - Useless assignment to variable - registry. Use _ or _registry as a variable name to indicate that it won't be used.
  • ⚠️ - Line 15, Col 5 - Lint/UselessAssignment - Useless assignment to variable - scheme. Use _ or _scheme as a variable name to indicate that it won't be used.
  • ⚠️ - Line 15, Col 59 - Lint/UselessAssignment - Useless assignment to variable - opaque. Use _ or _opaque as a variable name to indicate that it won't be used.
  • ⚠️ - Line 15, Col 67 - Lint/UselessAssignment - Useless assignment to variable - query. Use _ or _query as a variable name to indicate that it won't be used.
  • ⚠️ - Line 15, Col 74 - Lint/UselessAssignment - Useless assignment to variable - fragment. Use _ or _fragment as a variable name to indicate that it won't be used.
  • ⚠️ - Line 15, Col 95 - Lint/UriEscapeUnescape - URI.encode method is obsolete and should not be used. Instead, use CGI.escape, URI.encode_www_form or URI.encode_www_form_component depending on your specific use case.
  • ❗ - Line 4, Col 11 - Style/MutableConstant - Freeze mutable objects assigned to constants.

lib/mount/miq_smb_session.rb

  • ⚠️ - Line 17, Col 13 - Lint/UselessAssignment - Useless assignment to variable - userinfo. Use _ or _userinfo as a variable name to indicate that it won't be used.
  • ⚠️ - Line 17, Col 30 - Lint/UselessAssignment - Useless assignment to variable - port. Use _ or _port as a variable name to indicate that it won't be used.
  • ⚠️ - Line 17, Col 36 - Lint/UselessAssignment - Useless assignment to variable - registry. Use _ or _registry as a variable name to indicate that it won't be used.
  • ⚠️ - Line 17, Col 5 - Lint/UselessAssignment - Useless assignment to variable - scheme. Use _ or _scheme as a variable name to indicate that it won't be used.
  • ⚠️ - Line 17, Col 59 - Lint/UselessAssignment - Useless assignment to variable - opaque. Use _ or _opaque as a variable name to indicate that it won't be used.
  • ⚠️ - Line 17, Col 67 - Lint/UselessAssignment - Useless assignment to variable - query. Use _ or _query as a variable name to indicate that it won't be used.
  • ⚠️ - Line 17, Col 74 - Lint/UselessAssignment - Useless assignment to variable - fragment. Use _ or _fragment as a variable name to indicate that it won't be used.
  • ⚠️ - Line 17, Col 95 - Lint/UriEscapeUnescape - URI.encode method is obsolete and should not be used. Instead, use CGI.escape, URI.encode_www_form or URI.encode_www_form_component depending on your specific use case.
  • ❗ - Line 4, Col 11 - Style/MutableConstant - Freeze mutable objects assigned to constants.

lib/object_storage/miq_ftp_storage.rb

lib/object_storage/miq_s3_storage.rb

lib/object_storage/miq_swift_storage.rb

spec/lib/miq_ftp_lib_spec.rb

spec/support/custom_matchers/exist_on_ftp_server.rb

spec/support/custom_matchers/have_size_on_ftp_server_of.rb

spec/support/examples_group/with_ftp_server.rb

NickLaMuro added a commit to NickLaMuro/manageiq-cross_repo-tests that referenced this pull request Jan 21, 2020
This is a modified matrix that tests the changes in the following PRs:

- ManageIQ/manageiq#19547
- ManageIQ/manageiq-gems-pending#454

but replaces ManageIQ/manageiq#19547 with the following branch:

https://github.com/NickLaMuro/manageiq/tree/rebase_testing

Which does a few extra steps to attempt to graft commits prior to
rebasing.

Unsure of the `git` results, but before open a separate PR, this should
test to see if specs still pass.
NickLaMuro added a commit to NickLaMuro/manageiq-cross_repo-tests that referenced this pull request Jan 21, 2020
This is a modified matrix that tests the changes in the following PRs:

- ManageIQ/manageiq#19547
- ManageIQ/manageiq-gems-pending#454

but replaces ManageIQ/manageiq#19547 with the following branch:

https://github.com/NickLaMuro/manageiq/tree/rebase_testing

Which does a few extra steps to attempt to graft commits prior to
rebasing.

Unsure of the `git` results, but before open a separate PR, this should
test to see if specs still pass.
@miq-bot
Copy link
Member

miq-bot commented May 7, 2020

This pull request is not mergeable. Please rebase and repush.

@NickLaMuro
Copy link
Member Author

NickLaMuro commented May 7, 2020

Since #19742 is the replacement for this PR, and there is a high probability of that effort never being merged in favor of a different approach (see #19742 (comment) ), I am going to close this since it has just became unmergeable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.