-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🌱 Add Pathfinder assessment migration to CLI #507
Merged
Merged
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
c510317
Add Pathfinder assessment migration to CLI
aufi ffa9dba
Single POST transformed assessment
aufi a06f7d0
Update migration assessment on-flight
aufi b35baa7
:bug: Fix latest analayis report. (#504)
jortel 003362a
:ghost: Support seed build args. (#503)
jortel 720212e
:bug: Only marshal assessment sections if not empty (#506)
mansam c61ea49
:bug: PrepareForArchetype should use criteria and tags (#508)
mansam 697b502
:bug: Increment assessment counter when a new one is created (#511)
mansam 855f10d
:bug: Protect builtin questionnaires from update/delete (#512)
mansam 20a0875
:bug: Fix HTML analysis report. (#513)
jortel bf0d473
:bug: Fix business service filter of issues. (#515)
jortel e6678eb
:sparkles: Add `comment` string field to Section (#516)
mansam e8a2a4f
:bug: Static report app id needs to be a string. (#518)
jortel 23a290c
:seedling: Fix Identity API test Maven sample (#514)
aufi 57dbd78
:seedling: add ticket API test (#477)
khareyash05 de45958
:bug: Deduplicate tags on imported apps (#517)
mansam 1b52612
:bug: Bump x/net to 0.17.0 (#519)
dymurray 2f2a07b
:sparkles: Better pod OOM reporting. (#521)
jortel 71b7379
:ghost: Update 0.1.2 for License. (#522)
jortel d4b1822
:book: Update OpenAPI spec w/ Assessment changes (#520)
mansam ecb7206
:bug: Bucket getdir (#525)
jortel ceaccd8
:seedling: Delete identity associated with tracker (#529)
mguetta1 d827869
:sparkles: Add `confidence` and `risk` to Application and Archetype r…
mansam 8837c08
:sparkles: Surface latest analysis effort on Application resource (#537)
mansam b207a7d
:bug: Explicitly set yaml names for camelCased fields (#535)
mansam 5f5b669
:ghost: add hack to generate jwt token. (#534)
jortel 31807ab
:sparkles: Add Target.provider. (#541)
jortel efed198
:bug: Add missing Targets to ui.target.order after seeding (#536)
mansam 565e23e
:ghost: Add As/With to v11 Setting (#546)
mansam b3d8435
:bug: Add `yaml:"inline"` to Resource field tags (#543)
mansam a8edc31
:bug: Preload assessment stakeholder relationships (#544)
mansam 875009e
:book: Improve API documentation (#545)
mansam 410af4d
:bug: Fix typo in AssessmentList preload (#547)
mansam b65f7a5
:ghost: Preload Archetype assessment stakeholder relations (#548)
mansam d71406d
Merge branch 'konveyor:main' into cli-migrate-assessments
aufi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -126,6 +126,9 @@ def apiJSON(url, token, data=None, method='GET', ignoreErrors=False): | |
elif method == 'PATCH': | ||
debugPrint("PATCH data: %s" % json.dumps(data)) | ||
r = requests.patch(url, data=json.dumps(data), headers={"Authorization": "Bearer %s" % token, "Content-Type": "application/json"}, verify=False) | ||
elif method == 'PUT': | ||
debugPrint("PUT data: %s" % json.dumps(data)) | ||
r = requests.put(url, data=json.dumps(data), headers={"Authorization": "Bearer %s" % token, "Content-Type": "application/json"}, verify=False) | ||
else: # GET | ||
r = requests.get(url, headers={"Authorization": "Bearer %s" % token, "Content-Type": "application/json"}, verify=False) | ||
|
||
|
@@ -722,6 +725,65 @@ class TackleTool: | |
# Push the updated assessment | ||
apiJSON(self.tackle2Url + tackle2path("assessments/%d" % assmnt2['id']), self.tackle2Token, data=assmnt2, method='PATCH', ignoreErrors=ignoreErrors) | ||
|
||
def migrateAssessments(self, assessmentsFileName="assessments", ignoreErrors=False): | ||
cnt = 0 | ||
## Export Pathfinder data for each Application | ||
#for app in apiJSON(self.tackle2Url + "/hub/applications", self.tackle2Token): | ||
# for assm2 in apiJSON(self.tackle2Url + "/hub/pathfinder/assessments?applicationId=%d" % app.id, self.tackle2Token): | ||
# # Prepare Assessment and find its ID | ||
# assm = Tackle2Object() | ||
# assm.id = assm2['id'] | ||
# assm.applicationId = assm2['applicationId'] | ||
# assm.status = assm2['status'] | ||
# # Prepare Assessment questions and answers | ||
# asqa2 = apiJSON(self.tackle2Url + "/hub/pathfinder/assessments/%d" % assm.id, self.tackle2Token) | ||
# asqa = Tackle2Object() | ||
# asqa.id = asqa2['id'] | ||
# asqa.applicationId = asqa2['applicationId'] | ||
# asqa.status = asqa2['status'] | ||
# asqa.stakeholders = asqa2['stakeholders'] | ||
# asqa.stakeholderGroups = asqa2['stakeholderGroups'] | ||
# asqa.questionnaire = asqa2['questionnaire'] | ||
# self.add('assessments', asqa) | ||
#saveJSON(os.path.join(self.dataDir, assessmentsFileName), self.data["assessments"]) | ||
|
||
# Upload assessment to Konveyor (expecting Pathfinder hard-coded questionnaire) | ||
dictCollection = loadDump(os.path.join(self.dataDir, assessmentsFileName + '.json')) | ||
print("Processing Pathfinder assessments..") | ||
for passmnt in dictCollection: | ||
print("# Assessment for Application %s" % passmnt["applicationId"]) | ||
# Skip if Assessment for given Application already exists | ||
if len(apiJSON(self.tackle2Url + "/hub/applications/%d/assessments" % passmnt["applicationId"], self.tackle2Token, data={"questionnaire": {"id": 1}})) > 0: | ||
print(" Assessment already exists, skipping.") | ||
continue | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If there is an assessment on the application, import is skipped, so the script can run multiple times without failing. |
||
# Create new assessment | ||
assmnt = apiJSON(self.tackle2Url + "/hub/applications/%d/assessments" % passmnt["applicationId"], self.tackle2Token, data={"questionnaire": {"id": 1}}, method='POST') | ||
aufi marked this conversation as resolved.
Show resolved
Hide resolved
aufi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
for category in passmnt['questionnaire']['categories']: | ||
print("Category %s %s" % (category["order"], category["title"])) | ||
for question in category['questions']: | ||
print(" Question %s %s" % (question["order"], question["question"])) | ||
for option in question['options']: | ||
if option['checked'] == True: | ||
# Find corresponding option in newly created assessment and check it | ||
destSection = next(cat for cat in assmnt['sections'] if cat['order'] == category['order']) | ||
aufi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
destQuestion = next(que for que in destSection['questions'] if que['order'] == question['order']) | ||
destOption = next(opt for opt in destQuestion['answers'] if opt['order'] == option['order']) | ||
print(" Answer %d %s" % (destOption['order'], destOption['text'])) | ||
destOption['selected'] = True | ||
# Set remaining assessment attributes | ||
assmnt['status'] = passmnt['status'] | ||
assmnt['stakeholders'] = [] | ||
for sh in passmnt['stakeholders']: | ||
assmnt['stakeholders'].append({"id": sh}) | ||
assmnt['stakeholderGroups'] = [] | ||
for shg in passmnt['stakeholderGroups']: | ||
assmnt['stakeholderGroups'].append({"id": shg}) | ||
# Push the updated assessment | ||
apiJSON(self.tackle2Url + "/hub/assessments/%d" % assmnt['id'], self.tackle2Token, data=assmnt, method='PUT') | ||
cnt += 1 | ||
print("PUT filled Assessment.") | ||
return cnt | ||
|
||
def preImportCheck(self): | ||
# Compatibility checks | ||
# TagCategories on Hub API | ||
|
@@ -953,6 +1015,22 @@ if cmdWanted(args, "clean-all"): | |
print("Cleaning ALL data in Tackle2") | ||
tool.cleanAllTackle2() | ||
|
||
# Migrate Pathfinder Assessments to Konveyor Assessments | ||
if cmdWanted(args, "migrate-assessments"): | ||
cmdExecuted = True | ||
# Gather Keycloak access token for Tackle 2 | ||
token2 = getHubToken(c['url'], c['username'], c['password']) | ||
|
||
# Setup Tackle data migration object | ||
tool = TackleTool(args.data_dir, '', '', c['url'], token2) | ||
|
||
# Run the import | ||
print("Starting Pathfinder Assessments to Konveyor Assessment migration.") | ||
appCnt = tool.migrateAssessments(ignoreErrors=args.ignoreImportErrors) | ||
|
||
print("Done. %d new Assessment(s) for Application(s) were migrated!" % appCnt) | ||
|
||
|
||
# Print help if action was not specified | ||
if not cmdExecuted: | ||
print("Unknown action, use tackle --help to see usage.") | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jmontleon @jortel Thinking about usage in operator upgrade:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm trying to think about this from a workflow where the operator is orchestrating it. Option 2 sounds cleaner to me, from a logic standpoint. Is it possible to run the migration utility process within a pod (maybe a job would be appropriate) that can do the migration between pathfinder and hub?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A job sounds good to me too to execute the python migration script (@dymurray, was it planned to have the script part of the hub image or a special one?). It'd need to get endpoints for Pathfinder (probably the internal one) and Hub API to run the migration.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps I am being confused by the terminology. When I read execute the python migration script I feel like we are rolling something special for this.
I think the original plan is the simplest:
This is an application level migration that does should be performed/orchestrated by the operator, not the hub.
The workflow:
If the operator doing this in a job is better, it seems to just add moving parts but open to understanding why it's better.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Previous comments included also the "execution" part of the assessments migration that will be part of operator repo, so that was misleading, sorry for mixing it.
One more open question is about credentials/authorization - the tool expects username&password which is not known to the Hub (AFAIK), just Keycloak knows it. But as @mansam suggested, an addon token could be used for this assessments migration purpose. would this addon token (that is known to the operator too) be suitable for calling Hub API by the tool instead of login/password @jortel?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The hub has a builtin auth provider the is used for addon tokens.
Yes, we have the operator build/sign an auth token with assessment scopes and signed using the key in the secret. I will prototype that.