Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🌱 Add Pathfinder assessment migration to CLI #507

Merged
merged 35 commits into from
Oct 30, 2023
Merged
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
c510317
Add Pathfinder assessment migration to CLI
aufi Oct 6, 2023
ffa9dba
Single POST transformed assessment
aufi Oct 10, 2023
a06f7d0
Update migration assessment on-flight
aufi Oct 27, 2023
b35baa7
:bug: Fix latest analayis report. (#504)
jortel Oct 5, 2023
003362a
:ghost: Support seed build args. (#503)
jortel Oct 5, 2023
720212e
:bug: Only marshal assessment sections if not empty (#506)
mansam Oct 5, 2023
c61ea49
:bug: PrepareForArchetype should use criteria and tags (#508)
mansam Oct 6, 2023
697b502
:bug: Increment assessment counter when a new one is created (#511)
mansam Oct 10, 2023
855f10d
:bug: Protect builtin questionnaires from update/delete (#512)
mansam Oct 10, 2023
20a0875
:bug: Fix HTML analysis report. (#513)
jortel Oct 11, 2023
bf0d473
:bug: Fix business service filter of issues. (#515)
jortel Oct 11, 2023
e6678eb
:sparkles: Add `comment` string field to Section (#516)
mansam Oct 11, 2023
e8a2a4f
:bug: Static report app id needs to be a string. (#518)
jortel Oct 11, 2023
23a290c
:seedling: Fix Identity API test Maven sample (#514)
aufi Oct 12, 2023
57dbd78
:seedling: add ticket API test (#477)
khareyash05 Oct 12, 2023
de45958
:bug: Deduplicate tags on imported apps (#517)
mansam Oct 12, 2023
1b52612
:bug: Bump x/net to 0.17.0 (#519)
dymurray Oct 16, 2023
2f2a07b
:sparkles: Better pod OOM reporting. (#521)
jortel Oct 16, 2023
71b7379
:ghost: Update 0.1.2 for License. (#522)
jortel Oct 17, 2023
d4b1822
:book: Update OpenAPI spec w/ Assessment changes (#520)
mansam Oct 17, 2023
ecb7206
:bug: Bucket getdir (#525)
jortel Oct 17, 2023
ceaccd8
:seedling: Delete identity associated with tracker (#529)
mguetta1 Oct 19, 2023
d827869
:sparkles: Add `confidence` and `risk` to Application and Archetype r…
mansam Oct 19, 2023
8837c08
:sparkles: Surface latest analysis effort on Application resource (#537)
mansam Oct 24, 2023
b207a7d
:bug: Explicitly set yaml names for camelCased fields (#535)
mansam Oct 24, 2023
5f5b669
:ghost: add hack to generate jwt token. (#534)
jortel Oct 24, 2023
31807ab
:sparkles: Add Target.provider. (#541)
jortel Oct 25, 2023
efed198
:bug: Add missing Targets to ui.target.order after seeding (#536)
mansam Oct 26, 2023
565e23e
:ghost: Add As/With to v11 Setting (#546)
mansam Oct 27, 2023
b3d8435
:bug: Add `yaml:"inline"` to Resource field tags (#543)
mansam Oct 27, 2023
a8edc31
:bug: Preload assessment stakeholder relationships (#544)
mansam Oct 27, 2023
875009e
:book: Improve API documentation (#545)
mansam Oct 27, 2023
410af4d
:bug: Fix typo in AssessmentList preload (#547)
mansam Oct 27, 2023
b65f7a5
:ghost: Preload Archetype assessment stakeholder relations (#548)
mansam Oct 27, 2023
d71406d
Merge branch 'konveyor:main' into cli-migrate-assessments
aufi Oct 30, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions hack/tool/tackle
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,9 @@ def apiJSON(url, token, data=None, method='GET', ignoreErrors=False):
elif method == 'PATCH':
debugPrint("PATCH data: %s" % json.dumps(data))
r = requests.patch(url, data=json.dumps(data), headers={"Authorization": "Bearer %s" % token, "Content-Type": "application/json"}, verify=False)
elif method == 'PUT':
debugPrint("PUT data: %s" % json.dumps(data))
r = requests.put(url, data=json.dumps(data), headers={"Authorization": "Bearer %s" % token, "Content-Type": "application/json"}, verify=False)
else: # GET
r = requests.get(url, headers={"Authorization": "Bearer %s" % token, "Content-Type": "application/json"}, verify=False)

Expand Down Expand Up @@ -722,6 +725,65 @@ class TackleTool:
# Push the updated assessment
apiJSON(self.tackle2Url + tackle2path("assessments/%d" % assmnt2['id']), self.tackle2Token, data=assmnt2, method='PATCH', ignoreErrors=ignoreErrors)

def migrateAssessments(self, assessmentsFileName="assessments", ignoreErrors=False):
cnt = 0
## Export Pathfinder data for each Application
Copy link
Member Author

@aufi aufi Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jmontleon @jortel Thinking about usage in operator upgrade:

  • option 1 - Does it make sense first dump Pathfinder data on Konveyor 0.2 version, then upgrade Konveyor and import assessments to 0.3 as part of the upgrade steps (option 1) or
  • option 2 - upgrade Konveyor, but keep running Pathfinder, then run the assessment migration live directly from Pathfinder (without Hub proxy) to Hub API without creating temp files and kill Pathfinder after migration succeeded?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to think about this from a workflow where the operator is orchestrating it. Option 2 sounds cleaner to me, from a logic standpoint. Is it possible to run the migration utility process within a pod (maybe a job would be appropriate) that can do the migration between pathfinder and hub?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A job sounds good to me too to execute the python migration script (@dymurray, was it planned to have the script part of the hub image or a special one?). It'd need to get endpoints for Pathfinder (probably the internal one) and Hub API to run the migration.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps I am being confused by the terminology. When I read execute the python migration script I feel like we are rolling something special for this.
I think the original plan is the simplest:

  • Add functionality to the export/import tool to work with Assessments. This is what the tool is for.
  • Add support for hub API version 3 (for all things including assessment).
  • Package the tool with the operator.
  • The operator uses to tool to export/import assessments.

This is an application level migration that does should be performed/orchestrated by the operator, not the hub.

The workflow:

  1. The operator uses the tool to export assessments though the hub API (this is how the tool works).
  2. upgrade tackle.
  3. import the assessments
  4. On success, Get rid of pathfinder.

If the operator doing this in a job is better, it seems to just add moving parts but open to understanding why it's better.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previous comments included also the "execution" part of the assessments migration that will be part of operator repo, so that was misleading, sorry for mixing it.

One more open question is about credentials/authorization - the tool expects username&password which is not known to the Hub (AFAIK), just Keycloak knows it. But as @mansam suggested, an addon token could be used for this assessments migration purpose. would this addon token (that is known to the operator too) be suitable for calling Hub API by the tool instead of login/password @jortel?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The hub has a builtin auth provider the is used for addon tokens.
Yes, we have the operator build/sign an auth token with assessment scopes and signed using the key in the secret. I will prototype that.

#for app in apiJSON(self.tackle2Url + "/hub/applications", self.tackle2Token):
# for assm2 in apiJSON(self.tackle2Url + "/hub/pathfinder/assessments?applicationId=%d" % app.id, self.tackle2Token):
# # Prepare Assessment and find its ID
# assm = Tackle2Object()
# assm.id = assm2['id']
# assm.applicationId = assm2['applicationId']
# assm.status = assm2['status']
# # Prepare Assessment questions and answers
# asqa2 = apiJSON(self.tackle2Url + "/hub/pathfinder/assessments/%d" % assm.id, self.tackle2Token)
# asqa = Tackle2Object()
# asqa.id = asqa2['id']
# asqa.applicationId = asqa2['applicationId']
# asqa.status = asqa2['status']
# asqa.stakeholders = asqa2['stakeholders']
# asqa.stakeholderGroups = asqa2['stakeholderGroups']
# asqa.questionnaire = asqa2['questionnaire']
# self.add('assessments', asqa)
#saveJSON(os.path.join(self.dataDir, assessmentsFileName), self.data["assessments"])

# Upload assessment to Konveyor (expecting Pathfinder hard-coded questionnaire)
dictCollection = loadDump(os.path.join(self.dataDir, assessmentsFileName + '.json'))
print("Processing Pathfinder assessments..")
for passmnt in dictCollection:
print("# Assessment for Application %s" % passmnt["applicationId"])
# Skip if Assessment for given Application already exists
if len(apiJSON(self.tackle2Url + "/hub/applications/%d/assessments" % passmnt["applicationId"], self.tackle2Token, data={"questionnaire": {"id": 1}})) > 0:
print(" Assessment already exists, skipping.")
continue
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is an assessment on the application, import is skipped, so the script can run multiple times without failing.

# Create new assessment
assmnt = apiJSON(self.tackle2Url + "/hub/applications/%d/assessments" % passmnt["applicationId"], self.tackle2Token, data={"questionnaire": {"id": 1}}, method='POST')
aufi marked this conversation as resolved.
Show resolved Hide resolved
aufi marked this conversation as resolved.
Show resolved Hide resolved
for category in passmnt['questionnaire']['categories']:
print("Category %s %s" % (category["order"], category["title"]))
for question in category['questions']:
print(" Question %s %s" % (question["order"], question["question"]))
for option in question['options']:
if option['checked'] == True:
# Find corresponding option in newly created assessment and check it
destSection = next(cat for cat in assmnt['sections'] if cat['order'] == category['order'])
aufi marked this conversation as resolved.
Show resolved Hide resolved
destQuestion = next(que for que in destSection['questions'] if que['order'] == question['order'])
destOption = next(opt for opt in destQuestion['answers'] if opt['order'] == option['order'])
print(" Answer %d %s" % (destOption['order'], destOption['text']))
destOption['selected'] = True
# Set remaining assessment attributes
assmnt['status'] = passmnt['status']
assmnt['stakeholders'] = []
for sh in passmnt['stakeholders']:
assmnt['stakeholders'].append({"id": sh})
assmnt['stakeholderGroups'] = []
for shg in passmnt['stakeholderGroups']:
assmnt['stakeholderGroups'].append({"id": shg})
# Push the updated assessment
apiJSON(self.tackle2Url + "/hub/assessments/%d" % assmnt['id'], self.tackle2Token, data=assmnt, method='PUT')
cnt += 1
print("PUT filled Assessment.")
return cnt

def preImportCheck(self):
# Compatibility checks
# TagCategories on Hub API
Expand Down Expand Up @@ -953,6 +1015,22 @@ if cmdWanted(args, "clean-all"):
print("Cleaning ALL data in Tackle2")
tool.cleanAllTackle2()

# Migrate Pathfinder Assessments to Konveyor Assessments
if cmdWanted(args, "migrate-assessments"):
cmdExecuted = True
# Gather Keycloak access token for Tackle 2
token2 = getHubToken(c['url'], c['username'], c['password'])

# Setup Tackle data migration object
tool = TackleTool(args.data_dir, '', '', c['url'], token2)

# Run the import
print("Starting Pathfinder Assessments to Konveyor Assessment migration.")
appCnt = tool.migrateAssessments(ignoreErrors=args.ignoreImportErrors)

print("Done. %d new Assessment(s) for Application(s) were migrated!" % appCnt)


# Print help if action was not specified
if not cmdExecuted:
print("Unknown action, use tackle --help to see usage.")
Expand Down