diff --git a/hack/tool/README.md b/hack/tool/README.md index 21a159eea..db02fa3ba 100644 --- a/hack/tool/README.md +++ b/hack/tool/README.md @@ -6,7 +6,7 @@ For more details about the Tackle project, see [Tackle2-Hub README](https://gith ## Scenarios -### Export and Import data between Tackle 2 instances +### Export and Import data between Tackle/Konveyor instances Tackle 2 can be exported and imported to another Tackle 2 instance, expecting it is cleaned-up before import. This focuses on application inventory, but exports and imports most of Tackle 2 objects except TaskGroups&Tasks and Keycloak identitty provider data. @@ -24,21 +24,6 @@ Steps for import (tackle-config.yml file points to destination Tackle 2, existin **Note: This export/import functionality doesn't aim to be a backup/restore tool. E.g. it doesn't dump Keycloak data and it requires have the Tackle installation to be running on the destination cluster before importing the data.** -### Migrate data from running Tackle 1.2 to running Tackle 2 instance - -Migrate data updating refs to the seed objects matching to the destination Tackle 2 (tags, tag-types and job functions seeded). - -- ```tackle export-tackle1``` -- ```tackle import``` - -### Export data from Tackle 1.2 to be later imported to some Tackle 2 instance - -Exports full data dump including seeds to be later imported to currently not available Tackle 2 instance, which needs a cleanup before running the import. - -- ```tackle --skip-destination-check export-tackle1``` -- ```tackle clean-all``` -- ```tackle import``` - ### If the import failed The ```tackle import``` command could fail in a pre-import check phase which ensures there are no resources of given type with the same ID in the destination Tackle 2 (error after ```Checking tagtypes in destination Tackle2..``` etc.). In this case, run ```tackle clean``` command, which will remove such objects from the destination Tackle 2 API or remove it manually either from the destination Tackle 2 or from the JSON data files. @@ -72,7 +57,6 @@ Run the tackle tool: ### Supported actions - ```export``` exports Tackle 2 objects into local JSON files and bucket data directory -- ```export-tackle1``` exports Tackle 1.2 API objects into local JSON files - ```import``` creates objects in Tackle 2 from local JSON files and buckets data - ```clean``` deletes objects uploaded to Tackle 2 from local JSON files - ```clean-all``` deletes ALL data returned by Tackle 2 (including seeds, additional to ```clean```), skips some pathfinder stuff without index action @@ -85,12 +69,6 @@ The export dommand connects to Tackle2 Hub API and dumps relevant resources to l **Note: the encryption_passphase needs to be the same when running the import.** -### Export Tackle 1.2 - -Run ```tackle export-tackle1``` to get dump of Tackle 1.2 objects into JSON files in local directory ```./tackle-data```. - -The ```export-tackle1``` command looks into Tackle2 to grab seed resources first, then downloads all resources from Tackle 1.2 API, transforms it to format expected by the Tackle 2 Hub and re-map resources to seeds already existing in destination Tackle2 Hub API. The result is serialized and stored into local JSON files. - ### Import to Tackle 2 Check local JSON dump files in ```./tackle-data``` directory (if needed) and create objects in Tackle 2 Hub running ```tackle import```. @@ -133,7 +111,7 @@ usage: tackle [-h] [-c [CONFIG]] [-d [DATA_DIR]] [-v] [-s] [-w] [-i] [-n] [-b] [ Konveyor Tackle maintenance tool. positional arguments: - action One or more Tackle commands that should be executed, options: export export-tackle1 import clean clean-all + action One or more Tackle commands that should be executed, options: export import clean clean-all options: -h, --help show this help message and exit @@ -164,12 +142,6 @@ password: # Export of Identitiy (credentials) password and key fields should be encrypted, set the passphase encryption_passphase: -# Tackle 1.2 endpoint and credentials, e.g. to dump data and migrate to Tackle2 -tackle1: - url: https://tackle-tackle.apps.mta01.cluster.local - username: tackle - password: - ``` Unverified HTTPS warnings from Python could be hidden by ```export PYTHONWARNINGS="ignore:Unverified HTTPS request"``` or with ```-w``` command option. diff --git a/hack/tool/tackle b/hack/tool/tackle index 2decde9c9..ae8244061 100755 --- a/hack/tool/tackle +++ b/hack/tool/tackle @@ -301,196 +301,6 @@ class TackleTool: print("ERROR: %s record ID %d not found." % (objType, id)) exit(1) - # Gather Tackle 1.2 API objects and map seeded Tackle2 API objects - def dumpTackle1(self): - ### TAG TYPES & TAGS ### - collection = apiJSON(self.tackle1Url + "/api/controls/tag-type", self.tackle1Token) - for tt1 in collection: - # Temp holder for tags - tags = [] - # Prepare TagCategories's Tags - for tag1 in tt1['tags']: - tag = Tackle2Object(tag1) - tag.name = tag1['name'] - # TagCategory is injected from tagType processing few lines below - # Store Tag only if doesn't exist in Tackle2 destination already - self.add('origin-tags', tag) # tmp tags for merge with seed tags lookup - if tag.name.lower() not in self.destData['tags']: - self.add('tags', tag) - tags.append(tag) - # Prepare TagCategory - tt = Tackle2Object(tt1) - tt.name = tt1['name'] - tt.colour = tt1['colour'] - tt.rank = tt1['rank'] - tt.username = tt1['createUser'] - for tag in tags: - tag.tagCategory = copy.deepcopy(tt) - tt.tags = tags - # Store only if doesn't exist in Tackle2 destination already - if tt.name.lower() not in self.destData['tagcategories']: - self.add('tagcategories', tt) - - ### APPLICATION ### - collection = apiJSON(self.tackle1Url + "/api/application-inventory/application?page=0&size=10000", self.tackle1Token) - for app1 in collection: - # Temp holder for tags - tags = [] - # Prepare Tags - debugPrint(app1) - if app1['tags']: - for tagId in app1['tags']: - appTag = self.findById('origin-tags', int(tagId)) - # Check if Tag exists in Tackle2 destination - if appTag.name.lower() in self.destData['tags']: - # Re-map to existing Tackle2 Tag - tags.append(self.destData['tags'][appTag.name.lower()]) - else: - # Use imported Tag, creating a new one to cut association to Tag type - tag = Tackle2Object() - tag.id = appTag.id - tag.name = appTag.name - tags.append(tag) - # Prepare Application - app = Tackle2Object(app1) - app.name = app1['name'] - app.description = app1['description'] - app.tags = tags - if app1['businessService']: - app.businessService = {'id':int(app1['businessService'])} - else: - print("Warning: Application %d %s has not businessService, which is required by Tackle2. Set it manually in %s/applications.json" % (app.id, app.name, self.dataDir)) - ### APPLICATION REVIEW ### - if app1['review']: - rev = Tackle2Object(app1['review']) - rev.proposedAction = app1['review']['proposedAction'] - rev.effortEstimate = app1['review']['effortEstimate'] - rev.businessCriticality = app1['review']['businessCriticality'] - rev.workPriority = app1['review']['workPriority'] - rev.comments = app1['review']['comments'] - rev.application = {'id': app.id, 'name': app.name} - if app1['review']['copiedFromReviewId']: - rev.copiedFromReviewId = app1['review']['copiedFromReviewId'] - self.add('reviews', rev) - #app.repository = app1['repository'] # Not part of 1.2 API - #app.binary = app1['binary'] - #app.facts = app1['facts'] - self.add('applications', app) - - ### DEPENDENCIES ### - collection = apiJSON(self.tackle1Url + "/api/application-inventory/applications-dependency", self.tackle1Token) - for dep1 in collection: - # Prepare Dependency - dep = Tackle2Object(dep1) - dep.to = {'id': dep1['to']['id'], 'name': dep1['to']['name']} - setattr(dep, 'from', {'id': dep1['from']['id'], 'name': dep1['from']['name']}) # Cannot use "from" as an attribute name directly - self.add('dependencies', dep) - - ### ASSESSMENTS & RISKS (per Application) ### - for app in self.data['applications']: - collection = apiJSON(self.tackle1Url + "/api/pathfinder/assessments?applicationId=%d" % app.id, self.tackle1Token) - for assm1 in collection: - # Prepare Assessment - assm = Tackle2Object() - assm.id = assm1['id'] - assm.applicationId = assm1['applicationId'] - assm.status = assm1['status'] - # Prepare Assessment questions and answers - asqa1 = apiJSON(self.tackle1Url + "/api/pathfinder/assessments/%d" % assm.id, self.tackle1Token) - asqa = Tackle2Object() - asqa.id = asqa1['id'] - asqa.applicationId = asqa1['applicationId'] - asqa.status = asqa1['status'] - asqa.stakeholders = asqa1['stakeholders'] - asqa.stakeholderGroups = asqa1['stakeholderGroups'] - asqa.questionnaire = asqa1['questionnaire'] - self.add('assessments', asqa) - - collection = apiJSON(self.tackle1Url + "/api/pathfinder/assessments/assessment-risk", self.tackle1Token, data=[{"applicationId": app.id}], method='POST') - for assmr1 in collection: - # Prepare Assessment Risk - assmr = Tackle2Object() - assmr.assessmentId = assmr1['assessmentId'] - assmr.applicationId = assmr1['applicationId'] - assmr.risk = assmr1['risk'] - self.add('assessments--assessment-risk', assmr) - - collection = apiJSON(self.tackle1Url + "/api/pathfinder/assessments/confidence", self.tackle1Token, data=[{"applicationId": app.id}], method='POST') - for conf1 in collection: - # Prepare Confidence - conf = Tackle2Object() - conf.assessmentId = conf1['assessmentId'] - conf.applicationId = conf1['applicationId'] - conf.confidence = conf1['confidence'] - self.add('assessments--confidence', conf) - - ### STAKEHOLDER ### - collection = apiJSON(self.tackle1Url + "/api/controls/stakeholder", self.tackle1Token) - for sh1 in collection: - # Temp holder for stakeholder's groups - shgs = [] - # Prepare StakeholderGroups - for shg1 in sh1['stakeholderGroups']: - shg = Tackle2Object(shg1) - shg.name = shg1['name'] - shg.description = shg1['description'] - self.add('stakeholdergroups', shg) - shgs.append(shg) - # Prepare StakeHolder - sh = Tackle2Object(sh1) - sh.name = sh1['displayName'] - sh.email = sh1['email'] - sh.stakeholderGroups = shgs - if sh1['jobFunction']: - if sh1['jobFunction']['role'] in self.destData['jobfunctions']: - # Re-map to JobFunction existing in Tackle2 destination - sh.jobFunction = self.destData['jobfunctions'][sh1['jobFunction']['role']] - else: - # Prepare new JobFunction - jf = Tackle2Object(sh1['jobFunction']) - jf.name = sh1['jobFunction']['role'] - self.add('jobfunctions', jf) - sh.jobFunction = jf - self.add('stakeholders', sh) - - ### STAKEHOLDER GROUPS ### - collection = apiJSON(self.tackle1Url + "/api/controls/stakeholder-group", self.tackle1Token) - for shg1 in collection: - # Prepare StakeholderGroup - shg = Tackle2Object(shg1) - shg.name = shg1['name'] - shg.description = shg1['description'] - self.add('stakeholdergroups', shg) - - ### JOB FUNCTION ### - collection = apiJSON(self.tackle1Url + "/api/controls/job-function", self.tackle1Token) - for jf1 in collection: - # Temp holder for stakeholders - shs = [] - # Prepare JobFunction's Stakeholders - for sh1 in jf1['stakeholders']: - sh = Tackle2Object(sh1) - sh.name = sh1['displayName'] - sh.email = sh1['email'] - shs.append(sh) - # Prepare JobFunction - jf = Tackle2Object(jf1) - jf.name = jf1['role'] - jf.stakeholders = shs - # Store only if doesn't exist in Tackle2 destination already - if jf.name not in self.destData['jobfunctions']: - self.add('jobfunctions', jf) - - ### BUSINESS SERVICE ### - collection = apiJSON(self.tackle1Url + "/api/controls/business-service", self.tackle1Token) - for bs1 in collection: - # Prepare JobFunction - bs = Tackle2Object(bs1) - bs.name = bs1['name'] - bs.description = bs1['description'] - bs.owner = bs1['owner'] # Stakeholder - self.add('businessservices', bs) - # Gather Tackle 2 API objects def dumpTackle2(self): ### TAG TYPES & TAGS ### @@ -920,28 +730,6 @@ disableSSlWarnings(args.disableSslWarnings) c = loadConfig(args.config) cmdExecuted = False -# Tackle 1 export steps -if cmdWanted(args, "export-tackle1"): - cmdExecuted = True - # Gather Keycloak access tokens for Tackle1&2 - token1 = getKeycloakToken(c['tackle1']['url'], c['tackle1']['username'], c['tackle1']['password']) - token2 = "" - if not args.skipDestCheck: - token2 = getKeycloakToken(c['url'], c['username'], c['password']) - - # Setup Tackle 1.2->2.0 data migration object - tool = TackleTool(args.data_dir, c['tackle1']['url'], token1, c['url'], token2) - - # Run the export - if not args.skipDestCheck: - print("Loading seed objects from Tackle 2") - tool.loadTackle2Seeds() - print("Exporting Tackle 1.2 objects (this might take a while..)") - tool.dumpTackle1() - print("Writing JSON data files into %s" % args.data_dir) - tool.store() - print("Done. The data could be imported to Tackle 2 using command \"tackle import\"") - # Tackle 2 export steps if cmdWanted(args, "export"): cmdExecuted = True diff --git a/hack/tool/tackle-config.yml.example b/hack/tool/tackle-config.yml.example index cea23d73c..bee0d45a8 100644 --- a/hack/tool/tackle-config.yml.example +++ b/hack/tool/tackle-config.yml.example @@ -6,9 +6,3 @@ password: # Export of Identitiy (credentials) password and key fields should be encrypted, set the passphase encryption_passphase: - -# Tackle 1.2 endpoint and credentials, e.g. to dump data and migrate to Tackle2 -tackle1: - url: https://tackle-tackle.apps.mta01.cluster.local - username: tackle - password: