Releases: odpi/egeria-connector-ibm-information-server
Releases · odpi/egeria-connector-ibm-information-server
Job-level lineage
🎉 New features
- Changes mapping of IGC
data_file_fieldto a newTabularFileColumnrather than theTabularColumnsuper-type - Adds the option to configure a job-level lineage mode for the DataStage connector, if the stage- and column-level details are not desired
🐞 Bug fixes
- No longer erroneously returns results for metadata instances that are not homed in the repository being proxied (see #557)
🔒 Security fixes
- Dependency version updates
🔨 Dependencies
Bug fixes
🎉 New features
- Adds mapping of a DataStage job's transformation project as a Data Engine
Collection(with thanks to @marius-patrascu 📣) - Adds mapping of the primary category of a term in IGC to the
PrimaryCategoryclassification (with thanks to @alexandra-bucur 📣)
🐞 Bug fixes
- Various null-checking fixes to avoid potential NPEs (with thanks to @lpalashevski 📣)
- No longer relies on the use of
main_objectforSemanticAssignmentrelationship (no longer works in latest fixpacks of IGC) - Removes unused (no-op) DataStage mappings for PortImplementations and PortAliases (with thanks to @popa-raluca 📣)
- Updates the name-setting for the
SoftwareServerCapabilityof DataStage connector to use appropriate method (with thanks to @popa-raluca 📣) - Fixes various bugs related to edge cases in searches and complex nested searches
- Fixes mapping of the
minCardinalityproperty ofSchemaAttributefor use in searches
🔒 Security fixes
- Dependency version updates
🔨 Dependencies
v2.5 - Search and lineage improvements
🎉 New features
- Adds stage variable lineage handling to the DataStage connector
- Implements independent process hierarchy handling to improve efficiency of DataStage connector
- Improves paging consistency for IGC searches, by defaulting to a sort by GUID if no other sort criteria is specified
- Allows searching IGC based on instance header properties (eg.
updateTime,createdBy, etc) - Adds a
limitToLineageEnabledJobsparameter to the DataStage connector to only include jobs for which lineage is enabled
🐞 Bug fixes
- Fixes an issue where the lineage mappings included for a stage were not limited to that stage's input / output data stores
- Fixes issues handling embedded fields (
deflated:...) and consistency inqualifiedNames in the DataStage connector - Addresses edge cases searching for hosts and users in IGC
- Improves logging of underlying IGC REST client library (by propagating checked exceptions)
- Avoids producing any relationship from IGC where one end of the relationship is unmapped (avoiding relationships with null ends)
- Fixes issue where a search by property against IGC without criteria caused an NPE
🔒 Security fixes
- Adds a maximum length validation to
qualifiedNames before attempting to blindly tokenize them
🔨 Dependencies
v2.2 - DataStage connector improvements
🎉 New features
- Optimise the discovery of the oldest job since last sync, to apply the project limiters (previously ran across all jobs, ignoring limiters)
- Ensures stage variables are bulk-retrieved as part of the up-front caching of jobs
- Ensures that column-level lineage is traceable across both stage columns and stage variables
- Forces lineage detection before trying to retrieve job details, to avoid potential race conditions between the connector polling and IGC calculating its own lineage details (to expose via its APIs)
- Prefixes DataStage-owned qualifiedNames to be able to quickly bypass searches in IGC connector
- Adds docs clarifying capabilities and limitations of the connectors
- Extends IGC caching to the retrieval of full assets
🐞 Bug fixes
- Ensures any cache misses of links and stage columns are individually looked up rather than skipped
- Fixes issue with caching where a cache miss was not returning the information that was retrieved
- Encodes URL path variables to ensure any special characters are appropriately handled against IGC's REST API
- Fixes a potential NullPointerException when attempting to iterate through relationships that could be non-existent
- Fixes issues with qualifiedNames generated for various DataStage objects, to ensure they are unique
- Removes duplicative lineage mappings that were causing problems in certain stage-level jobs (ie. lookup stages)
🔨 Dependencies
v2.1 - performance optimizations
🎉 New features
- Adds a new project filter to the DataStage connector, to limit which projects are synchronized
- Switches event mapper to be considered an experimental feature: will only be used if the event mapper connector is configured -- this also now means that no administrative privileges are needed for the Information Server user used to run the connectors
- Implements a basic caching mechanism that operates at the level of an Egeria request (covering multiple underlying Information Server requests), to improve performance of connectors
- Updated documentation to reflect the default use of https over http (as of Egeria 2.0)
🐞 Bug fixes
- Fixes a potential NullPointerException related to qualifiedNames in the DataStage connector
- Searching on all types previously attempt to search non-searchable types: this is now avoided
🔨 Dependencies
v2.0 - dependency updates
v1.8 - search enhancements
🎉 New features
- Adds new
findEntitiessearch method - IGC model deprecations for v11.7.1 fp1 (11.7.1.1)
🐞 Bug fixes
- Now retries connections on both a Forbidden (403) and an Unauthorized (401) response
- Updates IGC model to better handle processing of DataStage shared containers
🔒 Security fixes
- Improves XML handling
- Extends cookie whitelist to handle cookies in 11.7.1 fp1 (11.7.1.1)
🔨 Dependencies
⚠️ Known issues
- There is a known issue with the way the connector works and the CTS in v1.8, which will result in this version of the connector failing a variety of tests against v1.8 -- this has been fixed for future releases
Bug fix release
🎉 New features
- Migrates to new error logging framework
🐞 Bug fixes
- Only retries connection on a forbidden response, as on other responses (such as bad requests) a retry otherwise risks overflowing the number of allowed sessions
- Avoids potential NullPointerExceptions in error handling
🔒 Security fixes
- Updates version of various dependencies, and removes unused dependencies
🔨 Dependencies
Configurability, extensibility and security updates
🎉 New features
- Adds basic virtual asset handling in the DataStage connector
- Adds option to run the DataStage connector standalone (without an IGC connector in the cohort), whereby the DataStage connector will create (and home) data assets like files and tables
- Automatically handles custom attributes in an IGC environment (ignoring them by default, if the extra attributes have not been added to the beans)
- Optimizes the retrieval of context and identity information to reduce unnecessary REST calls
- Makes it possible to extend the base connector with your own model extensions (custom attributes, OpenIGC assets) and mapping logic, without needing to maintain a separate fork of the base connector itself
🐞 Bug fixes
- Fixes a regression for IGC v11.7.0.0 where database column
lengthmetadata was not properly parsed - Fixes an issue where
anchorGUIDpointed at the wrong entity for SchemaElements (now properly points to the Asset entity) - Makes date formatting and parsing thread-safe
- Fixes a boolean parsing issue when handling data classes
- Fixes paging and sorting of relationship results for
getRelationshipsForEntitymethod - Fixes a paging issue present in v11.5.x of IGC specific to virtual assets
- Adds sorting in various places to give more deterministic behavior in the outputs produced
⚠️ Potentially breaking changes
- Most errors produced in the Information Analyzer client now throw exceptions rather than silently logging
🔒 Security fixes
- Adds a whitelist for validating cookies to avoid possible cookie injections
- Adds a whitelist for file names for bean generator to avoid potential for malicious injection
🔨 Dependencies
🧩 Open metadata type mappings added / changed in this release
(📦 Entities, 🔗 Relationships, 🏷 Classifications)
| 🧩 | OMRS type | IGC type(s) | Notes |
|---|---|---|---|
| 🔗 | AttachedNoteLog | main_object - main_object | generated relationship (all properties for both endpoints are on a single entity instance in IGC), which cannot be searched |
| 🔗 | AttachedNoteLogEntry | main_object - note | cannot be searched |
| 📦 | NoteLog | main_object | generated entity for those objects in IGC that can have notes left against them |
| 📦 | NoteEntry | note | cannot be searched |
Bug fix release
🎉 New features
- Adds refresh request handling (the publication of refresh events)
🐞 Bug fixes
- Fixes list-wrapped primitive IGC values
- Fixes search bugs:
- Fixes a bug where EntitySummary objects were not being properly initialized
- Fixes a bug where the EventMapper was not properly handling some relationships, causing NPEs
- Fixes a bug where classification searches did not use regular expressions for string properties
- Fixes a bug where SubjectArea searches did not follow the documented behavior for how SubjectArea classification has been implemented
- Fixes a bug where the
verifyAttributeTypeDefmethod would not throw any exception for an unsupported AttributeTypeDef (now throwsTypeDefNotSupportedException) - Fixes enumeration mappings to ensure they can be tested for equality
- Code and packaging cleanups
- Adds extensive testing to the build process
⚠️ Potentially breaking changes
IGCRestClientno longer starts its connection on construction, but must be explicitly started by invoking itsstart()method (which returns a boolean indicating success or failure of the startup). This also replaces the need for theisSuccessfullyInitialised()method, which has therefore been removed. If startup is unsuccessful, a runtime exception will be thrown indicating the reason.