OpenMetadata 0.10.1-release

What's Changed

• Support for Postgres as OpenMetadata Store #4601
• UI Improvements in 0.10.1 Release #4600
• Support JWT Token Generation for Bot Accounts #4637
• UI Ingestion Improvements - Support for Dashboards & Messaging Services #4843
• Security: Fix Azure SSO and support refresh tokens (#4988) by @vivekratnavel in #4989

New Contributors

• @cometta made their first contribution in #2916
• @snyk-bot made their first contribution in #3130
• @Beetelbrox made their first contribution in #3185
• @sfirke made their first contribution in #3361
• @mosiac1 made their first contribution in #3599
• @troyel made their first contribution in #3695
• @dvecerdea made their first contribution in #3800
• @Curiosady made their first contribution in #3892
• @barloc made their first contribution in #4037
• @Vj-L made their first contribution in #4092
• @TeraInferno made their first contribution in #3720
• @vitoravancini made their first contribution in #4733
• @habibrhmn made their first contribution in #4800
• @filip-xmargin made their first contribution in #4926
• @mohitdeuex made their first contribution in #4956

Full Changelog: 0.10.0-release...0.10.1-release

OpenMetadata 0.10.0-release

What's Changed

Support for Database Schema

OpenMetadata supports databases, service name databases, and tables. We’ve added Database Schema as part of the FQN. For each external data source, we ingest the database, as well as the tables that are contained underneath the schemas.

Support for Hard Delete

OpenMetadata supported soft deletions. Now, we also support the hard deletion of entities through the UI, APIs, and ingestion. Hard deleting an entity removes the entity and all of its relationships. This will also generate a change event.

Deploy Ingestion from UI

OpenMetadata has refactored the service connections to simplify the ingestion jobs from both the ingestion framework as well as the UI. We now use the pydantic models automatically generated from the JSON schemas for the connection definition. The ‘Add Service’ form is automatically generated in the UI based on the JSON schema specifications for the various connectors that are supported in OpenMetadata.

Download DBT Manifest Files from Amazon S3 or Google Cloud Storage

Previously, when ingesting the models and lineage from DBT, we passed the path of the DBT manifest and catalog files directly into the workflow. We’ve worked on improving the quality of life of DBT. Now, we can dynamically download these files from Amazon S3 or Google Cloud Storage. This way we can have any other process to connect to the DBT, extract the catalog, and put it into any cloud service. We just need the path name and workflow job details from the metadata extraction to be able to ingest metadata.

JSON Schema based Connection Definition

Each service (database, dashboard, messaging, or pipeline service) has its own configuration specifications, with some unique requirements for some services. Instead of the ad hoc definitions of the source module in Python for each connector, we’ve worked on the full refactoring of the ingestion framework. We now use the pydantic models automatically generated from the JSON schemas for the connection definition.

Airflow Rest APIs

The Airflow REST APIs have been refactored. With our API centric model, we are creating a custom airflow rest API directly on top of Airflow using plugins. This passes the connection information to automatically generate all the dags and prepares handy methods to help us test the connection to the source before creating the service.

UI Changes

• The UI improvements are directed towards providing a consistent user experience.
• Hard Deletion of Entities: With the support for the hard deletion of entities, we can permanently delete tables, topics, or services. When the entity is hard deleted, the entity and all its relationships are removed. This generates an ‘EntityDeleted’ change event.
• Dynamic “Add Service” Forms: The ‘Add Service’ form is automatically generated in the UI based on the JSON schema specifications for the various connectors that are supported in OpenMetadata.
• UI Support for Database Schema as part of FQN: The database schema has been introduced in the 0.10 release. All the entity pages now support Database Schema in the UI.
• Lineage Editor: Improvements have been made to the lineage editor.
• Teams: While signing up in OpenMetadata, the teams with restricted access are hidden and only the joinable teams are displayed.
• Team Owner: An Owner field has been added to the Team entity. Only team owners can update the teams.
• Activity Feeds: The Activity Feeds UI supports infinite scrolling.
• Add User: A user can be added from the Users page.

Security Changes

Support Refresh Tokens for Auth0 and Okta SSO

The JWT tokens generated by the SSO providers expire by default in about an hour, making the user re-login often. In this release, we’ve added support for refresh tokens for Auth0 and Okta SSO. The tokens are refreshed silently behind the scenes to provide an uninterrupted user experience. In the future releases, we’ll continue to stabilize authentication and add refresh tokens for the other SSO providers.

Custom OIDC SSO

OpenMetadata now supports integration with your custom built OIDC SSO for authentication. This is supported both on the front end for user authentication as well as on the ingestion side.

Azure SSO

Support has been added for Azure SSO on Airflow.

New Contributors

• @cometta made their first contribution in #2916
• @Beetelbrox made their first contribution in #3185
• @sfirke made their first contribution in #3361
• @OnkarVO7 made their first contribution in #3408
• @mosiac1 made their first contribution in #3599
• @troyel made their first contribution in #3695
• @dvecerdea made their first contribution in #3800
• @Curiosady made their first contribution in #3892
• @MilanBariya made their first contribution in #3949
• @barloc made their first contribution in #4037
• @Vj-L made their first contribution in #4092
• @TeraInferno made their first contribution in #3720

Full Changelog: 0.8.1-release...0.10.0-release

OpenMetadata 0.9.1-release

Collaboration

• Conversations in the main feed
• Users can ask each other questions, add suggestions and replies
• Table details - Click through on usage to see who or what services are using it, and what queries are pulling from it.

Data Quality

• Ability to create and monitor the test cases
• Data Quality Tests support with Json Schemas and APIs
• UI Integration to enable users to write tests and run them on Airflow

Glossary

• Glossaries are a Controlled Vocabulary in an organization used to define the concepts and terminologies specific to a particular domain.
• API & Schemas to support Glossary
• UI support to add Glossary and Glossary Terms.
• Support for using Glossary terms to annotate Entities and Search using Glossary Terms

Connectors

• Apache Iceberg
• Azure SQL
• Clickhouse
• Clickhouse Usage
• Databricks
• Databricks Usage
• Delta Lake
• DynamoDB
• IBM DB2
• Power BI
• MSSQL Usage
• SingleStore
• Apache Atlas, Import Metadata from Apache Atlas into OpenMetadata
• Amundsen, Import Metadata from Amundsen into OpenMetadata

Lineage

• DataSource SQL Parsing support to extract Lineage
• View Lineage support

Pipeline

• Capture pipeline status as it happens

Security

• Security policies through the UI
• Configuration personas and authorization based on policies
• AWS SSO support

New Contributors

• @cometta made their first contribution in #2916
• @Beetelbrox made their first contribution in #3185
• @sfirke made their first contribution in #3361

Full Changelog: 0.8.1-release...0.9.1-release

OpenMetadata 0.9.0-release

What's Changed

• Handle description and tag updates NPE by @harshach in #2856
• Generating Typescript Types from Json Schema. by @Sachin-chaurasiya in #2851
• Fix #2835, #2836 Lineage entities overlap and Descriptions in Lineage view do not have formatting. by @Sachin-chaurasiya in #2840
• Fix #2778 UI : Add Queries tab for table details page. by @Sachin-chaurasiya in #2793
• Fix: #2521 - Add owner support for all services and database by @darth-coder00 in #2858
• Tableau Lineage Implementation by @ayush-shah in #2850
• Table Constraints Added - Ingestion by @ayush-shah in #2854
• Add support for default role (#2631) by @mithmatt in #2676
• Fix #2837: Add a Feed Count API with entityLink filter support by @vivekratnavel in #2860
• Fix #2845 - Init Profiler Workflow by @pmbrull in #2862
• Fix access control rule userRoleAttr to point to roleName by @mithmatt in #2644
• Support swapping default roles by @mithmatt in #2868
• Extend team to support defaultRoles by @mithmatt in #2885
• fix: cherry pick for 0.8.3 by @akash-jain-10 in #2893
• fix #2869: snowflake-usage processor fixed by @ulixius9 in #2896
• Fix #2902 - Check for empty payload by @pmbrull in #2903
• Fix #2867: Add support for GET /feed/{id} by @vivekratnavel in #2892
• Fix #2894 - Profiler Processor & Metrics by @pmbrull in #2900
• Fix : Roles page is showing blank UI when switching to the users tab. by @Sachin-chaurasiya in #2922
• fix incorrect query by @cometta in #2916
• Fix #2890 Show columns icons in Lineage UI enhancement. by @Sachin-chaurasiya in #2924
• Fix #2901: fixed error message by @ulixius9 in #2917
• Removed ColumnDataTypeDisplay check with ColumnDataType by @ayush-shah in #2928
• Fix #2871: Make search magnifying glass clickable to search by @Sachin-chaurasiya in #2923
• Fixes #2899 - Not getting proper relation between parent and child gl… by @sureshms in #2941
• Fix #2953: NoopAuthorizer should create a anonymous user by @harshach in #2940
• Changing selenium version by @kshinde2512 in #2934
• Added Error Handling in Policy tags - Bigquery by @ayush-shah in #2933
• Fix #2905: Added MSSQL Usage Connector by @ulixius9 in #2948
• Refactor EntityReference list population with id list for a given entity type by @mithmatt in #2956
• Fix #2865: Thread should have its own message and posts should only be the replies by @vivekratnavel in #2944
• Support returning teams for GET /roles by @mithmatt in #2958
• Fixes #2942 - Fix bugs reported by Sonar issue on release 0.9 by @sureshms in #2943
• Fix #2878 & #2877 - Implement Metrics and Validate Composed Metrics by @pmbrull in #2926
• Bigquery lineage mapping by @codingwithabhi in #2849
• Fix #2897 - Profiler CLI by @pmbrull in #2967
• Fix #2873: DataQuality test specifications and APIs by @harshach in #2906
• fix: Bump up release version 0.8.4 for docker compose by @akash-jain-10 in #2968
• Fix #2539: redshift lineage mapping by @ulixius9 in #2828
• Adding Webhooks selenium tests by @kshinde2512 in #2911
• Assign default roles from Teams to User by @mithmatt in #2947
• Fix #2966: JWT filter should compare times in UTC to check the validity by @vivekratnavel in #2971
• Update all *.java in openmetadata-ui to use lombok.LOG by @mithmatt in #2973
• Use RequiredArgsConstructor in openmetadata-ui/*.java by @mithmatt in #2974
• fix #2809: Auto populate View Lineage to Tables by @ulixius9 in #2977
• Fix #2965 - Additional Properties set to False by @pmbrull in #2976
• Use lombok Getter to simplify openmetadata-ui/*.java files by @mithmatt in #2975
• Fix #2981 - Update Profile to match TableProfile by @pmbrull in #2982
• Fix #2991: Add ES support for glossary by @harshach in #2993
• Fixes #2962 - Clean up fields parameter choices in GET APIs by @sureshms in #2963
• Fix #2946: Presto Ingestion issues fix by @ulixius9 in #2986
• Fix #2937: added clickhouse usage connector by @ulixius9 in #2987
• Fix Okta Authentication and Validation - Ingestion by @ayush-shah in #2955
• Fix #2996: ChangeEventHandler throws NPE at times by @vivekratnavel in #2998
• Metabase lineage by @codingwithabhi in #3003
• Fix #2834: Merge the change events 'delete' and 'add' as 'update' for Activity Feed by @vivekratnavel in #3006
• updated chrome drivers and added options for headless chrome by @parthp2107 in #3009
• Fix #2807: Merge openmetadata.yaml and openmetadata-server.yaml by @harshach in #2808
• Fix #3012 - Dependency breaking Airflow Container by @pmbrull in #3015
• Fix: #2938 Remove serviceType from service UI route by @Sachin-chaurasiya in #2939
• Feat collaboration by @Sachin-chaurasiya in #2859
• Fix #2984: added azure sso auth by @ulixius9 in #3002
• Fixes #3021 - Restore a soft-deleted container entity along with the … by @sureshms in #3022
• Fixes #3007 - Clean up Sonar flagged issues by @sureshms in #3008
• Fix #3023: Keep ElasticSearch version to 7.13 by @harshach in #3024
• Power BI connector by @ayush-shah in #3019
• Feat : Display a chat icon next to data entities that have threads by @Sachin-chaurasiya in #3026
• Atlas connector by @codingwithabhi in #2855
• ES index: ignore any errors during the es index by @harshach in #1220
• WIP - Selenium fix by @parthp2107 in #3014
• Fix #3037: metadata --version doesn't work by @amiorin in #3038
• Fix #3040 - py_format recipe includes more paths by @pmbrull in #3043
• Fix: #2491 - improvements in google, okta and azure auth by @darth-coder00 in #2929
• fix #3035: handled none entity by @ulixius9 in #3036
• Add UpdateTeam permission support by @mithmatt in #2994
• Fix #3041: Handled AttributeError by @ulixius9 in #3042
• Fix: azure auth callback from UI by @darth-coder00 in #3048
• Fix #2989: Add delete apis for tests by @harshach in #3020
• Fix #3047: OpenMetadata Server fails to run when updating config from Environment variable by @harshach in #3049
• Fix #2866: Add support for PATCH to feed API to be able to resolve a thread by @vivekratnavel in #3027
  *...

OpenMetadata 0.8.4-release

This is a bug fix release for #2940.

OpenMetadata 0.8.3-release

This is a bug fix release for the #2490.

OpenMetadata 0.8.2-release

This is a bug fix release for the #2843

OpenMetadata 0.8.1-release

This is a bug fix release for the #2843

OpenMetadata 0.8.0-release

Access Control Policy

• New entities called ‘Role’ and ‘Policy’ have been added.
• A User has a ‘Role’. A ‘Policy’ can be assigned to a Role.
• A Policy has a set of ‘Rules’. Rules are used to provide access to functions like updateDescription, updateTags, updateOwner and so on.
• Can provide access to metadata operations on any entity.
• A standard set of Roles with their Policies have been added in the new release.
• ‘Admins’ and ‘Bots’ can perform any metadata operation on any entity.
• Admins can define policies through the Policy UI, and assign roles to the Users.

Manual Lineage

• Enhance the lineage captured from machine metadata with user knowledge.
• Users can edit the lineage and connect the entities with a no-code editor.
• Drag and drop UI has been designed to add lineage information manually for the table and column levels.
• Entities like table, pipeline, and dashboard can be dragged and dropped to the lineage graph to create a node.
• The required entity can be searched and clicked to insert into the graph.

Event Notification via Webhooks & Slack Integration

• Subscribe event notifications via webhooks.
• Send metadata change events as Slack notifications
• Provide timely updates to keep the data team informed of changes

Entity Deletion

• API support has been added for entity deletion, both for soft delete and hard delete.
• A deleted dataset is marked as deactivated in the OpenMetadata backend instead of hard deleting it.
• Ingestion support has been added to publish entity deletion.
• Enabled version support for deleted entities.

Version panel has been added for all the entities- Table, Topic, Pipeline, and Dashboard.

• Previously, we were getting the change descriptions for a limited set of fields for the Topic entity; several other fields have now been included.

New Connectors

• Supports Delta Lake, an open source project that enables building a Lakehouse architecture on top of data lakes.
• Worked on the refactor of SQL connectors to extract the lineage.
• Connector API was refactored to capture the configs on the OpenMetadata side and to schedule the ingestion via UI.

Other Features

• DataSource attribute has been added to the ML model entity.
• Python API has been updated to add lineage for ML Model entities.
• A new tab called ‘Bots’ has been added to group users with isBot set to true.
• Support Application Default Credentials or a keyless, default service account in BigQuery data ingestion.
• Includes a feature tour for new users.

OpenMetadata 0.7.1 Release - Bug Fix

1. Fixed migrate issue for ./bootstrap-storage.sh migrate-all
2. Fixed sql v2 file to drop if exists on dbt_table_entity