OpenMetadata 1.1.0 Release
UI Overhaul, New Connectors, Improved Lineage Parsing, PII Masking, and lots more
Written By: OpenMetadata Team
Less than two months have passed since our exciting OpenMetadata 1.0 Release, and we’re thrilled to announce the completion of Release 1.1 already! The OpenMetadata community thrives on pushing our limits; this latest release is a testament to it. Prepare to be amazed as we unveil a complete UI overhaul, meticulously designed to elevate the user experience across the entire platform. But that’s not all! We’ve also introduced four new connectors, implemented advanced PII masking, and significantly enhanced lineage parsing capabilities, just to name a few of the numerous features we’ve packed into this release. Stay tuned for an exceptional OpenMetadata experience like never before!
The OpenMetadata Community recently hosted a Webinar showcasing How the ingestion Framework Works. The bottom line: No Airflow? No Problem! While the one-click deploy and manage workflows in UI might not be there, you can still run the exact same processes from — literally — anywhere. In the webinar, we demo ingestion workflows executed from GitHub Actions.
In the upcoming 1.2 Release of OpenMetadata, we are thrilled to introduce exclusive new features specifically tailored for Collate SaaS. You can review Collate’s roadmap here and be as excited as we are 🚀
Community Updates
Thanks to the incredible OpenMetadata Community, our growth and activity have skyrocketed. Slack is buzzing with constant engagement, and we truly appreciate your code contributions, feedback, and feature requests. Our webinars are attracting more attendees, and the June community meeting was extra special, thanks to our first Community Spotlight: Gaétan Soulas from Solocal!
We are excited about our soaring community numbers!
Crossed 2400+ GitHub stars (+200 stars since the previous release)
The Slack community reached 3200+ members (+500 since the previous release)
168 Open-source GitHub developers (+8 since the previous release)
Merged 526 Commits into the 1.1 Release
OpenMetadata 1.1 Release Highlights
UI Overhaul
This release marks a significant milestone for the OpenMetadata platform, bringing many UI changes that are among the most substantial since the start of the project in 2021.
Our primary focus is to simplify the overall experience for users while building upon our already exceptional UI. We are incredibly excited to share these changes with you as they further enhance the platform’s discovery, collaboration, and data quality experience.
Refined Landing Page
One notable improvement is the revamped Landing Page, which aims to ensure users from diverse roles can navigate and explore metadata effortlessly. From the start, we want every user to have a seamless and enjoyable interaction with their data.
We will continue to refine and evolve the Landing Page components, allowing organizations to customize the user experience for different Personas. This flexibility enables highlighting specific areas of interest such as discovery, governance, management, and more.
We are just scratching the surface of what’s to come, so stay tuned for more updates.
Simplified Explore View for Effortless Discovery
The search experience has been enhanced with an improved Asset Details section and a simplified filtering experience with a unique top bar. Before, the explore page was packed with tons of information, and users had filtering options in different places on the screen.
With a centralized approach for filtering, we want users to be able to really focus on what’s important: the data.
Improved Lineage View
Data Lineage has been consistently enhanced in every release. In the current release, we have added support for column pagination and filtering. This will make the exploration of tables with a large number of columns simpler and faster.
Data Quality Redesign
Data Quality Tests have been redesigned for an improved end-user experience. You no longer need to select a test suite when adding tests to a table. This helps organize and schedule all the tests related to a table in a single place, which also helps prevent unnecessary duplication of tests.
The role of Test Suites is now to help users create their own control panels of the tests they’d like to monitor, even if they belong to multiple tables.
With these changes, we still maintain the flexibility of grouping tests based on your use case or domain while simplifying the creation and scheduling flows.
Test Case Failures Resolution
What do we do when a test fails? As an end user of a table, how do I know if an error is being reviewed or already solved? Communication is vital when managing failures or outages, and OpenMetadata now helps streamline this process to ensure no information is lost.
Adding the Resolution field to failed test cases will allow users to:
Acknowledge the failure once the situation is being analyzed,
Update the status to Resolved with the following reasons: Duplicates, False Positive, Missing Data, Out of Bounds, or. Other, if anything other than the above, applies.
Finally, provide any further explanations.
We will introduce more features and options on this Resolution flow. To allow users to explore the already resolved tests that could help identify ongoing issues.
Improved SQL Parsing
Open source thrives when communities grow together. SQLLineage and SQLFluff are the foundations of OpenMetadata’s lineage parsing capabilities. When ingesting lineage from database sources, we get the queries executed on that instance from the query history tables. These queries are then parsed through SQLFluff and SQLLineage to extract the involved tables and their relationship.
In this release, we have continued our efforts to improve the SQL parsing by directly contributing to both communities. We will continue to improve the open-source ecosystem release after release, with your help in identifying those queries that still need support.
Connectors
This release brings out a long list of new features and improvements for existing connectors. On top of that, three new connectors have been added, giving us a total of 66 connectors supported by OpenMetadata.
Here are the highlights of this release:
New Spline Connector to extract metadata and lineage from Spark jobs. Regardless of where the Spark execution happens, if you have configured the Spline Agent, we can send Spark metadata to OpenMetadata.
New SAP Hana Connector, our first integration to the SAP ecosystem.
New MongoDB Connector, extracting Collections as Tables.
New Oracle lineage and usage workflows based on the query history.
Added support for Databricks Unity Catalog for metadata and lineage extraction. If your Databricks instance supports the Unity Catalog, you can enable it in the Connection Details section to use this metadata extraction method instead of getting metadata out of the metastore and history APIs.
IAM role-based authentication for MySQL and Postgres RDS databases.
PII Masking
Crucial data must always be secure in the hands of the concerned stakeholders.
That is why we are introducing PII Masking capabilities to the platform. Let’s break down the scenarios:
We can tag the table or its columns as PII Sensitive for Tables. We will mask the sample data, profiler metrics, and test cases if tagged. The only users that can see the real value are the owners of the tables or admins. This feature goes perfectly with the Auto-Tagging capability introduced in 1.0!
We can apply the same tagging logic for Topics to mask the sample data completely. Finally, Queries will also be masked for non-owners.
In the 1.2 release, we will iterate on this behavior to increase flexibility based on roles and policies to manage who can view sensitive information.
FQN Hashing: Supports Longer Names!
While this is an internal detail, it took weeks of planning and implementation, and we are excited to share the same with you!
As you already know, OpenMetadata is based on a metadata standard powered by JSON Schema. When we bring metadata into the platform, it transforms into this standard and is safely stored in a database (for the curious, more information on the platform’s high-level design here).
Since we want the API to be blazing fast, how we define and manage indexes in the db was a limiting factor on how long the entity names in OpenMetadata could be. In this release, we have improved how we store the data. By storing the hashed names, we can now support longer names in the application without hitting the limits of index sizes.
This change is especially important for tables that come from the Datalake connector (named after the path) or containers from the Storage Services, which can now be nested further.
This is just another example of how we continuously expand the metadata standard and APIs to fit more use cases while retaining architectural simplicity and high performance.
Planned for Release 1.2
We have stated the highlights of our current release above, but more exciting updates are in store for the 1.2 Release! We’ll bring support for Domains, expand the Custom Properties for all entities, and further improve the Glossaries with approval workflows, among many other features! You can check the roadmap here.
Moreover, the Collate team will continue to expand the platform’s capabilities with features built for the SaaS version of OpenMetadata. Review the Collate roadmap here, and let us know your thoughts!
Thanks to our Contributors
We are grateful for the active participation of the OpenMetadata community in making this open-source project successful. It is good to see code contributions from the Community as well as the amazing feedback we have been receiving. Thank you so much for exploring the Sandbox and using OpenMetadata in your organizations. The GitHub issues are a validation of the increased adoption of OpenMetadata in diverse industry verticals.
Thank you, Schlameel, for the code contribution, i.e., adding methods to update Glossary and GlossaryTerm in Python SDK. Thanks to the following users for providing feedback on GitHub that made it to the 1.1 release: Alekseev Vadim, Allen Haozi, Artemnest, Benjamin Meyer, Bojan Vujanic, Chris Utz, Christian Medina, Cristian Calugaru, Daan Damhuis, Deepak Tripathi, Denis Kirichenko, Dhu-Gehang, Dimitri, DovileKr, Eeshan Deshpande, Felipe Caracciolo Gonçalves, Georg Heiler, Giacomo, Giovanni Corsetti, Gnomolio, Guangwei, Guangyu Qu, Haithem Souala, Hesamoddin Monfared, Jan-Niklas Mühlenbrock, Jan Picha, Jonathan Bonnaud, Joosep Vaikma, JRomandv, Karl Putland, Keat Min, Kevin Parijs, Laila Patel, Larspetter Madsstuen, learn-learn4, Martin Trillhaas, Martsec, Matheus Malta, Miki Horn, Mith Matt, Noel Aramis, Ph-eip, Presteniko Septi Rahadian, Rahul Sharma, Raja Mudhireddy, Raphael Solarski, Rufat, Sabine GI, Sai, SapirHi, Sebastian Arriagada, Sevde Gueldre, Sharp Lu, Stéphane Sol, Tyler Pegram, Yossi Ankri Yu Ishikawa, Yuqi Sun, and Zach All.
Please get in touch with us on Slack if you have any questions about code, installation, and docs. For feature requests, please file a GitHub issue or contact us on Slack for feature requests. Are you interested in contributing code? Here are some good starting issues to get you going.
And don’t forget to support us with a GitHub star if you like what we are doing! That will help OpenMetadata to reach a wider audience and to build a community to solve data problems together.
OpenMetadata 1.1.0 Release was originally published in OpenMetadata on Medium, where people are continuing the conversation by highlighting and responding to this story.