Apache Gravitino 0.7.0-incubating

Apache Gravitino 0.7.0 is the second major release after entering the ASF. In this release, the community mainly focused on strengthening cloud support to make Gravitino work better in cloud environments.

This release blog will briefly introduce the new features related to cloud support and other significant features and improvements. Please keep reading to learn more about what the community has worked on.

Cloud storage support for Gravitino

As more and more users run their data stacks in the cloud and use cloud object storage, cloud storage support has become an imperative requirement. In this release, the community has mainly focused on adding cloud storage support for Gravitino to make sure Gravitino itself and its connectors/sources can work smoothly with cloud storage.

In this release:

• Gravitino Iceberg REST catalog server now supports different cloud storage services, including AWS S3, Google GCS, Aliyun OSS. Users can simply configure it to make it work.
• Gravitino Fileset catalog now supports managing files (objects) stored in S3, GCS, and OSS. Gravitino provides both server-side pluggable framework and client-side Java / Python GVFS (Gravitino Virtual File System) SDK. Users can easily use their existing tools with the Gravitino provided bundled packages to access the data in the cloud. Also, Gravitino provides a pluggable framework for users to implement their own storage support.
• Gravitino’s Hive, Paimon, and Iceberg catalogs also adds and verifies the support with different cloud storage.
• Gravitino’s Spark and Trino connectors also verifies cloud storage support.

Overall, with the 0.7.0 release, Gravitino generally supports working with different cloud storage services. To know more, you can check our issue #4396. Also, we’re continuing to add more cloud storage support in future releases. Please stay tuned.

Credential vending support in Gravitino

Besides cloud storage support, credential vending support is also vital for Gravitino, especially to work with cloud storage. The traditional way of using AKSK is not convenient or safe. With credential vending technology, the Gravitino server will help users get temporary tokens for authentication, significantly simplifying the client-side configurations and centralizing authentications.

Gravitino 0.7.0 introduces a framework to support Credential vending and also adds S3 and GCS token support. This framework is integrated into Gravitino's Iceberg REST catalog service. So users can smoothly access Iceberg tables on S3 and GCS with authentication.

This is just the first step of credential vending, and we will add more integrations with Gravitino, like fileset support, connector support, etc, in the next release.

For the details of credential vending, please check the issue #4398 and the design document.

Unified access control improvements

Gravitino 0.6.0 introduced the alpha version of unified access control with Apache Ranger support (here), but this feature still needs improvement. In version 0.7.0, we added many improvements and bug fixes to make this end-to-end access control workable. Now, with the release of 0.7.0, the Gravitino unified access control works well with Spark and Ranger to secure end-to-end table access. To see what we have fixed, please check out our issue #4615. You can also try our playground to try out the unified access control feature.

Centralized audit log support

Thanks to the community, Gravitino now supports centralized audit logs. With this feature enabled, users can get the audit information in a centralized place, whether they’re accessing tables or filesets from various sources.

Gravitino’s audit log framework also supports different plugin formatters and writers, and users can implement their own log format and output destinations.

Please see the issues #4887 and #4021 to know more about Gravitino’s centralized audit log.

New data sources support

As a unified data catalog, the community always aims to add more data sources. In this version, Gravitino adds two new data sources. One is Apache Hudi, the other is OceanBase. You can now use Gravitino to manipulate Hudi and Oceanbase metadata in a unified manner.

Various core features

Apart from the features listed above, this version also improves a lot at its core, here are several important features:

• Add PostgreSQL support for storage backend #4101. Gravitino already supports using MySQL, H2 as its backend metadata storage. In 0.7.0, the community adds the PostgreSQL support to enlarge its adoption.
• Unify the catalog and metalake drop behavior #5031. In the previous version, we didn’t enforce the behavior of catalog and metadata drop operations. In this version, we redefine its behavior and make it much safer.
• Manage columns in Gravitino #4493. In 0.7.0, we introduce the column entity in Gravitino, which Gravitino can manage. With this feature introduced, Gravitino can now support tagging on columns, and in future, it will support column-level operations.
• Add an event listener for Iceberg REST catalog server #5204 and support pre-event for event listener #5112.

Other notable enhancements

Gravitino core

• Supporting storing column metadata in Gravitino #4493.
• Support pre-event for Gravitino #5049.
• Unify drop metalake and catalog behavior #5031.
• Add credential vending support in Gravitino #4398.
• Support audit log in Gravitino #4887.
• Shrink the package size of Gravitino #4513.

Iceberg REST catalog server

• Add credential vending for Iceberg REST server. #4993.
• Add an event listener for Iceberg REST server #5204.
• Support pre-event for event listener #5112.

Catalog related

• Add OSS support for fileset catalog #5173.
• Add GCS support for fileset catalog #5074.
• Add S3 support for fileset catalog #3379.
• Add pluggable storage support for fileset catalog #5019.
• Add S3 support for Paimon catalog #4938.
• Add catalog support for Hudi #4306.
• Add catalog support for OceanBase#4848.

API and client

• Add S3 fileset support for Python GVFS client #5188.
• Add GCS fileset support for Python GVFS client #5139.
• Add OSS fileset support for Python GVFS client #5221.
• Supports unified auditing of Fileset metadata and data operations #4021.
• Support OAuth2 in Python GVFS #3758.

UI

• Add UI support for operating fileset #5167.
• Add UI support for operating schema #5140.

All the resolved issues targeting the 0.7.0 release can be seen at https://github.com/apache/gravitino/issues?q=is%3Aissue+is%3Aclosed+label%3A0.7.0+.

Overall

Apache Gravitino 0.7.0 is the second ASF release. This version adds a bunch of new features. We thank the Gravitino community for their continued support and valuable contributions. Thanks to our users' feedback, we can continue to innovate and build,
so thanks to all those reading this!

To further explore the Gravitino 0.7.0 release, please check the documentation.
Your feedback is invaluable to the community and the project.

Credits

This release acknowledges the hard work and dedication of all contributors who have helped make this release possible.

@FANNG1, @LauraXia123, @LindaSummer, @LiuQhahah, @Naresh-kumar-Thodupunoori, @SeanAverS, @caican00, @coolderli, @diqiu50, @featherchen, @hanwxx, @jerqi, @jerryshao, @jingjia88, [@justinmclean...

Gravitino 0.6.1-incubating

We are pleased to announce the stable release of Gravitino 0.6.1-incubating, based on branch-0.6. This release brings a suite of new features and enhancements, particularly focusing on the unified access control system. Additionally, it includes various bug fixes and optimizations across other components.

Security

• Supports list users #3348
• Supports list roles #3346
• Supports list roles by object #4886
• Supports list group #4873
• Supports grant or revoke privileges for a role #4903
• Improved security with additional checks for privilege APIs #5054 #5070
• Fix Hive metastore authentication failed when creating a role #4960
• Remove role local cache #4246
• Addressed a response error in Ranger when calling the Ranger CREATE_GROUP API #4975

Gravitino Core

• Fixed an issue with updating comments in metalake or catalog operations #4845
• Introduced a basic framework to support multiple JDBC backends #4832 #4868
• Fixed a cleanup bug occurring after failed catalog creation attempts #5082

Tag

• Transitioned Tag REST APIs to Object path for improved management #5000

Catalogs

Iceberg

• Use unified logic to transform catalog backend name to handle the renaming of catalog #4718

Doris

• Fix the missing distribution information when loading Doris tables #4988

Trino Connector

• Corrected the default precision settings for Time and Timestamp column types in the Iceberg catalog #4743

UI

• Supports creating Paimon catalog #4742
• Improved user experience by showing an expand arrow when reloading tree nodes #5042

Build and Others

• Fix the env of openAPI lint plugin #4876
• Addressed an Out Of Memory (OOM) issue during Trino connector tests #4871
• Resolved a test failure in testCheckLinkDocs for the web module #4914
• Increase the Python timeout minutes to 45 minutes #5038
• Ensured that TestHiveTableOperations can be run independently #4851
• Added LICENSE and NOTICE files for the Iceberg REST binary to comply with licensing requirements #5010

Limitations and Known Issues

• Please be aware that the Ranger authorization plugin within the unified access control system may exhibit some limitations and known issues. For detailed information, refer to issue #5115.

Credits

We would like to thank the following contributors for their valuable contributions to this release:

@diqiu50 @FANNG1 @jerqi @jerryshao @justinmclean @LauraXia123 @LindaSummer @LiuQhahah @lsyulong @lw-yang @mchades @tyoushinya @yangyuxia @yuqi1129

Gravitino 0.6.0-incubating

This blog post will briefly introduce the new features and significant improvements. Keep reading to learn what the community has worked on and understand Gravitino’s use cases.

Introducing the unified RBAC model for Gravitino

Access control is a crucial feature for the enterprise use of a data catalog, providing users with unified and centralized authorization and authentication capabilities. This release introduces a role-based access control (RBAC) model in Gravitino to authorize different securable objects in a unified manner.

We use Privilege, SecurableObject, Role, User, and Group to define the permissions.

Privilege

Privilege defines the types of operations on different metadata objects, and is used to allow or deny a specific type of operation on a metadata object.

SecurableObject

SecurableObject binds multiple operation-specific types of privileges to a single metadata object.

Role

A Role is a collection of SecurableObjects, and a role represents multiple operation type permissions on multiple metadata objects.

User
Users are granted one or multiple roles, and users have different operating privileges depending on their roles.

Group

To make it easier to grant a single permission to multiple users, we can add users to a group, and then grant one or more roles to that user group. This process allows all users belonging to that user group to have the permissions in those roles.

More importantly, the privileges authorized by the user in Gravitino will be pushed down to the underlying permission system. Currently, we support push permissions to Apache Ranger, others like IAM are under development.

For more information about how our RBAC works, please check out our design document. To enable and use access control in Gravitino, please refer to the user document.

Our implementation of unified access control capability is still in the alpha stage, and we’re striving to add more features and make it stable as soon as possible, so please stay tuned.

Separation of the Iceberg REST catalog service

Apache Iceberg is a first-class citizen, and Gravitino has provided an embedded Iceberg REST catalog service since version 0.3. We have seen the increased demands and adoption of Iceberg REST catalog service as a standalone server. So, in version 0.6.0, we refactored the whole architecture and modularized the Iceberg REST catalog service as a standalone service, allowing it to be deployed with or without the Gravitino server. Besides the refactoring, we also bumped the supported version to Iceberg 1.5.2, added support for S3 cloud storage, and now support the registerTable interface.

Iceberg REST catalog support is crucial to Gravitino, and modularization is just the first step. In future releases, we will add more features like cloud storage support and integrating Gravitino’s RBAC model, credential vending, etc.

To use the Gravitino Iceberg REST catalog service, please check our user document. The umbrella issue is #4058.

Tagging support

Tagging on metadata objects is useful for data discovery, classification, and data governance. It can also be leveraged by query engines to provide tag-based access control. In Gravitino 0.6.0, we introduce tag support users can add tags on metadata objects like CATALOG, SCHEMA, TABLE, FILESET, and TOPIC. To know how our tag system is designed, please check out the design document and issue #3344. To use tags in both REST API and Java SDK, please see how to manage tags.

Apache Flink Gravitino connector

As an open data catalog, we want to be able to support all query engines. Therefore, alongside Trino and Apache Spark, we have added Apache Flink as our newest supported query engine.

In 0.6.0, we added a new Flink Gravitino connector #1354 and supported querying Hive tables using Flink with Gravitino. Hive support is just our first step, we will continue to add more table support.

To know how to use the Flink Gravitino connector, please refer to our documentation.

Apache Paimon table management in Gravitino

Apache Paimon has become quite popular this year, and many companies use Paimon to build their streaming warehouse or lakehouse. To manage all the lakehouse tables in a unified manner, Gravitino has added Paimon table management in 0.6.0 #1129. Users can use our unified API to manage Paimon tables as well as other tables. To know more about how to manage Paimon tables, please refer to Lakehouse Paimon Catalog document.

Add Python GVFS support for fileset

In Gravitino 0.5, we added a Java Hadoop Compatible Filesystem (HCFS) support (GVFS) for fileset read/write in Gravitino. The provided Java GVFS can be used by query engines like Apache Spark to read/write data from files or folders. Although this works well in big data, AI development is largely dominated by Python, which can create an obstacle and hinder users from using Fileset with AI frameworks.

In 0.6.0, we followed the Python fsspec to provide a Python GVFS package that can be used by popular Python frameworks like Apache Arrow, Pandas, Ray, LlamaIndex, and more. You can check out Python GVFS document for more information.

Notable enhancements

Gravitino core

• Support catalog reload after a property is altered #2267.
• Deprecate KV store and add H2 support as embedded storage backend #3968.

Catalog relate

• Add API test catalog connection #4107.
• Improve the type system to support unknown types #3427.
• Add Kerberos support for fileset Hadoop catalog #3462.
• Add S3 support for Iceberg #4264.
• Support cloud and region property when creating catalog #3966.
• Support multiple Kerberos authentication for Hive catalog #3906.
• Unify the behavior of purge for all the catalogs #3685.

API and client

• Refactor Java and Python API for better user experience #3626.
• Add missing error handlers in Python client #4225.

All the resolved issues targeting the 0.6.0 release can be seen at https://github.com/apache/gravitino/issues?page=12&q=is%3Aissue+is%3Aclosed+label%3A0.6.0.

Overall

Apache Gravitino 0.6.0 is the first ASF release, we would like to show appreciation to the Gravitino community for their continued support and valuable contributions. Thanks to the feedback of our users, we are able to continue to innovate and build, so thanks to all those reading this!

To explore Gravitino 0.6.0 release, please check the documentation. Your feedback is invaluable to the community and the project.

Credits

This release acknowledges the hard work and dedication of all contributors who have helped make this release possible.

@1996fanrui @BSSsunny @FANNG1 @IamSaker @JinsYin @JosefinaOller @LanceHsun @LauraXia123 @Leonidas963 @LindaSummer @MukarramHaq @Naresh-kumar-Thodupunoori @Nishtha-Jain-1119 @SteNicholas @TEOTEO520 @Vishesh-Paliwal @ashwin1596 @bknbkn @caican00 @ch3yne @charliecheng630 @coolderli @danhuawang @diqiu50 @featherchen @hanwxx @ian910297 @jenish-thapa @jerqi @jerryshao @jingjia88 @jtao1 @justinmclean [@kalencaya](https://github....

v0.5.1

Gravitino Release 0.5.1

Gravitino 0.5.1 is a stable release based on branch-0.5 that contains bug fixes and introduces some core features for Trino and Spark connectors and security. Additionally, the Python client is ready to use in version 0.5.1.

Core Features

1. Apache Spark Connector Enhancement: Supports multiple Spark (3.3, 3.4, 3.5) and Scala (2.12, 2.13) versions. Introduces several advanced features for Iceberg support like row-level operations, time travel query, procedures, etc.

2. Trino Connector Improvement: Improve the Gravitino trino connector to use Trino's dynamic catalog framework to better support queries in the distributed mode.

3. Security Improvement Supports the condition of privileges, support authentication between Gravitino and Hive, Hadoop catalog.

4. Python client Fixed several critical bugs, enhancing the usability of the Gravitino Python client for real environments.

Gravitino Core

• Supports time sliding window to record metrics #3152
• The semantics of the delete operation for relational storage should be aligned with kv storage #3092
• Provide specific classLoader for Partition operation #2997
• Support preassigned partitions when creating partitioned table #2266
• Remove log4j from client and client java runtime module #3649
• Improve compatibility issues with data type serde #3598

Security

• Privilege should have deny mechanism #3342
• Authentication between the Hadoop catalog and Gravitino. #3337
• Support Kerberos client authentication in gvfs #2711
• Add Hive user authentication end-to-end test #3489
• Add IT to test kerberosized HDFS #3432
• Fix the Hive catalog bug In multiple Kerberized HMS #3295

Catalogs

Iceberg

• Support Iceberg RestCatalog in Gravitino #3149
• Add reserved properties to Table Properties when load an Iceberg table #2962
• Bucket and truncate are missing width params in Gravitino SortOrder #2921

Doris

• Support creating Doris table with partition #3347
• Filter system database when list database for Doris catalogs #3275

MySQL

• jdbc-mysql catalogs failed on mysql 8.x driver #3269
• MysqlTypeConverter cannot handle the datetime data type #1761

Kafka

• Adding a new Apache Kafka catalog works but times out listing topics #3249
• Drop kafka catalog maybe failed after altering #3673

Trino Connector

• Gravitino Trino connector loads the catalog using Trino's dynamic catalog feature #3524
• Support automatically loading catalog to Trino by using CREATE CATALOG command #3601
• Support create Internal connector like hive/iceberg/mysql/pg/memory #3587
• [trino-connector] Support create Gravitino catalog by using Trino CREATE CATALOG command #3526

Spark Connector

• Support multiple Spark and Scala versions #1574
• Remove iceberg runtime from spark connector jar #3396
• Support passing catalog properties to spark connector #3292
• Support iceberg metadataColumns #2587
• Support row-level operations to iceberg Table #2543
• Support Iceberg time travel in SQL queries #3264
• Support Iceberg RestCatalog with HiveCatalog backend in spark-connector #3193
• Support Iceberg Spark Procedure #3186
• Alter table comment though table properties failed #3137
• Spark connector create hive table failed for using hive #3129

Python Client

• Correct type specification of create_fileset function in fileset_catalog.py #3508
• When calling hash() function in name_identifier with python client throw an exception #3521
• Add test coverage tool for client-python #3517
• Add Pylint Rules for client-python naming rules #3205
• Add auto formatter for client-python #3206
• Add Pylint Rules for client-python #3203
• Unspecified pylint version in client-python #3558

UI

• Gravitino icon doesn't show on UI #3413
• UI: the font family of gravitino title text in production mode does not match with dev mode #3386
• With only one metalake UI presents user with a choice #3285
• Gravitino UI when creating a new fileset catalog it seems to be missing location property #3257

Build and Others

• Unable to build Gravitino in Linux environment due to JAVA problem #3336
• ./gradlew test -PskipITs failed in python-client test #3319
• TrinoContainer#checkSyncCatalogFromGravitino will always return false #3237

Limitation and known issues

• The query in Trino connector for 0.5.1 has slower performance than version 0.5.0 #3738

Credits

This release acknowledges the hard work and dedication of all contributors who have helped make this release possible.

@caican00 @ch3yne @danhuawang @diqiu50 @FANNG1 @featherchen @jerqi @kalencaya @LauraXia123 @lw-yang @MukarramHaq @mchades @noidname01 @qqqttt123 @SteNicholas @TEOTEO520 @theoryxu @unknowntpo @xloya @xiaozcy @xunliu @yijhenlin @yuqi1129 @zhoukangcn

Gravitino 0.5.0 Latest

Gravitino Release 0.5.0

We are glad to announce the release of Gravitino 0.5.0. This release is a major milestone for Gravitino and includes over 240 issues that cover several new features, improvements, and bug fixes.

This release introduces several core features like Apache Spark connector support, messaging catalog support, general user and authority management system, event listener system, and Python client support. In the meantime, we have made a lot of improvements and bug fixes to the existing features.

Core Features

1. Apache Spark connector support: Gravitino now supports Spark connector. You can use Spark to read and write catalog metadata through Gravitino. #1227, for more, refer to spark-connector

2. Messaging catalog support: Gravitino now supports messaging catalogs such as Apache Kafka or Kafka-compatible streaming systems. You can use Messaging Catalog to manage your messaging catalog. #2369, for more, please refer to kafka-catalog

3. General user and access management: Gravitino now supports general user and access management. #2232. This feature is currently in an alpha phase and is not available for production usage.

4. Event listener system: Gravitino now supports an event listener system. You can use it to manage all operation events or use the hook mechanism for your own events, such as operation history auditing, operation monitoring, etc. etc. #2233, refer to event-listener-configuration for more information

5. Python client support: Gravitino now supports a Python client. Users can use Python to connect to Gravitino and operate the catalog directly. #2229. Currently, we only support fileset type catalogs with Python clients.

6. Doris catalog support: Gravitino now supports Apache Doris catalogs. #1339, for more information, you can refer to jdbc-doris-catalog

7. Support JDBC backend store: Gravitino now supports using a JDBC backend store besides RocksDB. If you want to use MySQL or PostgreSQL as the entity store, you can use the JDBC entity store. #1811, for more, refer to storage-configuration

8. Support fileset catalog: Gravitino now supports Fileset catalogs. with it, you can manage non-tabular data on HDFS, S3, or other Hadoop-compatible filesystems. #1241, for more, refer to fileset catalog

9. Introduce capability framework: This framework shows the capability of different catalogs such as whether the name is case sensitive, name specification, whether null values are supported or not, and so on. #2952

Gravitino core

• Handling multi-thread issues in Gravitino and introducing tree lock. #407
• Introduces the user system. #2232
• Support multiple kinds of entity types in the same namespace with the same name. #2697
• Improve KV GC collector. #1276, #2888
• Improve client API #1292, #2628, #839, #1635, #1793, #1759, #1758
• Separate Java client. #2478
• Make class loaders of catalog able to be GC. #2706
• Support UnparedType to handle an unresolvable type from the catalog. #2117

Catalog related

MySQL & PostgreSQL

• Add PostgreSQL support for array type conversion. #947
• Obtain MySQL table meta information from JDBC metadata. #2934
• Avoid using system tables for MySQL catalogs. #2085

Kafka

• Please see the section Core Features

Fileset

• Please see the section Core Features

Doris

• Please see the section Core Features

Trino connector

• Add datatype test cases for the Trino connector. #2034
• Optimize varchar/char mapping between Gravitino catalogs and the Trino server. #2356
• Support the system table catalog. #2416
• Support update catalog operations in the Trino connector. #2417
• Make the Gravitino Trino connector compatible with Trino 435. #2376

Spark connector

• Please see the section Core Features

Build, test, and CI

• Introduce the error-prone plugin to check the code quality. #2225
• Increase the retry interval of the container status check. #2365
• Isolate catalog class path in IT. #2397
• Extend sleep time in the testInternalCache unit test. #2745
• Add web UI support for the fileset catalog. #2883
• Add check mechanism in CI to validate gradle publish. #2655
• More tests added in the Gravitino web E2E test framework. #1503
• Separate output log of test containers. #2839
• Upload process logs of IT container. #2832
• Merging embedded and deploy test mode for frontend integration test. #2798

Web UI

• Verify whether the catalog exists before creating it. #2324
• Add web UI support for the Kafka catalog. #2614
• Add web UI support for the Fileset catalog. #2292

Documents

• Separate metadata operations into different docs. #2750
• Add a document about how to debug the Gravitino Trino connector locally. #2446

Limitation and known issues

• The Doris catalog does not support features like sort order, distribution, and partitioning, and these features are under development.

Credits

@bknbkn @caican00 @charliecheng630 @ch3yne @coolderli @Clearvive @danhuawang @diqiu50 @FANNG1 @hiirrxnn @ichuniq @jerryshao @justinmclean @lw-yang @Lanznx @LauraXia123 @mchades @MohitKambli @nk1506 @qqqttt123 @shaofengshi @SteNicholas @TEOTEO520 @unknowntpo @xiacongling @xiaozcy @xloya @xunliu @yijhenlin @yuqi1129 @Yangxuhao123 @YxAc @zhaoyongjie @zhoukangcn @zivali

Gravitino 0.4.0

Gravitino Release 0.4.0

Gravitino 0.4.0 is the fourth release since its inception and the second release since becoming open-source. With substantial contributions from the open-source community, this release has addressed over 280 issues.

This release is a stable release that mainly focuses on bug fixes. In the meantime, this release also introduces several features, like the public preview of Gravitino web UI, the support of unified partition management, the column default value and indexing support for table, the Kerberos support for client-server communication, the user impersonation support, the Kerberos support of hive catalog, the multiple Java version support, etc.

To download Gravitino 0.4.0, please visit the releases pages, For detailed changes, you can visit Gravitino milestone 0.4.0 for more details. We have also curated a list of high-level changes here, grouped by major modules.

Highlights

• Add the UI support for Gravitino #605
• Add partition support for Gravitino #1575
• Support default value for column #896
• Support auto increment property for column #1277
• Support index for table #1398 #1707 #1736
• Support Kerberos for client-server communication #1185 #1618
• Support user impersonation #596
• Support Kerberized Hive metastore #1415
• Support Java 8, 11 and 17 for Gravitino #1098
• Introduce fine-grained lock support in Gravitino #778
• Support partitioning, bucketing, sorting for Trino Gravitino connector #1510
• Support query optimization in Trino connector #1491

Gravitino core

• Improve the ConfigEntry to support configuration without default value #950
• Add CORS filter for Jetty server Support Java 8, 11 and 17 for Gravitino #42
• Support auto increment property for column #1277
• Ensure the correct semantics of dropTable and purgeTable #1436
• Support index for table #1398
• Support default value for column #896
• Add partition support for Gravitino #1575
• Support Kerberos for client-server communication #1185 #1618
• Introduce fine-grained lock support in Gravitino #778
• Refine Literal APIs #1379

Catalogs

Hive

• Add partition support for Hive #1575
• Add user impersonation support for Hive catalog #596
• Support Kerberized Hive metastore #1451

Iceberg

• Add the reigster IcebergCatalogConfig abstract class #874
• Support Distribution for Iceberg #1460
• Add the basic framework to store Iceberg metrics #1187
• Support struct column for Iceberg #1313 #1471

MySQL & PostgreSQL

• Support auto increment property for column #1388 #1279
• Support PostgreSQL index #1736
• Support MySQL index #1707
• Refactor to get metadata from JDBC #1737 #1468
• Add column default value support for MySQL and PostgreSQL #1531

Trino connector

• Add the Trino TPC-H and TPC-DS test sets to the Trino connector integration test #1291
• Optimize Trino connector's property mapping system #1455
• Create docker-compose container for Trino test #919
• Support partitioning, bucketing, sorting for Trino Gravitino connector #1512
• Support query optimization in Trino connector #1491

UI

• Add the UI support for Gravitino #605

Build and others

• Remove sequence dependencies in test with annotation Order improvement #995
• Support Java 8, 11 and 17 for Gravitino #1098
• Add multiple MySQL and PostgreSQL version support in integration test #1367 #1361
• Upgrade Jersey version to 2.41 #1756
• Upgrade Guava version 32.1.3-jre #1755
• Add test framework for web UI #1694
• Support query optimization in Trino connector #1491
• Pass the Trino-connector attributes when creating a Graviton catalog #1688

Fixed known CVEs

• Upgrade Jersey version to 2.41 #1756
• Upgrade Guava version 32.1.3-jre #1755

Know issues

• Varchar type of PG catalog is not handled correctly by Gravitino #2035
• OAuth mode, authentication will expire after a few minutes #2099
• Gravitino web UI has issues on Safari #2100

Limitation:

• Audit information is empty for MySQL schema #1767

Credits

@abant07 @amananand20 @austin362667 @brandboat @BWbwchen @ch3yne @Clearvive @coolderli @cottage14 @cxzl25 @diqiu50 @FANNG1 @FrankYang0529 @hiirrxnn @jerryshao @justinmclean @mchades @NIRMALraki @pan3793 @qqqttt123 @SophieTech88 @SteNicholas @TEOTEO520 @xnge @xunliu @yuqi1129 @YxAc @724thomas

Gravitino 0.3.1

Gravitino 0.3.1 is a maintenance release on branch-0.3, which fixes a bunch of UI, documentation and JDBC catalog issues.

For the details, please see the detailed change list below.

What's Changed

• [Minor] Minor formating and changes to fit with the style guide. by @github-actions in #1127
• [MINOR] fix(docs): Fix the wrong link in the README.md by @github-actions in #1130
• [MINOR] docs(lakehouse-iceberg): improvement lakehouse iceberg document by @github-actions in #1138
• [#1136] improvement(docs): Add the md suffix for every link in the markdown by @github-actions in #1140
• [Minor] improvement(docs): Optimized Trino related docs. by @github-actions in #1144
• [MINOR] fix(docs): Fix link of REST API page by @github-actions in #1148
• [MINOR] fix(docs): fix ./gradlew spotlessApply of Coding standards by @SteNicholas in #1153
• [Minor] improvement(docs): Optimize Trino related docs and fix some minor mistakes in docs by @github-actions in #1159
• [#1166] fix(docs): Fix typos graviton and syntax problems in docs by @github-actions in #1176
• [MINOR] fix(docs): Fix a typo by @github-actions in #1192
• [#1154] fix(trino-connector): Fix the issue with joins causing errors in PostgreSQL. by @github-actions in #1186
• [#1047] improvement(core): Fix the issue of StringIdentifier in metalake property by @github-actions in #1188
• [#1171] bugfix(postgresql): Fix bug where the same table name can be loaded under multiple schemas. by @github-actions in #1195
• [MINOR] fix(docs): Modify the required value for the document security.md by @github-actions in #1223
• [MINOR] fix(doc): Fix a JSON format error in doc manage-metadata-using-gravitino.md by @github-actions in #1229
• [#1112] fix(jdbc): deleteColumn throws exception for table with ifExisting = true. by @github-actions in #1230
• [#879][#925] Docker network Pool overlaps with other one on this address space by @github-actions in #1234
• [Minor] docs: Add more details about dropping schema and tables by @github-actions in #1240
• [MINOR] fix(docs): Modify the document according to the README of playground by @github-actions in #1250
• [#1231] Fix bugs in drop MySQL schemas by @github-actions in #1251
• [#1253] bugfix(jdbc): fix jdbc catalog store empty audit info issue by @github-actions in #1261
• [Minor] Improve English, use better names and data, simplify SQL by @github-actions in #1272
• [#1469] improvement(doc): Add doc about webUI operation by @github-actions in #1281
• [MINOR] improve(doc): Improve the index to reorganize the order of the docs by @github-actions in #1288
• Happy new year! by @github-actions in #1289
• [#1135] improvement(docs): Add docs about tables advanced feature like partitioning by @github-actions in #1296
• [#1299] Fix the copyright year by @github-actions in #1309
• [MINOR] doc: Improve English and add warning that web UI only works in Chrome by @github-actions in #1314
• [#1311] bugfix(docker): Unable to run datastrato/trino Docker in AWS by @github-actions in #1315
• Fix minor grammar errors and English mistakes in documentation. by @github-actions in #1320
• [MINOR] fix typo in URL by @github-actions in #1324
• [#1151] fix(UI): Fix status cache after modified a Metalake by @github-actions in #1325
• [MINOR] Add note on supported build systems by @github-actions in #1326
• [#1113][#1224] improvement(UI): TS support and improve global request handing by @github-actions in #1332
• [#1237] fix(trino): Fix dropping iceberg schema failure by @github-actions in #1333
• [#1150] fix(UI): Fix details comment error and improve components check by @github-actions in #1334
• [#1282] bugfix(CI) Automatically trigger CI for auto-cherry-pick workflow by @jerryshao in #1342
• [#1328] fix(trino-connector): Fix the failed 'drop catalog' test on Trino image gravitino-ci-trino:0.1.3. by @jerryshao in #1347
• [#1295] improvement(CI): separate trino image for CI and users by @jerryshao in #1344
• [#1267] fix: Add missing headers and license by @jerryshao in #1360
• [#1317] fix(trino-connector): Support Fixed type conversion by @jerryshao in #1376
• [#1001] fix(UI): Fix null requests in the web UI by @jerryshao in #1378
• [#1350] fix(UI): Fix fields validation when create a catalog in the web UI by @jerryshao in #1387
• [#1273] fix(UI): Fix cache values in the dialog when create a catalog by @jerryshao in #1382
• [#1391] fix(UI): Fix update metalake error in the web UI by @jerryshao in #1400
• [#1403] fix(UI): Remove unnecessary fields when creating a catalog in the web UI by @jerryshao in #1407
• [#1428] improvement(UI): Filtering empty keys when create or update by @jerryshao in #1430

Full Changelog: v0.3.0...v0.3.1

Gravitino 0.3.0

Release notes

Gravitino server

• Authorization support for Gravitino.
• Metrics system for Gravitino.
• Optimize and finalize the v1 storage layout.

Core metadata

• Add a unified type system.
• Add a unified expression system.

Catalog support

• Add new JDBC Catalog support (with MySQL and PostgreSQL support).

Trino connector

• Add Iceberg and JDBC connector support.
• Add catalog/schema/table level property support.
• Add full type system support.

Web UI

• Gravitino UI support.

Gravitino 0.2.0

This is the Gravitino 0.2.0 release. You can verify the release using the signature (.asc files) or hashes (.sha256 files).