Release Notes - 1.9.0

Highlight

• Fully verified with Java 8, 11, and 17, Scala 2.12 and 2.13
• Fully verified with Apache Spark 3.1 to 3.5, support for Spark 3.1 is deprecated and will be removed in the next release
• Fully verified with Apache Flink 1.16 to 1.19, support for Flink 1.16 is deprecated and will be removed in the next release
• Fully verified with Apache Hive 2.3, 3.1, and Cloudera Hive 2.1.1-cdh6, support running Hive engine on YARN (experimental)
• Spark engine supports save large query result to file before retrieving
• JDBC engine supports more dialects, including Doris, MySQL, PostgreSQL, Phoenix, StarRocks and Impala
• Kyuubis server metastore supports PostgreSQL as backend storage
• Introduce a basic SQL query editor in Kyuubi Web UI
• Continuously Spark AuthZ plugin improvements, including enhanced Iceberg support and new introduced Apache Hudi and Delta Lake support
• Continuously Kyuubi & Spark on Kubernetes improvements

Server

• [KYUUBI #5423] Support chaining SessionConfAdvisors
• [KYUUBI #5631][K8S][HELM] Set session affinity if needed in helm chart
• [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata
• [KYUUBI #5683] Add kyuubi.session.proxy.user as an alternative to hive.server2.proxy.user
• [KYUUBI #5704] Add unit tests for kyuubi.session.proxy.user in RESTful API
• [KYUUBI #5729] Use G1GC as Java option example in kyuubi-env template
• [KYUUBI #5797] Support to describe engine with command in current session
• [KYUUBI #5797][FOLLOWUP] Desc engine command support show engine registered attributes
• [KYUUBI #5873] Support to get operation progress with RESTful API
• [KYUUBI #5912] Cleanup unused config kyuubi.frontend.thrift.http.allow.user.substitution
• [KYUUBI #5921] EngineRef#deregister should return the value
• [KYUUBI #5934][K8S][HELM] Add Spark configuration support
• [KYUUBI #6003] Allow disabling user impersonation on launching engine
• [KYUUBI #6006][HELM] Support additional labels for service monitor
• [KYUUBI #6066] Fix PostgreSQL metastore migration script failure if indexes already exist
• [KYUUBI #6070] Improve perf on assembling row-based TRowSet
• [KYUUBI #6080] Improve instruction for deploy Kyuubi on K8s
• [KYUUBI #6081] RESTful client should retry on UnknownHostException
• [KYUUBI #6099] Remove support of Derby for Kyuubi metastore
• [KYUUBI #6126] Filter blank engine home env
• [KYUUBI #6140] Move getCodeSourceLocation to kyuubi-util

Spark Engine

• [KYUUBI #5377] Spark engine query result save to file
• [KYUUBI #5377][FOLLOWUP] Spark engine query result save to file
• [KYUUBI #5377][FOLLOWUP] Always try to cleanup session result path
• [KYUUBI #5377][FOLLOWUP] Get limit from more spark plan and regard result max rows
• [KYUUBI #5870] Directly mapping engine’s data type to Java type for TRowSet generation
• [KYUUBI #5877] Support Python magic syntax for notebook usage
• [KYUUBI #5877][FOLLOWUP] Add spark output mode to better support PySpark notebook
• [KYUUBI #5894] Separate closed and online sessions/statements in the SparkUI’s engine tab
• [KYUUBI #5899] Refine the python/scala interpreter error prompt
• [KYUUBI #6053][K8S] Show more details for pod/container status in application error
• [KYUUBI #6083] Bump Spark 3.3.3 to 3.3.4
• [KYUUBI #6091] Deprecate and remove building support for Spark 3.1
• [KYUUBI #6092] Enable DeltaTest for Spark 3.5
• [KYUUBI #6094] Bump Spark from 3.5.0 to 3.5.1
• [KYUUBI #6095] Enable PaimonTest for Spark 3.5
• [KYUUBI #6096] Enable PySpark tests for Spark 3.5
• [KYUUBI #6107][Spark] Collect and summarize the executorRunTime and executorCpuTime of the statement
• [KYUUBI #6107][FOLLOWUP] Add comments for the newly added parameters of the Spark event
• [KYUUBI #6108] Display the CPU time consumed by the statement in the Spark Engine tab
• [KYUUBI #6156] Remove flink. prefix for create session configurations
• [KYUUBI #6163] Set default Spark version to 3.5
• [KYUUBI #6166][MINOR] Fix wrong copy path about SPARK_HOME in docker
• [KYUUBI #6175][MINOR] Refactor copy path about SPARK_HOME in docker

Spark AuthZ Plugin

• [KYUUBI #5248][AUTHZ] Check previleges for Iceberg’s CALL to RewriteDataFilesProcedure procedure
• [KYUUBI #5255] Add an optional comment field to the authz specs for better recognition
• [KYUUBI #5284] Support Hudi Alter Table Command in Authz
• [KYUUBI #5323][AUTHZ] Drop Hive and Iceberg tables with PURGE option in tests
• [KYUUBI #5359][AUTHZ] Support Create Table Commands for Hudi
• [KYUUBI #5360] Support Hudi InsertIntoHoodieTableCommand
• [KYUUBI #5361][AUTHZ] Support Drop/Truncate Table Commands for Hudi
• [KYUUBI #5362] Remove Spark 3.0 support for Authz
• [KYUUBI #5399][AUTHZ] Cleanup Spark 3.0 specific implementation
• [KYUUBI #5407][AUTHZ] Tests for Iceberg system procedures of snapshot management
• [KYUUBI #5417] should not check in-subquery in permanent view
• [KYUUBI #5417] Authz should not check dependent subquery plan privilege
• [KYUUBI #5427][AUTHZ] Shade spark authz plugin
• [KYUUBI #5446][AUTHZ] Support Create/Drop/Show/Reresh index command for Hudi
• [KYUUBI #5447][AUTHZ] Support Hudi DeleteHoodieTableCommand/UpdateHoodieTableCommand/MergeIntoHoodieTableCommand
• [KYUUBI #5447][FOLLOWUP] Remove unrelated debug prints in TableIdentifierTableExtractor
• [KYUUBI #5452][AUTHZ] Support Compaction table commands for Hudi
• [KYUUBI #5455][AUTHZ] Support hudi CompactionHoodiePathCommand & CompactionShowHoodiePathCommand
• [KYUUBI #5457][AUTHZ] Support RepairTable Commands for Hudi
• [KYUUBI #5462][AUTHZ] Support create table command for Paimon
• [KYUUBI #5472] Permanent View should pass column when child plan no output
• [KYUUBI #5472][AUTHZ][FOLLOWUP] Check permanent view also need support merge projection
• [KYUUBI #5475][FOLLOWUP] Authz check permanent view’s subquery should check view’s correct privilege
• [KYUUBI #5478][AUTHZ] Support Hudi ShowHoodieTablePartitionsCommand
• [KYUUBI #5479][AUTHZ] Support Hudi CallProcedureHoodieCommand for stored procedures
• [KYUUBI #5492][AUTHZ] saveAsTable create DataSource table miss db info
• [KYUUBI #5496][AUTHZ] Authz shaded include the transitive jackson libs
• [KYUUBI #5497][AuthZ] Simplify debug message for missing field/method in ReflectUtils
• [KYUUBI #5506][AUTHZ][MINOR] Reuse loaded table extractor in Hudi table extractors
• [KYUUBI #5512][AuthZ] Remove the non-existent query specs in Deletes and Updates
• [KYUUBI #5529][AUTHZ] Support create table command for Delta Lake
• [KYUUBI #5503][AUTHZ] Check plan auth checked should not set tag to all child nodes
• [KYUUBI #5503][FOLLOWUP][AUTHZ] Authz should skip inner plan that have been verified
• [KYUUBI #5529][AUTHZ][FOLLOWUP] Fix Create table ut for Delta Lake does not trigger a sql operation
• [KYUUBI #5529][AUTHZ][FOLLOWUP] Remove useless org.apache.spark.sql.delta.commands.CreateDeltaTableCommand
• [KYUUBI #5532][AUTHZ] Support Alter table commands of updating table schema for Delta Lake
• [KYUUBI #5533][AUTHZ] Support merge into table command for Delta Lake
• [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake
• [KYUUBI #5535][AUTHZ] Support vacuum table command for Delta Lake
• [KYUUBI #5537][AuthZ] Order Command Spec by Command Class [group, classname]
• [KYUUBI #5543][AUTHZ] Generalize command specs and check duplicated class names of command spec
• [KYUUBI #5546][AUTHZ] Reorgnize the package names for rules
• [KYUUBI #5555][AUTHZ] Extractor common authorization rule class
• [KYUUBI #5556][AUTHZ] Support Alter table commands of set table properties
• [KYUUBI #5557][AUTHZ] Refactor code about handle PermanentViewMarker
• [KYUUBI #5565][AUTHZ] Support Delete/Insert/Update table command for Delta Lake
• [KYUUBI #5575][AUTHZ] Support InsertIntoDataSourceDir/InsertIntoHiveDirCommand path check
• [KYUUBI #5575][KYUUBI #5611][AUTHZ] Check path privilege should use correct action type
• [KYUUBI #5576][Bug] Fix wrong code in test case of dir command
• [KYUUBI #5579][AUTHZ] Support LogicalRelation don’t have CatalogTable but have HadoopFsRelation
• [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node’s input
• [KYUUBI #5611][AUTHZ] Authz support path privilege for SaveIntoDataSourceCommand
• [KYUUBI #5628][AUTHZ] Support path privilege check for resource command
• [KYUUBI #5630][Authz] Support path check of LoadDataCommand
• [KYUUBI #5638][AUTHZ] URI related privileges shall be READ/WRITE
• [KYUUBI #5638][FOLLOWUP] Remove unnecessary actionType in uri desc
• [KYUUBI #5642][AUTHZ] Support path privilege of CreateDatabaseCommand/AlterDatabaseSetLocationCommand/CreateNamespace/SetNamespaceLocation
• [KYUUBI #5642][FOLLOWUP] Fix unit test
• [KYUUBI #5652][AUTHZ] CreateTable related command support path privilege
• [KYUUBI #5657][AUTHZ] Support path privilege of AlterTableSetLocationCommand/AlterTableAddPartitionCommand
• [KYUUBI #5664][AUTHZ] Support create path-based table for Delta Lake
• [KYUUBI #5677][AUTHZ] Typeof expression miss column information
• [KYUUBI #5684][AUTHZ] Support delete/update path-based table for Delta Lake
• [KYUUBI #5690][AUTHZ] Support insert into/overwrite path-based table for Delta Lake in Authz
• [KYUUBI #5692][Bug] Authz not skip explain command
• [KYUUBI #5707][AUTHZ] Support merge into path-based table for Delta Lake in Authz
• [KYUUBI #5715][MINOR] Replace with .isDefined and .isEmpty
• [KYUUBI #5726][AUTHZ] Support optimize path-based table for Delta Lake in Authz
• [KYUUBI #5735][AUTHZ] Support vacuum path-based table for Delta Lake
• [KYUUBI #5743][AUTHZ] Improve AccessControlException verification of RangerSparkExtensionSuite
• [KYUUBI #5768][AUTHZ] Authz internal place holder should skip privilege check
• [KYUUBI #5780][AUTHZ] Treating PermanentViewMarker as LeafNode make code simple and got correct privilege object
• [KYUUBI #5780][AUTHZ][FOLLOWUP] Format PermanentViewMarker tree string
• [KYUUBI #5793][AUTHZ][BUG] PVM with nested scalar-subquery should not check src table privilege
• [KYUUBI #5884] PVM should inherit MultiInstance and wrap with new exprId
• [KYUUBI #5903] PVM should override computeStats method
• [KYUUBI #5913][Bug] After resolve PVM should mark all nodes as checked
• [KYUUBI #5937] PVM cause cache table not work
• [KYUUBI #5964][BUG] Avoid check not fully optimized query for InsertIntoDataSourceDirCommand and InsertIntoDataSourceCommand
• [KYUUBI #5965][AUTHZ] Supports check hoodie procedures show_commits resource privileges
• [KYUUBI #5985][AUTHZ][MINOR] Remove incorrect getUri method
• [KYUUBI #5997][AUTHZ] Avoid unnecessary loop of RuleEliminateTypeOf
• [KYUUBI #6007] AuthZ should check hoodie procedures path resource privileges

Other Spark Plugins

• [KYUUBI #5580] Support generate and window operators for InferRebalanceAndSortOrders
• [KYUUBI #5786] Disable spark script transformation
• [KYUUBI #5816] Change spark rule class to object or case class
• [KYUUBI #6125] Drop Kyuubi extension for Spark 3.1
• [KYUUBI #6168] Check if forcedMaxOutputRows is negative

Flink Engine

• [KYUUBI #5264][FLINK] Force disable Flink’s session timeout
• [KYUUBI #5489] Adjust shuffle partitions dynamically
• [KYUUBI #5505][FLINK] Support HELP command
• [KYUUBI #6097][KYUUBI #6098] Upgrade Flink 1.16.3 and 1.18.1
• [KYUUBI #6130] Stop engine immediately after close session for CONNECTION level FlinkSQLEngine
• [KYUUBI #6134] Support Flink 1.19
• [KYUUBI #6142] Deprecate Flink 1.16
• [KYUUBI #6152] Remove useless variable
• [KYUUBI #6158][MINOR] Remove confusing variable
• [KYUUBI #6171] Drop building support for Flink 1.16
• [KYUUBI #6171][FOLLOWUP] Restore flink-1.16 profile to recover Flink IT

Hive Engine

• [KYUUBI #5830] Test Hive engine with CDH Hive 2.1.1-cdh6.3.2
• [KYUUBI #5865] Support get SQL keywords from Hive engine through API
• [KYUUBI #5867] HiveEngine support run on YARN mode
• [KYUUBI #5867][FOLLOWUP] Remove unused config
• [KYUUBI #5867][FOLLOWUP] Fix generated configuration docs and typo

Trino Engine

• [KYUUBI #5968] Support set authentication user for Trino engine
• [KYUUBI #6147][TRINO] Use the overlay conf as session conf

JDBC Engine

• [KYUUBI #5375] JDBC Engine supports PostgreSQL
• [KYUUBI #5382][JDBC] Duplication cleanup improvement in JdbcDialect and schema helpers
• [KYUUBI #5404] JDBC Engine supports StarRocks
• [KYUUBI #5464] JDBC Engine supports MySQL
• [KYUUBI #5509] Add Apache Impala JDBC engine dialect
• [KYUUBI #5582] JDBC Engine supports configurable default fetchSize
• [KYUUBI #5851] Generalize TRowSet generators
• [KYUUBI #5861] Generalize TRowSet generator for JDBC engine with dialects
• [KYUUBI #5862] Use TestContainerForAll for testing JDBC engine with testcontainers
• [KYUUBI #5906][JDBC] Rebase Doris dialect implementation
• [KYUUBI #6037] Fix the JDBC engine provider name

Other Engine

• [KYUUBI #5384][KSCH] Hive connector supports Spark 3.5
• [KYUUBI #5709] Avoid exposing client keystore and truststore passwords in the JDBC URL
• [KYUUBI #5783] Switch to kyuubi-relocated-hive-service-rpc
• [KYUUBI #5828][CHAT] Kyuubi chat engine supports ernie bot(文心一言)
• [KYUUBI #6078] KSHC should handle the commit of the partitioned table as dynamic partition at write path
• [KYUUBI #6114] Properly handle KyuubiStatement#getMoreResults(CLOSE_CURRENT_RESULT)

Web UI

• [KYUUBI #5366][UI] Add submit SQL page in Kyuubi Web UI
• [KYUUBI #5517][UI] Initial implement the SQL Lab page
• [KYUUBI #5958][WEBUI] Bump follow-redirects from 1.15.1 to 1.15.4
• [KYUUBI #6002] Bump vite from 4.3.9 to 4.5.2 in web UI

Documentation

• [KYUUBI #5265][DOCS][FLINK] Add notice about Jobmanager HA to Flink deployment docs
• [KYUUBI #5500] Add Kyuubi Code Program to Doc
• [KYUUBI #5688][DOC] Update supported spark 3.5.x in authz doc
• [KYUUBI #5762][DOC] Introduce Kyuubi Web UI Engine Page
• [KYUUBI #5893][DOC] Update logos in layer picture on welcome page
• [KYUUBI #5905][DOC] Workaround for long inlined code sample in config page
• [KYUUBI #6010] Improve Gluten usage documentation
• [KYUUBI #6119][DOC] Update doc for HA/Zookeeper Configuration

Building, Infrastructure and Dependency

• [KYUUBI #4453][FOLLOW] Delete LICENSE for removed kubernetes client dependencies
• [KYUUBI #5104][TEST] BatchesResourceSuite - expose the metrics with operation type and current state
• [KYUUBI #5241] Bump Commons Lang3 from 3.12.0 to 3.13.0
• [KYUUBI #5245][INFRA] Daily deploy snapshot version on branch-1.8
• [KYUUBI #5266] Upgrade playground to Kyuubi 1.7.1
• [KYUUBI #5322][INFRA] Enable GitHub Projects
• [KYUUBI #5324] Bump latest 1.7.3 in playground and helm charts
• [KYUUBI #5342] Add label hacktoberfest to project
• [KYUUBI #5356][INFRA] GitHub Issue template for 2023 Kyuubi Code Contribution Program
• [KYUUBI #5395] Bump netty from 4.1.93.Final to 4.1.100.Final
• [KYUUBI #5449] Bump Delta Lake 3.0.0
• [KYUUBI #5501] Update codecov token and fix codecov reporting on PRs
• [KYUUBI #5509][FOLLOWUP] JDBC IT should always depends on kyuubi-hive-jdbc-shaded
• [KYUUBI #5600] Fix flaky test SessionsResourceSuite
• [KYUUBI #5602] Improve issue template for 2023 Kyuubi Code Contribution Program
• [KYUUBI #5660][TEST] Ensure SPI provider file content sorted in alphabet order
• [KYUUBI #5671] Bump axios from 0.27.2 to 1.6.0 in /kyuubi-server/web-ui
• [KYUUBI #5674][LICENSE][FOLLOWUP] Update license files
• [KYUUBI #5676] Bump Apache parent pom from 30 to 31
• [KYUUBI #5701] Enhance the PR Template to make PR desc informative
• [KYUUBI #5716][TEST] Fix match may not be exhaustive in Scala 2.13
• [KYUUBI #5774][CI] Bump actions/checkout from v3 to v4
• [KYUUBI #5779][CI] Upgrade Github actions for Docker
• [KYUUBI #5800][KYUUBI#5467] Integrate Intel Gluten with Spark engine
• [KYUUBI #5835] Remove issue template for 2023 Kyuubi Code Contribution Task
• [KYUUBI #5838][BUILD] Bump actions/setup-python from v4 to v5
• [KYUUBI #5839][BUILD] Bump actions/setup-java from v3 to v4
• [KYUUBI #5841][BUILD] Bump actions/stale from v7 to v9
• [KYUUBI #5842][BUILD] Bump actions/setup-node from v3 to v4
• [KYUUBI #5874][INFRA] Bump github labeler action to v5
• [KYUUBI #5879] Bump dropwizard metrics from 4.2.8 to 4.2.23
• [KYUUBI #5902] Bump doc build dependencies
• [KYUUBI #5914] Update layer diagram on welcome page
• [KYUUBI #5925] Kyuubi TPC-DS support running benchmark with skipping some queries
• [KYUUBI #5939] Bump Gluten version to recover Gluten IT
• [KYUUBI #5941] Drop Kubernetes Block Cleaner Tool from Kyuubi
• [KYUUBI #5944] Introduce Prometheus and Grafana for Kyuubi Playground
• [KYUUBI #5955][TEST][GLUTEN] Disable gluten ui when running integration test
• [KYUUBI #5973] Bump gRPC from 1.53.0 to 1.60.1
• [KYUUBI #5974] Bump jetcd 0.7.7
• [KYUUBI #5975] Bump trino-client 411
• [KYUUBI #5975][FOLLOWUP][TEST] Fix trino - get type info
• [KYUUBI #6043][TEST][GLUTEN] Gluten-it gluten package add arch suffix
• [KYUUBI #6061] Delete svn staging dir before checking out
• [KYUUBI #6063] Bump iceberg 1.4.3
• [KYUUBI #6067] Bump org.postgresql:postgresql from 42.6.0 to 42.7.2
• [KYUUBI #6068] Remove community section from user docs
• [KYUUBI #6069][TEST] TFrontendServiceSuite - close expired operations
• [KYUUBI #6071] Add .java-version into git ignore
• [KYUUBI #6074] Add a script to simplify the process of creating release notes
• [KYUUBI #6075] Update known_translations
• [KYUUBI #6076] Update Helm Chart and playground to use 1.8.1
• [KYUUBI #6085] Bump gRPC from 1.60.1 to 1.60.2
• [KYUUBI #6089] Bump Paimon from 0.5.0-incubating to 0.7.0-incubating
• [KYUUBI #6090] Bump Delta from 3.0.0 to 3.1.0 for Spark 3.5
• [KYUUBI #6101] Bump Jackson from 2.15.0 to 2.15.4
• [KYUUBI #6109] Fork beeline module from Apache Hive 3.1.3
• [KYUUBI #6117] Bump Kyuubi Shaded 0.3.0
• [KYUUBI #6118] Migrate to shaded HMS client for getting delegation token on server
• [KYUUBI #6129][BUILD] Remove the extension spark jars before build to avoid include unspecified version
• [KYUUBI #6131] Simplify Maven dependency management after dropping building support for Spark 3.1
• [KYUUBI #6139] Expose env vars and sys props to JUnit tests too
• [KYUUBI #6143] Add Dockerfile.HMS for testing
• [KYUUBI #6162] Cut out hive-common deps in beeline module
• [KYUUBI #6164] Limit max parallel for each GitHub Actions workflow
• [KYUUBI #6145] Use HiveMetaStore Docker for testing
• [KYUUBI #6145][FOLLOWUP] Remove unused dependency commons-lang
• [KYUUBI #6151] Remove invalid matrix spark 3.1 in GA
• [KYUUBI #6153] Cut out Hive JDBC and Hive Service deps in beeline module
• [KYUUBI #6160] Fix beeline test KyuubiBeeLineTest.testKyuubiBeelineComment
• [KYUUBI #6167] Drop JDK 11 on GitHub Actions
• [KYUUBI #6170] Merge JDBC Trino TPC jobs and Style Check jobs in GitHub Actions workflow
• [KYUUBI #6173] Bump maven-download-plugin from 1.7.1 to 1.8.1
• [KYUUBI #6174] Remove unused dependency management from POM
• [KYUUBI #6177] Bump BouncyCastle from 1.67 to 1.77
• [KYUUBI #6178] Improve Gluten integration test

Credits

Last but not least, this release would not have been possible without the following contributors:

• GitHub Bot
• Shaoyun Chen
• GGHWS
• Deepali Gupta
• Baokun Han
• Denis Krivenko
• Kwafoor
• Pengqi Li
• Bowen Liang
• Paul Lin
• Senmiao Liu
• Chenliang Lu
• Tigran Manasyan
• Hai Nguyen
• Cheng Pan
• XH Qian
• Zhen Wang
• Fei Wang
• Kang Wang
• Junjie Xu
• Binjie Yang
• Kent Yao
• Xianxun Ye
• Kaifei Yi
• David Yuan
• He Zhao
• Yangming Zhao
• Yifan Zhou
• Mingliang Zhu
• Yi Zhu
• Yaogai Zhu
• huangxiaopingRD
• kevinjmh
• mrtisttt
• BingKun Pan
• rhh777
• silencily
• sincewhen
• Vino Yang