Release Notes - 1.7.1

Highlight

Kyuubi requires Spark 3.1 and above
Upgrade Hadoop from 3.3.4 to 3.3.5
A series of improvements for Arrow based serialization
A series of improvements for batch job of Spark on K8s/YARN
Bug fix for ‘beeline -e’ with SQLs before ‘source’ statement
Spark Authz plugin fixes privilege checks on source tables of the permanent view in Spark 3.1

Server

[KYUUBI #4652][FOLLOWUP] Fix JaasConfiguration ClassNotFoundException for Hadoop 3.3.4 and previous
[KYUUBI #3887][FOLLOWUP] Fix kyuubiServerPrincipal logic in KyuubiCommands
[KYUUBI #4763][DOCS] Fix the Kyuubi JDBC kerberos parameters
[KYUUBI #4738] AdminResource.getEngineSpace should use primary group name on GROUP share level
[KYUUBI #4655][DOCS][FOLLOWUP] Recover the missing JDBC URL
[KYUUBI #4753][KYUUBI 4752][Improvement] KyuubiConf.unset should not log deprecation warning
[KYUUBI #4407][FOLLOWUP] Adapt SLF4J2
[KYUUBI #4746] Do not recreate async request executor if has been shutdown
[KYUUBI #4739] Add operation lock instead of locking state Enumeration
[KYUUBI #4719] Support submission timeout for yarn application manager and get the applicationInfo in-memory
[KYUUBI #4731] Support batch session conf advisor
[KYUUBI #4734][Docs] Fix typo in docs of custom event handler
[KYUUBI #4711] JDBC client should catch task failed exception instead of NPE in the incremental mode
[KYUUBI #4688] Fix the failure to read the operation log after executing catalog and database operation
[KYUUBI #4652] Upgrade Hadoop from 3.3.4 to 3.3.5
[KYUUBI #4691][REST] Configure FAIL_ON_UNKNOWN_PROPERTIES to false for KyuubiScalaObjectMapper
[KYUUBI #4658][FOLLOWUP] Improve unit tests
[KYUUBI #4669] LDAP authentication allows auth user contains domain when bind.dn/pw enabled
[KYUUBI #4653][KYUUBI 4650][Improvement] LogDivertAppender supports reading RollingFileAppender pattern
[KYUUBI #4623][FOLLOWUP] Fix malformed docs
[KYUUBI #4522] use:catalog should execute before than use:database
[KYUUBI #4655][DOCS] Enrich docs for Kyuubi Hive JDBC driver
[KYUUBI #4644] Manually terminate the Py4JServer during engine shutdown
[KYUUBI #4623][FOLLOWUP] Remove unused import
[KYUUBI #4623][K8S] KubernetesApplicationOperation uses Informer instead of list
[KYUUBI #3872][BATCH] Skip setting proxy user when keytab is provided
[KYUUBI #4639] Support to specify confOverlay when executing statement with RESTful API
[KYUUBI #4627][Docs] Fix a typo in rest_api.md.
[KYUUBI #4626][UI][CI] Fix Web UI Ci check fail with pnpm the lockfile needs updates
[KYUUBI #4624][Docs] Fix table headers in kyuubi_vs_hive.md
[KYUUBI #4620][KSHC] Cut off transitive dependencies
[KYUUBI #4599][DOCS] Simplify project version evaluation
[KYUUBI #4480] Engine alive probe should close thrift connection on engine lost
[KYUUBI #4593][DOCS] Keep promote the release step update-to-date
[KYUUBI #4579] Fix the unstable situation of test cases for Trino-Fronted-Service
[KYUUBI #4584] Post the session exception to EventBus in time
[KYUUBI #4563][Improvement] Format error log output in case of metrics json file not found
[KYUUBI #3739][FOLLOWUP][REST] Remove unused parameters in SessionsResource rest docs (#4542)
[KYUUBI #4581][TEST] Add KyuubiCommands parse python snippets unit test
[KYUUBI #4575] Fix the empty last line may causes the session to exit directly
[KYUUBI #4568] Make kyuubi-ctl doc enable variable ‘release’ automatic substitution
[KYUUBI #4526][FOLLOWUP] Defer evaluation for GroupProvider
[KYUUBI #4519] Update metadata when batch application state changed
[KYUUBI #4467][K8S][BATCH] Tolerate Driver Pod ephemerally invisible after submitting
[KYUUBI #4496] Call super.clusterManager() when batchConf does not have spark.master
[KYUUBI #4494] bin/kyuubi should use exec to run Kyuubi server
[KYUUBI #4491] Fix Trino typo
[KYUUBI #4489][REST] Fix missing org.apache.commons.logging by including jcl-over-slf4j dependency in REST client
[KYUUBI #4488][KSHC] Keep object original name defined in HiveBridgeHelper
[KYUUBI #4471] Prompted command for docs generation should skip Java tests
[KYUUBI #4482] Support to disable batch resource upload function
[KYUUBI #4452] Strip the redundant leading and tailing slash of getZooKeeperNamespace.
[KYUUBI #4451] Skip etcd test if no docker env
[KYUUBI #4433] Fix the case with create a view from view
[KYUUBI #4464] Simplify and improve log for JDBCMetadataStore
[KYUUBI #4462] Fix variable usage issue in SessionManager#stop
[KYUUBI #4443] Do not set engine session init sql for alive probe session

Spark Engine

[KYUUBI #4754][ARROW] Use KyuubiArrowConveters#toBatchIterator instead of ArrowConveters#toBatchIterator
[KYUUBI #4749] Fix flaky test issues in SchedulerPoolSuite
[KYUUBI #4722][DOCS] Kyuubi requires Spark 3.1 and above now
[KYUUBI #4720][ARROW] Fix java.lang.NoSuchFieldError: IpcOption.DEFAULT for Spark-3.1/3.2
[KYUUBI #4713][TEST] Fix false positive result in SchedulerPoolSuite (#4714)
[KYUUBI #4713][TEST] Fix false positive result in SchedulerPoolSuite
[KYUUBI #4710][ARROW] LocalTableScanExec should not trigger job
[KYUUBI #4702][ARROW] CommandResultExec should not trigger job
[KYUUBI #4662][ARROW] Arrow serialization should not introduce extra shuffle for outermost limit
[KYUUBI #4587][SPARK][EXT] Take care of table cache query stage

Spark Authz Plugin

[KYUUBI #4585] Authz policy file generation
[KYUUBI #4658][Authz][Bug] Fix InsertIntoHiveDirCommand classname so that we can extract the query in it when authorization.
[KYUUBI #4617][AUTHZ] Collect results for filtered show objects ahead to prevent holding unserializable spark plan
[KYUUBI #4504][Authz][Bug] Fix source table privilege requirement when querying permanent view in Spark 3.1 and below
[KYUUBI #4532][AUTHZ] Displays the columns involved in extracting the aggregation operator
[KYUUBI #4450] Ignore unknown fields policyPriority when reading policy json file

Other Spark Extensions

[KYUUBI #4727][DOC] kyuubi-spark-lineage has no transitive deps
[KYUUBI #4586][TEST][1.7] Fix to clean up views in lineage unit tests
[KYUUBI #4525][KSHC] Partitioning predicates should take effect to filter data
[KYUUBI #4331][KYUUBI #4431] Lineage supports Union and MergeInto

Flink Engine

[KYUUBI #4666] Support Flink varbinary type in query operation
[KYUUBI #4635] Support Flink time type in query operation
[KYUUBI #4614][DOCS] Rename Flink Table Store to Apache Paimon (Incubating) in docs Connectors for Flink SQL Query Engine
[KYUUBI #4607][DOCS] Rename Flink Table Store to Apache Paimon (Incubating) in docs Connectors for Hive SQL Query Engine
[KYUUBI #4608][DOCS] Rename Flink Table Store to Apache Paimon (Incubating) in docs Connectors for Trino SQL Query Engine
[KYUUBI #4598] Bump Flink-1.15 from 1.15.3 to 1.15.4
[KYUUBI #4583][DOCS] Rename Flink Table Store to Apache Paimon (Incubating)
[KYUUBI #4517][FLINK] Fix multiple executions lead to abnormal results on Flink 1.1
[KYUUBI #4446] Fix connections blocked by Flink insert statements

Client

[KYUUBI #4619] Fix beeline with -e When there are other SQL statements before the source statement, the source statement cannot be executed normally
[KYUUBI #4508][BEELINE] Beeline should reset stdin after consuming init SQL file
[KYUUBI #4492] Correct engine subdomain calculation in kyuubi-ctl
[KYUUBI #4479] Restore JDBC Kerberos authentication behavior for UGI.doAs
[KYUUBI #4438] Render the batch info map in kyuubi-ctl

Deployment

[KYUUBI #4706][K8S][HELM] Use template comments for the chart license header
[KYUUBI #4707][K8S][HELM] Fix README typo and minor reformatting
[KYUUBI #4606][K8S][HELM] Add command and args configuration support
[KYUUBI #4605][K8S][HELM] Set IfNotPresent pullPolicy by default
[KYUUBI #4589][K8S][HELM] Update template on port-foward usage guide NOTES.txt
[KYUUBI #4557][K8S][HELM] Kyuubi server should bind Pod IP by default
[KYUUBI #4484][K8S][HELM] Set specific Kyuubi version to the chart appVersion

Credits

Last but not least, this release would not have been possible without the following contributors:

Alex
Bowen Liang
Chao Chen
Cheng Pan
Derick Cypho
Fei Wang
Fu Chen
Karsonnel
Paul Lin
Ruguo Yu
Tianlin Liao
Xieming LI
Yikf
dnskr
fwang12
guanhua.lgh
huangzhir
maruilei
miaowang
minyk
odone
pengqli
phyyou
remzi
senmiaoliu
sychen
ulysses-you
wangyepeng2
xu.guo
zwangsheng

2023 May 5