Release Notes - 1.9.1

Highlights

• Experimental support for Spark 4.0
• Web UI supports Basic Authentication

Features and Improvements

• [KYUUBI #6424] TPC-H/DS connector support Spark 4.0
• [KYUUBI #6413] SPARK_HOME detection supports Spark 4
• [KYUUBI #6404] HiveResult.toHiveString compatible to Spark 4.0
• [KYUUBI #6405] Spark engine supports both javax and jakarta ws.rs namespaces
• [KYUUBI #6399] Spark engine supports both javax and jakarta servlet namespaces
• [KYUUBI #6392] Support javax.servlet and jakarta.servlet co-exist
• [KYUUBI #6297] Package Spark SQL engine both Scala 2.12 and 2.13
• [KYUUBI #4847] Exclude the alive probe sessions in terminating checker
• [KYUUBI #6172] Support to interrupt the thrift request immediately after marking the engine not alive
• [KYUUBI #6315] Spark 3.5: MaxScanStrategy supports DSv2
• [KYUUBI #6049] Support to filter sessions/operations with session type
• [KYUUBI #6366] Improve log message for YARN getApplicationInfoByTag
• [KYUUBI #6298] Make ServiceMonitor use selector labels in Helm charts
• [KYUUBI #6251] Improve kyuubi-beeline help message
• [KYUUBI #6290] Add custom exception serialization for SparkOperationEvent
• [KYUUBI #6079] Web UI supports Basic authN
• [KYUUBI #6338] Support connecting Kyuubi using Hive JDBC driver 4.0
• [KYUUBI #6248] Web SQL Editor supports run SQL snippets
• [KYUUBI #6216] Support to deny some client ips to make connection
• [KYUUBI #6283] Improve the message of distribution built without enabling web ui
• [KYUUBI #6207] Support to retrieve Spark UserDefinedType result
• [KYUUBI #6205] Backport SPARK-47300: quoteIfNeeded should quote identifier starts with digits
• [KYUUBI #6215] Improve DropIgnoreNonexistent rule for Spark 3.5
• [KYUUBI #6211] Check memory offHeap enabled for CustomResourceProfileExec

Bug Fixes

• [KYUUBI #6430] livenessProbe uses absolute path based on KYUUBI_HOME
• [KYUUBI #6428] kyuubi-server/web-ui should not to set a timeout for axios requests
• [KYUUBI #6422] Merge standard error to output
• [KYUUBI #6396] Add caching for KerberosAuthentication using ticketCache key
• [KYUUBI #6410] Check if asyncRequestExecutor is initialized before shutdown
• [KYUUBI #6394] Fix example SQL in Spark TPC-DS connector docs
• [KYUUBI #6400] Fix memory leak when using saveToFile
• [KYUUBI #6344] FlinkProcessBuilder prioritizes user configurations
• [KYUUBI #6346] Change Kyuubi JDBC Driver default port and make it effective
• [KYUUBI #6212] Added audit handler shutdown to the shutdown hook
• [KYUUBI #6370] For saveToFile, checking idx to prevent IndexOutOfBoundsException
• [KYUUBI #6320] Fix terminated application pods not deleted issue when kyuubi server restarted
• [KYUUBI #6291] Avoid NPE in MySQLErrPacket
• [KYUUBI #5767] Fix spark batch conf not convert issue
• [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue
• [KYUUBI #6287] Avoid missing match
• [KYUUBI #6183] KyuubiConf.getKubernetesConf should set namespace even if the Kubernetes context is empty
• [KYUUBI #6204] Fix kyuubi session limiter leak when opening session failed
• [KYUUBI #6182] JDBC engine using the overlay conf as session conf
• [KYUUBI #6221] Fix parameter replacement issue caused by incorrect sql split
• [KYUUBI #6228] Dropdown Selection and Parameter Parsing Issues in Kyuubi SQL Editor

Build, Test and Misc

• [KYUUBI #6432] Mention support for Spark 4.0 in docs
• [KYUUBI #6423] Fix Scalatest exclusion tags
• [KYUUBI #6419] Bump Jetty 9.4.54.v20240208
• [KYUUBI #6418] Bump Jersey to 2.40
• [KYUUBI #6416] Generate flattened POM
• [KYUUBI #6415] Manage Jersey in kyuubi-server module
• [KYUUBI #6417] Remove scala compile args -Yno-adapted-args
• [KYUUBI #6382] Bump Delta from 3.1.0 to 3.2.0 for Spark 3.5
• [KYUUBI #6348] Upgrade iceberg from 1.5.0 to 1.5.2
• [KYUUBI #6328] Bump Hive 2.3.10
• [KYUUBI #6373] Avoid NPE in get operation log api
• [KYUUBI #6377] Fix isCommand check and set min rows threshold for saveToFile
• [KYUUBI #6357] Bump org.bouncycastle:bcprov-jdk18on from 1.77 to 1.78
• [KYUUBI #6363] Increase CI concurrency
• [KYUUBI #6244] Fix test logs upload error
• [KYUUBI #6294] Simplify Netty and gRPC dependency management
• [KYUUBI #6252] Upgrade hive-service-rpc 4.0.0
• [KYUUBI #6268] Specify logDir for RollingFile filePattern
• [KYUUBI #6267] Remove unused dependency management in POM
• [KYUUBI #6271] Upgrade kafka-clients from 3.5.1 to 3.5.2
• [KYUUBI #6260] Clean up and improve comments for spark extensions
• [KYUUBI #6257] Bump vite from 4.5.2 to 4.5.3 in /kyuubi-server/web-ui
• [KYUUBI #6243] Add missing Impala alias for JDBC engine
• [KYUUBI #6234] Fix invalid code-block syntax in docs
• [KYUUBI #2782] Add missed UNION_TYPE back
• [KYUUBI #6206] Update comments for job group id of Spark broadcast jobs
• [KYUUBI #6197] Revise dependency management of Spark authZ plugin
• [KYUUBI #6196] Add log4j2 template for kyuubi repl commands
• [KYUUBI #6194] AuthZ shaded should include ranger-plugins-cred
• [KYUUBI #6188] Bump follow-redirects from 1.15.4 to 1.15.6 in web-ui
• [KYUUBI #6191] Update docs to mention support of Flink 1.19
• [KYUUBI #6134] Add profile and CI for Flink 1.19
• [KYUUBI #6193] Add known_transkations
• [KYUUBI #6179] Bump iceberg from 1.4.3 to 1.5.0

Credits

Last but not least, this release would not have been possible without the following contributors:

• Lennon Chin
• JiaLiangC
• Senmiao Liu
• Xiao Liu
• Cheng Pan
• PorterZhang2021
• TakawaAkirayo
• Zhen Wang
• Fei Wang
• Binjie Yang
• He Zhao
• Mingliang Zhu
• amanraj2520
• beryllw
• camper42
• dupen01
• lllong33
• mrpengbei
• sudohainguyen
• zRains