Releases Archive

    Release Notes - 1.10.1

    2024 Dec 26

    Changelogs

    • [KYUUBI #6840] Fix PodMonitor pods selection
    • [KYUUBI #6829] Add metrics for batch pending max elapse time
    • [KYUUBI #6836] Ship kafka-clients in binary distribution tarball without compression libs
    • [KYUUBI #6838] [INFRA] Update archive mailing list address
    • [KYUUBI #6837] [INFRA] Disable K8s CI image GHA cache
    • [KYUUBI #6828] Clean up and improve error message for KyuubiBaseResultSet
    • Revert “[KYUUBI #6639] Port HIVE-27815: Support update numModifiedRows”
    • [KYUUBI #6722] Fix AppState when Engine connection is terminated
    • [KYUUBI #6820] Explicitly disable attach-scaladocs for pure Java modules
    • [KYUUBI #6813] [BUILD] Replace java.version 1.8 with 8
    • [KYUUBI #6773] Bump commons-codec from 1.15 to 1.17.1
    • [KYUUBI #6808] [BUILD]Replace systemProperties with systemPropertyVariables in maven-surefire-plugin
    • [KYUUBI #6805] [K8S][HELM] Update default Kyuubi version to 1.10.0
    • [KYUUBI #6778] [BUILD] allow frontend plugin not to inherits Maven’s http proxy config
    • [KYUUBI #6786] Skip repeated checks on convert function in TColumnGenerator
    • [KYUUBI #6790] Fix engine cannot exit when gracefully stopped

    Credits

    Last but not least, this release would not have been possible without the following contributors:

    • Bowen Liang
    • pionCham
    • Denis Krivenko
    • Fei Wang
    • Zhen Wang
    • Cheng Pan
    • Joao Amaral
    • zhifanggao

    Release Notes - 1.9.3

    2024 Dec 2

    Changelogs

    • [KYUUBI #6828] Clean up and improve error message for KyuubiBaseResultSet
    • [KYUUBI #6741] [CI] update GHA actions/upload-artifact from v3 to v4
    • [KYUUBI #6722] Fix AppState when Engine connection is terminated
    • [KYUUBI #6820] Explicitly disable attach-scaladocs for pure Java modules
    • [KYUUBI #6813] [BUILD] Replace java.version 1.8 with 8
    • [KYUUBI #6790] Fix engine cannot exit when gracefully stopped
    • [KYUUBI #6772] Fix ProcessBuilder to properly handle Java opts as a list
    • [KYUUBI #6757] Pass build args to extra spark engine build command
    • [KYUUBI #6750] [REST] Using ForbiddenException instead of NotAllowedException
    • [KYUUBI #6728] [DOC] update Authz plugin docs of build command with -am option
    • [KYUUBI #6688] [SPARK] Avoid trigger execution when getting result schema
    • [KYUUBI #6638] Authz shaded should include jsr311-api
    • [KYUUBI #6720] K8s pod OOM Killed should be identified as Application failed state
    • [KYUUBI #6709] Fix seq and set equals check bug while check auth type in AuthenticationFilter.initAuthHandlers
    • [KYUUBI #6708] spark-sql-engine’s classes should not be visible in server module
    • [KYUUBI #6699] Bump Spark 4.0.0-preview2
    • [KYUUBI #6648] Support to show launch engine log for thrift http mode
    • [KYUUBI #6698] Exclude transitive deps of paimon
    • [KYUUBI #6680] Format the kyuubi-ctl batch command output
    • [KYUUBI #6668] Fix kyuubi batch state abnormal
    • [KYUUBI #6661] Improve perf for column-based TRowSet generation
    • [KYUUBI #6658] [DOCS] Fixed typo in REST API docs
    • [KYUUBI #6594] Port HIVE-26633: Make thrift client maxMessageSize configurable
    • [KYUUBI #6638] Shade jsr311-api in Authz
    • [KYUUBI #6635] Fix Hive 2.3.10 integration tests
    • [KYUUBI #6583] Support to cancel Spark python operation
    • [KYUUBI #6627] Using concurrent collections for session operation handle set
    • [KYUUBI #6626] Fix operation never expired issue with periodical listOperations api calls
    • [KYUUBI #6620] [K8S][HELM] Update default Kyuubi version to 1.9.2
    • [KYUUBI #6615] Make Jetty sending server version in response configurable
    • [KYUUBI #6609] Bump Spark 3.5.2
    • [KYUUBI #6606] Restore shell compatibility for build/mvn
    • [KYUUBI #6606] build/mvn checks project local downloaded binary ahead
    • [KYUUBI #6600] Correct usage of maven.scaladoc.skip
    • [KYUUBI #6598] Flink engine module supports building with Scala 2.13
    • [KYUUBI #6599] Disable Javadoc/Scaladoc by default
    • [KYUUBI #6596] Fix typos in architecture page
    • [KYUUBI #6591] Support authorization on swagger UI
    • [KYUUBI #6592] Add uploadWorkDir into local dir allow list and do not skip check for resource uploaded use case
    • [KYUUBI #6588] Get effective USER if env var is unavailable
    • [KYUUBI #6571] Bump apache parent pom from 31 to 33
    • [KYUUBI #6586] Bump commons-io to 2.16.1
    • [KYUUBI #6564] Insert into table check the privilege of table
    • [KYUUBI #6549] Correctly handle empty Java options for engines
    • [KYUUBI #6579] Show kyuubi batch app details on app state change
    • [KYUUBI #6364] Bump scala-maven-plugin from 4.8.0 to 4.9.2
    • [KYUUBI #6566] Fix typos in KyuubiConf
    • [KYUUBI #6541] [AUTHZ] Fix DataSourceV2RelationTableExtractor can’t get the ‘database’ attribute if it’s a Paimon plan.
    • [KYUUBI #6550] Delete updateStrategy.rollingUpdate.maxUnavailable because it is alpha feature
    • [KYUUBI #6543] Bump Kyuubi Shaded 0.4.1

    Credits

    Last but not least, this release would not have been possible without the following contributors:

    • promising-forever
    • Cheng Pan
    • Bowen Liang
    • lifulong
    • be-hase
    • DenineLu
    • Baokun Han
    • hh-cn
    • LiJie20190102
    • Senmiao Liu
    • Zhen Wang
    • Joao Amaral
    • Fei Wang
    • Mike Wilson
    • Madhukar525722
    • liujiayi771
    • paul8263
    • pionCham

    Release Notes - 1.8.3

    2024 Nov 27

    Changelogs

    [KYUUBI #6417] Remove scala compile args -Yno-adapted-args [KYUUBI #6820] Explicitly disable attach-scaladocs for pure Java modules [KYUUBI #6790] Fix engine cannot exit when gracefully stopped [KYUUBI #6709] Fix seq and set equals check bug while check auth type in AuthenticationFilter.initAuthHandlers [KYUUBI #6708] spark-sql-engine’s classes should not be visible in server module [KYUUBI #6658] Fixed typo in REST API docs [KYUUBI #6606] Restore shell compatibility for build/mvn [KYUUBI #6606] build/mvn checks project local downloaded binary ahead [KYUUBI #6600] Correct usage of maven.scaladoc.skip [KYUUBI #6591] Support authorization on swagger UI [KYUUBI #6592] Add uploadWorkDir into local dir allow list and do not skip check for resource uploaded use case [KYUUBI #6588] Get effective USER if env var is unavailable [KYUUBI #6564] Insert into table check the privilege of table [KYUUBI #6549] Correctly handle empty Java options for engines [KYUUBI #6542] KyuubiBatchService should wait for HTTP server started before picking jobs [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf [KYUUBI #6466] Bump minikube and kubernetes version of integration test [KYUUBI #6463] Release semaphore immediately after startup process exit [KYUUBI #5957] Flink engine should not load kyuubi-defaults.conf [KYUUBI #6440] Fix casing in kubectl commands for serviceaccount creation [KYUUBI #6430] livenessProbe uses absolute path based on KYUUBI_HOME [KYUUBI #6422] Merge standard error to output [KYUUBI #6410] Check if asyncRequestExecutor is initialized before shutdown [KYUUBI #6394] Fix example SQL in Spark TPC-DS connector docs [KYUUBI #4847] Exclude the alive probe sessions in terminating checker [KYUUBI #6172] Support to interrupt the thrift request immediately after marking the engine not alive [KYUUBI #6373] Avoid NPE in get operation log api [KYUUBI #6366] Improve log message for YARN getApplicationInfoByTag

    Credits

    Last but not least, this release would not have been possible without the following contributors:

    • Fei Wang
    • Zhen Wang
    • ic4y
    • lifulong
    • pionCham
    • liujiayi771
    • LiJie20190102
    • beryllw
    • huangzhir
    • Fengyu Cao
    • lllong33
    • Cheng Pan
    • Fu Chen
    • Qingbo Jiao

    Release Notes - 1.10.0

    2024 Oct 27

    Highlight

    • Fully verified with Java 8, 11, and 17, on Scala 2.12 and 2.13
    • Fully verified with Apache Spark 3.2 to 3.5, initial support for 4.0-preview, support for Spark 3.2 is deprecated and will be removed in the next release
    • Fully verified with Apache Flink 1.17 to 1.20
    • Support creating batch job with uploading extra resources via REST API
    • Support server choosing strategy for HA mode
    • Introduce Spark JVM quake plugin
    • Periodically cleanup expired temp files and operation logs on server
    • Performance improvement for Ranger access requests deduplication in AuthZ plugin

    Server

    • [KYUUBI #6034] Kyuubi Server HA&ZK get server from serverHosts support more strategy
    • [KYUUBI #6756] [REST] Check max file size of uploaded resource and extra resources in batch creation
    • [KYUUBI #6750] [REST] Using ForbiddenException instead of NotAllowedException
    • [KYUUBI #6731] [REST] Check all required extra resource files uploaded in creating batch request
    • [KYUUBI #6720] K8s pod OOM Killed should be identified as Application failed state
    • [KYUUBI #6681][FOLLOWUP] Log the kill batch request before closeSession and during closeOperation
    • [KYUUBI #6772] Fix ProcessBuilder to properly handle Java opts as a list
    • [KYUUBI #6753] Set hadoop fs delegation token renewer to empty
    • [KYUUBI #6704] Disable periodic gc if set interval to 0
    • [KYUUBI #6680] Format the kyuubi-ctl batch command output
    • [KYUUBI #6668] Fix kyuubi batch state abnormal
    • [KYUUBI #6681] Log the delete batch request in batch operation log
    • [KYUUBI #6669] Track the client ip for internal kyuubi RESTful requests
    • [KYUUBI #6645] Size based eviction for server-side temp files cleanup
    • [KYUUBI #6661] Improve perf for column-based TRowSet generation
    • [KYUUBI #6652] Support to list batches in descending order
    • [KYUUBI #6648] Support to show launch engine log for thrift http mode
    • [KYUUBI #6646] Fix RESTful API NPE when restarting Kyuubi server
    • [KYUUBI #6587] Periodically expire temp files and operation logs on server to avoid memeory leak by Files.deleteOnExit
    • [KYUUBI #6335][FOLLOWUP] Using sessionId for sessionUploadFolderPath
    • [KYUUBI #6627] Using concurrent collections for session operation handle set
    • [KYUUBI #6626] Fix operation never expired issue with periodical listOperations api calls
    • [KYUUBI #6615] Make Jetty sending server version in response configurable
    • [KYUUBI #6585] Support Multiple Configuration Files in FileSessionConfAdvisor
    • [KYUUBI #6592] Add uploadWorkDir into local dir allow list and do not skip check for resource uploaded use case
    • [KYUUBI #6588] Get effective USER if env var is unavailable
    • [KYUUBI #6335] [REST] Support uploading extra resources in creating batch jobs via REST API
    • [KYUUBI #6582] [KYUUBI-6581] Zorder clause syntax does not support special characters
    • [KYUUBI #6554] Delete redundant code related to zorder
    • [KYUUBI #6551] Allow insert zorder when global sort is false and the plan is Repartition or RepartitionByExpression.
    • [KYUUBI #6542] KyuubiBatchService should wait for HTTP server started before picking jobs
    • [KYUUBI #6534] Fix checkEngineConnectionAlive may throw NPE
    • [KYUUBI #6524] Trino engine supports insecure configuration
    • [KYUUBI #6008] RESTful API supports killing engine forcibly
    • [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf
    • [KYUUBI #6500] Remove service loader descriptor for relocated HMS DelegationTokenIdentifier
    • [KYUUBI #6499] Rewrite some utility methods in Java
    • [KYUUBI #6469] Lazily initialize RecordReaderIterator to avoid driver oom when fetching big result set
    • [KYUUBI #6476] Fix incomplete app events deserialization in SHS
    • [KYUUBI #6302] Call cancelJobGroup immediately after statement execution finished
    • [KYUUBI #4847][FOLLOWUP] Fix engine session never idle issue
    • [KYUUBI #6422] Merge standard error to output
    • [KYUUBI #6423] Fix Scalatest exclusion tags
    • [KYUUBI #6396][FOLLOWUP] Avoid NPE
    • [KYUUBI #6415] Manage Jersey in kyuubi-server module
    • [KYUUBI #6410] Check if asyncRequestExecutor is initialized before shutdown
    • [KYUUBI #6408] Change to Base64 RFC4648 for handle guid and credential
    • [KYUUBI #6400] Fix memory leak when using saveToFile
    • [KYUUBI #6376] Improve closeEngine method names and logs due to closeSession
    • [KYUUBI #6379] Return engine name and engine url within KyuubiSessionEvent/SessionData
    • [KYUUBI #4847][FOLLOWUP] Exclude the alive probe sessions in terminating checker
    • [KYUUBI #6172][TASK][EASY] Support to interrupt the thrift request immediately after marking the engine not alive
    • [KYUUBI #6373] Avoid NPE in get operation log api
    • [KYUUBI #6359] [REST] Return more fields within session/batch data
    • [KYUUBI #6366] Improve log message for YARN getApplicationInfoByTag
    • [KYUUBI #6244][TEST] Fix test logs upload error
    • [KYUUBI #6322] Expose more config items of server internal rest client to users
    • [KYUUBI #6049] Support to filter sessions/operations with session type
    • [KYUUBI #6320] Fix terminated application pods not deleted issue when kyuubi server restarted
    • [KYUUBI #6278] Update DB state when the query fallback to resource manager and the batch app is terminal (#6284)
    • [KYUUBI #6291] Avoid NPE in MySQLErrPacket
    • [KYUUBI #6216] Support to deny some client ips to make connection
    • [KYUUBI #6199] Support to run HiveSQLEngine on kerberized YARN
    • [KYUUBI #6204] Fix kyuubi session limiter leak when opening session failed
    • [KYUUBI #6165] Add some HttpClient options for EngineUIProxyServlet

    Spark Engine

    • [KYUUBI #6688] [SPARK] Avoid trigger execution when getting result schema
    • [KYUUBI #6708] spark-sql-engine’s classes should not be visible in server module
    • [KYUUBI #6699] Bump Spark 4.0.0-preview2
    • [KYUUBI #6698] Exclude transitive deps of paimon
    • [KYUUBI #6686] Ignore Spark pod container state if pod is terminated
    • [KYUUBI #6653] Persist spark pod name conf into batch metadata
    • [KYUUBI #6583] Support to cancel Spark python operation
    • [KYUUBI #6609] Bump Spark 3.5.2
    • [KYUUBI #6601] Bump Delta Lake 4.0.0rc1 and enable Delta Lake test for Spark 4.0
    • [KYUUBI #6574] Skip eagerly execute command in PlanOnly mode of Spark Engine
    • [KYUUBI #6549] Correctly handle empty Java options for engines
    • [KYUUBI #6579] Show kyuubi batch app details on app state change
    • [KYUUBI #6545] Deprecate and remove building support for Spark 3.2
    • [KYUUBI #6546] Update incorrect descriptions in Zorder related configurations
    • [KYUUBI #6531] Fix SPARK-EngineTab stop/gracefulstop not work
    • [KYUUBI #6446] Add tests for Spark saveToFile function
    • [KYUUBI #6518] Support extracting URL for Spark 4 on YARN
    • [KYUUBI #6516] Fix KyuubiSparkUtil.buildURI
    • [KYUUBI #6489] [PYTHON] PyKyuubi get_table_names also supports Spark SQL dialect
    • [KYUUBI #6302][FOLLOWUP] Skip spark job group cancellation on incremental collect mode
    • [KYUUBI #6453] Make KSHC support Spark 4.0 and enable CI for Spark 4.0
    • [KYUUBI #6452] Add cross test for Spark 4.0
    • [KYUUBI #6447] Use static regex Pattern instances in JavaUtils.timeStringAs and JavaUtils.byteStringAs
    • [KYUUBI #6437] Fix Spark engine query result save to HDFS
    • [KYUUBI #6441] Kyuubi Spark TPC-DS/H Connector cross version test
    • [KYUUBI #6247] Make KSHC binary compatible with multiple Spark versions
    • [KYUUBI #6425] Fix tests in spark engine and kyuubi server modules with Spark 4.0
    • [KYUUBI #6424] TPC-H/DS connector support Spark 4.0
    • [KYUUBI #6413] SPARK_HOME detection supports Spark 4
    • [KYUUBI #6404] Fix HiveResult.toHiveString compatibility for Spark 4.0
    • [KYUUBI #6405] Spark engine supports both javax and jakarta ws.rs namespaces
    • [KYUUBI #6399] Spark Kyuubi UI supports both javax and jakarta servlet namespaces
    • Revert “[KYUUBI #6390] Temporarily disable UI Tab for Spark 4.0 and above”
    • [KYUUBI #6390] Temporarily disable UI Tab for Spark 4.0 and above
    • [KYUUBI #6305][FOLLOWUP] Improve package Spark SQL engine both Scala 2.12 and 2.13
    • [KYUUBI #6377] Fix isCommand check and set min rows threshold for saveToFile
    • [KYUUBI #6370] For saveToFile, checking idx to prevent IndexOutOfBoundsException
    • [KYUUBI #6323] Upgrade Spark 3.4.3
    • [KYUUBI #6321] Support to get Spark Kubernetes app URL
    • [KYUUBI #6315] Spark 3.5: MaxScanStrategy supports DSv2
    • [KYUUBI #6297] Package Spark SQL engine both Scala 2.12 and 2.13
    • [KYUUBI #6290] Add custom exception serialization for SparkOperationEvent
    • [KYUUBI #6287] Avoid missing matche
    • [KYUUBI #6250][FOLLOWUP] Simplify code after dropping Spark 3.1
    • [KYUUBI #6250] Drop support for Spark 3.1
    • [KYUUBI #6260] Clean up and improve comments for spark extensions
    • [KYUUBI #5767][FOLLOWUP] Fix spark batch conf not convert issue
    • [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue
    • [KYUUBI #6215] Improve DropIgnoreNonexistent rule for Spark 3.5
    • [KYUUBI #6211] Check memory offHeap enabled for CustomResourceProfileExec
    • [KYUUBI #6207] Support to retrieve Spark UserDefinedType result
    • [KYUUBI #6205] Backport SPARK-47300: quoteIfNeeded should quote identifier starts with digits
    • [KYUUBI #6206] Update comments for job group id of Spark broadcast jobs
    • [KYUUBI #6180] Kyuubi Spark nightly testing should use JDK 17
    • [KYUUBI #6180] Kyuubi Spark nightly testing should use JDK 17

    Spark AuthZ Plugin

    • [KYUUBI #6754][AUTHZ] Improve the performance of Ranger access requests deduplication
    • [KYUUBI #6638][FOLLOWUP] Authz shaded should include jsr311-api
    • [KYUUBI #6709] Fix seq and set equals check bug while check auth type in AuthenticationFilter.initAuthHandlers
    • [KYUUBI #6666][AUTHZ]Upgrade Ranger plugin to 2.5.0
    • [KYUUBI #5402] Introduce Spark JVM quake plugin
    • [KYUUBI #6619][FOLLOWUP] Do not log auth audit log for thrift http cookie auth
    • [KYUUBI #6640] [AUTHZ] Adapt Derby 10.16 new JDBC driver package name
    • [KYUUBI #6638] Shade jsr311-api in Authz
    • [KYUUBI #6619] Support http bearer authentication for ThriftHttp protocol
    • [KYUUBI #6623] Support bearer token authentication on swagger UI
    • [KYUUBI #6618] Support http bearer token authentication for REST protocol
    • [KYUUBI #6564] Insert into table check the privilege of table
    • [KYUUBI #6541] [AUTHZ] Fix DataSourceV2RelationTableExtractor can’t get the ‘database’ attribute if it’s a Paimon plan.
    • [KYUUBI #6197] Revise dependency management of Spark authZ plugin
    • [KYUUBI #6194] AuthZ shaded should include ranger-plugins-cred

    Other Spark Plugins

    • [KYUUBI #6398] Fix lineage plugin UT for Spark 4.0
    • [KYUUBI #6368] Flink engine supports user impersonation
    • [KYUUBI #6598] Flink engine module supports building with Scala 2.13
    • [KYUUBI #6557] Support Flink 1.20
    • [KYUUBI #6537] Upgrade Flink from 1.19.0 to 1.19.1
    • [KYUUBI #5957] Flink engine should not load kyuubi-defaults.conf
    • [KYUUBI #6353] Catch exception for closing flink internal session
    • [KYUUBI #6367] Flink SQL engine supports RenewDelegationToken
    • [KYUUBI #6344] FlinkProcessBuilder prioritizes user configurations
    • [KYUUBI #6249] Drop support for Flink 1.16.
    • [KYUUBI #6134][FOLLOWUP] Add profile and CI for Flink 1.19

    Trino Engine

    • [KYUUBI #6696] Fix Trino Status Printer to Prevent Thread Leak
    • [KYUUBI #6485] Fix the Presto TABLE NOT FOUND error message that failed to match
    • [KYUUBI #6524] Trino engine supports insecure configuration
    • [KYUUBI #6387] [TRINO] Trino engine improve operation log

    JDBC Engine

    • Revert “[KYUUBI #6526] Kyuubi BeeLine wrongly process JDBC URL that contains --”
    • [KYUUBI #6526] Kyuubi BeeLine wrongly process JDBC URL that contains --
    • [KYUUBI #6339] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf
    • [KYUUBI #6396] Add caching for KerberosAuthentication using ticketCache key
    • [KYUUBI #6346] Change Kyuubi JDBC Driver default port and make it effective
    • [KYUUBI #6338] Support connecting Kyuubi using Hive JDBC driver 4.0
    • [KYUUBI #6316] Support switching jdbc dialect based on session conf
    • [KYUUBI #6253] Support running JDBC engine on YARN AM
    • [KYUUBI #5374][FOLLOWUP] Fix JDBC ClickHouse TRowSet Generator
    • [KYUUBI #6243] Add missing Impala alias for JDBC engine
    • [KYUUBI #5374] JDBC Engine supports ClickHouse
    • [KYUUBI #6221] Fix parameter replacement issue caused by incorrect sql split
    • [KYUUBI #2782][FOLLOWUP] Add missed UNION_TYPE back
    • [KYUUBI #6182] JDBC engine using the overlay conf as session conf

    Other Engine

    • [KYUUBI #6594] Port HIVE-26633: Make thrift client maxMessageSize configurable
    • [KYUUBI #6639] Port HIVE-27815: Support update numModifiedRows
    • [KYUUBI #6635] Fix Hive 2.3.10 integration tests
    • [KYUUBI #6201] Distinguish the collect mode of different engines

    Python client

    • [KYUUBI #6281][PY] Enable hive test in python client
    • [KYUUBI #6281][PY] Speed up testing with xdist plugin
    • [KYUUBI #6281][PY] Initialize github action for python unit testing
    • [KYUUBI #6567] Fix compatibility of pyhive with setuptools==72.0.0
    • [KYUUBI #6434] Add footnote about pyhive origin

    Web UI

    • [KYUUBI #6727] replace immutable empty list and map in BatchRequest initialization
    • ⬆️ Bump rollup from 3.29.4 to 3.29.5 in /kyuubi-server/web-ui (#6717)
    • [KYUUBI #6712] [UI] Bump pnpm to v9 and nodejs to v20 LTS
    • [KYUUBI #6660][UI] Improve the display of sql editor and query result
    • [KYUUBI #6591] Support authorization on swagger UI
    • [KYUUBI #6463] Release semaphore immediately after startup process exit
    • ⬆️ Bump braces from 3.0.2 to 3.0.3 in /kyuubi-server/web-ui (#6461)
    • [KYUUBI #6428] kyuubi-server/web-ui should not to set a timeout for axios requests
    • [KYUUBI #6299] Support disabling Web UI
    • [KYUUBI #6079] Web UI support Basic authN
    • [KYUUBI #6283] Improve the message of distribution built without enabling web ui
    • [KYUUBI #6248] Web SQL Editor supports run SQL snippets
    • [KYUUBI #6257] Bump vite from 4.5.2 to 4.5.3 in /kyuubi-server/web-ui
    • [KYUUBI #6228] Dropdown Selection and Parameter Parsing Issues in Kyuubi SQL Editor
    • [KYUUBI #6188] Bump follow-redirects from 1.15.4 to 1.15.6 in web-ui

    Documentation

    • [KYUUBI #6752] [DOC] Bump doc build requirements
    • [KYUUBI #6728] [DOC] update Authz plugin docs of build command with -am option
    • ⬆️ Bump get-func-name in /kyuubi-server/web-ui (#6735)
    • [KYUUBI #6734] [DOC] add authentication example in REST API docs
    • ⬆️ Bump micromatch from 4.0.5 to 4.0.8 in /kyuubi-server/web-ui (#6729)
    • [KYUUBI #6719] [DOC] Fix a couple of typos
    • [KYUUBI #6673] [DOC] Fix typos in logging.md
    • [KYUUBI #6671] [DOC] Fix typo in ENGINE SHARE LEVEL docs
    • [KYUUBI #6658] [DOCS] Fixed typo in REST API docs
    • [KYUUBI #6628] [DOCS] Improve docs for GROUP Share Level
    • [KYUUBI #6596] Fix typos in architecture page
    • [KYUUBI #6566] Fix typos in KyuubiConf
    • [KYUUBI #6512] Improve docs for KSHC
    • [KYUUBI #6514] Add link to Github Issues on Contributing Documentation page
    • [KYUUBI #6507] Correct Kyuubi BeeLine help message
    • [KYUUBI #6504] Improve docs for Solution for Large Query Results
    • [KYUUBI #6440] Fix casing in kubectl commands for serviceaccount creation
    • [KYUUBI #6432] Mention support for Spark 4.0 in docs
    • [KYUUBI #6394] Fix example SQL in Spark TPC-DS connector docs
    • [KYUUBI #6242] Remove block cleaner docs
    • [KYUUBI #6234] Fix invalid code-block syntax in docs
    • [KYUUBI #6191] Update docs to mention support of Flink 1.19

    Building, Infrastructure and Dependency

    • [KYUUBI #6775] Bump Github actions action/cache and actions/dependency-review-action v4
    • [KYUUBI #6768] [CI] skip explicitly setting check-latest to false
    • [KYUUBI #6762] [BUILD] cleanup dockerfile warnings
    • [KYUUBI #6763] [CI] Specify JDK 11 for spark engine on kubernetes IT
    • [KYUUBI #6741] [CI] update GHA actions/upload-artifact from v3 to v4
    • [KYUUBI #6739] [CI] Bump runner image from Ubuntu 22.04 to 24.04
    • [KYUUBI #6687] [BUILD] Use MAVEN_ARGS for passing Maven CLI arguments
    • [KYUUBI #6757] Pass build args to extra spark engine build command
    • [KYUUBI #6740] [CI] update GHA docker/build-push-action from v5 to v6 to support build summary
    • [KYUUBI #6755] Upgrade default flink version to 1.20.0
    • [KYUUBI #6742] Bump Jetty from 9.4.54 to 9.4.56
    • [KYUUBI #6743] Bump protobuf from 3.25.4 to 3.25.5
    • [KYUUBI #6744] Bump vite from 4.5.3 to 4.5.5
    • [KYUUBI #6745] Bump testcontainers-scala from 0.41.0 to 0.41.4
    • [KYUUBI #6746] Bump commons-lang3 from 3.13.0 to 3.17.0
    • [KYUUBI #6747] Bump Guava from 32.1.3 to 33.3.1
    • [KYUUBI #6737] Bump vertx-grpc and vertx-core from 4.5.1 to 4.5.3
    • ⬆️ Bump semver in /kyuubi-server/web-ui (#6736)
    • ⬆️ Bump axios from 1.6.0 to 1.7.4 in /kyuubi-server/web-ui (#6617)
    • [KYUUBI #6733] Bump SQLlite from 3.42 to 3.46
    • [KYUUBI #6682] Bump super-linter to v7
    • [KYUUBI #6677] Bump scala-collection-compat to 2.12.0
    • [KYUUBI #6676] Bump Scala to 2.12.19
    • [KYUUBI #6672] Cleanup unused Commons Lang 2 dependency
    • [KYUUBI #6664] [BUILD] Bump Maven from 3.8.8 to 3.9.9
    • [KYUUBI #6659] Bump Iceberg to 1.6.1
    • [KYUUBI #6644] Bump Paimon to 0.8
    • [KYUUBI #5314] Support JDK 21
    • [KYUUBI #6603] Upgrade Scala version from 2.13.8 to 2.13.14
    • [KYUUBI #6620] [K8S][HELM] Update default Kyuubi version to 1.9.2
    • [KYUUBI #6606][FOLLOWUP] Restore shell compatibility for build/mvn
    • [KYUUBI #6584] Upgrade flip-tables from 1.0.2 to 1.1.1
    • [KYUUBI #6593] bin/kyuubi supports kill command
    • [KYUUBI #6607] Bump gRPC 1.65.1 Protobuf 3.25.4 Guava 32.1.3
    • [KYUUBI #6606] build/mvn checks project local downloaded binary ahead
    • [KYUUBI #6600] Correct usage of maven.scaladoc.skip
    • [KYUUBI #6599] Disable Javadoc/Scaladoc by default
    • [KYUUBI #6571] Bump apache parent pom from 31 to 33
    • [KYUUBI #6580] [K8S][HELM] Improve metrics configuration
    • [KYUUBI #6586] Bump commons-io to 2.16.1
    • [KYUUBI #6364] Bump scala-maven-plugin from 4.8.0 to 4.9.2
    • [KYUUBI #6558] Bump Iceberg 1.6.0
    • [KYUUBI #6559] Various improvments for release scripts
    • [KYUUBI #6550] Delete updateStrategy.rollingUpdate.maxUnavailable because it is alpha feature
    • [KYUUBI #6543] Bump Kyuubi Shaded 0.4.1
    • [KYUUBI #6548] Bump enforcer extra mojo rules to 1.8.0 for recognizing higher Java bytecode versions
    • [KYUUBI #6523] Support configuration with --conf options
    • [KYUUBI #6529] Upgrade kubernetes-client from 6.8.1 to 6.13.1
    • [KYUUBI #6528] Bump Netty 4.1.108.Final
    • [KYUUBI #6515] Bumping up dropwizard metrics to 4.2.26
    • [KYUUBI #6506] kyuubi-beeline supports --conf
    • [KYUUBI #6493] [K8S][HELM] Update default Kyuubi version to 1.9.1
    • [KYUUBI #6484] Bump ws from 8.8.1 to 8.17.1 in web-ui
    • [KYUUBI #6466] Bump minikube and kubernetes version of integration test
    • [KYUUBI #6458] Remove commons-logging from binary release
    • [KYUUBI #6457] Strip dependencies from REST client
    • [KYUUBI #6456] Opens jdk.internal.ref
    • [KYUUBI #6445] Normalize extra name for optional Python distribution dependencies
    • [KYUUBI #6427] Extract data lake artifact names as maven properties
    • [KYUUBI #6451] Bump Hudi 0.15.0 and enable Hudi authZ test for Spark 3.5
    • [KYUUBI #6439] kyuubi-util-scala test jar leaked to compile scope
    • [KYUUBI #6430] livenessProbe uses absolute path based on KYUUBI_HOME
    • [KYUUBI #6419] Bump Jetty 9.4.54.v20240208
    • [KYUUBI #6418] Bump Jersey to 2.40
    • [KYUUBI #6416] Generate flattened POM
    • [KYUUBI #6417] Remove scala compile args -Yno-adapted-args
    • [KYUUBI #6397] Align Antlr version with Spark for Spark 4.0
    • [KYUUBI #6392] Support javax.servlet and jakarta.servlet co-exist
    • [KYUUBI #6391] Bump Arrow from 15.0.2 to 16.0.0
    • [KYUUBI #6382] Bump Delta from 3.1.0 to 3.2.0 for Spark 3.5
    • [KYUUBI #6348] Upgrade iceberg from 1.5.0 to 1.5.2
    • [KYUUBI #6328] Bump Hive 2.3.10
    • [KYUUBI #6357] Bump org.bouncycastle:bcprov-jdk18on from 1.77 to 1.78
    • [KYUUBI #6363] [INFRA] Increase CI concurrency
    • [KYUUBI #6294] Simplify Netty and gRPC dependency management
    • [KYUUBI #6312][HELM] Allow attaching annotations to service account
    • [KYUUBI #6293] Upgrade Arrow from 12.0.0 to 15.0.2
    • [KYUUBI #6298][HELM] Make ServiceMonitor use selector labels
    • [KYUUBI #6251] Improve kyuubi-beeline help message
    • [KYUUBI #6252] Upgrade hive-service-rpc 4.0.0
    • [KYUUBI #6266] Kyuubi BeeLine supports JSON output format.
    • [KYUUBI #5686][FOLLOWUP] Rename pyhive to python
    • [KYUUBI #6180][FOLLOWUP] Update the setup JDK name in nightly.yml
    • [KYUUBI #5686] Donate PyHive to Apache Kyuubi
    • [KYUUBI #6268] Specify logDir for RollingFile filePattern
    • [KYUUBI #6212] Added audit handler shutdown to the shutdown hook
    • [KYUUBI #6267] Remove unused dependency management in POM
    • [KYUUBI #6271] Upgrade kafka-clients from 3.5.1 to 3.5.2
    • [KYUUBI #6239] Rename beeline to kyuubi-beeline
    • [KYUUBI #6240] [INFRA] Add issue template for 2024H1 Kyuubi Code Contribution Task
    • [KYUUBI #6185] Audit kyuubi operation state change log
    • [KYUUBI #6196] Add log4j2 template for kyuubi repl commands
    • [KYUUBI #6195] Update Helm Chart and playground to use 1.9.0
    • [KYUUBI #6193] [INFRA] Add known_transkations
    • [KYUUBI #6183] [K8S] KyuubiConf.getKubernetesConf should set namespace even if the Kubernetes context is empty
    • [KYUUBI #6179] Bump iceberg from 1.4.3 to 1.5.0

    Credits

    Last but not least, this release would not have been possible without the following contributors:

    • Alexander Rashed
    • Binjie Yang
    • Bogdan
    • Bowen Liang
    • Bruce Wong
    • Cheng Pan
    • Fei Wang
    • Fu Chen
    • George314159
    • Harry
    • Lennon Chin
    • Liu Xiao
    • Lucas Resch
    • Mike Wilson
    • PorterZhang2021
    • Shilun Fan
    • SteNicholas
    • Wang, Fei
    • Zhigang
    • amanraj2520
    • artilexx
    • avishnus
    • axiangzheng
    • binjie yang
    • camper42
    • caoyu
    • chengpeiming
    • dependabot[bot]
    • dnskr
    • dupeng
    • felixzh
    • futureltl
    • hang.huang
    • hezhao2
    • huangxiaoping
    • huangzhir
    • ic4y
    • jeanlyn
    • jialiang
    • jiaoqingbo
    • joey.ljy
    • lifulong
    • lijie0203
    • liuxiao
    • lllong33
    • ludejiu
    • madlnu
    • minyk
    • pengbei
    • promising-forever
    • ryosuke-hasebe
    • senmiaoliu
    • tatian
    • taylor.fan
    • vinoyang
    • wangjunbo
    • wankunde
    • wenjie.wang01
    • wforget
    • xorsum
    • yanghua
    • yikaifei
    • zRain
    • zhang_yao
    • zhouyifan279
    • zml1206
    • 吴梓溢
    • 王龙

    Release Notes - 1.9.2

    2024 Jul 23

    Changelogs

    • [KYUUBI #6542] KyuubiBatchService should wait for HTTP server started before picking jobs
    • [KYUUBI #6534] Fix checkEngineConnectionAlive may throw NPE
    • [KYUUBI #6529] Upgrade kubernetes-client from 6.8.1 to 6.13.1
    • [KYUUBI #6531] Fix SPARK-EngineTab stop/gracefulstop not work
    • [KYUUBI #6518] Support extracting URL for Spark 4 on YARN
    • [KYUUBI #6524] Trino engine supports insecure configuration
    • [KYUUBI #6515] Bumping up dropwizard metrics to 4.2.26
    • [KYUUBI #6516] Fix KyuubiSparkUtil.buildURI
    • [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf
    • [KYUUBI #6512] Improve docs for KSHC
    • [KYUUBI #6507] Correct Kyuubi BeeLine help message
    • [KYUUBI #6500] Remove service loader descriptor for relocated HMS DelegationTokenIdentifier
    • [KYUUBI #6469] Lazily initialize RecordReaderIterator to avoid driver oom when fetching big result set
    • [KYUUBI #6476] Fix incomplete app events deserialization in SHS
    • [KYUUBI #6302] Skip spark job group cancellation on incremental collect mode
    • [KYUUBI #6302] Call cancelJobGroup immediately after statement execution finished
    • [KYUUBI #4847] Fix engine session never idle issue
    • [KYUUBI #6466] Bump minikube and kubernetes version of integration test
    • [KYUUBI #6463] Release semaphore immediately after startup process exit
    • [KYUUBI #6458] Remove commons-logging from binary release
    • [KYUUBI #6457] Strip dependencies from REST client
    • [KYUUBI #5957] Flink engine should not load kyuubi-defaults.conf
    • [KYUUBI #6353] Catch exception for closing flink internal session
    • [KYUUBI #6251] Replace kyuubi-beeline with beeline in beeline help message
    • [KYUUBI #6439] kyuubi-util-scala test jar leaked to compile scope
    • [KYUUBI #6437] Fix Spark engine query result save to HDFS
    • [KYUUBI #6440] Fix casing in kubectl commands for serviceaccount creation
    • [KYUUBI #6441] Kyuubi Spark TPC-DS/H Connector cross version test
    • [KYUUBI #6247] Make KSHC binary compatible with multiple Spark versions
    • [KYUUBI #6493] [K8S][HELM] Update default Kyuubi version to 1.9.1

    Credits

    Last but not least, this release would not have been possible without the following contributors:

    • Fu Chen
    • Fei Wang
    • Baokun Han
    • Madhukar525722
    • Cheng Pan
    • Zhen Wang
    • Z1Wu
    • Yifan Zhou
    • avishnus
    • camper42
    • felixzh2020
    • huangzhir
    • ic4y
    • jeanlyn
    • jiaoqingbo
    • promising-forever
    • zcx513566

    Release Notes - 1.9.1

    2024 Jun 3

    Highlights

    • Experimental support for Spark 4.0
    • Web UI supports Basic Authentication

    Features and Improvements

    • [KYUUBI #6424] TPC-H/DS connector support Spark 4.0
    • [KYUUBI #6413] SPARK_HOME detection supports Spark 4
    • [KYUUBI #6404] HiveResult.toHiveString compatible to Spark 4.0
    • [KYUUBI #6405] Spark engine supports both javax and jakarta ws.rs namespaces
    • [KYUUBI #6399] Spark engine supports both javax and jakarta servlet namespaces
    • [KYUUBI #6392] Support javax.servlet and jakarta.servlet co-exist
    • [KYUUBI #6297] Package Spark SQL engine both Scala 2.12 and 2.13
    • [KYUUBI #4847] Exclude the alive probe sessions in terminating checker
    • [KYUUBI #6172] Support to interrupt the thrift request immediately after marking the engine not alive
    • [KYUUBI #6315] Spark 3.5: MaxScanStrategy supports DSv2
    • [KYUUBI #6049] Support to filter sessions/operations with session type
    • [KYUUBI #6366] Improve log message for YARN getApplicationInfoByTag
    • [KYUUBI #6298] Make ServiceMonitor use selector labels in Helm charts
    • [KYUUBI #6251] Improve kyuubi-beeline help message
    • [KYUUBI #6290] Add custom exception serialization for SparkOperationEvent
    • [KYUUBI #6079] Web UI supports Basic authN
    • [KYUUBI #6338] Support connecting Kyuubi using Hive JDBC driver 4.0
    • [KYUUBI #6248] Web SQL Editor supports run SQL snippets
    • [KYUUBI #6216] Support to deny some client ips to make connection
    • [KYUUBI #6283] Improve the message of distribution built without enabling web ui
    • [KYUUBI #6207] Support to retrieve Spark UserDefinedType result
    • [KYUUBI #6205] Backport SPARK-47300: quoteIfNeeded should quote identifier starts with digits
    • [KYUUBI #6215] Improve DropIgnoreNonexistent rule for Spark 3.5
    • [KYUUBI #6211] Check memory offHeap enabled for CustomResourceProfileExec

    Bug Fixes

    • [KYUUBI #6430] livenessProbe uses absolute path based on KYUUBI_HOME
    • [KYUUBI #6428] kyuubi-server/web-ui should not to set a timeout for axios requests
    • [KYUUBI #6422] Merge standard error to output
    • [KYUUBI #6396] Add caching for KerberosAuthentication using ticketCache key
    • [KYUUBI #6410] Check if asyncRequestExecutor is initialized before shutdown
    • [KYUUBI #6394] Fix example SQL in Spark TPC-DS connector docs
    • [KYUUBI #6400] Fix memory leak when using saveToFile
    • [KYUUBI #6344] FlinkProcessBuilder prioritizes user configurations
    • [KYUUBI #6346] Change Kyuubi JDBC Driver default port and make it effective
    • [KYUUBI #6212] Added audit handler shutdown to the shutdown hook
    • [KYUUBI #6370] For saveToFile, checking idx to prevent IndexOutOfBoundsException
    • [KYUUBI #6320] Fix terminated application pods not deleted issue when kyuubi server restarted
    • [KYUUBI #6291] Avoid NPE in MySQLErrPacket
    • [KYUUBI #5767] Fix spark batch conf not convert issue
    • [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue
    • [KYUUBI #6287] Avoid missing match
    • [KYUUBI #6183] KyuubiConf.getKubernetesConf should set namespace even if the Kubernetes context is empty
    • [KYUUBI #6204] Fix kyuubi session limiter leak when opening session failed
    • [KYUUBI #6182] JDBC engine using the overlay conf as session conf
    • [KYUUBI #6221] Fix parameter replacement issue caused by incorrect sql split
    • [KYUUBI #6228] Dropdown Selection and Parameter Parsing Issues in Kyuubi SQL Editor

    Build, Test and Misc

    • [KYUUBI #6432] Mention support for Spark 4.0 in docs
    • [KYUUBI #6423] Fix Scalatest exclusion tags
    • [KYUUBI #6419] Bump Jetty 9.4.54.v20240208
    • [KYUUBI #6418] Bump Jersey to 2.40
    • [KYUUBI #6416] Generate flattened POM
    • [KYUUBI #6415] Manage Jersey in kyuubi-server module
    • [KYUUBI #6417] Remove scala compile args -Yno-adapted-args
    • [KYUUBI #6382] Bump Delta from 3.1.0 to 3.2.0 for Spark 3.5
    • [KYUUBI #6348] Upgrade iceberg from 1.5.0 to 1.5.2
    • [KYUUBI #6328] Bump Hive 2.3.10
    • [KYUUBI #6373] Avoid NPE in get operation log api
    • [KYUUBI #6377] Fix isCommand check and set min rows threshold for saveToFile
    • [KYUUBI #6357] Bump org.bouncycastle:bcprov-jdk18on from 1.77 to 1.78
    • [KYUUBI #6363] Increase CI concurrency
    • [KYUUBI #6244] Fix test logs upload error
    • [KYUUBI #6294] Simplify Netty and gRPC dependency management
    • [KYUUBI #6252] Upgrade hive-service-rpc 4.0.0
    • [KYUUBI #6268] Specify logDir for RollingFile filePattern
    • [KYUUBI #6267] Remove unused dependency management in POM
    • [KYUUBI #6271] Upgrade kafka-clients from 3.5.1 to 3.5.2
    • [KYUUBI #6260] Clean up and improve comments for spark extensions
    • [KYUUBI #6257] Bump vite from 4.5.2 to 4.5.3 in /kyuubi-server/web-ui
    • [KYUUBI #6243] Add missing Impala alias for JDBC engine
    • [KYUUBI #6234] Fix invalid code-block syntax in docs
    • [KYUUBI #2782] Add missed UNION_TYPE back
    • [KYUUBI #6206] Update comments for job group id of Spark broadcast jobs
    • [KYUUBI #6197] Revise dependency management of Spark authZ plugin
    • [KYUUBI #6196] Add log4j2 template for kyuubi repl commands
    • [KYUUBI #6194] AuthZ shaded should include ranger-plugins-cred
    • [KYUUBI #6188] Bump follow-redirects from 1.15.4 to 1.15.6 in web-ui
    • [KYUUBI #6191] Update docs to mention support of Flink 1.19
    • [KYUUBI #6134] Add profile and CI for Flink 1.19
    • [KYUUBI #6193] Add known_transkations
    • [KYUUBI #6179] Bump iceberg from 1.4.3 to 1.5.0

    Credits

    Last but not least, this release would not have been possible without the following contributors:

    • Lennon Chin
    • JiaLiangC
    • Senmiao Liu
    • Xiao Liu
    • Cheng Pan
    • PorterZhang2021
    • TakawaAkirayo
    • Zhen Wang
    • Fei Wang
    • Binjie Yang
    • He Zhao
    • Mingliang Zhu
    • amanraj2520
    • beryllw
    • camper42
    • dupen01
    • lllong33
    • mrpengbei
    • sudohainguyen
    • zRains

    Release Notes - 1.7.4

    2024 Apr 26

    Changlogs

    • [KYUUBI #5451] Ignore NoSuchFileException during OperationLog.close()
    • [KYUUBI #6291] Avoid NPE in MySQLErrPacket
    • [KYUUBI #2782] Add missed UNION_TYPE back
    • [KYUUBI #6204] Fix kyuubi session limiter leak when opening session failed
    • [KYUUBI #6061] Delete svn staging dir before checking out
    • [KYUUBI #6047] Free up disk space
    • [KYUUBI #5953] Update NOTICE
    • [KYUUBI #5951] Improve doc for Session Conf Advisor
    • [KYUUBI #5933] Happy New Year 2024
    • [KYUUBI #5922] Enhance YARN app state judgment
    • [KYUUBI #5832] Always perform closing action in OperationLog to avoid fd leak
    • [KYUUBI #5857] Dependencies CI check when dev/dependencyList changes
    • [KYUUBI #5849] Incorrectly parse JDBC URL while variable includes colon
    • [KYUUBI #5822] Fix the usage msg of KyuubiBeeLine
    • [KYUUBI #5819] Fix rst syntax of missing blank line after code-block
    • [KYUUBI #5785] Fix flaky test - JpsApplicationOperation with spark local mode
    • [KYUUBI #5750] Fix the inaccurate issue of obtaining COLUMN_SIZE in the decimal field jdbc of spark engine
    • [KYUUBI #5754] Fix engine id of the engine probe failures logging messages
    • [KYUUBI #5317] Hive Connector throws NotSerializableException on reading Hive Avro partitioned table
    • [KYUUBI #5720] Fix flaky test in TPCHQuerySuite
    • [KYUUBI #5713] Backport HIVE-27271: Client connection to HS2 fails when transportMode=http, ssl=true, sslTrustStore specified without trustStorePassword in the JDBC URL
    • [KYUUBI #5697] Fix incorrect url reference in code and docs
    • [KYUUBI #5599] Correct the parameter name of fetching op log in RESTful API
    • [KYUUBI #5499]Catch any exception when closing idle session
    • [KYUUBI #5414] KSHC reader should not pollut the global hiveConf instance
    • [KYUUBI #5432] Fix typo in README.md
    • [KYUUBI #5347] Fix java code errors in quick_start_with_jdbc
    • [KYUUBI #5333] Aaptive call fetchLaunchEngineResult
    • [KYUUBI #5308] Ensure release using Java 8

    Credits

    Last but not least, this release would not have been possible without the following contributors:

    • Shaoyun Chen
    • Chao Chen
    • Flyangz
    • Pengqi Li
    • Bowen Liang
    • Omkar Makhare
    • Drake Youngkun Min
    • Cheng Pan
    • Gianluca Principini
    • Zeyu Wang
    • Kang Wang
    • Fei Wang
    • Zhen Wang
    • Kaifei Yi
    • He Zhao
    • mrtisttt

    Release Notes - 1.8.2

    2024 Apr 26

    Changelogs

    • [KYUUBI #6320] Fix terminated application pods not deleted issue when kyuubi server restarted
    • [KYUUBI #6298] Make ServiceMonitor use selector labels
    • [KYUUBI #6291] Avoid NPE in MySQLErrPacket
    • [KYUUBI #6283] Improve the message of distribution built without enabling web ui
    • [KYUUBI #6287] Avoid missing matche
    • [KYUUBI #6268] Specify logDir for RollingFile filePattern
    • [KYUUBI #6212] Added audit handler shutdown to the shutdown hook
    • [KYUUBI #6271] Upgrade kafka-clients from 3.5.1 to 3.5.2
    • [KYUUBI #5767] Fix spark batch conf not convert issue
    • [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue
    • [KYUUBI #6234] Fix invalid code-block syntax in docs
    • [KYUUBI #6215] Improve DropIgnoreNonexistent rule for Spark 3.5
    • [KYUUBI #6221] Fix parameter replacement issue caused by incorrect sql split
    • [KYUUBI #2782] Add missed UNION_TYPE back
    • [KYUUBI #6207] Support to retrieve Spark UserDefinedType result
    • [KYUUBI #6206] Update comments for job group id of Spark broadcast jobs
    • [KYUUBI #6204] Fix kyuubi session limiter leak when opening session failed
    • [KYUUBI #6182] JDBC engine using the overlay conf as session conf
    • [KYUUBI #6193] Add known_transkations
    • [KYUUBI #6183] KyuubiConf.getKubernetesConf should set namespace even if the Kubernetes context is empty
    • [KYUUBI #6175] Refactor copy path about SPARK_HOME in docker
    • [KYUUBI #6166] Fix wrong copy path about SPARK_HOME in docker
    • [KYUUBI #6170] Merge JDBC Trino TPC jobs and Style Check jobs in GitHub Actions workflow
    • [KYUUBI #6164] Limit max parallel for each GitHub Actions workflow
    • [KYUUBI #6156] Remove flink. prefix for create session configurations
    • [KYUUBI #6173] Bump maven-download-plugin from 1.7.1 to 1.8.1
    • [KYUUBI #6168] Check if forcedMaxOutputRows is negative
    • [KYUUBI #6167] Drop JDK 11 on GitHub Actions
    • [KYUUBI #6158] Remove confusing variable
    • [KYUUBI #6152] Remove useless variable
    • [KYUUBI #6130] Stop engine immediately after close session for CONNECTION level FlinkSQLEngine
    • [KYUUBI #6147] Use the overlay conf as session conf
    • [KYUUBI #6078] KSHC should handle the commit of the partitioned table as dynamic partition at write path
    • [KYUUBI #6139] Expose env vars and sys props to JUnit tests too
    • [KYUUBI #6126] Filter blank engine home env
    • [KYUUBI #6114] Properly handle KyuubiStatement#getMoreResults(CLOSE_CURRENT_RESULT)
    • [KYUUBI #5902] Bump doc build dependencies
    • [KYUUBI #6081] RESTful client should retry on UnknownHostException
    • [KYUUBI #6071] Add .java-version into git ignore
    • [KYUUBI #6069] TFrontendServiceSuite - close expired operations
    • [KYUUBI #5104] BatchesResourceSuite - expose the metrics with operation type and current state
    • [KYUUBI #6061] Delete svn staging dir before checking out

    Credits

    Last but not least, this release would not have been possible without the following contributors:

    • Shaoyun Chen
    • Lennon Chin
    • Fei Wang
    • Bowen Liang
    • Xiao Liu
    • Senmiao Liu
    • Tigran Manasyan
    • Cheng Pan
    • TakawaAkirayo
    • Zhen Wang
    • Binjie Yang
    • Vino Yang
    • Kaifei Yi
    • He Zhao
    • amanraj2520
    • dupen01
    • sudohainguyen

    Release Notes - 1.9.0

    2024 Mar 19

    Highlight

    • Fully verified with Java 8, 11, and 17, Scala 2.12 and 2.13
    • Fully verified with Apache Spark 3.1 to 3.5, support for Spark 3.1 is deprecated and will be removed in the next release
    • Fully verified with Apache Flink 1.16 to 1.19, support for Flink 1.16 is deprecated and will be removed in the next release
    • Fully verified with Apache Hive 2.3, 3.1, and Cloudera Hive 2.1.1-cdh6, support running Hive engine on YARN (experimental)
    • Spark engine supports save large query result to file before retrieving
    • JDBC engine supports more dialects, including Doris, MySQL, PostgreSQL, Phoenix, StarRocks and Impala
    • Kyuubis server metastore supports PostgreSQL as backend storage
    • Introduce a basic SQL query editor in Kyuubi Web UI
    • Continuously Spark AuthZ plugin improvements, including enhanced Iceberg support and new introduced Apache Hudi and Delta Lake support
    • Continuously Kyuubi & Spark on Kubernetes improvements

    Server

    • [KYUUBI #5423] Support chaining SessionConfAdvisors
    • [KYUUBI #5631][K8S][HELM] Set session affinity if needed in helm chart
    • [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata
    • [KYUUBI #5683] Add kyuubi.session.proxy.user as an alternative to hive.server2.proxy.user
    • [KYUUBI #5704] Add unit tests for kyuubi.session.proxy.user in RESTful API
    • [KYUUBI #5729] Use G1GC as Java option example in kyuubi-env template
    • [KYUUBI #5797] Support to describe engine with command in current session
    • [KYUUBI #5797][FOLLOWUP] Desc engine command support show engine registered attributes
    • [KYUUBI #5873] Support to get operation progress with RESTful API
    • [KYUUBI #5912] Cleanup unused config kyuubi.frontend.thrift.http.allow.user.substitution
    • [KYUUBI #5921] EngineRef#deregister should return the value
    • [KYUUBI #5934][K8S][HELM] Add Spark configuration support
    • [KYUUBI #6003] Allow disabling user impersonation on launching engine
    • [KYUUBI #6006][HELM] Support additional labels for service monitor
    • [KYUUBI #6066] Fix PostgreSQL metastore migration script failure if indexes already exist
    • [KYUUBI #6070] Improve perf on assembling row-based TRowSet
    • [KYUUBI #6080] Improve instruction for deploy Kyuubi on K8s
    • [KYUUBI #6081] RESTful client should retry on UnknownHostException
    • [KYUUBI #6099] Remove support of Derby for Kyuubi metastore
    • [KYUUBI #6126] Filter blank engine home env
    • [KYUUBI #6140] Move getCodeSourceLocation to kyuubi-util

    Spark Engine

    • [KYUUBI #5377] Spark engine query result save to file
    • [KYUUBI #5377][FOLLOWUP] Spark engine query result save to file
    • [KYUUBI #5377][FOLLOWUP] Always try to cleanup session result path
    • [KYUUBI #5377][FOLLOWUP] Get limit from more spark plan and regard result max rows
    • [KYUUBI #5870] Directly mapping engine’s data type to Java type for TRowSet generation
    • [KYUUBI #5877] Support Python magic syntax for notebook usage
    • [KYUUBI #5877][FOLLOWUP] Add spark output mode to better support PySpark notebook
    • [KYUUBI #5894] Separate closed and online sessions/statements in the SparkUI’s engine tab
    • [KYUUBI #5899] Refine the python/scala interpreter error prompt
    • [KYUUBI #6053][K8S] Show more details for pod/container status in application error
    • [KYUUBI #6083] Bump Spark 3.3.3 to 3.3.4
    • [KYUUBI #6091] Deprecate and remove building support for Spark 3.1
    • [KYUUBI #6092] Enable DeltaTest for Spark 3.5
    • [KYUUBI #6094] Bump Spark from 3.5.0 to 3.5.1
    • [KYUUBI #6095] Enable PaimonTest for Spark 3.5
    • [KYUUBI #6096] Enable PySpark tests for Spark 3.5
    • [KYUUBI #6107][Spark] Collect and summarize the executorRunTime and executorCpuTime of the statement
    • [KYUUBI #6107][FOLLOWUP] Add comments for the newly added parameters of the Spark event
    • [KYUUBI #6108] Display the CPU time consumed by the statement in the Spark Engine tab
    • [KYUUBI #6156] Remove flink. prefix for create session configurations
    • [KYUUBI #6163] Set default Spark version to 3.5
    • [KYUUBI #6166][MINOR] Fix wrong copy path about SPARK_HOME in docker
    • [KYUUBI #6175][MINOR] Refactor copy path about SPARK_HOME in docker

    Spark AuthZ Plugin

    • [KYUUBI #5248][AUTHZ] Check previleges for Iceberg’s CALL to RewriteDataFilesProcedure procedure
    • [KYUUBI #5255] Add an optional comment field to the authz specs for better recognition
    • [KYUUBI #5284] Support Hudi Alter Table Command in Authz
    • [KYUUBI #5323][AUTHZ] Drop Hive and Iceberg tables with PURGE option in tests
    • [KYUUBI #5359][AUTHZ] Support Create Table Commands for Hudi
    • [KYUUBI #5360] Support Hudi InsertIntoHoodieTableCommand
    • [KYUUBI #5361][AUTHZ] Support Drop/Truncate Table Commands for Hudi
    • [KYUUBI #5362] Remove Spark 3.0 support for Authz
    • [KYUUBI #5399][AUTHZ] Cleanup Spark 3.0 specific implementation
    • [KYUUBI #5407][AUTHZ] Tests for Iceberg system procedures of snapshot management
    • [KYUUBI #5417] should not check in-subquery in permanent view
    • [KYUUBI #5417] Authz should not check dependent subquery plan privilege
    • [KYUUBI #5427][AUTHZ] Shade spark authz plugin
    • [KYUUBI #5446][AUTHZ] Support Create/Drop/Show/Reresh index command for Hudi
    • [KYUUBI #5447][AUTHZ] Support Hudi DeleteHoodieTableCommand/UpdateHoodieTableCommand/MergeIntoHoodieTableCommand
    • [KYUUBI #5447][FOLLOWUP] Remove unrelated debug prints in TableIdentifierTableExtractor
    • [KYUUBI #5452][AUTHZ] Support Compaction table commands for Hudi
    • [KYUUBI #5455][AUTHZ] Support hudi CompactionHoodiePathCommand & CompactionShowHoodiePathCommand
    • [KYUUBI #5457][AUTHZ] Support RepairTable Commands for Hudi
    • [KYUUBI #5462][AUTHZ] Support create table command for Paimon
    • [KYUUBI #5472] Permanent View should pass column when child plan no output
    • [KYUUBI #5472][AUTHZ][FOLLOWUP] Check permanent view also need support merge projection
    • [KYUUBI #5475][FOLLOWUP] Authz check permanent view’s subquery should check view’s correct privilege
    • [KYUUBI #5478][AUTHZ] Support Hudi ShowHoodieTablePartitionsCommand
    • [KYUUBI #5479][AUTHZ] Support Hudi CallProcedureHoodieCommand for stored procedures
    • [KYUUBI #5492][AUTHZ] saveAsTable create DataSource table miss db info
    • [KYUUBI #5496][AUTHZ] Authz shaded include the transitive jackson libs
    • [KYUUBI #5497][AuthZ] Simplify debug message for missing field/method in ReflectUtils
    • [KYUUBI #5506][AUTHZ][MINOR] Reuse loaded table extractor in Hudi table extractors
    • [KYUUBI #5512][AuthZ] Remove the non-existent query specs in Deletes and Updates
    • [KYUUBI #5529][AUTHZ] Support create table command for Delta Lake
    • [KYUUBI #5503][AUTHZ] Check plan auth checked should not set tag to all child nodes
    • [KYUUBI #5503][FOLLOWUP][AUTHZ] Authz should skip inner plan that have been verified
    • [KYUUBI #5529][AUTHZ][FOLLOWUP] Fix Create table ut for Delta Lake does not trigger a sql operation
    • [KYUUBI #5529][AUTHZ][FOLLOWUP] Remove useless org.apache.spark.sql.delta.commands.CreateDeltaTableCommand
    • [KYUUBI #5532][AUTHZ] Support Alter table commands of updating table schema for Delta Lake
    • [KYUUBI #5533][AUTHZ] Support merge into table command for Delta Lake
    • [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake
    • [KYUUBI #5535][AUTHZ] Support vacuum table command for Delta Lake
    • [KYUUBI #5537][AuthZ] Order Command Spec by Command Class [group, classname]
    • [KYUUBI #5543][AUTHZ] Generalize command specs and check duplicated class names of command spec
    • [KYUUBI #5546][AUTHZ] Reorgnize the package names for rules
    • [KYUUBI #5555][AUTHZ] Extractor common authorization rule class
    • [KYUUBI #5556][AUTHZ] Support Alter table commands of set table properties
    • [KYUUBI #5557][AUTHZ] Refactor code about handle PermanentViewMarker
    • [KYUUBI #5565][AUTHZ] Support Delete/Insert/Update table command for Delta Lake
    • [KYUUBI #5575][AUTHZ] Support InsertIntoDataSourceDir/InsertIntoHiveDirCommand path check
    • [KYUUBI #5575][KYUUBI #5611][AUTHZ] Check path privilege should use correct action type
    • [KYUUBI #5576][Bug] Fix wrong code in test case of dir command
    • [KYUUBI #5579][AUTHZ] Support LogicalRelation don’t have CatalogTable but have HadoopFsRelation
    • [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node’s input
    • [KYUUBI #5611][AUTHZ] Authz support path privilege for SaveIntoDataSourceCommand
    • [KYUUBI #5628][AUTHZ] Support path privilege check for resource command
    • [KYUUBI #5630][Authz] Support path check of LoadDataCommand
    • [KYUUBI #5638][AUTHZ] URI related privileges shall be READ/WRITE
    • [KYUUBI #5638][FOLLOWUP] Remove unnecessary actionType in uri desc
    • [KYUUBI #5642][AUTHZ] Support path privilege of CreateDatabaseCommand/AlterDatabaseSetLocationCommand/CreateNamespace/SetNamespaceLocation
    • [KYUUBI #5642][FOLLOWUP] Fix unit test
    • [KYUUBI #5652][AUTHZ] CreateTable related command support path privilege
    • [KYUUBI #5657][AUTHZ] Support path privilege of AlterTableSetLocationCommand/AlterTableAddPartitionCommand
    • [KYUUBI #5664][AUTHZ] Support create path-based table for Delta Lake
    • [KYUUBI #5677][AUTHZ] Typeof expression miss column information
    • [KYUUBI #5684][AUTHZ] Support delete/update path-based table for Delta Lake
    • [KYUUBI #5690][AUTHZ] Support insert into/overwrite path-based table for Delta Lake in Authz
    • [KYUUBI #5692][Bug] Authz not skip explain command
    • [KYUUBI #5707][AUTHZ] Support merge into path-based table for Delta Lake in Authz
    • [KYUUBI #5715][MINOR] Replace with .isDefined and .isEmpty
    • [KYUUBI #5726][AUTHZ] Support optimize path-based table for Delta Lake in Authz
    • [KYUUBI #5735][AUTHZ] Support vacuum path-based table for Delta Lake
    • [KYUUBI #5743][AUTHZ] Improve AccessControlException verification of RangerSparkExtensionSuite
    • [KYUUBI #5768][AUTHZ] Authz internal place holder should skip privilege check
    • [KYUUBI #5780][AUTHZ] Treating PermanentViewMarker as LeafNode make code simple and got correct privilege object
    • [KYUUBI #5780][AUTHZ][FOLLOWUP] Format PermanentViewMarker tree string
    • [KYUUBI #5793][AUTHZ][BUG] PVM with nested scalar-subquery should not check src table privilege
    • [KYUUBI #5884] PVM should inherit MultiInstance and wrap with new exprId
    • [KYUUBI #5903] PVM should override computeStats method
    • [KYUUBI #5913][Bug] After resolve PVM should mark all nodes as checked
    • [KYUUBI #5937] PVM cause cache table not work
    • [KYUUBI #5964][BUG] Avoid check not fully optimized query for InsertIntoDataSourceDirCommand and InsertIntoDataSourceCommand
    • [KYUUBI #5965][AUTHZ] Supports check hoodie procedures show_commits resource privileges
    • [KYUUBI #5985][AUTHZ][MINOR] Remove incorrect getUri method
    • [KYUUBI #5997][AUTHZ] Avoid unnecessary loop of RuleEliminateTypeOf
    • [KYUUBI #6007] AuthZ should check hoodie procedures path resource privileges

    Other Spark Plugins

    • [KYUUBI #5580] Support generate and window operators for InferRebalanceAndSortOrders
    • [KYUUBI #5786] Disable spark script transformation
    • [KYUUBI #5816] Change spark rule class to object or case class
    • [KYUUBI #6125] Drop Kyuubi extension for Spark 3.1
    • [KYUUBI #6168] Check if forcedMaxOutputRows is negative

    Hive Engine

    Trino Engine

    • [KYUUBI #5968] Support set authentication user for Trino engine
    • [KYUUBI #6147][TRINO] Use the overlay conf as session conf

    JDBC Engine

    Other Engine

    • [KYUUBI #5384][KSCH] Hive connector supports Spark 3.5
    • [KYUUBI #5709] Avoid exposing client keystore and truststore passwords in the JDBC URL
    • [KYUUBI #5783] Switch to kyuubi-relocated-hive-service-rpc
    • [KYUUBI #5828][CHAT] Kyuubi chat engine supports ernie bot(文心一言)
    • [KYUUBI #6078] KSHC should handle the commit of the partitioned table as dynamic partition at write path
    • [KYUUBI #6114] Properly handle KyuubiStatement#getMoreResults(CLOSE_CURRENT_RESULT)

    Web UI

    Documentation

    • [KYUUBI #5265][DOCS][FLINK] Add notice about Jobmanager HA to Flink deployment docs
    • [KYUUBI #5500] Add Kyuubi Code Program to Doc
    • [KYUUBI #5688][DOC] Update supported spark 3.5.x in authz doc
    • [KYUUBI #5762][DOC] Introduce Kyuubi Web UI Engine Page
    • [KYUUBI #5893][DOC] Update logos in layer picture on welcome page
    • [KYUUBI #5905][DOC] Workaround for long inlined code sample in config page
    • [KYUUBI #6010] Improve Gluten usage documentation
    • [KYUUBI #6119][DOC] Update doc for HA/Zookeeper Configuration

    Building, Infrastructure and Dependency

    • [KYUUBI #4453][FOLLOW] Delete LICENSE for removed kubernetes client dependencies
    • [KYUUBI #5104][TEST] BatchesResourceSuite - expose the metrics with operation type and current state
    • [KYUUBI #5241] Bump Commons Lang3 from 3.12.0 to 3.13.0
    • [KYUUBI #5245][INFRA] Daily deploy snapshot version on branch-1.8
    • [KYUUBI #5266] Upgrade playground to Kyuubi 1.7.1
    • [KYUUBI #5322][INFRA] Enable GitHub Projects
    • [KYUUBI #5324] Bump latest 1.7.3 in playground and helm charts
    • [KYUUBI #5342] Add label hacktoberfest to project
    • [KYUUBI #5356][INFRA] GitHub Issue template for 2023 Kyuubi Code Contribution Program
    • [KYUUBI #5395] Bump netty from 4.1.93.Final to 4.1.100.Final
    • [KYUUBI #5449] Bump Delta Lake 3.0.0
    • [KYUUBI #5501] Update codecov token and fix codecov reporting on PRs
    • [KYUUBI #5509][FOLLOWUP] JDBC IT should always depends on kyuubi-hive-jdbc-shaded
    • [KYUUBI #5600] Fix flaky test SessionsResourceSuite
    • [KYUUBI #5602] Improve issue template for 2023 Kyuubi Code Contribution Program
    • [KYUUBI #5660][TEST] Ensure SPI provider file content sorted in alphabet order
    • [KYUUBI #5671] Bump axios from 0.27.2 to 1.6.0 in /kyuubi-server/web-ui
    • [KYUUBI #5674][LICENSE][FOLLOWUP] Update license files
    • [KYUUBI #5676] Bump Apache parent pom from 30 to 31
    • [KYUUBI #5701] Enhance the PR Template to make PR desc informative
    • [KYUUBI #5716][TEST] Fix match may not be exhaustive in Scala 2.13
    • [KYUUBI #5774][CI] Bump actions/checkout from v3 to v4
    • [KYUUBI #5779][CI] Upgrade Github actions for Docker
    • [KYUUBI #5800][KYUUBI#5467] Integrate Intel Gluten with Spark engine
    • [KYUUBI #5835] Remove issue template for 2023 Kyuubi Code Contribution Task
    • [KYUUBI #5838][BUILD] Bump actions/setup-python from v4 to v5
    • [KYUUBI #5839][BUILD] Bump actions/setup-java from v3 to v4
    • [KYUUBI #5841][BUILD] Bump actions/stale from v7 to v9
    • [KYUUBI #5842][BUILD] Bump actions/setup-node from v3 to v4
    • [KYUUBI #5874][INFRA] Bump github labeler action to v5
    • [KYUUBI #5879] Bump dropwizard metrics from 4.2.8 to 4.2.23
    • [KYUUBI #5902] Bump doc build dependencies
    • [KYUUBI #5914] Update layer diagram on welcome page
    • [KYUUBI #5925] Kyuubi TPC-DS support running benchmark with skipping some queries
    • [KYUUBI #5939] Bump Gluten version to recover Gluten IT
    • [KYUUBI #5941] Drop Kubernetes Block Cleaner Tool from Kyuubi
    • [KYUUBI #5944] Introduce Prometheus and Grafana for Kyuubi Playground
    • [KYUUBI #5955][TEST][GLUTEN] Disable gluten ui when running integration test
    • [KYUUBI #5973] Bump gRPC from 1.53.0 to 1.60.1
    • [KYUUBI #5974] Bump jetcd 0.7.7
    • [KYUUBI #5975] Bump trino-client 411
    • [KYUUBI #5975][FOLLOWUP][TEST] Fix trino - get type info
    • [KYUUBI #6043][TEST][GLUTEN] Gluten-it gluten package add arch suffix
    • [KYUUBI #6061] Delete svn staging dir before checking out
    • [KYUUBI #6063] Bump iceberg 1.4.3
    • [KYUUBI #6067] Bump org.postgresql:postgresql from 42.6.0 to 42.7.2
    • [KYUUBI #6068] Remove community section from user docs
    • [KYUUBI #6069][TEST] TFrontendServiceSuite - close expired operations
    • [KYUUBI #6071] Add .java-version into git ignore
    • [KYUUBI #6074] Add a script to simplify the process of creating release notes
    • [KYUUBI #6075] Update known_translations
    • [KYUUBI #6076] Update Helm Chart and playground to use 1.8.1
    • [KYUUBI #6085] Bump gRPC from 1.60.1 to 1.60.2
    • [KYUUBI #6089] Bump Paimon from 0.5.0-incubating to 0.7.0-incubating
    • [KYUUBI #6090] Bump Delta from 3.0.0 to 3.1.0 for Spark 3.5
    • [KYUUBI #6101] Bump Jackson from 2.15.0 to 2.15.4
    • [KYUUBI #6109] Fork beeline module from Apache Hive 3.1.3
    • [KYUUBI #6117] Bump Kyuubi Shaded 0.3.0
    • [KYUUBI #6118] Migrate to shaded HMS client for getting delegation token on server
    • [KYUUBI #6129][BUILD] Remove the extension spark jars before build to avoid include unspecified version
    • [KYUUBI #6131] Simplify Maven dependency management after dropping building support for Spark 3.1
    • [KYUUBI #6139] Expose env vars and sys props to JUnit tests too
    • [KYUUBI #6143] Add Dockerfile.HMS for testing
    • [KYUUBI #6162] Cut out hive-common deps in beeline module
    • [KYUUBI #6164] Limit max parallel for each GitHub Actions workflow
    • [KYUUBI #6145] Use HiveMetaStore Docker for testing
    • [KYUUBI #6145][FOLLOWUP] Remove unused dependency commons-lang
    • [KYUUBI #6151] Remove invalid matrix spark 3.1 in GA
    • [KYUUBI #6153] Cut out Hive JDBC and Hive Service deps in beeline module
    • [KYUUBI #6160] Fix beeline test KyuubiBeeLineTest.testKyuubiBeelineComment
    • [KYUUBI #6167] Drop JDK 11 on GitHub Actions
    • [KYUUBI #6170] Merge JDBC Trino TPC jobs and Style Check jobs in GitHub Actions workflow
    • [KYUUBI #6173] Bump maven-download-plugin from 1.7.1 to 1.8.1
    • [KYUUBI #6174] Remove unused dependency management from POM
    • [KYUUBI #6177] Bump BouncyCastle from 1.67 to 1.77
    • [KYUUBI #6178] Improve Gluten integration test

    Credits

    Last but not least, this release would not have been possible without the following contributors:

    • GitHub Bot
    • Shaoyun Chen
    • GGHWS
    • Deepali Gupta
    • Baokun Han
    • Denis Krivenko
    • Kwafoor
    • Pengqi Li
    • Bowen Liang
    • Paul Lin
    • Senmiao Liu
    • Chenliang Lu
    • Tigran Manasyan
    • Hai Nguyen
    • Cheng Pan
    • XH Qian
    • Zhen Wang
    • Fei Wang
    • Kang Wang
    • Junjie Xu
    • Binjie Yang
    • Kent Yao
    • Xianxun Ye
    • Kaifei Yi
    • David Yuan
    • He Zhao
    • Yangming Zhao
    • Yifan Zhou
    • Mingliang Zhu
    • Yi Zhu
    • Yaogai Zhu
    • huangxiaopingRD
    • kevinjmh
    • mrtisttt
    • BingKun Pan
    • rhh777
    • silencily
    • sincewhen
    • Vino Yang

    Release Notes - 1.8.1

    2024 Feb 21

    Server

    • [KYUUBI #6058] Make Jetty server stop timeout configurable
    • [KYUUBI #6048] Assign serviceNode and add volatile for variables
    • [KYUUBI #6045] Sync the AdminRestApi with the AdminResource Apis
    • [KYUUBI #6028] Exited spark-submit process should not block batch submit queue
    • [KYUUBI #6001] Fix RESTful protocol security enabled evaluation
    • [KYUUBI #5999] loads the interval of sessionConfCache from configurable file in FileSessionConfAdvisor
    • [KYUUBI #5990] Always take the first declared SASL/PLAIN auth type
    • [KYUUBI #5922] Enhance YARN app state judgment
    • [KYUUBI #5900] Get the engine service node to delete instead of delete engine space
    • [KYUUBI #5900] Support to deregister the engine on open failure
    • [KYUUBI #5892] Metadata store JDBC url supports <KYUUBI_HOME> variable
    • [KYUUBI #5869] Add volatile to variables in KyuubiSessionImpl that may be operated by multiple threads
    • [KYUUBI #5244] Make engineAliveMaxFailCount configurable
    • [KYUUBI #5795] Using correct time unit for pod cleanup executor
    • [KYUUBI #5832] Always perform closing action in OperationLog to avoid fd leak
    • [KYUUBI #5853] Ensure closing the operation log for batch submission in fast-failed case
    • [KYUUBI #5833] Rename service registered endpoint key from serviceUri to serverUri
    • [KYUUBI #5795] Support to cleanup the spark driver pod periodically
    • [KYUUBI #5680] Optimize Spark engine pod name generation
    • [KYUUBI #5807] Simple norm the use of kyuubi.operation.result.arrow.timestampAsString and kyuubi.operation.result.format
    • [KYUUBI #5767] Extract common utils for assembling key value pairs with config option prefix in processbuilder
    • [KYUUBI #5765] Avoid array copy in generating process builder commands
    • [KYUUBI #5771] Cleanup the terminated app info trigger before clear kubernetes clients
    • [KYUUBI #5711] Fix typo for container states audit
    • [KYUUBI #5711] Fix typo and audit pod and container states
    • [KYUUBI #4847] Check engine alive for KyuubiSessionImpl only
    • [KYUUBI #5761] Pretty ProcBuilder generated command
    • [KYUUBI #5754] Fix engine id of the engine probe failures logging messages
    • [KYUUBI #5730] Tolerate execeptions for periodical scheduled tasks
    • [KYUUBI #5210] Close batch operation log even batch is in cancel state
    • [KYUUBI #5731] Support to cleanup the spark driver pod with specified cleanup strategy
    • [KYUUBI #5725] Handle the session exception for all kyuubi operations during running
    • [KYUUBI #2503] Catch all exceptions for sessions timeout check task
    • [KYUUBI #5717] Infer the proxy user automatically for delete batch operation
    • [KYUUBI #5711] Support to build application state and error from pod container
    • [KYUUBI #5714] Fix NPE when deleting terminated application pod
    • [KYUUBI #5714] Support to cleanup the spark driver pod after application terminates for retain period
    • [KYUUBI #5696] Return remote session id and remote operation id
    • [KYUUBI #5654] Introduce new config to forcibly rewrite pod name
    • [KYUUBI #5438] Add common method to get session level config

    Spark Engine, Connectors and Plugins

    • [KYUUBI #5952] Disconnect connections without running operations after engine maxlife time graceful period
    • [KYUUBI #6018] Speed up GetTables operation for Spark session catalog
    • [KYUUBI #5966] Error occurs on clicking on the Spark UI Kyuubi Query Engine tab
    • [KYUUBI #5831] Pre-allocate array list capacity for TColumns and TColumnValues in TRowSet generation
    • [KYUUBI #5811] Reuse time formatters instance in value serialization of TRowSet generation
    • [KYUUBI #5766] Default spark.yarn.maxAppAttempts to 1 for spark engine
    • [KYUUBI #5750] Fix the inaccurate issue of obtaining COLUMN_SIZE in the decimal field jdbc of spark engine
    • [KYUUBI #5331] Spark engine should throw an exception when it fails to start
    • [KYUUBI #5991] Error on reading Atlas properties composed of multi values

    Other Engines

    • [KYUUBI #5977] Fix argument position on constructing Trino ClientSession
    • [KYUUBI #5969] Trino engine add default schema
    • [KYUUBI #5878] Hive engine should recognize hiveserver2-site.xml
    • [KYUUBI #5756] Introduce specified initialized SQL to every engine
    • [KYUUBI #5814] Skip the max-rows limitation if the statement is not DQL in Flink engine
    • [KYUUBI #5799] Fix fetch timeout in session conf doesn’t support ISO-8601 in Flink engine
    • [KYUUBI #5789] Flink engine kyuubi.session.engine.flink.fetch.timeout parameter on the server is not effective
    • [KYUUBI #5784] Implement HiveTBinaryFrontendService#RenewDelegationToken
    • [KYUUBI #5782] Flink Engine GetInfo support CLI_ODBC_KEYWORDS
    • [KYUUBI #5778] Hive engine shall respect hive.server2.enable.doAs
    • [KYUUBI #5777] Include hive-service-rpc in Hive engine jar
    • [KYUUBI #5507] Support Initialize SQL in Flink Engine
    • [KYUUBI #5280] Hive engine compatible with Hive 2.3

    Client

    • [KYUUBI #6036] JDBC driver conditional sets fetchSize on opening session
    • [KYUUBI #6016] Conditional call GetInfo CLI_ODBC_KEYWORDS to restore compatible with lower version Kyuubi and HS2
    • [KYUUBI #5961] Prevent NPE when checking ticket cache exists
    • [KYUUBI #5961] Support to specify client kerberosAuthType as fromTicketCache
    • [KYUUBI #5849] Incorrectly parse JDBC URL while variable includes colon
    • [KYUUBI #5822] Fix the usage msg of KyuubiBeeLine
    • [KYUUBI #5752] JDBC driver implements getDate, getTimestamp and getTime with Calendar in ResultSet
    • [KYUUBI #5713] Backport HIVE-27271: Client connection to HS2 fails when transportMode=http, ssl=true, sslTrustStore specified without trustStorePassword in the JDBC URL

    Build, Test and Misc

    • [KYUUBI #5981] Deploy Spark Hive connector with Scala 2.13 to Maven Central
    • [KYUUBI #6047] Free up disk space
    • [KYUUBI #5987] Always export KYUUBI_HOME in load-kyuubi-env.sh
    • [KYUUBI #5987] Enhance KYUUBI_HOME detection in shell script to handle softlink cases
    • [KYUUBI #6000] Modify the incorrect configuration file in the trino-cli documentation
    • [KYUUBI #5978] Canonicalize Trino IT in GitHub Action workflow
    • [KYUUBI #5962] Bump Flink 1.17.1 to 1.17.2
    • [KYUUBI #5953] Update NOTICE
    • [KYUUBI #5951] Improve doc for Session Conf Advisor
    • [KYUUBI #5933] Happy New Year 2024
    • [KYUUBI #5244] Corrected ENGINE_ALIVE_MAX_FAILURES conf version
    • [KYUUBI #5856] Bump Spark 3.4.2
    • [KYUUBI #5857] Dependencies CI check when dev/dependencyList changes
    • [KYUUBI #5837] Only enable Jacoco with explicit profile
    • [KYUUBI #5827] Fix wrong test code about directory lineage
    • [KYUUBI #5819] Fix rst syntax of missing blank line after code-block
    • [KYUUBI #5802] Add scala-2.13 as supported Scala profile to the docs of building from source
    • [KYUUBI #5801] Add spark-3.4 and spark-3.5 as supported Spark profiles to the docs of building from source
    • [KYUUBI #5785] Fix flaky test - JpsApplicationOperation with spark local mode
    • [KYUUBI #5737] Correct JDBC URL demo in Flink docs
    • [KYUUBI #5617] Make flaky test spnego batch rest client reliable
    • [KYUUBI #5720] Fix flaky test in TPCHQuerySuite
    • [KYUUBI #5697] Fix incorrect url reference in code and docs
    • [KYUUBI #5599] Correct the parameter name of fetching op log in RESTful API
    • [KYUUBI #5641] Bump Kyuubi 1.8.0 in Helm charts
    • [KYUUBI #5640] Upgrade playground to Kyuubi 1.8.0 and Spark 3.4.1
    • [KYUUBI #5637] Add known contributor translation
    • [KYUUBI #5624] Remove npm run build warning about NODE_ENV=production
    • [KYUUBI #5634] Smoother the icon animation on collapsing sidebar
    • [KYUUBI #5626] Fix sytle linting violation in web ui

    Credits

    Last but not least, this release would not have been possible without the following contributors:

    • Shaoyun Chen
    • Chao Chen
    • Flyangz
    • Pengqi Li
    • Bowen Liang
    • Paul Lin
    • Senmiao Liu
    • Xiao Liu
    • Peiyue Liu
    • Junjie Ma
    • Ocean22
    • Cheng Pan
    • Gianluca Principini
    • Zhiming She
    • SwordyZhao
    • William Tong
    • Fei Wang
    • Zhen Wang
    • Tao Wang
    • Zeyu Wang
    • Kang Wang
    • Binjie Yang
    • Kaifei Yi
    • David Yuan
    • Yuwei Zhan
    • He Zhao
    • Yifan Zhou
    • Mingliang Zhu
    • Yi Zhu
    • liaoyt
    • mrtisttt