Release Notes - 1.8.0

Highlights

• Fully verified with Java 8, 11, and 17, Scala 2.12, initial support Scala 2.13
• Fully verified with Spark 3.1, 3.2, 3.3, and 3.4, initial support Spark 3.5
• Fully verified with Flink 1.16, 1.17, and 1.18
• New Web UI (experimental)
• New Chat engine with OpenAI ChatGPT backend
• Major improvements of Spark Batch API, optimized for large-scale job submission (experimental)
• Major improvements of Spark AuthZ plugin, including enhanced Iceberg and DSv2 operators support
• Major improvements of Flink engine, including YARN Application Mode support
• Introducing Atlas hook support in Spark Lineage plugin
• Continuously Kyuubi Helm Chart & Spark on K8s improvements

Spark Engine, Connectors and Plugins

• [KYUUBI #1265] OPTIMIZE where clause expression support
• [KYUUBI #3420] Expose UI url on registering engine service
• [KYUUBI #3444] Support the planOnly mode of kyuubi spark engine support SQL lineage
• [KYUUBI #3929] Refactor lineage plugin to add LineageDispatcher
• [KYUUBI #4167] [AUTHZ] Introduce function support in PrivilegeBuilder with Serde layers
• [KYUUBI #4171] Support skip retrieving table’s properties to speed up GetTables operation
• [KYUUBI #4186] Spark showProgress with JobInfo
• [KYUUBI #4330] Non-temporary views do not resolve to a specific real table
• [KYUUBI #4331] [KYUUBI #4431] Lineage supports Union and MergeInto
• [KYUUBI #4415] Align the operation handle in server/engine for ExecuteScala, ExecutePython and PlanOnlyStatement
• [KYUUBI #4430] Lineage supports the lateral view explode statement
• [KYUUBI #4433] Fix the case with create a view from view
• [KYUUBI #4443] Do not set engine session init sql for alive probe session
• [KYUUBI #4450] Ignore unknown fields policyPriority when reading policy json file
• [KYUUBI #4488] [KSHC] Keep object original name defined in HiveBridgeHelper
• [KYUUBI #4493] [AUTHZ] Enable Tests for Spark 3.1 with iceberg tables
• [KYUUBI #4497] Add RowFilteringTestBase to improve the test coverage
• [KYUUBI #4504] [AUTHZ] Fix source table privilege requirement when querying permanent view in Spark 3.1 and below
• [KYUUBI #4525] [KSHC] Partitioning predicates should take effect to filter data
• [KYUUBI #4530] [AUTHZ] Support non-English chars for MASK, MASK_SHOW_FIRST_4, and MASK_SHOW_FIRST_4
• [KYUUBI #4532] [AUTHZ] Displays the columns involved in extracting the aggregation operator
• [KYUUBI #4541] Support to customize the attributes to expose for Spark engine
• [KYUUBI #4560] [KSHC] Support Kerberized HMS in cluster mode w/o keytab
• [KYUUBI #4585] [AUTHZ] Authz policy file generation
• [KYUUBI #4587] Take care of table cache query stage
• [KYUUBI #4592] Support eagerly kill redundant executors
• [KYUUBI #4615] Support stage level schedule for final write stage
• [KYUUBI #4617] [AUTHZ] Collect results for filtered show objects ahead to prevent holding unserializable spark plan
• [KYUUBI #4620] [KSHC] Cut off transitive dependencies
• [KYUUBI #4636] Improve eagerly kill redundant executors
• [KYUUBI #4641] Add MaxFileSizeStrategy to limit max scan file size
• [KYUUBI #4644] Manually terminate the Py4JServer during engine shutdown
• [KYUUBI #4658] [AUTHZ] Fix InsertIntoHiveDirCommand classname so that we can extract the query in it when authorization.
• [KYUUBI #4662] [ARROW] Arrow serialization should not introduce extra shuffle for outermost limit
• [KYUUBI #4664] Fix empty relation when kill executors
• [KYUUBI #4672] [AUTHZ] Remove Support for Apache Ranger 0.6.0
• [KYUUBI #4676] [AUTHZ] Reuse users and namespaces in both tests and policy file generation
• [KYUUBI #4677] [AUTHZ] Check generated policy file in test suite
• [KYUUBI #4678] Improve FinalStageResourceManager kill executors
• [KYUUBI #4681] Set thread CreateSparkTimeoutChecker daemon
• [KYUUBI #4683] Update spark.sql.finalWriteStage.resourceIsolation.enabled version
• [KYUUBI #4702] [ARROW] CommandResultExec should not trigger job
• [KYUUBI #4710] [ARROW] LocalTableScanExec should not trigger job
• [KYUUBI #4710] [ARROW] Post driver-side metrics for LocalTableScanExec/CommandResultExec
• [KYUUBI #4716] [KYUUBI #4715][AUTHZ] Fix the incorrect class name of InsertIntoHiveDirCommand in table spec generator
• [KYUUBI #4717] [AUTHZ] Check Authz plugin’s spec json files in UT
• [KYUUBI #4720] [ARROW] Fix java.lang.NoSuchFieldError: IpcOption.DEFAULT for Spark-3.1/3.2
• [KYUUBI #4741] Kyuubi Spark Engine/TPC connectors support Spark 3.4
• [KYUUBI #4754] [ARROW] Use KyuubiArrowConverters#toBatchIterator instead of ArrowConverters#toBatchIterator
• [KYUUBI #4797] [ARROW] Reflective calls to the function ArrowUtils#toArrowSchema
• [KYUUBI #4812] Generalize case transformation method for string type config entry
• [KYUUBI #4834] Reduce the scope of method references in Authz plugin cleanup shutdown hook
• [KYUUBI #4838] Fix spark operation exception leak in withLocalProperties method
• [KYUUBI #4866] Add annotation for Iceberg tests in Authz plugin
• [KYUUBI #4869] [AUTHZ] Introduce table extractor for ResolvedIdentifier in Spark 3.4
• [KYUUBI #4871] [AUTHZ] Adapt plan changes for CreateNamespace and SetCatalogAndNamespace in Spark 3.4
• [KYUUBI #4873] [AUTHZ] Refactor Authz reflection with kyuubi-util’s DynMethods
• [KYUUBI #4874] [AUTHZ] Improve methods in AuthzUtils
• [KYUUBI #4875] [AUTHZ] Remove checking Spark v2 in tests since Spark v2 not supported
• [KYUUBI #4878] Delta support for Spark-3.4
• [KYUUBI #4880] [BUILD] Skip compiling useless files when making distribution
• [KYUUBI #4888] [AUTHZ] Remove filtering results for ShowDatabasesCommand in Spark 2.x
• [KYUUBI #4892] [AUTHZ] Make identifier part name comparison case insensitive in tests of PrivilegeBuilder
• [KYUUBI #4895] [ARROW] Reflective calls to the function ArrowConverters#fromBatchIterator
• [KYUUBI #4899] [AUTHZ] Extract function from FunctionIdentifier for CreateFunction and DropFunction in Spark 3.4
• [KYUUBI #4903] [AUTHZ] Fix NoSuchElementException when listing database in CatalogImpl in Spark 3.4
• [KYUUBI #4909] [AUTHZ] Enable authz plugin tests for Spark 3.4
• [KYUUBI #4910] Extract table from ResolvedIdentifier for DropTable in Spark 3.4
• [KYUUBI #4913] Eliminate unnecessary output in ut “union an unmasked table”
• [KYUUBI #4914] [AUTHZ] Reuse extractor singleton instance with generalized getter for supported extractor types
• [KYUUBI #4915] Adapt database not exist error message change for Spark 3.4
• [KYUUBI #4916] [AUTHZ] Support ReplaceData and compatible Spark 3.4 and 3.5
• [KYUUBI #4916] Fix Iceberg merge into UT
• [KYUUBI #4917] [AUTHZ] Table owner undefined in Iceberg 1.3.0 on Spark 3.4
• [KYUUBI #4923] [ARROW] Update arguments of ArrowUtils#toArrowSchema function
• [KYUUBI #4925] Add default catalog using spark_catalog with the lineage result
• [KYUUBI #4935] More than target num of executors may survive after FinalStageResourceManager did kill
• [KYUUBI #4937] Cleanup spark catalog shim and renamed to catalog utils
• [KYUUBI #4952] Enhance AWS Glue default database does not exist error message on opening session
• [KYUUBI #4995] Use hadoop conf and hive conf from catalog options
• [KYUUBI #4999] [KSHC] Kyuubi-Spark-Hive-Connector support Apache Spark 3.4
• [KYUUBI #5002] Fail the engine fast when no incoming connection in CONNECTION mode
• [KYUUBI #5005] Remove default settings spark.sql.execution.topKSortFallbackThreshold
• [KYUUBI #5007] Remove blank SparkSimpleStatsReportListener
• [KYUUBI #5009] Pass Spark Engine Log Path to Spark Conf
• [KYUUBI #5017] [KSHC] Support Parquet/Orc provider is splittable
• [KYUUBI #5018] Make kyuubi spark extension compatible with Spark 3.4
• [KYUUBI #5022] [KSHC] CreateTable should use the correct provider
• [KYUUBI #5023] [KSHC] TableIdentify don’t attach catalog
• [KYUUBI #5025] Use formatDuration instead of formatDurationVerbose in Spark UI’s engine tab
• [KYUUBI #5028] Update session hadoop conf to catalog hadoop conf
• [KYUUBI #5030] Support get query id in Spark engine
• [KYUUBI #5035] Spark engine session page display session end time and duration
• [KYUUBI #5045] Initialize EXECUTOR_POD_NAME_PREFIX_MAX_LENGTH before using
• [KYUUBI #5055] [AUTHZ] Support building function privileges in Spark 3.4
• [KYUUBI #5073] Correct the method name in SparkSQLLineageParserHelperSuite
• [KYUUBI #5075] Refine tests to adapt Spark 4.0
• [KYUUBI #5076] Add KDF engine_url
• [KYUUBI #5105] GetColumns gets duplicate column names when Spark integrates with Iceberg
• [KYUUBI #5136] Spark App may hang forever if FinalStageResourceManager killed all executors
• [KYUUBI #5148] Improve spark.driver.host assignment in Spark on K8s client mode
• [KYUUBI #5170] Identifier compatible with Spark 3.4
• [KYUUBI #5172] [AUTHZ] Check USE permissions for DESCRIBE FUNCTION
• [KYUUBI #5180] Replace deprecated classes by equivalent in scala.tools.nsc.interpreter package
• [KYUUBI #5184] Rename Kyuubi’s StageInfo to SparkStageInfo to fix class mismatch
• [KYUUBI #5188] Make server module and Spark sql engine module compilable on Scala 2.13
• [KYUUBI #5189] [AUTHZ] Make spark authz module compilable on Scala 2.13
• [KYUUBI #5191] Make Spark extension plugin compilable on Scala 2.13
• [KYUUBI #5192] Make Spark sql lineage plugin compilable on Scala 2.13
• [KYUUBI #5193] Make Spark hive connector plugin compilable on Scala 2.13
• [KYUUBI #5195] Make Spark TPC-H connector plugin compilable on Scala 2.13
• [KYUUBI #5196] Enable CI Test on Scala 2.13 and support custom or spark-core extracted Scala version for Spark’s engine
• [KYUUBI #5196] Extract spark core scala version lazily and respect engine env
• [KYUUBI #5199] Read all columns of metadata to prevent column missing
• [KYUUBI #5202] Switch to Spark 3.4 by default
• [KYUUBI #5224] Modify the README file of authz to make spark 3.4 the default version
• [KYUUBI #5225] [KSHC] Unify the exception handling of v1 and v2 during dropDatabase
• [KYUUBI #5232] In SparkOperation#cleanup always calls cancelJobGroup even though it’s in the completed state
• [KYUUBI #5257] [LINEAGE] Adapt CreateTableAsSelect plan to Spark 3.5 changes
• [KYUUBI #5268] [AUTHZ] Extract method for enabling authorization in single call mode
• [KYUUBI #5269] Duplicated AnalyzesTablesCommand in kyuubi authz
• [KYUUBI #5271] AnalyzeTableCommand should also add table write privilege
• [KYUUBI #5291] Verify Spark engine is compatible with Spark 3.5.0
• [KYUUBI #5317] [KSHC] Hive Connector throws NotSerializableException on reading Hive Avro partitioned table
• [KYUUBI #5336] Spark extension supports Spark 3.5
• [KYUUBI #5339] Avoid using kyuubi.session.engine.initialize.timeout string
• [KYUUBI #5414] [KSHC] Reader should not pollute the global hiveConf instance
• [KYUUBI #5426] [KSHC] Avoid use class.newInstance directly
• [KYUUBI #5550] Optimizing TPC-DS dataset generation for 10x speedup

Flink Engine

• [KYUUBI #1652] Support Flink YARN application mode
• [KYUUBI #4367] Support Flink 1.17
• [KYUUBI #4387] Remove support for Flink 1.14
• [KYUUBI #4446] Fix connections blocked by Flink insert statements
• [KYUUBI #4495] Support Flink job management statements
• [KYUUBI #4517] Fix multiple executions lead to abnormal results on Flink 1.14
• [KYUUBI #4635] Support flink time type in query operation
• [KYUUBI #4666] Support flink varbinary type in query operation
• [KYUUBI #4745] Support Flink’s LocalZonedTimestamp DataType
• [KYUUBI #4806] Support time-out incremental result fetch for Flink engine
• [KYUUBI #4861] Support Flink session idleness
• [KYUUBI #4938] Implement Kyuubi UDF in Flink engine
• [KYUUBI #5080] Fix EmbeddedExecutorFactory not thread-safe during bootstrap
• [KYUUBI #5106] Improve logs for fatal errors
• [KYUUBI #5108] Fix FileNotFoundException during Flink engine bootstrap
• [KYUUBI #5160] Refactor getNextRowSetInternal to support fetch streaming data
• [KYUUBI #5190] Explicitly name Flink bootstrap SQL in application mode
• [KYUUBI #5200] Optimize Flink application name generating
• [KYUUBI #5222] Remove unused provided dependencies in Flink SQL engine
• [KYUUBI #5238] Fix credentials may break Flink engine launch command
• [KYUUBI #5315] Propagate HIVE_CONF_DIR on launching Flink engine
• [KYUUBI #5405] Support Flink 1.18
• [KYUUBI #5531] Fix flaky FlinkOperationOnYarnSuite by enlarging the max rows setting

Other Engines

• [KYUUBI #3374] Support password authentication for Trino engine
• [KYUUBI #4412] Align the server/engine session handle for flink/hive/trino/jdbc engines
• [KYUUBI #4491] Fix Trino typo
• [KYUUBI #4522] use:catalog should execute before than use:database
• [KYUUBI #4544] Initial implement Kyuubi Chat Engine
• [KYUUBI #4548] Kyuubi Chat Engine supports Chinese questions and HTTP proxy
• [KYUUBI #4554] Code improvement in ChatGPTProvider
• [KYUUBI #4556] Refactor ChatGPTProvider to use openai-java client
• [KYUUBI #4558] Make ChatGPT model ID configurable
• [KYUUBI #4559] Pretty chat engine launch command
• [KYUUBI #4711] JDBC client should catch task failed exception instead of NPE in the incremental mode
• [KYUUBI #4809] Set session user in ChatGPT request
• [KYUUBI #4810] Request and use a single choice for chat completion
• [KYUUBI #4881] JDBCEngine performs initialization sql
• [KYUUBI #4940] Implement Kyuubi UDF in Hive engine
• [KYUUBI #5120] Allow using session’s user and password for connecting database in JDBC engine
• [KYUUBI #5282] Support configure Trino session conf in kyuubi-default.conf
• [KYUUBI #5333] Adaptive call fetchLaunchEngineResult
• [KYUUBI #5570] Fix memory leak when using incremental collect mode in JDBC engine

Server

• [KYUUBI #3653] AdminResource add list Kyuubi server api
• [KYUUBI #3739] Remove unused parameters in SessionsResource rest docs
• [KYUUBI #3872] Skip setting proxy user when keytab is provided on batch session
• [KYUUBI #3876] Update the rest api docs for open session response
• [KYUUBI #3920] Fix flaky test - distribute lock
• [KYUUBI #3957] Rename the config prefix from kyuubi.server.batch.limit to kyuubi.server.limit.batch
• [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
• [KYUUBI #4344] Expose exec pool work queue size metrics
• [KYUUBI #4364] Add metrics for user opened connections with session type
• [KYUUBI #4376] Support to config the Kyuubi service administrator with Kyuubi conf
• [KYUUBI #4388] Limit the max rows for get nextRowSet api
• [KYUUBI #4404] Support to list/close sessions in AdminResource
• [KYUUBI #4407] Adapt SLF4J2
• [KYUUBI #4424] Catch No Node Exception, when list kyuubi engines
• [KYUUBI #4439] Add list/close operation method for AdminResource
• [KYUUBI #4462] Fix variable usage issue in SessionManager#stop
• [KYUUBI #4464] Simplify and improve log for JDBCMetadataStore
• [KYUUBI #4467] Tolerate Driver Pod ephemerally invisible after submitting
• [KYUUBI #4472] add session/operation methods in AdminRestApi
• [KYUUBI #4480] Engine alive probe should close thrift connection on engine lost
• [KYUUBI #4482] Support to disable batch resource upload function
• [KYUUBI #4483] Enable REST frontend protocol by default
• [KYUUBI #4496] Call super.clusterManager() when batchConf does not have spark.master
• [KYUUBI #4498] Support to list sessions/operations with conditions in AdminRestApi
• [KYUUBI #4501] Allow administrator to specify the user for engine admin
• [KYUUBI #4506] Return sessionType in session data
• [KYUUBI #4516] Return kyuubi instance in SessionData and OperationData
• [KYUUBI #4518] Return engine id in SessionData and post kyuubi instance in KyuubiOperationEvent
• [KYUUBI #4519] Update metadata when batch application state changed
• [KYUUBI #4523] support close engine session gracefully
• [KYUUBI #4526] Defer evaluation for GroupProvider
• [KYUUBI #4540] Support to deny some users to make connection
• [KYUUBI #4540] Create session limiter if user unlimited list or deny list nonempty
• [KYUUBI #4563] Format error log output in case of metrics json file not found
• [KYUUBI #4575] Fix the empty last line may causes the session to exit directly
• [KYUUBI #4579] Fix the unstable situation of test cases for Trino-Fronted-Service
• [KYUUBI #4584] Post the session exception to EventBus in time
• [KYUUBI #4594] Support PUT method in REST client
• [KYUUBI #4611] support all sessions api in SessionRestApi
• [KYUUBI #4618] Admin Resource list operations with sessionHandle filter
• [KYUUBI #4623] KubernetesApplicationOperation uses Informer instead of list
• [KYUUBI #4639] Support to specify confOverlay when executing statement with RESTful API
• [KYUUBI #4652] Fix JaasConfiguration ClassNotFoundException for Hadoop 3.3.4 and previous
• [KYUUBI #4653] [KYUUBI 4650] LogDivertAppender supports reading RollingFileAppender pattern
• [KYUUBI #4669] LDAP authentication allows auth user contains domain when bind.dn/pw enabled
• [KYUUBI #4688] Fix the failure to read the operation log after executing catalog and database operation
• [KYUUBI #4691] Configure FAIL_ON_UNKNOWN_PROPERTIES to false for KyuubiScalaObjectMapper
• [KYUUBI #4713] Fix false positive result in SchedulerPoolSuite
• [KYUUBI #4719] Support submission timeout for yarn application manager and get the applicationInfo in-memory
• [KYUUBI #4731] Support batch session conf advisor
• [KYUUBI #4733] Introduce Kafka event logger for server events
• [KYUUBI #4738] AdminResource.getEngineSpace should use primary group name on GROUP share level
• [KYUUBI #4739] Add operation lock instead of locking state Enumeration
• [KYUUBI #4746] Do not recreate async request executor if has been shutdown
• [KYUUBI #4749] Fix flaky test issues in SchedulerPoolSuite
• [KYUUBI #4753] KyuubiConf.unset should not log deprecation warning
• [KYUUBI #4767] Correct the submit time for BatchJobSubmission and check applicationInfo if submitted application
• [KYUUBI #4777] Deregister event handlers when stopping server with event handler made auto-closeable
• [KYUUBI #4780] Get engine application info with interval to prevent frequent call to resource manager
• [KYUUBI #4786] Support yarn-client and yarn-cluster for YarnApplicationOperation
• [KYUUBI #4790] Initial implement Batch V2
• [KYUUBI #4791] Add helper method to simplify REST enabled judgment
• [KYUUBI #4792] Enhance hardcode session keywords and remove unused code
• [KYUUBI #4796] Expose JVM attributes to metrics
• [KYUUBI #4798] Allows BatchJobSubmission to run in sync mode
• [KYUUBI #4801] Using different engine submit timeout config for kubernetes and yarn
• [KYUUBI #4807] Fix flaky test “basic batch rest client”
• [KYUUBI #4807] Retry more times to fix flaky test “spnego batch rest client”
• [KYUUBI #4811] Do not update app info after batch or application terminated
• [KYUUBI #4813] Add the fetch orientation parameter to the /v1/operations/:operationId/log interface
• [KYUUBI #4814] Introduce Apache Atlas hook support in lineage plugin
• [KYUUBI #4816] Correct the implementation of cleanup K8s terminated appInfo
• [KYUUBI #4829] Support to expose operation metrics
• [KYUUBI #4830] Fix flaky test “support to interrupt the thrift request if remote engine is broken”
• [KYUUBI #4835] Using hive conf to check whether to apply HIVE_DELEGATION_TOKEN
• [KYUUBI #4836] Set UncaughtExceptionHandler for thread to log exception
• [KYUUBI #4840] Return cached appInfo iif both op and app are terminated
• [KYUUBI #4841] Log more info for closing session
• [KYUUBI #4842] Fix typo in etcd client config
• [KYUUBI #4843] Support multiple kubernetes contexts and namespaces
• [KYUUBI #4847] Close the session immediately when engine corrupt
• [KYUUBI #4847] Close the session immediately when engine connection closed
• [KYUUBI #4850] Change default EC spec to secp521r1 for internal session variable signature
• [KYUUBI #4855] Change socket “connection refused” message match pattern
• [KYUUBI #4857] Fix flaky test TFrontendServiceSuite
• [KYUUBI #4858] Relax test message assertion in “sync query causes engine crash”
• [KYUUBI #4859] HttpException message should not be null
• [KYUUBI #4865] Fix flaky test BackendServiceMetricSuite
• [KYUUBI #4876] fix Flaky test BackendServiceMetricSuite (#4924)
• [KYUUBI #4889] Admin command line supports list server command
• [KYUUBI #4890] Return operation metrics in OperationData and fix typo
• [KYUUBI #4893] Prevent null collection for rest dto
• [KYUUBI #4898] Fix logOperation multiple read with missing line
• [KYUUBI #4919] Fix the flaky tests in the “open batch session” functionality within the “BatchesResourceSuite”
• [KYUUBI #4920] Add OperationRestApi
• [KYUUBI #4946] Alter the order of initLoggerEventHandler
• [KYUUBI #4949] For operation getNextRowSet method, with operation lock required
• [KYUUBI #4950] Migrate Kyuubi embedded database from Derby to SQLite
• [KYUUBI #4950] Misc improvement for usage of SQLite
• [KYUUBI #4950] SQLite schema files should start from 1.8.0
• [KYUUBI #4951] Add reload4j support
• [KYUUBI #4977] Log error message when REST API invocation error occurs
• [KYUUBI #4978] Fix flaky test: close expired operations
• [KYUUBI #4978] Fix flaky test: close expired operations
• [KYUUBI #4979] Fix flaky test: support to interrupt the thrift request if remote engine is broken
• [KYUUBI #4981] Refactor code of closeBatchSession
• [KYUUBI #4985] Fix log level restore behavior of KyuubiFunSuite#withLogAppender
• [KYUUBI #4986] Always use Files#deleteIfExists
• [KYUUBI #4994] Respect the engine type and share level for listing all engines
• [KYUUBI #4994] Support listing all engines
• [KYUUBI #4996] Support to refresh kubernetes configs dynamically
• [KYUUBI #5004] Fix typo for kubernetes allowed context and namespace check
• [KYUUBI #5010] Make Kyuubi server’s connection URL configurable
• [KYUUBI #5029] Close the alive probe session after engine session closed
• [KYUUBI #5036] Fix Operation.close not update complete timestamp
• [KYUUBI #5043] Destroy the build process when waitCompletion is false and the engine is running in cluster mode
• [KYUUBI #5050] Add admin checking in getProxyUser
• [KYUUBI #5056] Return more helpful error message to http client
• [KYUUBI #5063] Support to filter batch with batch name
• [KYUUBI #5065] Call destroy first on killing Spark startup process to allows it release temp files
• [KYUUBI #5065] Graceful close the process when launch engine timeout
• [KYUUBI #5078] Make kyuubi_instance nullable in metadata table schema
• [KYUUBI #5081] Minor refactor JDBCMetadataStore
• [KYUUBI #5089] Limit maximum engine startup concurrency of kyuubi server
• [KYUUBI #5130] Allow setting Zookeeper SPN using Kyuubi configuration
• [KYUUBI #5131] Create index on metastore.create_time
• [KYUUBI #5138] Fix flaky test BatchesResourceSuite - get batch session list
• [KYUUBI #5145] Change embedded Zookeeper server to method-local variable
• [KYUUBI #5158] Allow embedded Zookeeper binding IP address
• [KYUUBI #5165] Build Spark Driver/Executor Pod Name(Prefix) in process
• [KYUUBI #5185] Use Set collection for order-insensitive configs
• [KYUUBI #5187] Fix info message in UserFilter
• [KYUUBI #5196] Extract spark core scala version lazily and respect engine env
• [KYUUBI #5201] Allow disabling batch impl v2 on globally
• [KYUUBI #5204] Thrift HTTP FE checks req conf nullable before evaluating proxy user
• [KYUUBI #5206] Try to kill pod with label if no ApplicationInfo found to prevent pod leak
• [KYUUBI #5210] Cancel operation will cause the log file node to leak
• [KYUUBI #5213] Check config value by enum values
• [KYUUBI #5216] Workaround for negative counter in SessionLimiter
• [KYUUBI #5218] Improve logs of KubernetesApplicationOperation
• [KYUUBI #5220] Batch submitter should only block submitting stage
• [KYUUBI #5220] Batch submitted considers application state
• [KYUUBI #5243] Distinguish metadata between batch impl v2 and recovery
• [KYUUBI #5289] RESTful API should always print audit log
• [KYUUBI #5290] Batch impl v2 should pick batch jobs in FIFO
• [KYUUBI #5300] Batch v2 should not update kyuubi_instance when open session
• [KYUUBI #5305] Return dummy log when batch job before being added to session manager
• [KYUUBI #5306] YarnApplicationOperation supports proxy user
• [KYUUBI #5310] Batch session recovery should start after HTTP server getting started
• [KYUUBI #5327] Add priority field in batch session metadata table definition
• [KYUUBI #5328] Batch supports priority scheduling
• [KYUUBI #5340] AdminResourceSuite make engine idle time 3min
• [KYUUBI #5380] Create PySpark batch jobs tests for RESTful API
• [KYUUBI #5381] Change the default metrics reporter to Prometheus
• [KYUUBI #5391] Close batch session after batch operation completion for idle timeout
• [KYUUBI #5392] Add query timeout monitor on server-side in ExecuteStatement
• [KYUUBI #5408] MetadataManager tries MySQL 8 driver class first
• [KYUUBI #5412] Resolve the relative zk configuration dir based on KYUUBI_HOME
• [KYUUBI #5441] Make the configuration kyuubi.zookeeper.embedded.data.log.dir effective
• [KYUUBI #5451] Ignore NoSuchFileException during OperationLog.close()
• [KYUUBI #5499] [KYUUBI #2503] Catch any exception when closing idle session
• [KYUUBI #5513] Always redirect delete batch request to Kyuubi instance that owns batch session
• [KYUUBI #5522] Ignore main class for PySpark batch job submission
• [KYUUBI #5528] Remove duplicate properties tag and respect KYUUBI_LOG_DIR
• [KYUUBI #5566] InternalRestClient respects kyuubi.engine.security.enabled to add HTTP auth header
• [KYUUBI #5566] Check InternalSecurityAccessor is initialized only when kyuubi.engine.security.enabled is true
• [KYUUBI #5568] Check administrator only when security is enabled
• [KYUUBI #5568] Fix security enabled judgement
• [KYUUBI #5606] Handle engine listing request properly for users who have not created engine

Client

• [KYUUBI #3887] Fix kyuubiServerPrincipal logic in KyuubiCommands
• [KYUUBI #4305] Backport HIVE-15820: comment at the head of beeline -e
• [KYUUBI #4438] Render the batch info map in kyuubi-ctl
• [KYUUBI #4439] Add dto class for operation data
• [KYUUBI #4452] Strip the redundant leading and tailing slash of getZooKeeperNamespace.
• [KYUUBI #4479] Restore JDBC Kerberos authentication behavior for UGI.doAs
• [KYUUBI #4489] Fix missing org.apache.commons.logging by including jcl-over-slf4j dependency in REST client
• [KYUUBI #4492] Correct engine subdomain calculation in kyuubi-ctl
• [KYUUBI #4508] Beeline should reset stdin after consuming init SQL file
• [KYUUBI #4581] Add KyuubiCommands parse python snippets unit test
• [KYUUBI #4609] get engineRefId in KyuubiConnection
• [KYUUBI #4619] Fix beeline with -e when there are other SQL statements before the source statement, the source statement cannot be executed normally
• [KYUUBI #4657] Building rest client to kyuubi instance including original host urls
• [KYUUBI #4962] Backport HIVE-19048: init script errors are ignored
• [KYUUBI #4965] Beeline supports --python-mode option and remove comments for non-python mode
• [KYUUBI #4969] Run JUnit tests on beeline module
• [KYUUBI #5054] Refine the kyuubi client version properties file to prevent conflicts
• [KYUUBI #5100] Fix unpaired quote in message pattern
• [KYUUBI #5101] Remove unused StringBuilders in ResultSets
• [KYUUBI #5129] KyuubiBeeline should redirect JDK logging
• [KYUUBI #5179] Use Iterable instead of Seq in kyuubi-ctl commands
• [KYUUBI #5297] New RetryableClient get http://null server uri from metadata.kyuubiInstance
• [KYUUBI #5480] Support setting kyuubi hive jdbc client protocol version
• [KYUUBI #5564] Support to config ticket cache location in JDBC URL

Web UI

• [KYUUBI #3420] Kyuubi Server Proxy Engine UI
• [KYUUBI #3646] Front-end style should bracket same line
• [KYUUBI #3646] Init Session Statistic Page
• [KYUUBI #3648] Add Session Detail Page
• [KYUUBI #3650] Add Operation Statistics Page
• [KYUUBI #3652] Add Kyuubi Server Management page
• [KYUUBI #3654] Add Engine Manager Page
• [KYUUBI #4537] Enable Vite proxy server for web ui development
• [KYUUBI #4565] Fix out-date README.md about Development Project
• [KYUUBI #4622] Session api use admin api for security
• [KYUUBI #4626] Fix Web UI CI check fail with pnpm the lockfile needs updates
• [KYUUBI #4757] Move Statistics to Management Menu
• [KYUUBI #4982] Add query string to forward requests to Engine UI
• [KYUUBI #5259] Web UI supports Swagger pages
• [KYUUBI #5463] Adjust the file directory structure
• [KYUUBI #5484] Remove legacy Web UI
• [KYUUBI #5573] Delete parts of the Kyuubi Web UI that are not useful
• [KYUUBI #5586] Kyuubi Web UI show current version
• [KYUUBI #5608] Kyuubi version should always use the light color
• [KYUUBI #5620] Remove dummy information card from Overview page

Documentation

• [KYUUBI #4308] Make README more welcoming
• [KYUUBI #4312] Fix over-matched README pattern for markdown styling
• [KYUUBI #4367] Kyuubi requires Flink 1.16/1.17
• [KYUUBI #4387] Update Flink version requirements
• [KYUUBI #4568] Make kyuubi-ctl doc enable variable ‘release’ automatic substitution
• [KYUUBI #4583] Rename Flink Table Store to Apache Paimon (Incubating)
• [KYUUBI #4593] Keep promote the release step update-to-date
• [KYUUBI #4599] Simplify project version evaluation
• [KYUUBI #4607] Rename Flink Table Store to Apache Paimon (Incubating) in docs Connectors for Hive SQL Query Engine
• [KYUUBI #4608] Rename Flink Table Store to Apache Paimon (Incubating) in docs Connectors for Trino SQL Query Engine
• [KYUUBI #4614] Rename Flink Table Store to Apache Paimon (Incubating) in docs Connectors for Flink SQL Query Engine
• [KYUUBI #4624] Fix table headers in kyuubi_vs_hive.md
• [KYUUBI #4627] Fix a typo in rest_api.md.
• [KYUUBI #4655] Enrich docs for Kyuubi Hive JDBC driver
• [KYUUBI #4655] Recover the missing JDBC URL
• [KYUUBI #4722] Kyuubi requires Spark 3.1 and above now
• [KYUUBI #4727] kyuubi-spark-lineage has no transitive deps
• [KYUUBI #4734] Fix typo in docs of custom event handler
• [KYUUBI #4742] Add docs for Flink application mode
• [KYUUBI #4763] Fix the Kyuubi JDBC kerberos parameters
• [KYUUBI #4800] Update readthedocs.yaml
• [KYUUBI #4864] Fix grammar in HA section
• [KYUUBI #4933] Mark spark.sql.optimizer.insertRepartitionNum config for Spark 3.1 only
• [KYUUBI #4954] Fix variable reference in quick_start_with_jdbc
• [KYUUBI #4960] Fix doc for pyhive client.
• [KYUUBI #5049] PyHive Kerberos usage doc
• [KYUUBI #5083] Add LDAP document
• [KYUUBI #5085] Update config section of deploy on kubernetes
• [KYUUBI #5094] Add Python JayDeBeApi doc
• [KYUUBI #5099] Document link failure
• [KYUUBI #5118] Add Flink KDF doc
• [KYUUBI #5122] Hive KDF usage
• [KYUUBI #5127] Improvements for markdown builder
• [KYUUBI #5146] Fix link of IntelliJ IDEA Setup Guide
• [KYUUBI #5153] Minor change w/ to with
• [KYUUBI #5154] Move configuration docs to the top level
• [KYUUBI #5162] Add statements about contributions co-authored with AIGC in developer guide page
• [KYUUBI #5178] Make MarkdownBuilder not extending Growable
• [KYUUBI #5227] Added deploy mode info
• [KYUUBI #5247] Correct building command for Spark authz/lineage plugins
• [KYUUBI #5273] Update release docs
• [KYUUBI #5275] Improve and fix comparison and regeneration for golden files
• [KYUUBI #5294] Update supported dialects for JDBC engine
• [KYUUBI #5335] Set markdown file EOL
• [KYUUBI #5345] Fix non-kerberized code description in quick_start_with_jdbc
• [KYUUBI #5347] Fix java code errors in quick_start_with_jdbc
• [KYUUBI #5409] Update config docs regeneration hints in developer guide
• [KYUUBI #5432] Fix typo in README.md
• [KYUUBI #5523] Update the Kyuubi supported components version

Building, Infrastructure and Dependency

• [KYUUBI #4078] Fix shellcheck violations in scripts of /bin folder
• [KYUUBI #4187] Windows build-info script supports revision time
• [KYUUBI #4278] Use new Apache ‘closer.lua’ syntax to obtain Maven
• [KYUUBI #4295] Introduce super-linter action for linting JSON, XML, ENV files and bash_exec
• [KYUUBI #4299] Bump JUnit to 4.13.2
• [KYUUBI #4299] Bump MySQL JDBC driver to 8.0.32
• [KYUUBI #4315] Check modules available before build step in dependency workflow
• [KYUUBI #4347] Bump maven download plugin from 1.6.6 to 1.6.8
• [KYUUBI #4348] Cache engine archives in CI jobs for maven download plugin
• [KYUUBI #4397] build/dist supports --web-ui
• [KYUUBI #4413] Build docker image should include web UI
• [KYUUBI #4447] Bump log4j from 2.19.0 to 2.20.0
• [KYUUBI #4448] Bump Netty from 4.1.87.Final to 4.1.89.Final
• [KYUUBI #4451] Skip etcd test if no docker env
• [KYUUBI #4453] Bump Kubernetes Client Version to 6.4.1
• [KYUUBI #4461] Add Kyuubi Kubernetes IT Module to CI style check
• [KYUUBI #4466] Use bitnami/minio docker image
• [KYUUBI #4471] Prompted command for docs generation should skip Java tests
• [KYUUBI #4484] [K8S][HELM] Set specific Kyuubi version to the chart appVersion
• [KYUUBI #4494] bin/kyuubi should use exec to run Kyuubi server
• [KYUUBI #4513] Bump Kyuubi 1.7.0 in Playground
• [KYUUBI #4546] Fully exclude metrics dir from git
• [KYUUBI #4557] [K8S][HELM] Kyuubi server should bind Pod IP by default
• [KYUUBI #4572] Bump Iceberg from 1.1.0 to 1.2.0
• [KYUUBI #4576] Multi arch build for ARM support in Docker Hub
• [KYUUBI #4589] [K8S][HELM] Update template on port-forward usage guide NOTES.txt
• [KYUUBI #4590] Bump delta from 2.2.0 to 2.3.0
• [KYUUBI #4598] Bump Flink-1.15 from 1.15.3 to 1.15.4
• [KYUUBI #4600] Bump Iceberg 1.2.0 for playground
• [KYUUBI #4601] Bump Hadoop 3.3.5 for playground
• [KYUUBI #4605] [K8S][HELM] Set IfNotPresent pullPolicy by default
• [KYUUBI #4606] [K8S][HELM] Add command and args configuration support
• [KYUUBI #4615] Bump Ranger from 2.3.0 to 2.4.0
• [KYUUBI #4628] Add Dependency management ISSUE_TEMPLATE
• [KYUUBI #4632] Add rule for labeler to tag module:ui
• [KYUUBI #4638] Update dependency.yml as label in not permitted in markdown block
• [KYUUBI #4652] Upgrade Hadoop from 3.3.4 to 3.3.5
• [KYUUBI #4695] Bump super-linter action from v4 to v5
• [KYUUBI #4696] Upgrade scalafmt from 3.7.1 to 3.7.3
• [KYUUBI #4697] [K8S][HELM] Add template rendering info to README
• [KYUUBI #4698] [K8S][HELM] Centralize Kyuubi labels definition
• [KYUUBI #4706] [K8S][HELM] Use template comments for the chart license header
• [KYUUBI #4707] [K8S][HELM] Fix README typo and minor reformatting
• [KYUUBI #4712] Bump Spark from 3.2.3 to 3.2.4
• [KYUUBI #4737] Restore Project & Community Status in README.md
• [KYUUBI #4744] Remove Hudi integration tests
• [KYUUBI #4747] Install and cache build/mvn in CI builds
• [KYUUBI #4768] Guidelines for Document and Code contributions
• [KYUUBI #4772] Bump Jersey from 2.39 to 2.39.1
• [KYUUBI #4783] Use pnpm v8 and update pnpm lockfile to v6 schema
• [KYUUBI #4784] Use nodejs v18 in CI builds and maven web-ui builds
• [KYUUBI #4788] [K8S][HELM] Use StatefulSet instead of Deployment
• [KYUUBI #4805] Bump openai-java from 0.11.1 to 0.12.0
• [KYUUBI #4808] Bump Jetty from 9.4.50 to 9.4.51
• [KYUUBI #4818] [K8S][HELM] Update default Kyuubi version to 1.7.1
• [KYUUBI #4819] [K8S][HELM] Refactor liveliness and readiness probes
• [KYUUBI #4824] Bump Jackson from 2.14.2 to 2.15.0
• [KYUUBI #4827] Bump Apache Arrow from 11.0.0 to 12.0.0
• [KYUUBI #4828] Exclude macOS tar extended metadata in build/dist
• [KYUUBI #4849] Open modules to enable JDK 17 support
• [KYUUBI #4851] Bump Maven 3.8.8
• [KYUUBI #4852] Switch to Kyuubi Shaded Zookeeper
• [KYUUBI #4856] Add restore-keys for action/cache
• [KYUUBI #4870] Add kyuubi-util and kyuubi-util-scala modules
• [KYUUBI #4877] Bump Flink 1.17 to 1.17.1
• [KYUUBI #4879] Refactor and promote reflection utils and cleanup similar reflection methods
• [KYUUBI #4883] Bump Iceberg 1.3.0
• [KYUUBI #4887] Refactor and add ut for ClassUtils
• [KYUUBI #4891] Bump Flink 1.16.2
• [KYUUBI #4894] Bump Mockito from 4.9.0 to 4.11.0 and ScalaTest from 3.2.15 to 3.2.16
• [KYUUBI #4904] Move AssertionUtils to kyuubi-util-scala module
• [KYUUBI #4905] Generalize util method for loading class from service loader
• [KYUUBI #4907] Add a maven profile ‘remote-debug’
• [KYUUBI #4908] Bump scalafmt from 3.7.3 to 3.7.4
• [KYUUBI #4911] Remove duplicate test annotations org.apache.kyuubi.tags.* in kyuubi-common
• [KYUUBI #4912] Replace Scala’s assert in tests with scalatest’s for prettified error message
• [KYUUBI #4931] Bump vite from 3.0.0 to 4.2.3 and vitest from 0.22.0 to 0.32.0
• [KYUUBI #4937] Remove redundant quoteIfNeeded method
• [KYUUBI #4943] Bump Guava from 31.1 to 32.0.1
• [KYUUBI #4944] Code improvement for Java
• [KYUUBI #4947] Bump pnpm version in maven’s frontend plugin to 8.6.1 for lockfileVersion 6.1
• [KYUUBI #4948] Reuse versions of NodeJS and pnpm from maven properties
• [KYUUBI #4959] Code improvements for Scala
• [KYUUBI #4966] Bump Apache parent pom from 29 to 30
• [KYUUBI #4968] Simplify Option conversion
• [KYUUBI #4970] Unified reflection methods invokeAs and getField
• [KYUUBI #4974] Eliminate raw-types compilation warning in KyuubiBeeLine
• [KYUUBI #4975] Fixed run test link in PR template
• [KYUUBI #4990] Bump Spark 3.4.1
• [KYUUBI #4992] Bump Netty 4.1.93.Final
• [KYUUBI #5001] Bump Scala from 2.12.17 to 2.12.18
• [KYUUBI #5003] [K8S][HELM] Add doc strings to the chart values.yaml
• [KYUUBI #5007] Bump scalafmt from 3.7.4 to 3.7.5
• [KYUUBI #5019] Shows details of compilation info in SparkUI’s engine tab
• [KYUUBI #5024] Bump grpc-protobuf from 1.48.0 to 1.53.0
• [KYUUBI #5034] Bump tough-cookie from 4.0.0 to 4.1.3 in /kyuubi-server/web-ui
• [KYUUBI #5039] Use semantic versions and remove redundant version comparison methods
• [KYUUBI #5058] [K8S][HELM] Allow the user to set the priorityClass used by Kyuubi server Pod
• [KYUUBI #5059] Upgrade kubernetes client version to 6.7.2
• [KYUUBI #5064] Audit kubernetes application state change in separate log files
• [KYUUBI #5068] Bump Spark in spark-master profile from 3.5.0-SNAPSHOT to 4.0.0-SNAPSHOT
• [KYUUBI #5070] Bump word-wrap from 1.2.3 to 1.2.4 in web UI
• [KYUUBI #5072] Fix KyuubiOperationWithEngineSecuritySuite and related issues
• [KYUUBI #5090] Fix AllKyuubiConfiguration to generate redundant blank lines in Windows
• [KYUUBI #5113] Remove dummy Spark Kudu connector module
• [KYUUBI #5116] Bump Hadoop 3.3.6
• [KYUUBI #5117] Code improvements for Java and Scala code
• [KYUUBI #5121] Extract dev scripts for regenerating the golden files
• [KYUUBI #5132] [K8S][HELM] Kyuubi add server metrics monitor
• [KYUUBI #5135] Exclude web-ui coverage reports in rat plugins
• [KYUUBI #5149] Correct error message of ReflectUtils’s invokeAs when method not found
• [KYUUBI #5150] Bump Spark 3.3.3
• [KYUUBI #5152] Check milestone and assignees when merging pull request
• [KYUUBI #5177] Use Scala binary version placeholder in Maven module’s artifactId suffix
• [KYUUBI #5183] Introducing profile scala-2.13 for development use
• [KYUUBI #5196] Enable CI Test on Scala 2.13 and support custom or spark-core extracted Scala version for Spark’s engine
• [KYUUBI #5202] Update upload_nexus_staging
• [KYUUBI #5205] Upgrade kubernetes-client to 6.8.1
• [KYUUBI #5209] Reformat Scala source code crossing versions
• [KYUUBI #5212] Fix configuration errors causing by helm charts of prometheus services
• [KYUUBI #5214] Correct file path for scala tests in util-scala module
• [KYUUBI #5234] Upgrade Jetty version from 9.4.51.v20230217 to 9.4.52.v20230823
• [KYUUBI #5235] Daily deploy snapshot version on master and branch-1.7
• [KYUUBI #5250] Bump Iceberg from 1.3.0 to 1.3.1
• [KYUUBI #5252] Remove incubator link
• [KYUUBI #5253] Upgrade testcontainers-scala from 0.40.12 to 0.41.0
• [KYUUBI #5253] Supply testcontainers-scala-scalatest deps
• [KYUUBI #5293] Upgrade snakeyaml from 1.33 to 2.2
• [KYUUBI #5305] Fix get local log UT
• [KYUUBI #5308] Ensure release using Java 8
• [KYUUBI #5318] Pin maven-source-plugin 3.2.1
• [KYUUBI #5358] Bump iceberg 1.4.0
• [KYUUBI #5365] Don’t use Log4j2’s extended throwable conversion pattern in default logging configurations
• [KYUUBI #5425] Add thread name and ID to log4j2-defaults.xml
• [KYUUBI #5435] Improve Kyuubi On Kubernetes IT
• [KYUUBI #5443] Add kyuubi-kubernetes-it module to reformat script
• [KYUUBI #5444] Check style with profiles of Spark 3.4 and 3.5
• [KYUUBI #5483] Fix deploy command for Scala 2.13 TPC-DS/H connectors
• [KYUUBI #5483] Release Spark TPC-H/DS Connectors with Scala 2.13
• [KYUUBI #5486] Bump Kafka client version from 3.4.0 to 3.5.1
• [KYUUBI #5508] Upgrade download-maven-plugin from 1.6.8 to 1.7.1
• [KYUUBI #5544] Release contains Spark 3.5 extension
• [KYUUBI #5561] Binary distribution artifact ships database schema scripts
• [KYUUBI #5561] Rename folder to db-scripts
• [KYUUBI #5569] Fix pnpm-lock file version

Credits

Last but not least, this release would not have been possible without the following contributors:

• Deng An
• GitHub Bot
• CavemanIV
• Fu Chen
• Shaoyun Chen
• Chao Chen
• Derick Cypho
• Yang Du
• FourSpaces
• Xu Guo
• Karsonnel
• Denis Krivenko
• Kyofin
• Pengqi Li
• Xieming Li
• Guanhua Li
• Bowen Liang
• Tianlin Liao
• Hanna Liashchuk
• Paul Lin
• Senmiao Liu
• Peiyue Liu
• Naijie Liu
• Chenliang Lu
• Ruilei Ma
• Omkar Makhare
• Drake Youngkun Min
• Sidhant Nagpal
• Cheng Pan
• QianyongY
• Anurag Rajawat
• Rojin Raju
• Fei Wang
• Zhen Wang
• Kang Wang
• Miao Wang
• Bruce Wong
• XorSum
• Junjie Xu
• Binjie Yang
• Remzi Yang
• Kent Yao
• Xianxun Ye
• Kaifei Yi
• Xiduo You
• Ruguo Yu
• David Yuan
• Qiang Zhang
• Yaodong Zhang
• Yangming Zhao
• He Zhao
• Min Zhao
• Yifan Zhou
• Yi Zhu
• Yaogai Zhu
• Alex Zou
• Junjie Zu
• biangjuang
• camper42
• fantasticKe
• huangzhir
• huapan123456
• liaoyt
• mans2singh
• mrtisttt
• phyyou
• rhh777
• runzhliu
• touchida
• uicosp
• wangyepeng2
• weixi62961
• yehere