Release Notes - 1.11.0

Highlight

• Fully verified with Java 8, 11, 17, and 21, on Scala 2.12 and 2.13
• Fully verified with Apache Spark 3.3 to 4.0, initial support Spark 4.1, support for Spark 3.3 and 3.4 is deprecated.
• Fully verified with Apache Flink 1.17 to 1.20, support for Flink 1.17 to 1.19 is deprecated.

Changelogs

• [KYUUBI #7289] Release requires Java 17 or 21
• [KYUUBI #7288] Bump Maven 3.9.12
• [KYUUBI #7284] Upgrade Flink and Spark to latest patched version
• [KYUUBI #7287] Fix checkstyle workflow
• [KYUUBI #7245] Fix arrow batch converter error
• [KYUUBI #7281] Support to enable asynchronous post event
• [KYUUBI #7277] Add UUID v7 generator
• [KYUUBI #7275] Kyuubi Spark extension supports Spark 4.1
• [KYUUBI #7254] Initial support Spark 4.1
• [KYUUBI #7274] Bump Scala 2.13.17
• [KYUUBI #7273] Bump Netty 4.2.7
• [KYUUBI #7259] Bump Netty 4.1.128 gRPC 1.76.2 Protobuf 3.25.8 Guava 33.4.8-jre
• [KYUUBI #7272] Improve build/dependency.sh script
• [KYUUBI #6884][FOLLOWUP] Support reassign batch with BatchRestApi
• [KYUUBI #6884][FOLLOWUP] Fix internal kyuubi instance ping failure
• [KYUUBI #7226] Support to wait the batch recovery appliction submission to throttle the load on the system
• [KYUUBI #7186] Introduce RuleFunctionAuthorization for persistent function calls authorization
• [KYUUBI #7266] Upgrade jersey version to 2.47
• [KYUUBI #7265] Recommend to configure kyuubi.session.local.dir.allowlist for multi-tenant clusters
• [KYUUBI #7261] Bump extra-enforcer-rules 1.11.0
• [KYUUBI #7260] Suppress rat-plugin deprecate warnings
• [KYUUBI #7257] Bump Jackson 2.20.1 and kubernetes-client 6.14.0
• [KYUUBI #7256] Enable authZ compile support for Spark 4.0 and refactor some test methods
• [KYUUBI #7248] JDBC engine should cancel the statement when receive cancel operation
• [KYUUBI #7251] Use RustFS instead of MinIO in Playground
• [KYUUBI #7250] Fix JDBCMetadataStore Scala 2.13 compile error
• [KYUUBI #7241] Transfer engine pool name to lower case
• [KYUUBI #7237] Replace recommonmark with myst
• [KYUUBI #7238] [DOC] Replace ‘Getting Started with Hive JDBC’ markdown with rst
• [KYUUBI #7224] [DOC] Fix “duplicate label usage” warnings
• [KYUUBI #7225] [CI] Skip Python Client unit tests when only documentation changes are made
• [KYUUBI #7229] Fix countMetadata if the params is empty
• [KYUUBI #7209] Bump mysql image to v8 for authZ plugin test suite on iceberg
• [KYUUBI #7222] [DOC] Fix “document isn’t included in any toctree” warnings
• [KYUUBI #7221] [DOC] Fix nonexisting document ‘client/advanced/features/engine_pool’
• [KYUUBI #7223] [CI] Skip running tests for documentation only changes
• [KYUUBI #7220] Proactive cleanup kubernetes terminated application info
• [KYUUBI #7219] Fix endless swagger openapi.json security items
• [KYUUBI #7217] Support customized session protocol version to support binary type
• [KYUUBI #7215] Make all BatchUtils variables to be final
• [KYUUBI #7214] Fix kubernetes container state
• [KYUUBI #7213] Enable test of Spark lineage plugin for Spark 4.0
• [KYUUBI #7190] Fix Presto SQLAlchemy dialect did not implement get_view_names
• [KYUUBI #7171] Fix empty list engine result when etcd is used as the service registry
• [KYUUBI #7205] Use Apache closer.lua syntax for kyuubi-download to obtain engine archives
• [KYUUBI #7211] [DOC] Remove unsupported ’logo_only’ and ’extra_navbar’ theme options
• [KYUUBI #7172][FOLLOWUP] kyuubi-spark-lineage-shaded artifactId should include scala.binary.version
• [KYUUBI #7212] [DOC] Use unordered list for Appendixes page
• [KYUUBI #7208] Pyhive remove thrift results columns assert
• [KYUUBI #7202] [AUTHZ] make FilteredShowObjectsExec extends V2CommandExec to avoid running executor side task
• [KYUUBI #7203] Bump Iceberg 1.10.0
• [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine
• [KYUUBI #7176] Cleanup metadata with batch size and interval
• [KYUUBI #7198] Remove support for Spark 3.2
• [KYUUBI #7199] Deprecate support for Flink 1.17 and 1.18
• [KYUUBI #7183][LINEAGE] Support collect all input tables by the plan
• [KYUUBI #7197] Bump Spark 4.0.1
• [KYUUBI #7196] Bump Maven 3.9.11
• [KYUUBI #7194] Speed up K8s integration tests in CI
• [KYUUBI #7174] Kyuubi Spark Extention supports Spark 4.0
• [KYUUBI #7175] Enhance the performance for metadata cleanup by leveraging create_time index
• [KYUUBI #7172] Shade kyuubi spark lineage module
• [KYUUBI #7153] Share JAAS configuration for Zookeeper client to avoid server OOM
• [KYUUBI #7126][LINEAGE] Support merge into syntax in row level catalog
• [KYUUBI #7168] Adapt PermanentViewMarker introduced by authz plugin in lineage plugin
• [KYUUBI #7164] Bump Byte Buddy 1.17.6
• [KYUUBI #7163][SPARK] Check whether engine context stopped in engine terminating checker
• [KYUUBI #7165] Remove banned action-shellcheck
• [KYUUBI #7157] Bump form-data from 4.0.0 to 4.0.4 in /kyuubi-server/web-ui
• [KYUUBI #7162] Bump kafka-clients 3.9.1
• [KYUUBI #7138] Respect kyuubi.session.engine.spark.initialize.sql set by client in shared engine mode
• [KYUUBI #7158] Spark engine respects session-level idle timeout threshold
• [KYUUBI #7160] Use single try-with-resources blocks in docs example
• [KYUUBI #7144] Remove direct usage of sun.misc.Signal
• [KYUUBI #7147] Enforce to use Kyuubi’s PlainSASLServer for PLAIN mechanism
• [KYUUBI #7151] Bump commons-lang3 3.18.0
• [KYUUBI #7148] Fix spark.kubernetes.file.upload.path permission
• [KYUUBI #7141] Support to get spark app url with pattern http://{{SPARK_DRIVER_POD_IP}}:{{SPARK_UI_PORT}}
• [KYUUBI #7139] Fix Spark extension rules to support RebalancePartitions
• [KYUUBI #7129] Support PARQUET hive table pushdown filter
• [KYUUBI #7132] Respect kyuubi.session.engine.startup.waitCompletion for wait engine completion
• [KYUUBI #7135] Fix cannot access /tmp/engine-archives: No such file or directory
• [KYUUBI #7131] Print cached engine archives
• [KYUUBI #7118] docs(client): Add comprehensive docs for Engine Pool feature
• [KYUUBI #7122] Support ORC hive table pushdown filter
• [KYUUBI #7121] Improve operation timeout management with configurable executors
• [KYUUBI #6876] Fix hadoopConf for autoCreateFileUploadPath
• [KYUUBI #7109] Ignore the ? in backticks
• [KYUUBI #7112] Enhance test ‘capture error from spark process builder’ for Spark 4.0
• [KYUUBI #7113] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory
• [KYUUBI #6924] Upgrade Spark Ranger plugin to 2.6.0
• [KYUUBI #7100] [#7099] Ranger Support Check Iceberg Alter Table Command & Change Iceberg Test Use JDBC Catalog
• [KYUUBI #7110] Fix serverOnlyPrefixConfigKeys is iterator issue
• [KYUUBI #7105] [K8S][HELM] Support additional labels for PrometheusRule
• [KYUUBI #7076] Update known_translations
• [KYUUBI #7106] Make response.results.columns optional
• [KYUUBI #6884] [FEATURE] Support to reassign the batches to alternative kyuubi instance in case kyuubi instance lost
• [KYUUBI #7101] Load the existing pods when initializing kubernetes client to cleanup terminated app pods
• [KYUUBI #7103] Bump Delta 4.0.0 and enable Delta tests for Spark 4.0
• [KYUUBI #7104] Bump Maven 3.9.10
• [KYUUBI #7098] Add ability to annotate pods and headless service in Kyuubi helm chart
• [KYUUBI #7072][FOLLOWUP] Fix engine startup permit Grafana panel unit
• [KYUUBI #7095] Respect terminated app state when building batch info from metadata
• [KYUUBI #7094] Add serverOnly flag for metrics config items
• [KYUUBI #7097] [INFRA] Write sorted authors for release contributors
• [KYUUBI #7093] Log the metadata cleanup count
• [KYUUBI #7072] Expose metrics of engine startup permit state
• [KYUUBI #7077] Spark 3.5: Enhance MaxScanStrategy for DSv2
• [KYUUBI #7068] Iceberg ranger support check branch and tag ddl
• [KYUUBI #6870][FOLLOWUP] Correct file name of grafana/README.md
• [KYUUBI #6928] Bump Spark 4.0.0
• [KYUUBI #7073] Retry 3 times on deploying to nexus
• [KYUUBI #7063] Bump Kyuubi Shaded 0.5.0
• [KYUUBI #7065] [#7066] Iceberg Support add partition field check
• [KYUUBI #6920][FOLLOWUP] Spark SQL engine supports Spark 4.0
• [KYUUBI #7062] Bump Delta Lake 3.3.1
• [KYUUBI #7055] Support to filter out server only configs with prefixes
• [KYUUBI #7051] Fix usage without sslTrustStore of JDBC driver
• [KYUUBI #7054] Add server only flag for more server/credentials/frontend/metadata configs
• [KYUUBI #6926] Add SERVER_LOCAL engine share level
• [KYUUBI #7048] Fix KeyError when parsing unknown Hive type_id in schema inspection
• [KYUUBI #7033] Treat YARN/Kubernetes application NOT_FOUND as failed to prevent data quality issue
• [KYUUBI #7028] Persist the kubernetes application terminate state into metastore for app info store fallback
• [KYUUBI #7046] Bump Dropwizard metrics version to 4.2.30
• [KYUUBI #7045] Expose jetty metrics
• [KYUUBI #7044] Bump jetty version to 9.4.57.v20241219
• [KYUUBI #7043] Support to construct the batch info from metadata directly
• [KYUUBI #7034][FOLLOWUP] Decouple the kubernetes pod name and app name
• [KYUUBI #7035] Close the operation by operation manager to prevent operation leak
• [KYUUBI #7041][FOLLOWUP] Fix build for SparkOnKubernetesTestsSuite
• [KYUUBI #7041] Fix NPE when getting metadtamanager in KubernetesApplicationOperation
• [KYUUBI #6828][FOLLOWUP] Fix NPE in KyuubiBaseResultSet::getBigDecimal
• [KYUUBI #7036] [DOCS] Improve docs for kyuubi-extension-spark-jdbc-dialect
• [KYUUBI #7034] [KUBERNETES] Prefer to use pod spark-app-name label as application name than pod name
• [KYUUBI #7025] [KYUUBI #6686][FOLLOWUP] Prefer terminated container app state than terminated pod state
• [KYUUBI #7032] Remove Umbrella/Subtask issue template
• [KYUUBI #7026] Audit the kubernetes pod event type and fix DELETE event process logical
• [KYUUBI #7027] Support to initialize kubernetes clients on kyuubi server startup
• [KYUUBI #7023] Upgrade kubernetes client version to 6.13.5
• [KYUUBI #7022] Update announcement mail template to contain download links
• [KYUUBI #7015] Record the session disconnected info into kyuubi session event
• [KYUUBI #7017] Using mutable JettyServer uri to prevent batch kyuubi instance mismatch
• [KYUUBI #7011] Set kyuubi session engine client after opening engine session successfully
• [KYUUBI #7009] Backport HIVE-26723: Configurable canonical name checking.
• [KYUUBI #7008] Backport HIVE-27817: Disable ssl hostname verification for 127.0.0.1
• [KYUUBI #7005] [DOC] Remove empty page “Getting Started with Jupyter Lap”
• [KYUUBI #7004] Include FastXML Jackson into authZ shaded jar
• [KYUUBI #6979] Support check paimon system producers
• [KYUUBI #7003] Cut out JNA dependencies for authZ plugin
• [KYUUBI #7000] Exclude aws-java-sdk-logs from kyuubi-spark-authz-shaded
• [KYUUBI #6999] Keep JNA in authz-shaded with Scala 2.13
• [KYUUBI #6998] [TEST] Harness SparkProcessBuilderSuite
• [KYUUBI #6997] Get the latest batch app info after submit process terminated to prevent batch ERROR due to engine submit timeout
• [KYUUBI #6990] Add rebalance before InsertIntoHiveDirCommand and InsertIntoDataSourceDirCommand to align with behaviors of hive
• [KYUUBI #6785] Shutdown the executor service in KubernetesApplicationOperation and prevent NPE
• [KYUUBI #6984] Fix ValueError when rendering MapType data
• [KYUUBI #6989] Calculate expected join partitions based on scanned table size
• [KYUUBI #6988] [INFRA] Foward GitHub discussions to ASF mailing list
• [KYUUBI #6941] Test Add new Column for paimon
• [KYUUBI #6951] Test changing column type
• [KYUUBI #6986] [DOC] Fix multiple Pygments lexer name issues
• [KYUUBI #6987] [DOC] Fix Unknown target name issues
• [KYUUBI #6983] Remove support for spark.sql.watchdog.forcedMaxOutputRows
• [KYUUBI #6981] [DOC] Fix nested lists
• [KYUUBI #6964] Fix typos in serveral docs
• [KYUUBI #6973][AUTHZ] Support Paimon DELETE FROM / UPDATE / MERGE INTO commands
• [KYUUBI #6975] Clean up code for Spark 3.5 extension
• [KYUUBI #6977] [DOC] Remove empty note block
• [KYUUBI #6947] Test Dropping Columns
• [KYUUBI #6972] [KYUUBI 6971]Test Table Query&Batch Time Travel
• [KYUUBI #6970] [DOC] Fix “nonexisting document” issues
• [KYUUBI #6969] [DOC] Fix “Title underline too short” issues
• [KYUUBI #6965] Bump Scala 2.13.16
• [KYUUBI #6948] Test Changing Column Comment
• [KYUUBI #6958] Test INSERT TABLE
• [KYUUBI #6960] Bump log4j version to 2.24.3 to fix ConcurrentModificationException in LoggerContext
• [KYUUBI #6936] Test RenameTable command
• [KYUUBI #6949] Test adding column position
• [KYUUBI #6940] Test Unset Table Properties Command
• [KYUUBI #6950] Test changing column position
• [KYUUBI #6939] Bump Spark 3.5.5
• [KYUUBI #6942] Test Rename Column Name for paimon
• [KYUUBI #6908] Connection class ssl context object paramater
• [KYUUBI #6938] [DOC] Refine monitoring docs
• [KYUUBI #6923] Test Create Partitioned Table for Paimon
• [KYUUBI #6932] Test ALTER TBLPROPERTIES for Paimon
• [KYUUBI #6674] Bump Scalafmt to 3.9.x
• [KYUUBI #6921][AUTHZ] Test CTAS for Paimon
• [KYUUBI #6925] Only run Paimon authz tests with Scala 2.12
• [KYUUBI #6920] Spark SQL engine supports Spark 4.0
• [KYUUBI #6918] Cache client ipAddress in kyuubi jdbc connection
• [KYUUBI #6914] Bump Iceberg 1.8.0
• [KYUUBI #6905] PyHive HTTP/HTTPS dialect to use the database name from url
• [KYUUBI #6912][LINEAGE] Properly handle empty attribute set on mergeRelationColumnLineage
• [KYUUBI #6916] Bump Delta 3.3.0 for Spark 3.5
• [KYUUBI #6917] Bump Hudi 1.0.1
• [KYUUBI #6915] Fix ClickHouse integration tests
• [KYUUBI #6901] Default policy for spark
• [KYUUBI #6814] [UI] Bump cross-spawn from 7.0.3 to 7.0.6
• [KYUUBI #6900] [DOCS] Correct spelling errors in ’large_query_results’ part
• [KYUUBI #6891] Fix get existing gauge issue
• [KYUUBI #6875] [K8S][HELM] Add Hadoop configuration files support
• [KYUUBI #6876] Support rolling spark.kubernetes.file.upload.path
• [KYUUBI #6883] Using withOauthTokenProvider instead of withOauthToken to support token refresh
• [KYUUBI #6843] [FOLLOWUP] Fix ‘query-timeout-thread’ thread leak
• [KYUUBI #6843] Fix ‘query-timeout-thread’ thread leak
• [KYUUBI #6870] Add more metrics for grafana dashboard
• [KYUUBI #6866][FOLLOWUP] Prevent register gauge conflicts if both thrift binary SSL and thrift http SSL enabled
• [KYUUBI #6866] Add metrics for SSL keystore expiration time
• [KYUUBI #6867] Refactor the grafana dashboard template
• [KYUUBI #6864][FOLLOWUP] Support to apply instance with existing labels
• [KYUUBI #6864] Support to return prometheus metrics with instance label
• [KYUUBI #6861] Configuration guide of structured logging for Kyuubi server
• [KYUUBI #6862] Spark 3.3: MaxScanStrategy supports DSv2
• [KYUUBI #6859] Exclude log4j12 from hive engine module classpath
• [KYUUBI #6615] Make Jetty sending server version in response configurable
• [KYUUBI #6857] Spark 3.4: MaxScanStrategy supports DSv2
• [KYUUBI #5834] Add Grafana dashboard template
• [KYUUBI #6842] Bump Spark 3.5.4
• [KYUUBI #6848] Fix Gluten CI
• [KYUUBI #6845] Support additional labels for PodMonitor to be discovered by Prometheus
• [KYUUBI #6840] Fix PodMonitor pods selection
• [KYUUBI #6726][FOLLOWUP] Fix compilation on scala-2.13
• [KYUUBI #6846] Enable gluten CI for kyuubi apache repo
• [KYUUBI #6839] Add example for service monitor in Helm chart
• [KYUUBI #6829] Add metrics for batch pending max elapse time
• [KYUUBI #6836] Ship kafka-clients in binary distribution tarball without compression libs
• [KYUUBI #6838] [INFRA] Update archive mailing list address
• [KYUUBI #6837] [INFRA] Disable K8s CI image GHA cache
• [KYUUBI #6815] JDBC Engine supports Oracle
• [KYUUBI #6828] Clean up and improve error message for KyuubiBaseResultSet
• [KYUUBI #6826] Bump log4j from 2.24.1 to 2.24.2
• [KYUUBI #6722] Fix AppState when Engine connection is terminated
• [KYUUBI #6521] [K8S][HELM] Implement new configuration approach
• [KYUUBI #6726] Support trino stage progress
• [KYUUBI #6818] Upgrade mysql jdbc version to 8.4.0
• [KYUUBI #6821] Restore the plain PULL_REQUEST_TEMPLATE
• [KYUUBI #6820] Explicitly disable attach-scaladocs for pure Java modules
• [KYUUBI #6813] [BUILD] Replace java.version 1.8 with 8
• [KYUUBI #6807] [BUILD] Bump maven download plugin from 1.8.1 to 1.11.3
• [KYUUBI #6773] Bump commons-codec from 1.15 to 1.17.1
• [KYUUBI #6765] [CI] speed up the script for freeing disk space by batching removals
• [KYUUBI #6806] [K8S][HELM] Headless service renders ports for enabled protocols only
• [KYUUBI #6804] Bump Iceberg from 1.6.1 to 1.7.0
• [KYUUBI #6774] Bump log4j from 2.20.0 to 2.24.1
• [KYUUBI #6808] [BUILD]Replace systemProperties with systemPropertyVariables in maven-surefire-plugin
• [KYUUBI #6805] [K8S][HELM] Update default Kyuubi version to 1.10.0
• [KYUUBI #6778] [BUILD] allow frontend plugin not to inherits Maven’s http proxy config
• [KYUUBI #6786] Skip repeated checks on convert function in TColumnGenerator
• [KYUUBI #6790] Fix engine cannot exit when gracefully stopped
• [KYUUBI #6787] Improve the compatibility of queryTimeout in more version clients
• Normalize local path in Kyuubi server
• [KYUUBI #7001] Bump nanoid from 3.3.6 to 3.3.11 in /kyuubi-server/web-ui
• [KYUUBI #6967] Bump axios from 1.7.4 to 1.8.2 in /kyuubi-server/web-ui

Credits

Last but not least, this release would not have been possible without the following contributors:

• 1358035421
• Akira Ajisaka
• Alex Wojtowicz
• Bowen Liang
• Cheng Pan
• Chenliang Lu
• cutiechi
• David Yuan
• Denis Krivenko
• echo567
• eric9204
• Fei Wang
• Frank Bertsch
• gabry-lab
• GitHub Bot
• He Zhao
• Igor Khrol
• Jan Willem
• jasonj
• Joao Amaral
• John Zhang
• Kent Yao
• Lennon Chin
• moelhoussein
• naive-zhang
• namanagraw
• Octavian Ciubotaru
• Pat Buxton
• paul8263
• Peiyue Liu
• pionCham
• Reese Feng
• ruanwenjun
• Senmiao Liu
• shuyouZZ
• taylor12805
• Tian Bao
• wangzhigang1999
• Wenjun Ruan
• wenxin-cn
• Xu Guang Lv
• Yaguang Jia
• z1131392774
• Zhen Wang
• zhifanggao
• Ziyi Wu