Releases Archive

Release Notes - 1.5.1-incubating

2022 Apr 21

ChangeList

• [KYUUBI #2354] Fix NPE in process builder log capture thread
• [KYUUBI #2296] Fix operation log file handler leak
• [KYUUBI #2266] The default value of frontend.connection.url.use.hostname should be set to true to be consistent with previous versions
• [KYUUBI #2255]The engine state of Spark’s EngineEvent is hardcoded with 0
• [KYUUBI #2008][FOLLOWUP] Support engine type and subdomain in kyuubi-ctl
• [KYUUBI #2156][FOLLOWUP] Fix configuration format in document
• [KYUUBI #2156] Change log to reflect exactly why getting token failed
• [KYUUBI #2134] Respect Spark bundled log4j in extension modules
• [KYUUBI #2150] [DOCS] Fix Getting Started With Kyuubi on Kubernetes

Credits

Last but not least, this release would not have been possible without the following contributors:

• Cheng Pan
• Fei Wang
• HYBG
• Nick-0723
• cxzl25
• jiaoqingbo
• ulysses

Release Notes - 1.5.0-incubating

2022 Mar 25

Highlight

• Support Flink SQL Engine (Beta)
• Support Trino/ Presto engine (Beta)
• Support Running Scala for Spark SQL Engine
• Upgrade Hive Service RPC Protocol to 3.1.2
• Engine Launch Process Switch to Async by default
• Add Session Conf Provider as Developer API
• Correctness - Fix float precision loss for Spark SQL
• Migrate from log4j1 to log4j2

ChangeList

• [KYUUBI #1936][FOLLOWUP] Send credentials when opening session and wait for completion
• [KYUUBI #2115][1.5] Update license and enhance collect_licenses script
• [KYUUBI #2125] closeSession should avoid sending RPC after openSession fails
• [KYUUBI #2103] [1.5] Revert “[KYUUBI #1948] Upgrade thrift version to 0.16.0”
• [KYUUBI #1866][FOLLOWUP] Add logging of Flink SQL Engine
• [KYUUBI #1866][DOCS] Add flink sql engine quick start
• [KYUUBI #2070][FLINK] Support Flink job submission on yarn-session mode
• [KYUUBI #2089] Add debugging instructions for Flink engine
• [KYUUBI #1948][FOLLOWUP] Relocate fb303 classes
• [KYUUBI #2078] logCaptureThread does not catch sparksubmit exception
• [KYUUBI #1936] Send credentials when opening session and wait for completion
• [KYUUBI #2079] Update kyuubi layer source file to add flink and trino…
• [KYUUBI #2075] Using thread-safe FastDateFormat instead of SimpleDateFormat
• [KYUUBI #2063] Fix engine idle timeout lose efficacy for Flink Engine
• [KYUUBI #1215][DOC] Document incremental collection
• [KYUUBI #2043] Upgrade log4j/2.x/ to 2.17.2
• [KYUUBI #2060] Clear job group for init SQL
• [KYUUBI #2054] [KYUUBI-1819] Support closing Flink SQL engine process
• [KYUUBI #2055] correct the log service name
• [KYUUBI #2047] Support more MySQL JDBC driver versions
• [KYUUBI #2012] rename EventLoggingService and creates an correct directory
• [KYUUBI #2042] [KYUUI #2036] Redirect Issues/PR Notifications to notifications@kyuubi.apache.org
• [KYUUBI #2039] Remove LegacyFlinkOperationSuite
• [KYUUBI #1961] Upgrade hive-service-rpc to 3.1.2
• [KYUUBI #2008] Support engine type and subdomain in kyuubi-ctl
• [KYUUBI #2006] Improve the readability of ServiceControl list/get operations
• [KYUUBI #1999] Update quick_start_with_helm.md
• [KYUUBI #2000][DEPS] Bump Hadoop 3.3.2
• [KYUUBI #2004] Sync contents for CONTRIBUTING & COMMUNITY between web and main repo
• [KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf
• [KYUUBI #1986] Apache Kyuubi(Incubating) Maturity Self Assessment
• [KYUUBI #1988] Tune initialization of vars depend on system level confs
• [KYUUBI #1992] Remove SPARK_HOME predication.
• [KYUUBI #1960] Skip creating sparksession and starting engine if the max initialization time exceeds
• [KYUUBI #1958] Fix flaky test: statement redact for engine tab
• [[KYUUBI #1983] [KYUUBI #1806][FOLLOWUP] Fix LaunchEngine may be closed multiple times
• [KYUUBI #1981] Remove unused artifacts for common and parent module
• [KYUUBI #1972] Fix flaky test: SchedulerPoolSuite: Scheduler pool
• [KYUUBI #1969] Fix race on some service during start and stop phase
• [KYUUBI #1979] Simplify Kyuubi Event
• [KYUUBI #1975] [BUILD] Run TPC-DS test using the default Spark version
• [KYUUBI #1974] Support merge small files in multi insert statement
• [KYUUBI #1963] Make plan only mode skippable for configuable plans
• [KYUUBI #1970] Replace mirror-cn profile with Apache officially suggested dlcdn
• [KYUUBI #1948][FOLLOWUP] Remove TFramedTransport in extension-spark-3-1 and extension-spark-3-2
• [KYUUBI #1965] [DOCS] Use the newly added official logo at asf svn in README
• [KYUUBI #1948] Upgrade thrift version to 0.16.0
• [KYUUBI #1964] [BUILD] Update scalatest.exclude.tags
• [KYUUBI #1883] Support max result rows for Flink queries
• [KYUUBI #1920] Skip the Plan parsing of UseStatement
• [KYUUBI #1944] Upgrade Spark 3.1.3
• [KYUUBI #1957] Skip html comments in merge commit test body from PR desc
• [KYUUBI #1947] Revamp Github Action settings for CI
• [KYUUBI #1950] Remove ambiguous SPARK_HADOOP_VERSION
• [KYUUBI #1949] [INFA] Add homepage for every module in maven central
• [KYUUBI #1946] Close the kyuubi connection on launch engine operation failure
• [KYUUBI #1918] Build and test on Linux ARM64
• [KYUUBI #1945] [BUILD] Remove spark-hadoop from matrix to simplify workflow
• [KYUUBI #1940] [TEST] Fix EngineSecurityAccessorSuite
• [KYUUBI #1876][FOLLOWUP] Update iceberg.name
• [KYUUBI #1937] Build: Use Spark 3.2 for cross version verification
• [KYUUBI #1939] Build: Bump Jacoco maven plugin 0.8.7
• [KYUUBI #1796] Destroy the process after the engine starts
• [KYUUBI #1931] AbstractSession._lastIdleTime should be reset in acquire method
• [KYUUBI #1919] Add more enum values for OperationModes
• [KYUUBI #1930] Export TRINO_ENGINE_HOME in load-kyuubi-env script
• [KYUUBI #1926] Remove useless constants in Constants
• [KYUUBI #1923] Add GetFunctions for trino engine
• [KYUUBI #1906] Add GetColumns for trino engine
• [KYUUBI #1914] Update metric doc
• [KYUUBI #1867] Support PlanOnlyStatement operation like Spark SQL engine
• [KYUUBI #1911] Add GetTypeInfo for trino engine
• [KYUUBI #1876] Set Spark 3.2.1 as default
• [KYUUBI #1536] Add session conf advisor
• [KYUUBI #1899] Bump Iceberg 0.13.1
• [KYUUBI #1887] Add GetTables for trino engine
• [KYUUBI #1909] [DOCS] Remove unnecessary docs link in readme
• [KYUUBI #1886] Add GetSchemas for trino engine
• [KYUUBI #1905] log4j2-defaults.properties is not working since log4j 2 is always intialized by default
• [KYUUBI #1908] Codecov for flink/trino/beeline/jdbc modules
• [KYUUBI #1903] Add GetTableTypes for trino engine
• [KYUUBI #1901] [GA] Fix checkout version
• [KYUUBI #1898] Enhance release guide add more information about how to…
• [KYUUBI #1896] Make Query Details and Failure Reason sortable on the Kyuubi Query Engine page
• [KYUUBI #1811] TPC-DS benchmark expose cli args breakdown and results-dir
• [KYUUBI #1893] Bump Delta 1.0.1
• [KYUUBI #1880] Fixed: forcedMaxOutputRows extension for join
• [KYUUBI #1889] Improve the user experience of the configuration parameters of the document
• [KYUUBI #1885] Add GetCatalogs for trino engine
• [KYUUBI #1821] Add trino ExecuteStatement
• [KYUUBI #1884] Fix compile SQL measurement
• [KYUUBI #1816] Implement KyuubiHistoryServerPlugin
• [KYUUBI #1873] Fix incorrect applicationId
• [KYUUBI #1878] Fix typo in log4j2-test.properties
• [KYUUBI #1838] Clean up query results after query operations finish
• [KYUUBI #1868] Upgrate netty from 4.1.68 to 4.1.73
• [KYUUBI #1872] Make ProcBuilder decouple with YARN
• [KYUUBI #1875] Upgrade codahale.metrics.version to 4.2.8 and its deps
• [KYUUBI #1870] Upgrate k8s client from 5.5.0 to 5.12.1
• [KYUUBI #1871] Upgrade minikube action from 2.4.2 to 2.4.3
• [KYUUBI #1804][FOLLOWUP][GA] Avoid push docker image for forked repos
• [KYUUBI #1844] Add step for generating release notes
• [KYUUBI #1849] Bump Iceberg 0.13.0 and enable Iceberg test with Spark 3.2
• [KYUUBI #1861] Migrate bind-api and activation-api from javax to jakarta
• [KYUUBI #1848] Bump Spark 3.2.1
• [KYUUBI #1769][FOLLOWUP] Exclude log4j 12 deps from spark-hive and fix detect deps change workflow
• [KYUUBI #1769][FOLLOWUP] Revert the dependency change for kyuubi-hive-jdbc(shaded) modules
• [KYUUBI #1846] Add improvement issue template
• [KYUUBI #1858] Rename test log4j2 properties to log4j2-test.properties
• [KYUUBI #1850] Bump Prometheus 0.14.1
• [KYUUBI #1855] Suppress logging to console in scalatest
• [KYUUBI #1847] Bump Flink 1.14.3
• [KYUUBI #1829] Support executing statement async for Flink SQL engine
• [KYUUBI #1852] Fix UT property of log4j2 conf
• [KYUUBI #1769][FOLLOWUP] Some cleanups for log4j properties
• [KYUUBI #1839] Minor refactor OperationUtil
• [KYUUBI #1813] Enable the internal secure authentication between Kyuubi server and engine
• [KYUUBI #1769] [BUILD] Migrate from log4j1 to log4j2
• [KYUUBI #1832] Fixed: forcedMaxOutputRows extension for subquery
• [KYUUBI #1836] [RELEASE] Enhance release guide
• [KYUUBI #1840] Minor fix for general_incubator_vote script
• [KYUUBI #1834] [BUILD] Add support of external mvn with default mvn jvm opts when making distribution
• [KYUUBI #1835] Fix error to helm manifest
• [KYUUBI #1786] Fix the Spark sql engine logger level changed to WARN issue
• [KYUUBI #1685] Support jar and lib start option
• [KYUUBI #1778] Support Flink Set/Reset Operations
• [KYUUBI #1825] Generate appName for Flink applications
• [KYUUBI #1831] [DOCS] Fix the jdbc connection url format doc
• [KYUUBI #1610] Rest frontend service jetty server has started at null
• [KYUUBI #1806] Fix LaunchEngine may be closed multiple times
• [KYUUBI #1698] Add kyuubi server operation state rate
• [KYUUBI #1808] Update the link in the PULL_REQUEST_TEMPLATE
• [KYUUBI #1800] Remove oom hook
• [KYUUBI #1780] Log error message on REST endpoints
• [KYUUBI #1784][FOLLOWUP][TRINO] Fix float types lose precision
• [KYUUBI #1784][FOLLOWUP][FLINK] Support float type
• [KYUUBI #1797] Only exposing jdbc url for server side
• [KYUUBI #1804]Avoid nightly build for forked repos
• [KYUUBI #1764] Reuse the workingDir in ProcBuilder
• [KYUUBI #1699] Improve codahale metric
• [KYUUBI #1784] Fix float types lose precision
• [KYUUBI #1787] Export FLINK_ENGINE_HOME in load-kyuubi-env script
• [KYUUBI #1774] Export FLINK_HOME in load-kyuubi-env script
• [KYUUBI #1775] Add kyuubi.server.name configuration
• [KYUUBI #1785] Stop Discovery services properly
• [KYUUBI #1643][FOLLOWUP] Fix Flink GetFunctions result schema
• [KYUUBI #1788] Update greetings.yml
• [KYUUBI #1776] Improve EnginePage
• [KYUUBI #1648] Implement Flink GetTypeInfo operation
• [KYUUBI #1771] Change rate unit in JsonReporterService
• [KYUUBI #1696] Add fetch logs and results rows rate
• [KYUUBI #1704] Complete Flink-Hive data type mapping
• [KYUUBI #1645] Implement Flink engine GetSchemas operation
• [KYUUBI #1714] Add executeScala api for KyuubiStatement
• [KYUUBI #1756] Clean up code of FlinkOperation
• [KYUUBI #1766] [TEST] Remove SessionManagerSuite to reduce duplicated tests
• [KYUUBI #1763] [INFRA] Remove scala installation inbuild/mvn and using warn level for the ci
• [KYUUBI #1643] Implement GetFunctions operation
• [KYUUBI #1761] [SubTask][THTTP] Make abstract TFrontendService for thrift http protocol support
• [KYUUBI #1757] Extract workingDir from engine specific process builde…
• [KYUUBI #1510] Add TRINO engine type and TrinoProcessBuilder
• [KYUUBI #1748] bin/kyuubi-zk-cli should has execute permission
• [KYUUBI #1759] Remove duplicate close operation api
• [KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
• [KYUUBI #1738] Unify kubernetes integration test
• [KYUUBI #1660] Add restart option for bin/kyuubi
• [KYUUBI #1746] Move sparkEngineMajorMinorVersion into SparkQueryTests to make HiveJDBCTestHelper spark free
• [KYUUBI #1743] Fix parallelism of DataGenerator and other enhancements
• [KYUUBI #1740] Use testcontainers scalatest in trino module
• [KYUUBI #1745] [KYUBBI #1744] Fix trino client flaky test
• [KYUUBI #1737] [WIP][KYUUBI #1731][FEATURE] Kyuubi server should not fail to start when Hadoop FileSystem class not found
• [KYUUBI #1716] Unify Hive deps for server and client
• [KYUUBI #1729] Extract java lint workflow to a single yml and disable it in other jobs
• [KYUUBI #1735] [CI] Add current ip address, long hostname and short hostname to /etc/hosts
• [KYUUBI #1736] Bump up GA actions/checkout to latest (v2.3.5)
• [KYUUBI #1732] Reduce spark driver memory configuration to protect fa…
• [KYUUBI #1730] Use getHostName instead of getCanonicalHostName
• [KYUUBI #1672] Add VersionInfo in REST API
• [KYUUBI #1726] Extract dep workflow to a single yml and disable it in other jobs
• [KYUUBI #1719] javax.activation:activation:jar duplicates
• [KYUUBI #1720] Extract scalastyle workflow to a single yml and disable it in other jobs
• [KYUUBI #1646] support flink get tables
• [KYUUBI #1723] Create greetings.yml
• [KYUUBI #1722] [NOTICE] Update NOTICE for 2022
• [KYUUBI #1721] Recover maven deplop plugin in all module
• [KYUUBI #1709] Introduce a IT module for Kyuubi and FlinkSQL Engine t…
• [KYUUBI #1718] Extract rat workflow to a single yml and disable it in other jobs
• [KYUUBI #1717] Extract rat configurations from pom to file
• [KYUUBI #1715] For kyuubi-beeline, using kyuubi-hive-jdbc instead of kyuubi-hive-jdbc-shaded
• [KYUUBI #1712] Bump up maven.version to 3.8.4
• [KYUUBI #1710] Support to specify OPERATION_LANGUAGE with TExecuteStatementReq confOverlay
• [KYUUBI #1647] Change GetCatalog returned schema key name to support JDBC specification
• [KYUUBI #1690] Checking whether launchEngineOpHandle is null for KyuubiConnection:: getExecLog
• [KYUUBI #1697] BackendService method avg call times and rate
• [KYUUBI #1681] Extend Flink ExecuteStatement to support more operations
• [KYUUBI #1705] Support decimal type for Flink SQL Engine
• [KYUUBI #1691] Implement api: /${version}/operations/${operation_identifier}/rowset
• [KYUUBI #1641] Add Trino client
• [KYUUBI #1647] Implement GetTableTypes operation
• [KYUUBI #1477] Use KyuubiHadoopUtils.newHadoopConf instead of new Con…
• [KYUUBI #1678] Fixed: The value of hasResultSet is always set to true in Kyuubi Server
• [KYUUBI #1679] Remove invalid column and field number check
• [KYUUBI #1673] [BUILD] Improve log4j configurations
• [KYUUBI #1568] [FOLLOWUP] Add getSessionEvent in session
• [KYUUBI #1675][FEATURE] Redact secret information from SparkSQLEngine log
• [KYUUBI #1674] Uncache cached tables when session closed
• [KYUUBI #1654][Bug] Flaky OperationsResourceSuite
• [KYUUBI #1665][FOLLOWUP] Rename FlinkEngineProcessBuilder to FlinkProcessBuilder to align with SparkProcessBuilder
• [KYUUBI #1594] SparkProcessBuild submit add –conf spark.driver.host=
• [KYUUBI #1665] Rename FlinkEngineProcessBuilder to FlinkProcessBuilder to align with SparkProcessBuilder
• [KYUUBI #1656] Fix dist failed with –spark-provided
• [KYUUBI #1658] Revamp swagger ui dependencies
• [KYUUBI #1662] Turn up maven options in mvn script
• [KYUUBI #1597] fix row value in Query Execution should be queryExecution
• [KYUUBI #1649] Determine the yarn command in stop-application.sh.
• [KYUUBI #1634][FEATURE] Redact secret information from SparkSQLEngine UI
• [KYUUBI #1639] Make ApiRequestContext provide KyuubiRestFrontendService directly
• [KYUUBI #1629] Flink backend implementation
• [KYUUBI #1631] Migrating existing rest fe tests to real cases
• [KYUUBI #1556] Upgrade swagger-ui version from 4.1.0 to 4.1.3
• [KYUUBI #1614] Fix hardcoded version in OpenAPIDefinition
• [KYUUBI #1623] KyuubiSessionEvent shall always have Id
• [KYUUBI #1618] Correct hadoop version in RELEASE file
• [KYUUBI #1581] Add SchemaHelper for trino engine
• [KYUUBI #1620] Implement api: /${version}/operations/${operation_identifier}/log
• [KYUUBI #1613] Record the engine’s applicationId in KyuubiSessionEvent
• [KYUUBI #1611] Fix swagger-ui redirect wrong to other service address
• [KYUUBI #1601] Align SparkStatementEvent to KyuubiOperationEvent
• [KYUUBI #1592] [TEST][ICEBERG][DELTA] Introduce row level operation test for data lake format
• [KYUUBI #1582] Use ClientTypeSignature to further analysis of trino column type
• [KYUUBI #1607] [DOC] Add operation and backend service metrics in metrics doc
• [KYUUBI #1579] Implement basic ability of executing statement in Flink engine
• [KYUUBI #1575] Implement api: /${version}/operations/${operation_identifier}/resultsetmetadata
• [KYUUBI #1593] use user set host or ip instead of read hostname from user set
• [KYUUBI #1591] Watchdog support for Spark-3.2
• [KYUUBI #1587] [SUB-TASK][KPIP-2] Bump Flink 1.14.2
• [KYUUBI #1577] Add DropIgnoreNonexistent Rule.
• [KYUUBI #1586] Add time metric on each KyuubiBackendService method
• [KYUUBI #1578] Introduce flink-sql-client dependency to reduce some duplicated code
• [KYUUBI #1499] Introduce DataFrameHolder for cli result fetching
• [KYUUBI #1550] Provide a specific user guide about connecting to kerberized kyuubi
• [KYUUBI #1555][FLINK] Bump flink version to 1.14.0
• [KYUUBI #1554] Add kyuubi.engine.pool.name configuration
• [KYUUBI #1460] Bump Hudi 0.10.0
• [KYUUBI #1559] Bump Jersey 2.34
• [KYUUBI #1568] Replace sessionDetail with sessionEvent
• [KYUUBI #1565] Move time functions to RowSetUtils and move schema to spark package
• [KYUUBI #1567] [Bug] Expose HADOOP_CONF_DIR to Kyuubi beeline to fix “GSS initiate failed” error
• [KYUUBI #1571] [BUILD] Correct build command in build/dist
• [KYUUBI #1558] Bump Jackson 2.12.5
• [KYUUBI #1556] Shorten kyuubi.session.engine.check.interval to one minute
• [KYUUBI #1560] Align all Jackson libs version
• [KYUUBI #1539] Replace operationDetail with KyuubiEvent
• [KYUUBI #1557] [DOC] The TTL Of Kyuubi Engines
• [KYUUBI #1174][FOLLOWUP] change configuration(ha.zookeeper.publish.configs) default value to false
• [KYUUBI #1515] Add RowSet for trino engine
• [KYUUBI #1525][FLINK] Implement GetCatalogs operation and do the further initialization
• [KYUUBI #1548] Get docs version from project.version
• [KYUUBI #1544] Update the mailing list to send announcement
• [KYUUBI #1542] Move FetchIterator into kyuubi-common for multi-engine support
• [KYUUBI #1541] Remove keys sync step for Kyuubi finalize release
• [KYUUBI #1533] Add documentation about Kyuubi docker-image-tool using Spark image as BASE_IMAGE
• [KYUUBI #1535] Remove duplicate hive client version property
• [KYUUBI #1531] Add kyuubi version in engineSpace
• [KYUUBI #1528] Record the kyuubi server ip address in event log
• [KYUUBI #1527] [DOC] Improve High Availability Guide
• [KYUUBI #1516] Implement api: /${version}/operations/${operation_identifier}
• [KYUUBI #1480] Fix bug in docker-image-tool & Provider image way to config SPARK instead of local copy
• [KYUUBI #1393][TASK-2] Add operation number metric
• [KYUUBI #1523] Refine usage about option ‘–namespace’ in kyuubi-ctl
• [KYUUBI #1522] Pass the client ip to engine
• [KYUUBI #1526] [DOC] Update kyuubi_layers.drawio to add driver layer
• [KYUUBI #1521] [BUILD] Migrate to actions/cache@v2
• [KYUUBI #1489] Support operation log for ExecuteScala
• [KYUUBI #1520] [BUILD] Switch to setup-java@v2
• [KYUUBI #1509] Make KYUUBI_WORK_DIR_ROOT as the default root path.
• [KYUUBI #1511] Initialize kyuubi trino engine module
• [KYUUBI #1513] Added support liveness and readiness for kyuubi pod
• [KYUUBI #1501] Introduce operationsResource
• [KYUUBI #1503] Introduce FlinkThriftBinaryFrontendService to do the further initialization
• [KYUUBI #1487] Correct the KEYS link and add script to generate vote and announcement
• [KYUUBI #1496] Support tpcds benchmark
• [KYUUBI #1508] Add Kyuubi version in UI
• [KYUUBI #1504] Support to capture console out/error into interpreter output for ExecuteScala
• [KYUUBI #1505] Move ServiceNodeInfo to a separate file
• [KYUUBI #1492] Initialize kyuubi flink sql engine module
• [KYUUBI #1445] [Kyuubi #1444] Implement api: /${version}/sessions/${session_identifier}/operations/${operation_identifier}
• [KYUUBI #1490] Introduce the basic framework for running scala
• [KYUUBI #1493] Make a reasonable thread pool size for listing file statuses
• [KYUUBI #1379] Make SESSION_ENGINE_LAUNCH_ASYNC true by default
• [KYUUBI #1486] Enable delta test on Spark 3.2 with delta 1.1.0
• [KYUUBI #1484] Add a dependency describe for kyuubi on azure delta-lake doc.
• [KYUUBI #1483] Revamp KyuubiSessionImpl and KyuubiSyncThriftClient
• [KYUUBI #1458] Delta lake table columns won’t show up in DBeaver.
• [KYUUBI #1382] Sync the flink-provided profile in docs and testing suite comments
• [KYUUBI #1481] Add z-order docs
• [KYUUBI #1449] Fix when KyuubiServer initialize fail but not exit
• [KYUUBI #1478] Docker-image-tool.sh awareness of incorrectly configured SPARK_HOME
• [KYUUBI #1472] allow the engine not to self-terminate.
• [KYUUBI #1473] Exit gracefully when engine idle
• [KYUUBI #1471] Improve log if user specify the spark principal and keytab but not same with proxy user
• [KYUUBI #1446] Decouple zookeeper from abstract ServiceDiscovery
• [KYUUBI #1470] Git ignore kyuubi-server metrics
• [KYUUBI #1467] Add The Doc of Intergrate Azure Blob Storage Based Delta Lake with Kyuubi
• [KYUUBI #1469] Add cancel vote template and add release notes into vote template
• [KYUUBI #1462] [DOC] Use relative paths instead of absolute paths
• [KYUUBI #1466] Unify the guava version for kyuubi server and kyuubi beeline
• [KYUUBI #1463] Enabled to preserve symbolic links when copying dist dir
• [KYUUBI #1454] Add Doc About Deploy Kyuubi On Kubernetes
• [KYUUBI #1459] Kyuubi Server Event Test change
• [KYUUBI #1461] [DOC] Remove deprecated dist repo directory for deprecated release candidates
• [KYUUBI #1396][K8S] Add docker image tools and replace Dockerfile
• [KYUUBI #1443] MaxPartitionStrategy support file source based table.
• [KYUUBI #1050] Add KyuubiServerInfoEvent
• [KYUUBI #1439] Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl.
• [KYUUBI #1442] Fix encode MySQLHandshakePacket UT

Credits

Last but not least, this release would not have been possible without the following contributors:

• Ada Wong
• Brian Yue
• Cheng Pan
• Fei Wang
• Fu Chen
• HYBG-1126
• I521853
• Jeff Zhang
• Kent Yao
• Martin Tzvetanov Grigorov
• Muru Muthusamy
• Paul Lam
• Paul Lin
• Rishi Kumar Ray
• Simon
• SteNicholas
• Tianlin Liao
• Wang Zhen
• fwang12
• hongdongdong
• hwk42
• jiaoqingbo
• jing10.gao
• mask
• minyk
• odone
• rayliu
• senmiaoliu
• shenbing
• simon
• sundapeng
• sunfangbin
• sychen
• ulysses-you
• wForget
• xifeng yang
• yanghua
• zhenjiaguo
• zhichaoleo
• zhouyifan279
• zwangsheng

Apache Kyuubi(Incubating) 1.4.1-incubating Release Notes

2022 Jan 31

Hi all,

Apache Kyuubi (Incubating) 1.4.1-incubating is now released. It is a maintenance release containing stability fixes which is based on branch-1.4. Mainly fix some issues of the event log.

Apache Kyuubi (Incubating) is a distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark and designed to support more engines (i.e. Apache Flink). Please refer to Apache Kyuubi (Incubating) to learn more.

To download Apache Kyuubi (Incubating) 1.4.1-incubating, visit the downloads page.

Highlight

• [KYUUBI #1449] Fix when KyuubiServer initialize fail but not exit

• [KYUUBI #1458] Delta lake table columns won’t show up in DBeaver.

• [KYUUBI #1473] Exit gracefully when engine idle

• [KYUUBI #1567] Expose HADOOP_CONF_DIR to Kyuubi beeline to fix “GSS initiate failed” error

• [KYUUBI #1439] Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl

• [KYUUBI #1806] Fix LaunchEngine may be closed multiple times

• [KYUUBI #1731] Kyuubi server should not fail to start when Hadoop FileSystem class not found

• [KYUUBI #1509] Make KYUUBI_WORK_DIR_ROOT as the default root path.

Others

• [KYUUBI #1800] [1.4] Remove oom hook

• [KYUUBI #1784] Fix float types lose precision

• [KYUUBI #1748] bin/kyuubi-zk-cli should has execute permission

• [KYUUBI #1722] [NOTICE] Update NOTICE for 2022

• [KYUUBI #1611] Fix swagger-ui redirect wrong to other service address

Credits

Thank you for the following contributors who helped to review and commit to Apache Kyuubi (Incubating) 1.4.1-incubating version, and the order is based on the lexicographical order:

Cheng Pan, Fei Wang, Fu Chen, Muru Muthusamy, Wang Zhen, sunfangbin, sychen, ulysses-you, yanghua, zhenjiaguo, zhouyifan279, zwangsheng

Apache Kyuubi(Incubating) 1.4.0-incubating Release Notes

2021 Dec 10

Hi all,

In this release, Kyuubi supports the latest Spark 3.2, Z-Order to optimize sql, long running SQL engine without principal and keytab provided, mysql protocol frontend and mysql client. Other major updates includes GROUP engine.share.level support, asynchronous opening session, user-friendly kyuubi beeline and zookeeper authentication enhancement.

To download Apache Kyuubi (Incubating) 1.4.0-incubating, visit the downloads page.

Highlight

Fully verified with the latest Spark 3.2

Support Z-Order to optimize SQL

Support long running SQL engine without principal and keytab provided

Support GROUP engine share level

Enhance USER engine share level support engine pool

Enhance zookeeper authentication and ACLs

Add more Kyuubi defined functions

Fully tested with all data lake platform Delta Lake, Iceberg and Hudi

Support both KERBEROS and PLAIN authentication at the same time

Support MySQL protocol frontend and MySQL client(experimental)

Support to launch query engine asynchronously during opening session and user-friendly Kyuubi Hive JDBC and BeeLine

Functionality

• [KYUUBI #1225] [KYUUBI #1228] Add kyuubi-extension-spark-3-2 module and profile to support latest spark-3.2

• [KYUUBI #939] Add Z-Order extensions to support optimize SQL

• [KYUUBI #913] Support long running SQL engine without principal and keytab provided

• [KYUUBI #1219] Support MySQL Protocol Frontend

• [KYUUBI #1349] Add new conf kyuubi.frontend.protocols

• [KYUUBI #1206] Support GROUP engine share level

• [KYUUBI #1018] Set spark.sql.execution.topKSortFallbackThreshold to 10000

• [KYUUBI #1059] Add Plan Only Operations

• [KYUUBI #1039] Application stop log redirect append to engineLog

• [KYUUBI #1159] Add Session stats on Kyuubi Query Engine Page

• [KYUUBI #1198] Support to incremental collect statement result

• [KYUUBI #1248] Change kyuubi.engine.session.initialize.sql default value to Nil

• [KYUUBI #1286] Increase kyuubi.session.engine.initialize.timeout default to 3min

• [KYUUBI #1323] Introduce EngineType to distinguish multiple engines

• [KYUUBI #1262] Support both KERBEROS and PLAIN authentication at the same time

• [KYUUBI #1160] Support to config operation log root dir both for kyuubi server and engine sides

• [KYUUBI #1204] Enhance zookeeper authentication and ACLs

• [KYUUBI #1039] Application stop log redirect append to engineLog

Integration

• [KYUUBI #703] Integrate test with Hudi 0.9.0

• [KYUUBI #1338] Integrate test with Iceberg 0.12.1

Hive JDBC

• [KYUUBI #1131] Rename KyuubiDriver to KyuubiHiveDriver

• [KYUUBI #1346] Support launch query engine asynchronously during opening session

• [KYUUBI #1377] Support to get launch engine log asynchronously with KyuubiConnection

• [KYUUBI #1414] Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement

Others

• [KYUUBI #1047] Use deployment to run kyuubi server HA on K8s

• [KYUUBI #1223] Remove dead module kyuubi-spark-monitor

• [KYUUBI #1277] Add UDF engine_id

• [KYUUBI #660] Add UDF session_user

Credits

Last but not least, this release would not have been possible without the following contributors: AnybodyHome, Brian Yue, Cheng Pan, Fei Wang, Fu Chen, Jagadesh Adireddi, Kent Yao, Min Zhao, Guoqing Yang, h, happytofly, hongdongdong, jiaoqingbo, qiuliang, qphien, senmiaoliu, simon, sunfangbin, sychen, timothy65535, ulysses-you, wFoget, xiebo03, xiongyinke, yanghua, yanyu34946, zhang1002, zhenjiaguo, zhouyifan279, zwangsheng, 张宇翔

Apache Kyuubi(Incubating) 1.3.1-incubating Release Notes

2021 Oct 25

Hi all,

Apache Kyuubi (Incubating) 1.3.1-incubating is now released. It is a maintenance release containing stability fixes which is based on branch-1.3. Mainly fix some issues of the event log.

Apache Kyuubi (Incubating) is a distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark and designed to support more engines (i.e. Apache Flink). Please refer to Apache Kyuubi (Incubating) to learn more.

To download Apache Kyuubi (Incubating) 1.3.1-incubating, visit the downloads page.

Highlight

• [KYUUBI #1190] Enhance error log detection in case lost the exception log

• [KYUUBI #1067] Catch exception if logRoot is not exist

• [KYUUBI #884] [FOLLOWUP] Fix catalog in KyuubiDatabaseMetaData#getTables

• [KYUUBI #1108] Register KyuubiDriver in ServiceLoader

• [KYUUBI #1094] [BUGFIX #1068]Support upload file to HDFS

• [KYUUBI #1034] Engine may deadlock when close operationLog

• [KYUUBI #1057] [KYUUBI#1056] Output error log when currentEngine is None

Others

• [KYUUBI #1158] [INFRA] Update ANNOUNCE mail template

• [KYUUBI #1107] [INFRA] Add release mail templates

• [KYUUBI #1141] Remove redundant unit

• [KYUUBI #1134] [TEST] DataLake suites should clean up withKyuubiConf instead of extraConfigs

• [KYUUBI #1130] [DOCS] Restore Kyuubi logo image size

• [KYUUBI #1125] [FOLLOWUP] Compress and remove duplicated new logo and keep legacy logo

• [KYUUBI #1125] A new version of logo flie, apache information included.

• [KYUUBI #1104] Fix release script finalize_svn

• [KYUUBI #1096] Remove never expired declaration in NOTICE

Knwon Issues

• [KYUUBI #1263] Kyuubi auxiliary UDF failed on Spark Yarn mode

• [KYUUBI #1267] Modify comments for the golden file test suite and fix it

Credits

Thank you for the following contributors who helped to review and commit to Apache Kyuubi (Incubating) 1.3.1-incubating version, and the order is based on the commit time:

qiuliang, hongdd, Yuxiang Zhang, Cheng Pan, happytofly, timothy, cxzl25, Xiduo You, Simon

Apache Kyuubi(Incubating) 1.3.0-incubating Release Notes

2021 Sep 14

To download Apache Kyuubi (Incubating) 1.3.0-incubating, visit the downloads page.

Highlight

• Fully verified with the latest Spark 3.0/3.1

• Support build on JDK8(default), JDK11

• Provide enhanced AQE in Kyuubi Spark Extensions (for Spark 3.1)

• Provide several auxiliary SQL functions as a supplement to Spark’s built-in Functions

• Major improvements of Spark on Kubernetes, include documents, integration tests, tools, etc.

• Support engine/session level configurable initialize SQL statements

• Support custom authentication implementation

• Support stop Spark engine through Web UI

• Support single session mode

• Several improvements of observability

• Fully tested with the all data lake platforms Delta, Iceberg

• Fully tested with MiniYARNCluster

Functionality

• [KYUUBI #631] Add kyuubi-spark-extensions-3-1 module

• [KYUUBI #657][KYUUBI #659] Provides auxiliary SQL functions as supplement to Spark’s built-in Functions

• [KYUUBI #897] Support stop Spark engine through Web UI

• [KYUUBI #762] Add Kyuubi Hive JDBC Module

• [KYUUBI #926] Support add repartition for OptimizedCreateHiveTableAsSelectCommand

• [KYUUBI #670] KyuubiSQLException support sqlState and errorCode

• [KYUUBI #707] Respect spark.sql.catalogImplementation in engine

• [KYUUBI #641] Support custom authentication

• [KYUUBI #751] Support single session mode

• [1.2][KYUUBI #457] Support configurable initialize sql statement for engine startup

• [KYUUBI #857] Support configurable initialize SQL statement for each session

Integration

• [KYUUBI #755] Default build against Spark 3.1

• [KYUUBI #693][KYUUBI #760] Support build against JDK11

• [KYUUBI #768][K8S] Add module shuffle-files-cleaner

• [KYUUBI #844][K8S] Exposing Kyuubi as a NodePort service

• [KYUUBI #859][K8S] Add kubernetes configmap

• [KYUUBI #656][K8S] Document how to deploy Kyuubi engines on Kubernetes

• [KYUUBI #776][K8S] Run kyuubi on Kubernetes - make basic dockerfile and example

• [KYUUBI #444][KYUUBI #854][TEST][K8S] Add minikube for K8s integration test

• [KYUUBI #712][LDAP] Add a new config property authentication.ldap.guidKey

• [KYUUBI #643][TEST] Test Detla 1.0.0 with Spark 3.1

• [KYUUBI #948][TEST] Test Iceberg 0.12.0 with Spark 3.0/3.1

• [KYUUBI #527][TEST][YARN] Test with MiniYARNCluster

Observability

• [KYUUBI #737] Add monitor module

• [KYUUBI #727] Add the listener for each SQL

• [KYUUBI #767] Add statement event tracking

Performance

• [1.2][KYUUBI #629] Refine distributed lock to lock only engine bootstrap

Others

• [KYUUBI #416] Detected Dependency List Change for better dependency management

• [KYUUBI #487][KYUUBI #566] Add TPC-DS DDL test cases

• [KYUUBI #871] Change module name kyuubi-main to kyuubi-server