Kyuubi is built based on Apache Maven,
./build/mvn clean package -DskipTests
This results in the creation of all sub-modules of Kyuubi project without running any unit test.
If you want to test it manually, you can start Kyuubi directly from the Kyuubi project root by running
bin/kyuubi start
For instance, you can build the Kyuubi Common module using:
build/mvn clean package -pl :kyuubi-common -DskipTests
For instance, you can build the Kyuubi Common module using:
build/mvn clean package -pl :kyuubi-common,:kyuubi-ha -DskipTests
For instance, you can build the Kyuubi modules without Kyuubi Codecov and Assembly modules using:
build/mvn clean install -pl '!:kyuubi-codecov,!:kyuubi-assembly' -DskipTests
Since v1.1.0, Kyuubi support building with different Spark profiles,
Profile | Default | Since |
---|---|---|
-Pspark-3.0 | No | 1.0.0 |
-Pspark-3.1 | No | 1.1.0 |
-Pspark-3.2 | Yes | 1.4.0 |
-Pspark-3.3 | No | 1.6.0 |
By default, we use https://archive.apache.org/dist/spark/
to download the built-in Spark release package,
but if you find it hard to reach, or the downloading speed is too slow, you can define the spark.archive.mirror
property to a suitable Apache mirror site. For instance,
build/mvn clean package -Dspark.archive.mirror=https://mirrors.bfsu.edu.cn/apache/spark/spark-3.2.1
Visit Apache Mirrors and choose a mirror based on your region.
Specifically for developers in China mainland, you can use the pre-defined profile named mirror-cdn
which use
mirrors.bfsu.edu.cn
to speed up Spark Binary downloading. For instance,
build/mvn clean package -Pmirror-cn
To create a Kyuubi distribution like those distributed by Kyuubi Release Page,
and that is laid out so as to be runnable, use ./build/dist
in the project root directory.
For more information on usage, run ./build/dist --help
./build/dist - Tool for making binary distributions of Kyuubi
Usage:
+------------------------------------------------------------------------------------------------------+
| ./build/dist [--name <custom_name>] [--tgz] [--flink-provided] [--spark-provided] [--hive-provided] |
| [--mvn <maven_executable>] <maven build options> |
+------------------------------------------------------------------------------------------------------+
name: - custom binary name, using project version if undefined
tgz: - whether to make a whole bundled package
flink-provided: - whether to make a package without Flink binary
spark-provided: - whether to make a package without Spark binary
hive-provided: - whether to make a package without Hive binary
mvn: - external maven executable location
For instance,
./build/dist --name custom-name --tgz
This results a Kyuubi distribution named kyuubi-{version}-bin-custom-name.tar.gz
for you.
If you are planing to deploy Kyuubi where spark
is provided, in other word, it’s not required to bundle spark binary, use
./build/dist --tgz --spark-provided
Then you will get a Kyuubi distribution without spark binary named kyuubi-{version}-bin.tar.gz
.
Follow the steps below and learn how to build the Kyuubi documentation as the one you are watching now.
virtualenv
Firstly, install virtualenv
, this is optional but recommended as it is useful to create an independent environment to resolve dependency issues for building the documentation.
pip install virtualenv
Switch to the docs
root directory.
cd $KTUUBI_HOME/docs
Create a virtual environment named ‘kyuubi’ or anything you like using virtualenv
if it’s not existing.
virtualenv kyuubi
Activate it,
source ./kyuubi/bin/activate
Install all dependencies enumerated in the requirements.txt
pip install -r requirements.txt
make html
If the build process succeed, the HTML pages are in _build/html
.
Open the _build/html/index.html
file in your favorite web browser.
Kyuubi can be tested based on Apache Maven and the ScalaTest Maven Plugin, please refer to the ScalaTest documentation,
The following is an example of a command to run all the tests:
./build/mvn clean test
./build/mvn clean test -pl :kyuubi-common
When developing locally, it’s convenient to run one single test, or a couple of tests, rather than all.
With Maven, you can use the -DwildcardSuites flag to run individual Scala tests:
./build/mvn test -Dtest=none -DwildcardSuites=org.apache.kyuubi.service.FrontendServiceSuite
If you want to make a single test that need integrate with kyuubi-spark-sql-engine module, please build the package for kyuubi-spark-sql-engine module at first.
You can leverage the ready-made tool for creating a binary distribution.
./build/dist
You can use the Java Debug Wire Protocol to debug Kyuubi with your favorite IDE tool, e.g. Intellij IDEA.
We can configure the JDWP agent in KYUUBI_JAVA_OPTS
for debugging.
For example,
KYUUBI_JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005 \
bin/kyuubi start
In the IDE, you set the corresponding parameters(host&port) in debug configurations, for example,
spark.driver.extraJavaOptions -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005
spark.executor.extraJavaOptions -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005
build/mvn versions:set -DgenerateBackupPoms=false
Whenever Project version updates, please also update the document version at docs/conf.py
to target the upcoming release.
For example,
release = '1.2.0'
Kyuubi uses the dev/dependencyList
file to indicates what upstream dependencies will actually go to the server-side classpath.
For Pull requests, a linter for dependency check will be automatically executed in GitHub Actions.
You can run build/dependency.sh
locally first to detect the potential dependency change first.
If the changes look expected, run build/dependency.sh --replace
to update dev/dependencyList
in your Pull request.