lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity

[HUDI-584] Relocate spark-avro dependency by maven-shade-plugin (#1290)

Browse Source
This commit is contained in:
Bhavani Sudha Saktheeswaran
2020-03-04 11:01:49 -08:00
committed by GitHub
parent 9d46ce380a
commit 5f85c26704
3 changed files with 43 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
README.md
View File

@@ -55,6 +55,11 @@ Prerequisites for building Apache Hudi:
# Checkout code and build
git clone https://github.com/apache/incubator-hudi.git && cd incubator-hudi
mvn clean package -DskipTests -DskipITs
# Start command
spark-2.4.4-bin-hadoop2.7/bin/spark-shell \
--jars `ls packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-*.*.*-SNAPSHOT.jar` \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
```
To build the Javadoc for all Java and Scala classes:
@@ -71,6 +76,22 @@ The default Scala version supported is 2.11. To build for Scala 2.12 version, bu
mvn clean package -DskipTests -DskipITs -Dscala-2.12
```
### Build without spark-avro module
The default hudi-jar bundles spark-avro module. To build without spark-avro module, build using `spark-shade-unbundle-avro` profile
```
# Checkout code and build
git clone https://github.com/apache/incubator-hudi.git && cd incubator-hudi
mvn clean package -DskipTests -DskipITs -Pspark-shade-unbundle-avro
# Start command
spark-2.4.4-bin-hadoop2.7/bin/spark-shell \
--packages org.apache.spark:spark-avro_2.11:2.4.4 \
--jars `ls packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-*.*.*-SNAPSHOT.jar` \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
```
## Quickstart
Please visit [https://hudi.apache.org/docs/quick-start-guide.html](https://hudi.apache.org/docs/quick-start-guide.html) to quickly explore Hudi's capabilities using spark-shell.