lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity

[HUDI-3681] Provision additional hudi-spark-bundle with different versions (#5171)

Browse Source
This commit is contained in:
Y Ethan Guo
2022-03-30 17:35:56 -07:00
committed by GitHub
parent 2d73c8ae86
commit 9830005e9b
3 changed files with 102 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
README.md
View File

@@ -70,24 +70,31 @@ To build the Javadoc for all Java and Scala classes:
mvn clean javadoc:aggregate -Pjavadocs
```
### Build with Scala 2.12
### Build with Different Spark versions
The default Scala version supported is 2.11. To build for Scala 2.12 version, build using `scala-2.12` profile
The default Spark version supported is 2.4.4. To build for different Spark versions and Scala 2.12, use the
corresponding profile
| Label | Artifact Name for Spark Bundle | Maven Profile Option | Notes |
|--|--|--|--|
| Spark 2.4, Scala 2.11 | hudi-spark2.4-bundle_2.11 | `-Pspark2.4` | For Spark 2.4.4, which is the same as the default |
| Spark 2.4, Scala 2.12 | hudi-spark2.4-bundle_2.12 | `-Pspark2.4,scala-2.12` | For Spark 2.4.4, which is the same as the default and Scala 2.12 |
| Spark 3.1, Scala 2.12 | hudi-spark3.1-bundle_2.12 | `-Pspark3.1` | For Spark 3.1.x |
| Spark 3.2, Scala 2.12 | hudi-spark3.2-bundle_2.12 | `-Pspark3.2` | For Spark 3.2.x |
| Spark 3, Scala 2.12 | hudi-spark3-bundle_2.12 | `-Pspark3` | This is the same as `Spark 3.2, Scala 2.12` |
| Spark, Scala 2.11 | hudi-spark-bundle_2.11 | Default | The default profile, supporting Spark 2.4.4 |
| Spark, Scala 2.12 | hudi-spark-bundle_2.12 | `-Pscala-2.12` | The default profile (for Spark 2.4.4) with Scala 2.12 |
For example,
```
mvn clean package -DskipTests -Dscala-2.12
```
# Build against Spark 3.2.x (the default build shipped with the public Spark 3 bundle)
mvn clean package -DskipTests -Pspark3.2
### Build with Spark 3
# Build against Spark 3.1.x
mvn clean package -DskipTests -Pspark3.1
The default Spark version supported is 2.4.4. To build for different Spark 3 versions, use the corresponding profile
```
# Build against Spark 3.2.1 (the default build shipped with the public Spark 3 bundle)
mvn clean package -DskipTests -Dspark3
# Build against Spark 3.1.2
mvn clean package -DskipTests -Dspark3.1.x
# Build against Spark 2.4.4 and Scala 2.12
mvn clean package -DskipTests -Pspark2.4,scala-2.12
```
### What about "spark-avro" module?