5ce45c440b
* Introduce hudi-spark3-common and hudi-spark2-common modules to place classes that would be reused in different spark versions, also introduce hudi-spark3.1.x to support spark 3.1.x. * Introduce hudi format under hudi-spark2, hudi-spark3, hudi-spark3.1.x modules and change the hudi format in original hudi-spark module to hudi_v1 format. * Manually tested on Spark 3.1.2 and Spark 3.2.0 SQL. * Added a README.md file under hudi-spark-datasource module.
39 lines
2.0 KiB
Markdown
39 lines
2.0 KiB
Markdown
<!--
|
|
* Licensed to the Apache Software Foundation (ASF) under one
|
|
* or more contributor license agreements. See the NOTICE file
|
|
* distributed with this work for additional information
|
|
* regarding copyright ownership. The ASF licenses this file
|
|
* to you under the Apache License, Version 2.0 (the
|
|
* "License"); you may not use this file except in compliance
|
|
* with the License. You may obtain a copy of the License at
|
|
*
|
|
* http://www.apache.org/licenses/LICENSE-2.0
|
|
*
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
* See the License for the specific language governing permissions and
|
|
-->
|
|
|
|
# Description of the relationship between each module
|
|
|
|
This repo contains the code that integrate Hudi with Spark. The repo is split into the following modules
|
|
|
|
`hudi-spark`
|
|
`hudi-spark2`
|
|
`hudi-spark3`
|
|
`hudi-spark3.1.x`
|
|
`hudi-spark2-common`
|
|
`hudi-spark3-common`
|
|
`hudi-spark-common`
|
|
|
|
* hudi-spark is the module that contains the code that both spark2 & spark3 version would share, also contains the antlr4
|
|
file that supports spark sql on spark 2.x version.
|
|
* hudi-spark2 is the module that contains the code that compatible with spark 2.x versions.
|
|
* hudi-spark3 is the module that contains the code that compatible with spark 3.2.0(and above) versions。
|
|
* hudi-spark3.1.x is the module that contains the code that compatible with spark3.1.x and spark3.0.x version.
|
|
* hudi-spark2-common is the module that contains the code that would be reused between spark2.x versions, right now the module
|
|
has no class since hudi only supports spark 2.4.4 version, and it acts as the placeholder when packaging hudi-spark-bundle module.
|
|
* hudi-spark3-common is the module that contains the code that would be reused between spark3.x versions.
|
|
* hudi-spark-common is the module that contains the code that would be reused between spark2.x and spark3.x versions.
|