a66fd40692
- This change makes sure MT records are updated appropriately on HDFS: previously after Log File append operations MT records were updated w/ just the size of the deltas being appended to the original files, which have been found to be the cause of issues in case of Rollbacks that were instead updating MT with records bearing the full file-size. - To make sure that we hedge against similar issues going f/w, this PR alleviates this discrepancy and streamlines the flow of MT table always ingesting records bearing full file-sizes.
Description of the relationship between each module
This repo contains the code that integrate Hudi with Spark. The repo is split into the following modules
hudi-spark
hudi-spark2
hudi-spark3
hudi-spark3.1.x
hudi-spark2-common
hudi-spark3-common
hudi-spark-common
- hudi-spark is the module that contains the code that both spark2 & spark3 version would share, also contains the antlr4 file that supports spark sql on spark 2.x version.
- hudi-spark2 is the module that contains the code that compatible with spark 2.x versions.
- hudi-spark3 is the module that contains the code that compatible with spark 3.2.0(and above) versions。
- hudi-spark3.1.x is the module that contains the code that compatible with spark3.1.x and spark3.0.x version.
- hudi-spark2-common is the module that contains the code that would be reused between spark2.x versions, right now the module has no class since hudi only supports spark 2.4.4 version, and it acts as the placeholder when packaging hudi-spark-bundle module.
- hudi-spark3-common is the module that contains the code that would be reused between spark3.x versions.
- hudi-spark-common is the module that contains the code that would be reused between spark2.x and spark3.x versions.