Files

xiarixiaoyao 444ff496a4 [RFC-33] [HUDI-2429][Stacked on HUDI-2560] Support full Schema evolution for Spark (#4910 )

* [HUDI-2560] introduce id_based schema to support full schema evolution.

* add test for FileBasedInternalSchemaStorageManger and rebase code

* add support for change column type and fix some test case

* fix some bugs encountered in the production env and delete useless code

* fix test error

* rebase code

* fixed some nested schema change bugs

* [HUDI-2429][Stacked On HUDI-2560]Support full schema evolution for spark

* [use dummyInternalSchema instead of null]

* add support for spark3.1.x

* remove support for spark3.1.x , sicne some compile fail

* support spark3.1.x

* rebase and prepare solve all comments

* address all comments

* rebase code

* fixed the count(*) bug

* try to get internalSchema by parser commit file/history file directly, not use metaclient which is time cost
address some comments

* fixed all comments

* fix new comments

* rebase code,fix UT failed

* fixed mistake

* rebase code ,fixed new comments

* rebase code , and prepare for address new comments

* address commits

* address new comments

* fix new issues

* control fallback original write logical

2022-04-01 13:20:24 -07:00

hudi-spark

[RFC-33] [HUDI-2429][Stacked on HUDI-2560] Support full Schema evolution for Spark (#4910 )

2022-04-01 13:20:24 -07:00

hudi-spark2

[RFC-33] [HUDI-2429][Stacked on HUDI-2560] Support full Schema evolution for Spark (#4910 )

2022-04-01 13:20:24 -07:00

hudi-spark2-common

[HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation (#4514 )

2022-01-14 13:42:35 +08:00

hudi-spark3

[RFC-33] [HUDI-2429][Stacked on HUDI-2560] Support full Schema evolution for Spark (#4910 )

2022-04-01 13:20:24 -07:00

hudi-spark3-common

[RFC-33] [HUDI-2429][Stacked on HUDI-2560] Support full Schema evolution for Spark (#4910 )

2022-04-01 13:20:24 -07:00

hudi-spark3.1.x

[RFC-33] [HUDI-2429][Stacked on HUDI-2560] Support full Schema evolution for Spark (#4910 )

2022-04-01 13:20:24 -07:00

hudi-spark-common

[RFC-33] [HUDI-2429][Stacked on HUDI-2560] Support full Schema evolution for Spark (#4910 )

2022-04-01 13:20:24 -07:00

pom.xml

[HUDI-3574] Improve maven module configs for different spark profiles (#4970 )

2022-03-08 01:01:05 -08:00

README.md

[HUDI-3221] Support querying a table as of a savepoint (#4720 )

2022-03-08 10:02:34 -08:00

README.md

Description of the relationship between each module

This repo contains the code that integrate Hudi with Spark. The repo is split into the following modules

hudi-spark hudi-spark2 hudi-spark3 hudi-spark3.1.x hudi-spark2-common hudi-spark3-common hudi-spark-common

hudi-spark is the module that contains the code that both spark2 & spark3 version would share, also contains the antlr4 file that supports spark sql on spark 2.x version.
hudi-spark2 is the module that contains the code that compatible with spark 2.x versions.
hudi-spark3 is the module that contains the code that compatible with spark 3.2.0(and above) versions。
hudi-spark3.1.x is the module that contains the code that compatible with spark3.1.x and spark3.0.x version.
hudi-spark2-common is the module that contains the code that would be reused between spark2.x versions, right now the module has no class since hudi only supports spark 2.4.4 version, and it acts as the placeholder when packaging hudi-spark-bundle module.
hudi-spark3-common is the module that contains the code that would be reused between spark3.x versions.
hudi-spark-common is the module that contains the code that would be reused between spark2.x and spark3.x versions.

Description of Time Travel

HoodieSpark3_2ExtendedSqlAstBuilder have comments in the spark3.2's code fork from org.apache.spark.sql.catalyst.parser.AstBuilder, and additional withTimeTravel method.
SqlBase.g4 have comments in the code forked from spark3.2's parser, and add SparkSQL Syntax TIMESTAMP AS OF and VERSION AS OF.

Time Travel Support Spark Version:

version	support
2.4.x	No
3.0.x	No
3.1.2	No
3.2.0	Yes

About upgrading Time Travel:

Spark3.3 support time travel syntax link SPARK-37219. Once Spark 3.3 released. The files in the following list will be removed:

hudi-spark3's HoodieSpark3_2ExtendedSqlAstBuilder.scala、HoodieSpark3_2ExtendedSqlParser.scala、TimeTravelRelation.scala、SqlBase.g4、HoodieSqlBase.g4