Files

Prashant Wason b28f0d6ceb [HUDI-3290] Different file formats for the partition metadata file. (#5179 )

* [HUDI-3290] Different file formats for the partition metadata file.

Partition metadata files are stored in each partition to help identify the base path of a table. These files are saved in the properties file format. Some query engines do not work when non Parquet/ORC files are found in a partition.

Added a new table config 'hoodie.partition.metafile.use.data.format' which when enabled (default false for backward compatibility) ensures that partition metafiles will be saved in the same format as the base files of a dataset.

For new datasets, the config can be set via hudi-cli. Deltastreamer has a new parameter --partition-metafile-use-data-format which will create a table with this setting.

* Code review comments

- Adding a new command to migrate from text to base file formats for meta file.
- Reimplementing readFromFS() to first read the text format, then base format
- Avoid extra exists() checks in readFromFS()
- Added unit tests, enabled parquet format across hoodie-hadoop-mr
- Code cleanup, restructuring, naming consistency.

* Wiring in all the other Spark code paths to respect this config

 - Turned on parquet meta format for COW data source tests
 - Removed the deltastreamer command line to keep it shorter

* populate HoodiePartitionMetadata#format after readFromFS()

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>

2022-04-04 08:08:20 -07:00

hudi-spark

[HUDI-3290] Different file formats for the partition metadata file. (#5179 )

2022-04-04 08:08:20 -07:00

hudi-spark2

[HUDI-3664] Fixing Column Stats Index composition (#5181 )

2022-04-02 17:15:52 -07:00

hudi-spark2-common

[HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation (#4514 )

2022-01-14 13:42:35 +08:00

hudi-spark3

[HUDI-3664] Fixing Column Stats Index composition (#5181 )

2022-04-02 17:15:52 -07:00

hudi-spark3-common

[RFC-33] [HUDI-2429][Stacked on HUDI-2560] Support full Schema evolution for Spark (#4910 )

2022-04-01 13:20:24 -07:00

hudi-spark3.1.x

[HUDI-3664] Fixing Column Stats Index composition (#5181 )

2022-04-02 17:15:52 -07:00

hudi-spark-common

[HUDI-3290] Different file formats for the partition metadata file. (#5179 )

2022-04-04 08:08:20 -07:00

pom.xml

[HUDI-3574] Improve maven module configs for different spark profiles (#4970 )

2022-03-08 01:01:05 -08:00

README.md

[HUDI-3221] Support querying a table as of a savepoint (#4720 )

2022-03-08 10:02:34 -08:00

README.md

Description of the relationship between each module

This repo contains the code that integrate Hudi with Spark. The repo is split into the following modules

hudi-spark hudi-spark2 hudi-spark3 hudi-spark3.1.x hudi-spark2-common hudi-spark3-common hudi-spark-common

hudi-spark is the module that contains the code that both spark2 & spark3 version would share, also contains the antlr4 file that supports spark sql on spark 2.x version.
hudi-spark2 is the module that contains the code that compatible with spark 2.x versions.
hudi-spark3 is the module that contains the code that compatible with spark 3.2.0(and above) versions。
hudi-spark3.1.x is the module that contains the code that compatible with spark3.1.x and spark3.0.x version.
hudi-spark2-common is the module that contains the code that would be reused between spark2.x versions, right now the module has no class since hudi only supports spark 2.4.4 version, and it acts as the placeholder when packaging hudi-spark-bundle module.
hudi-spark3-common is the module that contains the code that would be reused between spark3.x versions.
hudi-spark-common is the module that contains the code that would be reused between spark2.x and spark3.x versions.

Description of Time Travel

HoodieSpark3_2ExtendedSqlAstBuilder have comments in the spark3.2's code fork from org.apache.spark.sql.catalyst.parser.AstBuilder, and additional withTimeTravel method.
SqlBase.g4 have comments in the code forked from spark3.2's parser, and add SparkSQL Syntax TIMESTAMP AS OF and VERSION AS OF.

Time Travel Support Spark Version:

version	support
2.4.x	No
3.0.x	No
3.1.2	No
3.2.0	Yes

About upgrading Time Travel:

Spark3.3 support time travel syntax link SPARK-37219. Once Spark 3.3 released. The files in the following list will be removed:

hudi-spark3's HoodieSpark3_2ExtendedSqlAstBuilder.scala、HoodieSpark3_2ExtendedSqlParser.scala、TimeTravelRelation.scala、SqlBase.g4、HoodieSqlBase.g4