9e87d164b3
- Adding capability to fetch Metadata Records by key prefix so that Data Skipping could fetch only Column Stats - Index records pertaining to the columns being queried by, instead of reading out whole Index. - Fixed usages of HFileScanner in HFileReader. few code paths uses cached scanner if available. Other code paths uses its own HFileScanner w/ positional read. Brief change log - Rebasing ColumnStatsIndexSupport to rely on HoodieBackedTableMetadata in lieu of reading t/h Spark DS - Adding methods enabling key-prefix lookups to HoodiFileReader, HoodieHFileReader - Wiring key-prefix lookup t/h LogRecordScanner impls - Cleaning up HoodieHFileReader impl Co-authored-by: sivabalan <n.siva.b@gmail.com> Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
Description of the relationship between each module
This repo contains the code that integrate Hudi with Spark. The repo is split into the following modules
hudi-spark
hudi-spark2
hudi-spark3
hudi-spark3.1.x
hudi-spark2-common
hudi-spark3-common
hudi-spark-common
- hudi-spark is the module that contains the code that both spark2 & spark3 version would share, also contains the antlr4 file that supports spark sql on spark 2.x version.
- hudi-spark2 is the module that contains the code that compatible with spark 2.x versions.
- hudi-spark3 is the module that contains the code that compatible with spark 3.2.0(and above) versions。
- hudi-spark3.1.x is the module that contains the code that compatible with spark3.1.x and spark3.0.x version.
- hudi-spark2-common is the module that contains the code that would be reused between spark2.x versions, right now the module has no class since hudi only supports spark 2.4.4 version, and it acts as the placeholder when packaging hudi-spark-bundle module.
- hudi-spark3-common is the module that contains the code that would be reused between spark3.x versions.
- hudi-spark-common is the module that contains the code that would be reused between spark2.x and spark3.x versions.
Description of Time Travel
HoodieSpark3_2ExtendedSqlAstBuilderhave comments in the spark3.2's code fork fromorg.apache.spark.sql.catalyst.parser.AstBuilder, and additionalwithTimeTravelmethod.SqlBase.g4have comments in the code forked from spark3.2's parser, and add SparkSQL SyntaxTIMESTAMP AS OFandVERSION AS OF.
Time Travel Support Spark Version:
| version | support |
|---|---|
| 2.4.x | No |
| 3.0.x | No |
| 3.1.2 | No |
| 3.2.0 | Yes |
About upgrading Time Travel:
Spark3.3 support time travel syntax link SPARK-37219. Once Spark 3.3 released. The files in the following list will be removed:
- hudi-spark3's
HoodieSpark3_2ExtendedSqlAstBuilder.scala、HoodieSpark3_2ExtendedSqlParser.scala、TimeTravelRelation.scala、SqlBase.g4、HoodieSqlBase.g4