Currently when doing Hudi queries w/ Spark, it won't
load the external configurations. Say if customers enabled
metadata listing in their global config file, then this would
let them actually query w/o metadata feature enabled.
This PR fixes this issue and allows loading global
configs during the Hudi reading phase.
Co-authored-by: Wenning Ding <wenningd@amazon.com>
Currently, all Hudi Relations bear performance gap relative to Spark's HadoopFsRelation
and the reason to that is SchemaPruning optimization rule (pruning nested schemas)
that is unfortunately predicated on usage of HadoopFsRelation, meaning that it's
not applied in cases when any other relation is used.
This change is porting this rule to Hudi relations (MOR, Incremental, etc)
by the virtue of leveraging HoodieSparkSessionExtensions mechanism
injecting modified version of the original SchemaPruning rule
that is adopted to work w/ Hudi's custom relations.
- Added customOptimizerRules to HoodieAnalysis
- Added NestedSchemaPrunning Spark's Optimizer rule
- Handle Spark's Optimizer pruned data schema (to effectively prune nested schemas)
- Enable HoodieClientTestHarness to inject HoodieSparkSessionExtensions
- Injecting Spark Session extensions for TestMORDataSource, TestCOWDataSource
- Disabled fallback to HadoopFsRelation
- Move clean related configuration to HoodieCleanConfig
- Move Archival related configuration to HoodieArchivalConfig
- hoodie.compaction.payload.class move this to HoodiePayloadConfig
* [HUDI-4276] Reconcile schema-inject null values for missing fields and add new fields.
* fix comments
Co-authored-by: public (bdcee5037027) <mengtao0326@qq.com>
* [HUDI-3730] Improve meta sync class design and hierarchies (#5754)
* Implements class design proposed in RFC-55
Co-authored-by: jian.feng <fengjian428@gmial.com>
Co-authored-by: jian.feng <jian.feng@shopee.com>
* [HUDI-4285] add ByteBuffer#rewind after ByteBuffer#get in AvroDeserializer
* add ut
Co-authored-by: wangzixuan.wzxuan <wangzixuan.wzxuan@bytedance.com>
Fixed sequence determining whether Parquet's legacy-format writing property should be overridden to only kick in when it has not been explicitly specified by the caller
- Upgrade junit to 5.7.2
- Downgrade surefire and failsafe to 2.22.2
- Fix test failures that were previously not reported
- Improve azure pipeline configs
Co-authored-by: liujinhui1994 <965147871@qq.com>
Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com>
There are multiple issues with our current DataSource V2 integrations: b/c we advertise Hudi tables as V2, Spark expects it to implement certain APIs which are not implemented at the moment, instead we're using custom Resolution rule (in HoodieSpark3Analysis) to instead manually fallback to V1 APIs. This commit fixes the issue by reverting DSv2 APIs and making Spark use V1, except for schema evaluation logic.
