[HUDI-3396] Make sure BaseFileOnlyViewRelation only reads projected columns (#4818)
NOTE: This change is first part of the series to clean up Hudi's Spark DataSource related implementations, making sure there's minimal code duplication among them, implementations are consistent and performant This PR is making sure that BaseFileOnlyViewRelation only reads projected columns as well as avoiding unnecessary serde from Row to InternalRow Brief change log - Introduced HoodieBaseRDD as a base for all custom RDD impls - Extracted common fields/methods to HoodieBaseRelation - Cleaned up and streamlined HoodieBaseFileViewOnlyRelation - Fixed all of the Relations to avoid superfluous Row <> InternalRow conversions
This commit is contained in:
@@ -133,7 +133,7 @@ public class HoodieMetadataPayload implements HoodieRecordPayload<HoodieMetadata
|
||||
// This can be simplified using SpecificData.deepcopy once this bug is fixed
|
||||
// https://issues.apache.org/jira/browse/AVRO-1811
|
||||
//
|
||||
// NOTE: {@code HoodieMetadataRecord} has to always carry both "key" nad "type" fields
|
||||
// NOTE: {@code HoodieMetadataRecord} has to always carry both "key" and "type" fields
|
||||
// for it to be handled appropriately, therefore these fields have to be reflected
|
||||
// in any (read-)projected schema
|
||||
key = record.get(KEY_FIELD_NAME).toString();
|
||||
|
||||
Reference in New Issue
Block a user