[HUDI-2593] Virtual keys support for metadata table (#3968)
- Metadata table today has virtual keys disabled, thereby populating the metafields for each record written out and increasing the overall storage space used. Hereby adding virtual keys support for metadata table so that metafields are disabled for metadata table records. - Adding a custom KeyGenerator for Metadata table so as to not rely on the default Base/SimpleKeyGenerators which currently look for record key and partition field set in the table config. - AbstractHoodieLogRecordReader's version of processing next data block and createHoodieRecord() will be a generic version and making the derived class HoodieMetadataMergedLogRecordReader take care of the special creation of records from explictly passed in partition names.
This commit is contained in:
committed by
GitHub
parent
eba354e922
commit
459b34240b
@@ -1672,9 +1672,9 @@ public class TestHoodieLogFormat extends HoodieCommonTestHarness {
|
||||
Map<HeaderMetadataType, String> header) {
|
||||
switch (dataBlockType) {
|
||||
case AVRO_DATA_BLOCK:
|
||||
return new HoodieAvroDataBlock(records, header);
|
||||
return new HoodieAvroDataBlock(records, header, HoodieRecord.RECORD_KEY_METADATA_FIELD);
|
||||
case HFILE_DATA_BLOCK:
|
||||
return new HoodieHFileDataBlock(records, header);
|
||||
return new HoodieHFileDataBlock(records, header, HoodieRecord.RECORD_KEY_METADATA_FIELD);
|
||||
default:
|
||||
throw new RuntimeException("Unknown data block type " + dataBlockType);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user