[HUDI-1552] Improve performance of key lookups from base file in Metadata Table. (#2494)
* [HUDI-1552] Improve performance of key lookups from base file in Metadata Table. 1. Cache the KeyScanner across lookups so that the HFile index does not have to be read for each lookup. 2. Enable block caching in KeyScanner. 3. Move the lock to a limited scope of the code to reduce lock contention. 4. Removed reuse configuration * Properly close the readers, when metadata table is accessed from executors - Passing a reuse boolean into HoodieBackedTableMetadata - Preserve the fast return behavior when reusing and opening from multiple threads (no contention) - Handle concurrent close() and open readers, for reuse=false, by always synchronizing Co-authored-by: Vinoth Chandar <vinoth@apache.org>
This commit is contained in:
@@ -107,9 +107,8 @@ public abstract class HoodieTable<T extends HoodieRecordPayload, I, K, O> implem
|
||||
this.hadoopConfiguration = context.getHadoopConf();
|
||||
this.context = context;
|
||||
|
||||
// disable reuse of resources, given there is no close() called on the executors ultimately
|
||||
HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder().fromProperties(config.getMetadataConfig().getProps())
|
||||
.enableReuse(false).build();
|
||||
.build();
|
||||
this.metadata = HoodieTableMetadata.create(context, metadataConfig, config.getBasePath(),
|
||||
FileSystemViewStorageConfig.DEFAULT_VIEW_SPILLABLE_DIR);
|
||||
|
||||
|
||||
Reference in New Issue
Block a user