[HUDI-3760] Adding capability to fetch Metadata Records by prefix (#5208)
- Adding capability to fetch Metadata Records by key prefix so that Data Skipping could fetch only Column Stats - Index records pertaining to the columns being queried by, instead of reading out whole Index. - Fixed usages of HFileScanner in HFileReader. few code paths uses cached scanner if available. Other code paths uses its own HFileScanner w/ positional read. Brief change log - Rebasing ColumnStatsIndexSupport to rely on HoodieBackedTableMetadata in lieu of reading t/h Spark DS - Adding methods enabling key-prefix lookups to HoodiFileReader, HoodieHFileReader - Wiring key-prefix lookup t/h LogRecordScanner impls - Cleaning up HoodieHFileReader impl Co-authored-by: sivabalan <n.siva.b@gmail.com> Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
This commit is contained in:
@@ -2038,7 +2038,7 @@ public class TestHoodieLogFormat extends HoodieCommonTestHarness {
|
||||
* Utility to convert the given iterator to a List.
|
||||
*/
|
||||
private static List<IndexedRecord> getRecords(HoodieDataBlock dataBlock) {
|
||||
ClosableIterator<IndexedRecord> itr = dataBlock.getRecordItr();
|
||||
ClosableIterator<IndexedRecord> itr = dataBlock.getRecordIterator();
|
||||
|
||||
List<IndexedRecord> elements = new ArrayList<>();
|
||||
itr.forEachRemaining(elements::add);
|
||||
|
||||
Reference in New Issue
Block a user