* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- Today, base files have bloom filter at their footers and index lookups
have to load the base file to perform any bloom lookups. Though we have
interval tree based file purging, we still end up in significant amount
of base file read for the bloom filter for the end index lookups for the
keys. This index lookup operation can be made more performant by having
all the bloom filters in a new metadata partition and doing pointed
lookups based on keys.
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- Adding indexing support for clean, restore and rollback operations.
Each of these operations will now be converted to index records for
bloom filter and column stats additionally.
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- Making hoodie key consistent for both column stats and bloom index by
including fileId instead of fileName, in both read and write paths.
- Performance optimization for looking up records in the metadata table.
- Avoiding multi column sorting needed for HoodieBloomMetaIndexBatchCheckFunction
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- HoodieBloomMetaIndexBatchCheckFunction cleanup to remove unused classes
- Base file checking before reading the file footer for bloom or column stats
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- Updating the bloom index and column stats index to have full file name
included in the key instead of just file id.
- Minor test fixes.
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- Fixed flink commit method to handle metadata table all partition update records
- TestBloomIndex fixes
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- SparkHoodieBloomIndexHelper code simplification for various config modes
- Signature change for getBloomFilters() and getColumnStats(). Callers can
just pass in interested partition and file names, the index key is then
constructed internally based on the passed in parameters.
- KeyLookupHandle and KeyLookupResults code refactoring
- Metadata schema changes - removed the reserved field
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- Removing HoodieColumnStatsMetadata and using HoodieColumnRangeMetadata instead.
Fixed the users of the the removed class.
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- Extending meta index test to cover deletes, compactions, clean
and restore table operations. Also, fixed the getBloomFilters()
and getColumnStats() to account for deleted entries.
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- Addressing review comments - java doc for new classes, keys sorting for
lookup, index methods renaming.
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- Consolidated the bloom filter checking for keys in to one
HoodieMetadataBloomIndexCheckFunction instead of a spearate batch
and lazy mode. Removed all the configs around it.
- Made the metadata table partition file group count configurable.
- Fixed the HoodieKeyLookupHandle to have auto closable file reader
when checking bloom filter and range keys.
- Config property renames. Test fixes.
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- Enabling column stats indexing for all columns by default
- Handling column stat generation errors and test update
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- Metadata table partition file group count taken from the slices when
the table is bootstrapped.
- Prep records for the commit refactored to the base class
- HoodieFileReader interface changes for filtering keys
- Multi column and data types support for colums stats index
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- rebase to latest master and merge fixes for the build and test failures
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- Extending the metadata column stats type payload schema to include
more statistics about the column ranges to help query integration.
* [HUDI-1295] Metadata Index - Bloom filter and Column stats index to speed up index lookups
- Addressing review comments
This change is addressing issues in regards to Metadata Table observing ingesting duplicated records leading to it persisting incorrect file-sizes for the files referred to in those records.
There are multiple issues that were leading to that:
- [HUDI-3322] Incorrect Rollback Plan generation: Rollback Plan generated for MOR tables was overly expansively listing all log-files with the latest base-instant as the ones that have been affected by the rollback, leading to invalid MT records being ingested referring to those.
- [HUDI-3343] Metadata Table including Uncommitted Log Files during Bootstrap: Since MT is bootstrapped at the end of the commit operation execution (after FS activity, but before committing to the timeline), it was actually incorrectly ingesting some files that were part of the intermediate state of the operation being committed.
This change will unblock Stack of PRs based off #4556
* [HUDI-2763] Metadata table records - support for key deduplication and virtual keys
- The backing log format for the metadata table is HFile, a KeyValue type.
Since the key field in the metadata record payload is a duplicate of the
Key in the Cell, the redundant key field in the record can be emptied
to save on the cost.
- HoodieHFileWriter and HoodieHFileDataBlock will now serialize records
with the key field emptied by default. HFile writer tries to find if
the record has metadata payload schema field 'key' and if so it does
the key trimming from the record payload.
- HoodieHFileReader when reading the serialized records back from disk,
it materializes the missing keyFields if any. HFile reader tries to
find if the record has metadata payload schema fiels 'key' and if so
it does the key materialization in the record payload.
- Tests have been added to verify the default virtual keys and key
deduplication support for the metadata table records.
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
* [HUDI-3030] InProcessLockPovider as default when any async servcies enabled with no lock provider override
- Making InProcessLockProvider as the default lock provider when
any async services are enabled and when no lock provider is
explicitly set.
- This is the workaround for metadata table updates racing with
async table serice operations
* [HUDI-3030] InProcessLockPovider as default when any async servcies enabled with no lock provider override
- Renaming isAnyTableServicesInline/Async() to areAnyTableServicesInline/Async()
* [HUDI-3030] InProcessLockPovider as default when any async servcies enabled with no lock provider override
- Additionally checking for write config properties when verifying
the lock provider override. Updated the unit test for this case.
* [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator
Timestampbased key generator was returning diff values for row writer and non row writer path. this patch fixes it and is guarded by a config flag (`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled`)
- HoodieMetadataMergedLogRecordReader#getRecordsByKeys() and its parent class methods
are not thread safe. When multiple queries come in for gettting log records
by keys, they all operate on the same log record reader instance provided by
HoodieBackedTableMetadata#openReadersIfNeeded() and they trip over each other
as they clear/put/get the same class memeber records.
- The fix is to streamline the mutatation to class member records. Making
HoodieMetadataMergedLogRecordReader#getRecordsByKeys() a synchronized method
to avoid concurrent log records readers getting into NPE.
* [HUDI-2154] Add index key field to HoodieKey
* [HUDI-2157] Add the bucket index and its read/write implemention of Spark engine.
* revert HUDI-2154 add index key field to HoodieKey
* fix all comments and introduce a new tricky way to get index key at runtime
support double insert for bucket index
* revert spark read optimizer based on bucket index
* add the storage layout
* index tag, hash function and add ut
* fix ut
* address partial comments
* Code review feedback
* add layout config and docs
* fix ut
* rename hoodie.layout and rebase master
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
