* [HUDI-2795] Add mechanism to safely update,delete and recover table properties
- Fail safe mechanism, that lets queries succeed off a backup file
- Readers who are not upgraded to this version of code will just fail until recovery is done.
- Added unit tests that exercises all these scenarios.
- Adding CLI for recovery, updation to table command.
- [Pending] Add some hash based verfication to ensure any rare partial writes for HDFS
* Fixing upgrade/downgrade infrastructure to use new updation method
- Metadata table today has virtual keys disabled, thereby populating the metafields
for each record written out and increasing the overall storage space used. Hereby
adding virtual keys support for metadata table so that metafields are disabled
for metadata table records.
- Adding a custom KeyGenerator for Metadata table so as to not rely on the
default Base/SimpleKeyGenerators which currently look for record key
and partition field set in the table config.
- AbstractHoodieLogRecordReader's version of processing next data block and
createHoodieRecord() will be a generic version and making the derived class
HoodieMetadataMergedLogRecordReader take care of the special creation of
records from explictly passed in partition names.
- ExternalSpillableMap does the payload/value size estimation on the first put to
determine when to spill over to disk map. The payload size re-estimation also
happens after a minimum threshold of puts. This size re-estimation goes my the
current in-memory map size for calculating average payload size and does attempts
divide by zero operation when the map is size is empty. Avoiding the
ArithmeticException during the payload size re-estimate by checking the map size
upfront.
* [HUDI-2634] Improved the metadata table bootstrap for very large tables.
Following improvements are implemented:
1. Memory overhead reduction:
- Existing code caches FileStatus for each file in memory.
- Created a new class DirectoryInfo which is used to cache a director's file list with parts of the FileStatus (only filename and file len). This reduces the memory requirements.
2. Improved parallelism:
- Existing code collects all the listing to the Driver and then creates HoodieRecord on the Driver.
- This takes a long time for large tables (11million HoodieRecords to be created)
- Created a new function in SparkRDDWriteClient specifically for bootstrap commit. In it, the HoodieRecord creation is parallelized across executors so it completes fast.
3. Fixed setting to limit the number of parallel listings:
- Existing code had a bug wherein 1500 executors were hardcoded to perform listing. This leads to exception due to limit in the spark's result memory.
- Corrected the use of the config.
Result:
Dataset has 1299 partitions and 12Million files.
file listing time=1.5mins
HoodieRecord creation time=13seconds
deltacommit duration=2.6mins
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
* [HUDI-2101]support z-order for hudi
* Renaming some configs for consistency/simplicity.
* Minor code cleanups
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
* [HUDI-1295] Hash ID generator util for Hudi table columns, partition and files
- Adding a new utility class HashID to generate 32,64,128 bits hashes for any
given message of string or byte array type. This class internally uses
MessageDigest and xxhash libraries.
- Adding stateful hash holders for Hudi table columns, partition and files to
pass around for metaindex and to convert to base64encoded strings whenever
needed
- InetAddress.getLocalHost() can take up as much as 30+seconds if the network
configurations are not done right. This might be due to local hostname
missing IPv6 address mapping in /etc/hosts or network configs slowing down
any IPv6 name resolutions. If this API is used for logging verbose messages
and that too in the hot code path, it can lead to order of magnitude
slowness in the overall task completion.
* [HUDI-2285] Adding Synchronous updates to metadata before completion of commits in data timelime.
- This patch adds synchronous updates to metadata table. In other words, every write is first committed to metadata table followed by data table. While reading metadata table, we ignore any delta commits that are present only in metadata table and not in data table timeline.
- Compaction of metadata table is fenced by the condition that we trigger compaction only when there are no inflight requests in datatable. This ensures that all base files in metadata table is always in sync with data table(w/o any holes) and only there could be some extra invalid commits among delta log files in metadata table.
- Due to this, archival of data table also fences itself up until compacted instant in metadata table.
All writes to metadata table happens within the datatable lock. So, metadata table works in one writer mode only. This might be tough to loosen since all writers write to same FILES partition and so, will result in a conflict anyways.
- As part of this, have added acquiring locks in data table for those operations which were not before while committing (rollback, clean, compaction, cluster). To note, we were not doing any conflict resolution. All we are doing here is to commit by taking a lock. So that all writes to metadata table is always a single writer.
- Also added building block to add buckets for partitions, which will be leveraged by other indexes like record level index, etc. For now, FILES partition has only one bucket. In general, any number of buckets per partition is allowed and each partition has a fixed fileId prefix with incremental suffix for each bucket within each partition.
Have fixed [HUDI-2476]. This fix is about retrying a failed compaction if it succeeded in metadata for first time, but failed w/ data table.
- Enabling metadata table by default.
- Adding more tests for metadata table
Co-authored-by: Prashant Wason <pwason@uber.com>
