* `ZCurveOptimizeHelper` > `ZOrderingIndexHelper`;
Moved Z-index helper under `hudi.index.zorder` package
* Tidying up `ZOrderingIndexHelper`
* Fixing compilation
* Fixed index new/original table merging sequence to always prefer values from new index;
Cleaned up `HoodieSparkUtils`
* Added test for `mergeIndexSql`
* Abstracted Z-index name composition w/in `ZOrderingIndexHelper`;
* Fixed `DataSkippingUtils` to interrupt prunning in case data filter contains non-indexed column reference
* Properly handle exceptions origination during pruning in `HoodieFileIndex`
* Make sure no errors are logged upon encountering `AnalysisException`
* Cleaned up Z-index updating sequence;
Tidying up comments, java-docs;
* Fixed Z-index to properly handle changes of the list of clustered columns
* Tidying up
* `lint`
* Suppressing `JavaDocStyle` first sentence check
* Fixed compilation
* Fixing incorrect `DecimalType` conversion
* Refactored test `TestTableLayoutOptimization`
- Added Z-index table composition test (against fixtures)
- Separated out GC test;
Tidying up
* Fixed tests re-shuffling column order for Z-Index table `DataFrame` to align w/ the one by one loaded from JSON
* Scaffolded `DataTypeUtils` to do basic checks of Spark types;
Added proper compatibility checking b/w old/new index-tables
* Added test for Z-index tables merging
* Fixed import being shaded by creating internal `hudi.util` package
* Fixed packaging for `TestOptimizeTable`
* Revised `updateMetadataIndex` seq to provide Z-index updating process w/ source table schema
* Make sure existing Z-index table schema is sync'd to source table's one
* Fixed shaded refs
* Fixed tests
* Fixed type conversion of Parquet provided metadata values into Spark expected schemas
* Fixed `composeIndexSchema` utility to propose proper schema
* Added more tests for Z-index:
- Checking that Z-index table is built correctly
- Checking that Z-index tables are merged correctly (during update)
* Fixing source table
* Fixing tests to read from Parquet w/ proper schema
* Refactored `ParquetUtils` utility reading stats from Parquet footers
* Fixed incorrect handling of Decimals extracted from Parquet footers
* Worked around issues in javac failign to compile stream's collection
* Fixed handling of `Date` type
* Fixed handling of `DateType` to be parsed as `LocalDate`
* Updated fixture;
Make sure test loads Z-index fixture using proper schema
* Removed superfluous scheme adjusting when reading from Parquet, since Spark is actually able to perfectly restore schema (given Parquet was previously written by Spark as well)
* Fixing race-condition in Parquet's `DateStringifier` trying to share `SimpleDataFormat` object which is inherently not thread-safe
* Tidying up
* Make sure schema is used upon reading to validate input files are in the appropriate format;
Tidying up;
* Worked around javac (1.8) inability to infer expression type properly
* Updated fixtures;
Tidying up
* Fixing compilation after rebase
* Assert clustering have in Z-order layout optimization testing
* Tidying up exception messages
* XXX
* Added test validating Z-index lookup filter correctness
* Added more test-cases;
Tidying up
* Added tests for string expressions
* Fixed incorrect Z-index filter lookup translations
* Added more test-cases
* Added proper handling on complex negations of AND/OR expressions by pushing NOT operator down into inner expressions for appropriate handling
* Added `-target:jvm-1.8` for `hudi-spark` module
* Adding more tests
* Added tests for non-indexed columns
* Properly handle non-indexed columns by falling back to a re-write of containing expression as `TrueLiteral` instead
* Fixed tests
* Removing the parquet test files and disabling corresponding tests
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
* Rebased `DFSPropertiesConfiguration` to access Hadoop config in liue of FS to avoid confusion
* Fixed `readConfig` to take Hadoop's `Configuration` instead of FS;
Fixing usages
* Added test for local FS access
* Rebase to use `FSUtils.getFs`
* Combine properties provided as a file along w/ overrides provided from the CLI
* Added helper utilities to `HoodieClusteringConfig`;
Make sure corresponding config methods fallback to defaults;
* Fixed DeltaStreamer usage to respect properly combined configuration;
Abstracted `HoodieClusteringConfig.from` convenience utility to init Clustering config from `Properties`
* Tidying up
* `lint`
* Reverting changes to `HoodieWriteConfig`
* Tdiying up
* Fixed incorrect merge of the props
* Converted `HoodieConfig` to wrap around `Properties` into `TypedProperties`
* Fixed compilation
* Fixed compilation
* Making error -> warn logs from timeline server with concurrent writers for inconsistent state
* Fixing bad request response exception for timeline out of sync
* Addressing feedback. removed write concurrency mode depedency
* [HUDI-2443] Hudi KVComparator for all HFile writer usages
- Hudi relies on custom class shading for Hbase's KeyValue.KVComparator to
avoid versioning and class loading issues. There are few places which are
still using the Hbase's comparator class directly and version upgrades
would make them obsolete. Refactoring the HoodieKVComparator and making
all HFile writer creation using the same shaded class.
* [HUDI-2443] Hudi KVComparator for all HFile writer usages
- Moving HoodieKVComparator from common.bootstrap.index to common.util
* [HUDI-2443] Hudi KVComparator for all HFile writer usages
- Retaining the old HoodieKVComparatorV2 for boostrap case. Adding the
new comparator as HoodieKVComparatorV2 to differentiate from the old
one.
* [HUDI-2443] Hudi KVComparator for all HFile writer usages
- Renamed HoodieKVComparatorV2 to HoodieMetadataKVComparator and moved it
under the package org.apache.hudi.metadata.
* Make comparator classname configurable
* Revert new config and address other review comments
Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
* [HUDI-2332] Add clustering and compaction in Kafka Connect Sink
* Disable validation check on instant time for compaction and adjust configs
* Add javadocs
* Add clustering and compaction config
* Fix transaction causing missing records in the target table
* Add debugging logs
* Fix kafka offset sync in participant
* Adjust how clustering and compaction are configured in kafka-connect
* Fix clustering strategy
* Remove irrelevant changes from other published PRs
* Update clustering logic and others
* Update README
* Fix test failures
* Fix indentation
* Fix clustering config
* Add JavaCustomColumnsSortPartitioner and make async compaction enabled by default
* Add test for JavaCustomColumnsSortPartitioner
* Add more changes after IDE sync
* Update README with clarification
* Fix clustering logic after rebasing
* Remove unrelated changes
- Adds support for generating commit timestamps with millisecs granularity.
- Older commit timestamps (in secs granularity) will be suffixed with 999 and parsed with millisecs format.
* [HUDI-2795] Add mechanism to safely update,delete and recover table properties
- Fail safe mechanism, that lets queries succeed off a backup file
- Readers who are not upgraded to this version of code will just fail until recovery is done.
- Added unit tests that exercises all these scenarios.
- Adding CLI for recovery, updation to table command.
- [Pending] Add some hash based verfication to ensure any rare partial writes for HDFS
* Fixing upgrade/downgrade infrastructure to use new updation method
- Metadata table today has virtual keys disabled, thereby populating the metafields
for each record written out and increasing the overall storage space used. Hereby
adding virtual keys support for metadata table so that metafields are disabled
for metadata table records.
- Adding a custom KeyGenerator for Metadata table so as to not rely on the
default Base/SimpleKeyGenerators which currently look for record key
and partition field set in the table config.
- AbstractHoodieLogRecordReader's version of processing next data block and
createHoodieRecord() will be a generic version and making the derived class
HoodieMetadataMergedLogRecordReader take care of the special creation of
records from explictly passed in partition names.
- ExternalSpillableMap does the payload/value size estimation on the first put to
determine when to spill over to disk map. The payload size re-estimation also
happens after a minimum threshold of puts. This size re-estimation goes my the
current in-memory map size for calculating average payload size and does attempts
divide by zero operation when the map is size is empty. Avoiding the
ArithmeticException during the payload size re-estimate by checking the map size
upfront.
* [HUDI-2634] Improved the metadata table bootstrap for very large tables.
Following improvements are implemented:
1. Memory overhead reduction:
- Existing code caches FileStatus for each file in memory.
- Created a new class DirectoryInfo which is used to cache a director's file list with parts of the FileStatus (only filename and file len). This reduces the memory requirements.
2. Improved parallelism:
- Existing code collects all the listing to the Driver and then creates HoodieRecord on the Driver.
- This takes a long time for large tables (11million HoodieRecords to be created)
- Created a new function in SparkRDDWriteClient specifically for bootstrap commit. In it, the HoodieRecord creation is parallelized across executors so it completes fast.
3. Fixed setting to limit the number of parallel listings:
- Existing code had a bug wherein 1500 executors were hardcoded to perform listing. This leads to exception due to limit in the spark's result memory.
- Corrected the use of the config.
Result:
Dataset has 1299 partitions and 12Million files.
file listing time=1.5mins
HoodieRecord creation time=13seconds
deltacommit duration=2.6mins
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
* [HUDI-2101]support z-order for hudi
* Renaming some configs for consistency/simplicity.
* Minor code cleanups
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
