* [HUDI-1653] Add support for composite keys in NonpartitionedKeyGenerator
* update NonpartitionedKeyGenerator to support composite record keys
* update NonpartitionedKeyGenerator
* [HUDI-845] Added locking capability to allow multiple writers
1. Added LockProvider API for pluggable lock methodologies
2. Added Resolution Strategy API to allow for pluggable conflict resolution
3. Added TableService client API to schedule table services
4. Added Transaction Manager for wrapping actions within transactions
* [HUDI-1552] Improve performance of key lookups from base file in Metadata Table.
1. Cache the KeyScanner across lookups so that the HFile index does not have to be read for each lookup.
2. Enable block caching in KeyScanner.
3. Move the lock to a limited scope of the code to reduce lock contention.
4. Removed reuse configuration
* Properly close the readers, when metadata table is accessed from executors
- Passing a reuse boolean into HoodieBackedTableMetadata
- Preserve the fast return behavior when reusing and opening from multiple threads (no contention)
- Handle concurrent close() and open readers, for reuse=false, by always synchronizing
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
In order to support object storage, we need these changes:
* Use the Hadoop filesystem so that we can find the plugin filesystem
* Do not fetch file size until the file handle is closed
* Do not close the opened filesystem because we want to use the
filesystem cache
Parameterized test case like `org.apache.hudi.table.upgrade.TestUpgradeDowngrade#testUpgrade` incurs flakiness when org.apache.hadoop.fs.FileSystem#closeAll is invoked at BeforeEach; it should be invoked in AfterAll instead.
- introduce configs to control how compaction is triggered
- Compaction can be triggered using time, number of delta commits and/or combinations
- Default behaviour remains the same.
This is the #step 2 of RFC-24:
https://cwiki.apache.org/confluence/display/HUDI/RFC+-+24%3A+Hoodie+Flink+Writer+Proposal
This PR introduce a BucketAssigner that assigns bucket ID (partition
path & fileID) for each stream record.
There is no need to look up index and partition the records anymore in
the following pipeline for these records,
we actually decide the write target location before the write and each
record computes its location when the BucketAssigner receives it, thus,
the indexing is with streaming style.
Computing locations for a batch of records all at a time is resource
consuming so a pressure to the engine,
we should avoid that in streaming system.
* Added HoodieConcatHandle to skip merging for "insert" operation when the corresponding config is set
Co-authored-by: Sivabalan Narayanan <sivabala@uber.com>
