lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,328 Commits 1 Branch 0 Tags
0d8ca8da4e0f6651bc1f06dba5e7e37881225fdc
Commit Graph

710 Commits

Author SHA1 Message Date
Sivabalan Narayanan
98ec215079 [HUDI-3178] Fixing metadata table compaction so as to not include uncommitted data (#4530) 
- There is a chance that the actual write eventually failed in data table but commit was successful in Metadata table, and if compaction was triggered in MDT, compaction could have included the uncommitted data. But once compacted, it may never be ignored while reading from metadata table. So, this patch fixes the bug. Metadata table compaction is triggered before applying the commit to metadata table to circumvent this issue.
 2022-01-08 10:34:47 -05:00
Sagar Sumit
827549949c [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator (#4203) 
* [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator

Timestampbased key generator was returning diff values for row writer and non row writer path. this patch fixes it and is guarded by a config flag (`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled`)
 2022-01-08 10:22:44 -05:00
Sivabalan Narayanan
8718c30324 [HUDI-3165] Enabling InProcessLockProvider for all multi-writer tests instead of FileSystemBasedLockProviderTestClass (#4427) 2022-01-06 13:04:10 -05:00
Sivabalan Narayanan
2954027b92 [HUDI-52] Enabling savepoint and restore for MOR table (#4507) 
* Enabling restore for MOR table

* Fixing savepoint for compaction commits in MOR
 2022-01-06 21:26:08 +05:30
Sivabalan Narayanan
b6891d253f [HUDI-44] Adding support to preserve commit metadata for compaction (#4428) 2022-01-06 20:27:37 +05:30
Sagar Sumit
75133f9942 [HUDI-3170] Do not preserve filename when preserveCommitMetadata enabled (#4512) 2022-01-05 08:09:58 -05:00
harshal
2b2ae34cb9 [HUDI-2558] Fixing Clustering w/ sort columns with null values fails (#4404) 2022-01-03 12:19:43 +05:30
Yuwei XIAO
2444f40a4b [HUDI-3095] abstract partition filter logic to enable code reuse (#4454) 
* [HUDI-3095] abstract partition filter logic to enable code reuse

* [HUDI-3095] address reviews
 2021-12-31 11:07:52 +05:30
Shawy Geng
a4e622ac61 [HUDI-1951] Add bucket hash index, compatible with the hive bucket (#3173) 
* [HUDI-2154] Add index key field to HoodieKey

* [HUDI-2157] Add the bucket index and its read/write implemention of Spark engine.
* revert HUDI-2154 add index key field to HoodieKey
* fix all comments and introduce a new tricky way to get index key at runtime
support double insert for bucket index
* revert spark read optimizer based on bucket index
* add the storage layout
* index tag, hash function and add ut
* fix ut
* address partial comments
* Code review feedback
* add layout config and docs
* fix ut
* rename hoodie.layout and rebase master

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-12-30 12:38:26 -08:00
董可伦
436becf3ea [HUDI-2675] Fix the exception 'Not an Avro data file' when archive and clean (#4016) 2021-12-29 22:53:17 -05:00
Ron
674c149234 [HUDI-3083] Support component data types for flink bulk_insert (#4470) 
* [HUDI-3083] Support component data types for flink bulk_insert

* add nested row type test
 2021-12-30 11:15:54 +08:00
Sivabalan Narayanan
5c0e4ce005 Revert "[HUDI-3043] Revert async cleaner leak commit to unblock CI failure (#4343)" (#4465) 
This reverts commit 7e7ad1558c.
 2021-12-30 10:45:09 +08:00
Yann Byron
05942e018c [HUDI-2811] Support Spark 3.2 (#4270) 2021-12-28 00:12:44 -08:00
Danny Chan
c81df99e50 [HUDI-3102] Do not store rollback plan in inflight instant (#4445) 2021-12-25 18:10:43 +08:00
Danny Chan
7b07aac286 [HUDI-3101] Excluding compaction instants from pending rollback info (#4443) 2021-12-25 14:10:45 +08:00
Sivabalan Narayanan
1a5f8693aa [HUDI-3011] Adding ability to read entire data with HoodieIncrSource with empty checkpoint (#4334) 
* Adding ability to read entire data with HoodieIncrSource with empty checkpoint

* Addressing comments
 2021-12-22 15:43:06 +05:30
Danny Chan
f1286c2c76 [HUDI-3032] Do not clean the log files right after compaction for metadata table (#4336) 2021-12-22 11:10:27 +08:00
Raymond Xu
32a44bbe06 [HUDI-2970] Add test for archiving replace commit (#4345) 2021-12-21 00:01:59 -05:00
Manoj Govindassamy
4a48f99a59 [HUDI-3064][HUDI-3054] FileSystemBasedLockProviderTestClass tryLock fix and TestHoodieClientMultiWriter test fixes (#4384) 
- Made FileSystemBasedLockProviderTestClass thread safe and fixed the
   tryLock retry logic.

 - Made TestHoodieClientMultiWriter. testHoodieClientBasicMultiWriter
   deterministic in verifying the HoodieWriteConflictException.
 2021-12-19 13:31:02 -05:00
Sivabalan Narayanan
77abb5ccb9 [HUDI-3054] Fixing default lock configs for FileSystemBasedLock and fixing a flaky test (#4374) 2021-12-18 16:15:48 -05:00
Sivabalan Narayanan
dc40397fa9 [HUDI-3064] Fixing a bug in TransactionManager and FileSystemTestLock (#4372) 2021-12-18 11:52:11 -05:00
Manoj Govindassamy
733732be7b [HUDI-3029] Transaction manager: avoid deadlock when doing begin and end transactions (#4373) 2021-12-18 11:43:10 -05:00
Manoj Govindassamy
d1d48ed494 [HUDI-3029] Transaction manager: avoid deadlock when doing begin and end transactions (#4363) 
* [HUDI-3029] Transaction manager: avoid deadlock when doing begin and end transactions

 - Transaction manager has begin and end transactions as synchronized methods.
   Based on the lock provider implementaion, this can lead to deadlock
   situation when the underlying lock() calls are blocking or with a long timeout.

 - Fixing transaction manager begin and end transactions to not get to deadlock
   and to not assume anything on the lock provider implementation.
 2021-12-18 09:43:17 -05:00
Manoj Govindassamy
7784249e55 [HUDI-2962] InProcess lock provider to guard single writer process with async table operations (#4259) 
- Adding Local JVM process based lock provider implementation

 - This local lock provider can be used by a single writer process with async
   table operations to guard the metadata tabl against concurrent updates.
 2021-12-17 20:18:46 -05:00
xiarixiaoyao
9246b16492 [HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType (#4253) 2021-12-17 08:58:02 -05:00
Sivabalan Narayanan
7e7ad1558c [HUDI-3043] Revert async cleaner leak commit to unblock CI failure (#4343) 
* Revert "[HUDI-2959] Fix the thread leak of cleaning service (#4252)"
Reverting to unblock CI failure for now. will revisit this with the right fix
 2021-12-16 21:51:28 -05:00
xiarixiaoyao
294d712948 [HUDI-3001] Clean up the marker directory when finish bootstrap operation. (#4298) 2021-12-16 12:36:01 -08:00
zhangyue19921010
a8a192aef5 [Minor] Catch and ignore all the exceptions in quietDeleteMarkerDir (#4301) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-12-16 11:15:08 -08:00
Danny Chan
ea2eba1a55 [HUDI-3015] Implement #reset and #sync for metadata filesystem view (#4307) 2021-12-16 15:26:16 +08:00
WangMinChao
9a2030ab31 [HUDI-3024] Add explicit write handler for flink (#4329) 
Co-authored-by: wangminchao <wangminchao@asinking.com>
 2021-12-15 20:16:48 +08:00
Danny Chan
c8d6bd8898 [HUDI-2997] Skip the corrupt meta file for pending rollback action (#4296) 2021-12-14 14:08:13 +08:00
Manoj Govindassamy
b22c2c611b [HUDI-2938] Metadata table util to get latest file slices for reader/writers (#4218) 2021-12-11 20:42:36 -08:00
Y Ethan Guo
b5f05fd153 [HUDI-2906] Add a repair util to clean up dangling data and log files (#4278) 2021-12-11 00:16:05 -08:00
Danny Chan
9bdcee00c0 [HUDI-2959] Fix the thread leak of cleaning service (#4252) 2021-12-11 12:08:47 +08:00
rmahindra123
9797fdfbb2 [HUDI-2974] Make the prefix for metrics name configurable (#4274) 
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-12-10 19:42:20 -08:00
Manoj Govindassamy
c48a2a125a [HUDI-2527] Multi writer test with conflicting async table services (#4046) 2021-12-10 20:01:19 -05:00
Alexey Kudinkin
2d864f7524 [HUDI-2814] Make Z-index more generic Column-Stats Index (#4106) 2021-12-10 14:56:09 -08:00
zhangyue19921010
3ba2909690 [HUDI-2892][BUG] Pending Clustering may stain the ActiveTimeLine and lead to incomplete query results (#4172) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-12-10 09:57:01 -08:00
Sivabalan Narayanan
be368264f4 [HUDI-2952] Fixing metadata table for non-partitioned dataset (#4243) 2021-12-10 11:11:42 -05:00
Yuwei XIAO
f194566ed4 [HUDI-2849] Improve SparkUI job description for write path (#4222) 2021-12-10 23:22:37 +08:00
xiarixiaoyao
456d74ce4e [HUDI-2901] Fixed the bug clustering jobs cannot running in parallel (#4178) 2021-12-09 22:39:35 -08:00
Y Ethan Guo
2d66451a51 [MINOR] Fix partition path formatting in error log (#4168) 2021-12-06 11:11:44 -08:00
Sivabalan Narayanan
1d4fb827e7 [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight (#4206) 
* [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

* Fixing retry of pending compaction in metadata table and enhancing tests
 2021-12-03 21:44:50 -08:00
vinoth chandar
0fd6b2d71e [HUDI-2933] DISABLE Metadata table by default (#4213) 2021-12-03 21:12:35 -08:00
Raymond Xu
a799fae316 [MINOR] Mitigate CI jobs timeout issues (#4173) 
* skip shutdown zookeeper in `@AfterAll` in TestHBaseIndex

* rebalance CI tests
 2021-12-03 21:08:32 -08:00
Sivabalan Narayanan
e483f7c776 [HUDI-2902] Fixing populate meta fields with Hfile writers and Disabling virtual keys by default for metadata table (#4194) 2021-12-03 07:20:21 -05:00
rmahindra123
91d2e61433 [HUDI-2904] Fix metadata table archival overstepping between regular writers and table services (#4186) 
- Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
- Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-12-02 13:32:26 -05:00
Alexey Kudinkin
772f5ca24e Fixed partitions produced by layout optimization in case order-by key is composed of a single column (#4183) 2021-12-01 20:56:04 -08:00
Shawy Geng
5284730175 [HUDI-2881] Compact the file group with larger log files to reduce write amplification (#4152) 2021-12-02 09:41:04 +08:00
Alexey Kudinkin
24380c2060 Revert "[HUDI-2855] Change the default value of 'PAYLOAD_CLASS_NAME' to 'DefaultHoodieRecordPayload' (#4115)" (#4169) 
This reverts commit 88067f57a2.
 2021-11-30 17:47:16 -08:00