lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,282 Commits 1 Branch 0 Tags
3d7a8695cdd76cdf359f9d1bb7108a49b41d64c4
Commit Graph

2282 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Sivabalan Narayanan
3d7a8695cd Fixing dynamoDbLockConfig required prop check (#4422) 2021-12-28 15:56:30 +05:30
Yann Byron
05942e018c [HUDI-2811] Support Spark 3.2 (#4270) 2021-12-28 00:12:44 -08:00
ForwardXu
32505d5adb [HUDI-3106] Fix HiveSyncTool not sync schema (#4452) 2021-12-27 22:11:14 -08:00
Yann Byron
1f7afba5e4 [HUDI-3093] fix spark-sql query table that write with TimestampBasedKeyGenerator (#4416) 2021-12-27 21:39:52 -08:00
harshal
6409fc733d [HUDI-2374] Fixing AvroDFSSource does not use the overridden schema to deserialize Avro binaries (#4353) 2021-12-27 23:01:21 -05:00
ForwardXu
282aa68552 [HUDI-3099] Purge drop partition for spark sql (#4436) 2021-12-28 09:38:26 +08:00
Danny Chan
c81df99e50 [HUDI-3102] Do not store rollback plan in inflight instant (#4445) 2021-12-25 18:10:43 +08:00
Danny Chan
7b07aac286 [HUDI-3101] Excluding compaction instants from pending rollback info (#4443) 2021-12-25 14:10:45 +08:00
xuzifu666
4721073b43 [MINOR] Remove unused method in HoodieActiveTimeline (#4435) 2021-12-24 22:29:34 +08:00
xuzifu666
032b883bd1 [HUDI-3014] Add table option to set utc timezone (#4306) 2021-12-23 16:27:45 +08:00
Aimiyoo
57f43de1ea [MINOR] Fix DedupeSparkJob typo (#4418) 2021-12-22 11:51:26 -08:00
ForwardXu
5d93edc539 [HUDI-3060] drop table for spark sql (#4364) 2021-12-22 19:17:43 +08:00
Sivabalan Narayanan
1a5f8693aa [HUDI-3011] Adding ability to read entire data with HoodieIncrSource with empty checkpoint (#4334) 
* Adding ability to read entire data with HoodieIncrSource with empty checkpoint

* Addressing comments
 2021-12-22 15:43:06 +05:30
xiarixiaoyao
b5890cd17d Merge pull request #4308 from harsh1231/HUDI-3008 
[HUDI-3008] Fixing HoodieFileIndex partition column parsing for nested fields
 2021-12-22 16:46:57 +08:00
yuzhaojing
15eb7e81fc [HUDI-2547] Schedule Flink compaction in service (#4254) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-12-22 15:08:47 +08:00
Danny Chan
f1286c2c76 [HUDI-3032] Do not clean the log files right after compaction for metadata table (#4336) 2021-12-22 11:10:27 +08:00
Aimiyoo
92f54ce3d8 [HUDI-3027] Update hudi-examples README.md (#4330) 2021-12-21 13:36:03 -08:00
harshal patil
7d046f914a [HUDI-3008] Fixing HoodieFileIndex partition column parsing for nested fields 2021-12-21 11:54:52 +05:30
Raymond Xu
32a44bbe06 [HUDI-2970] Add test for archiving replace commit (#4345) 2021-12-21 00:01:59 -05:00
zhangyue19921010
f3f6112b75 [HUDI-3070] Add rerunFailingTestsCount for flakly testes (#4398) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-12-20 19:59:50 -08:00
Sivabalan Narayanan
982ae3d1eb [MINOR] Increasing CI timeout to 90 mins (#4407) 2021-12-20 20:27:22 -05:00
xuzifu666
f166ddad12 [MINOR] Remove unused method in HoodieActiveTimeline (#4401) 2021-12-20 22:19:37 +08:00
xuzifu666
3ca92108b2 remove unused import (#4349) 2021-12-20 16:32:41 +08:00
Manoj Govindassamy
4a48f99a59 [HUDI-3064][HUDI-3054] FileSystemBasedLockProviderTestClass tryLock fix and TestHoodieClientMultiWriter test fixes (#4384) 
- Made FileSystemBasedLockProviderTestClass thread safe and fixed the
   tryLock retry logic.

 - Made TestHoodieClientMultiWriter. testHoodieClientBasicMultiWriter
   deterministic in verifying the HoodieWriteConflictException.
 2021-12-19 13:31:02 -05:00
Sivabalan Narayanan
03f71ef1a2 [HUDI-2970] Adding tests for archival of replace commit actions (#4268) 2021-12-18 23:59:39 -08:00
Danny Chan
478f9f3695 [minor] fix NetworkUtils#getHostname (#4355) 2021-12-19 10:09:48 +08:00
Raymond Xu
bb99836841 [HUDI-3052] Fix flaky testJsonKafkaSourceResetStrategy (#4381) 2021-12-18 20:58:51 -05:00
Raymond Xu
f57e28fe39 [MINOR] Azure CI IT tasks clean up (#4337) 2021-12-18 17:00:56 -08:00
Sivabalan Narayanan
77abb5ccb9 [HUDI-3054] Fixing default lock configs for FileSystemBasedLock and fixing a flaky test (#4374) 2021-12-18 16:15:48 -05:00
Sivabalan Narayanan
dc40397fa9 [HUDI-3064] Fixing a bug in TransactionManager and FileSystemTestLock (#4372) 2021-12-18 11:52:11 -05:00
Manoj Govindassamy
733732be7b [HUDI-3029] Transaction manager: avoid deadlock when doing begin and end transactions (#4373) 2021-12-18 11:43:10 -05:00
Manoj Govindassamy
d1d48ed494 [HUDI-3029] Transaction manager: avoid deadlock when doing begin and end transactions (#4363) 
* [HUDI-3029] Transaction manager: avoid deadlock when doing begin and end transactions

 - Transaction manager has begin and end transactions as synchronized methods.
   Based on the lock provider implementaion, this can lead to deadlock
   situation when the underlying lock() calls are blocking or with a long timeout.

 - Fixing transaction manager begin and end transactions to not get to deadlock
   and to not assume anything on the lock provider implementation.
 2021-12-18 09:43:17 -05:00
Sivabalan Narayanan
47852446e8 [HUDI-3043] De-coupling multi writer tests (#4362) 2021-12-17 21:37:45 -05:00
Manoj Govindassamy
7784249e55 [HUDI-2962] InProcess lock provider to guard single writer process with async table operations (#4259) 
- Adding Local JVM process based lock provider implementation

 - This local lock provider can be used by a single writer process with async
   table operations to guard the metadata tabl against concurrent updates.
 2021-12-17 20:18:46 -05:00
Sivabalan Narayanan
6eba8345cb [HUDI-3043] Adding some test fixes to continuous mode multi writer tests (#4356) 2021-12-17 15:45:05 -05:00
xiarixiaoyao
9246b16492 [HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType (#4253) 2021-12-17 08:58:02 -05:00
yuzhaojing
e4cfb421c0 [HUDI-3046] Claim RFC number for RFC for Compaction / Clustering Service (#4347) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-12-17 14:52:45 +08:00
Danny Chan
d0087d4040 [HUDI-3037] Add back remote view storage config for flink (#4338) 2021-12-17 13:57:53 +08:00
Sivabalan Narayanan
7e7ad1558c [HUDI-3043] Revert async cleaner leak commit to unblock CI failure (#4343) 
* Revert "[HUDI-2959] Fix the thread leak of cleaning service (#4252)"
Reverting to unblock CI failure for now. will revisit this with the right fix
 2021-12-16 21:51:28 -05:00
xiarixiaoyao
294d712948 [HUDI-3001] Clean up the marker directory when finish bootstrap operation. (#4298) 2021-12-16 12:36:01 -08:00
zhangyue19921010
a8a192aef5 [Minor] Catch and ignore all the exceptions in quietDeleteMarkerDir (#4301) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-12-16 11:15:08 -08:00
Danny Chan
ea2eba1a55 [HUDI-3015] Implement #reset and #sync for metadata filesystem view (#4307) 2021-12-16 15:26:16 +08:00
Yuwei XIAO
f5b07a77bc [HUDI-2998] claiming rfc number for consistent hashing index (#4303) 
Co-authored-by: xiaoyuwei <xiaoyuwei.yw@alibaba-inc.com>
 2021-12-15 15:47:22 -08:00
Raymond Xu
27907dea4c [HUDI-3028] Use blob storage to speed up CI downloads (#4331) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-12-15 15:33:33 -08:00
Y Ethan Guo
3b89457e2b [HUDI-3025] Add additional wait time for namenode availability during IT tests initiatialization (#4328) 
- Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-12-15 13:44:42 -05:00
WangMinChao
9a2030ab31 [HUDI-3024] Add explicit write handler for flink (#4329) 
Co-authored-by: wangminchao <wangminchao@asinking.com>
 2021-12-15 20:16:48 +08:00
ForwardXu
dbec6c512b [HUDI-3022] Fix NPE for isDropPartition method (#4319) 
* [HUDI-3022] Fix NPE for isDropPartition method
 2021-12-15 19:38:02 +08:00
Manoj Govindassamy
bc8bf043d5 [HUDI-2995] Enabling metadata table by default (#4295) 
- Enabling metadata table by default
 2021-12-14 13:11:19 -08:00
Danny Chan
c8d6bd8898 [HUDI-2997] Skip the corrupt meta file for pending rollback action (#4296) 2021-12-14 14:08:13 +08:00
Fugle666
29bc5fd912 [HUDI-2996] Flink streaming reader 'skip_compaction' option does not work (#4304) 
close apache/hudi#4304
 2021-12-14 12:21:09 +08:00