lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,249 Commits 1 Branch 0 Tags
7784249e55c8dc6c1e5fa1142f7a93d78ca66471
Commit Graph

2249 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Manoj Govindassamy
7784249e55 [HUDI-2962] InProcess lock provider to guard single writer process with async table operations (#4259) 
- Adding Local JVM process based lock provider implementation

 - This local lock provider can be used by a single writer process with async
   table operations to guard the metadata tabl against concurrent updates.
 2021-12-17 20:18:46 -05:00
Sivabalan Narayanan
6eba8345cb [HUDI-3043] Adding some test fixes to continuous mode multi writer tests (#4356) 2021-12-17 15:45:05 -05:00
xiarixiaoyao
9246b16492 [HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType (#4253) 2021-12-17 08:58:02 -05:00
yuzhaojing
e4cfb421c0 [HUDI-3046] Claim RFC number for RFC for Compaction / Clustering Service (#4347) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-12-17 14:52:45 +08:00
Danny Chan
d0087d4040 [HUDI-3037] Add back remote view storage config for flink (#4338) 2021-12-17 13:57:53 +08:00
Sivabalan Narayanan
7e7ad1558c [HUDI-3043] Revert async cleaner leak commit to unblock CI failure (#4343) 
* Revert "[HUDI-2959] Fix the thread leak of cleaning service (#4252)"
Reverting to unblock CI failure for now. will revisit this with the right fix
 2021-12-16 21:51:28 -05:00
xiarixiaoyao
294d712948 [HUDI-3001] Clean up the marker directory when finish bootstrap operation. (#4298) 2021-12-16 12:36:01 -08:00
zhangyue19921010
a8a192aef5 [Minor] Catch and ignore all the exceptions in quietDeleteMarkerDir (#4301) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-12-16 11:15:08 -08:00
Danny Chan
ea2eba1a55 [HUDI-3015] Implement #reset and #sync for metadata filesystem view (#4307) 2021-12-16 15:26:16 +08:00
Yuwei XIAO
f5b07a77bc [HUDI-2998] claiming rfc number for consistent hashing index (#4303) 
Co-authored-by: xiaoyuwei <xiaoyuwei.yw@alibaba-inc.com>
 2021-12-15 15:47:22 -08:00
Raymond Xu
27907dea4c [HUDI-3028] Use blob storage to speed up CI downloads (#4331) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-12-15 15:33:33 -08:00
Y Ethan Guo
3b89457e2b [HUDI-3025] Add additional wait time for namenode availability during IT tests initiatialization (#4328) 
- Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-12-15 13:44:42 -05:00
WangMinChao
9a2030ab31 [HUDI-3024] Add explicit write handler for flink (#4329) 
Co-authored-by: wangminchao <wangminchao@asinking.com>
 2021-12-15 20:16:48 +08:00
ForwardXu
dbec6c512b [HUDI-3022] Fix NPE for isDropPartition method (#4319) 
* [HUDI-3022] Fix NPE for isDropPartition method
 2021-12-15 19:38:02 +08:00
Manoj Govindassamy
bc8bf043d5 [HUDI-2995] Enabling metadata table by default (#4295) 
- Enabling metadata table by default
 2021-12-14 13:11:19 -08:00
Danny Chan
c8d6bd8898 [HUDI-2997] Skip the corrupt meta file for pending rollback action (#4296) 2021-12-14 14:08:13 +08:00
Fugle666
29bc5fd912 [HUDI-2996] Flink streaming reader 'skip_compaction' option does not work (#4304) 
close apache/hudi#4304
 2021-12-14 12:21:09 +08:00
WangMinChao
46de25d5fd [HUDI-2994] Add judgement to existed partitionPath in the catch code block for HU… (#4294) 
* [HUDI-2994] Add judgement to existed partition path in the catch code block for HUDI-2743

Co-authored-by: wangminchao <wangminchao@asinking.com>
 2021-12-13 20:41:03 +08:00
ForwardXu
dd96129191 [HUDI-2990] Sync to HMS when deleting partitions (#4291) 2021-12-13 20:40:06 +08:00
Manoj Govindassamy
b22c2c611b [HUDI-2938] Metadata table util to get latest file slices for reader/writers (#4218) 2021-12-11 20:42:36 -08:00
wenningd
15444c951f [HUDI-2946] Upgrade maven plugins to be compatible with higher Java versions (#4232) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2021-12-11 20:18:39 -08:00
Danny Chan
8dd0444ef9 [HUDI-2984] Implement #close for AbstractTableFileSystemView (#4285) 2021-12-11 16:19:10 +08:00
Y Ethan Guo
b5f05fd153 [HUDI-2906] Add a repair util to clean up dangling data and log files (#4278) 2021-12-11 00:16:05 -08:00
Danny Chan
2dcb3f0062 [HUDI-2985] Shade jackson for hudi flink bundle jar (#4284) 2021-12-11 14:40:57 +08:00
Danny Chan
9bdcee00c0 [HUDI-2959] Fix the thread leak of cleaning service (#4252) 2021-12-11 12:08:47 +08:00
rmahindra123
9797fdfbb2 [HUDI-2974] Make the prefix for metrics name configurable (#4274) 
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-12-10 19:42:20 -08:00
Manoj Govindassamy
c48a2a125a [HUDI-2527] Multi writer test with conflicting async table services (#4046) 2021-12-10 20:01:19 -05:00
Alexey Kudinkin
2d864f7524 [HUDI-2814] Make Z-index more generic Column-Stats Index (#4106) 2021-12-10 14:56:09 -08:00
Y Ethan Guo
72901a33a1 [HUDI-2784] Add a hudi-trino-bundle for Trino (#4279) 2021-12-10 14:27:22 -08:00
zhangyue19921010
3ba2909690 [HUDI-2892][BUG] Pending Clustering may stain the ActiveTimeLine and lead to incomplete query results (#4172) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-12-10 09:57:01 -08:00
Sivabalan Narayanan
3ce0526924 Adding verbose output for metadata validate files command (#4166) 2021-12-10 09:38:38 -08:00
yuzhaojing
3ad9b121f1 [HUDI-2912] Fix CompactionPlanOperator typo (#4187) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-12-10 09:32:53 -08:00
Sivabalan Narayanan
be368264f4 [HUDI-2952] Fixing metadata table for non-partitioned dataset (#4243) 2021-12-10 11:11:42 -05:00
Yuwei XIAO
f194566ed4 [HUDI-2849] Improve SparkUI job description for write path (#4222) 2021-12-10 23:22:37 +08:00
Sagar Sumit
c7473a7b0c [HUDI-2936] Add data count checks in async clustering tests (#4236) 2021-12-10 09:25:37 -05:00
xiarixiaoyao
456d74ce4e [HUDI-2901] Fixed the bug clustering jobs cannot running in parallel (#4178) 2021-12-09 22:39:35 -08:00
Sivabalan Narayanan
ea154bcb5d Revert "Claiming RFC for data skipping index for updated version (#4271)" (#4272) 
This reverts commit 8321d20c2c.
 2021-12-10 00:46:26 -05:00
Sivabalan Narayanan
8321d20c2c Claiming RFC for data skipping index for updated version (#4271) 2021-12-10 10:07:42 +05:30
arunkc
3fb2f974ca [MINOR] FAQ link in SUPPORT_REQUEST template (#4266) 2021-12-09 14:43:36 -08:00
xiarixiaoyao
68f8597b12 [HUDI-2966] Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScaner when the query finished. (#4265) 
* [HUDI-2966] Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScaner when the query finished.
 2021-12-09 19:51:49 +08:00
RexAn
f612a20815 [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown (#4014) 2021-12-09 16:04:11 +05:30
leesf
5ac9ce7289 [MINOR] Fix Compile broken (#4263) 2021-12-09 13:12:18 +08:00
guanziyue
9c8ad0f0fa [HUDI-2665] Fix overflow of huge log file in HoodieLogFormatWriter (#3912) 
Co-authored-by: guanziyue.gzy <guanziyue.gzy@bytedance.com>
 2021-12-09 10:47:13 +08:00
Danny Chan
bd08470421 [HUDI-2957] Shade kryo jar for flink bundle jar (#4251) 2021-12-09 10:16:42 +08:00
Sivabalan Narayanan
7c3f0777aa [HUDI-2964] Fixing aws lock configs to inherit from HoodieConfig (#4258) 2021-12-08 19:17:56 -05:00
Vinoth Govindarajan
082faa3851 [HUDI-2832][RFC-41] Proposal to integrate Hudi on Snowflake platform (#4074) 
* [HUDI-2832][RFC-40] Proposal to integrate Hudi on Snowflake platform

* rebased and addressed review comments
 2021-12-08 11:27:19 -08:00
Danny Chan
c56d93e7b8 [MINOR] Update DOAP with 0.10.0 Release (#4246) 2021-12-08 17:55:22 +08:00
xuzifu666
c9e18d1e7d [HUDI-2942] add error message log in HoodieCombineHiveInputFormat (#4224) 2021-12-07 22:05:39 -08:00
Danny Chan
e8473b9a2b [HUDI-2951] Disable remote view storage config for flink (#4237) 2021-12-07 18:04:15 +08:00
Sagar Sumit
6dab307e6f [MINOR] Remove redundant and conflicting spark-hive dependency (#4228) 
Disable TestHiveSchemaProvider
 2021-12-06 17:48:32 -08:00