lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,236 Commits 1 Branch 0 Tags
dbec6c512b1e5a93a60034a9894e10a65db34ea4
Commit Graph

2236 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
ForwardXu
dbec6c512b [HUDI-3022] Fix NPE for isDropPartition method (#4319) 
* [HUDI-3022] Fix NPE for isDropPartition method
 2021-12-15 19:38:02 +08:00
Manoj Govindassamy
bc8bf043d5 [HUDI-2995] Enabling metadata table by default (#4295) 
- Enabling metadata table by default
 2021-12-14 13:11:19 -08:00
Danny Chan
c8d6bd8898 [HUDI-2997] Skip the corrupt meta file for pending rollback action (#4296) 2021-12-14 14:08:13 +08:00
Fugle666
29bc5fd912 [HUDI-2996] Flink streaming reader 'skip_compaction' option does not work (#4304) 
close apache/hudi#4304
 2021-12-14 12:21:09 +08:00
WangMinChao
46de25d5fd [HUDI-2994] Add judgement to existed partitionPath in the catch code block for HU… (#4294) 
* [HUDI-2994] Add judgement to existed partition path in the catch code block for HUDI-2743

Co-authored-by: wangminchao <wangminchao@asinking.com>
 2021-12-13 20:41:03 +08:00
ForwardXu
dd96129191 [HUDI-2990] Sync to HMS when deleting partitions (#4291) 2021-12-13 20:40:06 +08:00
Manoj Govindassamy
b22c2c611b [HUDI-2938] Metadata table util to get latest file slices for reader/writers (#4218) 2021-12-11 20:42:36 -08:00
wenningd
15444c951f [HUDI-2946] Upgrade maven plugins to be compatible with higher Java versions (#4232) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2021-12-11 20:18:39 -08:00
Danny Chan
8dd0444ef9 [HUDI-2984] Implement #close for AbstractTableFileSystemView (#4285) 2021-12-11 16:19:10 +08:00
Y Ethan Guo
b5f05fd153 [HUDI-2906] Add a repair util to clean up dangling data and log files (#4278) 2021-12-11 00:16:05 -08:00
Danny Chan
2dcb3f0062 [HUDI-2985] Shade jackson for hudi flink bundle jar (#4284) 2021-12-11 14:40:57 +08:00
Danny Chan
9bdcee00c0 [HUDI-2959] Fix the thread leak of cleaning service (#4252) 2021-12-11 12:08:47 +08:00
rmahindra123
9797fdfbb2 [HUDI-2974] Make the prefix for metrics name configurable (#4274) 
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-12-10 19:42:20 -08:00
Manoj Govindassamy
c48a2a125a [HUDI-2527] Multi writer test with conflicting async table services (#4046) 2021-12-10 20:01:19 -05:00
Alexey Kudinkin
2d864f7524 [HUDI-2814] Make Z-index more generic Column-Stats Index (#4106) 2021-12-10 14:56:09 -08:00
Y Ethan Guo
72901a33a1 [HUDI-2784] Add a hudi-trino-bundle for Trino (#4279) 2021-12-10 14:27:22 -08:00
zhangyue19921010
3ba2909690 [HUDI-2892][BUG] Pending Clustering may stain the ActiveTimeLine and lead to incomplete query results (#4172) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-12-10 09:57:01 -08:00
Sivabalan Narayanan
3ce0526924 Adding verbose output for metadata validate files command (#4166) 2021-12-10 09:38:38 -08:00
yuzhaojing
3ad9b121f1 [HUDI-2912] Fix CompactionPlanOperator typo (#4187) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-12-10 09:32:53 -08:00
Sivabalan Narayanan
be368264f4 [HUDI-2952] Fixing metadata table for non-partitioned dataset (#4243) 2021-12-10 11:11:42 -05:00
Yuwei XIAO
f194566ed4 [HUDI-2849] Improve SparkUI job description for write path (#4222) 2021-12-10 23:22:37 +08:00
Sagar Sumit
c7473a7b0c [HUDI-2936] Add data count checks in async clustering tests (#4236) 2021-12-10 09:25:37 -05:00
xiarixiaoyao
456d74ce4e [HUDI-2901] Fixed the bug clustering jobs cannot running in parallel (#4178) 2021-12-09 22:39:35 -08:00
Sivabalan Narayanan
ea154bcb5d Revert "Claiming RFC for data skipping index for updated version (#4271)" (#4272) 
This reverts commit 8321d20c2c.
 2021-12-10 00:46:26 -05:00
Sivabalan Narayanan
8321d20c2c Claiming RFC for data skipping index for updated version (#4271) 2021-12-10 10:07:42 +05:30
arunkc
3fb2f974ca [MINOR] FAQ link in SUPPORT_REQUEST template (#4266) 2021-12-09 14:43:36 -08:00
xiarixiaoyao
68f8597b12 [HUDI-2966] Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScaner when the query finished. (#4265) 
* [HUDI-2966] Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScaner when the query finished.
 2021-12-09 19:51:49 +08:00
RexAn
f612a20815 [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown (#4014) 2021-12-09 16:04:11 +05:30
leesf
5ac9ce7289 [MINOR] Fix Compile broken (#4263) 2021-12-09 13:12:18 +08:00
guanziyue
9c8ad0f0fa [HUDI-2665] Fix overflow of huge log file in HoodieLogFormatWriter (#3912) 
Co-authored-by: guanziyue.gzy <guanziyue.gzy@bytedance.com>
 2021-12-09 10:47:13 +08:00
Danny Chan
bd08470421 [HUDI-2957] Shade kryo jar for flink bundle jar (#4251) 2021-12-09 10:16:42 +08:00
Sivabalan Narayanan
7c3f0777aa [HUDI-2964] Fixing aws lock configs to inherit from HoodieConfig (#4258) 2021-12-08 19:17:56 -05:00
Vinoth Govindarajan
082faa3851 [HUDI-2832][RFC-41] Proposal to integrate Hudi on Snowflake platform (#4074) 
* [HUDI-2832][RFC-40] Proposal to integrate Hudi on Snowflake platform

* rebased and addressed review comments
 2021-12-08 11:27:19 -08:00
Danny Chan
c56d93e7b8 [MINOR] Update DOAP with 0.10.0 Release (#4246) 2021-12-08 17:55:22 +08:00
xuzifu666
c9e18d1e7d [HUDI-2942] add error message log in HoodieCombineHiveInputFormat (#4224) 2021-12-07 22:05:39 -08:00
Danny Chan
e8473b9a2b [HUDI-2951] Disable remote view storage config for flink (#4237) 2021-12-07 18:04:15 +08:00
Sagar Sumit
6dab307e6f [MINOR] Remove redundant and conflicting spark-hive dependency (#4228) 
Disable TestHiveSchemaProvider
 2021-12-06 17:48:32 -08:00
wenningd
4a437f25d3 [MINOR] Use maven-shade-plugin version for hudi-timeline-server-bundle from main pom.xml (#4209) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2021-12-06 12:29:18 -08:00
Y Ethan Guo
2d66451a51 [MINOR] Fix partition path formatting in error log (#4168) 2021-12-06 11:11:44 -08:00
xiarixiaoyao
57c4bf8152 [HUDI-2876] for hive/presto hudi should remove the temp file which created by HoodieMergedLogRecordSanner when the query finished. (#4139) 2021-12-06 21:33:10 +08:00
Ron
84b531ae75 [HUDI-2900] Fix corrupt block end position (#4181) 
* [HUDI-2900] Fix corrupt block end position

* add a test
 2021-12-06 20:38:39 +08:00
leesf
f0e46bf522 [HUDI-2916] Add IssueNavigationLink for IDEA (#4192) 2021-12-05 22:53:54 -08:00
冯健
734c9f5f2d [HUDI-2418] Support HiveSchemaProvider (#3671) 
Co-authored-by: jian.feng <fengjian428@gmial.com>
 2021-12-05 00:10:13 -08:00
ForwardXu
63b15607ff [HUDI-2937] Introduce a pulsar implementation of hoodie write commit … (#4217) 
* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback
 2021-12-05 11:51:06 +04:00
Ron
a8fb69656f [HUDI-2877] Support flink catalog to help user use flink table conveniently (#4153) 
* [HUDI-2877] Support flink catalog to help user use flink table conveniently

* Fix comment

* fix comment2
 2021-12-05 10:14:29 +08:00
vinoth chandar
36b69d8033 [HUDI-2935] Remove special casing of clustering in deltastreamer checkpoint retrival (#4216) 
- We now seek backwards to find the checkpoint
 - No need to return empty anymore
 2021-12-04 17:16:11 +08:00
fengli
568181a3e7 [HUDI-2934] Optimize RequestHandler code style 
close apache/hudi#4215
 2021-12-04 15:30:52 +08:00
Sivabalan Narayanan
1d4fb827e7 [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight (#4206) 
* [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

* Fixing retry of pending compaction in metadata table and enhancing tests
 2021-12-03 21:44:50 -08:00
rmahindra123
94f45e928c [HUDI-2890] Kafka Connect: Fix failed writes and avoid table service concurrent operations (#4211) 
* Fix kafka connect readme

* Fix handling of errors in write records for kafka connect

* By default, ensure we skip error records and keep the pipeline alive

* Fix indentation

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-12-03 21:30:32 -08:00
vinoth chandar
0fd6b2d71e [HUDI-2933] DISABLE Metadata table by default (#4213) 2021-12-03 21:12:35 -08:00