lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,762 Commits 1 Branch 0 Tags
60758b36ea033a5985eabc53da5f6036ffaa5c0d
Commit Graph

161 Commits

Author SHA1 Message Date
Danny Chan
60758b36ea [HUDI-2227] Only sync hive meta on successful commit for flink batch writer (#3351) 2021-07-27 20:10:08 +08:00
Danny Chan
9d2a65a6a6 [HUDI-2209] Bulk insert for flink writer (#3334) 2021-07-27 10:58:23 +08:00
xiang2102
024cf01f02 [MINOR] Correct the words accroding in the comments to according (#3343) 
Correct the words 'accroding' in the comments to 'according'
 2021-07-27 08:48:58 +08:00
rmahindra123
a14b19fdd5 [HUDI-1241] Automate the generation of configs webpage as configs are added to Hudi repo (#3302) 2021-07-23 21:33:34 -07:00
Xuedong Luan
b2f7fcb8c8 [MINOR] Replace deprecated method isDir with isDirectory (#3319) 2021-07-24 10:02:24 +08:00
pengzhiwei
2c910ee3af [HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table (#3332) 2021-07-23 15:21:57 +08:00
Danny Chan
c89bf1de20 [HUDI-2205] Rollback inflight compaction for flink writer (#3320) 2021-07-22 22:56:51 +08:00
swuferhong
fe5d2e7f53 [HUDI-2206] Fix checkpoint blocked because getLastPendingInstant() action after than restoreWriteMetadata() action (#3326) 2021-07-22 16:35:07 +08:00
Danny Chan
858e84b5b2 [HUDI-2198] Clean and reset the bootstrap events for coordinator when task failover (#3304) 2021-07-21 10:13:05 +08:00
yuzhaojing
634163a990 [HUDI-2145] Create new bucket when NewFileAssignState filled (#3258) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-07-20 17:46:45 +08:00
喻兆靖
2099bf41db [HUDI-2193] Remove state in BootstrapFunction 2021-07-19 18:14:06 +08:00
yuzhao.cyz
50c2b76d72 Revert "[HUDI-2087] Support Append only in Flink stream (#3252)" 
This reverts commit 783c9cb3
 2021-07-16 21:36:27 +08:00
yuzhao.cyz
c8aaf00819 [HUDI-2185] Remove the default parallelism of index bootstrap and bucket assigner 2021-07-16 15:44:15 +08:00
moranyuwen
23a4a96eb4 [HUDI-2153] Fix BucketAssignFunction Context NullPointerException 2021-07-15 19:54:49 +08:00
vinoth chandar
75040ee9e5 [HUDI-2149] Ensure and Audit docs for every configuration class in the codebase (#3272) 
- Added docs when missing
 - Rewrote, reworded as needed
 - Made couple more classes extend HoodieConfig
 2021-07-14 10:56:08 -07:00
vinoyang
52524b659d [HUDI-2165] Support Transformer for HoodieFlinkStreamer (#3270) 
* [HUDI-2165] Support Transformer for HoodieFlinkStreamer
 2021-07-14 23:01:52 +08:00
喻兆靖
aff1a1ed29 [HUDI-2171] Add parallelism conf for bootstrap operator 2021-07-13 17:55:12 +08:00
swuferhong
9b01d2a045 [HUDI-2142] Support setting bucket assign parallelism for flink write task (#3239) 2021-07-10 15:43:36 +08:00
Danny Chan
942a024e74 [HUDI-2143] Tweak the default compaction target IO to 500GB when flink async compaction is off (#3238) 2021-07-10 15:40:30 +08:00
yuzhaojing
783c9cb369 [HUDI-2087] Support Append only in Flink stream (#3252) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-07-10 14:49:35 +08:00
vinoyang
7c6eebf98c [MINOR] Fix some wrong assert reasons (#3248) 2021-07-10 14:35:40 +08:00
wangxianghu
3b2a4f2b6b [HUDI-2147] Remove unused class AvroConvertor in hudi-flink (#3243) 2021-07-10 10:16:33 +08:00
vinoth chandar
b4562e86e4 Revert "[HUDI-2087] Support Append only in Flink stream (#3174)" (#3251) 
This reverts commit 371526789d.
 2021-07-09 11:20:09 -07:00
yuzhaojing
371526789d [HUDI-2087] Support Append only in Flink stream (#3174) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-07-09 16:06:32 +08:00
wangxianghu
f2621da32f [HUDI-2093] Fix empty avro schema path caused by duplicate parameters (#3177) 
* [HUDI-2093] Fix empty avro schema path caused by duplicate parameters

* rename shcmea option key

* fix doc

* rename var name
 2021-07-06 15:14:30 +08:00
Danny Chan
1d6978cde4 [HUDI-2135] Add compaction schedule option for flink (#3226) 2021-07-06 14:11:20 +08:00
swuferhong
bc313727e3 [HUDI-2106] Fix flink batch compaction bug while user don't set compaction tasks (#3192) 2021-07-06 09:10:37 +08:00
Danny Chan
32bd8ce088 [HUDI-2132] Make coordinator events as POJO for efficient serialization (#3223) 2021-07-06 09:02:38 +08:00
Danny Chan
e6ee7bdb51 [HUDI-2129] StreamerUtil.medianInstantTime should return a valid date time string (#3221) 2021-07-05 20:56:24 +08:00
Danny Chan
98ec017bc8 [HUDI-2126] The coordinator send events to write function when there are no data for the checkpoint (#3219) 2021-07-05 11:34:18 +08:00
Danny Chan
d424fe6072 [HUDI-2121] Add operator uid for flink stateful operators (#3212) 2021-07-02 19:44:32 +08:00
Danny Chan
7462fdefc3 [HUDI-2112] Support reading pure logs file group for flink batch reader after compaction (#3202) 2021-07-02 16:29:22 +08:00
pengzhiwei
b34d53fa9c [HUDI-2088] Missing Partition Fields And PreCombineField In Hoodie Properties For Table Written By Flink (#3171) 2021-07-01 17:25:18 +08:00
vinoth chandar
d07def1290 [MINOR] Fix broken build due to FlinkOptions (#3198) 2021-06-30 20:34:58 -07:00
wenningd
d412fb2fe6 [HUDI-89] Add configOption & refactor all configs based on that (#2833) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2021-06-30 14:26:30 -07:00
yuzhaojing
07e93de8b4 [HUDI-2052] Support load logFile in BootstrapFunction (#3134) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-30 20:37:00 +08:00
yuzhaojing
1cbf43b6e7 [HUDI-2103] Add rebalance before index bootstrap (#3185) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-30 16:40:55 +08:00
wangxianghu
202887b8ca [HUDI-2092] Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value (#3176) 2021-06-30 09:21:06 +08:00
swuferhong
f665db071f [HUDI-2085] Support specify compaction paralleism and compaction target io for flink batch compaction (#3169) 2021-06-29 22:53:01 +08:00
swuferhong
5a7d1b3d6c [HUDI-2097] Fix Flink unable to read commit metadata error (#3180) 2021-06-29 22:43:47 +08:00
Danny Chan
b8a8f572d6 [HUDI-2094] Supports hive style partitioning for flink writer (#3178) 2021-06-29 15:34:26 +08:00
yuzhaojing
37b7c65d8a [HUDI-2084] Resend the uncommitted write metadata when start up (#3168) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-29 08:53:52 +08:00
Vinay Patil
34fc8a8880 [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig (#3151) 2021-06-28 19:26:08 +08:00
wangxianghu
9e61dad597 [MINOR] Drop duplicate keygenerator class configuration setting (#3167) 2021-06-28 17:11:32 +08:00
Danny Chan
d24341d10c [HUDI-2074] Use while loop instead of recursive call in MergeOnReadInputFormat#MergeIterator to avoid StackOverflow (#3159) 2021-06-28 16:03:10 +08:00
wangxianghu
f73bedd374 [MINOR] Remove unused methods (#3152) 2021-06-26 13:19:26 +08:00
Danny Chan
e64fe55054 [HUDI-2068] Skip the assign state for SmallFileAssign when the state can not assign initially (#3148) 2021-06-25 08:57:56 +08:00
yuzhaojing
218f2a6df8 [HUDI-2062] Catch FileNotFoundException in WriteProfiles #getCommitMetadata Safely (#3138) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-25 08:54:59 +08:00
yuzhaojing
380518e232 [HUDI-2038] Support rollback inflight compaction instances for CompactionPlanOperator (#3105) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-23 20:58:52 +08:00
Danny Chan
2687eab8f0 [HUDI-2054] Remove the duplicate name for flink write pipeline (#3135) 2021-06-23 14:49:38 +08:00