lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,776 Commits 1 Branch 0 Tags
c4e45a0010d3f13f84d90fbf417c4e5e6c547fda
Commit Graph

168 Commits

Author SHA1 Message Date
Danny Chan
c4e45a0010 [HUDI-2254] Builtin sort operator for flink bulk insert (#3372) 2021-07-30 16:58:11 +08:00
swuferhong
8b19ec9ca0 [HUDI-2252] Default consumes from the latest instant for flink streaming reader (#3368) 2021-07-30 14:25:05 +08:00
Danny Chan
efbbb67420 [HUDI-2241] Explicit parallelism for flink bulk insert (#3357) 2021-07-29 09:57:37 +08:00
swuferhong
7739518879 [HUDI-2228] Add option 'hive_sync.mode' for flink writer (#3352) 2021-07-28 19:45:50 +08:00
Danny Chan
91c2213412 [HUDI-2245] BucketAssigner generates the fileId evenly to avoid data skew (#3362) 2021-07-28 19:26:37 +08:00
rmahindra123
8fef50e237 [HUDI-2044] Integrate consumers with rocksDB and compression within External Spillable Map (#3318) 2021-07-28 01:31:03 -04:00
mincwang
00cd35f90a [HUDI-2215] Add rateLimiter when Flink writes to hudi. (#3338) 
Co-authored-by: wangminchao <wangminchao@asinking.com>
 2021-07-28 08:23:23 +08:00
Danny Chan
60758b36ea [HUDI-2227] Only sync hive meta on successful commit for flink batch writer (#3351) 2021-07-27 20:10:08 +08:00
Danny Chan
9d2a65a6a6 [HUDI-2209] Bulk insert for flink writer (#3334) 2021-07-27 10:58:23 +08:00
xiang2102
024cf01f02 [MINOR] Correct the words accroding in the comments to according (#3343) 
Correct the words 'accroding' in the comments to 'according'
 2021-07-27 08:48:58 +08:00
rmahindra123
a14b19fdd5 [HUDI-1241] Automate the generation of configs webpage as configs are added to Hudi repo (#3302) 2021-07-23 21:33:34 -07:00
Xuedong Luan
b2f7fcb8c8 [MINOR] Replace deprecated method isDir with isDirectory (#3319) 2021-07-24 10:02:24 +08:00
pengzhiwei
2c910ee3af [HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table (#3332) 2021-07-23 15:21:57 +08:00
Danny Chan
c89bf1de20 [HUDI-2205] Rollback inflight compaction for flink writer (#3320) 2021-07-22 22:56:51 +08:00
swuferhong
fe5d2e7f53 [HUDI-2206] Fix checkpoint blocked because getLastPendingInstant() action after than restoreWriteMetadata() action (#3326) 2021-07-22 16:35:07 +08:00
Danny Chan
858e84b5b2 [HUDI-2198] Clean and reset the bootstrap events for coordinator when task failover (#3304) 2021-07-21 10:13:05 +08:00
yuzhaojing
634163a990 [HUDI-2145] Create new bucket when NewFileAssignState filled (#3258) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-07-20 17:46:45 +08:00
喻兆靖
2099bf41db [HUDI-2193] Remove state in BootstrapFunction 2021-07-19 18:14:06 +08:00
yuzhao.cyz
50c2b76d72 Revert "[HUDI-2087] Support Append only in Flink stream (#3252)" 
This reverts commit 783c9cb3
 2021-07-16 21:36:27 +08:00
yuzhao.cyz
c8aaf00819 [HUDI-2185] Remove the default parallelism of index bootstrap and bucket assigner 2021-07-16 15:44:15 +08:00
moranyuwen
23a4a96eb4 [HUDI-2153] Fix BucketAssignFunction Context NullPointerException 2021-07-15 19:54:49 +08:00
vinoth chandar
75040ee9e5 [HUDI-2149] Ensure and Audit docs for every configuration class in the codebase (#3272) 
- Added docs when missing
 - Rewrote, reworded as needed
 - Made couple more classes extend HoodieConfig
 2021-07-14 10:56:08 -07:00
vinoyang
52524b659d [HUDI-2165] Support Transformer for HoodieFlinkStreamer (#3270) 
* [HUDI-2165] Support Transformer for HoodieFlinkStreamer
 2021-07-14 23:01:52 +08:00
喻兆靖
aff1a1ed29 [HUDI-2171] Add parallelism conf for bootstrap operator 2021-07-13 17:55:12 +08:00
swuferhong
9b01d2a045 [HUDI-2142] Support setting bucket assign parallelism for flink write task (#3239) 2021-07-10 15:43:36 +08:00
Danny Chan
942a024e74 [HUDI-2143] Tweak the default compaction target IO to 500GB when flink async compaction is off (#3238) 2021-07-10 15:40:30 +08:00
yuzhaojing
783c9cb369 [HUDI-2087] Support Append only in Flink stream (#3252) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-07-10 14:49:35 +08:00
vinoyang
7c6eebf98c [MINOR] Fix some wrong assert reasons (#3248) 2021-07-10 14:35:40 +08:00
wangxianghu
3b2a4f2b6b [HUDI-2147] Remove unused class AvroConvertor in hudi-flink (#3243) 2021-07-10 10:16:33 +08:00
vinoth chandar
b4562e86e4 Revert "[HUDI-2087] Support Append only in Flink stream (#3174)" (#3251) 
This reverts commit 371526789d.
 2021-07-09 11:20:09 -07:00
yuzhaojing
371526789d [HUDI-2087] Support Append only in Flink stream (#3174) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-07-09 16:06:32 +08:00
wangxianghu
f2621da32f [HUDI-2093] Fix empty avro schema path caused by duplicate parameters (#3177) 
* [HUDI-2093] Fix empty avro schema path caused by duplicate parameters

* rename shcmea option key

* fix doc

* rename var name
 2021-07-06 15:14:30 +08:00
Danny Chan
1d6978cde4 [HUDI-2135] Add compaction schedule option for flink (#3226) 2021-07-06 14:11:20 +08:00
swuferhong
bc313727e3 [HUDI-2106] Fix flink batch compaction bug while user don't set compaction tasks (#3192) 2021-07-06 09:10:37 +08:00
Danny Chan
32bd8ce088 [HUDI-2132] Make coordinator events as POJO for efficient serialization (#3223) 2021-07-06 09:02:38 +08:00
Danny Chan
e6ee7bdb51 [HUDI-2129] StreamerUtil.medianInstantTime should return a valid date time string (#3221) 2021-07-05 20:56:24 +08:00
Danny Chan
98ec017bc8 [HUDI-2126] The coordinator send events to write function when there are no data for the checkpoint (#3219) 2021-07-05 11:34:18 +08:00
Danny Chan
d424fe6072 [HUDI-2121] Add operator uid for flink stateful operators (#3212) 2021-07-02 19:44:32 +08:00
Danny Chan
7462fdefc3 [HUDI-2112] Support reading pure logs file group for flink batch reader after compaction (#3202) 2021-07-02 16:29:22 +08:00
pengzhiwei
b34d53fa9c [HUDI-2088] Missing Partition Fields And PreCombineField In Hoodie Properties For Table Written By Flink (#3171) 2021-07-01 17:25:18 +08:00
vinoth chandar
d07def1290 [MINOR] Fix broken build due to FlinkOptions (#3198) 2021-06-30 20:34:58 -07:00
wenningd
d412fb2fe6 [HUDI-89] Add configOption & refactor all configs based on that (#2833) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2021-06-30 14:26:30 -07:00
yuzhaojing
07e93de8b4 [HUDI-2052] Support load logFile in BootstrapFunction (#3134) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-30 20:37:00 +08:00
yuzhaojing
1cbf43b6e7 [HUDI-2103] Add rebalance before index bootstrap (#3185) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-30 16:40:55 +08:00
wangxianghu
202887b8ca [HUDI-2092] Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value (#3176) 2021-06-30 09:21:06 +08:00
swuferhong
f665db071f [HUDI-2085] Support specify compaction paralleism and compaction target io for flink batch compaction (#3169) 2021-06-29 22:53:01 +08:00
swuferhong
5a7d1b3d6c [HUDI-2097] Fix Flink unable to read commit metadata error (#3180) 2021-06-29 22:43:47 +08:00
Danny Chan
b8a8f572d6 [HUDI-2094] Supports hive style partitioning for flink writer (#3178) 2021-06-29 15:34:26 +08:00
yuzhaojing
37b7c65d8a [HUDI-2084] Resend the uncommitted write metadata when start up (#3168) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-29 08:53:52 +08:00
Vinay Patil
34fc8a8880 [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig (#3151) 2021-06-28 19:26:08 +08:00