lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,391 Commits 1 Branch 0 Tags
e00a9042e9a4e97f6f8ccaed4ac51a9aa8c2da76
Commit Graph

284 Commits

Author SHA1 Message Date
Danny Chan
9762e4c08c [MINOR] Some cosmetic changes for Flink (#3503) 2021-08-19 23:21:20 +08:00
swuferhong
1fed44af84 [HUDI-2316] Support Flink batch upsert (#3494) 2021-08-19 17:15:26 +08:00
leiqiang
b7a0d76fc9 [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException 
close apache/hudi#3402
 2021-08-18 15:40:51 +08:00
Danny Chan
66f951322a [HUDI-2191] Bump flink version to 1.13.1 (#3291) 2021-08-16 18:14:05 +08:00
Udit Mehrotra
3e301196bf Moving to 0.10.0-SNAPSHOT on master branch. 2021-08-14 18:51:09 -07:00
Danny Chan
6a4100bb91 [MINOR] Tweak change log more as FULL for flink streaming source (#3466) 2021-08-13 16:31:16 +08:00
Sagar Sumit
0544d70d8f [MINOR] Deprecate older configs (#3464) 
Rename and deprecate props in HoodieWriteConfig

Rename and deprecate older props
 2021-08-12 20:31:04 -07:00
Danny Chan
29332498af [HUDI-2298] The HoodieMergedLogRecordScanner should set up the operation of the chosen record (#3456) 2021-08-11 22:55:43 +08:00
swuferhong
21db6d7a84 [HUDI-1771] Propagate CDC format for hoodie (#3285) 2021-08-10 20:23:23 +08:00
yuzhaojing
11ea74958d [HUDI-2247] Filter file where length less than parquet MAGIC length (#3363) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-08-09 09:15:42 +08:00
Danny Chan
b7586a5632 [HUDI-2274] Allows INSERT duplicates for Flink MOR table (#3403) 2021-08-06 10:30:52 +08:00
yuzhaojing
b8b9d6db83 [HUDI-2087] Support Append only in Flink stream (#3390) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-08-04 17:53:20 +08:00
Danny Chan
02331fc223 [HUDI-2258] Metadata table for flink (#3381) 2021-08-04 10:54:55 +08:00
wenningd
91bb0d1318 [HUDI-2255] Refactor Datasource options (#3373) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2021-08-03 17:50:30 -07:00
Danny Chan
bec23bda50 [HUDI-2269] Release the disk map resource for flink streaming reader (#3384) 2021-08-03 13:55:35 +08:00
swuferhong
f7f5d4cc6d [HUDI-2184] Support setting hive sync partition extractor class based on flink configuration (#3284) 2021-07-30 17:24:00 +08:00
Danny Chan
c4e45a0010 [HUDI-2254] Builtin sort operator for flink bulk insert (#3372) 2021-07-30 16:58:11 +08:00
swuferhong
8b19ec9ca0 [HUDI-2252] Default consumes from the latest instant for flink streaming reader (#3368) 2021-07-30 14:25:05 +08:00
Danny Chan
efbbb67420 [HUDI-2241] Explicit parallelism for flink bulk insert (#3357) 2021-07-29 09:57:37 +08:00
swuferhong
7739518879 [HUDI-2228] Add option 'hive_sync.mode' for flink writer (#3352) 2021-07-28 19:45:50 +08:00
Danny Chan
91c2213412 [HUDI-2245] BucketAssigner generates the fileId evenly to avoid data skew (#3362) 2021-07-28 19:26:37 +08:00
rmahindra123
8fef50e237 [HUDI-2044] Integrate consumers with rocksDB and compression within External Spillable Map (#3318) 2021-07-28 01:31:03 -04:00
mincwang
00cd35f90a [HUDI-2215] Add rateLimiter when Flink writes to hudi. (#3338) 
Co-authored-by: wangminchao <wangminchao@asinking.com>
 2021-07-28 08:23:23 +08:00
Danny Chan
60758b36ea [HUDI-2227] Only sync hive meta on successful commit for flink batch writer (#3351) 2021-07-27 20:10:08 +08:00
Danny Chan
9d2a65a6a6 [HUDI-2209] Bulk insert for flink writer (#3334) 2021-07-27 10:58:23 +08:00
xiang2102
024cf01f02 [MINOR] Correct the words accroding in the comments to according (#3343) 
Correct the words 'accroding' in the comments to 'according'
 2021-07-27 08:48:58 +08:00
rmahindra123
a14b19fdd5 [HUDI-1241] Automate the generation of configs webpage as configs are added to Hudi repo (#3302) 2021-07-23 21:33:34 -07:00
Xuedong Luan
b2f7fcb8c8 [MINOR] Replace deprecated method isDir with isDirectory (#3319) 2021-07-24 10:02:24 +08:00
pengzhiwei
2c910ee3af [HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table (#3332) 2021-07-23 15:21:57 +08:00
Danny Chan
c89bf1de20 [HUDI-2205] Rollback inflight compaction for flink writer (#3320) 2021-07-22 22:56:51 +08:00
swuferhong
fe5d2e7f53 [HUDI-2206] Fix checkpoint blocked because getLastPendingInstant() action after than restoreWriteMetadata() action (#3326) 2021-07-22 16:35:07 +08:00
Danny Chan
858e84b5b2 [HUDI-2198] Clean and reset the bootstrap events for coordinator when task failover (#3304) 2021-07-21 10:13:05 +08:00
yuzhaojing
634163a990 [HUDI-2145] Create new bucket when NewFileAssignState filled (#3258) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-07-20 17:46:45 +08:00
喻兆靖
2099bf41db [HUDI-2193] Remove state in BootstrapFunction 2021-07-19 18:14:06 +08:00
yuzhao.cyz
50c2b76d72 Revert "[HUDI-2087] Support Append only in Flink stream (#3252)" 
This reverts commit 783c9cb3
 2021-07-16 21:36:27 +08:00
yuzhao.cyz
c8aaf00819 [HUDI-2185] Remove the default parallelism of index bootstrap and bucket assigner 2021-07-16 15:44:15 +08:00
moranyuwen
23a4a96eb4 [HUDI-2153] Fix BucketAssignFunction Context NullPointerException 2021-07-15 19:54:49 +08:00
vinoth chandar
75040ee9e5 [HUDI-2149] Ensure and Audit docs for every configuration class in the codebase (#3272) 
- Added docs when missing
 - Rewrote, reworded as needed
 - Made couple more classes extend HoodieConfig
 2021-07-14 10:56:08 -07:00
vinoyang
52524b659d [HUDI-2165] Support Transformer for HoodieFlinkStreamer (#3270) 
* [HUDI-2165] Support Transformer for HoodieFlinkStreamer
 2021-07-14 23:01:52 +08:00
喻兆靖
aff1a1ed29 [HUDI-2171] Add parallelism conf for bootstrap operator 2021-07-13 17:55:12 +08:00
swuferhong
9b01d2a045 [HUDI-2142] Support setting bucket assign parallelism for flink write task (#3239) 2021-07-10 15:43:36 +08:00
Danny Chan
942a024e74 [HUDI-2143] Tweak the default compaction target IO to 500GB when flink async compaction is off (#3238) 2021-07-10 15:40:30 +08:00
yuzhaojing
783c9cb369 [HUDI-2087] Support Append only in Flink stream (#3252) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-07-10 14:49:35 +08:00
vinoyang
7c6eebf98c [MINOR] Fix some wrong assert reasons (#3248) 2021-07-10 14:35:40 +08:00
wangxianghu
3b2a4f2b6b [HUDI-2147] Remove unused class AvroConvertor in hudi-flink (#3243) 2021-07-10 10:16:33 +08:00
vinoth chandar
b4562e86e4 Revert "[HUDI-2087] Support Append only in Flink stream (#3174)" (#3251) 
This reverts commit 371526789d.
 2021-07-09 11:20:09 -07:00
yuzhaojing
371526789d [HUDI-2087] Support Append only in Flink stream (#3174) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-07-09 16:06:32 +08:00
wangxianghu
f2621da32f [HUDI-2093] Fix empty avro schema path caused by duplicate parameters (#3177) 
* [HUDI-2093] Fix empty avro schema path caused by duplicate parameters

* rename shcmea option key

* fix doc

* rename var name
 2021-07-06 15:14:30 +08:00
Danny Chan
1d6978cde4 [HUDI-2135] Add compaction schedule option for flink (#3226) 2021-07-06 14:11:20 +08:00
swuferhong
bc313727e3 [HUDI-2106] Fix flink batch compaction bug while user don't set compaction tasks (#3192) 2021-07-06 09:10:37 +08:00