lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,561 Commits 1 Branch 0 Tags
7fa2f8ea827257e0320f670e5f22a114a52933cf
Commit Graph

1561 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Danny Chan
7fa2f8ea82 [HUDI-1921] Add target io option for flink compaction (#2980) 2021-06-02 10:10:35 +08:00
Danny Chan
bf1cfb5635 [HUDI-1949] Refactor BucketAssigner to make it more efficient (#3017) 
Add a process single class WriteProfile, the record and small files
profile re-construction can be more efficient if we reuse by same
checkpoint id.
 2021-06-02 09:12:35 +08:00
taylorliao
83c31e356f [HUDI-1927] Improve HoodieFlinkStreamer (#3019) 
Co-authored-by: enter58xuan <enter58xuan@zto.com>
 2021-06-01 18:35:14 +08:00
hk__lrzy
83b0301c1a [HUDI-1943] Lose properties when hoodieWriteConfig initializtion (#3006) 
* [hudi-flink]fix lose properties problem

Co-authored-by: haoke <haoke@bytedance.com>
 2021-06-01 16:09:48 +08:00
Wei
e6a71ea544 [MINOR] Access the static member getLastHeartbeatTime via the class instead (#3015) 2021-05-31 18:54:05 +08:00
Wei
219b92c8ae [MINOR] The collection can use forEach() directly (#3016) 2021-05-31 18:52:30 +08:00
Danny Chan
34ab756a40 [HUDI-1948] Shade kryo-shaded jar for hudi flink bundle (#3014) 2021-05-31 17:39:19 +08:00
Yao WANG
7a63175a70 fix the grammer err of the comment (#3013) 
Co-authored-by: ywang46 <ywang46@paypal.com>
 2021-05-31 11:44:25 +08:00
Wei
d965b0550f [MINOR] 'return' is unnecessary as the last statement in a 'void' method (#3012) 2021-05-31 11:43:10 +08:00
pengzhiwei
dcd7c331dc [HUDI-1879] Support Partition Prune For MergeOnRead Snapshot Table (#2926) 2021-05-29 07:50:24 -07:00
rmpifer
0709c62a6b [HUDI-1800] Exclude file slices in pending compaction when performing small file sizing (#2902) 
Co-authored-by: Ryan Pifer <ryanpife@amazon.com>
 2021-05-29 08:06:01 -04:00
wangxianghu
974b476180 [HUDI-1940] Add SqlQueryBasedTransformer unit test (#3004) 2021-05-28 22:30:30 +08:00
yuzhaojing
bc18c39835 [FLINK-1923] Exactly-once write for flink writer (#3002) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-05-28 14:58:21 +08:00
Danny Chan
7fed7352bd [HUDI-1865] Make embedded time line service singleton (#2899) 2021-05-27 13:38:33 +08:00
Vinay Patil
4eb6ef8144 [HUDI-1935] Updated Logger statement (#2996) 
Co-authored-by: veenaypatil <vinay18.patil@gmail.com>
 2021-05-26 15:04:58 +08:00
leesf
112732db81 [HUDI-1922] Bulk insert with row writer supports mor table (#2981) 2021-05-25 09:40:22 -07:00
Raymond Xu
afa6bc0b10 [HUDI-1723] Fix path selector listing files with the same mod date (#2845) 2021-05-25 10:19:10 -04:00
wangxianghu
e7020748b5 [HUDI-1920] Set archived as the default value of HOODIE_ARCHIVELOG_FOLDER_PROP_NAME (#2978) 2021-05-25 16:29:55 +08:00
Town
aba1eadbfc [HUDI-1919] Type mismatch when streaming read copy_on_write table using flink (#2986) 
* [HUDI-1919] Type mismatch when streaming read copy_on_write table using flink #2976

* Update ParquetSplitReaderUtil.java
 2021-05-25 11:36:43 +08:00
mpouttu
369a849337 [HUDI-1873] collect() call causing issues with very large upserts (#2907) 
Co-authored-by: Sivabalan Narayanan <sivabala@uber.com>
 2021-05-24 01:29:01 -04:00
wangxianghu
6539813733 [MINOR] Update the javadoc of EngineType (#2979) 2021-05-22 19:44:08 +08:00
Susu Dong
685f77b5dd [HUDI-1740] Fix insert-overwrite API archival (#2784) 
- fix problem of archiving replace commits
- Fix problem when getting empty replacecommit.requested
- Improved the logic of handling empty and non-empty requested/inflight commit files. Added unit tests to cover both empty and non-empty inflight files cases and cleaned up some unused test util methods

Co-authored-by: yorkzero831 <yorkzero8312@gmail.com>
Co-authored-by: zheren.yu <zheren.yu@paypay-corp.co.jp>
 2021-05-21 13:52:13 -07:00
zhangminglei
99b14a78e3 [HUDI-1918] Fix incorrect keyBy field cause serious data skew, to avoid multiple subtasks write to a partition at the same time (#2972) 2021-05-21 21:59:47 +08:00
Y Ethan Guo
a96034d38d [HUDI-1888] Fix NPE when the nested partition path field has null value (#2957) 2021-05-21 08:28:11 -04:00
Danny Chan
7c213f9f26 [HUDI-1917] Remove the metadata sync logic in HoodieFlinkWriteClient#preWrite because it is not thread safe (#2971) 2021-05-21 11:29:54 +08:00
xiarixiaoyao
081061e14b [HUDI-1719] hive on spark/mr,Incremental query of the mor table, the partition field is incorrect (#2720) 2021-05-20 11:00:08 -04:00
swuferhong
928b09ea0b [HUDI-1871] Fix hive conf for Flink writer hive meta sync (#2968) 2021-05-20 17:03:52 +08:00
Danny Chan
9b01d2f864 [HUDI-1915] Fix the file id for write data buffer before flushing (#2966) 2021-05-20 10:20:08 +08:00
wangxianghu
ced068e1ee [MINOR] Remove unused method in BaseSparkCommitActionExecutor (#2965) 2021-05-20 10:18:07 +08:00
zhangminglei
fe3f5c2d56 [HUDI-1913] Using streams instead of loops for input/output (#2962) 2021-05-19 09:13:38 +08:00
Sivabalan Narayanan
5d1f592395 [HUDI-1806] Honoring skipROSuffix in spark ds (#2882) 
* Honoring skipROSuffix in spark ds

* Adding tests

* fixing scala checkstype issue
 2021-05-18 16:11:39 -07:00
Danny Chan
7d2971d4e2 [HUDI-1911] Reuse the partition path and file group id for flink write data buffer (#2961) 
Reuse to reduce memory footprint.
 2021-05-18 17:47:22 +08:00
Danny Chan
46a2399a45 [HUDI-1902] Global index for flink writer (#2958) 
Supports deduplication for record keys with different partition path.
 2021-05-18 13:55:38 +08:00
Roc Marshal
fcedbfcb58 [MINOR][hudi-client] Code-cleanup,remove redundant variable declarations (#2956) 2021-05-17 13:34:42 +08:00
Danny Chan
8869b3b418 [HUDI-1902] Clean the corrupted files generated by FlinkMergeAndReplaceHandle (#2949) 
Make the intermediate files of FlinkMergeAndReplaceHandle hidden, when
committing the instant, clean these files in case there was some
corrupted files left(in normal case, the intermediate files should be cleaned
by the FlinkMergeAndReplaceHandle itself).
 2021-05-14 15:43:37 +08:00
xoln ann
12443e4187 [HUDI-1446] Support skip bootstrapIndex's init in abstract fs view init (#2520) 
Co-authored-by: zhongliang <zhongliang@kuaishou.com>
Co-authored-by: Sivabalan Narayanan <sivabala@uber.com>
 2021-05-14 00:29:26 -04:00
Danny Chan
ad77cf42ba [HUDI-1900] Always close the file handle for a flink mini-batch write (#2943) 
Close the file handle eagerly to avoid corrupted files as much as
possible.
 2021-05-14 10:25:18 +08:00
xiarixiaoyao
6f7ff7e8ca [HUDI-1722]Fix hive beeline/spark-sql query specified field on mor table occur NPE (#2722) 2021-05-12 20:52:37 +08:00
Danny Chan
b98c9ab439 [HUDI-1895] Close the file handles gracefully for flink write function to avoid corrupted files (#2938) 2021-05-12 18:44:10 +08:00
lw0090
5a8b2a4f86 [HUDI-1768] add spark datasource unit test for schema validate add column (#2776) 2021-05-11 16:49:18 -04:00
TeRS-K
be9db2c4f5 [HUDI-1055] Remove hardcoded parquet in tests (#2740) 
* Remove hardcoded parquet in tests
* Use DataFileUtils.getInstance
* Renaming DataFileUtils to BaseFileUtils

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-05-11 10:01:45 -07:00
Sivabalan Narayanan
ac72470e10 [HUDI-1851] Adding test suite long running automate scripts for docker (#2880) 2021-05-11 01:26:01 -07:00
hiscat
7a5af806cf [HUDI-1818] Validate required fields for Flink HoodieTable (#2930) 2021-05-11 11:11:19 +08:00
Danny Chan
42ec7e30d7 [HUDI-1890] FlinkCreateHandle and FlinkAppendHandle canWrite should always return true (#2933) 
The method #canWrite should always return true because they can already
write based on file size, e.g. the BucketAssigner.
 2021-05-11 09:14:51 +08:00
jsbali
aa398f77f1 [HUDI-1789] Support reading older snapshots (#2809) 
* [HUDI-1789] In HoodieParquetInoutFormat we currently default to the latest version of base files.
This PR attempts to add a new jobConf
 `hoodie.%s.consume.snapshot.time`

This new config will allow us to read older snapshots.

-  Reusing hoodie.%s.consume.commit for point in time snapshot queries as well.
-  Adding javadocs and some more tests
 2021-05-10 15:26:49 -07:00
Volodymyr Burenin
8a48d16e41 [HUDI-1707] Reduces log level for too verbose messages from info to debug level. (#2714) 
* Reduces log level for too verbose messages from info to debug level.
* Sort config output.
* Code Review : Small restructuring + rebasing to master
 - Fixing flaky multi delta streamer test
 - Using isDebugEnabled() checks
 - Some changes to shorten log message without moving to DEBUG

Co-authored-by: volodymyr.burenin <volodymyr.burenin@cloudkitchens.com>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-05-10 07:16:02 -07:00
hiscat
511ac4881d [MINOR] optimize FilePathUtils (#2931) 2021-05-10 06:47:56 -07:00
Danny Chan
c1b331bcff [HUDI-1886] Avoid to generates corrupted files for flink sink (#2929) 2021-05-10 10:43:03 +08:00
Danny Chan
bfbf993cbe [HUDI-1878] Add max memory option for flink writer task (#2920) 
Also removes the rate limiter because it has the similar functionality,
modify the create and merge handle cleans the retry files automatically.
 2021-05-08 14:27:56 +08:00
li36909
2c5a661a64 [HUDI-1759] Save one connection retry to hive metastore when hiveSyncTool run with useJdbc=false (#2759) 
* [HUDI-1759] Save one connection retry to hive metastore when hiveSyncTool run with useJdbc=false

* Fix review comment
 2021-05-07 15:30:26 -07:00