lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,392 Commits 1 Branch 0 Tags
02073235c3bbe72db32ed282b3e318e7daa0800a
Commit Graph

32 Commits

Author SHA1 Message Date
Raymond Xu
9437e0ddef [MINOR] Fix import in StreamerUtil.java (#2638) 2021-03-07 12:37:15 -08:00
pengzhiwei
bc883db5de [HUDI-1636] Support Builder Pattern To Build Table Properties For HoodieTableConfig (#2596) 2021-03-05 14:10:27 +08:00
Danny Chan
89003bc780 [HUDI-1647] Supports snapshot read for Flink (#2613) 2021-03-05 08:49:32 +08:00
ZhangChaoMing
0dde7f9185 [HUDI-1584] Modify maker file path, which should start with the target base path. (#2539) 2021-03-02 17:52:21 +08:00
Danny Chan
7a11de1276 [HUDI-1632] Supports merge on read write mode for Flink writer (#2593) 
Also supports async compaction with pluggable strategies.
 2021-03-01 12:29:41 +08:00
Danny Chan
06dc7c7fd8 [HUDI-1638] Some improvements to BucketAssignFunction (#2600) 
- The #initializeState executes before #open, thus, the
  #checkPartitionsLoaded may see null `initialPartitionsToLoad`
  - Only load the existing partitions
 2021-02-25 14:33:21 +08:00
Danny Chan
3ceb1b4c83 [HUDI-1624] The state based index should bootstrap from existing base files (#2581) 2021-02-23 13:37:44 +08:00
ZhangChaoMing
43a0776c7c [HUDI-1586] [Common Core] [Flink Integration] Reduce the coupling of hadoop. (#2540) 
Co-authored-by: zhangchaoming <zhangchaoming@360.com>
 2021-02-21 11:54:04 +08:00
n3nash
ffcfb58bac [HUDI-1486] Remove inline inflight rollback in hoodie writer (#2359) 
1. Refactor rollback and move cleaning failed commits logic into cleaner
2. Introduce hoodie heartbeat to ascertain failed commits
3. Fix test cases
 2021-02-19 20:12:22 -08:00
lamber-ken
c4bbcb7f0e [HUDI-1621] Gets the parallelism from context when init StreamWriteOperatorCoordinator (#2579) 2021-02-17 20:04:38 +08:00
Danny Chan
5d2491d10c [HUDI-1598] Write as minor batches during one checkpoint interval for the new writer (#2553) 2021-02-17 15:24:50 +08:00
lamber-ken
ff0e3f5669 [HUDI-1612] Fix write test flakiness in StreamWriteITCase (#2567) 
* [HUDI-1612] Fix write test flakiness in StreamWriteITCase
 2021-02-11 23:37:19 +08:00
Danny Chan
4c5b6923cc [HUDI-1557] Make Flink write pipeline write task scalable (#2506) 
This is the #step 2 of RFC-24:
https://cwiki.apache.org/confluence/display/HUDI/RFC+-+24%3A+Hoodie+Flink+Writer+Proposal

This PR introduce a BucketAssigner that assigns bucket ID (partition
path & fileID) for each stream record.

There is no need to look up index and partition the records anymore in
the following pipeline for these records,
we actually decide the write target location before the write and each
record computes its location when the BucketAssigner receives it, thus,
the indexing is with streaming style.

Computing locations for a batch of records all at a time is resource
consuming so a pressure to the engine,
we should avoid that in streaming system.
 2021-02-06 22:03:52 +08:00
ZhangChaoMing
291f92069e [MINOR] Fix wrong logic for checking state condition (#2524) 2021-02-06 16:40:31 +08:00
wangxianghu
647e9faf25 [HUDI-1547] CI intermittent failure: TestJsonStringToHoodieRecordMapF… (#2521) 2021-02-04 11:20:01 +08:00
Danny Chan
bc0325f6ea [HUDI-1522] Add a new pipeline for Flink writer (#2430) 
* [HUDI-1522] Add a new pipeline for Flink writer
 2021-01-28 08:53:13 +08:00
luokey
a54550d94f [MINOR]Fix NPE when using HoodieFlinkStreamer with multi parallelism (#2492) 2021-01-27 21:00:20 +08:00
wangxianghu
e302c6bc12 [HUDI-1453] Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi (#2474) 2021-01-23 10:27:40 +08:00
wangxianghu
748dcc9aae [MINOR] Remove InstantGeneratorOperator parallelism limit in HoodieFlinkStreamer and update docs (#2471) 2021-01-22 13:46:25 +08:00
luokey
b64d22e047 [HUDI-1511] InstantGenerateOperator support multiple parallelism (#2434) 2021-01-22 09:17:50 +08:00
vinoth chandar
5e30fc1b2b [MINOR] Disabling problematic tests temporarily to stabilize CI (#2468) 2021-01-20 14:24:34 -08:00
Vinoth Chandar
3719e7b388 Moving to 0.8.0-SNAPSHOT on master branch. 2021-01-20 11:31:22 -08:00
Sivabalan Narayanan
b9c2856d16 [HUDI-1535] Fix 0.7.0 snapshot (#2456) 
* Revert "[MINOR] Bumping snapshot version to 0.7.0 (#2435)"

This reverts commit a43e191d6c.

* Fixing 0.7.0 snapshot bump
 2021-01-19 12:20:43 -08:00
Sivabalan Narayanan
a43e191d6c [MINOR] Bumping snapshot version to 0.7.0 (#2435) 2021-01-16 09:56:28 -05:00
Udit Mehrotra
17df517b81 [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common (#2410) 2021-01-07 11:34:06 -08:00
Gary Li
c5e8a024f6 [HUDI-1418] Set up flink client unit test infra (#2281) 2020-12-31 08:57:22 +08:00
Sivabalan Narayanan
33d338f392 [HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue (#2311) 
* Added ability to pass in `properties` to payload methods, so they can perform table/record specific merges
* Added default methods so existing payload classes are backwards compatible. 
* Adding DefaultHoodiePayload to honor ordering while merging two records
* Fixing default payload based on feedback
 2020-12-19 19:19:42 -08:00
ChangLi
6a6b772c49 [MINOR] Fix error information in exception (#2341) 2020-12-16 19:37:01 +08:00
wangxianghu
007014c1ef [MINOR] Throw an exception when keyGenerator initialization failed (#2307) 2020-12-10 09:56:19 +08:00
wangxianghu
de2fbeac33 [HUDI-1412] Make HoodieWriteConfig support setting different default … (#2278) 
* [HUDI-1412] Make HoodieWriteConfig support setting different default value according to engine type
 2020-12-07 09:29:53 +08:00
Gary Li
c8d5ea2752 [MINOR] clean up and add comments to flink client (#2261) 2020-11-19 15:27:52 +08:00
wangxianghu
4d05680038 [HUDI-1327] Introduce base implemetation of hudi-flink-client (#2176) 2020-11-18 17:57:11 +08:00