Danny Chan
5d2491d10c
[HUDI-1598] Write as minor batches during one checkpoint interval for the new writer ( #2553 )
2021-02-17 15:24:50 +08:00
lamber-ken
ff0e3f5669
[HUDI-1612] Fix write test flakiness in StreamWriteITCase ( #2567 )
...
* [HUDI-1612] Fix write test flakiness in StreamWriteITCase
2021-02-11 23:37:19 +08:00
Danny Chan
4c5b6923cc
[HUDI-1557] Make Flink write pipeline write task scalable ( #2506 )
...
This is the #step 2 of RFC-24:
https://cwiki.apache.org/confluence/display/HUDI/RFC+-+24%3A+Hoodie+Flink+Writer+Proposal
This PR introduce a BucketAssigner that assigns bucket ID (partition
path & fileID) for each stream record.
There is no need to look up index and partition the records anymore in
the following pipeline for these records,
we actually decide the write target location before the write and each
record computes its location when the BucketAssigner receives it, thus,
the indexing is with streaming style.
Computing locations for a batch of records all at a time is resource
consuming so a pressure to the engine,
we should avoid that in streaming system.
2021-02-06 22:03:52 +08:00
ZhangChaoMing
291f92069e
[MINOR] Fix wrong logic for checking state condition ( #2524 )
2021-02-06 16:40:31 +08:00
wangxianghu
647e9faf25
[HUDI-1547] CI intermittent failure: TestJsonStringToHoodieRecordMapF… ( #2521 )
2021-02-04 11:20:01 +08:00
Danny Chan
bc0325f6ea
[HUDI-1522] Add a new pipeline for Flink writer ( #2430 )
...
* [HUDI-1522] Add a new pipeline for Flink writer
2021-01-28 08:53:13 +08:00
luokey
a54550d94f
[MINOR]Fix NPE when using HoodieFlinkStreamer with multi parallelism ( #2492 )
2021-01-27 21:00:20 +08:00
wangxianghu
e302c6bc12
[HUDI-1453] Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi ( #2474 )
2021-01-23 10:27:40 +08:00
wangxianghu
748dcc9aae
[MINOR] Remove InstantGeneratorOperator parallelism limit in HoodieFlinkStreamer and update docs ( #2471 )
2021-01-22 13:46:25 +08:00
luokey
b64d22e047
[HUDI-1511] InstantGenerateOperator support multiple parallelism ( #2434 )
2021-01-22 09:17:50 +08:00
vinoth chandar
5e30fc1b2b
[MINOR] Disabling problematic tests temporarily to stabilize CI ( #2468 )
2021-01-20 14:24:34 -08:00
Vinoth Chandar
3719e7b388
Moving to 0.8.0-SNAPSHOT on master branch.
2021-01-20 11:31:22 -08:00
Sivabalan Narayanan
b9c2856d16
[HUDI-1535] Fix 0.7.0 snapshot ( #2456 )
...
* Revert "[MINOR] Bumping snapshot version to 0.7.0 (#2435 )"
This reverts commit a43e191d6c
2021-01-19 12:20:43 -08:00
Sivabalan Narayanan
a43e191d6c
[MINOR] Bumping snapshot version to 0.7.0 ( #2435 )
2021-01-16 09:56:28 -05:00
Udit Mehrotra
17df517b81
[HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common ( #2410 )
2021-01-07 11:34:06 -08:00
Gary Li
c5e8a024f6
[HUDI-1418] Set up flink client unit test infra ( #2281 )
2020-12-31 08:57:22 +08:00
Sivabalan Narayanan
33d338f392
[HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue ( #2311 )
...
* Added ability to pass in `properties` to payload methods, so they can perform table/record specific merges
* Added default methods so existing payload classes are backwards compatible.
* Adding DefaultHoodiePayload to honor ordering while merging two records
* Fixing default payload based on feedback
2020-12-19 19:19:42 -08:00
ChangLi
6a6b772c49
[MINOR] Fix error information in exception ( #2341 )
2020-12-16 19:37:01 +08:00
wangxianghu
007014c1ef
[MINOR] Throw an exception when keyGenerator initialization failed ( #2307 )
2020-12-10 09:56:19 +08:00
wangxianghu
de2fbeac33
[HUDI-1412] Make HoodieWriteConfig support setting different default … ( #2278 )
...
* [HUDI-1412] Make HoodieWriteConfig support setting different default value according to engine type
2020-12-07 09:29:53 +08:00
Gary Li
c8d5ea2752
[MINOR] clean up and add comments to flink client ( #2261 )
2020-11-19 15:27:52 +08:00
wangxianghu
4d05680038
[HUDI-1327] Introduce base implemetation of hudi-flink-client ( #2176 )
2020-11-18 17:57:11 +08:00
