1
0
Commit Graph

55 Commits

Author SHA1 Message Date
Danny Chan
4c5b6923cc [HUDI-1557] Make Flink write pipeline write task scalable (#2506)
This is the #step 2 of RFC-24:
https://cwiki.apache.org/confluence/display/HUDI/RFC+-+24%3A+Hoodie+Flink+Writer+Proposal

This PR introduce a BucketAssigner that assigns bucket ID (partition
path & fileID) for each stream record.

There is no need to look up index and partition the records anymore in
the following pipeline for these records,
we actually decide the write target location before the write and each
record computes its location when the BucketAssigner receives it, thus,
the indexing is with streaming style.

Computing locations for a batch of records all at a time is resource
consuming so a pressure to the engine,
we should avoid that in streaming system.
2021-02-06 22:03:52 +08:00
wangxianghu
647e9faf25 [HUDI-1547] CI intermittent failure: TestJsonStringToHoodieRecordMapF… (#2521) 2021-02-04 11:20:01 +08:00
Danny Chan
bc0325f6ea [HUDI-1522] Add a new pipeline for Flink writer (#2430)
* [HUDI-1522] Add a new pipeline for Flink writer
2021-01-28 08:53:13 +08:00
vinoth chandar
5e30fc1b2b [MINOR] Disabling problematic tests temporarily to stabilize CI (#2468) 2021-01-20 14:24:34 -08:00
Gary Li
c5e8a024f6 [HUDI-1418] Set up flink client unit test infra (#2281) 2020-12-31 08:57:22 +08:00