lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
3,049 Commits 1 Branch 0 Tags
80368a049d0764b45ae0ce998a0e7d5897858615
Commit Graph

31 Commits

Author SHA1 Message Date
Danny Chan
0faa562b6f [HUDI-4403] Fix the end input metadata for bounded source (#6116) 2022-07-16 12:02:17 +08:00
Danny Chan
e70a427956 [HUDI-4391] Incremental read from archived commits for flink (#6096) 2022-07-14 15:19:26 +08:00
HunterXHunter
994c561488 [HUDI-4298] When reading the mor table with QUERY_TYPE_SNAPSHOT,Unabl… (#5937) 
* [HUDI-4298] Add test case for reading mor table

Signed-off-by: LinMingQiang <1356469429@qq.com>
 2022-07-12 14:49:44 +08:00
Danny Chan
a998586396 [minor] following 4152, refactor the clazz about plan selection strategy (#6060) 2022-07-08 09:56:10 +08:00
Lanyuanxiaoyao
e74ad324c3 [HUDI-4152] Flink offline compaction support compacting multi compaction plan at once (#5677) 
* [HUDI-4152] Flink offline compaction allow compact multi compaction plan at once

* [HUDI-4152] Fix exception for duplicated uid when multi compaction plan are compacted

* [HUDI-4152] Provider UT & IT for compact multi compaction plan

* [HUDI-4152] Put multi compaction plans into one compaction plan source

* [HUDI-4152] InstantCompactionPlanSelectStrategy allow multi instant by using comma

* [HUDI-4152] Add IT for InstantCompactionPlanSelectStrategy
 2022-07-07 14:11:26 +08:00
Danny Chan
7eeaff9ee0 [HUDI-4357] Support flink 1.15.x (#6050) 2022-07-06 13:42:58 +08:00
Shiyan Xu
c0e1587966 [HUDI-3730] Improve meta sync class design and hierarchies (#5854) 
* [HUDI-3730] Improve meta sync class design and hierarchies (#5754)
* Implements class design proposed in RFC-55

Co-authored-by: jian.feng <fengjian428@gmial.com>
Co-authored-by: jian.feng <jian.feng@shopee.com>
 2022-07-03 14:47:25 +05:30
Danny Chan
47792a3186 [HUDI-4353] Column stats data skipping for flink (#6026) 2022-07-03 08:29:31 +08:00
JerryYue-M
bdf73b2650 [HUDI-3953]Flink Hudi module should support low-level source and sink api (#5445) 
Co-authored-by: jerryyue <jerryyue@didiglobal.com>
 2022-07-02 08:38:46 +08:00
luokey
59978ef4a9 [HUDI-4260] Change KEYGEN_CLASS_NAME without default value (#5877) 
* Change KEYGEN_CLASS_NAME without default value

Co-authored-by: 854194341@qq.com <loukey_7821>
 2022-06-24 15:05:03 +08:00
Zhaojing Yu
6456bd3a51 [HUDI-4273] Support inline schedule clustering for Flink stream (#5890) 
* [HUDI-4273] Support inline schedule clustering for Flink stream

* delete deprecated clustering plan strategy and add clustering ITTest
 2022-06-24 11:28:06 +08:00
Danny Chan
1dbd9d407a [minor] following 4270, add unit tests for the keys lost case (#5918) 2022-06-22 16:56:06 +08:00
Alexander Trushev
f1103281d2 [HUDI-4258] Fix when HoodieTable removes data file before the end of Flink job (#5876) 
* [HUDI-4258] Fix when HoodieTable removes data file before the end of Flink job
 2022-06-20 17:07:49 +08:00
Danny Chan
22c45a7704 [HUDI-4188] Fix flaky ITTestDataSTreamWrite.testWriteCopyOnWrite (#5749) 2022-06-06 12:12:48 +08:00
喻兆靖
c20db99a7b [HUDI-2207] Support independent flink hudi clustering function 2022-05-24 20:16:48 +08:00
Danny Chan
43e08193ef [HUDI-4098] Metadata table heartbeat for instant has expired, last heartbeat 0 (#5583) 2022-05-16 17:40:08 +08:00
Bo Cui
7fb436d3cf [HUDI-4078][HUDI-FLINK]BootstrapOperator contains the pending compact… (#5545) 
* [HUDI-4078][HUDI-FLINK]BootstrapOperator contains the pending compaction files
 2022-05-13 14:32:48 +08:00
Bo Cui
701f8c039d [HUDI-3336][HUDI-FLINK]Support custom hadoop config for flink (#5528) 
* [HUDI-3336][HUDI-FLINK]Support custom hadoop config for flink
 2022-05-13 09:50:11 +08:00
xicm
6b47ef6ed2 [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti… (#5526) 
* [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOptimized

Co-authored-by: xicm <xicm@asiainfo.com>
 2022-05-09 16:35:50 +08:00
ForwardXu
4c70840275 [MINOR] Fixing close for HoodieCatalog's test (#5531) 
* [MINOR] Fixing close for HoodieCatalog's test
 2022-05-09 15:17:24 +08:00
Wangyh
33ff4752ba [HUDI-3978] Fix use of partition path field as hive partition field in flink (#5434) 
* Fix partition path fields as hive sync partition fields error
 2022-04-29 20:58:54 -07:00
Danny Chan
e1ccf2e00b [HUDI-3977] Flink hudi table with date type partition path throws HoodieNotSupportedException (#5432) 2022-04-27 13:19:55 +08:00
董可伦
b8e465fdfc [MINOR] Fix typos in log4j-surefire.properties (#5212) 2022-04-15 13:33:37 -07:00
Danny Chan
0281725c6b [MINOR] Inline the partition path logic into the builder (#5310) 2022-04-13 16:54:39 +05:30
Sagar Sumit
df87095ef0 [HUDI-3454] Fix partition name in all code paths for LogRecordScanner (#5252) 
* Depend on FSUtils#getRelativePartitionPath(basePath, logFilePath.getParent) 
to get the partition.

* If the list of log file paths in the split is empty, then fallback to usual behaviour.
 2022-04-08 09:59:36 +05:30
Danny Chan
b9fbada2f2 [minor] Follow 3178, fix the flink metadata table compaction (#5175) 2022-03-30 20:45:29 +08:00
Danny Chan
5c1b482a1b [HUDI-3741] Fix flink bucket index bulk insert generates too many small files (#5164) 2022-03-30 08:18:36 +08:00
Danny Chan
4d940bbf8a [HUDI-3716] OOM occurred when use bulk_insert cow table with flink BUCKET index (#5135) 2022-03-27 09:13:58 +08:00
Danny Chan
5e86cdd1e9 [HUDI-3701] Flink bulk_insert support bucket hash index (#5118) 2022-03-25 09:01:42 +08:00
wxp4532
26e5d2e6fc [HUDI-3559] Flink bucket index with COW table throws NoSuchElementException 
Actually method FlinkWriteHelper#deduplicateRecords does not guarantee the records sequence, but there is a
implicit constraint: all the records in one bucket should have the same bucket type(instant time here),
the BucketStreamWriteFunction breaks the rule and fails to comply with this constraint.

close apache/hudi#5018
 2022-03-21 17:34:54 +08:00
Danny Chan
799c78e688 [HUDI-3665] Support flink multiple versions (#5072) 2022-03-21 10:34:50 +08:00