lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,354 Commits 1 Branch 0 Tags
5d2491d10c70e4e5fc9b7aeb62cc64bcaaf6043f
Commit Graph

1354 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Danny Chan
5d2491d10c [HUDI-1598] Write as minor batches during one checkpoint interval for the new writer (#2553) 2021-02-17 15:24:50 +08:00
vinoyang
302bd29dab [MINOR] Add clustering to feature list (#2568) 2021-02-13 07:39:14 -08:00
Raymond Xu
527175ab0b [MINOR] Default to empty list for unset datadog tags property (#2574) 2021-02-13 15:52:03 +08:00
Sivabalan Narayanan
d5f202821b Adding fixes to test suite framework. Adding clustering node and validate async operations node. (#2400) 2021-02-12 09:29:21 -08:00
lamber-ken
ff0e3f5669 [HUDI-1612] Fix write test flakiness in StreamWriteITCase (#2567) 
* [HUDI-1612] Fix write test flakiness in StreamWriteITCase
 2021-02-11 23:37:19 +08:00
teeyog
26da4f5462 [HUDI-1526] Translate the api partitionBy in spark datasource to hoodie.datasource.write.partitionpath.field (#2431) 2021-02-10 12:07:54 -05:00
vinoyang
a2f85d90de [MINOR] Fix the wrong comment for HoodieJavaWriteClientExample (#2559) 2021-02-09 10:33:34 -08:00
Gary Li
7a98b1c878 [HUDI-1603] fix DefaultHoodieRecordPayload serialization failure (#2556) 2021-02-09 10:53:45 -05:00
Sun Ke
c30481f4b0 [HUDI-1545] Add test cases for INSERT_OVERWRITE Operation (#2483) 
Co-authored-by: sunke.03 <sunke.03@bytedance.com>
 2021-02-07 21:47:01 -08:00
Danny Chan
4c5b6923cc [HUDI-1557] Make Flink write pipeline write task scalable (#2506) 
This is the #step 2 of RFC-24:
https://cwiki.apache.org/confluence/display/HUDI/RFC+-+24%3A+Hoodie+Flink+Writer+Proposal

This PR introduce a BucketAssigner that assigns bucket ID (partition
path & fileID) for each stream record.

There is no need to look up index and partition the records anymore in
the following pipeline for these records,
we actually decide the write target location before the write and each
record computes its location when the BucketAssigner receives it, thus,
the indexing is with streaming style.

Computing locations for a batch of records all at a time is resource
consuming so a pressure to the engine,
we should avoid that in streaming system.
 2021-02-06 22:03:52 +08:00
ZhangChaoMing
291f92069e [MINOR] Fix wrong logic for checking state condition (#2524) 2021-02-06 16:40:31 +08:00
n3nash
b2c47a24be [HUDI-1589] Fix Rollback Metadata AVRO backwards incompatiblity (#2543) 2021-02-05 16:03:34 -08:00
Sivabalan Narayanan
b5d4a046bb [HUDI-1571] Adding commit_show_records_info to display record sizes for commit (#2514) 2021-02-05 07:53:24 -05:00
hiscat
b51b3a39a8 [HUDI-1420] HoodieTableMetaClient.getMarkerFolderPath works incorrectly on windows client with hdfs server for wrong file seperator (#2526) 
* Fix HUDI-1420

FIX https://issues.apache.org/jira/browse/HUDI-1420

* fix(hudi-common): fix HUDI-1420 HoodieTableMetaClient.getMarkerFolderPath works incorrectly on windows client with hdfs server for wrong file seperator

Co-authored-by: 谢波 <xiebo1@yonghui.cn>
 2021-02-05 16:24:35 +08:00
Sivabalan Narayanan
4a5683d54a [MINOR] Fixing the default value for source ordering field for payload config (#2516) 2021-02-04 08:43:03 -05:00
wangxianghu
647e9faf25 [HUDI-1547] CI intermittent failure: TestJsonStringToHoodieRecordMapF… (#2521) 2021-02-04 11:20:01 +08:00
Volodymyr Burenin
17802569fd [HUDI-1538] Try to init class trying different signatures instead of checking its name (#2476) 
* [HUDI-1538] Try to init class trying different signatures instead of checking its name.

* Removed unused imports

Co-authored-by: volodymyr.burenin <volodymyr.burenin@cloudkitchens.com>
 2021-02-03 12:29:08 -08:00
Sivabalan Narayanan
eb91e5ba70 [HUDI-1523] Call mkdir(partition) only if not exists (#2501) 2021-02-03 09:02:37 -05:00
wangxianghu
d74d8e2084 [HUDI-1335] Introduce FlinkHoodieSimpleIndex to hudi-flink-client (#2271) 2021-02-03 08:59:49 +08:00
vinoyang
50ff9ab2d2 [MINOR] Rename FileSystemViewHandler to RequestHandler and corrected the class comment (#2458) 2021-02-02 09:15:53 -08:00
jackiehff
ec950b4cfe [MINOR] Fix method comment typo (#2518) 
Co-authored-by: 黄飞飞 <huangfeifei@mininglamp.com>
 2021-02-02 19:23:29 +08:00
pengzhiwei
0d8a4d0a56 [HUDI-1550] Honor ordering field for MOR Spark datasource reader (#2497) 2021-02-01 21:04:27 +08:00
steven zhang
f159c0c49a [HUDI-1519] Improve minKey/maxKey computation in HoodieHFileWriter (#2427) 
Co-authored-by: zhang wen <steven@stevendeMac-mini.local>
 2021-02-01 07:51:57 -05:00
jiangjiguang
5d053b495b [MINOR] Quickstart.generateUpdates method add check (#2505) 2021-01-30 10:28:00 +08:00
satishkotha
9cb6cb8189 [HUDI-1266] Add unit test for validating replacecommit rollback (#2418) 2021-01-29 10:28:08 -08:00
satishkotha
2d2d5c83b1 [HUDI-1555] Remove isEmpty to improve clustering execution performance (#2502) 2021-01-29 10:27:09 -08:00
wangxianghu
23f2ef3efb [HUDI-623] Remove UpgradePayloadFromUberToApache (#2455) 2021-01-28 17:48:50 -08:00
Danny Chan
bc0325f6ea [HUDI-1522] Add a new pipeline for Flink writer (#2430) 
* [HUDI-1522] Add a new pipeline for Flink writer
 2021-01-28 08:53:13 +08:00
wangxianghu
7b2e658ac0 [MINOR] Add Jira URL and Mailing List (#2404) 2021-01-27 19:48:42 -05:00
SteNicholas
2ee1c3fb0c [HUDI-1234] Insert new records to data files without merging for "Insert" operation. (#2111) 
* Added HoodieConcatHandle to skip merging for "insert" operation when the corresponding config is set

Co-authored-by: Sivabalan Narayanan <sivabala@uber.com>
 2021-01-27 13:09:51 -05:00
luokey
a54550d94f [MINOR]Fix NPE when using HoodieFlinkStreamer with multi parallelism (#2492) 2021-01-27 21:00:20 +08:00
vinoth chandar
c8ee40f8ae [MINOR] Update doap with 0.7.0 release (#2491) 2021-01-26 09:28:22 -08:00
Shen Hong
c4afd179c1 [HUDI-1476] Introduce unit test infra for java client (#2478) 2021-01-24 11:17:19 -08:00
vinoth chandar
81836f0309 Removing spring repos from pom (#2481) 
- These are being deprecated
- Causes build issues when .m2 does not have this cached already
 2021-01-24 07:42:52 -08:00
Raymond Xu
84df26323d [MINOR] Use skipTests flag for skip.hudi-spark2.unit.tests property (#2477) 2021-01-24 21:36:41 +08:00
wangxianghu
e302c6bc12 [HUDI-1453] Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi (#2474) 2021-01-23 10:27:40 +08:00
wangxianghu
d3ea0f957e [HOTFIX] Revert upgrade flink verison to 1.12.0 (#2473) 2021-01-22 10:55:46 -08:00
cooper
048633da1a [MINOR] Improve code readability,remove the continue keyword (#2459) 2021-01-22 13:47:14 +08:00
wangxianghu
748dcc9aae [MINOR] Remove InstantGeneratorOperator parallelism limit in HoodieFlinkStreamer and update docs (#2471) 2021-01-22 13:46:25 +08:00
Xiang Yang
641abe8ab7 [HUDI-1332] Introduce FlinkHoodieBloomIndex to hudi-flink-client (#2375) 
* [HUDI] Add bloom index for hudi-flink-client

Co-authored-by: yangxiang <yangxiang@oppo.com>
 2021-01-22 10:36:28 +08:00
luokey
b64d22e047 [HUDI-1511] InstantGenerateOperator support multiple parallelism (#2434) 2021-01-22 09:17:50 +08:00
wenningd
976420c49a [HUDI-1512] Fix spark 2 unit tests failure with Spark 3 (#2412) 
* [HUDI-1512] Fix spark 2 unit tests failure with Spark 3

* resolve comments

Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2021-01-21 07:04:28 -08:00
vinoth chandar
81ccb0c71a [MINOR] Make a separate travis CI job for hudi-utilities (#2469) 2021-01-20 21:46:05 -08:00
vinoth chandar
5e30fc1b2b [MINOR] Disabling problematic tests temporarily to stabilize CI (#2468) 2021-01-20 14:24:34 -08:00
Vinoth Chandar
3719e7b388 Moving to 0.8.0-SNAPSHOT on master branch. 2021-01-20 11:31:22 -08:00
liujinhui
244f6def9c [MINOR] Fix dataSource cannot use hoodie.datasource.hive_sync.auto_create_database (#2444) 
fix dataSource cannot use hoodie.datasource.hive_sync.auto_create_database
 2021-01-20 22:58:18 +08:00
teeyog
c931dc5406 [MINOR] Remove redundant judgments (#2466) 2021-01-20 20:41:09 +08:00
vinoth chandar
5ca0625b27 [HUDI 1308] Harden RFC-15 Implementation based on production testing (#2441) 
Addresses leaks, perf degradation observed during testing. These were regressions from the original rfc-15 PoC implementation.

* Pass a single instance of HoodieTableMetadata everywhere
* Fix tests and add config for enabling metrics
 - Removed special casing of assumeDatePartitioning inside FSUtils#getAllPartitionPaths()
 - Consequently, IOException is never thrown and many files had to be adjusted
- More diligent handling of open file handles in metadata table
 - Added config for controlling reuse of connections
 - Added config for turning off fallback to listing, so we can see tests fail
 - Changed all ipf listing code to cache/amortize the open/close for better performance
 - Timelineserver also reuses connections, for better performance
 - Without timelineserver, when metadata table is opened from executors, reuse is not allowed
 - HoodieMetadataConfig passed into HoodieTableMetadata#create as argument.
 -  Fix TestHoodieBackedTableMetadata#testSync
 2021-01-19 21:20:28 -08:00
Sivabalan Narayanan
e23967b9e9 [HUDI-1540] Fixing commons codec shading in spark bundle (#2460) 2021-01-20 00:00:13 -05:00
Sivabalan Narayanan
91b9cb53d3 [MINOR] Fixing setting defaults for index config (#2457) 2021-01-19 18:16:25 -05:00