lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,391 Commits 1 Branch 0 Tags
e00a9042e9a4e97f6f8ccaed4ac51a9aa8c2da76
Commit Graph

731 Commits

Author SHA1 Message Date
rmpifer
0709c62a6b [HUDI-1800] Exclude file slices in pending compaction when performing small file sizing (#2902) 
Co-authored-by: Ryan Pifer <ryanpife@amazon.com>
 2021-05-29 08:06:01 -04:00
Danny Chan
7fed7352bd [HUDI-1865] Make embedded time line service singleton (#2899) 2021-05-27 13:38:33 +08:00
wangxianghu
e7020748b5 [HUDI-1920] Set archived as the default value of HOODIE_ARCHIVELOG_FOLDER_PROP_NAME (#2978) 2021-05-25 16:29:55 +08:00
Susu Dong
685f77b5dd [HUDI-1740] Fix insert-overwrite API archival (#2784) 
- fix problem of archiving replace commits
- Fix problem when getting empty replacecommit.requested
- Improved the logic of handling empty and non-empty requested/inflight commit files. Added unit tests to cover both empty and non-empty inflight files cases and cleaned up some unused test util methods

Co-authored-by: yorkzero831 <yorkzero8312@gmail.com>
Co-authored-by: zheren.yu <zheren.yu@paypay-corp.co.jp>
 2021-05-21 13:52:13 -07:00
Y Ethan Guo
a96034d38d [HUDI-1888] Fix NPE when the nested partition path field has null value (#2957) 2021-05-21 08:28:11 -04:00
Danny Chan
7c213f9f26 [HUDI-1917] Remove the metadata sync logic in HoodieFlinkWriteClient#preWrite because it is not thread safe (#2971) 2021-05-21 11:29:54 +08:00
Danny Chan
9b01d2f864 [HUDI-1915] Fix the file id for write data buffer before flushing (#2966) 2021-05-20 10:20:08 +08:00
wangxianghu
ced068e1ee [MINOR] Remove unused method in BaseSparkCommitActionExecutor (#2965) 2021-05-20 10:18:07 +08:00
Roc Marshal
fcedbfcb58 [MINOR][hudi-client] Code-cleanup,remove redundant variable declarations (#2956) 2021-05-17 13:34:42 +08:00
Danny Chan
8869b3b418 [HUDI-1902] Clean the corrupted files generated by FlinkMergeAndReplaceHandle (#2949) 
Make the intermediate files of FlinkMergeAndReplaceHandle hidden, when
committing the instant, clean these files in case there was some
corrupted files left(in normal case, the intermediate files should be cleaned
by the FlinkMergeAndReplaceHandle itself).
 2021-05-14 15:43:37 +08:00
xoln ann
12443e4187 [HUDI-1446] Support skip bootstrapIndex's init in abstract fs view init (#2520) 
Co-authored-by: zhongliang <zhongliang@kuaishou.com>
Co-authored-by: Sivabalan Narayanan <sivabala@uber.com>
 2021-05-14 00:29:26 -04:00
Danny Chan
ad77cf42ba [HUDI-1900] Always close the file handle for a flink mini-batch write (#2943) 
Close the file handle eagerly to avoid corrupted files as much as
possible.
 2021-05-14 10:25:18 +08:00
Danny Chan
b98c9ab439 [HUDI-1895] Close the file handles gracefully for flink write function to avoid corrupted files (#2938) 2021-05-12 18:44:10 +08:00
lw0090
5a8b2a4f86 [HUDI-1768] add spark datasource unit test for schema validate add column (#2776) 2021-05-11 16:49:18 -04:00
TeRS-K
be9db2c4f5 [HUDI-1055] Remove hardcoded parquet in tests (#2740) 
* Remove hardcoded parquet in tests
* Use DataFileUtils.getInstance
* Renaming DataFileUtils to BaseFileUtils

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-05-11 10:01:45 -07:00
Danny Chan
42ec7e30d7 [HUDI-1890] FlinkCreateHandle and FlinkAppendHandle canWrite should always return true (#2933) 
The method #canWrite should always return true because they can already
write based on file size, e.g. the BucketAssigner.
 2021-05-11 09:14:51 +08:00
Danny Chan
c1b331bcff [HUDI-1886] Avoid to generates corrupted files for flink sink (#2929) 2021-05-10 10:43:03 +08:00
Danny Chan
bfbf993cbe [HUDI-1878] Add max memory option for flink writer task (#2920) 
Also removes the rate limiter because it has the similar functionality,
modify the create and merge handle cleans the retry files automatically.
 2021-05-08 14:27:56 +08:00
Danny Chan
528f4ca988 [HUDI-1880] Support streaming read with compaction and cleaning (#2921) 2021-05-07 20:04:35 +08:00
Sivabalan Narayanan
0284cdecce [HUDI-1876] wiring in Hadoop Conf with AvroSchemaConverters instantiation (#2914) 2021-05-05 21:31:44 -07:00
Raymond Xu
3418a92de8 [HUDI-1620] Fix Metrics UT (#2894) 
Make sure shutdown Metrics between unit test cases to ensure isolation
 2021-04-30 11:20:41 -07:00
satishkotha
386767693d [HUDI-1833] rollback pending clustering even if there is greater commit (#2863) 
* [HUDI-1833] rollback pending clustering even if there are greater commits
 2021-04-27 14:21:42 -07:00
satishkotha
2999586509 [HUDI-1690] use jsc union instead of rdd union (#2872) 2021-04-26 23:35:01 -07:00
Roc Marshal
9bbb458e88 [MINOR] Remove redundant method-calling. (#2881) 2021-04-27 09:34:09 +08:00
Danny Chan
d047e91d86 [HUDI-1837] Add optional instant range to log record scanner for log (#2870) 2021-04-26 16:53:18 +08:00
Chanh Le
a1e636dc6b [HUDI-1551] Add support for BigDecimal and Integer when partitioning based on time. (#2851) 
Co-authored-by: trungchanh.le <trungchanh.le@bybit.com>
 2021-04-22 21:56:20 +08:00
jsbali
b31c520c66 [HUDI-1714] Added tests to TestHoodieTimelineArchiveLog for the archival of compl… (#2677) 
* Added tests to TestHoodieTimelineArchiveLog for the archival of completed clean and rollback actions.

* Adding code review changes

* [HUDI-1714] Minor Fixes
 2021-04-21 10:27:43 -07:00
Sebastian Bernauer
9a288ccbeb [MINOR] Added metric reporter Prometheus to HoodieBackedTableMetadataWriter (#2842) 2021-04-19 16:04:59 -07:00
li36909
6b4b878d08 [HUDI-1744] rollback fails on mor table when the partition path hasn't any files (#2749) 
Co-authored-by: lrz <lrz@lrzdeMacBook-Pro.local>
 2021-04-19 15:44:11 -07:00
Aditya Tiwari
ec2334ceac [HUDI-1716]: Resolving default values for schema from dataframe (#2765) 
- Adding default values and setting null as first entry in UNION data types in avro schema. 

Co-authored-by: Aditya Tiwari <aditya.tiwari@flipkart.com>
 2021-04-19 10:05:20 -04:00
Danny Chan
dab5114f16 [HUDI-1804] Continue to write when Flink write task restart because of container killing (#2843) 
The `FlinkMergeHande` creates a marker file under the metadata path
each time it initializes, when a write task restarts from killing, it
tries to create the existing file and reports error.

To solve this problem, skip the creation and use the original data file
as base file to merge.
 2021-04-19 19:43:41 +08:00
Danny Chan
b6d949b48a [HUDI-1801] FlinkMergeHandle rolling over may miss to rename the latest file handle (#2831) 
The FlinkMergeHandle may rename the N-1 th file handle instead of the
latest one, thus to cause data duplication.
 2021-04-16 11:40:53 +08:00
Danny Chan
ab4a7b0b4a [HUDI-1788] Insert overwrite (table) for Flink writer (#2808) 
Supports `INSERT OVERWRITE` and `INSERT OVERWRITE TABLE` for Flink
writer.
 2021-04-14 10:23:37 +08:00
wangxianghu
040756d8c0 [HUDI-1785] Move OperationConverter to hudi-client-common for code reuse (#2798) 2021-04-12 16:22:33 +08:00
hj2016
1da16dfd2e [HUDI-1784] Added print detailed stack log when hbase connection error (#2799) 2021-04-12 13:46:06 +08:00
hongdd
ecdbd2517f [HUDI-699] Fix CompactionCommand and add unit test for CompactionCommand (#2325) 2021-04-08 15:35:33 +08:00
Simon
18459d4045 [MINOR] Some unit test code optimize (#2782) 
* Optimized code

* Optimized code
 2021-04-08 13:35:03 +08:00
Danny Chan
9c369c607d [HUDI-1757] Assigns the buckets by record key for Flink writer (#2757) 
Currently we assign the buckets by record partition path which could
cause hotspot if the partition field is datetime type. Changes to assign
buckets by grouping the record whth their key first, the assignment is
valid if only there is no conflict(two task write to the same bucket).

This patch also changes the coordinator execution to be asynchronous.
 2021-04-06 19:06:41 +08:00
Roc Marshal
94a5e72f16 [HUDI-1737][hudi-client] Code Cleanup: Extract common method in HoodieCreateHandle & FlinkCreateHandle (#2745) 2021-04-02 11:39:05 +08:00
pengzhiwei
684622c7c9 [HUDI-1591] Implement Spark's FileIndex for Hudi to support queries via Hudi DataSource using non-globbed table path and partition pruning (#2651) 2021-04-01 11:12:28 -07:00
Danny Chan
9804662bc8 [HUDI-1738] Emit deletes for flink MOR table streaming read (#2742) 
Current we did a soft delete for DELETE row data when writes into hoodie
table. For streaming read of MOR table, the Flink reader detects the
delete records and still emit them if the record key semantics are still
kept.

This is useful and actually a must for streaming ETL pipeline
incremental computation.
 2021-04-01 15:25:31 +08:00
vinoyang
fe16d0de7c [MINOR] Delete useless UpsertPartitioner for flink integration (#2746) 2021-03-31 16:36:42 +08:00
Sebastian Bernauer
aa0da72c59 Preparation for Avro update (#2650) 2021-03-30 21:50:17 -07:00
leo-Iamok
8bc65b9318 [HUDI-1731] Rename UpsertPartitioner in hudi-java-client (#2734) 
Co-authored-by: lei.zhu <lei.zhu@envisioncn.com>
 2021-03-31 11:06:04 +08:00
Gary Li
452f5e2d66 [HOTFIX] close spark session in functional test suite and disable spark3 test for spark2 (#2727) 2021-03-29 06:04:48 -07:00
Danny Chan
d415d45416 [HUDI-1729] Asynchronous Hive sync and commits cleaning for Flink writer (#2732) 2021-03-29 10:47:29 +08:00
Shen Hong
ecbd389a3f [HUDI-1478] Introduce HoodieBloomIndex to hudi-java-client (#2608) 2021-03-28 20:28:40 +08:00
n3nash
bec70413c0 [HUDI-1728] Fix MethodNotFound for HiveMetastore Locks (#2731) 2021-03-27 10:07:10 -07:00
garyli1019
6e803e08b1 Moving to 0.9.0-SNAPSHOT on master branch. 2021-03-24 21:37:14 +08:00
n3nash
d7b18783bd [HUDI-1709] Improving config names and adding hive metastore uri config (#2699) 2021-03-22 01:22:06 -07:00