lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,494 Commits 1 Branch 0 Tags
63fa2b618626b2b8e9530ec105e0d4995eebb643
Commit Graph

1494 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
hiscat
63fa2b6186 [HUDI-1836] Logging consuming instant to StreamReadOperator#processSplits (#2867) 2021-04-27 14:00:59 +08:00
Danny Chan
5be3997f70 [HUDI-1841] Tweak the min max commits to keep when setting up cleaning retain commits for Flink (#2875) 2021-04-27 10:58:06 +08:00
Roc Marshal
9bbb458e88 [MINOR] Remove redundant method-calling. (#2881) 2021-04-27 09:34:09 +08:00
Nick Young
f4e3b94971 [HUDI-1742] Improve table level config priority for HoodieMultiTableDeltaStreamer (#2744) 2021-04-26 22:05:06 +08:00
Danny Chan
d047e91d86 [HUDI-1837] Add optional instant range to log record scanner for log (#2870) 2021-04-26 16:53:18 +08:00
Sivabalan Narayanan
3e4fa170cf [HUDI-1835] Fixing kafka native config param for auto offset reset (#2864) 2021-04-25 12:16:09 -04:00
Danny Chan
1b27259b53 [HUDI-1844] Add option to flush when total buckets memory exceeds the threshold (#2877) 
Current code supports flushing as per-bucket memory usage, while the
buckets may still take too much memory for bootstrap from history data.

When the threshold hits, flush out half of the buckets with bigger
buffer size.
 2021-04-25 23:06:53 +08:00
Danny Chan
a5789c4067 [HUDI-1829] Use while loop instead of recursive call in MergeOnReadInputFormat to avoid StackOverflow (#2862) 
Recursive all is risky for StackOverflow when there are too many.
 2021-04-23 09:59:36 +08:00
Chanh Le
a1e636dc6b [HUDI-1551] Add support for BigDecimal and Integer when partitioning based on time. (#2851) 
Co-authored-by: trungchanh.le <trungchanh.le@bybit.com>
 2021-04-22 21:56:20 +08:00
jsbali
4a3431866d [HUDI-1746] Added support for replace commits in commit showpartitions, commit show_write_stats, commit showfiles (#2678) 
* Added support for replace commits in commit showpartitions, commit show_write_stats, commit showfiles

* Adding CR changes

* [HUDI-1746] Code review changes
 2021-04-21 10:31:35 -07:00
jsbali
b31c520c66 [HUDI-1714] Added tests to TestHoodieTimelineArchiveLog for the archival of compl… (#2677) 
* Added tests to TestHoodieTimelineArchiveLog for the archival of completed clean and rollback actions.

* Adding code review changes

* [HUDI-1714] Minor Fixes
 2021-04-21 10:27:43 -07:00
vinoyang
c24d90d25a [MINOR] Expose the detailed exception object (#2861) 2021-04-21 22:41:42 +08:00
hiscat
cc81ddde01 [HUDI-1812] Add explicit index state TTL option for Flink writer (#2853) 2021-04-21 20:13:30 +08:00
Danny Chan
ac3589f006 [HUDI-1814] Non partitioned table for Flink writer (#2859) 2021-04-21 20:07:27 +08:00
pengzhiwei
aacb8be521 [HUDI-1415] Read Hoodie Table As Spark DataSource Table (#2283) 2021-04-20 14:21:38 -07:00
Jintao Guan
3253079507 [HUDI-1764] Add Hudi-CLI support for clustering (#2773) 
* tmp base

* update

* update unit test

* update

* update

* update CLI parameters

* linting

* update doSchedule in HoodieClusteringJob

* update

* update diff according to comments
 2021-04-20 09:46:42 -07:00
Danny Chan
d6d52c6063 [HUDI-1809] Flink merge on read input split uses wrong base file path for default merge type (#2846) 2021-04-20 21:27:09 +08:00
Sebastian Bernauer
9a288ccbeb [MINOR] Added metric reporter Prometheus to HoodieBackedTableMetadataWriter (#2842) 2021-04-19 16:04:59 -07:00
li36909
6b4b878d08 [HUDI-1744] rollback fails on mor table when the partition path hasn't any files (#2749) 
Co-authored-by: lrz <lrz@lrzdeMacBook-Pro.local>
 2021-04-19 15:44:11 -07:00
Thinking Chen
d21753d903 [HUDI-1802] Timeline Server Bundle need to include com.esotericsoftware package (#2835) 2021-04-19 09:27:58 -07:00
Aditya Tiwari
ec2334ceac [HUDI-1716]: Resolving default values for schema from dataframe (#2765) 
- Adding default values and setting null as first entry in UNION data types in avro schema. 

Co-authored-by: Aditya Tiwari <aditya.tiwari@flipkart.com>
 2021-04-19 10:05:20 -04:00
Danny Chan
dab5114f16 [HUDI-1804] Continue to write when Flink write task restart because of container killing (#2843) 
The `FlinkMergeHande` creates a marker file under the metadata path
each time it initializes, when a write task restarts from killing, it
tries to create the existing file and reports error.

To solve this problem, skip the creation and use the original data file
as base file to merge.
 2021-04-19 19:43:41 +08:00
Roc Marshal
f7b6b68063 [MINOR][hudi-sync] Fix typos (#2844) 2021-04-19 16:27:13 +08:00
satishkotha
4e050cc2ba [MINOR] Add jackson module to presto bundle (#2816) 2021-04-17 13:26:07 -07:00
Xu Guang Lv
1d53d6e6c2 [HUDI-1803] Support BAIDU AFS storage format in hudi (#2836) 2021-04-16 16:43:14 +08:00
hj2016
62b8a341dd [HUDI-1792] flink-client query error when processing files larger than 128mb (#2814) 
Co-authored-by: huangjing <huangjing@clinbrain.com>
 2021-04-16 13:59:19 +08:00
Danny Chan
b6d949b48a [HUDI-1801] FlinkMergeHandle rolling over may miss to rename the latest file handle (#2831) 
The FlinkMergeHandle may rename the N-1 th file handle instead of the
latest one, thus to cause data duplication.
 2021-04-16 11:40:53 +08:00
MINCWANG
191470d1fc [HUDI-1797] Remove the com.google.guave jar from hudi-flink-bundle to avoid conflicts. (#2828) 
Co-authored-by: wangminchao <wangminchao@asinking.com>
 2021-04-15 15:16:33 +08:00
hiscat
6d1aec604f [HUDI-1798] Flink streaming reader should always monitor the delta commits files (#2825) 
The streaming reader should only monitor the delta log files, if there are parquet commits but we recognize as logs, the reader would report FileNotFound exception.
 2021-04-15 13:50:17 +08:00
Roc Marshal
62bb9e10d9 [Hotfix][utilities] Optimized codes (#2821) 2021-04-15 09:40:14 +08:00
Sivabalan Narayanan
8d29863c86 [HUDI-1615] Fixing usage of NULL schema for delete operation in HoodieSparkSqlWriter (#2777) 2021-04-14 15:35:39 +08:00
Danny Chan
ab4a7b0b4a [HUDI-1788] Insert overwrite (table) for Flink writer (#2808) 
Supports `INSERT OVERWRITE` and `INSERT OVERWRITE TABLE` for Flink
writer.
 2021-04-14 10:23:37 +08:00
xiarixiaoyao
65844a8d29 [HUDI-1720] Fix RealtimeCompactedRecordReader StackOverflowError (#2721) 2021-04-13 18:23:26 +08:00
hiscat
e16d31dce2 [HUDI-1787] Remove the rocksdb jar from hudi-flink-bundle (#2807) 
Remove the RocksDB jar from hudi-flink-bundle to avoid conflicts.
 2021-04-13 10:31:16 +08:00
Danny Chan
1ff99ca7d7 [HUDI-1786] Add option for merge max memory (#2805) 2021-04-12 17:03:58 +08:00
wangxianghu
040756d8c0 [HUDI-1785] Move OperationConverter to hudi-client-common for code reuse (#2798) 2021-04-12 16:22:33 +08:00
hj2016
1da16dfd2e [HUDI-1784] Added print detailed stack log when hbase connection error (#2799) 2021-04-12 13:46:06 +08:00
wangxianghu
f3777f44fe [MINOR] Remove unused imports and some other checkstyle issues (#2800) 2021-04-11 21:42:34 +08:00
Roc Marshal
b554835053 [MINOR] fix typo. (#2804) 2021-04-11 10:31:07 +08:00
xiarixiaoyao
8d4a7fe33e [HUDI-1783] Support Huawei Cloud Object Storage (#2796) 2021-04-10 13:02:11 +08:00
Danny Chan
6786581c48 [HUDI-1775] Add option for compaction parallelism (#2785) 2021-04-09 13:46:19 +08:00
Vinoth Govindarajan
08e82c469c [HUDI-1762] Added HiveStylePartitionExtractor to support Hive style partitions (#2769) 2021-04-09 01:00:11 -04:00
Gary Li
cf3d2e21eb [MINOR] Update doap with 0.8.0 release (#2772) 2021-04-08 11:06:13 -04:00
hiscat
5b3608f149 [HUDI-1778] Add setter to CompactionPlanEvent and CompactionCommitEvent to have better SE/DE performance for Flink (#2789) 2021-04-08 19:40:37 +08:00
hongdd
ecdbd2517f [HUDI-699] Fix CompactionCommand and add unit test for CompactionCommand (#2325) 2021-04-08 15:35:33 +08:00
Simon
18459d4045 [MINOR] Some unit test code optimize (#2782) 
* Optimized code

* Optimized code
 2021-04-08 13:35:03 +08:00
hiscat
3a926aacf6 [HUDI-1773] HoodieFileGroup code optimize (#2781) 2021-04-07 18:16:03 +08:00
hiscat
f4f9dd9d83 [HUDI-1772] HoodieFileGroupId compareTo logical error(fileId self compare) (#2780) 2021-04-07 18:10:38 +08:00
li36909
dadd081d45 [HUDI-1751] DeltaStreamer print many unnecessary warn log (#2754) 2021-04-07 00:47:03 -07:00
hiscat
d035fcbb3c [HUDI-1767] Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink (#2779) 2021-04-07 14:13:31 +08:00