lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,535 Commits 1 Branch 0 Tags
928b09ea0b2d97bc5055170c8c87fa0103615b4c
Commit Graph

1535 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
swuferhong
928b09ea0b [HUDI-1871] Fix hive conf for Flink writer hive meta sync (#2968) 2021-05-20 17:03:52 +08:00
Danny Chan
9b01d2f864 [HUDI-1915] Fix the file id for write data buffer before flushing (#2966) 2021-05-20 10:20:08 +08:00
wangxianghu
ced068e1ee [MINOR] Remove unused method in BaseSparkCommitActionExecutor (#2965) 2021-05-20 10:18:07 +08:00
zhangminglei
fe3f5c2d56 [HUDI-1913] Using streams instead of loops for input/output (#2962) 2021-05-19 09:13:38 +08:00
Sivabalan Narayanan
5d1f592395 [HUDI-1806] Honoring skipROSuffix in spark ds (#2882) 
* Honoring skipROSuffix in spark ds

* Adding tests

* fixing scala checkstype issue
 2021-05-18 16:11:39 -07:00
Danny Chan
7d2971d4e2 [HUDI-1911] Reuse the partition path and file group id for flink write data buffer (#2961) 
Reuse to reduce memory footprint.
 2021-05-18 17:47:22 +08:00
Danny Chan
46a2399a45 [HUDI-1902] Global index for flink writer (#2958) 
Supports deduplication for record keys with different partition path.
 2021-05-18 13:55:38 +08:00
Roc Marshal
fcedbfcb58 [MINOR][hudi-client] Code-cleanup,remove redundant variable declarations (#2956) 2021-05-17 13:34:42 +08:00
Danny Chan
8869b3b418 [HUDI-1902] Clean the corrupted files generated by FlinkMergeAndReplaceHandle (#2949) 
Make the intermediate files of FlinkMergeAndReplaceHandle hidden, when
committing the instant, clean these files in case there was some
corrupted files left(in normal case, the intermediate files should be cleaned
by the FlinkMergeAndReplaceHandle itself).
 2021-05-14 15:43:37 +08:00
xoln ann
12443e4187 [HUDI-1446] Support skip bootstrapIndex's init in abstract fs view init (#2520) 
Co-authored-by: zhongliang <zhongliang@kuaishou.com>
Co-authored-by: Sivabalan Narayanan <sivabala@uber.com>
 2021-05-14 00:29:26 -04:00
Danny Chan
ad77cf42ba [HUDI-1900] Always close the file handle for a flink mini-batch write (#2943) 
Close the file handle eagerly to avoid corrupted files as much as
possible.
 2021-05-14 10:25:18 +08:00
xiarixiaoyao
6f7ff7e8ca [HUDI-1722]Fix hive beeline/spark-sql query specified field on mor table occur NPE (#2722) 2021-05-12 20:52:37 +08:00
Danny Chan
b98c9ab439 [HUDI-1895] Close the file handles gracefully for flink write function to avoid corrupted files (#2938) 2021-05-12 18:44:10 +08:00
lw0090
5a8b2a4f86 [HUDI-1768] add spark datasource unit test for schema validate add column (#2776) 2021-05-11 16:49:18 -04:00
TeRS-K
be9db2c4f5 [HUDI-1055] Remove hardcoded parquet in tests (#2740) 
* Remove hardcoded parquet in tests
* Use DataFileUtils.getInstance
* Renaming DataFileUtils to BaseFileUtils

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-05-11 10:01:45 -07:00
Sivabalan Narayanan
ac72470e10 [HUDI-1851] Adding test suite long running automate scripts for docker (#2880) 2021-05-11 01:26:01 -07:00
hiscat
7a5af806cf [HUDI-1818] Validate required fields for Flink HoodieTable (#2930) 2021-05-11 11:11:19 +08:00
Danny Chan
42ec7e30d7 [HUDI-1890] FlinkCreateHandle and FlinkAppendHandle canWrite should always return true (#2933) 
The method #canWrite should always return true because they can already
write based on file size, e.g. the BucketAssigner.
 2021-05-11 09:14:51 +08:00
jsbali
aa398f77f1 [HUDI-1789] Support reading older snapshots (#2809) 
* [HUDI-1789] In HoodieParquetInoutFormat we currently default to the latest version of base files.
This PR attempts to add a new jobConf
 `hoodie.%s.consume.snapshot.time`

This new config will allow us to read older snapshots.

-  Reusing hoodie.%s.consume.commit for point in time snapshot queries as well.
-  Adding javadocs and some more tests
 2021-05-10 15:26:49 -07:00
Volodymyr Burenin
8a48d16e41 [HUDI-1707] Reduces log level for too verbose messages from info to debug level. (#2714) 
* Reduces log level for too verbose messages from info to debug level.
* Sort config output.
* Code Review : Small restructuring + rebasing to master
 - Fixing flaky multi delta streamer test
 - Using isDebugEnabled() checks
 - Some changes to shorten log message without moving to DEBUG

Co-authored-by: volodymyr.burenin <volodymyr.burenin@cloudkitchens.com>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-05-10 07:16:02 -07:00
hiscat
511ac4881d [MINOR] optimize FilePathUtils (#2931) 2021-05-10 06:47:56 -07:00
Danny Chan
c1b331bcff [HUDI-1886] Avoid to generates corrupted files for flink sink (#2929) 2021-05-10 10:43:03 +08:00
Danny Chan
bfbf993cbe [HUDI-1878] Add max memory option for flink writer task (#2920) 
Also removes the rate limiter because it has the similar functionality,
modify the create and merge handle cleans the retry files automatically.
 2021-05-08 14:27:56 +08:00
li36909
2c5a661a64 [HUDI-1759] Save one connection retry to hive metastore when hiveSyncTool run with useJdbc=false (#2759) 
* [HUDI-1759] Save one connection retry to hive metastore when hiveSyncTool run with useJdbc=false

* Fix review comment
 2021-05-07 15:30:26 -07:00
Danny Chan
528f4ca988 [HUDI-1880] Support streaming read with compaction and cleaning (#2921) 2021-05-07 20:04:35 +08:00
hiscat
0a5863939b [HUDI-1821] Remove legacy code for Flink writer (#2868) 2021-05-07 10:58:49 +08:00
Sivabalan Narayanan
0284cdecce [HUDI-1876] wiring in Hadoop Conf with AvroSchemaConverters instantiation (#2914) 2021-05-05 21:31:44 -07:00
xiarixiaoyao
1db904a12e [HUDI-1718] When query incr view of mor table which has Multi level partitions, the query failed (#2716) 2021-05-05 00:34:20 -04:00
dijie
c5220b96e9 [HUDI-1781] Fix Flink streaming reader throws ClassCastException (#2900) 2021-05-01 19:13:15 +08:00
Nick Young
ea14d687da [HUDI-1852] Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX and SCHEMA_REGISTRY_TARGET_URL_SUFFIX property (#2884) 2021-05-01 10:02:00 +08:00
Raymond Xu
3418a92de8 [HUDI-1620] Fix Metrics UT (#2894) 
Make sure shutdown Metrics between unit test cases to ensure isolation
 2021-04-30 11:20:41 -07:00
Raymond Xu
46de9e0f3f [HUDI-1810] Fix azure setting for integ tests (#2889) 2021-04-30 11:17:36 -07:00
Raymond Xu
faf3785a2d [HUDI-1811] Fix TestHoodieRealtimeRecordReader (#2873) 
Pass basePath with scheme 'file://' to HoodieRealtimeFileSplit
 2021-04-30 11:16:55 -07:00
xiarixiaoyao
929eca43fe [HUDI-1817] Fix getting incorrect partition path while using incr query by spark-sql (#2858) 2021-04-30 14:57:52 +08:00
Danny Chan
6848a683bd [HUDI-1867] Streaming read for Flink COW table (#2895) 
Supports streaming read for Copy On Write table.
 2021-04-29 20:44:45 +08:00
Danny Chan
6e9c5dd765 [HUDI-1863] Add rate limiter to Flink writer to avoid OOM for bootstrap (#2891) 2021-04-29 20:32:10 +08:00
pengzhiwei
c9bcb5e33f [HUDI-1845] Exception Throws When Sync Non-Partitioned Table To Hive With MultiPartKeysValueExtractor (#2876) 2021-04-28 19:11:46 -07:00
dijie
3ca9030256 [HUDI-1858] Fix cannot create table due to jar conflict (#2886) 
Co-authored-by: 狄杰 <shenjinxin@accesscorporate.com.cn>
 2021-04-28 14:10:04 +08:00
satishkotha
386767693d [HUDI-1833] rollback pending clustering even if there is greater commit (#2863) 
* [HUDI-1833] rollback pending clustering even if there are greater commits
 2021-04-27 14:21:42 -07:00
Roc Marshal
e4fd195d9f [MINOR] Refactor method up to parent-class (#2822) 2021-04-27 21:32:32 +08:00
satishkotha
2999586509 [HUDI-1690] use jsc union instead of rdd union (#2872) 2021-04-26 23:35:01 -07:00
hiscat
63fa2b6186 [HUDI-1836] Logging consuming instant to StreamReadOperator#processSplits (#2867) 2021-04-27 14:00:59 +08:00
Danny Chan
5be3997f70 [HUDI-1841] Tweak the min max commits to keep when setting up cleaning retain commits for Flink (#2875) 2021-04-27 10:58:06 +08:00
Roc Marshal
9bbb458e88 [MINOR] Remove redundant method-calling. (#2881) 2021-04-27 09:34:09 +08:00
Nick Young
f4e3b94971 [HUDI-1742] Improve table level config priority for HoodieMultiTableDeltaStreamer (#2744) 2021-04-26 22:05:06 +08:00
Danny Chan
d047e91d86 [HUDI-1837] Add optional instant range to log record scanner for log (#2870) 2021-04-26 16:53:18 +08:00
Sivabalan Narayanan
3e4fa170cf [HUDI-1835] Fixing kafka native config param for auto offset reset (#2864) 2021-04-25 12:16:09 -04:00
Danny Chan
1b27259b53 [HUDI-1844] Add option to flush when total buckets memory exceeds the threshold (#2877) 
Current code supports flushing as per-bucket memory usage, while the
buckets may still take too much memory for bootstrap from history data.

When the threshold hits, flush out half of the buckets with bigger
buffer size.
 2021-04-25 23:06:53 +08:00
Danny Chan
a5789c4067 [HUDI-1829] Use while loop instead of recursive call in MergeOnReadInputFormat to avoid StackOverflow (#2862) 
Recursive all is risky for StackOverflow when there are too many.
 2021-04-23 09:59:36 +08:00
Chanh Le
a1e636dc6b [HUDI-1551] Add support for BigDecimal and Integer when partitioning based on time. (#2851) 
Co-authored-by: trungchanh.le <trungchanh.le@bybit.com>
 2021-04-22 21:56:20 +08:00