lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,719 Commits 1 Branch 0 Tags
632bfd1a65f55deff60bd56e514738b9c8730140
Commit Graph

1719 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Danny Chan
632bfd1a65 Merge pull request #3268 from yuzhaojing/HUDI-2171 
[HUDI-2171] Add parallelism conf for bootstrap operator
 2021-07-14 17:01:30 +08:00
Danny Chan
ac75bda929 [HUDI-1969] Support reading logs for MOR Hive rt table (#3033) 2021-07-13 23:43:30 -07:00
pengzhiwei
f0a2f378ea Merge pull request #3120 from pengzhiwei2018/dev_metasync 
[HUDI-2045] Support Read Hoodie As DataSource Table For Flink And DeltaStreamer
 2021-07-13 22:37:20 +08:00
Vinay Patil
7395a56dfb [HUDI-2168] Fix for AccessControlException for anonymous user (#3264) 2021-07-13 08:56:51 -04:00
喻兆靖
aff1a1ed29 [HUDI-2171] Add parallelism conf for bootstrap operator 2021-07-13 17:55:12 +08:00
Sagar Sumit
b0089b894a [MINOR] Fix EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION config (#3250) 2021-07-13 00:24:40 -04:00
zhangyue19921010
c8a2033c27 [HUDI-2144]Bug-Fix:Offline clustering(HoodieClusteringJob) will cause insert action losing data (#3240) 
* fixed

* add testUpsertPartitionerWithSmallFileHandlingAndClusteringPlan ut

* fix CheckStyle

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-07-12 18:14:17 -07:00
pengzhiwei
ca440ccf88 [HUDI-2107] Support Read Log Only MOR Table For Spark (#3193) 2021-07-12 17:31:23 +08:00
pengzhiwei
ffa934182a [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And DeltaStreamer 2021-07-12 13:03:14 +08:00
Sagar Sumit
5804ad8e32 [HUDI-1483] Support async clustering for deltastreamer and Spark streaming (#3142) 
- Integrate async clustering service with HoodieDeltaStreamer and HoodieStreamingSink
- Added methods in HoodieAsyncService to reuse code
 2021-07-11 14:43:38 -04:00
swuferhong
9b01d2a045 [HUDI-2142] Support setting bucket assign parallelism for flink write task (#3239) 2021-07-10 15:43:36 +08:00
Danny Chan
942a024e74 [HUDI-2143] Tweak the default compaction target IO to 500GB when flink async compaction is off (#3238) 2021-07-10 15:40:30 +08:00
yuzhaojing
783c9cb369 [HUDI-2087] Support Append only in Flink stream (#3252) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-07-10 14:49:35 +08:00
vinoyang
7c6eebf98c [MINOR] Fix some wrong assert reasons (#3248) 2021-07-10 14:35:40 +08:00
wangxianghu
3b2a4f2b6b [HUDI-2147] Remove unused class AvroConvertor in hudi-flink (#3243) 2021-07-10 10:16:33 +08:00
vinoth chandar
b4562e86e4 Revert "[HUDI-2087] Support Append only in Flink stream (#3174)" (#3251) 
This reverts commit 371526789d.
 2021-07-09 11:20:09 -07:00
yuzhaojing
371526789d [HUDI-2087] Support Append only in Flink stream (#3174) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-07-09 16:06:32 +08:00
swuferhong
047d956e01 [HUDI-2136] Fix conflict when flink-sql-connector-hive and hudi-flink-bundle are both in flink lib (#3227) 2021-07-09 10:10:21 +08:00
vinoth chandar
c50c24908a [MINOR] Fix build broken from #3186 (#3245) 2021-07-08 14:23:52 -07:00
xiarixiaoyao
de07e61382 [HUDI-2099]hive lock which state is WATING should be released, otherwise this hive lock will be locked forever (#3186) 2021-07-08 10:30:48 -04:00
Sivabalan Narayanan
8c0dbaa9b3 [HUDI-2009] Fixing extra commit metadata in row writer path (#3075) 2021-07-08 03:07:27 -04:00
Yungthuis
1d3cd06572 [HUDI-2134]Add generics to avoif forced conversion in BaseSparkCommitActionExecutor#partition (#3232) 2021-07-08 13:31:38 +08:00
Sivabalan Narayanan
16e90d30ea [HUDI-1105] Adding dedup support for Bulk Insert w/ Rows (#2206) 2021-07-07 17:38:26 -04:00
Sebastian Bernauer
8f7ad8b178 [HUDI-2069] Refactored String constants (#3172) 2021-07-07 14:22:00 -04:00
Sivabalan Narayanan
ea9e5d0e8b [HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows (#3149) 2021-07-07 11:15:25 -04:00
Shawy Geng
55ecbc662e [HUDI-2115] FileSlices in the filegroup is not descending by timestamp (#3206) 2021-07-07 22:24:36 +08:00
Prashant Wason
990820476a [HUDI-2140] Fixed the unit test TestHoodieBackedMetadata.testOnlyValidPartitionsAdded. (#3234) 2021-07-06 23:50:27 -07:00
Prashant Wason
221ddd9bf3 [HUDI-2016] Fixed bootstrap of Metadata Table when some actions are in progress. (#3083) 
Metadata Table cannot be bootstrapped when any action is in progress. This is detected by the presence of inflight or requested instants. The bootstrapping is initiated in preWrite and postWrite of each commit. So bootstrapping will be retried again until it succeeds.
Also added metrics for when the bootstrapping fails or a table is re-bootstrapped. This will help detect tables which are not getting bootstrapped.
 2021-07-06 08:08:46 -07:00
Shawy Geng
6e24434682 [HUDI-2113] Fix integration testing failure caused by sql results out of order (#3204) 2021-07-06 00:35:12 -07:00
wangxianghu
f2621da32f [HUDI-2093] Fix empty avro schema path caused by duplicate parameters (#3177) 
* [HUDI-2093] Fix empty avro schema path caused by duplicate parameters

* rename shcmea option key

* fix doc

* rename var name
 2021-07-06 15:14:30 +08:00
Randal Boyle
60e0254e67 [HUDI-1996] Adding functionality to allow the providing of basic auth creds for confluent cloud schema registry (#3097) 
* adding support for basic auth with confluent cloud schema registry
 2021-07-05 23:40:23 -07:00
dwshmilyss
2b21ae1775 [HUDI-2046] Loaded too many classes like sun/reflect/GeneratedSerializationConstructorAccessor in JVM metaspace (#3121) 
Loaded too many classes when use kryo of spark to hudi

Co-authored-by: weiwei.duan <weiwei.duan@linkflowtech.com>
 2021-07-05 23:36:55 -07:00
Sebastian Bernauer
05d6e18190 [HUDI-2055] Added deltastreamer metric for time of lastSync (#3129) 2021-07-05 23:34:46 -07:00
Danny Chan
1d6978cde4 [HUDI-2135] Add compaction schedule option for flink (#3226) 2021-07-06 14:11:20 +08:00
rmahindra123
a4dcbb5c5a [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap (#3194) 
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-07-05 23:03:41 -07:00
xiarixiaoyao
a0f598d371 [HUDI-2089]fix the bug that metatable cannot support non_partition table (#3182) 2021-07-06 11:14:05 +08:00
swuferhong
0bd20827ab [HUDI-2133] Support hive1 metadata sync for flink writer (#3225) 2021-07-06 11:01:57 +08:00
swuferhong
bc313727e3 [HUDI-2106] Fix flink batch compaction bug while user don't set compaction tasks (#3192) 2021-07-06 09:10:37 +08:00
Danny Chan
32bd8ce088 [HUDI-2132] Make coordinator events as POJO for efficient serialization (#3223) 2021-07-06 09:02:38 +08:00
wangxianghu
650c4455c6 [HUDI-2122] Improvement in packaging insert into smallfiles (#3213) 2021-07-05 09:30:57 -07:00
pengzhiwei
287d2dd79c [HUDI-2131] Exception Throw Out When MergeInto With Decimal Type Field (#3224) 2021-07-05 22:28:57 +08:00
Danny Chan
e6ee7bdb51 [HUDI-2129] StreamerUtil.medianInstantTime should return a valid date time string (#3221) 2021-07-05 20:56:24 +08:00
xiarixiaoyao
2cecb75187 [HUDI-2058]support incremental query for insert_overwrite_table/insert_overwrite operation on cow table (#3139) 2021-07-05 18:54:05 +08:00
Shawy Geng
2033d35dc3 [HUDI-2127] Initialize the maxMemorySizeInBytes in log scanner (#3220) 2021-07-05 11:53:18 +08:00
Danny Chan
98ec017bc8 [HUDI-2126] The coordinator send events to write function when there are no data for the checkpoint (#3219) 2021-07-05 11:34:18 +08:00
xiarixiaoyao
6a71412f78 [HUDI-2116] Support batch synchronization of partition datas to hive metastore to avoid oom problem (#3209) 2021-07-04 22:30:36 +08:00
wangxianghu
62a1ad8b3a [HUDI-1930] Bootstrap support configure KeyGenerator by type (#3170) 
* [HUDI-1930] Bootstrap support configure KeyGenerator by type
 2021-07-03 20:27:37 +08:00
pengzhiwei
4f215e2938 [HUDI-2057] CTAS Generate An External Table When Create Managed Table (#3146) 2021-07-03 15:55:36 +08:00
Prashant Wason
7173d1338a [HUDI-2124] A Grafana dashboard for HUDI. (#3216) 2021-07-02 18:48:37 -07:00
pengzhiwei
70d9c2e747 [HUDI-2123] Exception When Merge With Null-Value Field (#3214) 2021-07-02 22:46:52 +08:00