lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,704 Commits 1 Branch 0 Tags
b4562e86e4b58d6151fdeea12e727b8c8881a213
Commit Graph

1704 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
vinoth chandar
b4562e86e4 Revert "[HUDI-2087] Support Append only in Flink stream (#3174)" (#3251) 
This reverts commit 371526789d.
 2021-07-09 11:20:09 -07:00
yuzhaojing
371526789d [HUDI-2087] Support Append only in Flink stream (#3174) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-07-09 16:06:32 +08:00
swuferhong
047d956e01 [HUDI-2136] Fix conflict when flink-sql-connector-hive and hudi-flink-bundle are both in flink lib (#3227) 2021-07-09 10:10:21 +08:00
vinoth chandar
c50c24908a [MINOR] Fix build broken from #3186 (#3245) 2021-07-08 14:23:52 -07:00
xiarixiaoyao
de07e61382 [HUDI-2099]hive lock which state is WATING should be released, otherwise this hive lock will be locked forever (#3186) 2021-07-08 10:30:48 -04:00
Sivabalan Narayanan
8c0dbaa9b3 [HUDI-2009] Fixing extra commit metadata in row writer path (#3075) 2021-07-08 03:07:27 -04:00
Yungthuis
1d3cd06572 [HUDI-2134]Add generics to avoif forced conversion in BaseSparkCommitActionExecutor#partition (#3232) 2021-07-08 13:31:38 +08:00
Sivabalan Narayanan
16e90d30ea [HUDI-1105] Adding dedup support for Bulk Insert w/ Rows (#2206) 2021-07-07 17:38:26 -04:00
Sebastian Bernauer
8f7ad8b178 [HUDI-2069] Refactored String constants (#3172) 2021-07-07 14:22:00 -04:00
Sivabalan Narayanan
ea9e5d0e8b [HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows (#3149) 2021-07-07 11:15:25 -04:00
Shawy Geng
55ecbc662e [HUDI-2115] FileSlices in the filegroup is not descending by timestamp (#3206) 2021-07-07 22:24:36 +08:00
Prashant Wason
990820476a [HUDI-2140] Fixed the unit test TestHoodieBackedMetadata.testOnlyValidPartitionsAdded. (#3234) 2021-07-06 23:50:27 -07:00
Prashant Wason
221ddd9bf3 [HUDI-2016] Fixed bootstrap of Metadata Table when some actions are in progress. (#3083) 
Metadata Table cannot be bootstrapped when any action is in progress. This is detected by the presence of inflight or requested instants. The bootstrapping is initiated in preWrite and postWrite of each commit. So bootstrapping will be retried again until it succeeds.
Also added metrics for when the bootstrapping fails or a table is re-bootstrapped. This will help detect tables which are not getting bootstrapped.
 2021-07-06 08:08:46 -07:00
Shawy Geng
6e24434682 [HUDI-2113] Fix integration testing failure caused by sql results out of order (#3204) 2021-07-06 00:35:12 -07:00
wangxianghu
f2621da32f [HUDI-2093] Fix empty avro schema path caused by duplicate parameters (#3177) 
* [HUDI-2093] Fix empty avro schema path caused by duplicate parameters

* rename shcmea option key

* fix doc

* rename var name
 2021-07-06 15:14:30 +08:00
Randal Boyle
60e0254e67 [HUDI-1996] Adding functionality to allow the providing of basic auth creds for confluent cloud schema registry (#3097) 
* adding support for basic auth with confluent cloud schema registry
 2021-07-05 23:40:23 -07:00
dwshmilyss
2b21ae1775 [HUDI-2046] Loaded too many classes like sun/reflect/GeneratedSerializationConstructorAccessor in JVM metaspace (#3121) 
Loaded too many classes when use kryo of spark to hudi

Co-authored-by: weiwei.duan <weiwei.duan@linkflowtech.com>
 2021-07-05 23:36:55 -07:00
Sebastian Bernauer
05d6e18190 [HUDI-2055] Added deltastreamer metric for time of lastSync (#3129) 2021-07-05 23:34:46 -07:00
Danny Chan
1d6978cde4 [HUDI-2135] Add compaction schedule option for flink (#3226) 2021-07-06 14:11:20 +08:00
rmahindra123
a4dcbb5c5a [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap (#3194) 
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-07-05 23:03:41 -07:00
xiarixiaoyao
a0f598d371 [HUDI-2089]fix the bug that metatable cannot support non_partition table (#3182) 2021-07-06 11:14:05 +08:00
swuferhong
0bd20827ab [HUDI-2133] Support hive1 metadata sync for flink writer (#3225) 2021-07-06 11:01:57 +08:00
swuferhong
bc313727e3 [HUDI-2106] Fix flink batch compaction bug while user don't set compaction tasks (#3192) 2021-07-06 09:10:37 +08:00
Danny Chan
32bd8ce088 [HUDI-2132] Make coordinator events as POJO for efficient serialization (#3223) 2021-07-06 09:02:38 +08:00
wangxianghu
650c4455c6 [HUDI-2122] Improvement in packaging insert into smallfiles (#3213) 2021-07-05 09:30:57 -07:00
pengzhiwei
287d2dd79c [HUDI-2131] Exception Throw Out When MergeInto With Decimal Type Field (#3224) 2021-07-05 22:28:57 +08:00
Danny Chan
e6ee7bdb51 [HUDI-2129] StreamerUtil.medianInstantTime should return a valid date time string (#3221) 2021-07-05 20:56:24 +08:00
xiarixiaoyao
2cecb75187 [HUDI-2058]support incremental query for insert_overwrite_table/insert_overwrite operation on cow table (#3139) 2021-07-05 18:54:05 +08:00
Shawy Geng
2033d35dc3 [HUDI-2127] Initialize the maxMemorySizeInBytes in log scanner (#3220) 2021-07-05 11:53:18 +08:00
Danny Chan
98ec017bc8 [HUDI-2126] The coordinator send events to write function when there are no data for the checkpoint (#3219) 2021-07-05 11:34:18 +08:00
xiarixiaoyao
6a71412f78 [HUDI-2116] Support batch synchronization of partition datas to hive metastore to avoid oom problem (#3209) 2021-07-04 22:30:36 +08:00
wangxianghu
62a1ad8b3a [HUDI-1930] Bootstrap support configure KeyGenerator by type (#3170) 
* [HUDI-1930] Bootstrap support configure KeyGenerator by type
 2021-07-03 20:27:37 +08:00
pengzhiwei
4f215e2938 [HUDI-2057] CTAS Generate An External Table When Create Managed Table (#3146) 2021-07-03 15:55:36 +08:00
Prashant Wason
7173d1338a [HUDI-2124] A Grafana dashboard for HUDI. (#3216) 2021-07-02 18:48:37 -07:00
pengzhiwei
70d9c2e747 [HUDI-2123] Exception When Merge With Null-Value Field (#3214) 2021-07-02 22:46:52 +08:00
Danny Chan
d424fe6072 [HUDI-2121] Add operator uid for flink stateful operators (#3212) 2021-07-02 19:44:32 +08:00
pengzhiwei
ac65189458 [HUDI-2114] Spark Query MOR Table Written By Flink Return Incorrect Timestamp Value (#3208) 2021-07-02 17:39:57 +08:00
Danny Chan
7462fdefc3 [HUDI-2112] Support reading pure logs file group for flink batch reader after compaction (#3202) 2021-07-02 16:29:22 +08:00
pengzhiwei
6403547431 [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql (#3126) 2021-07-02 01:08:36 -07:00
pengzhiwei
6eca06d074 [HUDI-2105] Compaction Failed For MergeInto MOR Table (#3190) 2021-07-01 23:40:14 +08:00
wangxianghu
b376cefc3e [MINOR] Add Documentation to KEYGENERATOR_TYPE_PROP (#3196) 2021-07-01 18:48:59 +08:00
pengzhiwei
b34d53fa9c [HUDI-2088] Missing Partition Fields And PreCombineField In Hoodie Properties For Table Written By Flink (#3171) 2021-07-01 17:25:18 +08:00
vinoth chandar
d07def1290 [MINOR] Fix broken build due to FlinkOptions (#3198) 2021-06-30 20:34:58 -07:00
vinoth chandar
7895a3586e [MINOR] Update .asf.yaml to codify notification settings, turn on jira comments, gh discussions (#3164) 
- Turn on comment for jira, so we can track PR activity better
- Create a notification settings that match https://gitbox.apache.org/schemes.cgi?hudi
- Try and turn on "discussions" on Github, to experiment
 2021-06-30 14:56:56 -07:00
wenningd
d412fb2fe6 [HUDI-89] Add configOption & refactor all configs based on that (#2833) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2021-06-30 14:26:30 -07:00
yuzhaojing
07e93de8b4 [HUDI-2052] Support load logFile in BootstrapFunction (#3134) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-30 20:37:00 +08:00
Vinay Patil
94f0f40fec [HUDI-1944] Support Hudi to read from committed offset (#3175) 
* [HUDI-1944] Support Hudi to read from committed offset

* [HUDI-1944] Adding group option to KafkaResetOffsetStrategies

* [HUDI-1944] Update Exception msg
 2021-06-30 16:41:28 +08:00
yuzhaojing
1cbf43b6e7 [HUDI-2103] Add rebalance before index bootstrap (#3185) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-30 16:40:55 +08:00
Sivabalan Narayanan
5564c7ec01 [HUDI-2006] Adding more yaml templates to test suite (#3073) 2021-06-29 23:05:46 -04:00
wangxianghu
202887b8ca [HUDI-2092] Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value (#3176) 2021-06-30 09:21:06 +08:00