lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,792 Commits 1 Branch 0 Tags
1df5ded433eced3bbb66137bc741e616beab9a70
Commit Graph

115 Commits

Author SHA1 Message Date
Sivabalan Narayanan
1df5ded433 [HUDI-2273] Migrating some long running tests to functional test profile (#3398) 2021-08-04 19:08:50 -04:00
pengzhiwei
5574e092fb [HUDI-2232] [SQL] MERGE INTO fails with table having nested struct (#3379) 2021-08-04 18:20:29 +08:00
wenningd
91bb0d1318 [HUDI-2255] Refactor Datasource options (#3373) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2021-08-03 17:50:30 -07:00
Udit Mehrotra
1ff2d3459a [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL (#2893) 2021-08-03 14:47:40 -07:00
vinoth chandar
b21ae68e67 [MINOR] Improving runtime of TestStructuredStreaming by 2 mins (#3382) 2021-08-02 13:42:46 -07:00
Sivabalan Narayanan
fe508376fa [HUDI-2177][HUDI-2200] Adding virtual keys support for MOR table (#3315) 2021-08-02 09:45:09 -04:00
Sivabalan Narayanan
7bdae69053 [HUDI-2253] Refactoring few tests to reduce runningtime. DeltaStreamer and MultiDeltaStreamer tests. Bulk insert row writer tests (#3371) 
Co-authored-by: Sivabalan Narayanan <nsb@Sivabalans-MBP.attlocal.net>
 2021-07-29 22:22:26 -07:00
pengzhiwei
c2370402ea [HUDI-2251] Fix Exception Cause By Table Name Case Sensitivity For Append Mode Write (#3367) 2021-07-29 17:36:56 -04:00
Shawy Geng
44e41dc9bb [HUDI-2117] Unpersist the input rdd after the commit is completed to … (#3207) 
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-07-29 08:16:58 -07:00
pengzhiwei
f109c6cb0d [MINOR] fix check style error (#3365) 2021-07-29 17:29:10 +08:00
pengzhiwei
bbadac7de1 [HUDI-1425] Performance loss with the additional hoodieRecords.isEmpty() in HoodieSparkSqlWriter#write (#2296) 2021-07-28 21:30:18 -07:00
pengzhiwei
59ff8423f9 [HUDI-2223] Fix Alter Partitioned Table Failed (#3350) 2021-07-27 20:01:04 +08:00
Gary Li
925873bb3c [HUDI-2217] Fix no value present in incremental query on MOR (#3340) 2021-07-27 17:30:01 +08:00
Sivabalan Narayanan
61148c1c43 [HUDI-2176, 2178, 2179] Adding virtual key support to COW table (#3306) 2021-07-26 17:21:04 -04:00
xiarixiaoyao
5353243449 [HUDI-2214]residual temporary files after clustering are not cleaned up (#3335) 2021-07-26 10:26:20 -07:00
董可伦
a91296f14a [HUDI-2216] Correct the words fiels in the comments to fields (#3339) 2021-07-25 12:15:57 +08:00
pengzhiwei
2c910ee3af [HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table (#3332) 2021-07-23 15:21:57 +08:00
pengzhiwei
5a2f3d439e [HUDI-2139] MergeInto MOR Table May Result InCorrect Result (#3230) 2021-07-23 10:19:43 +08:00
pengzhiwei
151f22e43a [HUDI-2195] Sync Hive Failed When Execute CTAS In Spark2 And Spark3 (#3299) 2021-07-22 15:33:38 +08:00
Vinay Patil
5a94b6bf54 [HUDI-2192] Clean up Multiple versions of scala libraries detected Warning (#3292) 2021-07-21 00:33:27 -07:00
Sivabalan Narayanan
d5026e9a24 [HUDI-2161] Adding support to disable meta columns with bulk insert operation (#3247) 2021-07-19 20:43:48 -04:00
pengzhiwei
572a214412 [HUDI-1884] MergeInto Support Partial Update For COW (#3154) 2021-07-17 12:59:18 +08:00
Jintao Guan
38cd74b563 [MINOR] Allow users to choose ORC as base file format in Spark SQL (#3279) 2021-07-16 12:24:41 +08:00
vinoth chandar
75040ee9e5 [HUDI-2149] Ensure and Audit docs for every configuration class in the codebase (#3272) 
- Added docs when missing
 - Rewrote, reworded as needed
 - Made couple more classes extend HoodieConfig
 2021-07-14 10:56:08 -07:00
pengzhiwei
f0a2f378ea Merge pull request #3120 from pengzhiwei2018/dev_metasync 
[HUDI-2045] Support Read Hoodie As DataSource Table For Flink And DeltaStreamer
 2021-07-13 22:37:20 +08:00
pengzhiwei
ca440ccf88 [HUDI-2107] Support Read Log Only MOR Table For Spark (#3193) 2021-07-12 17:31:23 +08:00
pengzhiwei
ffa934182a [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And DeltaStreamer 2021-07-12 13:03:14 +08:00
Sagar Sumit
5804ad8e32 [HUDI-1483] Support async clustering for deltastreamer and Spark streaming (#3142) 
- Integrate async clustering service with HoodieDeltaStreamer and HoodieStreamingSink
- Added methods in HoodieAsyncService to reuse code
 2021-07-11 14:43:38 -04:00
Sivabalan Narayanan
8c0dbaa9b3 [HUDI-2009] Fixing extra commit metadata in row writer path (#3075) 2021-07-08 03:07:27 -04:00
Sivabalan Narayanan
16e90d30ea [HUDI-1105] Adding dedup support for Bulk Insert w/ Rows (#2206) 2021-07-07 17:38:26 -04:00
Sebastian Bernauer
8f7ad8b178 [HUDI-2069] Refactored String constants (#3172) 2021-07-07 14:22:00 -04:00
Sivabalan Narayanan
ea9e5d0e8b [HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows (#3149) 2021-07-07 11:15:25 -04:00
xiarixiaoyao
a0f598d371 [HUDI-2089]fix the bug that metatable cannot support non_partition table (#3182) 2021-07-06 11:14:05 +08:00
pengzhiwei
287d2dd79c [HUDI-2131] Exception Throw Out When MergeInto With Decimal Type Field (#3224) 2021-07-05 22:28:57 +08:00
xiarixiaoyao
2cecb75187 [HUDI-2058]support incremental query for insert_overwrite_table/insert_overwrite operation on cow table (#3139) 2021-07-05 18:54:05 +08:00
xiarixiaoyao
6a71412f78 [HUDI-2116] Support batch synchronization of partition datas to hive metastore to avoid oom problem (#3209) 2021-07-04 22:30:36 +08:00
pengzhiwei
4f215e2938 [HUDI-2057] CTAS Generate An External Table When Create Managed Table (#3146) 2021-07-03 15:55:36 +08:00
pengzhiwei
70d9c2e747 [HUDI-2123] Exception When Merge With Null-Value Field (#3214) 2021-07-02 22:46:52 +08:00
pengzhiwei
ac65189458 [HUDI-2114] Spark Query MOR Table Written By Flink Return Incorrect Timestamp Value (#3208) 2021-07-02 17:39:57 +08:00
pengzhiwei
6403547431 [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql (#3126) 2021-07-02 01:08:36 -07:00
pengzhiwei
6eca06d074 [HUDI-2105] Compaction Failed For MergeInto MOR Table (#3190) 2021-07-01 23:40:14 +08:00
wenningd
d412fb2fe6 [HUDI-89] Add configOption & refactor all configs based on that (#2833) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2021-06-30 14:26:30 -07:00
Sebastian Bernauer
b32855545b [HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection (#3111) 
[HUDI-2069] KafkaAvroSchemaDeserializer should get sourceSchema passed instead using Reflection
 2021-06-24 09:08:21 -04:00
pengzhiwei
84dd3ca18b [HUDI-2053] Insert Static Partition With DateType Return Incorrect Partition Value (#3133) 2021-06-24 19:09:37 +08:00
pengzhiwei
7e50f9a5a6 [HUDI-2061] Incorrect Schema Inference For Schema Evolved Table (#3137) 2021-06-23 22:48:01 -07:00
pengzhiwei
69c0d9e2d0 [HUDI-1883] Support Truncate Table For Hoodie (#3098) 2021-06-22 22:33:20 +08:00
pengzhiwei
4fd8a88b7e [HUDI-1776] Support AlterCommand For Hoodie (#3086) 2021-06-21 22:58:43 +08:00
pengzhiwei
b9e28e5292 [HUDI-2033] ClassCastException Throw When PreCombineField Is String Type (#3099) 2021-06-17 23:21:20 +08:00
pengzhiwei
ad53cf450e [HUDI-1879] Fix RO Tables Returning Snapshot Result (#2925) 2021-06-17 04:18:21 -07:00
Jintao Guan
b8fe5b91d5 [HUDI-764] [HUDI-765] ORC reader writer Implementation (#2999) 
Co-authored-by: Qingyun (Teresa) Kang <kteresa@uber.com>
 2021-06-15 15:21:43 -07:00