lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,842 Commits 1 Branch 0 Tags
b7a0d76fc99144fc2eb223cda62ac9d67528c6a2
Commit Graph

137 Commits

Author SHA1 Message Date
liujinhui
5ee35a0a92 HUDI-1674 (#3488) 2021-08-18 13:45:48 +08:00
Raymond Xu
4d508ef673 [MINOR] Fix SelectPackages in HoodieSparkFunctionalTestSuite (#3476) 2021-08-15 10:17:00 -07:00
Udit Mehrotra
3e301196bf Moving to 0.10.0-SNAPSHOT on master branch. 2021-08-14 18:51:09 -07:00
Y Ethan Guo
23dca6c237 [HUDI-2268] Add upgrade and downgrade to and from 0.9.0 (#3470) 
- Added upgrade and downgrade step to and from 0.9.0. Upgrade adds few table properties. Downgrade recreates timeline server based marker files if any.
 2021-08-14 20:20:23 -04:00
vinoth chandar
18e6b79947 [MINOR] Adding back all old default val members to DataSourceOptions (#3474) 
- Added @Deprecated
 - Added @deprecated javadoc to keys and defaults suggested how to migrate
 - Moved all deprecated members to bottom to improve readability
 2021-08-14 14:49:22 -07:00
liujinhui
b7da6cb33d [HUDI-2307] When using delete_partition with ds should not rely on the primary key (#3469) 
- Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-08-14 02:53:39 -04:00
Sivabalan Narayanan
642b1b671d [HUDI-2151] Flipping defaults (#3452) 2021-08-13 19:29:22 -04:00
Sagar Sumit
9689278014 [HUDI-1363] Provide option to drop partition columns (#3465) 
- Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-08-13 13:01:26 -04:00
董可伦
6602e55cd2 [HUDI-2279]Support column name matching for insert * and update set * in merge into (#3415) 2021-08-13 14:10:07 +08:00
Sagar Sumit
0544d70d8f [MINOR] Deprecate older configs (#3464) 
Rename and deprecate props in HoodieWriteConfig

Rename and deprecate older props
 2021-08-12 20:31:04 -07:00
liujinhui
c0fc9cdaf3 MINOR (#3459) 
MOVE hoodie Deltrstreamer to hudi-utilties
 2021-08-12 18:19:05 +08:00
Sivabalan Narayanan
c9fa3cffaf [HUDI-1774] Adding support for delete_partitions to spark data source (#3437) 2021-08-11 01:03:01 -04:00
Shawy Geng
a5e496fe23 [HUDI-2292] MOR should not predicate pushdown when reading with payload_combine type (#3443) 2021-08-11 12:17:39 +08:00
swuferhong
5448cdde7e [HUDI-2170] [HUDI-1763] Always choose the latest record for HoodieRecordPayload (#3401) 2021-08-11 10:20:55 +08:00
Sivabalan Narayanan
1196736185 [HUDI-1129] Improving schema evolution support in hudi (#2927) 
* Adding support to ingest records with old schema after table's schema is evolved

* Rebasing against latest master

- Trimming test file to be < 800 lines
- Renaming config names

* Addressing feedback

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-08-10 09:15:37 -07:00
zhangyue19921010
73d898322b [MINOR] Fix travis from errors (#3432) 2021-08-10 08:25:49 -07:00
pengzhiwei
41a9986a76 [HUDI-2208] Support Bulk Insert For Spark Sql (#3328) 2021-08-09 00:18:31 -04:00
pengzhiwei
32a50d8ddb [HUDI-2243] Support Time Travel Query For Hoodie Table (#3360) 2021-08-07 19:07:22 -04:00
pengzhiwei
55d2e786db [HUDI-1842] Spark Sql Support For pre-existing Hoodie Table (#3393) 2021-08-07 07:49:26 -04:00
pengzhiwei
9ce548edb1 [MINOR] fix compile error in compaction command (#3421) 2021-08-06 16:18:19 +08:00
pengzhiwei
3f8ca1a355 [HUDI-2182] Support Compaction Command For Spark Sql (#3277) 2021-08-06 15:12:10 +08:00
pengzhiwei
0dcd6a8fca [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql (#3387) 2021-08-05 09:57:22 -04:00
Sivabalan Narayanan
1df5ded433 [HUDI-2273] Migrating some long running tests to functional test profile (#3398) 2021-08-04 19:08:50 -04:00
pengzhiwei
5574e092fb [HUDI-2232] [SQL] MERGE INTO fails with table having nested struct (#3379) 2021-08-04 18:20:29 +08:00
wenningd
91bb0d1318 [HUDI-2255] Refactor Datasource options (#3373) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2021-08-03 17:50:30 -07:00
Udit Mehrotra
1ff2d3459a [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL (#2893) 2021-08-03 14:47:40 -07:00
vinoth chandar
b21ae68e67 [MINOR] Improving runtime of TestStructuredStreaming by 2 mins (#3382) 2021-08-02 13:42:46 -07:00
Sivabalan Narayanan
fe508376fa [HUDI-2177][HUDI-2200] Adding virtual keys support for MOR table (#3315) 2021-08-02 09:45:09 -04:00
Sivabalan Narayanan
7bdae69053 [HUDI-2253] Refactoring few tests to reduce runningtime. DeltaStreamer and MultiDeltaStreamer tests. Bulk insert row writer tests (#3371) 
Co-authored-by: Sivabalan Narayanan <nsb@Sivabalans-MBP.attlocal.net>
 2021-07-29 22:22:26 -07:00
pengzhiwei
c2370402ea [HUDI-2251] Fix Exception Cause By Table Name Case Sensitivity For Append Mode Write (#3367) 2021-07-29 17:36:56 -04:00
Shawy Geng
44e41dc9bb [HUDI-2117] Unpersist the input rdd after the commit is completed to … (#3207) 
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-07-29 08:16:58 -07:00
pengzhiwei
f109c6cb0d [MINOR] fix check style error (#3365) 2021-07-29 17:29:10 +08:00
pengzhiwei
bbadac7de1 [HUDI-1425] Performance loss with the additional hoodieRecords.isEmpty() in HoodieSparkSqlWriter#write (#2296) 2021-07-28 21:30:18 -07:00
pengzhiwei
59ff8423f9 [HUDI-2223] Fix Alter Partitioned Table Failed (#3350) 2021-07-27 20:01:04 +08:00
Gary Li
925873bb3c [HUDI-2217] Fix no value present in incremental query on MOR (#3340) 2021-07-27 17:30:01 +08:00
Sivabalan Narayanan
61148c1c43 [HUDI-2176, 2178, 2179] Adding virtual key support to COW table (#3306) 2021-07-26 17:21:04 -04:00
xiarixiaoyao
5353243449 [HUDI-2214]residual temporary files after clustering are not cleaned up (#3335) 2021-07-26 10:26:20 -07:00
董可伦
a91296f14a [HUDI-2216] Correct the words fiels in the comments to fields (#3339) 2021-07-25 12:15:57 +08:00
pengzhiwei
2c910ee3af [HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table (#3332) 2021-07-23 15:21:57 +08:00
pengzhiwei
5a2f3d439e [HUDI-2139] MergeInto MOR Table May Result InCorrect Result (#3230) 2021-07-23 10:19:43 +08:00
pengzhiwei
151f22e43a [HUDI-2195] Sync Hive Failed When Execute CTAS In Spark2 And Spark3 (#3299) 2021-07-22 15:33:38 +08:00
Vinay Patil
5a94b6bf54 [HUDI-2192] Clean up Multiple versions of scala libraries detected Warning (#3292) 2021-07-21 00:33:27 -07:00
Sivabalan Narayanan
d5026e9a24 [HUDI-2161] Adding support to disable meta columns with bulk insert operation (#3247) 2021-07-19 20:43:48 -04:00
pengzhiwei
572a214412 [HUDI-1884] MergeInto Support Partial Update For COW (#3154) 2021-07-17 12:59:18 +08:00
Jintao Guan
38cd74b563 [MINOR] Allow users to choose ORC as base file format in Spark SQL (#3279) 2021-07-16 12:24:41 +08:00
vinoth chandar
75040ee9e5 [HUDI-2149] Ensure and Audit docs for every configuration class in the codebase (#3272) 
- Added docs when missing
 - Rewrote, reworded as needed
 - Made couple more classes extend HoodieConfig
 2021-07-14 10:56:08 -07:00
pengzhiwei
f0a2f378ea Merge pull request #3120 from pengzhiwei2018/dev_metasync 
[HUDI-2045] Support Read Hoodie As DataSource Table For Flink And DeltaStreamer
 2021-07-13 22:37:20 +08:00
pengzhiwei
ca440ccf88 [HUDI-2107] Support Read Log Only MOR Table For Spark (#3193) 2021-07-12 17:31:23 +08:00
pengzhiwei
ffa934182a [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And DeltaStreamer 2021-07-12 13:03:14 +08:00
Sagar Sumit
5804ad8e32 [HUDI-1483] Support async clustering for deltastreamer and Spark streaming (#3142) 
- Integrate async clustering service with HoodieDeltaStreamer and HoodieStreamingSink
- Added methods in HoodieAsyncService to reuse code
 2021-07-11 14:43:38 -04:00