lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,464 Commits 1 Branch 0 Tags
8d29863c86aed57dc8f1a0a450bce3b256de2960
Commit Graph

31 Commits

Author SHA1 Message Date
Sivabalan Narayanan
8d29863c86 [HUDI-1615] Fixing usage of NULL schema for delete operation in HoodieSparkSqlWriter (#2777) 2021-04-14 15:35:39 +08:00
Danny Chan
ab4a7b0b4a [HUDI-1788] Insert overwrite (table) for Flink writer (#2808) 
Supports `INSERT OVERWRITE` and `INSERT OVERWRITE TABLE` for Flink
writer.
 2021-04-14 10:23:37 +08:00
pengzhiwei
684622c7c9 [HUDI-1591] Implement Spark's FileIndex for Hudi to support queries via Hudi DataSource using non-globbed table path and partition pruning (#2651) 2021-04-01 11:12:28 -07:00
Liulietong
ce3e8ec870 [HUDI-1667]: Fix a null value related bug for spark vectorized reader. (#2636) 2021-03-20 07:54:20 -07:00
xiarixiaoyao
d429169ff7 [HUDI-1688]hudi write should uncache rdd， when the write operation is finnished (#2673) 2021-03-18 10:19:18 -07:00
n3nash
74241947c1 [HUDI-845] Added locking capability to allow multiple writers (#2374) 
* [HUDI-845] Added locking capability to allow multiple writers
1. Added LockProvider API for pluggable lock methodologies
2. Added Resolution Strategy API to allow for pluggable conflict resolution
3. Added TableService client API to schedule table services
4. Added Transaction Manager for wrapping actions within transactions
 2021-03-16 16:43:53 -07:00
Sivabalan Narayanan
b038623ed3 [HUDI 1615] Fixing null schema in bulk_insert row writer path (#2653) 
* [HUDI-1615] Avoid passing in null schema from row writing/deltastreamer
* Fixing null schema in bulk insert row writer path
* Fixing tests

Co-authored-by: vc <vinoth@apache.org>
 2021-03-16 09:44:11 -07:00
pengzhiwei
bc883db5de [HUDI-1636] Support Builder Pattern To Build Table Properties For HoodieTableConfig (#2596) 2021-03-05 14:10:27 +08:00
Raymond Xu
899ae70fdb [HUDI-1587] Add latency and freshness support (#2541) 
Save min and max of event time in each commit and compute the latency and freshness metrics.
 2021-03-03 20:13:12 -08:00
liujinhui
8c2197ae5e [HUDI-1269] Make whether the failure of connect hive affects hudi ingest process configurable (#2443) 
Co-authored-by: Sivabalan Narayanan <sivabala@uber.com>
 2021-02-25 10:09:32 -05:00
Sivabalan Narayanan
c9fcf964b2 [HUDI-1315] Adding builder for HoodieTableMetaClient initialization (#2534) 2021-02-20 09:54:26 +08:00
pengzhiwei
37972071ff [HUDI-1109] Support Spark Structured Streaming read from Hudi table (#2485) 2021-02-17 03:36:29 -08:00
teeyog
26da4f5462 [HUDI-1526] Translate the api partitionBy in spark datasource to hoodie.datasource.write.partitionpath.field (#2431) 2021-02-10 12:07:54 -05:00
pengzhiwei
0d8a4d0a56 [HUDI-1550] Honor ordering field for MOR Spark datasource reader (#2497) 2021-02-01 21:04:27 +08:00
jiangjiguang
5d053b495b [MINOR] Quickstart.generateUpdates method add check (#2505) 2021-01-30 10:28:00 +08:00
liujinhui
244f6def9c [MINOR] Fix dataSource cannot use hoodie.datasource.hive_sync.auto_create_database (#2444) 
fix dataSource cannot use hoodie.datasource.hive_sync.auto_create_database
 2021-01-20 22:58:18 +08:00
lw0090
de42adc230 [HUDI-1520] add configure for spark sql overwrite use INSERT_OVERWRITE_TABLE (#2428) 2021-01-11 09:07:47 -08:00
Gary Li
23e93d05c0 [MINOR] fix spark 3 build for incremental query on MOR (#2425) 2021-01-09 21:08:55 -08:00
Gary Li
79ec7b4894 [HUDI-920] Support Incremental query for MOR table (#1938) 2021-01-09 08:02:08 -08:00
Udit Mehrotra
17df517b81 [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common (#2410) 2021-01-07 11:34:06 -08:00
Udit Mehrotra
4e64226844 [HUDI-1450] Use metadata table for listing in HoodieROTablePathFilter (apache#2326) 
[HUDI-1394] [RFC-15] Use metadata table (if present) to get all partition paths (apache#2351)
 2021-01-04 07:59:47 -08:00
pengzhiwei
b83d1d3e61 [HUDI-1484] Escape the partition value in HiveSyncTool (#2363) 2020-12-28 23:02:36 -05:00
wenningd
286055ce34 [HUDI-1451] Support bulk insert v2 with Spark 3.0.0 (#2328) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>

- Added support for bulk insert v2 with datasource v2 api in Spark 3.0.0.
 2020-12-25 09:43:34 -05:00
wenningd
89f482eaf2 [HUDI-1489] Fix null pointer exception when reading updated written bootstrap table (#2370) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2020-12-23 11:26:24 -08:00
wangxianghu
f8ccb2872d [HUDI-1471] Make QuickStartUtils generate deletes according to specific ts (#2357) 2020-12-22 21:14:18 +08:00
lw0090
8b5d6f9430 [HUDI-1437] support more accurate spark JobGroup for better performance tracking (#2322) 2020-12-17 15:20:13 -08:00
wangxianghu
4ddfc61d70 [MINOR] Make QuickstartUtil generate random timestamp instead of 0 (#2340) 2020-12-17 18:00:23 +08:00
wenningd
26cdc457f6 [HUDI-1376] Drop Hudi metadata cols at the beginning of Spark datasource writing (#2233) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2020-12-15 16:20:48 -08:00
wangxianghu
6cf25d5c8a [MINOR] Minor improve in IncrementalRelation (#2314) 2020-12-10 20:16:00 +08:00
Danny Chan
4bc45a391a [HUDI-1445] Refactor AbstractHoodieLogRecordScanner to use Builder (#2313) 2020-12-10 20:02:02 +08:00
wenningd
fce1453fa6 [HUDI-1040] Make Hudi support Spark 3 (#2208) 
* Fix flaky MOR unit test

* Update Spark APIs to make it be compatible with both spark2 & spark3

* Refactor bulk insert v2 part to make Hudi be able to compile with Spark3

* Add spark3 profile to handle fasterxml & spark version

* Create hudi-spark-common module & refactor hudi-spark related modules

Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2020-12-09 15:52:23 -08:00