wangxianghu
b593f10629
[MINOR] Rename unit test package of hudi-spark3 from scala to java ( #2411 )
2021-01-06 23:07:24 +08:00
Ryan Pifer
4b94529aaf
[HUDI-1325] [RFC-15] Merge updates of unsynced instants to metadata table (apache#2342)
...
[RFC-15] Fix partition key in metadata table when bootstrapping from file system (apache#2387)
Co-authored-by: Ryan Pifer <ryanpife@amazon.com >
2021-01-04 07:59:47 -08:00
Udit Mehrotra
4e64226844
[HUDI-1450] Use metadata table for listing in HoodieROTablePathFilter (apache#2326)
...
[HUDI-1394] [RFC-15] Use metadata table (if present) to get all partition paths (apache#2351)
2021-01-04 07:59:47 -08:00
Gary Li
c5e8a024f6
[HUDI-1418] Set up flink client unit test infra ( #2281 )
2020-12-31 08:57:22 +08:00
pengzhiwei
b83d1d3e61
[HUDI-1484] Escape the partition value in HiveSyncTool ( #2363 )
2020-12-28 23:02:36 -05:00
lw0090
9e6889a8ce
[HUDI-1481] add structured streaming and delta streamer clustering unit test ( #2360 )
2020-12-27 20:27:09 -08:00
lw0090
e807bb895e
[HUDI-1487] fix unit test testCopyOnWriteStorage random failed ( #2364 )
2020-12-25 09:54:23 -08:00
wenningd
286055ce34
[HUDI-1451] Support bulk insert v2 with Spark 3.0.0 ( #2328 )
...
Co-authored-by: Wenning Ding <wenningd@amazon.com >
- Added support for bulk insert v2 with datasource v2 api in Spark 3.0.0.
2020-12-25 09:43:34 -05:00
wenningd
89f482eaf2
[HUDI-1489] Fix null pointer exception when reading updated written bootstrap table ( #2370 )
...
Co-authored-by: Wenning Ding <wenningd@amazon.com >
2020-12-23 11:26:24 -08:00
wangxianghu
f8ccb2872d
[HUDI-1471] Make QuickStartUtils generate deletes according to specific ts ( #2357 )
2020-12-22 21:14:18 +08:00
Sivabalan Narayanan
33d338f392
[HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue ( #2311 )
...
* Added ability to pass in `properties` to payload methods, so they can perform table/record specific merges
* Added default methods so existing payload classes are backwards compatible.
* Adding DefaultHoodiePayload to honor ordering while merging two records
* Fixing default payload based on feedback
2020-12-19 19:19:42 -08:00
lw0090
8b5d6f9430
[HUDI-1437] support more accurate spark JobGroup for better performance tracking ( #2322 )
2020-12-17 15:20:13 -08:00
wangxianghu
4ddfc61d70
[MINOR] Make QuickstartUtil generate random timestamp instead of 0 ( #2340 )
2020-12-17 18:00:23 +08:00
wenningd
26cdc457f6
[HUDI-1376] Drop Hudi metadata cols at the beginning of Spark datasource writing ( #2233 )
...
Co-authored-by: Wenning Ding <wenningd@amazon.com >
2020-12-15 16:20:48 -08:00
wangxianghu
6cf25d5c8a
[MINOR] Minor improve in IncrementalRelation ( #2314 )
2020-12-10 20:16:00 +08:00
Danny Chan
4bc45a391a
[HUDI-1445] Refactor AbstractHoodieLogRecordScanner to use Builder ( #2313 )
2020-12-10 20:02:02 +08:00
wenningd
fce1453fa6
[HUDI-1040] Make Hudi support Spark 3 ( #2208 )
...
* Fix flaky MOR unit test
* Update Spark APIs to make it be compatible with both spark2 & spark3
* Refactor bulk insert v2 part to make Hudi be able to compile with Spark3
* Add spark3 profile to handle fasterxml & spark version
* Create hudi-spark-common module & refactor hudi-spark related modules
Co-authored-by: Wenning Ding <wenningd@amazon.com >
2020-12-09 15:52:23 -08:00
