lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,566 Commits 1 Branch 0 Tags
4e09545be4aedea231f5d8f512ba53181ca12e33
Commit Graph

32 Commits

Author SHA1 Message Date
Sivabalan Narayanan
f8945eca08 [MINOR] Adding more test props to integ tests (#4935) 2022-03-02 08:10:43 -05:00
Sivabalan Narayanan
1379300b5b [HUDI-3483] Adding insert override nodes to integ test suite and few clean ups (#4895) 2022-02-26 08:00:15 -05:00
Sivabalan Narayanan
62605be413 [HUDI-3480][HUDI-3481] Enchancements to integ test suite (#4884) 2022-02-23 15:56:35 -05:00
Sivabalan Narayanan
0ab1a8ec80 [HUDI-3312] Fixing spark yaml and adding hive validation to integ test suite (#4731) 2022-02-08 00:40:36 -05:00
Alexey Kudinkin
4bea758738 [HUDI-3191] Rebasing Hive's FileInputFormat onto AbstractHoodieTableFileIndex (#4531) 2022-01-18 14:54:51 -08:00
Sivabalan Narayanan
a818020f72 [HUDI-2530] Adding async compaction support to integ test suite framework (#3750) 2021-10-08 11:30:48 -04:00
Satish M
c7a5c8273b [HUDI-2267] Update docs and infra test configs, add support for graphite (#3482) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-09-17 10:10:15 -04:00
Y Ethan Guo
5d60491f5b [HUDI-2388] Add DAG nodes for Spark SQL in integration test suite (#3583) 
- Fixed validation in integ test suite for both deltastreamer and write client path. 

Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-09-13 11:53:13 -04:00
Sagar Sumit
cf15431852 [HUDI-2393] Add yamls for large scale testing (#3594) 2021-09-10 09:02:01 -04:00
Sivabalan Narayanan
15bf01dcb7 [HUDI-2349] Adding spark delete node to integ test suite (#3528) 2021-08-24 10:58:47 -04:00
Udit Mehrotra
09e625becd [HOT-FIX] Add apache license to spark_command.txt.template (#3477) 2021-08-15 07:08:55 -04:00
Sivabalan Narayanan
5564c7ec01 [HUDI-2006] Adding more yaml templates to test suite (#3073) 2021-06-29 23:05:46 -04:00
Sivabalan Narayanan
ac72470e10 [HUDI-1851] Adding test suite long running automate scripts for docker (#2880) 2021-05-11 01:26:01 -07:00
Gary Li
050626ad6c [MINOR] Add Missing Apache License to test files (#2736) 2021-03-29 07:17:23 -07:00
Sivabalan Narayanan
d5f202821b Adding fixes to test suite framework. Adding clustering node and validate async operations node. (#2400) 2021-02-12 09:29:21 -08:00
Sivabalan Narayanan
8cf6a7223f [HUDI-1331] Adding support for validating entire dataset and long running tests in test suite framework (#2168) 
* trigger rebuild

* [HUDI-1156] Remove unused dependencies from HoodieDeltaStreamerWrapper Class (#1927)

* Adding support for validating records and long running tests in test sutie framework

* Adding partial validate node

* Fixing spark session initiation in Validate nodes

* Fixing validation

* Adding hive table validation to ValidateDatasetNode

* Rebasing with latest commits from master

* Addressing feedback

* Addressing comments

Co-authored-by: lamber-ken <lamberken@163.com>
Co-authored-by: linshan-ma <mabin194046@163.com>
 2020-12-26 09:29:24 -08:00
Sivabalan Narayanan
a205dd10fa [HUDI-1338] Adding Delete support to test suite framework (#2172) 
- Adding Delete support to test suite. 
         Added DeleteNode 
         Added support to generate delete records
 2020-11-01 00:15:41 -04:00
n3nash
e109a61803 1. Fix merge on read DAG to make docker demo pass (#2092) 
1. Fix merge on read DAG to make docker demo pass (#2092)
2. Fix repeat_count, rollback node
 2020-10-28 22:34:26 -04:00
shenh062326
581d54097c [HUDI-1143] Change timestamp field in HoodieTestDataGenerator from double to long 2020-09-15 20:58:29 -07:00
Abhishek Modi
53d1e55110 Test Suite should work with Docker + Unit Tests 2020-09-08 22:41:14 -07:00
Dongwook
8d19ebfd0f [HUDI-993] Let delete API use "hoodie.delete.shuffle.parallelism" (#1703) 
For Delete API, "hoodie.delete.shuffle.parallelism" isn't used as opposed to "hoodie.upsert.shuffle.parallelism" is used for upsert, this creates the performance difference between delete by upsert API with "EmptyHoodieRecordPayload" and delete API for certain cases.

This patch makes the following fixes in this regard. 
- Let deduplicateKeys method use "hoodie.delete.shuffle.parallelism"
- Repartition inputRDD as "hoodie.delete.shuffle.parallelism" in case "hoodie.combine.before.delete=false"
 2020-09-01 12:55:31 -04:00
n3nash
727f1df62c [MINOR] Suppressing spark logs for hudi-integ and hudi-utilities (#1894) 2020-07-31 19:01:25 -07:00
Nishith Agarwal
2fc2b01d86 [HUDI-394] Provide a basic implementation of test suite 2020-07-30 21:21:15 -07:00
hongdd
fa419213f6 [HUDI-703] Add test for HoodieSyncCommand (#1774) 2020-07-28 08:31:43 +08:00
lamber-ken
11fb2c2614 [HUDI-580] Fix incorrect license header in files 2020-02-25 08:54:26 -08:00
wenningd
292c1e2ff4 [HUDI-238] Make Hudi support Scala 2.12 (#1226) 
* [HUDI-238] Rename scala related artifactId & add maven profile to support Scala 2.12
 2020-01-17 14:02:21 -08:00
Balaji Varadarajan
58623631d4 [HUDI-249] Update Release-notes. Add sign-artifacts to POM and release related scripts. Add missing license headers 2019-09-13 08:41:29 -07:00
vinoth chandar
6edf0b9def [HUDI-68] Pom cleanup & demo automation (#846) 
- [HUDI-172] Cleanup Maven POM/Classpath
  - Fix ordering of dependencies in poms, to enable better resolution
  - Idea is to place more specific ones at the top
  - And place dependencies which use them below them
- [HUDI-68] : Automate demo steps on docker setup
 - Move hive queries from hive cli to beeline
 - Standardize on taking query input from text command files
 - Deltastreamer ingest, also does hive sync in a single step
 - Spark Incremental Query materialized as a derived Hive table using datasource
 - Fix flakiness in HDFS spin up and output comparison
 - Code cleanup around streamlining and loc reduction
 - Also fixed pom to not shade some hive classs in spark, to enable hive sync
 2019-08-22 20:18:50 -07:00
Balaji Varadarajan
479908fd20 HUDI-125 : Change License for all source files and update RAT configurations 2019-06-09 11:41:55 -07:00
Balaji Varadarajan
9c8f8212ef HUDI-134 - Disable inline compaction for Hoodie Demo 2019-05-28 11:19:48 -07:00
Balaji Varadarajan
64fec64097 Timeline Service with Incremental View Syncing support 2019-05-16 13:25:33 -07:00
Balaji Varadarajan
f3418e4718 Docker Container Build and Run setup with foundations for adding docker integration tests. Docker images built with Hadoop 2.8.4 Hive 2.3.3 and Spark 2.3.1 and published to docker-hub 
Look at quickstart document for how to setup docker and run demo
 2018-10-02 09:28:21 +05:30