1
0
Commit Graph

12 Commits

Author SHA1 Message Date
Y Ethan Guo
53f75f84b8 [HUDI-2785] Add Trino setup in Docker Demo (#4300)
* [HUDI-2785] Add Trino setup in Docker Demo

* Update docker account and remove unnecessary configs

* Adjust sparkadhoc Dockerfile
2022-01-14 22:08:55 +05:30
Satish M
c7a5c8273b [HUDI-2267] Update docs and infra test configs, add support for graphite (#3482)
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
2021-09-17 10:10:15 -04:00
Prashant Wason
788d236c44 [HUDI-1303] Some improvements for the HUDI Test Suite. (#2128)
1. Use the DAG Node's label from the yaml as its name instead of UUID names which are not descriptive when debugging issues from logs.
2. Fix CleanNode constructor which is not correctly implemented
3. When generating upsets, allows more granualar control over the number of inserts and upserts - zero or more inserts and upserts can be specified instead of always requiring both inserts and upserts.
4. Fixed generation of records of specific size
   - The current code was using a class variable "shouldAddMore" which was reset to false after the first record generation causing subsequent records to be of minimum size.
   - In this change, we pre-calculate the extra size of the complex fields. When generating records, for complex fields we read the field size from this map.
5. Refresh the timeline of the DeltaSync service before calling readFromSource. This ensures that only the newest generated data is read and data generated in the older Dag Nodes is ignored (as their AVRO files will have an older timestamp).
6. Making --workload-generator-classname an optional parameter as most probably the default will be used
2020-10-07 08:33:51 -04:00
lamber-ken
11fb2c2614 [HUDI-580] Fix incorrect license header in files 2020-02-25 08:54:26 -08:00
Balaji Varadarajan
f7c2f8cedc [HUDI-329] Presto Containers for integration test must allow newly built local jars to override 2019-11-13 17:35:34 -08:00
Mehrotra
92c69f5703 Migrate integration tests to spark 2.4.4 2019-11-13 16:53:24 -08:00
Balaji Varadarajan
a6390aefc4 [HUDI-312] Make docker hdfs cluster ephemeral. This is needed to fix flakiness in integration tests. Also, Fix DeltaStreamer hanging issue due to uncaught exception 2019-11-01 11:49:59 -07:00
Balaji Varadarajan
58623631d4 [HUDI-249] Update Release-notes. Add sign-artifacts to POM and release related scripts. Add missing license headers 2019-09-13 08:41:29 -07:00
leesf
8b150a3c6b [HUDI-230] Add missing Apache License in some files 2019-08-30 09:38:28 -07:00
Bhavani Sudha Saktheeswaran
92eed6aca8 [HUDI-82] Adds Presto integration in Docker demo (#847) 2019-08-22 19:40:36 -07:00
Balaji Varadarajan
243c58f77c Move to apachehudi dockerhub repository & use openjdk docker containers 2019-04-17 16:37:58 -07:00
Balaji Varadarajan
f3418e4718 Docker Container Build and Run setup with foundations for adding docker integration tests. Docker images built with Hadoop 2.8.4 Hive 2.3.3 and Spark 2.3.1 and published to docker-hub
Look at quickstart document for how to setup docker and run demo
2018-10-02 09:28:21 +05:30