1
0

[HUDI-2006] Adding more yaml templates to test suite (#3073)

This commit is contained in:
Sivabalan Narayanan
2021-06-29 23:05:46 -04:00
committed by GitHub
parent 202887b8ca
commit 5564c7ec01
6 changed files with 322 additions and 28 deletions

View File

@@ -488,3 +488,33 @@ Spark submit with the flag:
--saferSchemaEvolution
```
## Automated tests for N no of yamls in Local Docker environment
Hudi provides a script to assist you in testing N no of yamls automatically. Checkout the script under
hudi_root/docker folder.
generate_test_suite.sh
Example command : // execute the command from within docker folder.
./generate_test_suite.sh --execute_test_suite false --include_medium_test_suite_yaml true --include_long_test_suite_yaml true
By default, generate_test_suite will run sanity test. In addition it supports 3 more yamls.
medium_test_suite, long_test_suite and clustering_test_suite. Users can add the required yamls via command line as per thier
necessity.
Also, "--execute_test_suite" false will generate all required files and yamls in a local staging directory if users want to inspect them.
To go ahead and execute the same, you can give "--execute_test_suite true".
staging dir: docker/demo/config/test-suite/staging
Also, there are other additional configs which users can override depending on their needs.
Some of the options are
--table_type COPY_ON_WRITE/MERGE_ON_READ // refers to table type.
--medium_num_iterations 20 // refers to total iterations medium test suite should run.
--long_num_iterations 100 // refers to total iterations long test suite should run.
--intermittent_delay_mins 1 // refers to delay between successive runs within a single test suite job.
--cluster_num_itr 30 // refers to total iterations for clustering test suite.
--cluster_delay_mins 2 // refers to delay between successive runs for clustering test suite job.
--cluster_exec_itr_count 15 // refers to the iteration at which clustering needs to be triggered.