lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,933 Commits 1 Branch 0 Tags
f52cb32f5f0c9e31b5addf29adbb886ca6d167dd
Commit Graph

1933 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
jsbali
f52cb32f5f [HUDI-2248] Fixing the closing of hms client (#3364) 
* [HUDI-2248] Fixing the closing of hms client

* [HUDI-2248] Using Hive.closeCurrent() over client.close()
 2021-09-23 13:45:24 -07:00
Shawy Geng
b4b23e401b [HUDI-2383] Clean the marker files after compaction (#3576) 2021-09-23 15:40:58 -04:00
Sagar Sumit
eeafd24f4c [HUDI-2395] Metadata tests rewrite (#3695) 
- Added commit metadata infra to test table so that we can test entire metadata using test table itself. These tests don't care about the contents of files as such and hence we should be able to test all code paths for metadata using test table.

Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-09-23 15:40:11 -04:00
Danny Chan
5515a0d319 [HUDI-2479] HoodieFileIndex throws NPE for FileSlice with pure log files (#3702) 2021-09-23 15:14:30 +08:00
Danny Chan
e813dae36d [MINOR] Cosmetic changes for flink (#3701) 2021-09-22 12:18:02 +08:00
Jimmy.Zhou
55df8f61e1 [MINOR] Fix typo."funcitons" corrected to "functions" (#3681) 2021-09-21 20:30:13 -04:00
董可伦
5a94043f38 [HUDI-2343]Fix the exception for mergeInto when the primaryKey and preCombineField of source table and target table differ in case only (#3517) 2021-09-21 22:11:52 +08:00
Sivabalan Narayanan
5091ab7311 [HUDI-2444] Fixing delete files corner cases wrt cleaning and rollback when applying changes to metadata (#3678) 2021-09-20 11:05:31 -04:00
Danny Chan
3354fac42f [HUDI-2449] Incremental read for Flink (#3686) 2021-09-19 09:06:46 +08:00
Satish M
c7a5c8273b [HUDI-2267] Update docs and infra test configs, add support for graphite (#3482) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-09-17 10:10:15 -04:00
董可伦
3a150ee181 [HUDI-2447] Extract common business logic & Fix typo (#3683) 2021-09-17 19:45:22 +08:00
liujinhui
61d0096088 [HUDI-2434] Make periodSeconds of GraphiteReporter configurable (#3667) 2021-09-17 19:39:55 +08:00
董可伦
8a652171cf [MINOR] Fix typo,'compatiblity' corrected to 'compatibility' (#3675) 2021-09-17 09:43:23 +08:00
vinoth chandar
57d5da68aa [HUDI-2330][HUDI-2335] Adding support for merge-on-read tables (#3679) 
- Inserts go into logs, hashed by Kafka and Hudi partitions
 - Fixed issues with the setupKafka script
 - Bumped up the default commit interval to 300 seconds
 - Minor renaming
 2021-09-16 15:24:34 -07:00
Sivabalan Narayanan
b8dad628e5 [HUDI-2422] Adding rollback plan and rollback requested instant (#3651) 
- This patch introduces rollback plan and rollback.requested instant. Rollback will be done in two phases, namely rollback plan and rollback action. In planning, we prepare the rollback plan and serialize it to rollback.requested. In the rollback action phase, we fetch details from the plan and just delete the files as per the plan. This will ensure final rollback commit metadata will contain all files that got rolled back even if rollback failed midway and retried again.
 2021-09-16 11:16:06 -04:00
Sarah Witt
4deaa30c8d [HUDI-2404] Add metrics-jmx to spark and flink bundles (#3632) 2021-09-16 09:53:16 -04:00
liujinhui
2791fb9a96 [HUDI-2423] Separate some config logic from HoodieMetricsConfig into HoodieMetricsGraphiteConfig HoodieMetricsJmxConfig (#3652) 2021-09-16 15:08:10 +08:00
zhangyue19921010
2d5ac55195 [HUDI-2355][Bug]Archive service executed after cleaner finished. (#3545) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-09-15 19:00:04 -04:00
Y Ethan Guo
916f12b7dd [HUDI-2433] Refactor rollback actions in hudi-client module (#3664) 2021-09-15 18:52:43 -04:00
liujinhui
86a7351c39 [MINOR] Delete Redundant code (#3661) 2021-09-15 14:46:11 +08:00
liujinhui
76554aa31a [MINOR] Add document for DataSourceReadOptions (#3653) 2021-09-15 14:33:43 +08:00
Danny Chan
627f20f9c5 [HUDI-2430] Make decimal compatible with hudi for flink writer (#3658) 2021-09-15 12:04:46 +08:00
Vinoth Chandar
d90fd1f68c [MINOR] Update Kafka connect sink readme 2021-09-14 10:36:37 -07:00
rmahindra123
9735f4b8ef [HUDI-2428] Fix protocol and other issues after stress testing Hudi Kafka Connect (#3656) 
* Fixes based on tests and some improvements
* Fix the issues after running stress tests
* Fixing checkstyle issues and updating README

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-09-14 07:14:58 -07:00
Y Ethan Guo
5d60491f5b [HUDI-2388] Add DAG nodes for Spark SQL in integration test suite (#3583) 
- Fixed validation in integ test suite for both deltastreamer and write client path. 

Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-09-13 11:53:13 -04:00
liujinhui
35a04c43a5 [HUDI-2425] TestHoodieMultiTableDeltaStreamer CI failed due to exception (#3654) 2021-09-13 06:57:04 -07:00
Danny Chan
89651c9408 [HUDI-2421] Catch the throwable when scheduling the cleaning task for flink writer (#3650) 2021-09-13 20:43:44 +08:00
liujinhui
9f3c4a2a7f [HUDI-2410] Fix getDefaultBootstrapIndexClass logical error (#3633) 2021-09-13 16:10:17 +08:00
K.I. (Dennis) Jung
c79017cb74 [HUDI-2397] Add --enable-sync parameter (#3608) 
* add meta-sync config

* update test

* keep enableMetaSync same with enableHiveSync

* Switch check logic to use `enableMetaSync`
 2021-09-13 12:04:49 +05:30
Danny Chan
280f66e0f8 [MINOR] Fix the default parallelism of write task (#3649) 2021-09-13 11:41:49 +08:00
Ankush Kanungo
4f991ee352 [HUDI-2398] Collect event time for inserts in DefaultHoodieRecordPayload (#3602) 2021-09-11 20:27:40 -07:00
Danny Chan
9d5c3e5cb9 [HUDI-2415] Add more info log for flink streaming reader (#3642) 2021-09-12 10:00:17 +08:00
董可伦
6228b17a3d [MINOR] Fix typo, 'requried' corrected to 'required' (#3643) 2021-09-11 15:46:24 +08:00
董可伦
dbcf60f370 [MINOR] fix typo (#3640) 2021-09-11 15:45:49 +08:00
Danny Chan
b30c5bdaef [HUDI-2412] Add timestamp based partitioning for flink writer (#3638) 2021-09-11 13:17:16 +08:00
zhangyue19921010
06240417e9 [HUDI-2354] Fix TimelineServer error because of replacecommit archive (#3536) 
* bug fixed

* done

* done

* travis fix

* code reviewed

* code review

* done

* code reviewed

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-09-10 21:26:04 -07:00
rmahindra123
e528dd798a [HUDI-2394] Implement Kafka Sink Protocol for Hudi for Ingesting Immutable Data (#3592) 
- Fixing packaging, naming of classes
 - Use of log4j over slf4j for uniformity
- More follow-on fixes
 - Added a version to control/coordinator events.
 - Eliminated the config added to write config
 - Fixed fetching of checkpoints based on table type
 - Clean up of naming, code placement

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-09-10 18:20:26 -07:00
Sagar Sumit
bd1d2d4952 [MINOR] Add avro schema evolution test with (non)nullable column and with(out) default value (#3639) 2021-09-10 22:03:35 +08:00
Sagar Sumit
cf15431852 [HUDI-2393] Add yamls for large scale testing (#3594) 2021-09-10 09:02:01 -04:00
wangxianghu
44b9bc145e [HUDI-2411] Remove unnecessary method overriden and note (#3636) 2021-09-10 18:58:34 +08:00
SteNicholas
512ca42d14 [MINOR] Correct the comment for the parallelism of tasks in FlinkOptions (#3634) 2021-09-10 13:42:11 +08:00
Y Ethan Guo
56d08fbe70 [HUDI-2351] Extract common FS and IO utils for marker mechanism (#3529) 2021-09-09 14:45:28 -04:00
Raymond Xu
57c8113ee1 [HUDI-2408] Deprecate FunctionalTestHarness to avoid init DFS (#3628) 2021-09-09 11:29:04 -04:00
Wei
4abcb4f659 [MINOR] Remove unused variables (#3631) 2021-09-09 23:21:16 +08:00
liujinhui
3c4eb60913 Add the document to the PUSHGATEWAY configuration item (#3627) 2021-09-09 15:53:58 +08:00
Danny Chan
db2ab9a150 [HUDI-2403] Add metadata table listing for flink query source (#3618) 2021-09-08 14:52:39 +08:00
vinoth chandar
81acb4cafe [MINOR] Remove commenting from Github, JIRA bridge (#3620) 2021-09-07 21:54:58 -07:00
Danny Chan
cf3a2ead32 [HUDI-2401] Load archived instants for flink streaming reader (#3610) 2021-09-08 10:43:54 +08:00
vinoth chandar
ea59a7ff5f [HUDI-2080] Move to ubuntu-18.04 for Azure CI (#3409) 
Update Azure CI ubuntu from 16.04 to 18.04 due to 16.04 will be removed soon

Fixed some consistently failed tests

* fix TestCOWDataSourceStorage TestMORDataSourceStorage
* reset mocks

Also update readme badge



Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>
 2021-09-07 09:44:30 -07:00
liujinhui
eb5e7eec0a MINOR_CHECKSTYLE (#3616) 
Fix checkstyle
 2021-09-07 18:19:39 +08:00