lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,949 Commits 1 Branch 0 Tags
46808dcb1fe22491326a9e831dd4dde4c70796fb
Commit Graph

1949 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Y Ethan Guo
46808dcb1f [HUDI-2497] Refactor clean and restore actions in hudi-client module (#3734) 2021-09-30 18:20:25 -04:00
Vinay Patil
73e8ba7620 [HUDI-2499] Making jdbc-url, user and pass as non-required field for other sync modes (#3732) 2021-09-30 11:41:15 -04:00
vinoyang
47ed917999 [HUDI-2440] Add dependency change diff script for dependency governace (#3674) 2021-09-30 16:56:11 +08:00
tangyoupeng
def08d703d [MINOR] Support JuiceFileSystem (#3729) 2021-09-30 12:50:46 +08:00
董可伦
2f07e1267f [MINOR] Fix typo Hooodie corrected to Hoodie & reuqired corrected to required (#3730) 2021-09-30 09:55:32 +08:00
zhangyue19921010
dd1bd62684 [HUDI-2277] HoodieDeltaStreamer reading ORC files directly using ORCDFSSource (#3413) 
* add ORCDFSSource to support reading orc file into hudi format && add UTs

* remove ununsed import

* simplify tes

* code review

* code review

* code review

* code review

* code review

* code review

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-09-29 08:54:12 -07:00
vinoth chandar
2aa660f99d [MINOR] Add a RFC template and folder (#3726) 2021-09-28 10:33:27 -06:00
Sivabalan Narayanan
f0585facd6 [HUDI-2474] Refreshing timeline for every operation in Hudi when metadata is enabled (#3698) 2021-09-28 05:16:52 -04:00
qianchutao
9067657a5f [HUDI-2487] Fix JsonKafkaSource cannot filter empty messages from kafka (#3715) 2021-09-28 13:47:15 +08:00
董可伦
36be287121 [MINOR] Fix typo,'Kakfa' corrected to 'Kafka' & 'parquest' corrected to 'parquet' (#3717) 2021-09-26 21:53:39 +08:00
qianchutao
7e887b54d7 [MINOR] fix typo,'SPAKR' corrected to 'SPARK' (#3721) 2021-09-26 21:52:35 +08:00
Carl-Zhou-CN
aa546554ff [HUDI-2451] On windows client with hdfs server for wrong file separator (#3687) 
Co-authored-by: yao.zhou <yao.zhou@linkflowtech.com>
 2021-09-26 21:51:27 +08:00
Sagar Sumit
bc4966ea73 [HUDI-2484] Fix hive sync mode setting in Deltastreamer (#3712) 2021-09-24 13:05:42 -04:00
Danny Chan
31a301f0aa [HUDI-2485] Consume as mini-batch for flink stream reader (#3710) 2021-09-24 23:44:01 +08:00
Danny Chan
440525ccbb [HUDI-2483] Infer changelog mode for flink compactor (#3706) 2021-09-24 14:52:27 +08:00
Shawy Geng
06c2cc2c8b [HUDI-2385] Make parquet dictionary encoding configurable (#3578) 
Co-authored-by: leesf <leesf@apache.org>
 2021-09-24 13:33:34 +08:00
jsbali
f52cb32f5f [HUDI-2248] Fixing the closing of hms client (#3364) 
* [HUDI-2248] Fixing the closing of hms client

* [HUDI-2248] Using Hive.closeCurrent() over client.close()
 2021-09-23 13:45:24 -07:00
Shawy Geng
b4b23e401b [HUDI-2383] Clean the marker files after compaction (#3576) 2021-09-23 15:40:58 -04:00
Sagar Sumit
eeafd24f4c [HUDI-2395] Metadata tests rewrite (#3695) 
- Added commit metadata infra to test table so that we can test entire metadata using test table itself. These tests don't care about the contents of files as such and hence we should be able to test all code paths for metadata using test table.

Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-09-23 15:40:11 -04:00
Danny Chan
5515a0d319 [HUDI-2479] HoodieFileIndex throws NPE for FileSlice with pure log files (#3702) 2021-09-23 15:14:30 +08:00
Danny Chan
e813dae36d [MINOR] Cosmetic changes for flink (#3701) 2021-09-22 12:18:02 +08:00
Jimmy.Zhou
55df8f61e1 [MINOR] Fix typo."funcitons" corrected to "functions" (#3681) 2021-09-21 20:30:13 -04:00
董可伦
5a94043f38 [HUDI-2343]Fix the exception for mergeInto when the primaryKey and preCombineField of source table and target table differ in case only (#3517) 2021-09-21 22:11:52 +08:00
Sivabalan Narayanan
5091ab7311 [HUDI-2444] Fixing delete files corner cases wrt cleaning and rollback when applying changes to metadata (#3678) 2021-09-20 11:05:31 -04:00
Danny Chan
3354fac42f [HUDI-2449] Incremental read for Flink (#3686) 2021-09-19 09:06:46 +08:00
Satish M
c7a5c8273b [HUDI-2267] Update docs and infra test configs, add support for graphite (#3482) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-09-17 10:10:15 -04:00
董可伦
3a150ee181 [HUDI-2447] Extract common business logic & Fix typo (#3683) 2021-09-17 19:45:22 +08:00
liujinhui
61d0096088 [HUDI-2434] Make periodSeconds of GraphiteReporter configurable (#3667) 2021-09-17 19:39:55 +08:00
董可伦
8a652171cf [MINOR] Fix typo,'compatiblity' corrected to 'compatibility' (#3675) 2021-09-17 09:43:23 +08:00
vinoth chandar
57d5da68aa [HUDI-2330][HUDI-2335] Adding support for merge-on-read tables (#3679) 
- Inserts go into logs, hashed by Kafka and Hudi partitions
 - Fixed issues with the setupKafka script
 - Bumped up the default commit interval to 300 seconds
 - Minor renaming
 2021-09-16 15:24:34 -07:00
Sivabalan Narayanan
b8dad628e5 [HUDI-2422] Adding rollback plan and rollback requested instant (#3651) 
- This patch introduces rollback plan and rollback.requested instant. Rollback will be done in two phases, namely rollback plan and rollback action. In planning, we prepare the rollback plan and serialize it to rollback.requested. In the rollback action phase, we fetch details from the plan and just delete the files as per the plan. This will ensure final rollback commit metadata will contain all files that got rolled back even if rollback failed midway and retried again.
 2021-09-16 11:16:06 -04:00
Sarah Witt
4deaa30c8d [HUDI-2404] Add metrics-jmx to spark and flink bundles (#3632) 2021-09-16 09:53:16 -04:00
liujinhui
2791fb9a96 [HUDI-2423] Separate some config logic from HoodieMetricsConfig into HoodieMetricsGraphiteConfig HoodieMetricsJmxConfig (#3652) 2021-09-16 15:08:10 +08:00
zhangyue19921010
2d5ac55195 [HUDI-2355][Bug]Archive service executed after cleaner finished. (#3545) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-09-15 19:00:04 -04:00
Y Ethan Guo
916f12b7dd [HUDI-2433] Refactor rollback actions in hudi-client module (#3664) 2021-09-15 18:52:43 -04:00
liujinhui
86a7351c39 [MINOR] Delete Redundant code (#3661) 2021-09-15 14:46:11 +08:00
liujinhui
76554aa31a [MINOR] Add document for DataSourceReadOptions (#3653) 2021-09-15 14:33:43 +08:00
Danny Chan
627f20f9c5 [HUDI-2430] Make decimal compatible with hudi for flink writer (#3658) 2021-09-15 12:04:46 +08:00
Vinoth Chandar
d90fd1f68c [MINOR] Update Kafka connect sink readme 2021-09-14 10:36:37 -07:00
rmahindra123
9735f4b8ef [HUDI-2428] Fix protocol and other issues after stress testing Hudi Kafka Connect (#3656) 
* Fixes based on tests and some improvements
* Fix the issues after running stress tests
* Fixing checkstyle issues and updating README

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-09-14 07:14:58 -07:00
Y Ethan Guo
5d60491f5b [HUDI-2388] Add DAG nodes for Spark SQL in integration test suite (#3583) 
- Fixed validation in integ test suite for both deltastreamer and write client path. 

Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-09-13 11:53:13 -04:00
liujinhui
35a04c43a5 [HUDI-2425] TestHoodieMultiTableDeltaStreamer CI failed due to exception (#3654) 2021-09-13 06:57:04 -07:00
Danny Chan
89651c9408 [HUDI-2421] Catch the throwable when scheduling the cleaning task for flink writer (#3650) 2021-09-13 20:43:44 +08:00
liujinhui
9f3c4a2a7f [HUDI-2410] Fix getDefaultBootstrapIndexClass logical error (#3633) 2021-09-13 16:10:17 +08:00
K.I. (Dennis) Jung
c79017cb74 [HUDI-2397] Add --enable-sync parameter (#3608) 
* add meta-sync config

* update test

* keep enableMetaSync same with enableHiveSync

* Switch check logic to use `enableMetaSync`
 2021-09-13 12:04:49 +05:30
Danny Chan
280f66e0f8 [MINOR] Fix the default parallelism of write task (#3649) 2021-09-13 11:41:49 +08:00
Ankush Kanungo
4f991ee352 [HUDI-2398] Collect event time for inserts in DefaultHoodieRecordPayload (#3602) 2021-09-11 20:27:40 -07:00
Danny Chan
9d5c3e5cb9 [HUDI-2415] Add more info log for flink streaming reader (#3642) 2021-09-12 10:00:17 +08:00
董可伦
6228b17a3d [MINOR] Fix typo, 'requried' corrected to 'required' (#3643) 2021-09-11 15:46:24 +08:00
董可伦
dbcf60f370 [MINOR] fix typo (#3640) 2021-09-11 15:45:49 +08:00