lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,964 Commits 1 Branch 0 Tags
7601e9e4c71f86a0aa9657d97f3bf51d51833ed0
Commit Graph

20 Commits

Author SHA1 Message Date
董可伦
b8e465fdfc [MINOR] Fix typos in log4j-surefire.properties (#5212) 2022-04-15 13:33:37 -07:00
董可伦
23b31225df [HUDI-3769] Optimize the logs of HoodieMergeHandle and BufferedConnectWriter (#5200) 2022-04-01 21:17:49 +08:00
Raymond Xu
6ccbae4d2a [HUDI-2757] Implement Hudi AWS Glue sync (#5076) 2022-03-28 14:54:59 -04:00
Rajesh Mahindra
5f570ea151 [HUDI-2883] Refactor hive sync tool / config to use reflection and standardize configs (#4175) 
- Refactor hive sync tool / config to use reflection and standardize configs

Co-authored-by: sivabalan <n.siva.b@gmail.com>
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>
 2022-03-21 22:56:31 -04:00
Y Ethan Guo
b8601a9f58 [HUDI-2656] Generalize HoodieIndex for flexible record data type (#3893) 
Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>
 2022-02-03 20:24:04 -08:00
Y Ethan Guo
bc95571caa [HUDI-2735] Allow empty commits in Kafka Connect Sink for Hudi (#4544) 2022-01-10 15:31:25 -05:00
Thinking Chen
e9a7f49f55 [HUDI-3112] Fix KafkaConnect cannot sync to Hive Problem (#4458) 2022-01-09 15:31:57 -08:00
Thinking Chen
0d8ca8da4e [HUDI-3104] Kafka-connect support of hadoop config environments and properties (#4451) 2022-01-08 23:10:17 -08:00
rmahindra123
94f45e928c [HUDI-2890] Kafka Connect: Fix failed writes and avoid table service concurrent operations (#4211) 
* Fix kafka connect readme

* Fix handling of errors in write records for kafka connect

* By default, ensure we skip error records and keep the pipeline alive

* Fix indentation

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-12-03 21:30:32 -08:00
Y Ethan Guo
ea009b55a3 [HUDI-2891] Fix write configs for Java engine in Kafka Connect Sink (#4161) 2021-11-30 09:45:50 -05:00
rmahindra123
9028e6e1e4 [HUDI-2864] Fix README and scripts with current limitations of hive sync (#4129) 
* Fix README with current limitations of hive sync

* Fix README with current limitations of hive sync

* Fix dep issue

* Fix Copy on Write flow

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-11-26 15:09:32 -08:00
rmahindra123
90f2ea2f12 [HUDI-2671] Fix kafka offset handling in Kafka Connect protocol (#4021) 
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-11-24 10:03:58 -08:00
rmahindra123
fbff0799b9 [HUDI-2325] Add hive sync support to kafka connect (#3660) 
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-11-23 15:48:06 -08:00
Y Ethan Guo
ca9bfa2a40 [HUDI-2332] Add clustering and compaction in Kafka Connect Sink (#3857) 
* [HUDI-2332] Add clustering and compaction in Kafka Connect Sink

* Disable validation check on instant time for compaction and adjust configs

* Add javadocs

* Add clustering and compaction config

* Fix transaction causing missing records in the target table

* Add debugging logs

* Fix kafka offset sync in participant

* Adjust how clustering and compaction are configured in kafka-connect

* Fix clustering strategy

* Remove irrelevant changes from other published PRs

* Update clustering logic and others

* Update README

* Fix test failures

* Fix indentation

* Fix clustering config

* Add JavaCustomColumnsSortPartitioner and make async compaction enabled by default

* Add test for JavaCustomColumnsSortPartitioner

* Add more changes after IDE sync

* Update README with clarification

* Fix clustering logic after rebasing

* Remove unrelated changes
 2021-11-23 14:23:28 +05:30
rmahindra123
3686c25fae [HUDI-2469] [Kafka Connect] Replace json based payload with protobuf for Transaction protocol. (#3694) 
* Substitue Control Event with protobuf

* Fix tests

* Fix unit tests

* Add javadocs

* Add javadocs

* Address reviewer comments

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-10-19 14:29:48 -07:00
Jimmy.Zhou
9aa7cfb802 [MINOR] fix typo,'seprarated' corrected to 'separated' (#3789) 2021-10-15 13:26:16 -07:00
董可伦
36be287121 [MINOR] Fix typo,'Kakfa' corrected to 'Kafka' & 'parquest' corrected to 'parquet' (#3717) 2021-09-26 21:53:39 +08:00
vinoth chandar
57d5da68aa [HUDI-2330][HUDI-2335] Adding support for merge-on-read tables (#3679) 
- Inserts go into logs, hashed by Kafka and Hudi partitions
 - Fixed issues with the setupKafka script
 - Bumped up the default commit interval to 300 seconds
 - Minor renaming
 2021-09-16 15:24:34 -07:00
rmahindra123
9735f4b8ef [HUDI-2428] Fix protocol and other issues after stress testing Hudi Kafka Connect (#3656) 
* Fixes based on tests and some improvements
* Fix the issues after running stress tests
* Fixing checkstyle issues and updating README

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-09-14 07:14:58 -07:00
rmahindra123
e528dd798a [HUDI-2394] Implement Kafka Sink Protocol for Hudi for Ingesting Immutable Data (#3592) 
- Fixing packaging, naming of classes
 - Use of log4j over slf4j for uniformity
- More follow-on fixes
 - Added a version to control/coordinator events.
 - Eliminated the config added to write config
 - Fixed fetching of checkpoints based on table type
 - Clean up of naming, code placement

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-09-10 18:20:26 -07:00