lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,391 Commits 1 Branch 0 Tags
e00a9042e9a4e97f6f8ccaed4ac51a9aa8c2da76
Commit Graph

11 Commits

Author SHA1 Message Date
Y Ethan Guo
76a72641f1 [HUDI-3188] Update quick start guide for Kafka Connect Sink for Hudi (#4527) 2022-01-07 07:56:08 -05:00
rmahindra123
94f45e928c [HUDI-2890] Kafka Connect: Fix failed writes and avoid table service concurrent operations (#4211) 
* Fix kafka connect readme

* Fix handling of errors in write records for kafka connect

* By default, ensure we skip error records and keep the pipeline alive

* Fix indentation

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-12-03 21:30:32 -08:00
rmahindra123
9028e6e1e4 [HUDI-2864] Fix README and scripts with current limitations of hive sync (#4129) 
* Fix README with current limitations of hive sync

* Fix README with current limitations of hive sync

* Fix dep issue

* Fix Copy on Write flow

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-11-26 15:09:32 -08:00
rmahindra123
fbff0799b9 [HUDI-2325] Add hive sync support to kafka connect (#3660) 
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-11-23 15:48:06 -08:00
Y Ethan Guo
ca9bfa2a40 [HUDI-2332] Add clustering and compaction in Kafka Connect Sink (#3857) 
* [HUDI-2332] Add clustering and compaction in Kafka Connect Sink

* Disable validation check on instant time for compaction and adjust configs

* Add javadocs

* Add clustering and compaction config

* Fix transaction causing missing records in the target table

* Add debugging logs

* Fix kafka offset sync in participant

* Adjust how clustering and compaction are configured in kafka-connect

* Fix clustering strategy

* Remove irrelevant changes from other published PRs

* Update clustering logic and others

* Update README

* Fix test failures

* Fix indentation

* Fix clustering config

* Add JavaCustomColumnsSortPartitioner and make async compaction enabled by default

* Add test for JavaCustomColumnsSortPartitioner

* Add more changes after IDE sync

* Update README with clarification

* Fix clustering logic after rebasing

* Remove unrelated changes
 2021-11-23 14:23:28 +05:30
Y Ethan Guo
6aa710eae0 [MINOR] Add more configuration to Kafka setup script (#3992) 
* [MINOR] Add more configuration to Kafka setup script

* Add option to reuse Kafka topic

* Minor fixes to README
 2021-11-23 07:33:38 +05:30
Y Ethan Guo
643b1a4df2 [MINOR] Fix README for hudi-kafka-connect (#3858) 2021-10-26 14:45:52 -07:00
vinoth chandar
57d5da68aa [HUDI-2330][HUDI-2335] Adding support for merge-on-read tables (#3679) 
- Inserts go into logs, hashed by Kafka and Hudi partitions
 - Fixed issues with the setupKafka script
 - Bumped up the default commit interval to 300 seconds
 - Minor renaming
 2021-09-16 15:24:34 -07:00
Vinoth Chandar
d90fd1f68c [MINOR] Update Kafka connect sink readme 2021-09-14 10:36:37 -07:00
rmahindra123
9735f4b8ef [HUDI-2428] Fix protocol and other issues after stress testing Hudi Kafka Connect (#3656) 
* Fixes based on tests and some improvements
* Fix the issues after running stress tests
* Fixing checkstyle issues and updating README

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-09-14 07:14:58 -07:00
rmahindra123
e528dd798a [HUDI-2394] Implement Kafka Sink Protocol for Hudi for Ingesting Immutable Data (#3592) 
- Fixing packaging, naming of classes
 - Use of log4j over slf4j for uniformity
- More follow-on fixes
 - Added a version to control/coordinator events.
 - Eliminated the config added to write config
 - Fixed fetching of checkpoints based on table type
 - Clean up of naming, code placement

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-09-10 18:20:26 -07:00