This set of changes makes sure that all builtin KeyGenerators properly implement Spark-specific APIs in a performant way (minimizing key-generators overhead)
- Move clean related configuration to HoodieCleanConfig
- Move Archival related configuration to HoodieArchivalConfig
- hoodie.compaction.payload.class move this to HoodiePayloadConfig
* Fix kafka connect readme
* Fix handling of errors in write records for kafka connect
* By default, ensure we skip error records and keep the pipeline alive
* Fix indentation
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
* Fix README with current limitations of hive sync
* Fix README with current limitations of hive sync
* Fix dep issue
* Fix Copy on Write flow
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
* [HUDI-2332] Add clustering and compaction in Kafka Connect Sink
* Disable validation check on instant time for compaction and adjust configs
* Add javadocs
* Add clustering and compaction config
* Fix transaction causing missing records in the target table
* Add debugging logs
* Fix kafka offset sync in participant
* Adjust how clustering and compaction are configured in kafka-connect
* Fix clustering strategy
* Remove irrelevant changes from other published PRs
* Update clustering logic and others
* Update README
* Fix test failures
* Fix indentation
* Fix clustering config
* Add JavaCustomColumnsSortPartitioner and make async compaction enabled by default
* Add test for JavaCustomColumnsSortPartitioner
* Add more changes after IDE sync
* Update README with clarification
* Fix clustering logic after rebasing
* Remove unrelated changes
- Inserts go into logs, hashed by Kafka and Hudi partitions
- Fixed issues with the setupKafka script
- Bumped up the default commit interval to 300 seconds
- Minor renaming
* Fixes based on tests and some improvements
* Fix the issues after running stress tests
* Fixing checkstyle issues and updating README
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
- Fixing packaging, naming of classes
- Use of log4j over slf4j for uniformity
- More follow-on fixes
- Added a version to control/coordinator events.
- Eliminated the config added to write config
- Fixed fetching of checkpoints based on table type
- Clean up of naming, code placement
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
