* Update Hive sync timestamp when change detected
Only update the last commit timestamp on the Hive table when the table schema
has changed or a partition is created/updated.
When using AWS Glue Data Catalog as the metastore for Hive this will ensure
that table versions are substantive (including schema and/or partition
changes). Prior to this change when a Hive sync is performed without schema
or partition changes the table in the Glue Data Catalog would have a new
version published with the only change being the timestamp property.
https://issues.apache.org/jira/browse/HUDI-1932
* add conditional sync flag
* fix testSyncWithoutDiffs
* fix HiveSyncConfig
Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>
* [HUDI-1848] Adding support for HMS for running DDL queries in hive-sync-tool
* [HUDI-1848] Fixing test cases
* [HUDI-1848] CR changes
* [HUDI-1848] Fix checkstyle violations
* [HUDI-1848] Fixed a bug when metastore api fails for complex schemas with multiple levels.
* [HUDI-1848] Adding the complex schema and resolving merge conflicts
* [HUDI-1848] Adding some more javadocs
* [HUDI-1848] Added javadocs for DDLExecutor impls
* [HUDI-1848] Fixed style issue
[global-hive-sync-tool] Add a global hive sync tool to sync hudi table across clusters. Add a way to rollback the replicated time stamp if we fail to sync or if we partly sync
Co-authored-by: Jagmeet Bali <jsbali@uber.com>
HiveTestUtil static setup mini servers caused connection refused issue in Azure CI environment, as TestHiveSyncTool and TestHiveMetastoreBasedLockProvider share the same test facilities. Moving TestHiveMetastoreBasedLockProvider (the easier one) to functional test with a separate and improved mini server setup resolved the issue.
Also cleaned up dfs cluster from HiveTestUtil.
The next step is to move TestHiveSyncTool to functional as well.
* [HUDI-845] Added locking capability to allow multiple writers
1. Added LockProvider API for pluggable lock methodologies
2. Added Resolution Strategy API to allow for pluggable conflict resolution
3. Added TableService client API to schedule table services
4. Added Transaction Manager for wrapping actions within transactions
- Generalize the hive-sync module for syncing to multiple metastores
- Added new options for datasource
- Added new command line for delta streamer
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
