* [HUDI-1937] Rollback unfinished replace commit to allow updates while clustering
* Revert and delete requested replacecommit too
* Rollback pending clustering instants transactionally
* No double locking and add a config to enable rollback
* Update config to be clear about rollback only on conflict
- Adds support for generating commit timestamps with millisecs granularity.
- Older commit timestamps (in secs granularity) will be suffixed with 999 and parsed with millisecs format.
* [HUDI-1870] Add more Spark CI build tasks
- build for spark3.0.x
- build for spark-shade-unbundle-avro
- fix build failures
- delete unnecessary assertion for spark 3.0.x
- use AvroConversionUtils#convertAvroSchemaToStructType instead of calling SchemaConverters#toSqlType directly to solve the compilation failures with spark-shade-unbundle-avro (#5)
Co-authored-by: Yann <biyan900116@gmail.com>
* Update Hive sync timestamp when change detected
Only update the last commit timestamp on the Hive table when the table schema
has changed or a partition is created/updated.
When using AWS Glue Data Catalog as the metastore for Hive this will ensure
that table versions are substantive (including schema and/or partition
changes). Prior to this change when a Hive sync is performed without schema
or partition changes the table in the Glue Data Catalog would have a new
version published with the only change being the timestamp property.
https://issues.apache.org/jira/browse/HUDI-1932
* add conditional sync flag
* fix testSyncWithoutDiffs
* fix HiveSyncConfig
Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>
* [HUDI-2795] Add mechanism to safely update,delete and recover table properties
- Fail safe mechanism, that lets queries succeed off a backup file
- Readers who are not upgraded to this version of code will just fail until recovery is done.
- Added unit tests that exercises all these scenarios.
- Adding CLI for recovery, updation to table command.
- [Pending] Add some hash based verfication to ensure any rare partial writes for HDFS
* Fixing upgrade/downgrade infrastructure to use new updation method
- Metadata table today has virtual keys disabled, thereby populating the metafields
for each record written out and increasing the overall storage space used. Hereby
adding virtual keys support for metadata table so that metafields are disabled
for metadata table records.
- Adding a custom KeyGenerator for Metadata table so as to not rely on the
default Base/SimpleKeyGenerators which currently look for record key
and partition field set in the table config.
- AbstractHoodieLogRecordReader's version of processing next data block and
createHoodieRecord() will be a generic version and making the derived class
HoodieMetadataMergedLogRecordReader take care of the special creation of
records from explictly passed in partition names.
* Enabling timeline server based markers
* Enabling timeline server based markers and marker based rollback
* Removing constraint that timeline server can be enabled only for hdfs
* Fixing tests
