lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity

[HUDI-2286] Handle the case of failed deltacommit on the metadata table. (#3428)

Browse Source 
A failed deltacommit on the metadata table will be automatically rolled back. Assuming the failed commit was "t10", the rollback will happen the next time at "t11". Post rollback, when we try to sync the dataset to the metadata table, we should look for all unsynched instants including t11. Current code ignores t11 since the latest commit timestamp on metadata table is t11 (due to rollback).
This commit is contained in:
Prashant Wason
2021-08-11 07:39:48 -07:00
committed by GitHub
parent c9fa3cffaf
commit aa11989ead
3 changed files with 87 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java
View File

@@ -299,9 +299,11 @@ public class HoodieBackedTableMetadata extends BaseTableMetadata {
return Collections.EMPTY_LIST;
}
// All instants on the data timeline, which are greater than the last instant on metadata timeline
// are candidates for sync.
String latestMetadataInstantTime = metaClient.getActiveTimeline().filterCompletedInstants().lastInstant().get().getTimestamp();
// All instants on the data timeline, which are greater than the last deltacommit instant on metadata timeline
// are candidates for sync. We only consider delta-commit instants as each actions on dataset leads to a
// deltacommit on the metadata table.
String latestMetadataInstantTime = metaClient.getActiveTimeline().getDeltaCommitTimeline().filterCompletedInstants()
.lastInstant().get().getTimestamp();
HoodieDefaultTimeline candidateTimeline = datasetMetaClient.getActiveTimeline().findInstantsAfter(latestMetadataInstantTime, Integer.MAX_VALUE);
Option<HoodieInstant> earliestIncompleteInstant = ignoreIncompleteInstants ? Option.empty()
: candidateTimeline.filterInflightsAndRequested().firstInstant();