lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity

[HUDI-2286] Handle the case of failed deltacommit on the metadata table. (#3428)

Browse Source 
A failed deltacommit on the metadata table will be automatically rolled back. Assuming the failed commit was "t10", the rollback will happen the next time at "t11". Post rollback, when we try to sync the dataset to the metadata table, we should look for all unsynched instants including t11. Current code ignores t11 since the latest commit timestamp on metadata table is t11 (due to rollback).
This commit is contained in:
Prashant Wason
2021-08-11 07:39:48 -07:00
committed by GitHub
parent c9fa3cffaf
commit aa11989ead
3 changed files with 87 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
hudi-common/src/test/java/org/apache/hudi/common/testutils/FileCreateUtils.java
View File

@@ -274,6 +274,30 @@ public class FileCreateUtils {
return markerFilePath.toAbsolutePath().toString();
}
private static void removeMetaFile(String basePath, String instantTime, String suffix) throws IOException {
Path parentPath = Paths.get(basePath, HoodieTableMetaClient.METAFOLDER_NAME);
Path metaFilePath = parentPath.resolve(instantTime + suffix);
if (Files.exists(metaFilePath)) {
Files.delete(metaFilePath);
}
}
public static void deleteCommit(String basePath, String instantTime) throws IOException {
removeMetaFile(basePath, instantTime, HoodieTimeline.COMMIT_EXTENSION);
}
public static void deleteRequestedCommit(String basePath, String instantTime) throws IOException {
removeMetaFile(basePath, instantTime, HoodieTimeline.REQUESTED_COMMIT_EXTENSION);
}
public static void deleteInflightCommit(String basePath, String instantTime) throws IOException {
removeMetaFile(basePath, instantTime, HoodieTimeline.INFLIGHT_COMMIT_EXTENSION);
}
public static void deleteDeltaCommit(String basePath, String instantTime) throws IOException {
removeMetaFile(basePath, instantTime, HoodieTimeline.DELTA_COMMIT_EXTENSION);
}
public static long getTotalMarkerFileCount(String basePath, String partitionPath, String instantTime, IOType ioType) throws IOException {
Path parentPath = Paths.get(basePath, HoodieTableMetaClient.TEMPFOLDER_NAME, instantTime, partitionPath);
if (Files.notExists(parentPath)) {