lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity

[HUDI-2480] FileSlice after pending compaction-requested instant-time… (#3703)

Browse Source 
* [HUDI-2480] FileSlice after pending compaction-requested instant-time is ignored by MOR snapshot reader

* include file slice after a pending compaction for spark reader

Co-authored-by: garyli1019 <yanjia.gary.li@gmail.com>
This commit is contained in:
Danny Chan
2021-11-25 22:30:09 +08:00
committed by GitHub
parent 88067f57a2
commit a2eb2b0b0a
4 changed files with 32 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java
View File

@@ -303,8 +303,8 @@ public class HoodieTableSource implements
}
HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient,
metaClient.getActiveTimeline().getCommitsTimeline()
.filterCompletedInstants(), fileStatuses);
// file-slice after pending compaction-requested instant-time is also considered valid
metaClient.getCommitsAndCompactionTimeline().filterCompletedAndCompactionInstants(), fileStatuses);
String latestCommit = fsView.getLastInstant().get().getTimestamp();
final String mergeType = this.conf.getString(FlinkOptions.MERGE_TYPE);
final AtomicInteger cnt = new AtomicInteger(0);