lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity

[HUDI-2767] Enabling timeline-server-based marker as default (#4112)

Browse Source 
- Changes the default config of marker type (HoodieWriteConfig.MARKERS_TYPE or hoodie.write.markers.type) from DIRECT to TIMELINE_SERVER_BASED for Spark Engine.
- Adds engine-specific marker type configs: Spark -> TIMELINE_SERVER_BASED, Flink -> DIRECT, Java -> DIRECT.
- Uses DIRECT markers as well for Spark structured streaming due to timeline server only available for the first mini-batch.
- Fixes the marker creation method for non-partitioned table in TimelineServerBasedWriteMarkers.
- Adds the fallback to direct markers even when TIMELINE_SERVER_BASED is configured, in WriteMarkersFactory: when HDFS is used, or embedded timeline server is disabled, the fallback to direct markers happens.
- Fixes the closing of timeline service.
- Fixes tests that depend on markers, mainly by starting the timeline service for each test.
This commit is contained in:
Y Ethan Guo
2021-11-26 13:41:05 -08:00
committed by GitHub
parent f8e0176eb0
commit d1e83e4ba0
35 changed files with 529 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/TimelineService.java
View File

@@ -321,8 +321,10 @@ public class TimelineService {
if (requestHandler != null) {
this.requestHandler.stop();
}
this.app.stop();
this.app = null;
if (this.app != null) {
this.app.stop();
this.app = null;
}
this.fsViewsManager.close();
LOG.info("Closed Timeline Service");
}