Y Ethan Guo d1e83e4ba0 [HUDI-2767] Enabling timeline-server-based marker as default (#4112) ...

- Changes the default config of marker type (HoodieWriteConfig.MARKERS_TYPE or hoodie.write.markers.type) from DIRECT to TIMELINE_SERVER_BASED for Spark Engine.
- Adds engine-specific marker type configs: Spark -> TIMELINE_SERVER_BASED, Flink -> DIRECT, Java -> DIRECT.
- Uses DIRECT markers as well for Spark structured streaming due to timeline server only available for the first mini-batch.
- Fixes the marker creation method for non-partitioned table in TimelineServerBasedWriteMarkers.
- Adds the fallback to direct markers even when TIMELINE_SERVER_BASED is configured, in WriteMarkersFactory: when HDFS is used, or embedded timeline server is disabled, the fallback to direct markers happens.
- Fixes the closing of timeline service.
- Fixes tests that depend on markers, mainly by starting the timeline service for each test.

2021-11-26 16:41:05 -05:00

..

hudi-spark

[HUDI-2767] Enabling timeline-server-based marker as default (#4112)

2021-11-26 16:41:05 -05:00

hudi-spark2

[HUDI-1869] Upgrading Spark3 To 3.1 (#3844)

2021-11-02 18:25:12 -07:00

hudi-spark3

[HUDI-1870] Add more Spark CI build tasks (#4022)

2021-11-22 02:16:45 -08:00

hudi-spark-common

[HUDI-2767] Enabling timeline-server-based marker as default (#4112)

2021-11-26 16:41:05 -05:00

pom.xml

Moving to 0.10.0-SNAPSHOT on master branch.

2021-08-14 18:51:09 -07:00