From e0fc4ec38e686e7a5634c267e978072af6bf35cc Mon Sep 17 00:00:00 2001 From: Vinoth Chandar Date: Sun, 26 Mar 2017 17:51:51 -0700 Subject: [PATCH] Documentation update + helper method for WriteConfig builder --- docs/community.md | 2 +- docs/configurations.md | 2 ++ .../main/java/com/uber/hoodie/config/HoodieWriteConfig.java | 5 +++++ 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/community.md b/docs/community.md index 6238dcb39..4f379f181 100644 --- a/docs/community.md +++ b/docs/community.md @@ -12,5 +12,5 @@ issues or pull requests against this repo. Before you do so, please sign the [Uber CLA](https://docs.google.com/a/uber.com/forms/d/1pAwS_-dA1KhPlfxzYLBqK6rsSWwRwH95OCCZrcsY5rk/viewform). Also, be sure to write unit tests for your bug fix or feature to show that it works as expected. - +If you want to participate in day-day conversations, please join our [slack group](https://hoodielib.slack.com/x-147852474016-157730502112/signup). diff --git a/docs/configurations.md b/docs/configurations.md index 7042d8237..9b4cf5d0a 100644 --- a/docs/configurations.md +++ b/docs/configurations.md @@ -21,6 +21,8 @@ summary: "Here we list all possible configurations and what they mean" HoodieWriteClient.insert and HoodieWriteClient.upsert returns a persisted RDD[WriteStatus], this is because the Client can choose to inspect the WriteStatus and choose and commit or not based on the failures. This is a configuration for the storage level for this RDD - [withAutoCommit](#withAutoCommit) (autoCommit = true)
Should HoodieWriteClient autoCommit after insert and upsert. The client can choose to turn off auto-commit and commit on a "defined success condition" + - [withAssumeDatePartitioning](#withAssumeDatePartitioning) (assumeDatePartitioning = false)
+ Should HoodieWriteClient assume the data is partitioned by dates, i.e three levels from base path. This is a stop-gap to support tables created by versions < 0.3.1. Will be removed eventually - [withIndexConfig](#withIndexConfig) (HoodieIndexConfig)
Hoodie uses a index to help find the FileID which contains an incoming record key. This is pluggable to have a external index (HBase) or use the default bloom filter stored in the Parquet files diff --git a/hoodie-client/src/main/java/com/uber/hoodie/config/HoodieWriteConfig.java b/hoodie-client/src/main/java/com/uber/hoodie/config/HoodieWriteConfig.java index a6790be05..bb190fda7 100644 --- a/hoodie-client/src/main/java/com/uber/hoodie/config/HoodieWriteConfig.java +++ b/hoodie-client/src/main/java/com/uber/hoodie/config/HoodieWriteConfig.java @@ -327,6 +327,11 @@ public class HoodieWriteConfig extends DefaultHoodieConfig { return this; } + public Builder withAssumeDatePartitioning(boolean assumeDatePartitioning) { + props.setProperty(HOODIE_ASSUME_DATE_PARTITIONING_PROP, String.valueOf(assumeDatePartitioning)); + return this; + } + public HoodieWriteConfig build() { HoodieWriteConfig config = new HoodieWriteConfig(props); // Check for mandatory properties