lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity

[HUDI-3972] Fixing hoodie.properties/tableConfig for no preCombine field with writes (#5424)

Browse Source 
Fixed instantiation of new table to set the null for preCombine if not explicitly set by the user.
This commit is contained in:
Sivabalan Narayanan
2022-04-25 23:03:10 -04:00
committed by GitHub
parent f2ba0fead2
commit 762623a15c
4 changed files with 127 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
View File

@@ -150,7 +150,9 @@ object HoodieSparkSqlWriter {
.setBaseFileFormat(baseFileFormat)
.setArchiveLogFolder(archiveLogFolder)
.setPayloadClassName(hoodieConfig.getString(PAYLOAD_CLASS_NAME))
.setPreCombineField(hoodieConfig.getStringOrDefault(PRECOMBINE_FIELD, null))
// we can't fetch preCombine field from hoodieConfig object, since it falls back to "ts" as default value,
// but we are interested in what user has set, hence fetching from optParams.
.setPreCombineField(optParams.getOrElse(PRECOMBINE_FIELD.key(), null))
.setPartitionFields(partitionColumns)
.setPopulateMetaFields(populateMetaFields)
.setRecordKeyFields(hoodieConfig.getString(RECORDKEY_FIELD))