lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity

[HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types

Browse Source 
- Upgrade Spark to 2.4.4, Parquet to 1.10.1, Avro to 1.8.2
- Remove spark-avro from hudi-spark-bundle. Users need to provide --packages org.apache.spark:spark-avro:2.4.4 when running spark-shell or spark-submit
- Replace com.databricks:spark-avro with org.apache.spark:spark-avro
- Shade avro in hudi-hadoop-mr-bundle to make sure it does not conflict with hive's avro version.
This commit is contained in:
Udit Mehrotra
2020-01-12 15:03:11 -08:00
committed by Balaji Varadarajan
parent d9675c4ec0
commit ad50008a59
11 changed files with 128 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
LICENSE
View File

@@ -241,27 +241,17 @@ This product includes code from https://github.com/twitter/commons/blob/master/s
limitations under the License.
=================================================================================================
This product includes code from Databricks spark-avro with the below license
This product includes code from Apache Spark
* org.apache.hudi.AvroConversionHelper copied from classes in com/databricks/spark/avro package
* org.apache.hudi.AvroConversionHelper copied from classes in org/apache/spark/sql/avro package
Copyright 2014 Databricks
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Copyright: 2014 and onwards The Apache Software Foundation
Home page: http://spark.apache.org/
License: http://www.apache.org/licenses/LICENSE-2.0
--------------------------------------------------------------------------------
This product includes code from https://github.com/big-data-europe/README
This product includes code from https://github.com/big-data-europe/README
* docker/hoodie/hadoop/base/entrypoint.sh copied from https://github.com/big-data-europe/docker-hadoop/blob/master/base/entrypoint.sh