lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity

[HUDI-249] Update Release-notes. Add sign-artifacts to POM and release related scripts. Add missing license headers

Browse Source
This commit is contained in:
Balaji Varadarajan
2019-09-03 19:02:39 -07:00
committed by Balaji Varadarajan
parent 895d732a14
commit 58623631d4
46 changed files with 1154 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
RELEASE_NOTES.md
View File

@@ -1,3 +1,107 @@
Release 0.5.0-incubating
------------------------------------
### Highlights
* First Apache Incubating Release
* Package Renames from com.uber.hoodie to org.apache.hudi (See https://cwiki.apache.org/confluence/display/HUDI/Migration+Guide+From+com.uber.hoodie+to+org.apache.hudi)
* Complete Redo of Hudi Jar bundling
* Bug fixes in query side integration, DeltaStreamer, compaction, rollbacks, restore
### Full PR List
* **Balaji Varadarajan** [HUDI-249] Update Release-notes. Add sign-artifacts to POM and release related scripts. Add missing license headers
* **Bhavani Sudha Saktheeswaran** [HUDI-164] Fixes incorrect averageBytesPerRecord
* **Balaji Varadarajan** [HUDI-243] Rename HoodieInputFormat and HoodieRealtimeInputFormat to HoodieParquetInputFormat and HoodieParquetRealtimeInputFormat
* **Vinoth Chandar** [HUDI-143] Excluding javax.* from utilities and spark bundles
* **vinoth chandar** [HUDI-159] Redesigning bundles for lighter-weight integrations
* **Mehrotra** Fix logging in HoodieSparkSqlWriter
* **leesf** [hotfix] fix typo
* **leesf** [HUDI-236] Failed to close stream
* **Alex Filipchik** Support nested ordering fields
* **leesf** [HUDI-230] Add missing Apache License in some files
* **Balaji Varadarajan** [HUDI-227] : DeltaStreamer Improvements : Commit empty input batch with progressing checkpoints and allow users to override configs through properties. Original PR : PR-805 and PR-806 (#863)
* **Balaji Varadarajan** HUDI-170 Updating hoodie record before inserting it into ExternalSpillableMap (#866)
* **leesf** [HUDI-229] Fix mvn notice:generate issue in windows
* **leesf** [HUDI-225] Create Hudi Timeline Server Fat Jar
* **Balaji Varadarajan** HUDI-124 : Exclude jdk.tools from hadoop-common and update Notice files (#858)
* **leesf** [hotfix] change hoodie-timeline-*.jar to hudi-timeline-*.jar
* **leesf** [HUDI-222] Rename main class path to org.apache.hudi.timeline.service.TimelineService in run_server.sh
* **Alex Filipchik** Fixed unit test
* **Alex Filipchik** Addressing comments
* **Alex Filipchik** Ignore duplicate of a compaction file
* **Alexander Filipchik** [HUDI-223] Adding a way to infer target schema from the dataset after the transformation (#854)
* **Vinoth Chandar** [HUDI-159] Precursor cleanup to reduce build warnings
* **Balaji Varadarajan** HUDI-128 Preparing POM for release and snapshot builds (#851)
* **vinoth chandar** [HUDI-159]: Pom cleanup and removal of com.twitter.parquet
* **vinoth chandar** [HUDI-68] Pom cleanup & demo automation (#846)
* **Bhavani Sudha Saktheeswaran** [HUDI-82] Adds Presto integration in Docker demo (#847)
* **leesf** HUDI-212: Specify Charset to UTF-8 for IOUtils.toString (#837)
* **vinoyang** [HUDI-205] Let checkstyle ban Java and Guava Optional instead of using Option provided by Hudi (#834)
* **Balaji Varadarajan** HUDI-204 : Make MOR rollback idempotent and disable using rolling stats for small file selection (#833)
* **Nishith Agarwal** Adding GPG Keys
* **Balaji Varadarajan** HUDI-123 Rename code packages/constants to org.apache.hudi (#830)
* **yanghua** [HUDI-153] Use com.uber.hoodie.common.util.Option instead of Java and Guava Optional
* **garyli1019** HUDI-171 delete tmp file in addShutDownHook
* **Balaji Varadarajan** HUDI-149 - Remove platform dependencies and update NOTICE plugin
* **n3nash** - Adding HoodieCombineHiveInputFormat for COW tables (#811)
* **n3nash** - Fix realtime queries by removing COLUMN_ID and COLUMN_NAME cache in inputformat (#814)
* **venkatr** Cache RDD to avoid recomputing data ingestion. Return result RDD after updating index so that this step is not skipped by chained actions on the same RDD
* **Balaji Varadarajan** HUDI-197 Hive Sync and othe CLIs using bundle picking sources jar instead of binary jar
* **vinothchandar** HUDI-178 : Add keys for vinoth to KEYS file
* **Anbu Cheeralan** Update Keys with anchee@apache.org
* **Luke Zhu** Fix typo in hoodie-presto-bundle (#818)
* **Balaji Varadarajan** Generate Source Jars for bundle packages (#810)
* **Vinoth Chandar** HUDI-92 : Making deltastreamer with DistributedTestSource also run locally
* **vinoyang** [HUDI-181] Fix the Bold markdown grammar issue of README file (#808)
* **eisig** HUDI-175 - add an option to manually override the DeltaStreamer checkpoint (#798)
* **Balaji Varadarajan** Add balaji gpg key to KEYS file
* **Balaji Varadarajan** Allow HoodieWrapperFileSystem to wrap other proxy file-system implementations with no getScheme implementation (#793)
* **Balaji Varadarajan** HUDI-140 : GCS: Log File Reading not working due to difference in seek() behavior for EOF
* **eisig** add jssc.stop() (#797)
* **n3nash** Fixing default value for avro 1.7 which assumes NULL value instead of a jsonnode that is null (#792)
* **Balaji Varadarajan** HUDI-168 Ensure getFileStatus calls for files getting written is done after close() is called (#788)
* **eisig** fix HoodieLogFileReader (#787)
* **Balaji Varadarajan** HUDI-162 : File System view must be built with correct timeline actions
* **Balaji Varadarajan** HUDI-138 - Meta Files handling also need to support consistency guard
* **Yihua Guo** [HUDI-161] Remove --key-generator-class CLI arg in HoodieDeltaStreamer and use key generator class specified in datasource properties. (#781)
* **Ho Tien Vu** Fixed TableNotFoundException when write with structured streaming (#778)
* **Thinking Chen** when column type is decimal, should add precision and scale (#753)
* **Balaji Varadarajan** Fixing bugs found during running hoodie demo (#760)
* **Ho Tien Vu** Added preemptive check for 'spark.scheduler.mode'
* **Jaimin Shah** adding support for complex keys (#728)
* **Ron Barabash** Adding support for optional skipping single archiving failures
* **Balaji Varadarajan** Reword confusing message and reducing the severity level
* **Balaji Varadarajan** Add maprfs to storage schemes
* **Balaji Varadarajan** Ensure TableMetaClient and FileSystem instances have exclusive copy of Configuration
* **Balaji Varadarajan** HUDI-70 : Making DeltaStreamer run in continuous mode with concurrent compaction
* **Balaji Varadarajan** Disable Notice Plugin
* **Balaji Varadarajan** HUDI-148 Small File selection logic for MOR must skip fileIds selected for pending compaction correctly
* **vinoth chandar** Update README.md
* **Nishith Agarwal** Reading baseCommitTime from the latest file slice as opposed to the tagged record value
* **Nishith Agarwal** - Ugrading to Hive 2.x - Eliminating in-memory deltaRecordsMap - Use writerSchema to generate generic record needed by custom payloads - changes to make tests work with hive 2.x
* **Balaji Varadarajan** All Opened hoodie clients in tests needs to be closed TestMergeOnReadTable must use embedded timeline server
* **Balaji Varadarajan** TestMergeOnReadTable must use embedded timeline server
* **Balaji Varadarajan** Turn on embedded server for all client tests
* **Balaji Varadarajan** Ensure log files are consistently ordered when scanning
* **Vinoth Chandar** Introduce HoodieReadHandle abstraction into index
* **Balaji Varadarajan** Close Hoodie Clients which are opened to properly shutdown embedded timeline service
* **Balaji Varadarajan** HUDI-147 Compaction Inflight Rollback not deleting Marker directory
* **Balaji Varadarajan** HUDI-125 : Change License for all source files and update RAT configurations
* **Balaji Varadarajan** Changes related to Licensing work 1. Go through dependencies list one round to ensure compliance. Generated current NOTICE list in all submodules (other apache projects like flink does this). To be on conservative side regarding licensing, NOTICE.txt lists all dependencies including transitive. Pending Compliance questions reported in https://issues.apache.org/jira/browse/LEGAL-461 2. Automate generating NOTICE.txt files to allow future package compliance issues be identified early as part of code-review process. 3. Added NOTICE.txt and LICENSE.txt to all HUDI jars
* **guanjianhui** exlude fasterxml and parquet from presto bundle
* **guanjianhui** set codehaus.jackson modules to the same version 1.9.13
* **Balaji Varadarajan** Auto generated Slack Channel Notifications setup
* **Balaji Varadarajan** Replace Non-Compliant dnl.utils package with Apache 2.0 licensed alternative
* **Balaji Varadarajan** LogFile comparator must handle log file names without write token for backwards compatibility
* **Thinking** fix spark-shell add jar problem
* **Vinoth Chandar** Move depedency repos to https urls
* **Vinoth Chandar** Create hoodie-utilities-bundle to host the shaded jar
* **Vinoth Chandar** Turn off noisy test
* **Vinoth Chandar** Add support for maven deploy plugin to make snapshot releases
* **guanjianhui** fix maven pom
* **Balaji Varadarajan** HUDI-139 Compaction running twice due to duplicate "map" transformation while finalizing compaction
Release 0.4.7
------------------------------------

18
docker/compose/hadoop.env
View File

@@ -1,3 +1,21 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
HIVE_SITE_CONF_javax_jdo_option_ConnectionURL=jdbc:postgresql://hive-metastore-postgresql/metastore
HIVE_SITE_CONF_javax_jdo_option_ConnectionDriverName=org.postgresql.Driver
HIVE_SITE_CONF_javax_jdo_option_ConnectionUserName=hive

18
docker/demo/compaction.commands
View File

@@ -1,3 +1,21 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
connect --path /user/hive/warehouse/stock_ticks_mor
compactions show all
compaction schedule

17
docker/demo/config/schema.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{
"type":"record",
"name":"stock_ticks",

18
docker/demo/hive-batch1.commands
View File

@@ -1,3 +1,21 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
add jar ${hudi.hadoop.bundle};
select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';

18
docker/demo/hive-batch2-after-compaction.commands
View File

@@ -1,3 +1,21 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
add jar ${hudi.hadoop.bundle};
select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG';

18
docker/demo/hive-incremental.commands
View File

@@ -1,3 +1,21 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
add jar ${hudi.hadoop.bundle};
set hoodie.stock_ticks_cow.consume.mode=INCREMENTAL;

18
docker/demo/hive-table-check.commands
View File

@@ -1,3 +1,21 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
add jar ${hudi.hadoop.bundle};
show tables;

18
docker/hoodie/hadoop/prestobase/bin/entrypoint.sh
View File

@@ -1,5 +1,23 @@
#!/bin/bash
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
set -eo pipefail
wait_until() {

20
docker/hoodie/hadoop/prestobase/bin/mustache.sh
View File

@@ -1,5 +1,23 @@
#!/bin/sh
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
# `mustache.sh`, Mustache in POSIX shell.
set -e
@@ -12,4 +30,4 @@ set -e
# Because it doesn't accept the `--compile` or `--tokens` command-line
# options and does not accept input file(s) as arguments, this program
# is called `mustache.sh`(1), not `mustache`(1).
mustache
mustache

17
docker/hoodie/hadoop/prestobase/etc/catalog/hive.properties
View File

@@ -1,3 +1,20 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
connector.name=hive-hadoop2
hive.metastore-cache-ttl=1s
hive.metastore-refresh-interval=1m

17
docker/hoodie/hadoop/prestobase/etc/catalog/jmx.properties
View File

@@ -1,3 +1,20 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
connector.name=jmx
jmx.dump-tables=java.lang:type=Runtime,com.facebook.presto.execution.scheduler:name=NodeScheduler
jmx.dump-period=10s

17
docker/hoodie/hadoop/prestobase/etc/catalog/localfile.properties
View File

@@ -1,3 +1,20 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
connector.name=localfile
presto-logs.http-request-log.location=/var/log/presto
presto-logs.http-request-log.pattern=http-request.*

17
docker/hoodie/hadoop/prestobase/etc/coordinator.properties.mustache
View File

@@ -1,3 +1,20 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=8090

19
docker/hoodie/hadoop/prestobase/etc/jvm.config.mustache
View File

@@ -1,3 +1,20 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
-server
-Xmx{{PRESTO_JVM_MAX_HEAP}}
-XX:+UseG1GC
@@ -6,4 +23,4 @@
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:OnOutOfMemoryError=kill -9 %p
-DHADOOP_USER_NAME=hive
-DHADOOP_USER_NAME=hive

17
docker/hoodie/hadoop/prestobase/etc/log.properties
View File

@@ -1 +1,18 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
com.facebook.presto=INFO

17
docker/hoodie/hadoop/prestobase/etc/node.properties.mustache
View File

@@ -1,3 +1,20 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
node.environment=production
node.id={{PRESTO_NODE_ID}}
node.data-dir={{PRESTO_LOG_DIR}}

17
docker/hoodie/hadoop/prestobase/etc/worker.properties.mustache
View File

@@ -1,3 +1,20 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
coordinator=false
http-server.http.port=8090
query.max-memory={{PRESTO_QUERY_MAX_MEMORY}}

20
docker/hoodie/hadoop/prestobase/lib/mustache.sh
View File

@@ -1,3 +1,21 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
# `mustache.sh`, Mustache in POSIX shell.
set -e
@@ -249,4 +267,4 @@ _mustache_tag() {
">") _mustache_die "{{>$_M_TAG}} syntax not implemented";;
esac
}
}

18
docker/hoodie/hadoop/prestobase/pom.xml
View File

@@ -1,4 +1,22 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!--
~ Copyright (c) 2016 Uber Technologies, Inc. (hoodie-dev-group@uber.com)
~

17
hudi-client/src/test/resources/exampleEvolvedSchema.txt
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{
"namespace": "example.schema",
"type": "record",

17
hudi-client/src/test/resources/exampleSchema.txt
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{
"namespace": "example.schema",
"type": "record",

17
hudi-common/src/main/avro/HoodieArchivedMetaEntry.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{
"type":"record",
"name":"HoodieArchivedMetaEntry",

17
hudi-common/src/main/avro/HoodieCleanMetadata.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{"namespace": "org.apache.hudi.avro.model",
"type": "record",
"name": "HoodieCleanMetadata",

17
hudi-common/src/main/avro/HoodieCommitMetadata.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{
"namespace":"org.apache.hudi.avro.model",
"type":"record",

17
hudi-common/src/main/avro/HoodieCompactionMetadata.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{
"namespace":"org.apache.hudi.avro.model",
"type":"record",

17
hudi-common/src/main/avro/HoodieCompactionOperation.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{
"namespace":"org.apache.hudi.avro.model",
"type":"record",

17
hudi-common/src/main/avro/HoodieRestoreMetadata.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{"namespace": "org.apache.hudi.avro.model",
"type": "record",
"name": "HoodieRestoreMetadata",

17
hudi-common/src/main/avro/HoodieRollbackMetadata.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{"namespace": "org.apache.hudi.avro.model",
"type": "record",
"name": "HoodieRollbackMetadata",

17
hudi-common/src/main/avro/HoodieSavePointMetadata.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{"namespace": "org.apache.hudi.avro.model",
"type": "record",
"name": "HoodieSavepointMetadata",

17
hudi-common/src/test/resources/complex-test-evolved.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{
"namespace": "example.avro",
"type": "record",

17
hudi-common/src/test/resources/simple-test-evolved.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{
"namespace": "example.avro",
"type": "record",

17
hudi-common/src/test/resources/simple-test.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{
"namespace": "example.avro",
"type": "record",

17
hudi-hadoop-mr/src/test/resources/sample1.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{
"type" : "record",
"name" : "testRecord",

18
hudi-integ-test/pom.xml
View File

@@ -1,4 +1,22 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>hudi</artifactId>

15
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/SchedulerConfGenerator.java
View File

@@ -1,18 +1,19 @@
/*
* Copyright (c) 2019 Uber Technologies, Inc. (hoodie-dev-group@uber.com)
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package org.apache.hudi.utilities.deltastreamer;

18
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/AbstractBaseTestSource.java
View File

@@ -1,3 +1,21 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hudi.utilities.sources;
import java.io.File;

16
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/DistributedTestDataSource.java
View File

@@ -1,19 +1,19 @@
/*
* Copyright (c) 2019 Uber Technologies, Inc. (hoodie-dev-group@uber.com)
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*
*/
package org.apache.hudi.utilities.sources;

16
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/config/TestSourceConfig.java
View File

@@ -1,19 +1,19 @@
/*
* Copyright (c) 2019 Uber Technologies, Inc. (hoodie-dev-group@uber.com)
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*
*/
package org.apache.hudi.utilities.sources.config;

17
hudi-utilities/src/test/resources/delta-streamer-config/source.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{
"type" : "record",
"name" : "triprec",

17
hudi-utilities/src/test/resources/delta-streamer-config/target.avsc
View File

@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
{
"type" : "record",
"name" : "triprec",

14
pom.xml
View File

@@ -940,6 +940,20 @@
<additionalparam>-Xdoclint:none</additionalparam>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-gpg-plugin</artifactId>
<version>1.4</version>
<executions>
<execution>
<id>sign-artifacts</id>
<phase>verify</phase>
<goals>
<goal>sign</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>

82
release/scripts/create_source_release.sh Executable file
View File

@@ -0,0 +1,82 @@
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
##
## Variables with defaults (if not overwritten by environment)
##
MVN=${MVN:-mvn}
# fail immediately
set -o errexit
set -o nounset
# print command before executing
set -o xtrace
CURR_DIR=`pwd`
if [[ `basename $CURR_DIR` != "release" ]] ; then
echo "You have to call the script from the release/ dir"
exit 1
fi
RELEASE_VERSION=`grep -A 5 "<artifactId>hudi</artifactId>" ../pom.xml | grep '<version>' | sed -e 's/<version>//' -e 's/<\/version>//' -e 's/ //g'`
if [ -z "${RELEASE_VERSION}" ]; then
echo "RELEASE_VERSION was not set."
exit 1
fi
echo "RELEASE_VERSION=${RELEASE_VERSION}"
if [ "$(uname)" == "Darwin" ]; then
SHASUM="shasum -a 512"
else
SHASUM="sha512sum"
fi
###########################
cd ..
HUDI_DIR=`pwd`
RELEASE_DIR=${HUDI_DIR}/src_release
CLONE_DIR=${RELEASE_DIR}/hudi-tmp-clone
echo "Creating source package"
rm -rf ${RELEASE_DIR}
mkdir -p ${RELEASE_DIR}
# create a temporary git clone to ensure that we have a pristine source release
git clone ${HUDI_DIR} ${CLONE_DIR}
cd ${CLONE_DIR}
rsync -a \
--exclude ".git" --exclude ".gitignore" --exclude ".gitattributes" --exclude ".travis.yml" \
--exclude ".github" --exclude "target" \
--exclude ".idea" --exclude "*.iml" --exclude ".DS_Store" --exclude "build-target" \
--exclude "docs/content" --exclude ".rubydeps" \
. hudi-$RELEASE_VERSION
tar czf ${RELEASE_DIR}/hudi-${RELEASE_VERSION}.src.tgz hudi-$RELEASE_VERSION
gpg --armor --detach-sig ${RELEASE_DIR}/hudi-${RELEASE_VERSION}.src.tgz
cd ${RELEASE_DIR}
$SHASUM hudi-${RELEASE_VERSION}.src.tgz > hudi-${RELEASE_VERSION}.src.tgz.sha512
cd ${CURR_DIR}
rm -rf ${CLONE_DIR}

138
release/scripts/cut_release_branch.sh Executable file
View File

@@ -0,0 +1,138 @@
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# This script will update apache hudi (incubating) master branch with next release version
# and cut release branch for current development version.
# Parse parameters passing into the script
set -e
function clean_up(){
echo "Do you want to clean local clone repo? [y|N]"
read confirmation
if [[ $confirmation = "y" ]]; then
cd ~
rm -rf ${LOCAL_CLONE_DIR}
echo "Clean up local repo."
fi
}
if [[ $# -eq 1 && $1 = "-h" ]]; then
echo "This script will update apache hudi(incubating) master branch with next release version and cut release branch for current development version."
echo "There are two params required:"
echo "--release=\${CURRENT_RELEASE_VERSION}"
echo "--next_release=\${NEXT_RELEASE_VERSION}"
exit
else
for param in "$@"
do
if [[ $param =~ --release\=([0-9]\.[0-9]*\.[0-9]) ]]; then
RELEASE=${BASH_REMATCH[1]}
fi
if [[ $param =~ --next_release\=([0-9]\.[0-9]*\.[0-9]) ]]; then
NEXT_VERSION_IN_BASE_BRANCH=${BASH_REMATCH[1]}
fi
done
fi
if [[ -z "$RELEASE" || -z "$NEXT_VERSION_IN_BASE_BRANCH" ]]; then
echo "This sricpt needs to be ran with params, please run with -h to get more instructions."
exit
fi
MASTER_BRANCH=master
RELEASE_BRANCH=release-${RELEASE}
GITHUB_REPO_URL=https://gitbox.apache.org/repos/asf/incubator-hudi.git
HUDI_ROOT_DIR=incubator-hudi
LOCAL_CLONE_DIR=hudi_release_${RELEASE}
echo "=====================Environment Variables====================="
echo "version: ${RELEASE}"
echo "next_release: ${NEXT_VERSION_IN_BASE_BRANCH}"
echo "working master branch: ${MASTER_BRANCH}"
echo "working release branch: ${RELEASE_BRANCH}"
echo "local repo dir: ~/${LOCAL_CLONE_DIR}/${HUDI_ROOT_DIR}"
echo "==============================================================="
cd ~
if [[ -d ${LOCAL_CLONE_DIR} ]]; then
rm -rf ${LOCAL_CLONE_DIR}
fi
mkdir ${LOCAL_CLONE_DIR}
cd ${LOCAL_CLONE_DIR}
git clone ${GITHUB_REPO_URL}
cd ${HUDI_ROOT_DIR}
# Create local release branch
git branch ${RELEASE_BRANCH}
git checkout ${MASTER_BRANCH}
echo "====================Current working branch====================="
echo ${MASTER_BRANCH}
echo "==============================================================="
# Update master branch
mvn versions:set -DnewVersion=${NEXT_VERSION_IN_BASE_BRANCH}-SNAPSHOT
echo "==============Update master branch as following================"
git diff
echo "==============================================================="
echo "Please make sure all changes above are expected. Do you confirm to commit?: [y|N]"
read confirmation
if [[ $confirmation != "y" ]]; then
echo "Exit without committing any changes on master branch."
clean_up
exit
fi
git commit -am "Moving to ${NEXT_VERSION_IN_BASE_BRANCH}-SNAPSHOT on master branch."
#if git push origin ${MASTER_BRANCH}; then
# break
#else
# clean_up
# exit
#fi
# Checkout and update release branch - Add incubating and remove snapshot
git checkout ${RELEASE_BRANCH}
mvn versions:set -DnewVersion=${RELEASE}-incubating
echo "==================Current working branch======================="
echo ${RELEASE_BRANCH}
echo "==============================================================="
echo "===============Update release branch as following=============="
git diff
echo "==============================================================="
echo "Please make sure all changes above are expected. Do you confirm to commit?: [y|N]"
read confirmation
if [[ $confirmation != "y" ]]; then
echo "Exit without committing any changes on release branch."
clean_up
exit
fi
git commit -am "Create release branch for version ${RELEASE}."
#git push --set-upstream origin ${RELEASE_BRANCH}
clean_up

44
release/scripts/deploy_staging_jars.sh Executable file
View File

@@ -0,0 +1,44 @@
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
##
## Variables with defaults (if not overwritten by environment)
##
MVN=${MVN:-mvn}
# fail immediately
set -o errexit
set -o nounset
# print command before executing
set -o xtrace
CURR_DIR=`pwd`
if [[ `basename $CURR_DIR` != "release" ]] ; then
echo "You have to call the script from the release/ dir"
exit 1
fi
###########################
cd ..
echo "Deploying to repository.apache.org"
COMMON_OPTIONS="-Prelease -DskipTests -DretryFailedDeploymentCount=10 -DdeployArtifacts=true"
$MVN clean deploy $COMMON_OPTIONS

104
release/scripts/preparation_before_release.sh Executable file
View File

@@ -0,0 +1,104 @@
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# This script will install and configure GPG key.
set -e
LOCAL_SVN_DIR=local_svn_dir
ROOT_SVN_URL=https://dist.apache.org/repos/dist/
DEV_REPO=dev
RELEASE_REPO=release
BEAM_REPO=hudi
cd ~
echo "=================Checking GPG Key===================="
echo "You need a GPG key which reflects your Apache account."
echo "Do you want to generate a new GPG key associated with your Apache account? [y|N]"
read confirmation
if [[ $confirmation = "y" ]]; then
echo "===============Generating new GPG key================"
sudo apt-get install rng-tools
sudo rngd -r /dev/urandom
gpg --full-generate-key
fi
echo "================Listing all GPG keys================="
gpg --list-keys
echo "Please copy the public key which is associated with your Apache account:"
read pub_key
echo "===========Configuring git signing key==============="
git config --global user.signingkey $pub_key
git config --list
echo "===========Adding your key into KEYS file============"
echo "It's required to append your key into KEYS file in dist.apache.org"
echo "Have you put your key in KEYS? [y|N]"
read confirmation
if [[ $confirmation != "y" ]]; then
echo "Only PMC member can write into dist.apache.org. Are you a PMC member? [y|N]"
read pmc_permission
if [[ $pmc_permission != "y" ]]; then
echo "Please ask a PMC member to help you add your key in dev@ list."
echo "Skip adding key into dist.apache.org/KEYS file."
else
echo "Please input your name: "
read name
echo "======Starting updating KEYS file in dev repo===="
if [[ -d ${LOCAL_SVN_DIR} ]]; then
rm -rf ${LOCAL_SVN_DIR}
fi
mkdir ${LOCAL_SVN_DIR}
cd ${LOCAL_SVN_DIR}
svn co ${ROOT_SVN_URL}/${DEV_REPO}/${BEAM_REPO}
cd ${BEAM_REPO}
(gpg --list-sigs ${name} && gpg --armor --export ${name}) >> KEYS
svn status
echo "Please review all changes. Do you confirm to commit? [y|N]"
read commit_confirmation
if [[ $commit_confirmation = "y" ]]; then
svn commit --no-auth-cache KEYS
else
echo "Not commit new changes into ${ROOT_SVN_URL}/${DEV_REPO}/${BEAM_REPO}${DEV_REPO}/KEYS"
fi
cd ~/${LOCAL_SVN_DIR}
echo "===Starting updating KEYS file in release repo==="
svn co ${ROOT_SVN_URL}/${RELEASE_REPO}/${BEAM_REPO}
cd ${BEAM_REPO}
(gpg --list-sigs ${name} && gpg --armor --export ${name}) >> KEYS
svn status
echo "Please review all changes. Do you confirm to commit? [y|N]"
read commit_confirmation
if [[ $commit_confirmation = "y" ]]; then
svn commit --no-auth-cache KEYS
else
echo "Not commit new changes into ${ROOT_SVN_URL}/${DEV_REPO}/${BEAM_REPO}${RELEASE_REPO}/KEYS"
fi
cd ~
rm -rf ${LOCAL_SVN_DIR}
fi
fi
echo "================Setting up gpg agent================="
eval $(gpg-agent --daemon --no-grab --write-env-file $HOME/.gpg-agent-info)
export GPG_TTY=$(tty)
export GPG_AGENT_INFO