lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity

[HUDI-1174] Changes for bootstrapped tables to work with presto (#1944)

Browse Source 
The purpose of this pull request is to implement changes required on Hudi side to get Bootstrapped tables integrated with Presto. The testing was done against presto 0.232 and following changes were identified to make it work:

Annotation UseRecordReaderFromInputFormat is required on HoodieParquetInputFormat as well, because the reading for bootstrapped tables needs to happen through record reader to be able to perform the merge. On presto side, this annotation is already handled.

We need to internally maintain VIRTUAL_COLUMN_NAMES because presto's internal hive version hive-apache-1.2.2 has VirutalColumn as a class, versus the one we depend on in hudi which is an enum. 

Dependency changes in hudi-presto-bundle to avoid runtime exceptions.
This commit is contained in:
Udit Mehrotra
2020-08-12 17:51:31 -07:00
committed by GitHub
parent 8b928e9bca
commit 8d04268264
5 changed files with 79 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

58
packaging/hudi-presto-bundle/pom.xml
View File

@@ -76,7 +76,12 @@
<include>org.apache.hbase:hbase-common</include>
<include>org.apache.hbase:hbase-protocol</include>
<include>org.apache.hbase:hbase-server</include>
<include>org.apache.hbase:hbase-annotations</include>
<include>org.apache.htrace:htrace-core</include>
<include>com.yammer.metrics:metrics-core</include>
<include>com.google.guava:guava</include>
<include>commons-lang:commons-lang</include>
<include>com.google.protobuf:protobuf-java</include>
</includes>
</artifactSet>
<relocations>
@@ -105,6 +110,22 @@
<pattern> org.apache.htrace.</pattern>
<shadedPattern>org.apache.hudi.org.apache.htrace.</shadedPattern>
</relocation>
<relocation>
<pattern>com.yammer.metrics.</pattern>
<shadedPattern>org.apache.hudi.com.yammer.metrics.</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.common.</pattern>
<shadedPattern>${presto.bundle.bootstrap.shade.prefix}com.google.common.</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.lang.</pattern>
<shadedPattern>${presto.bundle.bootstrap.shade.prefix}org.apache.commons.lang.</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.protobuf.</pattern>
<shadedPattern>${presto.bundle.bootstrap.shade.prefix}com.google.protobuf.</shadedPattern>
</relocation>
</relocations>
<createDependencyReducedPom>false</createDependencyReducedPom>
<filters>
@@ -159,5 +180,42 @@
<artifactId>avro</artifactId>
<scope>compile</scope>
</dependency>
<!--Guava needs to be shaded because HBase 1.2.3 depends on an earlier guava version i.e 12.0.1 and hits runtime
issues with the guava version present in Presto runtime-->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>12.0.1</version>
<scope>${presto.bundle.bootstrap.scope}</scope>
</dependency>
<!--commons-lang needs to be shaded because HBase 1.2.3 needs it at runtime, but Presto runtime does not have this
dependency-->
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.6</version>
<scope>${presto.bundle.bootstrap.scope}</scope>
</dependency>
<!--protobuf needs to be shaded because HBase 1.2.3 needs it at runtime, but Presto runtime does not have this
dependency-->
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.5.0</version>
<scope>${presto.bundle.bootstrap.scope}</scope>
</dependency>
</dependencies>
<profiles>
<profile>
<id>presto-shade-unbundle-bootstrap</id>
<properties>
<presto.bundle.bootstrap.scope>provided</presto.bundle.bootstrap.scope>
<presto.bundle.bootstrap.shade.prefix></presto.bundle.bootstrap.shade.prefix>
</properties>
</profile>
</profiles>
</project>