lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity

[HUDI-1897] Deltastreamer source for AWS S3 (#3433)

Browse Source 
- Added two sources for two stage pipeline. a. S3EventsSource that fetches events from SQS and ingests to a meta hoodie table. b. S3EventsHoodieIncrSource reads S3 events from this meta hoodie table, fetches actual objects from S3 and ingests to sink hoodie table. 
- Added selectors to assist in S3EventsSource. 

Co-authored-by: Satish M <84978833+satishmittal1111@users.noreply.github.com>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
This commit is contained in:
Sagar Sumit
2021-08-14 17:55:10 +05:30
committed by GitHub
parent 9056c68744
commit 5cc96e85c1
12 changed files with 1348 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
hudi-utilities/pom.xml
View File

@@ -402,6 +402,14 @@
<scope>test</scope>
</dependency>
<!-- AWS Services -->
<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-sqs -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-sqs</artifactId>
<version>${aws.sdk.version}</version>
</dependency>
<!-- Hive - Test -->
<dependency>
<groupId>${hive.groupid}</groupId>