lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity

[HUDI-1897] Deltastreamer source for AWS S3 (#3433)

Browse Source 
- Added two sources for two stage pipeline. a. S3EventsSource that fetches events from SQS and ingests to a meta hoodie table. b. S3EventsHoodieIncrSource reads S3 events from this meta hoodie table, fetches actual objects from S3 and ingests to sink hoodie table. 
- Added selectors to assist in S3EventsSource. 

Co-authored-by: Satish M <84978833+satishmittal1111@users.noreply.github.com>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
This commit is contained in:
Sagar Sumit
2021-08-14 17:55:10 +05:30
committed by GitHub
parent 9056c68744
commit 5cc96e85c1
12 changed files with 1348 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
pom.xml
View File

@@ -153,6 +153,7 @@
<shadeSources>true</shadeSources>
<zk-curator.version>2.7.1</zk-curator.version>
<antlr.version>4.7</antlr.version>
<aws.sdk.version>1.12.22</aws.sdk.version>
</properties>
<scm>