lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
473 Commits 1 Branch 0 Tags
11c4121f739d1d00a4ec66b4e243e47602d6ffb4
Commit Graph

68 Commits

Author SHA1 Message Date
Nishith Agarwal
129e433641 - Ugrading to Hive 2.x 
- Eliminating in-memory deltaRecordsMap
- Use writerSchema to generate generic record needed by custom payloads
- changes to make tests work with hive 2.x
 2019-06-13 12:46:14 -07:00
Balaji Varadarajan
479908fd20 HUDI-125 : Change License for all source files and update RAT configurations 2019-06-09 11:41:55 -07:00
Balaji Varadarajan
30b0f2636f Changes related to Licensing work 
1. Go through dependencies list one round to ensure compliance. Generated current NOTICE list in all submodules (other apache projects like flink does this).
   To be on conservative side regarding licensing, NOTICE.txt lists all dependencies including transitive. Pending Compliance questions reported in https://issues.apache.org/jira/browse/LEGAL-461
2. Automate generating NOTICE.txt files to allow future package compliance issues be identified early as part of code-review process.
3. Added NOTICE.txt and LICENSE.txt to all HUDI jars
 2019-06-07 17:58:57 -07:00
vinothchandar
66c0b81b49 [maven-release-plugin] prepare for next development iteration 2019-05-28 19:17:26 -07:00
vinothchandar
227785c022 [maven-release-plugin] prepare release hoodie-0.4.7 2019-05-28 19:17:15 -07:00
vinothchandar
446f99aa0f [maven-release-plugin] prepare for next development iteration 2019-05-14 07:29:22 -07:00
vinothchandar
cc38abecc8 [maven-release-plugin] prepare release hoodie-0.4.6 2019-05-14 07:29:11 -07:00
Balaji Varadarajan
adc8cac743 Fix hive sync (libfb version mismatch) and deltastreamer issue (missing cmdline argument) in demo 2019-03-13 16:14:32 -07:00
Vinoth Chandar
363df2c12e Upgrade various jar, gem versions for maintenance 2019-03-01 10:14:00 -08:00
vinothchandar
687395e40f [maven-release-plugin] prepare for next development iteration 2019-02-27 07:16:27 -08:00
vinothchandar
bbf40ef987 [maven-release-plugin] prepare release hoodie-0.4.5 2019-02-27 07:16:15 -08:00
Balaji Varadarajan
3a0044216c New Features in DeltaStreamer : 
(1) Apply transformation when using delta-streamer to ingest data.
 (2) Add Hudi Incremental Source for Delta Streamer
 (3) Allow delta-streamer config-property to be passed as command-line
 (4) Add Hive Integration to Delta-Streamer and address Review comments
 (5) Ensure MultiPartKeysValueExtractor  handle hive style partition description
 (6) Reuse same spark session on both source and transformer
 (7) Support extracting partition fields from _hoodie_partition_path for HoodieIncrSource
 (8) Reuse Binary Avro coders
 (9) Add push down filter for Incremental source
 (10) Add Hoodie DeltaStreamer metrics to track total time taken
 2019-02-11 18:22:05 -08:00
vinothchandar
7ba842c0fe [maven-release-plugin] prepare for next development iteration 2018-09-28 11:27:00 +05:30
vinothchandar
5847b61f44 [maven-release-plugin] prepare release hoodie-0.4.4 2018-09-28 11:26:15 +05:30
Vinoth Chandar
bd5af89f12 [maven-release-plugin] rollback the release of hoodie-0.4.4 2018-09-13 15:01:53 +05:30
Vinoth Chandar
d1cc864a43 [maven-release-plugin] prepare for next development iteration 2018-09-12 23:59:47 +05:30
Vinoth Chandar
b748bc836d [maven-release-plugin] prepare release hoodie-0.4.4 2018-09-12 23:59:34 +05:30
Vinoth Chandar
a5359662be Moving depedencies off cdh to apache + Hive2 support 
- Tests redone in the process
 - Main changes are to RealtimeRecordReader and how it treats maps/arrays
 - Make hive sync work with Hive 1/2 and CDH environments
 - Fixes to make corner cases for Hive queries
 - Spark Hive integration - Working version across Apache and CDH versions
 - Known Issue - https://github.com/uber/hudi/issues/439
 2018-09-11 11:03:30 +05:30
Vinoth Chandar
89cd6b0726 [maven-release-plugin] prepare for next development iteration 2018-08-22 21:30:05 -07:00
Vinoth Chandar
8d305c5a86 [maven-release-plugin] prepare release hoodie-0.4.3 2018-08-22 21:29:53 -07:00
Vinoth Chandar
34827d50e1 [maven-release-plugin] prepare for next development iteration 2018-06-11 08:59:13 -07:00
Vinoth Chandar
43ef385730 [maven-release-plugin] prepare release hoodie-0.4.2 2018-06-11 08:59:02 -07:00
Vinoth Chandar
73534d467f [maven-release-plugin] prepare for next development iteration 2018-03-07 21:04:10 -08:00
Vinoth Chandar
f2e5c6f9f8 [maven-release-plugin] prepare release hoodie-0.4.1 2018-03-07 21:04:00 -08:00
Vinoth Chandar
e45679f5e2 Reformatting code per Google Code Style all over 2017-11-12 23:19:02 -08:00
Vinoth Chandar
e1fe3ab937 [maven-release-plugin] prepare for next development iteration 2017-10-02 22:42:54 -07:00
Vinoth Chandar
50139fe904 [maven-release-plugin] prepare release hoodie-0.4.0 2017-10-02 22:42:32 -07:00
Prasanna Rajaperumal
7d3963b4ab Pushing master to 0.4.0 as we continue to make minor releases over 0.3.8 (MVP for MOR) 2017-06-30 11:41:23 -07:00
Nishith Agarwal
3eba812a1b [maven-release-plugin] prepare for next development iteration 2017-06-30 11:17:07 -07:00
Nishith Agarwal
06d44daea3 [maven-release-plugin] prepare release hoodie-0.3.9 2017-06-30 11:16:58 -07:00
Prasanna Rajaperumal
0ed3fac5e3 [maven-release-plugin] prepare for next development iteration 2017-06-16 11:03:17 -07:00
Prasanna Rajaperumal
45732e440c [maven-release-plugin] prepare release hoodie-0.3.8 2017-06-16 10:59:58 -07:00
Prasanna Rajaperumal
4b26be9f61 Fixes to RealtimeInputFormat and RealtimeRecordReader and update documentation for HiveSyncTool 2017-06-15 18:21:07 -07:00
Prasanna Rajaperumal
db6150c5ef Refactor hoodie-hive 2017-06-09 13:06:33 -07:00
Prasanna Rajaperumal
933cc8071f [maven-release-plugin] prepare for next development iteration 2017-05-24 14:02:50 -07:00
Prasanna Rajaperumal
bebae06b5b [maven-release-plugin] prepare release hoodie-0.3.7 2017-05-24 14:02:41 -07:00
Prasanna Rajaperumal
c3258039f0 [maven-release-plugin] prepare for next development iteration 2017-04-27 11:00:56 -07:00
Prasanna Rajaperumal
de1bdad756 [maven-release-plugin] prepare release hoodie-0.3.6 2017-04-27 11:00:45 -07:00
Vinoth Chandar
542d622e49 Adding HiveSyncTool to sync hoodie dataset schema/partitions to Hive 
- Designed to be run by your workflow manager after hoodie upsert
 - Assumes jdbc connectivity via HiveServer2, which should work with all major distros
 2017-04-03 21:27:49 -07:00
Prasanna Rajaperumal
aee136777b Fixes needed to run merge-on-read testing on production scale data 2017-04-02 22:25:47 -07:00
Prasanna Rajaperumal
57ab7a2405 [maven-release-plugin] prepare for next development iteration 2017-03-31 14:58:55 -07:00
Prasanna Rajaperumal
803c635098 [maven-release-plugin] prepare release hoodie-0.3.5 2017-03-31 14:58:46 -07:00
Prasanna Rajaperumal
f4bb44c1b1 Update snapshot version to 0.3.5-SNAPSHOT 2017-03-31 14:54:54 -07:00
Prasanna Rajaperumal
48fbb0f425 Implement reliable log file management for Merge on read, which is fault tolerant and allows random block level access on avro file 2017-02-21 16:23:53 -08:00
Prasanna Rajaperumal
283269e57f [maven-release-plugin] prepare for next development iteration 2017-02-20 16:52:25 -08:00
Prasanna Rajaperumal
d5a5f2ddff [maven-release-plugin] prepare release hoodie-0.3.0 2017-02-20 16:52:04 -08:00
Prasanna Rajaperumal
be1dd9444f [maven-release-plugin] prepare for next development iteration 2017-02-20 16:09:05 -08:00
Prasanna Rajaperumal
47583e280f [maven-release-plugin] prepare release hoodie-0.2.14 2017-02-20 16:08:45 -08:00
Prasanna Rajaperumal
2d49711cce Changing the current development version to 0.2.14-SNAPSHOT 2017-02-20 16:01:24 -08:00
Prasanna Rajaperumal
cc58a4c3e0 [maven-release-plugin] prepare for next development iteration 2017-02-20 15:49:45 -08:00