lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
188 Commits 1 Branch 0 Tags
6a3c94aaa30cb00675ff416d0d33ba3a1e868ccc
Commit Graph

13 Commits

Author SHA1 Message Date
Nishith Agarwal
19c22b231e 1. Use HoodieLogFormat to archive commits and other actions 2. Introduced avro schema for commits and compactions and an avro wrapper schema 2017-07-26 14:27:44 -07:00
Nishith Agarwal
e5d9b818bc Sync Tool registers 2 tables, RO and RT Tables 2017-06-28 15:41:36 -07:00
Vinoth Chandar
c00f1a9ed9 Refactoring HoodieTableFileSystemView using FileGroups/FileSlices 
- Merged all filter* and get* methods
 - new constructor takes filestatus[]
 - All existing tests pass
 - FileGroup is all files that belong to a fileID within a partition
 - FileSlice is a generation of data and log files, starting at a base commit
 2017-06-22 17:16:13 -07:00
gekath
db7311f85e Writes relative paths to .commit files instead of absolute paths 
Clean up code

Removed commented out code

Fixed merge conflict with master
 2017-06-16 12:51:19 -07:00
Prasanna Rajaperumal
4b26be9f61 Fixes to RealtimeInputFormat and RealtimeRecordReader and update documentation for HiveSyncTool 2017-06-15 18:21:07 -07:00
Prasanna Rajaperumal
db6150c5ef Refactor hoodie-hive 2017-06-09 13:06:33 -07:00
Prasanna Rajaperumal
240c91241b Implement HoodieLogFormat replacing Avro as the default log format 2017-05-23 08:35:11 -07:00
Vinoth Chandar
82b211d2e6 Rebase with generic partition support 2017-04-03 21:27:49 -07:00
Vinoth Chandar
542d622e49 Adding HiveSyncTool to sync hoodie dataset schema/partitions to Hive 
- Designed to be run by your workflow manager after hoodie upsert
 - Assumes jdbc connectivity via HiveServer2, which should work with all major distros
 2017-04-03 21:27:49 -07:00
Prasanna Rajaperumal
48fbb0f425 Implement reliable log file management for Merge on read, which is fault tolerant and allows random block level access on avro file 2017-02-21 16:23:53 -08:00
Prasanna Rajaperumal
b6d8e857f0 Fix javadoc issues. Prelude to pushing libs to maven central 2016-12-21 17:34:29 -08:00
Prasanna Rajaperumal
5d44ae3dbd Make hoodie run on travis-ci 
Test logs > 4MB, which is a limit for travis-ci. Reducing the logs by setting appropriate log levels for tests
Add sudo: required on travis.yml to get more memory for running the tests. (https://github.com/travis-ci/travis-ci/issues/5926)
Fixed requirement that fsclient.lastDataFileForDataset always returns files in order
 2016-12-20 19:26:48 -08:00
Prasanna Rajaperumal
388457b6b2 Add hoodie-hive module to support hive registration of hoodie datasets 2016-12-19 23:05:39 -08:00