lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
216 Commits 1 Branch 0 Tags
21ce846f1807ffb94063098e2e4f4e4f9e345f0c
Commit Graph

15 Commits

Author SHA1 Message Date
Vinoth Chandar
0cd186c899 Multi FS Support 
- Reviving PR 191, to make FileSystem creation off actual path
 - Streamline all filesystem access to HoodieTableMetaClient
 - Hadoop Conf from Spark Context serialized & passed to executor code too
 - Pick up env vars prefixed with HOODIE_ENV_ into Configuration object
 - Cleanup usage of FSUtils.getFS, piggybacking off HoodieTableMetaClient.getFS
 - Adding s3a to supported schemes & support escaping "." in env vars
 - Tests use HoodieTestUtils.getDefaultHadoopConf
 2018-01-17 23:34:21 -08:00
Nishith Agarwal
44839b88c6 Removing compaction action type and associated compaction timeline operations, replace with commit action type 2018-01-09 09:56:15 -08:00
Nishith Agarwal
051f600b7f Enable hive sync even if there is no compaction commit 2017-11-30 18:22:58 -08:00
Vinoth Chandar
e45679f5e2 Reformatting code per Google Code Style all over 2017-11-12 23:19:02 -08:00
Nishith Agarwal
c7d63a7622 1) Separated rollback as a table operation 2) Implement rollback for MOR 2017-10-12 07:36:46 -07:00
Nishith Agarwal
e5d9b818bc Sync Tool registers 2 tables, RO and RT Tables 2017-06-28 15:41:36 -07:00
Vinoth Chandar
c00f1a9ed9 Refactoring HoodieTableFileSystemView using FileGroups/FileSlices 
- Merged all filter* and get* methods
 - new constructor takes filestatus[]
 - All existing tests pass
 - FileGroup is all files that belong to a fileID within a partition
 - FileSlice is a generation of data and log files, starting at a base commit
 2017-06-22 17:16:13 -07:00
gekath
db7311f85e Writes relative paths to .commit files instead of absolute paths 
Clean up code

Removed commented out code

Fixed merge conflict with master
 2017-06-16 12:51:19 -07:00
Prasanna Rajaperumal
4b26be9f61 Fixes to RealtimeInputFormat and RealtimeRecordReader and update documentation for HiveSyncTool 2017-06-15 18:21:07 -07:00
Prasanna Rajaperumal
db6150c5ef Refactor hoodie-hive 2017-06-09 13:06:33 -07:00
Vinoth Chandar
82b211d2e6 Rebase with generic partition support 2017-04-03 21:27:49 -07:00
Vinoth Chandar
542d622e49 Adding HiveSyncTool to sync hoodie dataset schema/partitions to Hive 
- Designed to be run by your workflow manager after hoodie upsert
 - Assumes jdbc connectivity via HiveServer2, which should work with all major distros
 2017-04-03 21:27:49 -07:00
Prasanna Rajaperumal
b6d8e857f0 Fix javadoc issues. Prelude to pushing libs to maven central 2016-12-21 17:34:29 -08:00
Prasanna Rajaperumal
5d44ae3dbd Make hoodie run on travis-ci 
Test logs > 4MB, which is a limit for travis-ci. Reducing the logs by setting appropriate log levels for tests
Add sudo: required on travis.yml to get more memory for running the tests. (https://github.com/travis-ci/travis-ci/issues/5926)
Fixed requirement that fsclient.lastDataFileForDataset always returns files in order
 2016-12-20 19:26:48 -08:00
Prasanna Rajaperumal
388457b6b2 Add hoodie-hive module to support hive registration of hoodie datasets 2016-12-19 23:05:39 -08:00