Nishith Agarwal
6a3c94aaa3
suppressing logs (under 4MB) for jenkins
2017-08-15 16:30:51 -07:00
Nishith Agarwal
5ee4ac40ae
Use CompletedFileSystemView instead of CompactedView considering deltacommits
2017-08-07 12:26:42 -07:00
Vinoth Chandar
45dd8980c3
Temporary fix for build break after rebase
2017-08-04 17:36:39 -07:00
Vinoth Chandar
86209640f7
Adding range based pruning to bloom index
...
- keys compared lexicographically using String::compareTo
- Range metadata additionally written into parquet file footers
- Trim fat & few optimizations to speed up indexing
- Add param to control whether input shall be cached, to speed up lookup
- Add param to turn on/off range pruning
- Auto compute of parallelism now simply factors in amount of comparisons done
- More accurate parallelism computation when range pruning is on
- tests added & hardened, docs updated
2017-08-04 13:22:13 -07:00
Nishith Agarwal
0b26b60a5c
fix for cleaning log files(mor)
2017-08-02 11:54:42 -07:00
Nishith Agarwal
19c22b231e
1. Use HoodieLogFormat to archive commits and other actions 2. Introduced avro schema for commits and compactions and an avro wrapper schema
2017-07-26 14:27:44 -07:00
Nishith Agarwal
616c9a68c3
Enabled deletes in merge_on_read
2017-07-26 13:37:27 -07:00
Vinoth Chandar
cf1dde0323
Add recent talks/presentations to documentation
2017-07-08 22:47:15 -07:00
Vinoth Chandar
e8b3ddd7cb
Add note on community engagement to committership guidelines
2017-07-08 22:47:15 -07:00
Prasanna Rajaperumal
7d3963b4ab
Pushing master to 0.4.0 as we continue to make minor releases over 0.3.8 (MVP for MOR)
2017-06-30 11:41:23 -07:00
Nishith Agarwal
3eba812a1b
[maven-release-plugin] prepare for next development iteration
2017-06-30 11:17:07 -07:00
Nishith Agarwal
06d44daea3
[maven-release-plugin] prepare release hoodie-0.3.9
2017-06-30 11:16:58 -07:00
Nishith Agarwal
348250d960
Using FsUtils instead of Files API to extract file extension
2017-06-29 19:26:31 -07:00
Nishith Agarwal
e5d9b818bc
Sync Tool registers 2 tables, RO and RT Tables
2017-06-28 15:41:36 -07:00
Prasanna Rajaperumal
5cc071f74e
Savepoint should not create a hole in the commit timeline
2017-06-27 16:36:09 -07:00
Jian Xu
29b906b763
Fix TimestampBasedKeyGenerator when DATE_STRING is used for partitionpath.field
2017-06-27 13:02:06 -07:00
Vinoth Chandar
754ab88a2d
Introduce ReadOptimizedView & RealtimeView out of TableFileSystemView
...
- Usage now marks code as clearly using either RO or RT views, for future evolution
- Tests on all of FileGroups and FileSlices
2017-06-22 17:16:13 -07:00
Vinoth Chandar
c00f1a9ed9
Refactoring HoodieTableFileSystemView using FileGroups/FileSlices
...
- Merged all filter* and get* methods
- new constructor takes filestatus[]
- All existing tests pass
- FileGroup is all files that belong to a fileID within a partition
- FileSlice is a generation of data and log files, starting at a base commit
2017-06-22 17:16:13 -07:00
Vinoth Chandar
23e7badd8a
Rename IO Handles & introduce stub for BucketedIndex
...
- UpdateHandle -> MergeHandle, InsertHandle -> CreateHandle
- Also bunch of code cleanup in different places
2017-06-22 17:16:13 -07:00
prazanna
b0a2a23372
Adding Nishith to Contributors list
2017-06-20 15:48:43 -07:00
prazanna
649475c5cb
Adding Kaushik to contributors list
2017-06-20 15:47:05 -07:00
Kaushik Devarajaiah
3aa8083913
Correct clean bug that causes clean failure when partitionPaths are empty
2017-06-20 15:45:32 -07:00
prazanna
7ef76a4de0
Adding Kathy Ge to the contributors list
2017-06-16 12:52:54 -07:00
gekath
52c507f83e
Writes relative paths to .commit files
...
Handle case where path is read in as null from commit file
Merged with updated release
2017-06-16 12:51:19 -07:00
gekath
db7311f85e
Writes relative paths to .commit files instead of absolute paths
...
Clean up code
Removed commented out code
Fixed merge conflict with master
2017-06-16 12:51:19 -07:00
Prasanna Rajaperumal
0ed3fac5e3
[maven-release-plugin] prepare for next development iteration
2017-06-16 11:03:17 -07:00
Prasanna Rajaperumal
45732e440c
[maven-release-plugin] prepare release hoodie-0.3.8
2017-06-16 10:59:58 -07:00
Prasanna Rajaperumal
e44f9b889b
Added CHANGELOG.md and updated community contributions guideline
2017-06-16 10:48:37 -07:00
Prasanna Rajaperumal
4b26be9f61
Fixes to RealtimeInputFormat and RealtimeRecordReader and update documentation for HiveSyncTool
2017-06-15 18:21:07 -07:00
Kaushik Devarajaiah
521555c576
Parallelize file version deletes during clean and related tests
2017-06-15 18:20:42 -07:00
Prasanna Rajaperumal
dda28c0b4b
Rollback inflight commits as well when rolling back to savepoint
2017-06-14 11:03:27 -07:00
Prasanna Rajaperumal
db6150c5ef
Refactor hoodie-hive
2017-06-09 13:06:33 -07:00
Danny Chen
c192dd60b4
Change from deprecated closeQuietly to try with resources
2017-06-05 19:11:53 -07:00
n3nash
36fa6f66ac
Merge pull request #180 from n3nash/record-reader-with-compaction
...
updated HoodieRealtimeRecordReader to use HoodieCompactedLogRecordSca…
2017-06-02 11:49:32 -07:00
Nishith Agarwal
ba050973e3
updated HoodieRealtimeRecordReader to use HoodieCompactedLogRecordScanner, added test for recordreader
2017-06-02 11:33:59 -07:00
Prasanna Rajaperumal
933cc8071f
[maven-release-plugin] prepare for next development iteration
2017-05-24 14:02:50 -07:00
Prasanna Rajaperumal
bebae06b5b
[maven-release-plugin] prepare release hoodie-0.3.7
2017-05-24 14:02:41 -07:00
Prasanna Rajaperumal
bae98efeee
Delete other instant files (.clean) as well during commit archival
2017-05-24 13:51:49 -07:00
prazanna
e1d13f2bc8
https://repository.cloudera.com/artifactory/repo/ has been changed to https://repository.cloudera.com/artifactory/public/
2017-05-23 12:05:01 -07:00
Prasanna Rajaperumal
240c91241b
Implement HoodieLogFormat replacing Avro as the default log format
2017-05-23 08:35:11 -07:00
Nishith Agarwal
3c984447da
view scheme added
2017-05-22 12:27:40 -07:00
Prasanna Rajaperumal
70dd7a25ea
Clean should not create a .inflight file
2017-05-22 10:48:35 -07:00
Vinoth Chandar
7014670795
Update contributor list
2017-05-18 10:48:42 -07:00
Zeeshan Qureshi
43a55b09fd
Add GCS to supported filesystems
2017-05-18 10:30:34 -07:00
prazanna
21e334592f
Update java version to 8 in travis.yml
2017-05-17 13:43:11 -07:00
vinoth chandar
1b0a027942
Update community.md with committership guidelines
2017-05-04 17:25:54 -07:00
Vinoth Chandar
b4e787ce1d
Update docs
2017-05-01 21:48:27 -07:00
Vinoth Chandar
da17c5c607
Introduce getCommitsAndCompactionsTimeline() explicitly & adjust usage across code base
2017-05-01 21:48:27 -07:00
Vinoth Chandar
bae0528013
Cleanup calls to HoodieTimeline.compareTimeStamps
2017-05-01 21:48:27 -07:00
Vinoth Chandar
7b1446548f
Initial impl of HoodieRealtimeInputFormat
...
- Works end-end for flat schemas
- Schema evolution & hardening remains
- HoodieClientExample can now write mor tables as well
2017-05-01 21:48:27 -07:00
