Nishith Agarwal
3da063f83b
Adding ability for inserts to be written to log files
2018-06-11 14:08:59 -07:00
Balaji Varadarajan
dfc0c61eb7
Support union mode in HoodieRealtimeRecordReader for pure insert workloads
...
Also Replace BufferedIteratorPayload abstraction with function passing
2018-05-10 17:39:56 -07:00
Nishith Agarwal
93f345a032
Minor fixes for MergeOnRead MVP release readiness
2018-05-09 07:23:58 -07:00
Nishith Agarwal
04655e9e85
Adding metrics for MOR and COW
2018-04-26 09:32:45 -07:00
Sunil Ramaiah
4d1fba24c9
Fix for updating duplicate records in same/different files in same parition
2018-04-25 13:09:15 -07:00
Nishith Agarwal
720e42f52a
Parallelized read-write operations in Hoodie Merge phase
2018-04-12 11:46:42 -07:00
Balaji Varadarajan
788e4f2d2e
CodeStyle formatting to conform to basic Checkstyle rules.
...
The code-style rules follow google style with some changes:
1. Increase line length from 100 to 120
2. Disable JavaDoc related checkstyles as this needs more manual work.
Both source and test code are checked for code-style
2018-03-30 11:09:40 -07:00
Nishith Agarwal
987f5d6b96
Making ExternalSpillableMap generic for any datatype
...
- Introduced concept of converters to be able to serde generic datatype for SpillableMap
- Fixed/Added configs to Hoodie Configs
- Changed HoodieMergeHandle to start using SpillableMap
2018-03-28 07:56:07 -07:00
Jian Xu
d3df32fa03
Add back UseTempFolder changes in HoodieMergeHandle
2018-03-15 17:11:15 -07:00
Omkar Joshi
c5b4cb1b75
Spawning parallel writer thread to separate reading records from spark and writing records to parquet file
2018-03-15 16:58:14 -07:00
Nishith Agarwal
6fec9655a8
Added support for Disk Spillable Compaction to prevent OOM issues
2018-02-26 16:00:35 -08:00
Nishith Agarwal
7076c2e9f0
refactor classes to accept Map passed by RealtimeCompactor to avoid multiple map creations in HoodieMergeHandle
2018-02-07 11:16:01 -08:00
Jian Xu
15e669c60c
Incorporating code review feedback for finalizeWrite for COW #4
2018-02-02 11:38:25 -08:00
Jian Xu
363e35bb0f
Add finalizeWrite support for HoodieMergeHandle
2018-02-02 11:38:25 -08:00
Nishith Agarwal
e10100fe32
Reducing list status calls from listing logfile versions, some associated refactoring
2018-01-29 08:26:39 -08:00
Vinoth Chandar
e45679f5e2
Reformatting code per Google Code Style all over
2017-11-12 23:19:02 -08:00
Kaushik Devarajaiah
c98ee057fc
capture record metadata before deflating for record counting
2017-10-02 10:46:06 -07:00
Vinoth Chandar
c00f1a9ed9
Refactoring HoodieTableFileSystemView using FileGroups/FileSlices
...
- Merged all filter* and get* methods
- new constructor takes filestatus[]
- All existing tests pass
- FileGroup is all files that belong to a fileID within a partition
- FileSlice is a generation of data and log files, starting at a base commit
2017-06-22 17:16:13 -07:00
Vinoth Chandar
23e7badd8a
Rename IO Handles & introduce stub for BucketedIndex
...
- UpdateHandle -> MergeHandle, InsertHandle -> CreateHandle
- Also bunch of code cleanup in different places
2017-06-22 17:16:13 -07:00
