Prasanna Rajaperumal
8ee777a9bb
Refactor hoodie-common and create right abstractions for Hoodie Storage V2.0
...
The following is the gist of changes done
- All low-level operation of creating a commit code was in HoodieClient which made it hard to share code if there was a compaction commit.
- HoodieTableMetadata contained a mix of metadata and filtering files. (Also few operations required FileSystem to be passed in because those were called from TaskExecutors and others had FileSystem as a global variable). Since merge-on-read requires a lot of that code, but will have to change slightly on how it operates on the metadata and how it filters the files. The two set of operation are split into HoodieTableMetaClient and TableFileSystemView.
- Everything (active commits, archived commits, cleaner log, save point log and in future delta and compaction commits) in HoodieTableMetaClient is a HoodieTimeline. Timeline is a series of instants, which has an in-built concept of inflight and completed commit markers.
- A timeline can be queries for ranges, contains and also use to create new datapoint (create a new commit etc). Commit (and all the above metadata) creation/deletion is streamlined in a timeline
- Multiple timelines can be merged into a single timeline, giving us an audit timeline to whatever happened in a hoodie dataset. This also helps with #55 .
- Move to java 8 and introduce java 8 succinct syntax in refactored code
2017-02-21 16:23:53 -08:00
Prasanna Rajaperumal
283269e57f
[maven-release-plugin] prepare for next development iteration
2017-02-20 16:52:25 -08:00
Prasanna Rajaperumal
d5a5f2ddff
[maven-release-plugin] prepare release hoodie-0.3.0
2017-02-20 16:52:04 -08:00
Prasanna Rajaperumal
0e234ac0ef
Moving to Spark 2.1.0
2017-02-20 16:47:52 -08:00
Prasanna Rajaperumal
be1dd9444f
[maven-release-plugin] prepare for next development iteration
2017-02-20 16:09:05 -08:00
Prasanna Rajaperumal
47583e280f
[maven-release-plugin] prepare release hoodie-0.2.14
2017-02-20 16:08:45 -08:00
Prasanna Rajaperumal
2d49711cce
Changing the current development version to 0.2.14-SNAPSHOT
2017-02-20 16:01:24 -08:00
Prasanna Rajaperumal
4a47d26818
Fixing a javadoc lint issue
2017-02-20 15:57:58 -08:00
Prasanna Rajaperumal
cc58a4c3e0
[maven-release-plugin] prepare for next development iteration
2017-02-20 15:49:45 -08:00
Prasanna Rajaperumal
dd03038254
[maven-release-plugin] prepare release hoodie-0.2.13
2017-02-20 15:49:20 -08:00
Prasanna Rajaperumal
7178cb5a3f
Fixing a javadoc lint issue
2017-02-20 15:41:32 -08:00
Prasanna Rajaperumal
57a0b7a781
[maven-release-plugin] prepare for next development iteration
2017-02-20 15:35:19 -08:00
Prasanna Rajaperumal
9828bd8019
[maven-release-plugin] prepare release hoodie-0.2.12
2017-02-20 15:35:03 -08:00
Prasanna Rajaperumal
8f12163166
[maven-release-plugin] prepare for next development iteration
2017-02-20 15:00:35 -08:00
Prasanna Rajaperumal
6e6f6efb94
[maven-release-plugin] prepare release hoodie-0.2.11
2017-02-20 15:00:16 -08:00
Vinoth Chandar
33a85900f8
Adding admin guide, guide for sql queries and incr processing
2017-02-19 20:33:21 -08:00
Vinoth Chandar
dcc15d5d6f
Adding docs for running sql queries on hoodie datasets
2017-02-19 20:33:21 -08:00
vinoth chandar
66e272e9eb
Docs for performance section ( #80 )
...
* Adding performance section
* minor edit to perf section
2017-02-17 18:30:56 -08:00
vinoth chandar
c7a8e15c78
Docs for impl & comparison ( #79 )
...
* Initial version of comparison, implementation
* Finished doc for comparison to other systems
2017-02-17 08:25:17 -08:00
Prasanna Rajaperumal
a725382549
Add Configurations to Documentation
2017-02-06 14:12:18 -08:00
vinoth chandar
186d70713f
Adding documentation around Hoodie concepts ( #71 )
2017-01-29 15:22:57 -08:00
vinoth chandar
8e72ed69b4
Split insert() into insert() & bulkInsert() ( #69 )
...
- Behaviour change for existing insert() users
- Made the current insert() implementation, as something to use for bulkInsert()
- Normal inserts now share a lot of code with upsert, which provides benefits like small file handling
- Refactored/Cleaned up code in HoodieWriteClient for reuse
- Added a unit test, switching few tests to call bulkInsert() and few to call insert()
2017-01-27 10:51:00 -08:00
vinoth chandar
54409b07ea
Implement HoodieROTablePathFilter for use with Spark 2.0 ( #66 )
...
- Unit tested
- Tested with few queries on Spark 2.x at Uber
2017-01-26 11:13:33 -08:00
prazanna
0004ae62ad
Merge pull request #61 from zqureshi/format-comment-hoodiekey
...
Format param comment in HoodieKey
2017-01-16 19:38:53 -08:00
Zeeshan Qureshi
8a39730273
Format param comment in HoodieKey
2017-01-16 22:15:14 -05:00
Prasanna Rajaperumal
693d751506
[maven-release-plugin] prepare for next development iteration
2017-01-10 22:27:35 -08:00
Prasanna Rajaperumal
e9866bb4d9
[maven-release-plugin] prepare release hoodie-0.2.10
2017-01-10 22:27:28 -08:00
prazanna
e4e3395f3e
Make commit a public method. Introduce a auto-commit config. Relates issue https://github.com/uber/hoodie/issues/58 ( #60 )
2017-01-10 22:14:40 -08:00
vinoth chandar
9d09a58a18
Merge pull request #56 from vinothchandar/master
...
Shorten README and point to site
2017-01-09 11:32:35 -08:00
Vinoth Chandar
40a63fcab4
Shorten README and point to site
2017-01-09 11:30:46 -08:00
vinoth chandar
1559a3826f
Minor fixes to use_cases.md
2017-01-06 00:06:15 -08:00
vinoth chandar
534c898c6b
Merge pull request #34 from vinothchandar/master
...
Fixing the eyesore red font color over sidebar for docs
2017-01-05 23:57:48 -08:00
Vinoth Chandar
5a7d408c3c
Fixing the eyesore red font color over sidebar for docs
2017-01-05 23:55:53 -08:00
Prasanna Rajaperumal
1ced46ab3e
[maven-release-plugin] prepare for next development iteration
2017-01-05 20:04:35 -08:00
Prasanna Rajaperumal
e9f0d4d0bf
[maven-release-plugin] prepare release hoodie-0.2.9
2017-01-05 20:04:28 -08:00
Prasanna Rajaperumal
1d72b164c8
Javadoc generation fails because of invalid html tags
2017-01-05 19:56:41 -08:00
Prasanna Rajaperumal
7171ea6909
[maven-release-plugin] prepare for next development iteration
2017-01-05 19:43:31 -08:00
Prasanna Rajaperumal
c1f2d1e456
[maven-release-plugin] prepare release hoodie-0.2.8
2017-01-05 19:43:25 -08:00
prazanna
9071220a30
Merge pull request #33 from vinothchandar/master
...
Changing Update record failure semantics to be consistent with inserts
2017-01-05 19:28:59 -08:00
Vinoth Chandar
b2c958519e
Changing Update record failure semantics to be consistent with inserts
...
- Don't skip, but writes the same old record again now
- Marks the correspoinding HoodieRecord as failure to be handed back to the client
2017-01-05 17:26:47 -08:00
vinoth chandar
93de14f2cf
Merge pull request #32 from prazanna/master
...
Make sure to fail with HoodieUpsertException if HoodieUpdateHandle is…
2017-01-05 16:59:31 -08:00
Prasanna Rajaperumal
6dd603352e
Make sure to fail with HoodieUpsertException if HoodieUpdateHandle is not properly initialized
2017-01-05 16:47:27 -08:00
vinoth chandar
ee9be707e5
Merge pull request #26 from prazanna/master
...
Build hoodie-hadoop-mr as a fat jar - include hoodie common
2017-01-05 16:45:08 -08:00
vinoth chandar
64858239d1
Update use_cases.md
2017-01-04 23:58:09 -08:00
vinoth chandar
5d5e346c61
Merge pull request #28 from vinothchandar/add-docs
...
Adding Documentation for Getting Started Section
2017-01-04 20:54:39 -08:00
Vinoth Chandar
958f7ceda6
Adding Documentation for Getting Started Section
...
- Overview, Use Cases, Powered By are very detailed
- Cleaned up QuickStart
- Redistribute the content from README to correct pages to be improved upon
- Switch to blue theme
2017-01-04 20:50:44 -08:00
Prasanna Rajaperumal
2475610246
Build hoodie-hadoop-mr as a fat jar - include hoodie common
2017-01-04 14:52:38 -08:00
Prasanna Rajaperumal
362a84377b
Merge remote-tracking branch 'uber/master'
2017-01-04 14:42:05 -08:00
prazanna
83d4163c75
Merge pull request #25 from vinothchandar/master
...
Annotate HoodieInputFormat with UseFileSplitsFromInputFormat
2017-01-04 14:12:16 -08:00
Vinoth Chandar
dd8638c2cc
Annotate HoodieInputFormat with UseFileSplitsFromInputFormat
...
- Tested with corresponding Presto changes (pending PR)
2017-01-04 11:19:25 -08:00
