lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
839 Commits 1 Branch 0 Tags
f1d7bb381d4a370beeedb45132b24c2cac00aabf
Commit Graph

839 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
lamber-ken
8a3a50309b [MINOR] Fix missing @Override annotation on BufferedRandomAccessFile method (#1236) 2020-01-16 11:14:39 -08:00
Y Ethan Guo
b39458b008 [MINOR] Make constant fields final in HoodieTestDataGenerator (#1234) 2020-01-16 12:42:30 +08:00
Scheller
1daba24065 Add GlobalDeleteKeyGenerator 
Adds new GlobalDeleteKeyGenerator for record_key deletes with global indices. Also refactors key generators into their own package.
 2020-01-15 17:01:29 -08:00
Balajee Nagasubramaniam
dd09abb56d [HUDI-335] Improvements to DiskBasedMap used by ExternalSpillableMap, for write and random/sequential read paths, by introducing bufferedRandmomAccessFile 2020-01-15 16:45:45 -08:00
lamber-ken
7aa3ce31e6 [MINOR] Fix redundant judgment statement (#1231) 2020-01-15 16:30:14 -08:00
Sivabalan Narayanan
2248fd9aea Fixing checkstyle issues 2020-01-15 14:21:26 -08:00
Sivabalan Narayanan
2b2f23aa60 Fixing delete util method 2020-01-15 14:21:26 -08:00
Sivabalan Narayanan
87fdb769f0 Adding util methods to assist in adding deletion support to Quick Start 2020-01-15 14:21:26 -08:00
lamber-ken
9b2944a9a2 [MINOR] Refactor unnecessary boxing inside TypedProperties code (#1227) 2020-01-14 19:27:53 -08:00
Mehrotra
2bb0c21a3d Fix conversion of Spark struct type to Avro schema 
cr https://code.amazon.com/reviews/CR-17184364
 2020-01-14 00:27:56 -08:00
lamber-ken
fd8f1c70c0 [MINOR] Reuse random object (#1222) 2020-01-13 18:26:04 -08:00
liujianhui
c1f8acab34 [HUDI-526] fix the HoodieAppendHandle 2020-01-13 10:44:31 -08:00
openopen2
a44c61b813 [HUDI-502] provide a custom time zone definition for TimestampBasedKeyGenerator (#1188) 2020-01-12 15:45:23 -08:00
harveyyue
971c7d41bd [HUDI-322] DeltaSteamer should pick checkpoints off only deltacommits for MOR tables 2020-01-12 15:11:47 -08:00
Udit Mehrotra
ad50008a59 [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types 
- Upgrade Spark to 2.4.4, Parquet to 1.10.1, Avro to 1.8.2
- Remove spark-avro from hudi-spark-bundle. Users need to provide --packages org.apache.spark:spark-avro:2.4.4 when running spark-shell or spark-submit
- Replace com.databricks:spark-avro with org.apache.spark:spark-avro
- Shade avro in hudi-hadoop-mr-bundle to make sure it does not conflict with hive's avro version.
 2020-01-12 15:03:11 -08:00
lamber-ken
d9675c4ec0 [HUDI-522] Use the same version jcommander uniformly (#1214) 2020-01-12 10:48:52 -08:00
lamber-ken
017ee8e661 [MINOR] Fix partition typo (#1209) 2020-01-12 09:35:55 +08:00
lamber-ken
e103165083 [CLEAN] replace utf-8 constant with StandardCharsets.UTF_8 2020-01-10 16:23:29 -08:00
Thinking
b95367d82a [HUDI-469] Fix: HoodieCommitMetadata only show first commit insert rows. 2020-01-10 16:17:11 -08:00
leesf
04afac977d [HUDI-248] CLI doesn't allow rolling back a Delta commit 2020-01-10 16:10:35 -08:00
pratyakshsharma
3c90d252cc [HUDI-114]: added option to overwrite payload implementation in hoodie.properties file 2020-01-09 22:34:40 -08:00
hongdd
5af3dc6aed [HUDI-331]Fix java docs for all public apis in HoodieWriteClient (#1111) 2020-01-09 16:00:53 +08:00
Bhavani Sudha Saktheeswaran
d09eacdc13 [HUDI-25] Optimize HoodieInputformat.listStatus() for faster Hive incremental queries on Hoodie 
Summary:
    - InputPathHandler class classifies  inputPaths into incremental, non incremental and non hoodie paths.
    - Incremental queries leverage HoodieCommitMetadata to get partitions that are affected and only lists those partitions as opposed to listing all partitions
    - listStatus() processes each category separately
 2020-01-08 14:53:05 -08:00
Y Ethan Guo
480fc7869d [HUDI-319] Add a new maven profile to generate unified Javadoc for all Java and Scala classes (#1195) 
* Add javadoc build command in README, links to javadoc plugin and rename profile.
* Make java version configurable in one place.
 2020-01-08 10:38:09 -08:00
Wenning Ding
aba83876e7 Update deprecated HBase API 2020-01-08 10:26:47 -08:00
lamber-ken
9884972a3a [MINOR] Remove old jekyll config file (#1198) 2020-01-07 16:45:19 -08:00
vinoth chandar
9706f659db [HUDI-508] Standardizing on "Table" instead of "Dataset" across code (#1197) 
- Docs were talking about storage types before, cWiki moved to "Table"
 - Most of code already has HoodieTable, HoodieTableMetaClient - correct naming
 - Replacing renaming use of dataset across code/comments
 - Few usages in comments and use of Spark SQL DataSet remain unscathed
 2020-01-07 12:52:32 -08:00
Balaji Varadarajan
8306f749a2 [HUDI-417] Refactor HoodieWriteClient so that commit logic can be shareable by both bootstrap and normal write operations (#1166) 2020-01-06 20:11:48 -08:00
Abhishek Modi
b5df6723a2 [HUDI-464] Use Hive Exec Core for tests (#1125) 2020-01-06 16:32:55 -08:00
lamber-ken
75c3f630d4 [HUDI-405] Remove HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from DataSource 2020-01-06 14:25:38 -08:00
yuehan124
c78092d2d3 [HUDI-501] Execute docker/setup_demo.sh in any directory 2020-01-06 10:26:06 -08:00
Pratyaksh Sharma
8f935e779a [HUDI-406]: added default partition path in TimestampBasedKeyGenerator 2020-01-06 09:38:06 -08:00
hongdd
2d5b79d96f [HUDI-438] Merge duplicated code fragment in HoodieSparkSqlWriter (#1114) 2020-01-06 22:51:22 +08:00
hejinbiao123
b9fab0b933 Revert "[HUDI-455] Redo hudi-client log statements using SLF4J (#1145)" (#1181) 
This reverts commit e637d9ed26.
 2020-01-06 21:13:29 +08:00
lamber-ken
28ccf8c521 [HUDI-484] Fix NPE when reading IncrementalPull.sqltemplate in HiveIncrementalPuller (#1167) 2020-01-04 23:53:47 -08:00
Sivabalan Narayanan
7031445eb3 [HUDI-377] Adding Delete() support to DeltaStreamer (#1073) 
- Provides ability to perform hard deletes by writing delete marker records into the source data
- if the record contains a special field _hoodie_delete_marker set to true, deletes are performed
 2020-01-04 11:07:31 -08:00
SteNicholas
726ae47ce2 [MINOR]Optimize hudi-client module (#1139) 2020-01-04 10:57:08 -08:00
SteNicholas
a733f4ef72 [MINOR] Optimize hudi-cli module (#1136) 2020-01-04 09:05:50 -08:00
Pratyaksh Sharma
290278fc6c [HUDI-118]: Options provided for passing properties to Cleaner, compactor and importer commands 2020-01-03 16:00:57 -08:00
hongdongdong
ff1113f3b7 [HUDI-492]Fix show env all in hudi-cli 2020-01-03 15:50:20 -08:00
lamber-ken
e1e5fe3324 [MINOR] Fix error usage of String.format (#1169) 2020-01-02 09:11:15 +08:00
Pratyaksh Sharma
dde21e7315 [HUDI-402]: code clean up in test cases 2019-12-31 11:10:49 -08:00
Suneel Marthi
98c0d8cf60 Merge pull request #1160 from smarthi/HUDI-343 
[HUDI-343]: Create a DOAP file for Hudi
 2019-12-31 04:21:31 -05:00
Suneel Marthi
47c1f746e2 [HUDI-343]: Create a DOAP file for Hudi 2019-12-31 03:49:21 -05:00
Suneel Marthi
add4b1e329 Merge pull request #1143 from BigDataArtisans/outoflimit 
[MINOR] Fix out of limits for results
 2019-12-31 02:08:54 -05:00
lamber-ken
619f501054 Clean up code 2019-12-31 13:59:26 +08:00
hejinbiao123
e637d9ed26 [HUDI-455] Redo hudi-client log statements using SLF4J (#1145) 
* [HUDI-455] Redo hudi-client log statements using SLF4J
 2019-12-31 13:49:34 +08:00
lamber-ken
ab6ae5cebb [HUDI-482] Fix missing @Override annotation on methods (#1156) 
* [HUDI-482] Fix missing @Override annotation on methods
 2019-12-31 11:44:56 +08:00
dengziming
2a823f32ee [MINOR]: alter some wrong params which bring fatal exception 2019-12-30 16:50:12 -08:00
lamber-ken
e4ea7a2971 Update comment 2019-12-29 19:03:56 -08:00