lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
756 Commits 1 Branch 0 Tags
8e3d81cff118f81a29e286053ba48a1a33eb658c
Commit Graph

756 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Balaji Varadarajan
8e3d81cff1 [MINOR] Change deploy_staging_jars script to take in scala version (#1270) 2020-01-22 14:28:30 +08:00
Balaji Varadarajan
a46fea9ff4 [MINOR] Change deploy_staging_jars script to take in scala version (#1269) 2020-01-22 14:19:47 +08:00
lamber-ken
a54535ed5a [MINOR] Fix invalid maven repo address (#1265) 2020-01-21 04:41:59 -08:00
Mathieu
b6e2993ceb [MINOR] Update the javadoc of HoodieTableMetaClient#scanFiles (#1263) 
[MINOR] Update the javadoc of HoodieTableMetaClient#scanFiles
 2020-01-21 15:50:40 +08:00
Balaji Varadarajan
924bf514e2 [MINOR] Download KEYS file when validating release candidate (#1259) 2020-01-21 09:20:47 +08:00
leesf
6e59c1c777 Moving to 0.5.2-SNAPSHOT on master branch. 2020-01-20 10:51:33 -08:00
Balaji Varadarajan
ba54a7e973 [HUDI-559] : Make the timeline layout version default to be null version 2020-01-20 00:02:55 -08:00
leesf
7087e7d766 [HUDI-556] Add lisence for PR#1233 2020-01-19 09:26:14 -08:00
Y Ethan Guo
9489d0fb84 [HUDI-551] Abstract a test case class for DFS Source to make it extensible (#1239) 2020-01-19 18:50:12 +08:00
Y Ethan Guo
d0ee95ed16 [HUDI-552] Fix the schema mismatch in Row-to-Avro conversion (#1246) 2020-01-18 16:40:56 -08:00
Suneel Marthi
3f4966ddea [MINOR] Fix PMC in DOAP] (#1247) 2020-01-18 11:03:29 -05:00
leesf
5471d8f0c2 [MINOR] Add toString method to TimelineLayoutVersion to make it more readable (#1244) 2020-01-17 20:22:55 -05:00
wenningd
292c1e2ff4 [HUDI-238] Make Hudi support Scala 2.12 (#1226) 
* [HUDI-238] Rename scala related artifactId & add maven profile to support Scala 2.12
 2020-01-17 14:02:21 -08:00
Balaji Varadarajan
923e2b4a1e [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues (#1229) 2020-01-17 10:56:35 -08:00
Prashant Wason
0a07752dc0 [HUDI-527] scalastyle-maven-plugin moved to pluginManagement as it is only used in hoodie-spark and hoodie-cli modules. 
This fixes compile warnings as well as unnecessary plugin invocation for most of the modules which do not have scala code.
 2020-01-17 10:46:10 -08:00
vinoth chandar
baa6b5e889 [HUDI-537] Introduce repair overwrite-hoodie-props CLI command (#1241) 2020-01-17 01:21:44 -08:00
vinoth chandar
c2c0f6b13d [HUDI-509] Renaming code in sync with cWiki restructuring (#1212) 
- Storage Type replaced with Table Type (remaining instances)
 - View types replaced with query types;
 - ReadOptimized view referred as Snapshot Query
 - TableFileSystemView sub interfaces renamed to BaseFileOnly and Slice Views
 - HoodieDataFile renamed to HoodieBaseFile
 - Hive Sync tool will register RO tables for MOR with a `_ro` suffix
 - Datasource/Deltastreamer options renamed accordingly
 - Support fallback to old config values as well, so migration is painless
 - Config for controlling _ro suffix addition
 - Renaming DataFile to BaseFile across DTOs, HoodieFileSlice and AbstractTableFileSystemView
 2020-01-16 23:58:47 -08:00
lamber-ken
8a3a50309b [MINOR] Fix missing @Override annotation on BufferedRandomAccessFile method (#1236) 2020-01-16 11:14:39 -08:00
Y Ethan Guo
b39458b008 [MINOR] Make constant fields final in HoodieTestDataGenerator (#1234) 2020-01-16 12:42:30 +08:00
Scheller
1daba24065 Add GlobalDeleteKeyGenerator 
Adds new GlobalDeleteKeyGenerator for record_key deletes with global indices. Also refactors key generators into their own package.
 2020-01-15 17:01:29 -08:00
Balajee Nagasubramaniam
dd09abb56d [HUDI-335] Improvements to DiskBasedMap used by ExternalSpillableMap, for write and random/sequential read paths, by introducing bufferedRandmomAccessFile 2020-01-15 16:45:45 -08:00
lamber-ken
7aa3ce31e6 [MINOR] Fix redundant judgment statement (#1231) 2020-01-15 16:30:14 -08:00
Sivabalan Narayanan
2248fd9aea Fixing checkstyle issues 2020-01-15 14:21:26 -08:00
Sivabalan Narayanan
2b2f23aa60 Fixing delete util method 2020-01-15 14:21:26 -08:00
Sivabalan Narayanan
87fdb769f0 Adding util methods to assist in adding deletion support to Quick Start 2020-01-15 14:21:26 -08:00
lamber-ken
9b2944a9a2 [MINOR] Refactor unnecessary boxing inside TypedProperties code (#1227) 2020-01-14 19:27:53 -08:00
Mehrotra
2bb0c21a3d Fix conversion of Spark struct type to Avro schema 
cr https://code.amazon.com/reviews/CR-17184364
 2020-01-14 00:27:56 -08:00
lamber-ken
fd8f1c70c0 [MINOR] Reuse random object (#1222) 2020-01-13 18:26:04 -08:00
liujianhui
c1f8acab34 [HUDI-526] fix the HoodieAppendHandle 2020-01-13 10:44:31 -08:00
openopen2
a44c61b813 [HUDI-502] provide a custom time zone definition for TimestampBasedKeyGenerator (#1188) 2020-01-12 15:45:23 -08:00
harveyyue
971c7d41bd [HUDI-322] DeltaSteamer should pick checkpoints off only deltacommits for MOR tables 2020-01-12 15:11:47 -08:00
Udit Mehrotra
ad50008a59 [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types 
- Upgrade Spark to 2.4.4, Parquet to 1.10.1, Avro to 1.8.2
- Remove spark-avro from hudi-spark-bundle. Users need to provide --packages org.apache.spark:spark-avro:2.4.4 when running spark-shell or spark-submit
- Replace com.databricks:spark-avro with org.apache.spark:spark-avro
- Shade avro in hudi-hadoop-mr-bundle to make sure it does not conflict with hive's avro version.
 2020-01-12 15:03:11 -08:00
lamber-ken
d9675c4ec0 [HUDI-522] Use the same version jcommander uniformly (#1214) 2020-01-12 10:48:52 -08:00
lamber-ken
017ee8e661 [MINOR] Fix partition typo (#1209) 2020-01-12 09:35:55 +08:00
lamber-ken
e103165083 [CLEAN] replace utf-8 constant with StandardCharsets.UTF_8 2020-01-10 16:23:29 -08:00
Thinking
b95367d82a [HUDI-469] Fix: HoodieCommitMetadata only show first commit insert rows. 2020-01-10 16:17:11 -08:00
leesf
04afac977d [HUDI-248] CLI doesn't allow rolling back a Delta commit 2020-01-10 16:10:35 -08:00
pratyakshsharma
3c90d252cc [HUDI-114]: added option to overwrite payload implementation in hoodie.properties file 2020-01-09 22:34:40 -08:00
hongdd
5af3dc6aed [HUDI-331]Fix java docs for all public apis in HoodieWriteClient (#1111) 2020-01-09 16:00:53 +08:00
Bhavani Sudha Saktheeswaran
d09eacdc13 [HUDI-25] Optimize HoodieInputformat.listStatus() for faster Hive incremental queries on Hoodie 
Summary:
    - InputPathHandler class classifies  inputPaths into incremental, non incremental and non hoodie paths.
    - Incremental queries leverage HoodieCommitMetadata to get partitions that are affected and only lists those partitions as opposed to listing all partitions
    - listStatus() processes each category separately
 2020-01-08 14:53:05 -08:00
Y Ethan Guo
480fc7869d [HUDI-319] Add a new maven profile to generate unified Javadoc for all Java and Scala classes (#1195) 
* Add javadoc build command in README, links to javadoc plugin and rename profile.
* Make java version configurable in one place.
 2020-01-08 10:38:09 -08:00
Wenning Ding
aba83876e7 Update deprecated HBase API 2020-01-08 10:26:47 -08:00
lamber-ken
9884972a3a [MINOR] Remove old jekyll config file (#1198) 2020-01-07 16:45:19 -08:00
vinoth chandar
9706f659db [HUDI-508] Standardizing on "Table" instead of "Dataset" across code (#1197) 
- Docs were talking about storage types before, cWiki moved to "Table"
 - Most of code already has HoodieTable, HoodieTableMetaClient - correct naming
 - Replacing renaming use of dataset across code/comments
 - Few usages in comments and use of Spark SQL DataSet remain unscathed
 2020-01-07 12:52:32 -08:00
Balaji Varadarajan
8306f749a2 [HUDI-417] Refactor HoodieWriteClient so that commit logic can be shareable by both bootstrap and normal write operations (#1166) 2020-01-06 20:11:48 -08:00
Abhishek Modi
b5df6723a2 [HUDI-464] Use Hive Exec Core for tests (#1125) 2020-01-06 16:32:55 -08:00
lamber-ken
75c3f630d4 [HUDI-405] Remove HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from DataSource 2020-01-06 14:25:38 -08:00
yuehan124
c78092d2d3 [HUDI-501] Execute docker/setup_demo.sh in any directory 2020-01-06 10:26:06 -08:00
Pratyaksh Sharma
8f935e779a [HUDI-406]: added default partition path in TimestampBasedKeyGenerator 2020-01-06 09:38:06 -08:00
hongdd
2d5b79d96f [HUDI-438] Merge duplicated code fragment in HoodieSparkSqlWriter (#1114) 2020-01-06 22:51:22 +08:00