lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
777 Commits 1 Branch 0 Tags
462fd025563b0ae8a4d4f28d366a9bbfca070d3f
Commit Graph

82 Commits

Author SHA1 Message Date
Satish Kotha
462fd02556 [HUDI-571] Add 'commits show archived' command to CLI 2020-02-05 11:25:34 -08:00
lamber-ken
46842f4e92 [MINOR] Remove the declaration of thrown RuntimeException (#1305) 2020-02-05 23:23:20 +08:00
Prashant Wason
4de0fcfcb5 [HUDI-566] Added new test cases for class HoodieTimeline, HoodieDefaultTimeline and HoodieActiveTimeline. 2020-02-04 18:55:04 -08:00
Suneel Marthi
594da28fbf [HUDI-595] code cleanup, refactoring code out of PR# 1159 (#1302) 2020-02-04 21:52:03 +08:00
Suneel Marthi
5b7bb142dc [HUDI-583] Code Cleanup, remove redundant code, and other changes (#1237) 2020-02-02 18:03:44 +08:00
Prashant Wason
f27c7a16c6 [HUDI-564] Added new test cases for HoodieLogFormat and HoodieLogFormatVersion. 2020-01-30 13:53:18 -08:00
Balajee Nagasubramaniam
6f34be1b8d HUDI-117 Close file handle before throwing an exception due to append failure. 
Add test cases to handle/verify stage failure scenarios.
 2020-01-29 15:28:51 -08:00
hmatu
362a9b9aab [MINOR] Remove junit-dep dependency 2020-01-28 10:31:55 -08:00
Mathieu
b6e2993ceb [MINOR] Update the javadoc of HoodieTableMetaClient#scanFiles (#1263) 
[MINOR] Update the javadoc of HoodieTableMetaClient#scanFiles
 2020-01-21 15:50:40 +08:00
leesf
6e59c1c777 Moving to 0.5.2-SNAPSHOT on master branch. 2020-01-20 10:51:33 -08:00
Balaji Varadarajan
ba54a7e973 [HUDI-559] : Make the timeline layout version default to be null version 2020-01-20 00:02:55 -08:00
leesf
5471d8f0c2 [MINOR] Add toString method to TimelineLayoutVersion to make it more readable (#1244) 2020-01-17 20:22:55 -05:00
Balaji Varadarajan
923e2b4a1e [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues (#1229) 2020-01-17 10:56:35 -08:00
vinoth chandar
c2c0f6b13d [HUDI-509] Renaming code in sync with cWiki restructuring (#1212) 
- Storage Type replaced with Table Type (remaining instances)
 - View types replaced with query types;
 - ReadOptimized view referred as Snapshot Query
 - TableFileSystemView sub interfaces renamed to BaseFileOnly and Slice Views
 - HoodieDataFile renamed to HoodieBaseFile
 - Hive Sync tool will register RO tables for MOR with a `_ro` suffix
 - Datasource/Deltastreamer options renamed accordingly
 - Support fallback to old config values as well, so migration is painless
 - Config for controlling _ro suffix addition
 - Renaming DataFile to BaseFile across DTOs, HoodieFileSlice and AbstractTableFileSystemView
 2020-01-16 23:58:47 -08:00
lamber-ken
8a3a50309b [MINOR] Fix missing @Override annotation on BufferedRandomAccessFile method (#1236) 2020-01-16 11:14:39 -08:00
Balajee Nagasubramaniam
dd09abb56d [HUDI-335] Improvements to DiskBasedMap used by ExternalSpillableMap, for write and random/sequential read paths, by introducing bufferedRandmomAccessFile 2020-01-15 16:45:45 -08:00
lamber-ken
9b2944a9a2 [MINOR] Refactor unnecessary boxing inside TypedProperties code (#1227) 2020-01-14 19:27:53 -08:00
openopen2
a44c61b813 [HUDI-502] provide a custom time zone definition for TimestampBasedKeyGenerator (#1188) 2020-01-12 15:45:23 -08:00
lamber-ken
017ee8e661 [MINOR] Fix partition typo (#1209) 2020-01-12 09:35:55 +08:00
lamber-ken
e103165083 [CLEAN] replace utf-8 constant with StandardCharsets.UTF_8 2020-01-10 16:23:29 -08:00
Thinking
b95367d82a [HUDI-469] Fix: HoodieCommitMetadata only show first commit insert rows. 2020-01-10 16:17:11 -08:00
pratyakshsharma
3c90d252cc [HUDI-114]: added option to overwrite payload implementation in hoodie.properties file 2020-01-09 22:34:40 -08:00
vinoth chandar
9706f659db [HUDI-508] Standardizing on "Table" instead of "Dataset" across code (#1197) 
- Docs were talking about storage types before, cWiki moved to "Table"
 - Most of code already has HoodieTable, HoodieTableMetaClient - correct naming
 - Replacing renaming use of dataset across code/comments
 - Few usages in comments and use of Spark SQL DataSet remain unscathed
 2020-01-07 12:52:32 -08:00
Pratyaksh Sharma
dde21e7315 [HUDI-402]: code clean up in test cases 2019-12-31 11:10:49 -08:00
lamber-ken
ab6ae5cebb [HUDI-482] Fix missing @Override annotation on methods (#1156) 
* [HUDI-482] Fix missing @Override annotation on methods
 2019-12-31 11:44:56 +08:00
lamber-ken
e4ea7a2971 Update comment 2019-12-29 19:03:56 -08:00
lamber-ken
8440482977 Fix empty content clean plan 2019-12-29 19:03:56 -08:00
Mathieu
01c25d6aff [MINOR] Update the java doc of HoodieTableType (#1148) 2019-12-29 09:57:19 +08:00
hongdd
8affdf8bcb [HUDI-416] Improve hint information for cli (#1110) 2019-12-25 20:19:12 +08:00
dengziming
94aec965f5 [minor] Fix few typos in the java docs (#1132) 2019-12-24 20:44:11 -08:00
comsir
dd06660183 [MINOR] fix typo 2019-12-24 20:40:00 -08:00
vinoth chandar
350b0ecb4d [HUDI-311] : Support for AWS Database Migration Service in DeltaStreamer 
- Add a transformer class, that adds `Op` fiels if not found in input frame
 - Add a payload implementation, that issues deletes when Op=D
 - Remove Parquet as a top level source type, consolidate with RowSource
 - Made delta streamer work without a property file, simply using overridden cli options
 - Unit tests for transformer/payload classes
 2019-12-23 20:56:55 -08:00
Sivabalan Narayanan
14881e99e0 [HUDI-106] Adding support for DynamicBloomFilter (#976) 
- Introduced configs for bloom filter type
- Implemented dynamic bloom filter with configurable max number of keys
- BloomFilterFactory abstractions; Defaults to current simple bloom filter
 2019-12-17 19:06:24 -08:00
Balaji Varadarajan
9a1f698eef [HUDI-308] Avoid Renames for tracking state transitions of all actions on dataset 2019-12-15 21:26:30 -08:00
lamber-ken
ba514cfea0 [MINOR] Remove redundant plus operator (#1097) 2019-12-12 05:42:05 +08:00
lamber-ken
d447e2d751 [checkstyle] Unify LOG form (#1092) 2019-12-10 19:23:38 +08:00
lamber-ken
2745b7552f [HUDI-379] Refactor the codes based on new JavadocStyle code style rule (#1079) 2019-12-06 12:59:28 +08:00
lamber-ken
c06d89b648 [HUDI-378] Refactor the rest codes based on new ImportOrder code style rule (#1078) 2019-12-05 17:25:03 +08:00
lamber-ken
b3e0ebbc4a [checkstyle] Add ConstantName java checkstyle rule (#1066) 
* add SimplifyBooleanExpression java checkstyle rule
* collapse empty tags in scalastyle file
 2019-12-04 18:59:15 +08:00
vinoyang
84602c8882 [HUDI-355] Refactor hudi-common based on new comment and code style rules (#1049) 
[HUDI-355] Refactor hudi-common based on new comment and code style rules
 2019-12-03 20:49:13 -08:00
leesf
98ab33bb6e [HUDI-294] Delete Paths written in Cleaner plan needs to be relative to partition-path (#1062) 
[HUDI-294] Delete Paths written in Cleaner plan needs to be relative to partition-path
 2019-12-03 10:11:03 -08:00
lamber-ken
784e3ad0b6 [HUDI-370] Refactor hudi-common based on new ImportOrder code style rule (#1063) 2019-12-02 06:59:09 +08:00
谢磊
b77fad39b5 [HUDI-364] Refactor hudi-hive based on new ImportOrder code style rule (#1048) 
[HUDI-364] Refactor hudi-hive based on new ImportOrder code style rule
 2019-11-27 16:30:37 +08:00
wenningd
d6e83e8f49 [HUDI-325] Fix Hive partition error for updated HDFS Hudi table (#1001) 2019-11-26 21:18:39 -08:00
bschell
60fed21dc7 [HUDI-327] Add null/empty checks to key generators (#1040) 
* Adds null and empty checks to all key generators. 
* Also improves error messaging for key generator issues.
 2019-11-26 02:37:16 -08:00
Sivabalan Narayanan
c3355109b1 [HUDI-328] Adding delete api to HoodieWriteClient (#1004) 
[HUDI-328]  Adding delete api to HoodieWriteClient and Spark DataSource
 2019-11-22 15:05:25 -08:00
hongdd
7bc08cbfdc [HUDI-345] Fix used deprecated function (#1024) 
- Schema.parse() with new Schema.Parser().parse
- FSDataOutputStream constructor
 2019-11-22 03:32:09 -08:00
谢磊
804e348d0e [HUDI-346] Set allowMultipleEmptyLines to false for EmptyLineSeparator rule (#1025) 2019-11-19 18:44:42 +08:00
b_rousseau
e806eb797f [HUDI-339] Add support of Azure cloud storage (#1019) 
- Add Azure WASB (BLOB) and ADLS storage in StorageSchemes enum
- Update testStorageSchemes to test new added storage
 2019-11-17 14:29:24 -08:00
Nishith Agarwal
f82e58994e - Ensure that rollback instant is always created before the next commit instant. 
This especially affects IncrementalPull for MOR tables since we can end up pulling in
  log blocks for uncommitted data
- Ensure that generated commit instants are 1 second apart
 2019-11-17 14:11:26 -08:00