1
0
Commit Graph

94 Commits

Author SHA1 Message Date
Shaofeng Shi
78b3194e82 [HUDI-751] Fix some coding issues reported by FindBugs (#1470) 2020-03-31 21:19:32 +08:00
Edwin Guo
9ecf0ccfb2 [HUDI-742] Fix Java Math Exception (#1466) 2020-03-31 12:56:20 +08:00
lamber-ken
dbc9acd23a [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean (#1432) 2020-03-30 11:19:17 -07:00
ffcchi
1f5b0c77d6 [HUDI-724] Parallelize getSmallFiles for partitions (#1421)
Co-authored-by: Feichi Feng <feicfeng@amazon.com>
2020-03-30 00:14:38 -07:00
Suneel Marthi
fa36082554 [HUDI-746] Reduce build warnings < 10 (#1465) 2020-03-30 11:46:52 +08:00
vinoth chandar
e057c27603 [HUDI-744] Restructure hudi-common and clean up files under util packages (#1462)
- Brings more order and cohesion to the classes in hudi-common
 - Utils classes related to a particular concept (avro, timeline,...) are placed near to the package
 - common.fs package now contains all the filesystem level classes including wrapper filesystem
 - bloom.filter package renamed to just bloom
 - config package contains classes that help store properties
 - common.fs.inline package contains all the inline filesystem classes/impl
 - common.table.timeline now consolidates all timeline related classes
 - common.table.view consolidates all the classes related to filesystem view metadata
 - common.table.timeline.versioning contains all classes related to versioning of timeline
 - Fix few unit tests as a result
 - Moved the test packages around to match the source file move
 - Rename AvroUtils to TimelineMetadataUtils & minor fixes/typos
2020-03-29 10:58:49 -07:00
leesf
07c3c5d797 [HUDI-679] Make io package Spark free (#1460)
* [HUDI-679] Make io package Spark free
2020-03-29 16:54:00 +08:00
Suneel Marthi
04449f33fe [HUDI-743]: Remove FileIOUtils.close() (#1461) 2020-03-28 18:03:15 +08:00
Suneel Marthi
8c3001363d HUDI-479: Eliminate or Minimize use of Guava if possible (#1159) 2020-03-28 03:11:32 -04:00
Raymond Xu
1713f686f8 [MINOR] Add error message when check arguments (#1451) 2020-03-27 10:21:38 +08:00
leesf
8b0a4009a9 [HUDI-678] Make config package spark free (#1418) 2020-03-26 08:30:27 -07:00
Mathieu
5eed6c98a8 [MINOR] Fix javadoc of InsertBucket (#1445) 2020-03-25 22:25:47 +08:00
hongdd
cafc87041b [HUDI-697]Add unit test for ArchivedCommitsCommand (#1424) 2020-03-23 13:46:10 +08:00
Zhiyuan Zhao
0241b21f77 [HUDI-65] commitTime rename to instantTime (#1431) 2020-03-22 18:06:00 -07:00
Zhiyuan Zhao
06652aa935 [MINOR] Add omissive param desc on method doc and cleanup redundant code (#1437) 2020-03-22 21:39:33 +08:00
satishkotha
83fb9651f3 [HUDI-650] Modify handleUpdate path to validate partitionPath (#1368) 2020-03-20 08:37:22 -07:00
ForwardXu
1e321c2fc0 [HUDI-209] Implement JMX metrics reporter (#1106) 2020-03-19 20:10:35 +08:00
leesf
0a4902ecce [HUDI-437] Support user-defined index (#1408)
* [hotfix] set default value for index class config
* class config takes precedence over `hoodie.index.type`
2020-03-17 19:27:40 -07:00
Suneel Marthi
99b7e9eb9e [HUDI-629]: Replace Guava's Hashing with an equivalent in NumericUtils.java (#1350)
* [HUDI-629]: Replace Guava's Hashing with an equivalent in NumericUtils.java
2020-03-13 20:28:05 -04:00
Prashant Wason
7d66831444 [MINOR] Removing code which is duplicated from the base class HoodieWriteHandle. (#1399) 2020-03-11 16:43:04 -07:00
hongdd
f93e64fee4 [HUDI-681]Remove embeddedTimelineService from HoodieReadClient (#1388)
* [HUDI-681]Remove embeddedTimelineService from HoodieReadClient
2020-03-09 18:31:04 +08:00
vinoyang
ee5b32f5d4 [HUDI-652] Decouple HoodieReadClient and AbstractHoodieClient to break the inheritance chain (#1372)
* Removed timeline server support
* Removed try-with-resource
2020-03-06 09:59:35 -08:00
hongdd
8306205d7a [HUDI-332]Add operation type (insert/upsert/bulkinsert/delete) to HoodieCommitMetadata (#1157)
[HUDI-332]Add operation type (insert/upsert/bulkinsert/delete) to HoodieCommitMetadata (#1157)
2020-03-03 10:10:29 -08:00
vinoth chandar
71170fafe7 [HUDI-554] Cleanup package structure in hudi-client (#1346)
- Just package, class moves and renames with the following intent
 - `client` now has all the various client classes, that do the transaction management
 - `func` renamed to `execution` and some helpers moved to `client/utils`
 - All compaction code under `io` now under `table/compact`
 - Rollback code under `table/rollback` and in general all code for individual operations under `table`
 - `exception` `config`, `metrics` left untouched
 - Moved the tests also accordingly
 - Fixed some flaky tests
2020-02-27 08:05:58 -08:00
Suneel Marthi
078d4825d9 [HUDI-624]: Split some of the code from PR for HUDI-479 (#1344) 2020-02-21 14:22:21 +08:00
Suneel Marthi
f9d2f66dc1 [HUDI-622]: Remove VisibleForTesting annotation and import from code (#1343)
* HUDI:622: Remove VisibleForTesting annotation and import from code
2020-02-20 15:17:53 +08:00
Sivabalan Narayanan
00493235f5 [HUDI-108] Removing 2GB spark partition limitations in HoodieBloomIndex with spark 2.4.4 (#1315) 2020-02-18 11:12:20 -08:00
wangxianghu
aaa6cf9a98 [MINOR] Fix some typos 2020-02-15 09:49:25 +08:00
Raymond Xu
c1516df8ac [HUDI-499] Allow update partition path with GLOBAL_BLOOM (#1187)
* Handle partition path update by deleting a record from the old partition and
  insert into the new one
* Add a new configuration "hoodie.bloom.index.update.partition.path" to
  enable the behavior
* Add a new unit test case for global bloom index
2020-02-05 09:33:33 -08:00
Suneel Marthi
594da28fbf [HUDI-595] code cleanup, refactoring code out of PR# 1159 (#1302) 2020-02-04 21:52:03 +08:00
Suneel Marthi
5b7bb142dc [HUDI-583] Code Cleanup, remove redundant code, and other changes (#1237) 2020-02-02 18:03:44 +08:00
Balaji Varadarajan
ba54a7e973 [HUDI-559] : Make the timeline layout version default to be null version 2020-01-20 00:02:55 -08:00
Balaji Varadarajan
923e2b4a1e [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues (#1229) 2020-01-17 10:56:35 -08:00
vinoth chandar
c2c0f6b13d [HUDI-509] Renaming code in sync with cWiki restructuring (#1212)
- Storage Type replaced with Table Type (remaining instances)
 - View types replaced with query types;
 - ReadOptimized view referred as Snapshot Query
 - TableFileSystemView sub interfaces renamed to BaseFileOnly and Slice Views
 - HoodieDataFile renamed to HoodieBaseFile
 - Hive Sync tool will register RO tables for MOR with a `_ro` suffix
 - Datasource/Deltastreamer options renamed accordingly
 - Support fallback to old config values as well, so migration is painless
 - Config for controlling _ro suffix addition
 - Renaming DataFile to BaseFile across DTOs, HoodieFileSlice and AbstractTableFileSystemView
2020-01-16 23:58:47 -08:00
Balajee Nagasubramaniam
dd09abb56d [HUDI-335] Improvements to DiskBasedMap used by ExternalSpillableMap, for write and random/sequential read paths, by introducing bufferedRandmomAccessFile 2020-01-15 16:45:45 -08:00
lamber-ken
7aa3ce31e6 [MINOR] Fix redundant judgment statement (#1231) 2020-01-15 16:30:14 -08:00
liujianhui
c1f8acab34 [HUDI-526] fix the HoodieAppendHandle 2020-01-13 10:44:31 -08:00
pratyakshsharma
3c90d252cc [HUDI-114]: added option to overwrite payload implementation in hoodie.properties file 2020-01-09 22:34:40 -08:00
hongdd
5af3dc6aed [HUDI-331]Fix java docs for all public apis in HoodieWriteClient (#1111) 2020-01-09 16:00:53 +08:00
Wenning Ding
aba83876e7 Update deprecated HBase API 2020-01-08 10:26:47 -08:00
vinoth chandar
9706f659db [HUDI-508] Standardizing on "Table" instead of "Dataset" across code (#1197)
- Docs were talking about storage types before, cWiki moved to "Table"
 - Most of code already has HoodieTable, HoodieTableMetaClient - correct naming
 - Replacing renaming use of dataset across code/comments
 - Few usages in comments and use of Spark SQL DataSet remain unscathed
2020-01-07 12:52:32 -08:00
Balaji Varadarajan
8306f749a2 [HUDI-417] Refactor HoodieWriteClient so that commit logic can be shareable by both bootstrap and normal write operations (#1166) 2020-01-06 20:11:48 -08:00
hejinbiao123
b9fab0b933 Revert "[HUDI-455] Redo hudi-client log statements using SLF4J (#1145)" (#1181)
This reverts commit e637d9ed26.
2020-01-06 21:13:29 +08:00
SteNicholas
726ae47ce2 [MINOR]Optimize hudi-client module (#1139) 2020-01-04 10:57:08 -08:00
hejinbiao123
e637d9ed26 [HUDI-455] Redo hudi-client log statements using SLF4J (#1145)
* [HUDI-455] Redo hudi-client log statements using SLF4J
2019-12-31 13:49:34 +08:00
lamber-ken
ab6ae5cebb [HUDI-482] Fix missing @Override annotation on methods (#1156)
* [HUDI-482] Fix missing @Override annotation on methods
2019-12-31 11:44:56 +08:00
lamber-ken
8440482977 Fix empty content clean plan 2019-12-29 19:03:56 -08:00
lamber-ken
2f254163d4 Skip setting commit metadata 2019-12-29 19:03:56 -08:00
lamber-ken
179837e8ef Fix checkstyle 2019-12-29 19:03:56 -08:00
lamber-ken
58c5bed40a [HUDI-453] Fix throw failed to archive commits error when writing data to MOR/COW table 2019-12-29 19:03:56 -08:00