lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
868 Commits 1 Branch 0 Tags
fa36082554373dd4dce3e3d3159ab87300a4601d
Commit Graph

868 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Suneel Marthi
fa36082554 [HUDI-746] Reduce build warnings < 10 (#1465) 2020-03-30 11:46:52 +08:00
vinoth chandar
fad4bd377b [HUDI-745] CI should fail PRs with unapproved license files (#1464) 2020-03-29 10:59:40 -07:00
vinoth chandar
e057c27603 [HUDI-744] Restructure hudi-common and clean up files under util packages (#1462) 
- Brings more order and cohesion to the classes in hudi-common
 - Utils classes related to a particular concept (avro, timeline,...) are placed near to the package
 - common.fs package now contains all the filesystem level classes including wrapper filesystem
 - bloom.filter package renamed to just bloom
 - config package contains classes that help store properties
 - common.fs.inline package contains all the inline filesystem classes/impl
 - common.table.timeline now consolidates all timeline related classes
 - common.table.view consolidates all the classes related to filesystem view metadata
 - common.table.timeline.versioning contains all classes related to versioning of timeline
 - Fix few unit tests as a result
 - Moved the test packages around to match the source file move
 - Rename AvroUtils to TimelineMetadataUtils & minor fixes/typos
 2020-03-29 10:58:49 -07:00
leesf
07c3c5d797 [HUDI-679] Make io package Spark free (#1460) 
* [HUDI-679] Make io package Spark free
 2020-03-29 16:54:00 +08:00
Sivabalan Narayanan
ac73bdcdc3 [HUDI-430] Adding InlineFileSystem to support embedding any file format as an InlineFile (#1176) 
* Adding InlineFileSystem to support embedding any file format (parquet, hfile, etc). Supports reading the embedded file using respective readers.
 2020-03-28 12:13:35 -04:00
Suneel Marthi
04449f33fe [HUDI-743]: Remove FileIOUtils.close() (#1461) 2020-03-28 18:03:15 +08:00
Suneel Marthi
8c3001363d HUDI-479: Eliminate or Minimize use of Guava if possible (#1159) 2020-03-28 03:11:32 -04:00
Raymond Xu
1713f686f8 [MINOR] Add error message when check arguments (#1451) 2020-03-27 10:21:38 +08:00
leesf
8b0a4009a9 [HUDI-678] Make config package spark free (#1418) 2020-03-26 08:30:27 -07:00
Suneel Marthi
e101ea9bd4 [MINOR] Update DOAP with 0.5.2 Release (#1448) 2020-03-25 23:37:32 -04:00
Mathieu
5eed6c98a8 [MINOR] Fix javadoc of InsertBucket (#1445) 2020-03-25 22:25:47 +08:00
Raymond Xu
bc82e2be6c [HUDI-711] Refactor exporter main logic (#1436) 
* Refactor exporter main logic
* break main method into multiple readable methods
* fix bug of passing wrong file list
* avoid deleting output path when exists
* throw exception to early abort on multiple cases
* use JavaSparkContext instead of SparkSession
* improve unit test for expected exceptions
 2020-03-25 18:02:24 +08:00
hongdd
cafc87041b [HUDI-697]Add unit test for ArchivedCommitsCommand (#1424) 2020-03-23 13:46:10 +08:00
Zhiyuan Zhao
0241b21f77 [HUDI-65] commitTime rename to instantTime (#1431) 2020-03-22 18:06:00 -07:00
lamber-ken
38c3ccc51a [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly (#1377) 2020-03-22 10:31:48 -07:00
Pratyaksh Sharma
1e1d9e1d34 [HUDI-616] Fixed parquet files getting created on local FS (#1434) 2020-03-22 22:19:47 +08:00
Zhiyuan Zhao
06652aa935 [MINOR] Add omissive param desc on method doc and cleanup redundant code (#1437) 2020-03-22 21:39:33 +08:00
Zhiyuan Zhao
8b00791ef4 [MINOR] cleanup redundant comment and unused variable and fix typo (#1435) 2020-03-21 20:12:06 -07:00
vinoyang
c5030f77a0 [HUDI-720] NOTICE file needs to add more content based on the NOTICE files of the ASF projects that hudi bundles (#1417) 
* [HUDI-720] NOTICE file needs to add more content based on the NOTICE files of the ASF projects that hudi bundles
 2020-03-21 10:54:04 +08:00
satishkotha
83fb9651f3 [HUDI-650] Modify handleUpdate path to validate partitionPath (#1368) 2020-03-20 08:37:22 -07:00
Mathieu
eeab532d79 [HUDI-725] Remove init log in the constructor of DeltaSync (#1425) 2020-03-20 17:47:59 +08:00
Mathieu
21c45e1051 [HUDI-726]Delete unused method in HoodieDeltaStreamer (#1426) 2020-03-20 17:44:16 +08:00
Zhiyuan Zhao
14e0c95206 [HUDI-400] Check upgrade from old plan to new plan for compaction (#1422) 
* Fix NPE when DataFile is null
* Check from old plan upgrade to new plan
 2020-03-20 15:13:17 +08:00
Sivabalan Narayanan
a752b7b18c Merge pull request #1165 from yihua/HUDI-76-deltastreamer-csv-source 
[HUDI-76] Add CSV Source support for Hudi Delta Streamer
 2020-03-19 10:00:53 -04:00
ForwardXu
1e321c2fc0 [HUDI-209] Implement JMX metrics reporter (#1106) 2020-03-19 20:10:35 +08:00
Raymond Xu
779edc0688 [HUDI-344] Add partitioner param to Exporter (#1405) 2020-03-18 19:24:04 +08:00
leesf
0a4902ecce [HUDI-437] Support user-defined index (#1408) 
* [hotfix] set default value for index class config
* class config takes precedence over `hoodie.index.type`
 2020-03-17 19:27:40 -07:00
vinoth chandar
e3019031d8 [HUDI-539] Make ROPathFilter conf member serializable (#1415) 2020-03-17 12:52:48 -07:00
hongdd
f1d7bb381d [HUDI-695]Add unit test for TableCommand (#1411) 2020-03-17 14:15:30 +08:00
bschell
418f9bb2e9 Add constructor to HoodieROTablePathFilter (#1413) 
Allows HoodieROTablePathFilter to accept a configuration for
initializing the filesystem. This fixes a bug with Presto's use of this
pathfilter.

Co-authored-by: Brandon Scheller <bschelle@amazon.com>
 2020-03-16 15:19:16 -07:00
hongdd
3ef9e885ca [HUDI-715] Fix duplicate name in TableCommand (#1410) 2020-03-16 17:19:57 +08:00
hongdd
55e6d34815 [HUDI-694]Add unit test for SparkEnvCommand (#1401) 
* Add test for SparkEnvCommand
 2020-03-16 11:52:40 +08:00
Y Ethan Guo
cf765df606 [HUDI-76] Add CSV Source support for Hudi Delta Streamer 2020-03-15 19:03:37 -07:00
Balaji Varadarajan
23afe7a487 [HUDI-710] Fixing failure in Staging Validation Script (#1403) 2020-03-15 22:13:20 +08:00
Raymond Xu
14323cb100 [HUDI-344] Improve exporter tests (#1404) 2020-03-15 20:24:30 +08:00
Suneel Marthi
99b7e9eb9e [HUDI-629]: Replace Guava's Hashing with an equivalent in NumericUtils.java (#1350) 
* [HUDI-629]: Replace Guava's Hashing with an equivalent in NumericUtils.java
 2020-03-13 20:28:05 -04:00
vinoth chandar
fb7fba365f [HUDI-646] fix failing test due to improper filesytem cleanup (#1373) 2020-03-12 23:59:09 -07:00
Udit Mehrotra
c40a0d4e91 [HUDI-656][Performance] Return a dummy Spark relation after writing the DataFrame (#1394) 
Co-authored-by: Mehrotra <uditme@amazon.com>
 2020-03-11 20:27:46 -07:00
hongdd
0f892ef62c [HUDI-692] Add delete savepoint for cli (#1397) 
* Add delete savepoint for cli
* Add check
* Move JavaSparkContext to try
 2020-03-11 16:49:02 -07:00
Prashant Wason
7d66831444 [MINOR] Removing code which is duplicated from the base class HoodieWriteHandle. (#1399) 2020-03-11 16:43:04 -07:00
satishkotha
7194514aff [HUDI-689] Change CLI command names to not have overlap (#1392) 2020-03-11 16:29:54 -07:00
Sivabalan Narayanan
1ca912af09 [HUDI-667] Fixing delete tests for DeltaStreamer (#1395) 2020-03-11 16:19:23 -07:00
vinoth chandar
dd7cf38a13 [HUDI-688] Paring down the NOTICE file to minimum required notices (#1391) 
- Based on analysis, we don't need to call out anything
 - We only do source releases at this time
 - Fix typo in LICENSE
 2020-03-11 08:24:07 -04:00
Prashant Wason
cf0a4c19bc [HUDI-670] Added test cases for TestDiskBasedMap. (#1379) 
* [HUDI-670] Added test cases for TestDiskBasedMap.

* Update TestDiskBasedMap.java

Co-authored-by: Suneel Marthi <smarthi@apache.org>
 2020-03-11 08:03:03 -04:00
Prashant Wason
77d5b92d88 [HUDI-668] Added additional unit-tests for HUDI metrics. (#1380) 2020-03-09 23:15:42 -04:00
openopen2
44700d531a [HUDI-344] Hudi Dataset Snapshot Exporter (#1360) 
Co-authored-by: jason1993 <261049174@qq.com>
 2020-03-10 09:17:51 +08:00
hongdd
f93e64fee4 [HUDI-681]Remove embeddedTimelineService from HoodieReadClient (#1388) 
* [HUDI-681]Remove embeddedTimelineService from HoodieReadClient
 2020-03-09 18:31:04 +08:00
vinoyang
2137ecfa22 [HUDI-676] Address issues towards removing use of WIP Disclaimer (#1386) 
* rename DISCLAIMER-STANDARD TO DISCLAIMER
 2020-03-08 23:14:49 -07:00
Suneel Marthi
415882f902 [HUDI-581] NOTICE need more work as it missing content form included 3rd party ALv2 licensed NOTICE files (#1354) 
* [HUDI-581] - Add 3rd party library NOTICE

* [HUDI-581]: NOTICE need more work as it missing content form included 3rd party ALv2 licensed NOTICE files
 2020-03-07 22:08:35 -05:00
Prashant Wason
5f8bf97005 [HUDI-671] Added unit-test for HBaseIndex (#1381) 2020-03-07 16:48:43 -08:00