lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,078 Commits 1 Branch 0 Tags
9fe2d2b14a2a0f465d75e39fdfdcc4d5e0101ee7
Commit Graph

1078 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
vinoyang
194e20e661 [MINOR] Fix label issue in .asf.yaml (#1478) 2020-04-02 15:51:51 +08:00
Raymond Xu
5b53b0d85e [HUDI-731] Add ChainedTransformer (#1440) 
* [HUDI-731] Add ChainedTransformer
 2020-04-01 23:21:31 +08:00
Trevor
2a611f4ad3 [HUDI-749] Fix hudi-timeline-server-bundle run_server.sh start error (#1477) 2020-04-01 22:19:54 +08:00
vinoyang
c146ca90fd [HUDI-754] Configure .asf.yaml for Hudi Github repository (#1472) 
* [HUDI-754] Configure .asf.yaml for Hudi Github repository
 2020-04-01 10:02:47 +08:00
Shaofeng Shi
78b3194e82 [HUDI-751] Fix some coding issues reported by FindBugs (#1470) 2020-03-31 21:19:32 +08:00
Edwin Guo
9ecf0ccfb2 [HUDI-742] Fix Java Math Exception (#1466) 2020-03-31 12:56:20 +08:00
wenningd
ce0a4c64d0 [HUDI-713] Fix conversion of Spark array of struct type to Avro schema (#1406) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2020-03-30 15:52:15 -07:00
lamber-ken
dbc9acd23a [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean (#1432) 2020-03-30 11:19:17 -07:00
Prashant Wason
9f51b99174 [MINOR] Updated HoodieMergeOnReadTestUtils for future testing requirements (#1456) 
1. getRecordsUsingInputFormat() can take a custom Configuration which can be used to specify HUDI table properties (e.g. <table>.consume.mode or <table>.consume.start.timestamp)
2. Fixed the return to return an empty List rather than raise an Exception if no records are found
 2020-03-30 07:36:12 -07:00
ffcchi
1f5b0c77d6 [HUDI-724] Parallelize getSmallFiles for partitions (#1421) 
Co-authored-by: Feichi Feng <feicfeng@amazon.com>
 2020-03-30 00:14:38 -07:00
Suneel Marthi
fa36082554 [HUDI-746] Reduce build warnings < 10 (#1465) 2020-03-30 11:46:52 +08:00
vinoth chandar
fad4bd377b [HUDI-745] CI should fail PRs with unapproved license files (#1464) 2020-03-29 10:59:40 -07:00
vinoth chandar
e057c27603 [HUDI-744] Restructure hudi-common and clean up files under util packages (#1462) 
- Brings more order and cohesion to the classes in hudi-common
 - Utils classes related to a particular concept (avro, timeline,...) are placed near to the package
 - common.fs package now contains all the filesystem level classes including wrapper filesystem
 - bloom.filter package renamed to just bloom
 - config package contains classes that help store properties
 - common.fs.inline package contains all the inline filesystem classes/impl
 - common.table.timeline now consolidates all timeline related classes
 - common.table.view consolidates all the classes related to filesystem view metadata
 - common.table.timeline.versioning contains all classes related to versioning of timeline
 - Fix few unit tests as a result
 - Moved the test packages around to match the source file move
 - Rename AvroUtils to TimelineMetadataUtils & minor fixes/typos
 2020-03-29 10:58:49 -07:00
leesf
07c3c5d797 [HUDI-679] Make io package Spark free (#1460) 
* [HUDI-679] Make io package Spark free
 2020-03-29 16:54:00 +08:00
Sivabalan Narayanan
ac73bdcdc3 [HUDI-430] Adding InlineFileSystem to support embedding any file format as an InlineFile (#1176) 
* Adding InlineFileSystem to support embedding any file format (parquet, hfile, etc). Supports reading the embedded file using respective readers.
 2020-03-28 12:13:35 -04:00
Suneel Marthi
04449f33fe [HUDI-743]: Remove FileIOUtils.close() (#1461) 2020-03-28 18:03:15 +08:00
Suneel Marthi
8c3001363d HUDI-479: Eliminate or Minimize use of Guava if possible (#1159) 2020-03-28 03:11:32 -04:00
Raymond Xu
1713f686f8 [MINOR] Add error message when check arguments (#1451) 2020-03-27 10:21:38 +08:00
leesf
8b0a4009a9 [HUDI-678] Make config package spark free (#1418) 2020-03-26 08:30:27 -07:00
Suneel Marthi
e101ea9bd4 [MINOR] Update DOAP with 0.5.2 Release (#1448) 2020-03-25 23:37:32 -04:00
Mathieu
5eed6c98a8 [MINOR] Fix javadoc of InsertBucket (#1445) 2020-03-25 22:25:47 +08:00
Raymond Xu
bc82e2be6c [HUDI-711] Refactor exporter main logic (#1436) 
* Refactor exporter main logic
* break main method into multiple readable methods
* fix bug of passing wrong file list
* avoid deleting output path when exists
* throw exception to early abort on multiple cases
* use JavaSparkContext instead of SparkSession
* improve unit test for expected exceptions
 2020-03-25 18:02:24 +08:00
hongdd
cafc87041b [HUDI-697]Add unit test for ArchivedCommitsCommand (#1424) 2020-03-23 13:46:10 +08:00
Zhiyuan Zhao
0241b21f77 [HUDI-65] commitTime rename to instantTime (#1431) 2020-03-22 18:06:00 -07:00
lamber-ken
38c3ccc51a [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly (#1377) 2020-03-22 10:31:48 -07:00
Pratyaksh Sharma
1e1d9e1d34 [HUDI-616] Fixed parquet files getting created on local FS (#1434) 2020-03-22 22:19:47 +08:00
Zhiyuan Zhao
06652aa935 [MINOR] Add omissive param desc on method doc and cleanup redundant code (#1437) 2020-03-22 21:39:33 +08:00
Zhiyuan Zhao
8b00791ef4 [MINOR] cleanup redundant comment and unused variable and fix typo (#1435) 2020-03-21 20:12:06 -07:00
vinoyang
c5030f77a0 [HUDI-720] NOTICE file needs to add more content based on the NOTICE files of the ASF projects that hudi bundles (#1417) 
* [HUDI-720] NOTICE file needs to add more content based on the NOTICE files of the ASF projects that hudi bundles
 2020-03-21 10:54:04 +08:00
satishkotha
83fb9651f3 [HUDI-650] Modify handleUpdate path to validate partitionPath (#1368) 2020-03-20 08:37:22 -07:00
Mathieu
eeab532d79 [HUDI-725] Remove init log in the constructor of DeltaSync (#1425) 2020-03-20 17:47:59 +08:00
Mathieu
21c45e1051 [HUDI-726]Delete unused method in HoodieDeltaStreamer (#1426) 2020-03-20 17:44:16 +08:00
Zhiyuan Zhao
14e0c95206 [HUDI-400] Check upgrade from old plan to new plan for compaction (#1422) 
* Fix NPE when DataFile is null
* Check from old plan upgrade to new plan
 2020-03-20 15:13:17 +08:00
Sivabalan Narayanan
a752b7b18c Merge pull request #1165 from yihua/HUDI-76-deltastreamer-csv-source 
[HUDI-76] Add CSV Source support for Hudi Delta Streamer
 2020-03-19 10:00:53 -04:00
ForwardXu
1e321c2fc0 [HUDI-209] Implement JMX metrics reporter (#1106) 2020-03-19 20:10:35 +08:00
Raymond Xu
779edc0688 [HUDI-344] Add partitioner param to Exporter (#1405) 2020-03-18 19:24:04 +08:00
leesf
0a4902ecce [HUDI-437] Support user-defined index (#1408) 
* [hotfix] set default value for index class config
* class config takes precedence over `hoodie.index.type`
 2020-03-17 19:27:40 -07:00
vinoth chandar
e3019031d8 [HUDI-539] Make ROPathFilter conf member serializable (#1415) 2020-03-17 12:52:48 -07:00
hongdd
f1d7bb381d [HUDI-695]Add unit test for TableCommand (#1411) 2020-03-17 14:15:30 +08:00
bschell
418f9bb2e9 Add constructor to HoodieROTablePathFilter (#1413) 
Allows HoodieROTablePathFilter to accept a configuration for
initializing the filesystem. This fixes a bug with Presto's use of this
pathfilter.

Co-authored-by: Brandon Scheller <bschelle@amazon.com>
 2020-03-16 15:19:16 -07:00
hongdd
3ef9e885ca [HUDI-715] Fix duplicate name in TableCommand (#1410) 2020-03-16 17:19:57 +08:00
hongdd
55e6d34815 [HUDI-694]Add unit test for SparkEnvCommand (#1401) 
* Add test for SparkEnvCommand
 2020-03-16 11:52:40 +08:00
Y Ethan Guo
cf765df606 [HUDI-76] Add CSV Source support for Hudi Delta Streamer 2020-03-15 19:03:37 -07:00
Balaji Varadarajan
23afe7a487 [HUDI-710] Fixing failure in Staging Validation Script (#1403) 2020-03-15 22:13:20 +08:00
Raymond Xu
14323cb100 [HUDI-344] Improve exporter tests (#1404) 2020-03-15 20:24:30 +08:00
Suneel Marthi
99b7e9eb9e [HUDI-629]: Replace Guava's Hashing with an equivalent in NumericUtils.java (#1350) 
* [HUDI-629]: Replace Guava's Hashing with an equivalent in NumericUtils.java
 2020-03-13 20:28:05 -04:00
vinoth chandar
fb7fba365f [HUDI-646] fix failing test due to improper filesytem cleanup (#1373) 2020-03-12 23:59:09 -07:00
Udit Mehrotra
c40a0d4e91 [HUDI-656][Performance] Return a dummy Spark relation after writing the DataFrame (#1394) 
Co-authored-by: Mehrotra <uditme@amazon.com>
 2020-03-11 20:27:46 -07:00
hongdd
0f892ef62c [HUDI-692] Add delete savepoint for cli (#1397) 
* Add delete savepoint for cli
* Add check
* Move JavaSparkContext to try
 2020-03-11 16:49:02 -07:00
Prashant Wason
7d66831444 [MINOR] Removing code which is duplicated from the base class HoodieWriteHandle. (#1399) 2020-03-11 16:43:04 -07:00