lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
882 Commits 1 Branch 0 Tags
deb95ad9962bd40dc3edcc66281e8d333f32bab3
Commit Graph

882 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Prashant Wason
deb95ad996 [HUDI-748] Adding .codecov.yml to set exclusions for code coverage reports. (#1468) 2020-04-03 16:25:01 -07:00
Prashant Wason
6808559b01 [HUDI-717] Fixed usage of HiveDriver for DDL statements. (#1416) 
When using HiveDriver mode in HudiHiveClient, Hive 2.x DDL operations like ALTER PARTITION may fail. This is because Hive 2.x doesn't like `db`.`table_name` for operations. In this fix, we set the name of the database in the SessionState create for the Driver.
 2020-04-03 16:23:05 -07:00
Ramachandran Madtas Subramaniam
639ec20412 [HUDI-562] Enable testing at debug log level 
This is to ensure that tests will execute all code paths, even the ones
written under DEBUG log levels. This will improve coverage as well as
ensure there are no surprised when DEBUG log level is enabled in
production.
 2020-04-02 11:14:35 -07:00
yanghua
bd716ece18 [MINIOR] Add license header for .asf.yaml and adjust labels 2020-04-02 16:14:35 +08:00
vinoyang
194e20e661 [MINOR] Fix label issue in .asf.yaml (#1478) 2020-04-02 15:51:51 +08:00
Raymond Xu
5b53b0d85e [HUDI-731] Add ChainedTransformer (#1440) 
* [HUDI-731] Add ChainedTransformer
 2020-04-01 23:21:31 +08:00
Trevor
2a611f4ad3 [HUDI-749] Fix hudi-timeline-server-bundle run_server.sh start error (#1477) 2020-04-01 22:19:54 +08:00
vinoyang
c146ca90fd [HUDI-754] Configure .asf.yaml for Hudi Github repository (#1472) 
* [HUDI-754] Configure .asf.yaml for Hudi Github repository
 2020-04-01 10:02:47 +08:00
Shaofeng Shi
78b3194e82 [HUDI-751] Fix some coding issues reported by FindBugs (#1470) 2020-03-31 21:19:32 +08:00
Edwin Guo
9ecf0ccfb2 [HUDI-742] Fix Java Math Exception (#1466) 2020-03-31 12:56:20 +08:00
wenningd
ce0a4c64d0 [HUDI-713] Fix conversion of Spark array of struct type to Avro schema (#1406) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2020-03-30 15:52:15 -07:00
lamber-ken
dbc9acd23a [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean (#1432) 2020-03-30 11:19:17 -07:00
Prashant Wason
9f51b99174 [MINOR] Updated HoodieMergeOnReadTestUtils for future testing requirements (#1456) 
1. getRecordsUsingInputFormat() can take a custom Configuration which can be used to specify HUDI table properties (e.g. <table>.consume.mode or <table>.consume.start.timestamp)
2. Fixed the return to return an empty List rather than raise an Exception if no records are found
 2020-03-30 07:36:12 -07:00
ffcchi
1f5b0c77d6 [HUDI-724] Parallelize getSmallFiles for partitions (#1421) 
Co-authored-by: Feichi Feng <feicfeng@amazon.com>
 2020-03-30 00:14:38 -07:00
Suneel Marthi
fa36082554 [HUDI-746] Reduce build warnings < 10 (#1465) 2020-03-30 11:46:52 +08:00
vinoth chandar
fad4bd377b [HUDI-745] CI should fail PRs with unapproved license files (#1464) 2020-03-29 10:59:40 -07:00
vinoth chandar
e057c27603 [HUDI-744] Restructure hudi-common and clean up files under util packages (#1462) 
- Brings more order and cohesion to the classes in hudi-common
 - Utils classes related to a particular concept (avro, timeline,...) are placed near to the package
 - common.fs package now contains all the filesystem level classes including wrapper filesystem
 - bloom.filter package renamed to just bloom
 - config package contains classes that help store properties
 - common.fs.inline package contains all the inline filesystem classes/impl
 - common.table.timeline now consolidates all timeline related classes
 - common.table.view consolidates all the classes related to filesystem view metadata
 - common.table.timeline.versioning contains all classes related to versioning of timeline
 - Fix few unit tests as a result
 - Moved the test packages around to match the source file move
 - Rename AvroUtils to TimelineMetadataUtils & minor fixes/typos
 2020-03-29 10:58:49 -07:00
leesf
07c3c5d797 [HUDI-679] Make io package Spark free (#1460) 
* [HUDI-679] Make io package Spark free
 2020-03-29 16:54:00 +08:00
Sivabalan Narayanan
ac73bdcdc3 [HUDI-430] Adding InlineFileSystem to support embedding any file format as an InlineFile (#1176) 
* Adding InlineFileSystem to support embedding any file format (parquet, hfile, etc). Supports reading the embedded file using respective readers.
 2020-03-28 12:13:35 -04:00
Suneel Marthi
04449f33fe [HUDI-743]: Remove FileIOUtils.close() (#1461) 2020-03-28 18:03:15 +08:00
Suneel Marthi
8c3001363d HUDI-479: Eliminate or Minimize use of Guava if possible (#1159) 2020-03-28 03:11:32 -04:00
Raymond Xu
1713f686f8 [MINOR] Add error message when check arguments (#1451) 2020-03-27 10:21:38 +08:00
leesf
8b0a4009a9 [HUDI-678] Make config package spark free (#1418) 2020-03-26 08:30:27 -07:00
Suneel Marthi
e101ea9bd4 [MINOR] Update DOAP with 0.5.2 Release (#1448) 2020-03-25 23:37:32 -04:00
Mathieu
5eed6c98a8 [MINOR] Fix javadoc of InsertBucket (#1445) 2020-03-25 22:25:47 +08:00
Raymond Xu
bc82e2be6c [HUDI-711] Refactor exporter main logic (#1436) 
* Refactor exporter main logic
* break main method into multiple readable methods
* fix bug of passing wrong file list
* avoid deleting output path when exists
* throw exception to early abort on multiple cases
* use JavaSparkContext instead of SparkSession
* improve unit test for expected exceptions
 2020-03-25 18:02:24 +08:00
hongdd
cafc87041b [HUDI-697]Add unit test for ArchivedCommitsCommand (#1424) 2020-03-23 13:46:10 +08:00
Zhiyuan Zhao
0241b21f77 [HUDI-65] commitTime rename to instantTime (#1431) 2020-03-22 18:06:00 -07:00
lamber-ken
38c3ccc51a [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly (#1377) 2020-03-22 10:31:48 -07:00
Pratyaksh Sharma
1e1d9e1d34 [HUDI-616] Fixed parquet files getting created on local FS (#1434) 2020-03-22 22:19:47 +08:00
Zhiyuan Zhao
06652aa935 [MINOR] Add omissive param desc on method doc and cleanup redundant code (#1437) 2020-03-22 21:39:33 +08:00
Zhiyuan Zhao
8b00791ef4 [MINOR] cleanup redundant comment and unused variable and fix typo (#1435) 2020-03-21 20:12:06 -07:00
vinoyang
c5030f77a0 [HUDI-720] NOTICE file needs to add more content based on the NOTICE files of the ASF projects that hudi bundles (#1417) 
* [HUDI-720] NOTICE file needs to add more content based on the NOTICE files of the ASF projects that hudi bundles
 2020-03-21 10:54:04 +08:00
satishkotha
83fb9651f3 [HUDI-650] Modify handleUpdate path to validate partitionPath (#1368) 2020-03-20 08:37:22 -07:00
Mathieu
eeab532d79 [HUDI-725] Remove init log in the constructor of DeltaSync (#1425) 2020-03-20 17:47:59 +08:00
Mathieu
21c45e1051 [HUDI-726]Delete unused method in HoodieDeltaStreamer (#1426) 2020-03-20 17:44:16 +08:00
Zhiyuan Zhao
14e0c95206 [HUDI-400] Check upgrade from old plan to new plan for compaction (#1422) 
* Fix NPE when DataFile is null
* Check from old plan upgrade to new plan
 2020-03-20 15:13:17 +08:00
Sivabalan Narayanan
a752b7b18c Merge pull request #1165 from yihua/HUDI-76-deltastreamer-csv-source 
[HUDI-76] Add CSV Source support for Hudi Delta Streamer
 2020-03-19 10:00:53 -04:00
ForwardXu
1e321c2fc0 [HUDI-209] Implement JMX metrics reporter (#1106) 2020-03-19 20:10:35 +08:00
Raymond Xu
779edc0688 [HUDI-344] Add partitioner param to Exporter (#1405) 2020-03-18 19:24:04 +08:00
leesf
0a4902ecce [HUDI-437] Support user-defined index (#1408) 
* [hotfix] set default value for index class config
* class config takes precedence over `hoodie.index.type`
 2020-03-17 19:27:40 -07:00
vinoth chandar
e3019031d8 [HUDI-539] Make ROPathFilter conf member serializable (#1415) 2020-03-17 12:52:48 -07:00
hongdd
f1d7bb381d [HUDI-695]Add unit test for TableCommand (#1411) 2020-03-17 14:15:30 +08:00
bschell
418f9bb2e9 Add constructor to HoodieROTablePathFilter (#1413) 
Allows HoodieROTablePathFilter to accept a configuration for
initializing the filesystem. This fixes a bug with Presto's use of this
pathfilter.

Co-authored-by: Brandon Scheller <bschelle@amazon.com>
 2020-03-16 15:19:16 -07:00
hongdd
3ef9e885ca [HUDI-715] Fix duplicate name in TableCommand (#1410) 2020-03-16 17:19:57 +08:00
hongdd
55e6d34815 [HUDI-694]Add unit test for SparkEnvCommand (#1401) 
* Add test for SparkEnvCommand
 2020-03-16 11:52:40 +08:00
Y Ethan Guo
cf765df606 [HUDI-76] Add CSV Source support for Hudi Delta Streamer 2020-03-15 19:03:37 -07:00
Balaji Varadarajan
23afe7a487 [HUDI-710] Fixing failure in Staging Validation Script (#1403) 2020-03-15 22:13:20 +08:00
Raymond Xu
14323cb100 [HUDI-344] Improve exporter tests (#1404) 2020-03-15 20:24:30 +08:00
Suneel Marthi
99b7e9eb9e [HUDI-629]: Replace Guava's Hashing with an equivalent in NumericUtils.java (#1350) 
* [HUDI-629]: Replace Guava's Hashing with an equivalent in NumericUtils.java
 2020-03-13 20:28:05 -04:00