lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
857 Commits 1 Branch 0 Tags
bc82e2be6cf080ab99092758368e91f509a2004c
Commit Graph

857 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Raymond Xu
bc82e2be6c [HUDI-711] Refactor exporter main logic (#1436) 
* Refactor exporter main logic
* break main method into multiple readable methods
* fix bug of passing wrong file list
* avoid deleting output path when exists
* throw exception to early abort on multiple cases
* use JavaSparkContext instead of SparkSession
* improve unit test for expected exceptions
 2020-03-25 18:02:24 +08:00
hongdd
cafc87041b [HUDI-697]Add unit test for ArchivedCommitsCommand (#1424) 2020-03-23 13:46:10 +08:00
Zhiyuan Zhao
0241b21f77 [HUDI-65] commitTime rename to instantTime (#1431) 2020-03-22 18:06:00 -07:00
lamber-ken
38c3ccc51a [HUDI-663] Fix HoodieDeltaStreamer offset not handled correctly (#1377) 2020-03-22 10:31:48 -07:00
Pratyaksh Sharma
1e1d9e1d34 [HUDI-616] Fixed parquet files getting created on local FS (#1434) 2020-03-22 22:19:47 +08:00
Zhiyuan Zhao
06652aa935 [MINOR] Add omissive param desc on method doc and cleanup redundant code (#1437) 2020-03-22 21:39:33 +08:00
Zhiyuan Zhao
8b00791ef4 [MINOR] cleanup redundant comment and unused variable and fix typo (#1435) 2020-03-21 20:12:06 -07:00
vinoyang
c5030f77a0 [HUDI-720] NOTICE file needs to add more content based on the NOTICE files of the ASF projects that hudi bundles (#1417) 
* [HUDI-720] NOTICE file needs to add more content based on the NOTICE files of the ASF projects that hudi bundles
 2020-03-21 10:54:04 +08:00
satishkotha
83fb9651f3 [HUDI-650] Modify handleUpdate path to validate partitionPath (#1368) 2020-03-20 08:37:22 -07:00
Mathieu
eeab532d79 [HUDI-725] Remove init log in the constructor of DeltaSync (#1425) 2020-03-20 17:47:59 +08:00
Mathieu
21c45e1051 [HUDI-726]Delete unused method in HoodieDeltaStreamer (#1426) 2020-03-20 17:44:16 +08:00
Zhiyuan Zhao
14e0c95206 [HUDI-400] Check upgrade from old plan to new plan for compaction (#1422) 
* Fix NPE when DataFile is null
* Check from old plan upgrade to new plan
 2020-03-20 15:13:17 +08:00
Sivabalan Narayanan
a752b7b18c Merge pull request #1165 from yihua/HUDI-76-deltastreamer-csv-source 
[HUDI-76] Add CSV Source support for Hudi Delta Streamer
 2020-03-19 10:00:53 -04:00
ForwardXu
1e321c2fc0 [HUDI-209] Implement JMX metrics reporter (#1106) 2020-03-19 20:10:35 +08:00
Raymond Xu
779edc0688 [HUDI-344] Add partitioner param to Exporter (#1405) 2020-03-18 19:24:04 +08:00
leesf
0a4902ecce [HUDI-437] Support user-defined index (#1408) 
* [hotfix] set default value for index class config
* class config takes precedence over `hoodie.index.type`
 2020-03-17 19:27:40 -07:00
vinoth chandar
e3019031d8 [HUDI-539] Make ROPathFilter conf member serializable (#1415) 2020-03-17 12:52:48 -07:00
hongdd
f1d7bb381d [HUDI-695]Add unit test for TableCommand (#1411) 2020-03-17 14:15:30 +08:00
bschell
418f9bb2e9 Add constructor to HoodieROTablePathFilter (#1413) 
Allows HoodieROTablePathFilter to accept a configuration for
initializing the filesystem. This fixes a bug with Presto's use of this
pathfilter.

Co-authored-by: Brandon Scheller <bschelle@amazon.com>
 2020-03-16 15:19:16 -07:00
hongdd
3ef9e885ca [HUDI-715] Fix duplicate name in TableCommand (#1410) 2020-03-16 17:19:57 +08:00
hongdd
55e6d34815 [HUDI-694]Add unit test for SparkEnvCommand (#1401) 
* Add test for SparkEnvCommand
 2020-03-16 11:52:40 +08:00
Y Ethan Guo
cf765df606 [HUDI-76] Add CSV Source support for Hudi Delta Streamer 2020-03-15 19:03:37 -07:00
Balaji Varadarajan
23afe7a487 [HUDI-710] Fixing failure in Staging Validation Script (#1403) 2020-03-15 22:13:20 +08:00
Raymond Xu
14323cb100 [HUDI-344] Improve exporter tests (#1404) 2020-03-15 20:24:30 +08:00
Suneel Marthi
99b7e9eb9e [HUDI-629]: Replace Guava's Hashing with an equivalent in NumericUtils.java (#1350) 
* [HUDI-629]: Replace Guava's Hashing with an equivalent in NumericUtils.java
 2020-03-13 20:28:05 -04:00
vinoth chandar
fb7fba365f [HUDI-646] fix failing test due to improper filesytem cleanup (#1373) 2020-03-12 23:59:09 -07:00
Udit Mehrotra
c40a0d4e91 [HUDI-656][Performance] Return a dummy Spark relation after writing the DataFrame (#1394) 
Co-authored-by: Mehrotra <uditme@amazon.com>
 2020-03-11 20:27:46 -07:00
hongdd
0f892ef62c [HUDI-692] Add delete savepoint for cli (#1397) 
* Add delete savepoint for cli
* Add check
* Move JavaSparkContext to try
 2020-03-11 16:49:02 -07:00
Prashant Wason
7d66831444 [MINOR] Removing code which is duplicated from the base class HoodieWriteHandle. (#1399) 2020-03-11 16:43:04 -07:00
satishkotha
7194514aff [HUDI-689] Change CLI command names to not have overlap (#1392) 2020-03-11 16:29:54 -07:00
Sivabalan Narayanan
1ca912af09 [HUDI-667] Fixing delete tests for DeltaStreamer (#1395) 2020-03-11 16:19:23 -07:00
vinoth chandar
dd7cf38a13 [HUDI-688] Paring down the NOTICE file to minimum required notices (#1391) 
- Based on analysis, we don't need to call out anything
 - We only do source releases at this time
 - Fix typo in LICENSE
 2020-03-11 08:24:07 -04:00
Prashant Wason
cf0a4c19bc [HUDI-670] Added test cases for TestDiskBasedMap. (#1379) 
* [HUDI-670] Added test cases for TestDiskBasedMap.

* Update TestDiskBasedMap.java

Co-authored-by: Suneel Marthi <smarthi@apache.org>
 2020-03-11 08:03:03 -04:00
Prashant Wason
77d5b92d88 [HUDI-668] Added additional unit-tests for HUDI metrics. (#1380) 2020-03-09 23:15:42 -04:00
openopen2
44700d531a [HUDI-344] Hudi Dataset Snapshot Exporter (#1360) 
Co-authored-by: jason1993 <261049174@qq.com>
 2020-03-10 09:17:51 +08:00
hongdd
f93e64fee4 [HUDI-681]Remove embeddedTimelineService from HoodieReadClient (#1388) 
* [HUDI-681]Remove embeddedTimelineService from HoodieReadClient
 2020-03-09 18:31:04 +08:00
vinoyang
2137ecfa22 [HUDI-676] Address issues towards removing use of WIP Disclaimer (#1386) 
* rename DISCLAIMER-STANDARD TO DISCLAIMER
 2020-03-08 23:14:49 -07:00
Suneel Marthi
415882f902 [HUDI-581] NOTICE need more work as it missing content form included 3rd party ALv2 licensed NOTICE files (#1354) 
* [HUDI-581] - Add 3rd party library NOTICE

* [HUDI-581]: NOTICE need more work as it missing content form included 3rd party ALv2 licensed NOTICE files
 2020-03-07 22:08:35 -05:00
Prashant Wason
5f8bf97005 [HUDI-671] Added unit-test for HBaseIndex (#1381) 2020-03-07 16:48:43 -08:00
Aki Tanaka
fdcd3b18b6 [HUDI-680] Update Jackson databind to 2.6.7.3 (#1385) 2020-03-07 17:22:19 -05:00
lamber-ken
90227eeda7 [HUDI-673] Rename hudi-hive-bundle to hudi-hive-sync-bundle 2020-03-07 21:44:35 +08:00
lamber-ken
170ee88457 [HUDI-553] Building/Running Hudi on higher java versions (#1369) 2020-03-07 01:27:40 -08:00
vinoyang
ee5b32f5d4 [HUDI-652] Decouple HoodieReadClient and AbstractHoodieClient to break the inheritance chain (#1372) 
* Removed timeline server support
* Removed try-with-resource
 2020-03-06 09:59:35 -08:00
Satish Kotha
3d3781810c [CLI] Add export to table 2020-03-06 08:53:23 -08:00
lamber-ken
ccbf543607 [HUDI-654] Rename hudi-hive to hudi-hive-sync 2020-03-06 22:13:16 +08:00
Bhavani Sudha Saktheeswaran
5f85c26704 [HUDI-584] Relocate spark-avro dependency by maven-shade-plugin (#1290) 2020-03-04 11:01:49 -08:00
Ramachandran M S
9d46ce380a [HUDI -409] Match header and footer block length to improve corrupted block detection (#1332) 2020-03-03 13:26:54 -08:00
hongdd
8306205d7a [HUDI-332]Add operation type (insert/upsert/bulkinsert/delete) to HoodieCommitMetadata (#1157) 
[HUDI-332]Add operation type (insert/upsert/bulkinsert/delete) to HoodieCommitMetadata (#1157)
 2020-03-03 10:10:29 -08:00
Udit Mehrotra
2d04014581 [HUDI-607] Fix to allow creation/syncing of Hive tables partitioned by Date type columns (#1330) 2020-03-01 10:42:58 -08:00
yanghua
0dc8e493aa Moving to 0.6.0-SNAPSHOT on master branch. 2020-03-01 15:08:30 +08:00