lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,017 Commits 1 Branch 0 Tags
3b9a30528bd6a6369181702303f3384162b04a7f
Commit Graph

1017 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Raymond Xu
3b9a30528b [HUDI-996] Add functional test suite for hudi-utilities (#1746) 
- Share resources for functional tests
- Add suite for functional test classes from hudi-utilities
 2020-07-05 16:44:31 -07:00
Cory Locklear
574dcf920c [MINOR] Relocate jetty during shading/packaging for Databricks runtime (#1781) 2020-07-03 16:22:52 -07:00
andreitaleanu
37ea79566d [HUDI-539] Make HoodieROTablePathFilter implement Configurable (#1784) 
Co-authored-by: Andrei Taleanu <taleanu@adobe.com>
 2020-07-03 13:39:53 -07:00
baobaoyeye
2be924fd3a [HUDI-760]Remove Rolling Stat management from Hudi Writer (#1739) 2020-06-30 20:07:09 -07:00
Balaji Varadarajan
8919be6a5d [HUDI-855] Run Cleaner async with writing (#1577) 
- Cleaner can now run concurrently with write operation 
- Configs to turn on/off

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2020-06-28 02:04:50 -07:00
Raymond Xu
31247e9b34 [HUDI-896] Report test coverage by modules & parallelize CI (#1753) 
- use codecov flags for each module to report coverage
- parallelize CI jobs for shorter time
- add a testcase for MetricsReporterFactory (to trigger codecov comment)
 2020-06-27 23:16:12 -07:00
Prashant Wason
2603cfb33e [HUDI-684] Introduced abstraction for writing and reading different types of base file formats. (#1687) 
Notable changes:
    1. HoodieFileWriter and HoodieFileReader abstractions for writer/reader side of a base file format
    2. HoodieDataBlock abstraction for creation specific data blocks for base file formats. (e.g. Parquet has HoodieAvroDataBlock)
    3. All hardocded references to Parquet / Parquet based classes have been abstracted to call methods which accept a base file format
    4. HiveSyncTool accepts the base file format as a CLI parameter
    5. HoodieDeltaStreamer accepts the base file format as a CLI parameter
    6. HoodieSparkSqlWriter accepts the base file format as a parameter
 2020-06-25 23:46:55 -07:00
wangxianghu
5e47673341 [HUDI-1035] Remove unused class KeyLookupResult (#1754) 2020-06-23 17:01:03 -07:00
Shen Hong
89e37d5273 [HUDI-908] Add some data types to HoodieTestDataGenerator and fix some some bugs. (#1690) 2020-06-22 08:13:28 -07:00
wangxianghu
68a656b016 [HUDI-1032] Remove unused code in HoodieCopyOnWriteTable and code clean (#1750) 2020-06-21 07:34:47 -07:00
Raymond Xu
8a9fdd603e [HUDI-1023] Add validation error messages in delta sync (#1710) 
- Remove explicitly specifying BLOOM_INDEX since thats the default anyway
 2020-06-19 12:12:35 -07:00
Raymond Xu
ab724af5c4 [MINOR] Rename TestSourceConfig to SourceConfigs (#1749) 2020-06-19 12:08:19 -07:00
hongdd
f3a701757b [HUDI-696] Add unit test for CommitsCommand (#1724) 2020-06-18 21:42:13 +08:00
hongdd
5099a91edd [HUDI-709] Add unit test for UtilsCommand (#1686) 2020-06-18 19:54:14 +08:00
Sivabalan Narayanan
2a04647f5e [MINOR] Updating doap file for 0.5.3 release (#1740) 2020-06-16 12:47:30 -07:00
Yajun Luo
043eb564c2 [HUDI-1003] Handle partitions correctly for syncing hudi non-parititioned table to hive (#1720) 2020-06-15 19:02:03 +08:00
Litianye
ede6c9bda4 [HUDI-1006] Deltastreamer use kafkaSource with offset reset strategy:latest can't consume data (#1719) 2020-06-14 18:01:44 +08:00
vinoyang
31ef4acc59 [MINOR] Fix the ordered list for the hudi-examples README file (#1733) 2020-06-14 16:27:26 +08:00
hongdd
fcabc8fbca [HUDI-1019] Clean refresh command in CLI (#1725) 2020-06-14 14:30:28 +08:00
Satish Kotha
a7fd331624 Add unit test for snapshot reads in hadoop-mr 2020-06-13 10:23:05 -07:00
sathyaprakashg
df2e0c760e HUDI-942 Increase default value number of delta commits for inline compaction (#1664) 
Co-authored-by: Sathyaprakash Govindasamy <sathyaprakashg@zillowgroup.com>
 2020-06-10 16:16:44 -07:00
Gary Li
37838cea60 [HUDI-822] decouple Hudi related logics from HoodieInputFormat (#1592) 
- Refactoring business logic out of InputFormat into Utils helpers.
 2020-06-09 06:10:16 -07:00
shenhong
3387b3841f [HUDI-1005] fix NPE in HoodieWriteClient.clean 2020-06-09 05:57:04 -07:00
Shen Hong
6318e943d1 [HUDI-1016] Code optimization in MergeOnReadRollbackActionExecutor(#1718) 2020-06-09 19:14:26 +08:00
garyli1019
22cd824d99 HUDI-494 fix incorrect record size estimation 2020-06-08 20:29:29 -07:00
lw0090
9e07cebece [HUDI-974] Fix fields out of order in MOR mode when using Hive (#1711) 2020-06-09 09:22:06 +08:00
Wenning Ding
7d40f19f39 HUDI-515 Resolve API conflict for Hive 2 & Hive 3 2020-06-08 14:18:38 -07:00
liujinhui
97ab97b726 [HUDI-918] Fix kafkaOffsetGen can not read kafka data bug (#1652) 2020-06-08 20:46:47 +08:00
Shen Hong
2901f5423a [HUDI-1002] Ignore case when setting incremental mode in hive query (#1715) 2020-06-08 19:38:32 +08:00
hj2016
e0a5e0d343 [HUDI-1000] Fix incremental query for COW non-partitioned table with no data (#1708) 2020-06-08 15:34:42 +08:00
garyli1019
e9cab67b80 [HUDI-988] Fix More Unit Test Flakiness 2020-06-07 23:14:46 -07:00
Balaji Varadarajan
fb283934a3 [HUDI-990] Timeline API : filterCompletedAndCompactionInstants needs to handle requested state correctly. Also ensure timeline gets reloaded after we revert committed transactions 2020-06-04 02:52:21 -07:00
Balaji Varadarajan
a68180b179 [HUDI-988] Fix Unit Test Flakiness : Ensure all instantiations of HoodieWriteClient is closed properly. Fix bug in TestRollbacks. Make CLI unit tests for Hudi CLI check skip redering strings 2020-06-04 02:52:21 -07:00
Yajun Luo
a9a97d6af4 [HUDI-934] Add processing logic for the decimal LogicalType (#1677) 2020-06-02 19:50:55 +08:00
Raymond Xu
742c204099 [HUDI-811] Restructure test packages in hudi-client/cli (#1689) 2020-06-02 10:25:42 +08:00
cxzl25
7c59095314 [HUDI-975] Add unit tests in TestHoodieTableFileSystemView to test view for non-partitioned table (#1692) 2020-06-01 07:23:28 -07:00
Vinoth Govindarajan
8cb86b4d36 Added python3 to the spark_base docker image to support pyspark (#1632) 2020-05-31 22:53:50 -07:00
Sivabalan Narayanan
1a9fef48e7 Fixing binary file checks and disclaimer file checks (#1691) 2020-05-30 20:09:21 -07:00
Sivabalan Narayanan
7fb1251c6e [HUDI-980] Fixing dir path for metastore_db and derby.logs generated in hive tests (#1688) 2020-05-29 20:42:20 -04:00
Sivabalan Narayanan
5a0d3f1cf9 [HUDI-786] Fixing read beyond inline length in InlineFS (#1616) 2020-05-28 12:59:11 -07:00
cxzl25
3574a89232 [HUDI-973] RemoteHoodieTableFileSystemView supports non-partitioned table queries (#1674) 2020-05-28 10:50:47 -07:00
dengziming
bde7a7043e [HUDI-476]: Add hudi-examples module (#1151) 
add hoodie delta streamer mock source example and dfs source and kafka source examples

Signed-off-by: dengziming <dengziming1993@gmail.com>

add defaultSparkConf utils method

change version of hudi-examples to 0.5.2-SNAPSHOT
change the artifcatId of hudi-spark and hudi-utilities
alter some code to adapt kafka2.0

Update scritps

Add license
 2020-05-28 01:44:39 +08:00
Raymond Xu
03f136361a [HUDI-811] Restructure test packages in hudi-common (#1644) 
* [HUDI-811] Restructure test packages in hudi-common
 2020-05-27 16:28:17 +08:00
Bhavani Sudha Saktheeswaran
9697fbf71e [HUDI-936] Fix fetch ordering val in HoodieSparkSqlWriter to remove unnecessary conversion to String (#1659) 2020-05-26 21:09:02 -07:00
Raymond Xu
6c450957ce [HUDI-690] Filter out inflight compaction in exporter (#1667) 2020-05-26 09:23:34 -07:00
vinoth chandar
46572d04c4 [MINOR] Fix PMC tag in doap file (#1666) 2020-05-25 09:12:29 -07:00
leesf
492f324bc7 [HUDI-938] Removing incubating/incubator from project (#1658) 2020-05-24 18:28:13 +08:00
leesf
f22c3e933e [HUDI-926] Removing DISCLAIMER from the repo (#1657) 2020-05-24 18:27:08 +08:00
sathyaprakashg
d3edac4612 HUDI-921 Remove inlineCompactionEvery method in HoodieCompactionConfig.Builder (#1654) 
Co-authored-by: Sathyaprakash Govindasamy <sathyaprakashg@zillowgroup.com>
 2020-05-24 01:09:18 -07:00
vinoth chandar
45acccdb8a [MINOR] Remove incubating from README 2020-05-23 14:51:58 -07:00