lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
980 Commits 1 Branch 0 Tags
1a9fef48e767c2bb56b9fba5cac48540c5d6c6c1
Commit Graph

980 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Sivabalan Narayanan
1a9fef48e7 Fixing binary file checks and disclaimer file checks (#1691) 2020-05-30 20:09:21 -07:00
Sivabalan Narayanan
7fb1251c6e [HUDI-980] Fixing dir path for metastore_db and derby.logs generated in hive tests (#1688) 2020-05-29 20:42:20 -04:00
Sivabalan Narayanan
5a0d3f1cf9 [HUDI-786] Fixing read beyond inline length in InlineFS (#1616) 2020-05-28 12:59:11 -07:00
cxzl25
3574a89232 [HUDI-973] RemoteHoodieTableFileSystemView supports non-partitioned table queries (#1674) 2020-05-28 10:50:47 -07:00
dengziming
bde7a7043e [HUDI-476]: Add hudi-examples module (#1151) 
add hoodie delta streamer mock source example and dfs source and kafka source examples

Signed-off-by: dengziming <dengziming1993@gmail.com>

add defaultSparkConf utils method

change version of hudi-examples to 0.5.2-SNAPSHOT
change the artifcatId of hudi-spark and hudi-utilities
alter some code to adapt kafka2.0

Update scritps

Add license
 2020-05-28 01:44:39 +08:00
Raymond Xu
03f136361a [HUDI-811] Restructure test packages in hudi-common (#1644) 
* [HUDI-811] Restructure test packages in hudi-common
 2020-05-27 16:28:17 +08:00
Bhavani Sudha Saktheeswaran
9697fbf71e [HUDI-936] Fix fetch ordering val in HoodieSparkSqlWriter to remove unnecessary conversion to String (#1659) 2020-05-26 21:09:02 -07:00
Raymond Xu
6c450957ce [HUDI-690] Filter out inflight compaction in exporter (#1667) 2020-05-26 09:23:34 -07:00
vinoth chandar
46572d04c4 [MINOR] Fix PMC tag in doap file (#1666) 2020-05-25 09:12:29 -07:00
leesf
492f324bc7 [HUDI-938] Removing incubating/incubator from project (#1658) 2020-05-24 18:28:13 +08:00
leesf
f22c3e933e [HUDI-926] Removing DISCLAIMER from the repo (#1657) 2020-05-24 18:27:08 +08:00
sathyaprakashg
d3edac4612 HUDI-921 Remove inlineCompactionEvery method in HoodieCompactionConfig.Builder (#1654) 
Co-authored-by: Sathyaprakash Govindasamy <sathyaprakashg@zillowgroup.com>
 2020-05-24 01:09:18 -07:00
vinoth chandar
45acccdb8a [MINOR] Remove incubating from README 2020-05-23 14:51:58 -07:00
Raymond Xu
f34de3fb27 [HUDI-836] Implement datadog metrics reporter (#1572) 
- Adds support for emitting metrics to datadog
- Tests, configs..
 2020-05-22 09:14:21 -07:00
hongdd
802d16c8c9 [HUDI-707] Add unit test for StatsCommand (#1645) 2020-05-21 18:28:04 +08:00
Pratyaksh Sharma
6a0aa9a645 [HUDI-803] Replaced used of NullNode with JsonProperties.NULL_VALUE in HoodieAvroUtils (#1538) 
- added more test cases in TestHoodieAvroUtils.class

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2020-05-20 09:04:43 -07:00
Balaji Varadarajan
74ecc27e92 [HUDI-846][HUDI-848] Enable Incremental cleaning and embedded timeline-server by default (#1634) 2020-05-20 05:29:43 -07:00
Raymond Xu
f802d4400b [MINOR] Fix resource cleanup in TestTableSchemaEvolution (#1640) 
- Remove Xms it is not needed. 
- extending process exit timeout from 30 to 120 sec should be safe to do
 2020-05-20 05:07:30 -07:00
rolandjohann
244d47494e [HUDI-888] fix NullPointerException in HoodieCompactor (#1622) 2020-05-20 04:22:35 -07:00
wenningd
0dc2fa6172 [MINOR] Fix HoodieCompactor config abbreviation (#1642) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2020-05-19 21:03:54 -07:00
hongdd
161a798337 [HUDI-706] Add unit test for SavepointsCommand (#1624) 2020-05-19 18:36:01 +08:00
Balaji Varadarajan
e6f3bf10cf [HUDI-858] Allow multiple operations to be executed within a single commit (#1633) 2020-05-18 19:27:24 -07:00
Joey
2600d2de8d [MINOR] Fix apache-rat violations (#1639) 
* MINOR Fix apache-rat violations. Also, enabling RAT for hudi-utilities and hudi-integ-test
 2020-05-18 11:16:49 -07:00
rolandjohann
459356e292 [HUDI-863] get decimal properties from derived spark DataType (#1596) 2020-05-18 04:28:27 -07:00
hongdd
57132f79bb [HUDI-705] Add unit test for RollbacksCommand (#1611) 2020-05-18 14:04:06 +08:00
Sivabalan Narayanan
29edf4b3b8 [HUDI-407] Adding Simple Index to Hoodie. (#1402) 
This index finds the location by joining incoming records with records from base files.
 2020-05-17 18:32:24 -07:00
Balaji Varadarajan
3c9da2e5f0 [HUDI-895] Remove unnecessary listing .hoodie folder when using timeline server (#1636) 2020-05-17 18:18:53 -07:00
Mathieu
25a0080b2f [HUDI-714]Add javadoc and comments to hudi write method link (#1409) 
* [HUDI-714] Add javadoc and comments to hudi write method link
 2020-05-16 08:36:51 -04:00
Raymond Xu
148b2458f6 [MINOR] Increase heap space for surefire (#1623) 2020-05-16 01:11:57 -07:00
Raymond Xu
2ada2ef50f [HUDI-902] Avoid exception when getSchemaProvider (#1584) 
* When no new input data, don't throw exception for null SchemaProvider
* Return the newly added NullSchemaProvider instead
 2020-05-15 21:33:02 -07:00
Alexander Filipchik
25e0b75b3d [HUDI-723] Register avro schema if infered from SQL transformation (#1518) 
* Register avro schema if infered from SQL transformation
* Make HoodieWriteClient creation done lazily always. Handle setting schema-provider and avro-schemas correctly when using SQL transformer

Co-authored-by: Alex Filipchik <alex.filipchik@csscompany.com>
Co-authored-by: Balaji Varadarajan <varadarb@uber.com>
 2020-05-15 12:44:03 -07:00
Gary Li
a64afdfd17 HUDI-528 Handle empty commit in incremental pulling (#1612) 2020-05-14 22:55:25 -07:00
Alexander Filipchik
f094f42857 [HUDI-843] Add ability to specify time unit for TimestampBasedKeyGenerator (#1541) 
Co-authored-by: Alex Filipchik <alex.filipchik@csscompany.com>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2020-05-14 13:37:59 -07:00
hongdd
3a2fe13fcb [HUDI-701] Add unit test for HDFSParquetImportCommand (#1574) 2020-05-14 19:15:49 +08:00
Alexander Filipchik
83796b3189 [HUDI-793] Adding proper default to hudi metadata fields and proper handling to rewrite routine (#1513) 
* Adding proper default to hudi metadata fields and proper handling to rewrite routine
* Handle fields declared with a null default

Co-authored-by: Alex Filipchik <alex.filipchik@csscompany.com>
 2020-05-13 18:04:38 -07:00
Raymond Xu
0d4848b68b [HUDI-811] Restructure test packages (#1607) 
* restructure hudi-spark tests
* restructure hudi-timeline-service tests
* restructure hudi-hadoop-mr hudi-utilities tests
* restructure hudi-hive-sync tests
 2020-05-13 15:37:03 -07:00
cxzl25
32bada29dc [HUDI-889] Writer supports useJdbc configuration when hive synchronization is enabled (#1627) 2020-05-14 00:20:13 +08:00
liujinhui
32ea4c70ff [HUDI-869] Add support for alluxio (#1608) 2020-05-13 21:00:34 +08:00
Udit Mehrotra
404c7e82d9 [HUDI-884] Shade avro and parquet-avro in hudi-hive-sync-bundle (#1618) 
Co-authored-by: Mehrotra <uditme@amazon.com>
 2020-05-12 11:40:31 -07:00
Shen Hong
e8ffc6f0aa [HUDI-881] Replace part of spark context by hadoop configuration in AbstractHoodieClient and HoodieReadClient (#1620) 2020-05-12 09:33:29 -07:00
Shen Hong
b54517aad0 [HUDI-886] Replace jsc.hadoopConfiguration by hadoop configuration in hudi-client testcase (#1621) 2020-05-12 08:51:31 -07:00
Shen Hong
295d00beea [HUDI-880] Replace part of spark context by hadoop configuration in HoodieTable. (#1614) 2020-05-11 23:33:57 -07:00
liujinhui
5d37e66b7e [MINOR] Fix HoodieNotSupportedException description in KafkaOffsetGen (#1615) 2020-05-11 23:14:43 +08:00
Shen Hong
6dac10115c [HUDI-870] Remove spark context in ClientUtils and HoodieIndex (#1609) 2020-05-11 19:05:36 +08:00
Balaji Varadarajan
8d0e23173b [HUDI-820] cleaner repair command should only inspect clean metadata files (#1542) 2020-05-11 09:25:54 +08:00
vinoth chandar
f92b9fdcc4 [MINOR] Fix hardcoding of ports in TestHoodieJmxMetrics (#1606) 2020-05-10 19:23:26 -04:00
Carm
fa6aba751d [MINOR] fixed building IndexFileFilter with a wrong condition in HoodieGlobalBloomIndex class (#1537) 2020-05-10 09:45:07 +08:00
Udit Mehrotra
d54b4b8a52 [HUDI-838] Support schema from HoodieCommitMetadata for HiveSync (#1559) 
Co-authored-by: Mehrotra <uditme@amazon.com>
 2020-05-07 16:33:09 -07:00
Alexander Filipchik
e783ab1749 [HUDI-784] Adressing issue with log reader on GCS (#1516) 
[HUDI-784] Adressing issue with log reader on GCS (#1516)
Co-authored-by: Alex Filipchik <alex.filipchik@csscompany.com>
 2020-05-07 13:05:32 -07:00
hongdd
f921469afc [HUDI-704] Add test for RepairsCommand (#1554) 2020-05-07 23:02:28 +08:00