lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
968 Commits 1 Branch 0 Tags
45acccdb8a1aadc8aa499f0eb564f98247880357
Commit Graph

968 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
vinoth chandar
45acccdb8a [MINOR] Remove incubating from README 2020-05-23 14:51:58 -07:00
Raymond Xu
f34de3fb27 [HUDI-836] Implement datadog metrics reporter (#1572) 
- Adds support for emitting metrics to datadog
- Tests, configs..
 2020-05-22 09:14:21 -07:00
hongdd
802d16c8c9 [HUDI-707] Add unit test for StatsCommand (#1645) 2020-05-21 18:28:04 +08:00
Pratyaksh Sharma
6a0aa9a645 [HUDI-803] Replaced used of NullNode with JsonProperties.NULL_VALUE in HoodieAvroUtils (#1538) 
- added more test cases in TestHoodieAvroUtils.class

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2020-05-20 09:04:43 -07:00
Balaji Varadarajan
74ecc27e92 [HUDI-846][HUDI-848] Enable Incremental cleaning and embedded timeline-server by default (#1634) 2020-05-20 05:29:43 -07:00
Raymond Xu
f802d4400b [MINOR] Fix resource cleanup in TestTableSchemaEvolution (#1640) 
- Remove Xms it is not needed. 
- extending process exit timeout from 30 to 120 sec should be safe to do
 2020-05-20 05:07:30 -07:00
rolandjohann
244d47494e [HUDI-888] fix NullPointerException in HoodieCompactor (#1622) 2020-05-20 04:22:35 -07:00
wenningd
0dc2fa6172 [MINOR] Fix HoodieCompactor config abbreviation (#1642) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2020-05-19 21:03:54 -07:00
hongdd
161a798337 [HUDI-706] Add unit test for SavepointsCommand (#1624) 2020-05-19 18:36:01 +08:00
Balaji Varadarajan
e6f3bf10cf [HUDI-858] Allow multiple operations to be executed within a single commit (#1633) 2020-05-18 19:27:24 -07:00
Joey
2600d2de8d [MINOR] Fix apache-rat violations (#1639) 
* MINOR Fix apache-rat violations. Also, enabling RAT for hudi-utilities and hudi-integ-test
 2020-05-18 11:16:49 -07:00
rolandjohann
459356e292 [HUDI-863] get decimal properties from derived spark DataType (#1596) 2020-05-18 04:28:27 -07:00
hongdd
57132f79bb [HUDI-705] Add unit test for RollbacksCommand (#1611) 2020-05-18 14:04:06 +08:00
Sivabalan Narayanan
29edf4b3b8 [HUDI-407] Adding Simple Index to Hoodie. (#1402) 
This index finds the location by joining incoming records with records from base files.
 2020-05-17 18:32:24 -07:00
Balaji Varadarajan
3c9da2e5f0 [HUDI-895] Remove unnecessary listing .hoodie folder when using timeline server (#1636) 2020-05-17 18:18:53 -07:00
Mathieu
25a0080b2f [HUDI-714]Add javadoc and comments to hudi write method link (#1409) 
* [HUDI-714] Add javadoc and comments to hudi write method link
 2020-05-16 08:36:51 -04:00
Raymond Xu
148b2458f6 [MINOR] Increase heap space for surefire (#1623) 2020-05-16 01:11:57 -07:00
Raymond Xu
2ada2ef50f [HUDI-902] Avoid exception when getSchemaProvider (#1584) 
* When no new input data, don't throw exception for null SchemaProvider
* Return the newly added NullSchemaProvider instead
 2020-05-15 21:33:02 -07:00
Alexander Filipchik
25e0b75b3d [HUDI-723] Register avro schema if infered from SQL transformation (#1518) 
* Register avro schema if infered from SQL transformation
* Make HoodieWriteClient creation done lazily always. Handle setting schema-provider and avro-schemas correctly when using SQL transformer

Co-authored-by: Alex Filipchik <alex.filipchik@csscompany.com>
Co-authored-by: Balaji Varadarajan <varadarb@uber.com>
 2020-05-15 12:44:03 -07:00
Gary Li
a64afdfd17 HUDI-528 Handle empty commit in incremental pulling (#1612) 2020-05-14 22:55:25 -07:00
Alexander Filipchik
f094f42857 [HUDI-843] Add ability to specify time unit for TimestampBasedKeyGenerator (#1541) 
Co-authored-by: Alex Filipchik <alex.filipchik@csscompany.com>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2020-05-14 13:37:59 -07:00
hongdd
3a2fe13fcb [HUDI-701] Add unit test for HDFSParquetImportCommand (#1574) 2020-05-14 19:15:49 +08:00
Alexander Filipchik
83796b3189 [HUDI-793] Adding proper default to hudi metadata fields and proper handling to rewrite routine (#1513) 
* Adding proper default to hudi metadata fields and proper handling to rewrite routine
* Handle fields declared with a null default

Co-authored-by: Alex Filipchik <alex.filipchik@csscompany.com>
 2020-05-13 18:04:38 -07:00
Raymond Xu
0d4848b68b [HUDI-811] Restructure test packages (#1607) 
* restructure hudi-spark tests
* restructure hudi-timeline-service tests
* restructure hudi-hadoop-mr hudi-utilities tests
* restructure hudi-hive-sync tests
 2020-05-13 15:37:03 -07:00
cxzl25
32bada29dc [HUDI-889] Writer supports useJdbc configuration when hive synchronization is enabled (#1627) 2020-05-14 00:20:13 +08:00
liujinhui
32ea4c70ff [HUDI-869] Add support for alluxio (#1608) 2020-05-13 21:00:34 +08:00
Udit Mehrotra
404c7e82d9 [HUDI-884] Shade avro and parquet-avro in hudi-hive-sync-bundle (#1618) 
Co-authored-by: Mehrotra <uditme@amazon.com>
 2020-05-12 11:40:31 -07:00
Shen Hong
e8ffc6f0aa [HUDI-881] Replace part of spark context by hadoop configuration in AbstractHoodieClient and HoodieReadClient (#1620) 2020-05-12 09:33:29 -07:00
Shen Hong
b54517aad0 [HUDI-886] Replace jsc.hadoopConfiguration by hadoop configuration in hudi-client testcase (#1621) 2020-05-12 08:51:31 -07:00
Shen Hong
295d00beea [HUDI-880] Replace part of spark context by hadoop configuration in HoodieTable. (#1614) 2020-05-11 23:33:57 -07:00
liujinhui
5d37e66b7e [MINOR] Fix HoodieNotSupportedException description in KafkaOffsetGen (#1615) 2020-05-11 23:14:43 +08:00
Shen Hong
6dac10115c [HUDI-870] Remove spark context in ClientUtils and HoodieIndex (#1609) 2020-05-11 19:05:36 +08:00
Balaji Varadarajan
8d0e23173b [HUDI-820] cleaner repair command should only inspect clean metadata files (#1542) 2020-05-11 09:25:54 +08:00
vinoth chandar
f92b9fdcc4 [MINOR] Fix hardcoding of ports in TestHoodieJmxMetrics (#1606) 2020-05-10 19:23:26 -04:00
Carm
fa6aba751d [MINOR] fixed building IndexFileFilter with a wrong condition in HoodieGlobalBloomIndex class (#1537) 2020-05-10 09:45:07 +08:00
Udit Mehrotra
d54b4b8a52 [HUDI-838] Support schema from HoodieCommitMetadata for HiveSync (#1559) 
Co-authored-by: Mehrotra <uditme@amazon.com>
 2020-05-07 16:33:09 -07:00
Alexander Filipchik
e783ab1749 [HUDI-784] Adressing issue with log reader on GCS (#1516) 
[HUDI-784] Adressing issue with log reader on GCS (#1516)
Co-authored-by: Alex Filipchik <alex.filipchik@csscompany.com>
 2020-05-07 13:05:32 -07:00
hongdd
f921469afc [HUDI-704] Add test for RepairsCommand (#1554) 2020-05-07 23:02:28 +08:00
Raymond Xu
366bb10d8c [HUDI-812] Migrate hudi common tests to JUnit 5 (#1590) 
* [HUDI-812] Migrate hudi-common tests to JUnit 5
 2020-05-06 19:15:20 +08:00
bschell
e21441ad83 Add changes for presto mor queries (#1578) 
Adds the neccessary changes to hudi for support of presto querying hudi
merge-on-read table's realtime view.

Co-authored-by: Brandon Scheller <bschelle@amazon.com>
 2020-05-04 11:27:14 -07:00
AakashPradeep
5e0f5e5521 [HUDI-852] adding check for table name for Append Save mode (#1580) 
* adding check for table name for Append Save mode

* adding existing table validation for delete and upsert operation

Co-authored-by: Aakash Pradeep <apradeep@twilio.com>
 2020-05-03 23:09:17 -07:00
Raymond Xu
096f7f55b2 [HUDI-813] Migrate hudi-utilities tests to JUnit 5 (#1589) 2020-05-04 12:43:42 +08:00
Balaji Varadarajan
506447fd4f [HUDI-850] Avoid unnecessary listings in incremental cleaning mode (#1576) 2020-05-01 21:37:21 -07:00
vinoth chandar
c4b71622b9 [MINOR] Reorder HoodieTimeline#compareTimestamp arguments for better readability (#1575) 
- reads nicely as (instantTime1, GREATER_THAN_OR_EQUALS, instantTime2) etc
 2020-04-30 09:19:39 -07:00
hongdd
9059bce977 [HUDI-702] Add test for HoodieLogFileCommand (#1522) 2020-04-29 18:47:27 +08:00
Raymond Xu
69b16309c8 [HUDI-814] Migrate hudi-client tests to JUnit 5 (#1570) 2020-04-29 13:57:28 +08:00
Raymond Xu
06dae30297 [HUDI-810] Migrate ClientTestHarness to JUnit 5 (#1553) 2020-04-28 23:38:16 +08:00
satishkotha
6de9f5d9e5 [HUDI-819] Fix a bug with MergeOnReadLazyInsertIterable. 
Variable declared here[1] masks protected statuses variable. So although hoodie writes data, will not include writestatus in the completed section. This can cause duplicates being written (#1540)
[1] https://github.com/apache/incubator-hudi/blob/master/hudi-client/src/main/java/org/apache/hudi/execution/MergeOnReadLazyInsertIterable.java#L53
 2020-04-27 12:50:39 -07:00
vinoth chandar
19ca0b5629 [HUDI-785] Refactor compaction/savepoint execution based on ActionExector abstraction (#1548) 
- Savepoint and compaction classes moved to table.action.* packages
 - HoodieWriteClient#savepoint(...) returns void
 - Renamed HoodieCommitArchiveLog -> HoodieTimelineArchiveLog
 - Fixed tests to take into account the additional validation done
 - Moved helper code into CompactHelpers and SavepointHelpers
 2020-04-25 18:26:44 -07:00
dengziming
19cc15c098 [MINOR]: Fix cli docs for DeltaStreamer (#1547) 2020-04-22 11:37:17 -07:00