lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
646 Commits 1 Branch 0 Tags
845e26165848c97798ba5dd8fc534fdfa9552552
Commit Graph

646 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
leesf
845e261658 [MINOR] Update some urls from http to https in the README file (#1074) 2019-12-03 17:35:27 +08:00
lamber-ken
89f0968097 [DOCS] Update the build source link (#1071) 2019-12-03 09:56:09 +08:00
lamber-ken
ff688107fa [DOCS] Update Hudi Readme (#1058) 
- Add build status 
- Clean up layout
 2019-12-02 09:25:43 -08:00
lamber-ken
784e3ad0b6 [HUDI-370] Refactor hudi-common based on new ImportOrder code style rule (#1063) 2019-12-02 06:59:09 +08:00
Gurudatt Kulkarni
75132c139f [HUDI-357] Refactor hudi-cli based on new comment and code style rules (#1051) 2019-11-30 11:12:41 -08:00
hongdd
b65a897856 [HUDI-374] Unable to generateUpdates in QuickstartUtils (#1059) 2019-11-30 11:11:00 -08:00
lamber-ken
024230fbd2 [HUDI-372] Support the shortName for Hudi DataSource (#1054) 
- Ability to do `spark.write.format("hudi")...`
 2019-11-30 08:02:33 -08:00
ForwardXu
0b52ae3ac2 [HUDI-209] Implement JMX metrics reporter (#1045) 2019-11-28 19:17:34 +08:00
lamber-ken
da8d1334ee [HUDI-373] Refactor hudi-client based on new ImportOrder code style rule (#1056) 2019-11-28 09:25:56 +08:00
谢磊
f9139c0f61 [HUDI-366] Refactor some module codes based on new ImportOrder code style rule (#1055) 
[HUDI-366] Refactor hudi-hadoop-mr / hudi-timeline-service / hudi-spark / hudi-integ-test / hudi- utilities based on new ImportOrder code style rule
 2019-11-27 21:32:43 +08:00
谢磊
b77fad39b5 [HUDI-364] Refactor hudi-hive based on new ImportOrder code style rule (#1048) 
[HUDI-364] Refactor hudi-hive based on new ImportOrder code style rule
 2019-11-27 16:30:37 +08:00
wenningd
d6e83e8f49 [HUDI-325] Fix Hive partition error for updated HDFS Hudi table (#1001) 2019-11-26 21:18:39 -08:00
bschell
60fed21dc7 [HUDI-327] Add null/empty checks to key generators (#1040) 
* Adds null and empty checks to all key generators. 
* Also improves error messaging for key generator issues.
 2019-11-26 02:37:16 -08:00
Pratyaksh Sharma
2a4cfb47c7 [HUDI-340]: made max events to read from kafka source configurable (#1039) 2019-11-26 18:34:02 +08:00
hongdd
a7e07cd910 [HUDI-359] Add hudi-env for hudi-cli module (#1042) 2019-11-25 13:25:42 -08:00
谢磊
212282c8aa [HUDI-358] Add Java-doc and importOrder checkstyle rule (#1043) 
- import groups are separated by one blank line
- org.apache.hudi.* at the top location
 2019-11-25 11:36:23 -08:00
hongdd
44823041a3 [HUDI-362] Adds a check for the existence of field (#1047) 2019-11-25 11:31:07 -08:00
filippo balicchia
845a0509b3 [MINOR] Some minor optimizations in HoodieJavaStreamingApp (#1046) 2019-11-25 18:49:13 +08:00
Sivabalan Narayanan
c3355109b1 [HUDI-328] Adding delete api to HoodieWriteClient (#1004) 
[HUDI-328]  Adding delete api to HoodieWriteClient and Spark DataSource
 2019-11-22 15:05:25 -08:00
hongdd
7bc08cbfdc [HUDI-345] Fix used deprecated function (#1024) 
- Schema.parse() with new Schema.Parser().parse
- FSDataOutputStream constructor
 2019-11-22 03:32:09 -08:00
Gurudatt Kulkarni
17eaf41c54 [HUDI-348] Add Issue template for the project (#1029) 2019-11-22 03:25:52 -08:00
Pratyaksh Sharma
1e14390719 [HUDI-350]: updated default value of config.getCleanerCommitsRetained() in javadocs 2019-11-20 06:50:04 -08:00
lamber-ken
d9fbe33339 [HOTFIX] Fix error configuration item of dockerfile-maven-plugin 2019-11-19 16:30:03 -08:00
谢磊
804e348d0e [HUDI-346] Set allowMultipleEmptyLines to false for EmptyLineSeparator rule (#1025) 2019-11-19 18:44:42 +08:00
谢磊
66492498f7 [HUDI-342] add pull request template for hudi project (#1022) 2019-11-19 14:06:01 +08:00
b_rousseau
e806eb797f [HUDI-339] Add support of Azure cloud storage (#1019) 
- Add Azure WASB (BLOB) and ADLS storage in StorageSchemes enum
- Update testStorageSchemes to test new added storage
 2019-11-17 14:29:24 -08:00
Nishith Agarwal
f82e58994e - Ensure that rollback instant is always created before the next commit instant. 
This especially affects IncrementalPull for MOR tables since we can end up pulling in
  log blocks for uncommitted data
- Ensure that generated commit instants are 1 second apart
 2019-11-17 14:11:26 -08:00
Nishith Agarwal
3a05edab01 - Fixing RT queries for HiveOnSpark that causes race conditions 
- Adding more comments to understand usage of reader/writer schema
 2019-11-16 13:46:47 -08:00
谢磊
22315a887f [HOTFIX] fix missing version of rat-plugin (#1015) 2019-11-13 21:18:46 -08:00
Balaji Varadarajan
f7c2f8cedc [HUDI-329] Presto Containers for integration test must allow newly built local jars to override 2019-11-13 17:35:34 -08:00
Mehrotra
92c69f5703 Migrate integration tests to spark 2.4.4 2019-11-13 16:53:24 -08:00
lamber-ken
045fa87a3d [HUDI-330] add EmptyStatement java checkstyle rule 2019-11-13 14:11:11 -08:00
Balaji Varadarajan
8ff06ddb0f [HUDI-80] Leverage Commit metadata to figure out partitions to be cleaned for Cleaning by commits mode (#1008) 2019-11-12 06:12:44 -08:00
Udit Mehrotra
0bb5999f79 [HUDI-306] Support Glue catalog and other hive metastore implementations (#961) 
- Support Glue catalog and other metastore implementations
- Remove shading from hudi utilities bundle
- Add maven profile to optionally shade hive in utilities bundle
 2019-11-11 17:27:31 -08:00
Balaji Varadarajan
1032fc3e54 [HUDI-137] Hudi cleaning state changes should be consistent with compaction actions 
Before this change, Cleaner performs cleaning of old file versions and then stores the deleted files in .clean files.
With this setup, we will not be able to track file deletions if a cleaner fails after deleting files but before writing .clean metadata.
This is fine for regular file-system view generation but Incremental timeline syncing relies on clean/commit/compaction metadata to keep a consistent file-system view.

Cleaner state transitions is now similar to that of compaction.

1. Requested : HoodieWriteClient.scheduleClean() selects the list of files that needs to be deleted and stores them in metadata
2. Inflight : HoodieWriteClient marks the state to be inflight before it starts deleting
3. Completed : HoodieWriteClient marks the state after completing the deletion according to the cleaner plan
 2019-11-11 10:40:16 -08:00
Sivabalan Narayanan
23b303e4b1 [HUDI-218] Adding Presto support to Integration Test (#1003) 2019-11-11 06:21:49 -08:00
Pratyaksh Sharma
5f1309407a [HUDI-253]: added validations for schema provider class (#995) 2019-11-11 06:03:44 -08:00
vinoth chandar
1483b97018 [DOCS] Change Hudi acronyms to plural 2019-11-10 12:39:58 -08:00
Jeff G
1ce3d891ce [DOCS] Update to align with original Uber whitepaper (#999) 2019-11-10 12:38:13 -08:00
pratyakshsharma
0863b1cfd9 [HUDI-245]: replaced instances of getInstants() and reverse() with getReverseOrderedInstants() (#1000) 2019-11-07 08:42:48 -08:00
pratyakshsharma
20871a17b2 [HUDI-302]: simplified countInstants() method in HoodieDefaultTimeline (#997) 2019-11-06 12:56:09 -08:00
Gurudatt Kulkarni
71ac2c0d5e [HUDI-324] TimestampKeyGenerator should support milliseconds (#993) 2019-11-05 04:22:14 -08:00
Bhavani Sudha Saktheeswaran
04834817c8 [MINOR] Add features and instructions to build Hudi in README (#992) 2019-11-03 01:48:06 -08:00
Raymond Xu
91740635b2 [HUDI-321] Support bulkinsert in HDFSParquetImporter (#987) 
- Add bulk insert feature
- Fix some minor issues
 2019-11-02 23:12:44 -07:00
Wenning Ding
bd77dc792c Add MOR integration testing 2019-11-02 19:49:04 -07:00
Wenning Ding
b6057c5e0e [HUDI-314] Fix multi partition keys error when querying a realtime table 2019-11-02 19:49:04 -07:00
Balaji Varadarajan
a6390aefc4 [HUDI-312] Make docker hdfs cluster ephemeral. This is needed to fix flakiness in integration tests. Also, Fix DeltaStreamer hanging issue due to uncaught exception 2019-11-01 11:49:59 -07:00
dependabot[bot]
144ea4eedf Bump httpclient from 4.3.2 to 4.3.6 (#980) 
Bumps httpclient from 4.3.2 to 4.3.6.

Signed-off-by: dependabot[bot] <support@github.com>
 2019-11-01 05:22:31 -07:00
dependabot[bot]
74d8e625c5 Bump checkstyle from 8.8 to 8.18 (#981) 
Bumps [checkstyle](https://github.com/checkstyle/checkstyle) from 8.8 to 8.18.
- [Release notes](https://github.com/checkstyle/checkstyle/releases)
- [Commits](https://github.com/checkstyle/checkstyle/compare/checkstyle-8.8...checkstyle-8.18)

Signed-off-by: dependabot[bot] <support@github.com>
 2019-11-01 05:06:03 -07:00
Wenning Ding
ee0fd06de7 synchronized lock on conf object instead of class 2019-10-31 21:54:27 -07:00