lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,884 Commits 1 Branch 0 Tags
eb5e7eec0a080989a4ad3c867699b814dc502ba1
Commit Graph

1884 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
liujinhui
eb5e7eec0a MINOR_CHECKSTYLE (#3616) 
Fix checkstyle
 2021-09-07 18:19:39 +08:00
Raymond Xu
cf002b6918 [HUDI-2079] Make CLI command tests functional (#3601) 
Make all tests in org.apache.hudi.cli.commands extend org.apache.hudi.cli.functional.CLIFunctionalTestHarness and tag as "functional".

This also resolves a blocker where DFS init consistently failed when moving to ubuntu 18.04
 2021-09-06 15:53:53 -07:00
Sivabalan Narayanan
f218693f5d [MINOR] Fixing some functional tests by moving to right packages (#3596) 2021-09-06 00:07:55 -04:00
Raymond Xu
7592ddd776 [HUDI-2399] Rebalance CI jobs for shorter wait time (#3604) 2021-09-05 09:25:57 -07:00
Danny Chan
e9bf1c1186 [HUDI-2380] The default archive folder should be 'archived' (#3568) 2021-09-04 15:53:55 +08:00
Raymond Xu
073c318d9f [HUDI-1989] Disable HDFSParquetImporter related tests (#3597) 
Also mark HDFSParquetImportCommand and HDFSParquetImporter as deprecated.
 2021-09-03 23:08:11 -04:00
Raymond Xu
6bd3ca98d6 [HUDI-1989] Fix flakiness in TestHoodieMergeOnReadTable (#3574) 
* [HUDI-1989] Refactor clustering tests for MoR table

* refactor assertion helper

* add CheckedFunction

* SparkClientFunctionalTestHarness.java

* put back original test case

* move testcases out from TestHoodieMergeOnReadTable.java

* add TestHoodieSparkMergeOnReadTableRollback.java

* use SparkClientFunctionalTestHarness

* add tag
 2021-09-03 13:17:17 -07:00
Raymond Xu
11398e8480 [MINOR] Skip checkstyle and rat in Azure (#3593) 
- make tests run through without being blocked by style issues
- let GitHub Actions tasks give quick feedback on build, style and other checks
 2021-09-03 09:18:18 -07:00
Danny Chan
79b896f071 [HUDI-2392] Do not send partition delete record when changelog mode enabled (#3586) 2021-09-02 20:58:12 +08:00
yuzhaojing
7a1bd225ca [HUDI-2376] Add pipeline for Append mode (#3573) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-09-02 16:32:40 +08:00
Shawy Geng
21fd6edfe7 [HUDI-2384] Change log file size config to long (#3577) 2021-09-02 11:14:09 +08:00
Raymond Xu
38c9b85aa8 [HUDI-2280] Use GitHub Actions to build different scala spark versions (#3556) 2021-09-01 08:51:00 -07:00
Danny Chan
f66e1ce9bf [HUDI-2379] Include the pending compaction file groups for flink (#3567) 
streaming reader
 2021-09-01 16:47:52 +08:00
rmahindra123
d59c8044f8 [HUDI-2378] Add configs for common and pre validate (#3564) 
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-08-30 23:28:35 -04:00
董可伦
bf5a52e51b [HUDI-2320] Add support ByteArrayDeserializer in AvroKafkaSource (#3502) 2021-08-30 10:01:15 +08:00
Danny Chan
57668d02a0 [HUDI-2371] Improvement flink streaming reader (#3552) 
- Support reading empty table
- Fix filtering by partition path
- Support reading from earliest commit
 2021-08-28 20:16:54 +08:00
wenningd
69cbcc9516 Merge pull request #3541 from rahil-c/rahil-c/HUDI-2359 
[HUDI-2359] Add basic "hoodie_is_deleted" unit tests to TestDataSource classes
 2021-08-27 16:28:51 -07:00
董可伦
562e28f079 [HUDI-2365]Optimizing overwriteField method with Objects.equals (#3542) 
Optimizing overwriteField method with Objects.equals
 2021-08-27 17:17:22 +08:00
mikewu
9850e90e2e [HUDI-2229] Refact HoodieFlinkStreamer to reuse the pipeline of HoodieTableSink (#3495) 
Co-authored-by: mikewu <xingbo.wxb@alibaba-inc.com>
 2021-08-27 10:14:04 +08:00
Satish M
55a80a817d [HUDI-2264] Refactor HoodieSparkSqlWriterSuite to add setup and teardown (#3544) 2021-08-26 10:01:48 -04:00
Danny Chan
0f39137ba8 [HUDI-2321] Use the caller classloader for ReflectionUtils (#3535) 
Based on the discussion on stackoverflow:
https://stackoverflow.com/questions/1771679/difference-between-threads-context-class-loader-and-normal-classloader

The Thread.currentThread().getContextClassLoader() should never be used
because the context classloader is not immutable, user can overwrite it
when thread switches, it is also nullable.

The objection here: https://stackoverflow.com/a/36228195 says the
Thread.currentThread().getContextClassLoader() is a JDK design error
and the context classloader is never suggested to be used. The API that
needs classloader should ask the user to set up the right classloader.
 2021-08-26 21:00:30 +08:00
yuzhaojing
73fdcf37df [HUDI-2368] Catch Throwable in BoundedInMemoryExecutor (#3546) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-08-26 20:34:05 +08:00
pengzhiwei
cc5256a7d8 [HUDI-2357] MERGE INTO doesn't work for tables created using CTAS (#3534) 2021-08-26 16:54:41 +08:00
ayachi_nene
be57e42200 [HUDI-2366] fix too many logs (#3543) 2021-08-26 16:45:52 +08:00
Rahil Chertara
694300477f [HUDI-2359] Add basic "hoodie_is_deleted" unit tests to TestDataSource classes 2021-08-25 16:35:35 -07:00
Udit Mehrotra
486bc7dc3b [MINOR] Update DOAP with 0.9.0 Release (#3537) 2021-08-25 16:57:05 -04:00
Danny Chan
a60fab3a5c [HUDI-2352] The upgrade downgrade action of flink writer should be singleton (#3531) 2021-08-25 10:56:14 +08:00
Satish M
04ede8eecf [HUDI-2262] reduce build warnings (#3481) 2021-08-24 13:06:38 -04:00
Sivabalan Narayanan
15bf01dcb7 [HUDI-2349] Adding spark delete node to integ test suite (#3528) 2021-08-24 10:58:47 -04:00
zhangyue19921010
de94787a85 [HUDI-2345] Hoodie columns sort partitioner for bulk insert (#3523) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-08-24 21:45:17 +08:00
Danny Chan
05e6f44d53 [MINOR] Fix BatchBootstrapOperator initialization (#3520) 2021-08-22 13:03:22 +08:00
董可伦
be8c1e499f Support referencing subquery with column aliases by table alias in merge into (#3380) 2021-08-21 21:53:16 +08:00
yuzhaojing
ab3fbb8895 [HUDI-2342] Optimize Bootstrap operator (#3516) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-08-21 20:03:03 +08:00
Danny Chan
c7c517f14c [HUDI-2340] Merge the data set for flink bounded source when changelog mode turns off (#3513) 2021-08-21 07:21:35 +08:00
Udit Mehrotra
e39d0a2f28 Keep non-conflicting names for common configs between DataSourceOptions and HoodieWriteConfig (#3511) 2021-08-20 02:42:59 -07:00
pengzhiwei
49829f8822 [HUDI-2339] Create Table If Not Exists Failed After Alter Table (#3510) 2021-08-20 14:21:10 +08:00
Udit Mehrotra
c350d05dd3 Restore 0.8.0 config keys with deprecated annotation (#3506) 
Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-08-19 13:36:40 -07:00
Sagar Sumit
37c29e75dc [HUDI-2322] Use correct meta columns while preparing dataset for bulk insert (#3504) 2021-08-19 12:07:12 -04:00
Danny Chan
9762e4c08c [MINOR] Some cosmetic changes for Flink (#3503) 2021-08-19 23:21:20 +08:00
Sivabalan Narayanan
7dddd54406 [MINOR] Fixing release validation script (#3493) 2021-08-19 07:54:56 -04:00
Sagar Sumit
b15c3cf9a9 [HUDI-1363] Include _hoodie_operation meta column in removeMetadataFields (#3501) 2021-08-19 07:03:54 -04:00
swuferhong
1fed44af84 [HUDI-2316] Support Flink batch upsert (#3494) 2021-08-19 17:15:26 +08:00
leiqiang
b7a0d76fc9 [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException 
close apache/hudi#3402
 2021-08-18 15:40:51 +08:00
liujinhui
5ee35a0a92 HUDI-1674 (#3488) 2021-08-18 13:45:48 +08:00
ayachi_nene
99663d370b [HUDI-2301] fix FileSliceMetrics utils bug (#3487) 2021-08-17 11:09:53 -07:00
Danny Chan
66f951322a [HUDI-2191] Bump flink version to 1.13.1 (#3291) 2021-08-16 18:14:05 +08:00
Raymond Xu
4d508ef673 [MINOR] Fix SelectPackages in HoodieSparkFunctionalTestSuite (#3476) 2021-08-15 10:17:00 -07:00
Udit Mehrotra
09e625becd [HOT-FIX] Add apache license to spark_command.txt.template (#3477) 2021-08-15 07:08:55 -04:00
Udit Mehrotra
3e301196bf Moving to 0.10.0-SNAPSHOT on master branch. 2021-08-14 18:51:09 -07:00
Y Ethan Guo
23dca6c237 [HUDI-2268] Add upgrade and downgrade to and from 0.9.0 (#3470) 
- Added upgrade and downgrade step to and from 0.9.0. Upgrade adds few table properties. Downgrade recreates timeline server based marker files if any.
 2021-08-14 20:20:23 -04:00