lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,992 Commits 1 Branch 0 Tags
26c967bac60de394ca78e79d27647067680ed630
Commit Graph

2992 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
ForwardXu
26c967bac6 [HUDI-3504] Support bootstrap command based on Call Produce Command (#5977) 2022-06-27 13:06:50 +08:00
leesf
8f4e2a189e [HUDI-4315] Do not throw exception in BaseSpark3Adapter#toTableIdentifier (#5957) 2022-06-27 12:50:58 +08:00
cxzl25
72fa19bcc9 [HUDI-4316] Support for spillable diskmap configuration when constructing HoodieMergedLogRecordScanner (#5959) 2022-06-27 11:09:30 +08:00
cxzl25
7a6eb0f6e1 [HUDI-4309] Spark3.2 custom parser should not throw exception (#5947) 2022-06-27 09:37:23 +08:00
Sivabalan Narayanan
0a9e568ff5 [HUDI-5246] Bumping mysql connector version due to security vulnerability (#5851) 2022-06-26 16:54:57 -07:00
Shiyan Xu
559b26fb7c [MINOR] Remove -T option from CI build (#5972) 2022-06-26 08:34:05 -07:00
ForwardXu
1c43c590ac [HUDI-3502] Support hdfs parquet import command based on Call Produce Command (#5956) 2022-06-26 11:27:14 +08:00
xiarixiaoyao
142adf4ccb [HUDI-4296] Fix the bug that TestHoodieSparkSqlWriter.testSchemaEvolutionForTableType is flaky (#5973) 2022-06-25 21:03:19 +08:00
Alexey Kudinkin
c86edfc28e [HUDI-4319] Fixed Parquet's PLAIN_DICTIONARY encoding not being applied when bulk-inserting (#5966) 
* Fixed Dictionary encoding config not being properly propagated to Parquet writer (making it unable to apply it, substantially bloating the storage footprint)
 2022-06-24 23:52:28 -04:00
xiarixiaoyao
360df576a9 Revert "[TEST][DO_NOT_MERGE]fix random failed for ci (#5948)" (#5971) 
This reverts commit e8fbd4daf4.
 2022-06-25 11:23:17 +08:00
xiarixiaoyao
e8fbd4daf4 [TEST][DO_NOT_MERGE]fix random failed for ci (#5948) 2022-06-25 10:15:08 +08:00
jiz
eeafaeacd2 [HUDI-3512] Add call procedure for StatsCommand (#5955) 
Co-authored-by: zhanshaoxiong <shaoxiong0001@@gmail.com>
 2022-06-25 09:43:23 +08:00
luokey
59978ef4a9 [HUDI-4260] Change KEYGEN_CLASS_NAME without default value (#5877) 
* Change KEYGEN_CLASS_NAME without default value

Co-authored-by: 854194341@qq.com <loukey_7821>
 2022-06-24 15:05:03 +08:00
xi chaomin
30ebdc708b [HUDI-3735] TestHoodieSparkMergeOnReadTableRollback is flaky (#5874) 2022-06-24 02:47:36 -04:00
Zhaojing Yu
6456bd3a51 [HUDI-4273] Support inline schedule clustering for Flink stream (#5890) 
* [HUDI-4273] Support inline schedule clustering for Flink stream

* delete deprecated clustering plan strategy and add clustering ITTest
 2022-06-24 11:28:06 +08:00
jiz
af9f09047d [HUDI-3509] Add call procedure for HoodieLogFileCommand (#5949) 
Co-authored-by: zhanshaoxiong <jiimmyzhan@tencent.com>
 2022-06-24 10:16:54 +08:00
Sagar Sumit
eeb78f23e6 [HUDI-4290] Fix fetchLatestBaseFiles to filter replaced filegroups (#5941) 
* [HUDI-4290] Fix fetchLatestBaseFiles to filter replaced filegroups

* Separate out incremental sync fsview test with clustering
 2022-06-23 19:40:08 +05:30
Forus
38ff18a199 [HUDI-4299] Fix problem about hudi-example-java run failed on idea. (#5936) 2022-06-23 21:46:22 +08:00
jiz
1bb017d396 [HUDI-3508] Add call procedure for FileSystemViewCommand (#5929) 
* [HUDI-3508] Add call procedure for FileSystemView

* minor

Co-authored-by: jiimmyzhan <jiimmyzhan@tencent.com>
 2022-06-22 17:50:20 +08:00
Danny Chan
1dbd9d407a [minor] following 4270, add unit tests for the keys lost case (#5918) 2022-06-22 16:56:06 +08:00
LinMingQiang
c9590790f8 [HUDI-4279] Strength the remote fs view lagging check when latest commit refresh is enabled (#5917) 
Signed-off-by: LinMingQiang <1356469429@qq.com>
 2022-06-22 10:32:21 +08:00
Zhaojing Yu
c7e430bb46 Revert master (#5925) 
* Revert "udate"

This reverts commit 092e35c1e3.

* Revert "[HUDI-3475] Initialize hudi table management module."

This reverts commit 4640a3bbb8.
 2022-06-21 16:58:50 +08:00
喻兆靖
092e35c1e3 udate 2022-06-21 15:22:04 +08:00
喻兆靖
4640a3bbb8 [HUDI-3475] Initialize hudi table management module. 2022-06-21 15:21:30 +08:00
Bo Cui
7c4aaa9715 [HUDI-4270] Bootstrap op data loading missing (#5888) 2022-06-21 11:47:39 +08:00
Shawn Chang
5c204f1416 [HUDI-4177] Fix hudi-cli rollback with rollbackUsingMarkers method call (#5734) 
* Fix hudi-cli rollback with rollbackUsingMarkers method call
* Add test for hudi-cli rollbackUsingMarkers

Co-authored-by: Shawn Chang <yxchang@amazon.com>
 2022-06-21 10:54:12 +08:00
Forus
ba4d5bd847 [HUDI-4251] Fix the problem that the command 'commits sync' description does not match. (#5881) 2022-06-20 16:03:58 -07:00
RexAn
17ac5a4573 [HUDI-4173] Fix wrong results if the user read no base files hudi table by glob paths (#5723) 2022-06-20 23:02:34 +05:30
Y Ethan Guo
7601e9e4c7 [MINOR] Update DOAP with 0.11.1 Release (#5908) 2022-06-20 09:27:35 -07:00
Alexander Trushev
f1103281d2 [HUDI-4258] Fix when HoodieTable removes data file before the end of Flink job (#5876) 
* [HUDI-4258] Fix when HoodieTable removes data file before the end of Flink job
 2022-06-20 17:07:49 +08:00
luokey
7c6bedff25 [HUDI-4259] Flink create avro schema not conformance to standards (#5878) 
* flink create avro schema not conformance to standards

Co-authored-by: 854194341@qq.com <loukey_7821>
 2022-06-20 15:41:23 +08:00
felixYyu
d7facb8cb8 fix remove redundant Variable (#5806) 2022-06-20 15:21:49 +08:00
Shizhi Chen
7481eacf23 [HUDI-4277] supoort flink table source with computed column (#5897) 
Co-authored-by: chenshizhi <chenshizhi@bilibili.com>
 2022-06-20 15:19:32 +08:00
5herhom
efafb79eeb [MINOR] Add "spillable_map_path" in FlinkCompactionConfig. To avoid the disk space of "/tmp" full when compacting offline. (#5905) 2022-06-20 15:15:23 +08:00
huberylee
d4f0326b4b [HUDI-4275] Refactor rollback inflight instant for clustering/compaction to reuse some code (#5894) 2022-06-20 14:29:21 +08:00
ForwardXu
c5c4cfec91 [HUDI-3507] Support export command based on Call Produce Command (#5901) 2022-06-19 18:48:22 +08:00
huberylee
fec49dc12b [HUDI-4165] Support Create/Drop/Show/Refresh Index Syntax for Spark SQL (#5761) 
* Support Create/Drop/Show/Refresh Index Syntax for Spark SQL
 2022-06-17 18:33:58 +08:00
董可伦
7689e62cd9 [HUDI-4265] Deprecate useless targetTableName parameter in HoodieMultiTableDeltaStreamer (#5883) 2022-06-17 16:57:14 +08:00
KnightChess
0ff34b6974 [HUDI-4214] improve repeat init write schema in ExpressionPayload (#5820) 
* [HUDI-4214] improve repeat init write schema in ExpressionPayload
 2022-06-16 17:58:37 +08:00
KnightChess
2bf0a1906d [HUDI-4217] improve repeat init object in ExpressionPayload (#5825) 2022-06-15 20:21:28 +08:00
董可伦
c291b05699 [HUDI-4218] [HUDI-4218] Expose the real exception information when an exception occurs in the tableExists method (#5827) 2022-06-15 18:10:35 +08:00
superche
7b946cf351 [HUDI-3499] Add Call Procedure for show rollbacks (#5848) 
* Add Call Procedure for show rollbacks

* fix

* add ut for show_rollback_detail and exception handle

Co-authored-by: superche <superche@tencent.com>
 2022-06-15 16:50:15 +08:00
Danny Chan
0811bb38fb [HUDI-4255] Make the flink merge and replace handle intermediate file visible (#5866) 2022-06-15 14:23:23 +08:00
Danny Chan
25bbff64cf [minor] Following HUDI-4207, remote the new wrapper #init method (#5865) 2022-06-15 08:48:13 +08:00
felixYyu
f16b1e8982 [MINOR] Fix typo of DisruptorExecutor in RFC 53 (#5860) 2022-06-13 23:30:17 -07:00
HunterXHunter
264b15df87 [HUDI-4207] HoodieFlinkWriteClient.getOrCreateWriteHandle throws an e… (#5788) 
Adding more logs to assist in debugging with HoodieFlinkWriteClient.getOrCreateWriteHandle throwing exception
 2022-06-13 10:36:06 -04:00
Qi Ji
4774c4248f [HUDI-4006] failOnDataLoss on delta-streamer kafka sources (#5718) 
add new config key hoodie.deltastreamer.source.kafka.enable.failOnDataLoss
when failOnDataLoss=false (current behaviour, the default), log a warning instead of seeking to earliest silently
when failOnDataLoss is set, fail explicitly
 2022-06-13 10:31:57 -04:00
luoyajun
0d859fe58b [HUDI-3863] Add UT for drop partition column in deltastreamer testsuite (#5727) 2022-06-13 10:29:32 -04:00
xi chaomin
e89f5627e4 [HUDI-3682] testReaderFilterRowKeys fails in TestHoodieOrcReaderWriter (#5790) 
TestReaderFilterRowKeys needs to get the key from RECORD_KEY_METADATA_FIELD, but the writer in current UT does not populate the meta field and the schema does not contains meta fields.

This fix writes data with schema which contains meta fields and calls writeAvroWithMetadata for writing.

Co-authored-by: xicm <xicm@asiainfo.com>
 2022-06-13 10:22:12 -04:00
superche
14d8735a1c Strip extra spaces when creating new configuration (#5849) 
Co-authored-by: superche <superche@tencent.com>
 2022-06-13 19:10:38 +08:00