lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
3,001 Commits 1 Branch 0 Tags
e3eb14ad2d5bd25fcfd6bb2eb6e991558c0684ed
Commit Graph

3001 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Teng
e3eb14ad2d [HUDI-4334] close SparkRDDWriteClient after usage in Create/Delete/RollbackSavepointsProcedure (#5994) 2022-06-29 06:13:29 +08:00
bschell
fd7d25ab63 [HUDI-1176] Upgrade hudi to log4j2 (#5366) 
* Move to log4j2

cr: https://code.amazon.com/reviews/CR-71010705

* Upgrade unit tests to log4j2

* update exclusion

Co-authored-by: Brandon Scheller <bschelle@amazon.com>
 2022-06-28 12:54:23 -07:00
Alexey Kudinkin
ed823f1c6f [HUDI-4320] Make sure HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED could be specified by the writer (#5970) 
Fixed sequence determining whether Parquet's legacy-format writing property should be overridden to only kick in when it has not been explicitly specified by the caller
 2022-06-28 12:27:32 -07:00
BruceLin
efb9719018 [HUDI-4332] The current instant may be wrong under some extreme conditions in AppendWriteFunction. (#5988) 2022-06-28 20:42:26 +08:00
ForwardXu
08eba914ed [HUDI-4333] fix HoodieFileIndex's listFiles method log print skipping percent NaN (#5990) 2022-06-28 15:08:48 +08:00
KnightChess
09dc001430 [HUDI-4325] fix spark sql procedure cause ParseException with semicolon (#5982) 
* [HUDI-4325] fix saprk sql procedure cause ParseException with semicolon
 2022-06-28 09:44:41 +08:00
superche
b14ed47f21 [HUDI-3506] Add call procedure for CommitsCommand (#5974) 
* [HUDI-3506] Add call procedure for CommitsCommand

Co-authored-by: superche <superche@tencent.com>
 2022-06-28 09:43:36 +08:00
Sagar Sumit
8846849a03 [HUDI-4291] Fix flaky TestCleanPlanExecutor#testKeepLatestFileVersions (#5930) 2022-06-27 17:27:16 +05:30
吴祥平
3a1fd22841 [HUDI-4311] Fix Flink lose data on some rollback scene (#5950) 2022-06-27 16:09:44 +08:00
ForwardXu
26c967bac6 [HUDI-3504] Support bootstrap command based on Call Produce Command (#5977) 2022-06-27 13:06:50 +08:00
leesf
8f4e2a189e [HUDI-4315] Do not throw exception in BaseSpark3Adapter#toTableIdentifier (#5957) 2022-06-27 12:50:58 +08:00
cxzl25
72fa19bcc9 [HUDI-4316] Support for spillable diskmap configuration when constructing HoodieMergedLogRecordScanner (#5959) 2022-06-27 11:09:30 +08:00
cxzl25
7a6eb0f6e1 [HUDI-4309] Spark3.2 custom parser should not throw exception (#5947) 2022-06-27 09:37:23 +08:00
Sivabalan Narayanan
0a9e568ff5 [HUDI-5246] Bumping mysql connector version due to security vulnerability (#5851) 2022-06-26 16:54:57 -07:00
Shiyan Xu
559b26fb7c [MINOR] Remove -T option from CI build (#5972) 2022-06-26 08:34:05 -07:00
ForwardXu
1c43c590ac [HUDI-3502] Support hdfs parquet import command based on Call Produce Command (#5956) 2022-06-26 11:27:14 +08:00
xiarixiaoyao
142adf4ccb [HUDI-4296] Fix the bug that TestHoodieSparkSqlWriter.testSchemaEvolutionForTableType is flaky (#5973) 2022-06-25 21:03:19 +08:00
Alexey Kudinkin
c86edfc28e [HUDI-4319] Fixed Parquet's PLAIN_DICTIONARY encoding not being applied when bulk-inserting (#5966) 
* Fixed Dictionary encoding config not being properly propagated to Parquet writer (making it unable to apply it, substantially bloating the storage footprint)
 2022-06-24 23:52:28 -04:00
xiarixiaoyao
360df576a9 Revert "[TEST][DO_NOT_MERGE]fix random failed for ci (#5948)" (#5971) 
This reverts commit e8fbd4daf4.
 2022-06-25 11:23:17 +08:00
xiarixiaoyao
e8fbd4daf4 [TEST][DO_NOT_MERGE]fix random failed for ci (#5948) 2022-06-25 10:15:08 +08:00
jiz
eeafaeacd2 [HUDI-3512] Add call procedure for StatsCommand (#5955) 
Co-authored-by: zhanshaoxiong <shaoxiong0001@@gmail.com>
 2022-06-25 09:43:23 +08:00
luokey
59978ef4a9 [HUDI-4260] Change KEYGEN_CLASS_NAME without default value (#5877) 
* Change KEYGEN_CLASS_NAME without default value

Co-authored-by: 854194341@qq.com <loukey_7821>
 2022-06-24 15:05:03 +08:00
xi chaomin
30ebdc708b [HUDI-3735] TestHoodieSparkMergeOnReadTableRollback is flaky (#5874) 2022-06-24 02:47:36 -04:00
Zhaojing Yu
6456bd3a51 [HUDI-4273] Support inline schedule clustering for Flink stream (#5890) 
* [HUDI-4273] Support inline schedule clustering for Flink stream

* delete deprecated clustering plan strategy and add clustering ITTest
 2022-06-24 11:28:06 +08:00
jiz
af9f09047d [HUDI-3509] Add call procedure for HoodieLogFileCommand (#5949) 
Co-authored-by: zhanshaoxiong <jiimmyzhan@tencent.com>
 2022-06-24 10:16:54 +08:00
Sagar Sumit
eeb78f23e6 [HUDI-4290] Fix fetchLatestBaseFiles to filter replaced filegroups (#5941) 
* [HUDI-4290] Fix fetchLatestBaseFiles to filter replaced filegroups

* Separate out incremental sync fsview test with clustering
 2022-06-23 19:40:08 +05:30
Forus
38ff18a199 [HUDI-4299] Fix problem about hudi-example-java run failed on idea. (#5936) 2022-06-23 21:46:22 +08:00
jiz
1bb017d396 [HUDI-3508] Add call procedure for FileSystemViewCommand (#5929) 
* [HUDI-3508] Add call procedure for FileSystemView

* minor

Co-authored-by: jiimmyzhan <jiimmyzhan@tencent.com>
 2022-06-22 17:50:20 +08:00
Danny Chan
1dbd9d407a [minor] following 4270, add unit tests for the keys lost case (#5918) 2022-06-22 16:56:06 +08:00
LinMingQiang
c9590790f8 [HUDI-4279] Strength the remote fs view lagging check when latest commit refresh is enabled (#5917) 
Signed-off-by: LinMingQiang <1356469429@qq.com>
 2022-06-22 10:32:21 +08:00
Zhaojing Yu
c7e430bb46 Revert master (#5925) 
* Revert "udate"

This reverts commit 092e35c1e3.

* Revert "[HUDI-3475] Initialize hudi table management module."

This reverts commit 4640a3bbb8.
 2022-06-21 16:58:50 +08:00
喻兆靖
092e35c1e3 udate 2022-06-21 15:22:04 +08:00
喻兆靖
4640a3bbb8 [HUDI-3475] Initialize hudi table management module. 2022-06-21 15:21:30 +08:00
Bo Cui
7c4aaa9715 [HUDI-4270] Bootstrap op data loading missing (#5888) 2022-06-21 11:47:39 +08:00
Shawn Chang
5c204f1416 [HUDI-4177] Fix hudi-cli rollback with rollbackUsingMarkers method call (#5734) 
* Fix hudi-cli rollback with rollbackUsingMarkers method call
* Add test for hudi-cli rollbackUsingMarkers

Co-authored-by: Shawn Chang <yxchang@amazon.com>
 2022-06-21 10:54:12 +08:00
Forus
ba4d5bd847 [HUDI-4251] Fix the problem that the command 'commits sync' description does not match. (#5881) 2022-06-20 16:03:58 -07:00
RexAn
17ac5a4573 [HUDI-4173] Fix wrong results if the user read no base files hudi table by glob paths (#5723) 2022-06-20 23:02:34 +05:30
Y Ethan Guo
7601e9e4c7 [MINOR] Update DOAP with 0.11.1 Release (#5908) 2022-06-20 09:27:35 -07:00
Alexander Trushev
f1103281d2 [HUDI-4258] Fix when HoodieTable removes data file before the end of Flink job (#5876) 
* [HUDI-4258] Fix when HoodieTable removes data file before the end of Flink job
 2022-06-20 17:07:49 +08:00
luokey
7c6bedff25 [HUDI-4259] Flink create avro schema not conformance to standards (#5878) 
* flink create avro schema not conformance to standards

Co-authored-by: 854194341@qq.com <loukey_7821>
 2022-06-20 15:41:23 +08:00
felixYyu
d7facb8cb8 fix remove redundant Variable (#5806) 2022-06-20 15:21:49 +08:00
Shizhi Chen
7481eacf23 [HUDI-4277] supoort flink table source with computed column (#5897) 
Co-authored-by: chenshizhi <chenshizhi@bilibili.com>
 2022-06-20 15:19:32 +08:00
5herhom
efafb79eeb [MINOR] Add "spillable_map_path" in FlinkCompactionConfig. To avoid the disk space of "/tmp" full when compacting offline. (#5905) 2022-06-20 15:15:23 +08:00
huberylee
d4f0326b4b [HUDI-4275] Refactor rollback inflight instant for clustering/compaction to reuse some code (#5894) 2022-06-20 14:29:21 +08:00
ForwardXu
c5c4cfec91 [HUDI-3507] Support export command based on Call Produce Command (#5901) 2022-06-19 18:48:22 +08:00
huberylee
fec49dc12b [HUDI-4165] Support Create/Drop/Show/Refresh Index Syntax for Spark SQL (#5761) 
* Support Create/Drop/Show/Refresh Index Syntax for Spark SQL
 2022-06-17 18:33:58 +08:00
董可伦
7689e62cd9 [HUDI-4265] Deprecate useless targetTableName parameter in HoodieMultiTableDeltaStreamer (#5883) 2022-06-17 16:57:14 +08:00
KnightChess
0ff34b6974 [HUDI-4214] improve repeat init write schema in ExpressionPayload (#5820) 
* [HUDI-4214] improve repeat init write schema in ExpressionPayload
 2022-06-16 17:58:37 +08:00
KnightChess
2bf0a1906d [HUDI-4217] improve repeat init object in ExpressionPayload (#5825) 2022-06-15 20:21:28 +08:00
董可伦
c291b05699 [HUDI-4218] [HUDI-4218] Expose the real exception information when an exception occurs in the tableExists method (#5827) 2022-06-15 18:10:35 +08:00