lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
3,017 Commits 1 Branch 0 Tags
6187622178e4e802f4a32a5fa8c856846bf12bd5
Commit Graph

3017 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Shiyan Xu
6187622178 [MINOR] Improve variable names (#6039) 2022-07-04 18:03:50 -07:00
voonhous
c091e4cc30 [HUDI-3730] Add ConfigTool#toMap UT (#6035) 
Co-authored-by: voonhou.su <voonhou.su@shopee.com>
 2022-07-04 15:07:19 -07:00
superche
e0954040a9 [HUDI-3511] Add call procedure for MetadataCommand (#6018) 2022-07-03 21:44:56 +08:00
Shiyan Xu
c0e1587966 [HUDI-3730] Improve meta sync class design and hierarchies (#5854) 
* [HUDI-3730] Improve meta sync class design and hierarchies (#5754)
* Implements class design proposed in RFC-55

Co-authored-by: jian.feng <fengjian428@gmial.com>
Co-authored-by: jian.feng <jian.feng@shopee.com>
 2022-07-03 14:47:25 +05:30
superche
c00ea84985 [HUDI-3505] Add call procedure for UpgradeOrDowngradeCommand (#6012) 
Co-authored-by: superche <superche@tencent.com>
 2022-07-03 08:47:48 +08:00
Danny Chan
47792a3186 [HUDI-4353] Column stats data skipping for flink (#6026) 2022-07-03 08:29:31 +08:00
JerryYue-M
bdf73b2650 [HUDI-3953]Flink Hudi module should support low-level source and sink api (#5445) 
Co-authored-by: jerryyue <jerryyue@didiglobal.com>
 2022-07-02 08:38:46 +08:00
RexAn
62a0c962ac [HUDI-3634] Could read empty or partial HoodieCommitMetaData in downstream if using HDFS (#5048) 
Add the differentiated logic of creating immutable file in HDFS by first creating the file.tmp and then renaming the file
 2022-06-30 11:07:40 -07:00
miomiocat
397fd30142 [HUDI-3984] Remove mandatory check of partiton path for cli command (#5458) 2022-06-30 10:00:13 -07:00
komao
8547899a39 [HUDI-4285] add ByteBuffer#rewind after ByteBuffer#get in AvroDeseria… (#5907) 
* [HUDI-4285] add ByteBuffer#rewind after ByteBuffer#get in AvroDeserializer

* add ut

Co-authored-by: wangzixuan.wzxuan <wangzixuan.wzxuan@bytedance.com>
 2022-06-30 20:48:50 +08:00
RexAn
cdaaa3c4c7 [HUDI-4346] Fix params not update BULKINSERT_ARE_PARTITIONER_RECORDS_SORTED (#5999) 2022-06-29 19:26:00 -07:00
cxzl25
6a01f7029c [MINOR] Following #2070, Fix BindException when running tests on shared machines. (#5951) 2022-06-29 19:20:59 -07:00
luoyajun
3948b8935a [HUDI-4336] Fix records overwritten bug with binary primary key (#5996) 2022-06-30 09:12:00 +08:00
wenningd
03a94d9ff5 [HUDI-4331] Allow loading external config file from class loader (#5987) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2022-06-29 17:04:34 -07:00
YueZhang
e71f04768e [MINOR] Make CLI 'commit rollback' using rollbackUsingMarkers false as default (#5174) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-06-29 10:12:46 -07:00
YueZhang
637660b7aa [HUDI-1575] Claim RFC-56: Early Conflict Detection For Multi-writer (#6002) 
Co-authored-by: yuezhang <yuezhang@yuezhang-mac.freewheelmedia.net>
 2022-06-29 01:43:31 -07:00
Teng
e3eb14ad2d [HUDI-4334] close SparkRDDWriteClient after usage in Create/Delete/RollbackSavepointsProcedure (#5994) 2022-06-29 06:13:29 +08:00
bschell
fd7d25ab63 [HUDI-1176] Upgrade hudi to log4j2 (#5366) 
* Move to log4j2

cr: https://code.amazon.com/reviews/CR-71010705

* Upgrade unit tests to log4j2

* update exclusion

Co-authored-by: Brandon Scheller <bschelle@amazon.com>
 2022-06-28 12:54:23 -07:00
Alexey Kudinkin
ed823f1c6f [HUDI-4320] Make sure HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED could be specified by the writer (#5970) 
Fixed sequence determining whether Parquet's legacy-format writing property should be overridden to only kick in when it has not been explicitly specified by the caller
 2022-06-28 12:27:32 -07:00
BruceLin
efb9719018 [HUDI-4332] The current instant may be wrong under some extreme conditions in AppendWriteFunction. (#5988) 2022-06-28 20:42:26 +08:00
ForwardXu
08eba914ed [HUDI-4333] fix HoodieFileIndex's listFiles method log print skipping percent NaN (#5990) 2022-06-28 15:08:48 +08:00
KnightChess
09dc001430 [HUDI-4325] fix spark sql procedure cause ParseException with semicolon (#5982) 
* [HUDI-4325] fix saprk sql procedure cause ParseException with semicolon
 2022-06-28 09:44:41 +08:00
superche
b14ed47f21 [HUDI-3506] Add call procedure for CommitsCommand (#5974) 
* [HUDI-3506] Add call procedure for CommitsCommand

Co-authored-by: superche <superche@tencent.com>
 2022-06-28 09:43:36 +08:00
Sagar Sumit
8846849a03 [HUDI-4291] Fix flaky TestCleanPlanExecutor#testKeepLatestFileVersions (#5930) 2022-06-27 17:27:16 +05:30
吴祥平
3a1fd22841 [HUDI-4311] Fix Flink lose data on some rollback scene (#5950) 2022-06-27 16:09:44 +08:00
ForwardXu
26c967bac6 [HUDI-3504] Support bootstrap command based on Call Produce Command (#5977) 2022-06-27 13:06:50 +08:00
leesf
8f4e2a189e [HUDI-4315] Do not throw exception in BaseSpark3Adapter#toTableIdentifier (#5957) 2022-06-27 12:50:58 +08:00
cxzl25
72fa19bcc9 [HUDI-4316] Support for spillable diskmap configuration when constructing HoodieMergedLogRecordScanner (#5959) 2022-06-27 11:09:30 +08:00
cxzl25
7a6eb0f6e1 [HUDI-4309] Spark3.2 custom parser should not throw exception (#5947) 2022-06-27 09:37:23 +08:00
Sivabalan Narayanan
0a9e568ff5 [HUDI-5246] Bumping mysql connector version due to security vulnerability (#5851) 2022-06-26 16:54:57 -07:00
Shiyan Xu
559b26fb7c [MINOR] Remove -T option from CI build (#5972) 2022-06-26 08:34:05 -07:00
ForwardXu
1c43c590ac [HUDI-3502] Support hdfs parquet import command based on Call Produce Command (#5956) 2022-06-26 11:27:14 +08:00
xiarixiaoyao
142adf4ccb [HUDI-4296] Fix the bug that TestHoodieSparkSqlWriter.testSchemaEvolutionForTableType is flaky (#5973) 2022-06-25 21:03:19 +08:00
Alexey Kudinkin
c86edfc28e [HUDI-4319] Fixed Parquet's PLAIN_DICTIONARY encoding not being applied when bulk-inserting (#5966) 
* Fixed Dictionary encoding config not being properly propagated to Parquet writer (making it unable to apply it, substantially bloating the storage footprint)
 2022-06-24 23:52:28 -04:00
xiarixiaoyao
360df576a9 Revert "[TEST][DO_NOT_MERGE]fix random failed for ci (#5948)" (#5971) 
This reverts commit e8fbd4daf4.
 2022-06-25 11:23:17 +08:00
xiarixiaoyao
e8fbd4daf4 [TEST][DO_NOT_MERGE]fix random failed for ci (#5948) 2022-06-25 10:15:08 +08:00
jiz
eeafaeacd2 [HUDI-3512] Add call procedure for StatsCommand (#5955) 
Co-authored-by: zhanshaoxiong <shaoxiong0001@@gmail.com>
 2022-06-25 09:43:23 +08:00
luokey
59978ef4a9 [HUDI-4260] Change KEYGEN_CLASS_NAME without default value (#5877) 
* Change KEYGEN_CLASS_NAME without default value

Co-authored-by: 854194341@qq.com <loukey_7821>
 2022-06-24 15:05:03 +08:00
xi chaomin
30ebdc708b [HUDI-3735] TestHoodieSparkMergeOnReadTableRollback is flaky (#5874) 2022-06-24 02:47:36 -04:00
Zhaojing Yu
6456bd3a51 [HUDI-4273] Support inline schedule clustering for Flink stream (#5890) 
* [HUDI-4273] Support inline schedule clustering for Flink stream

* delete deprecated clustering plan strategy and add clustering ITTest
 2022-06-24 11:28:06 +08:00
jiz
af9f09047d [HUDI-3509] Add call procedure for HoodieLogFileCommand (#5949) 
Co-authored-by: zhanshaoxiong <jiimmyzhan@tencent.com>
 2022-06-24 10:16:54 +08:00
Sagar Sumit
eeb78f23e6 [HUDI-4290] Fix fetchLatestBaseFiles to filter replaced filegroups (#5941) 
* [HUDI-4290] Fix fetchLatestBaseFiles to filter replaced filegroups

* Separate out incremental sync fsview test with clustering
 2022-06-23 19:40:08 +05:30
Forus
38ff18a199 [HUDI-4299] Fix problem about hudi-example-java run failed on idea. (#5936) 2022-06-23 21:46:22 +08:00
jiz
1bb017d396 [HUDI-3508] Add call procedure for FileSystemViewCommand (#5929) 
* [HUDI-3508] Add call procedure for FileSystemView

* minor

Co-authored-by: jiimmyzhan <jiimmyzhan@tencent.com>
 2022-06-22 17:50:20 +08:00
Danny Chan
1dbd9d407a [minor] following 4270, add unit tests for the keys lost case (#5918) 2022-06-22 16:56:06 +08:00
LinMingQiang
c9590790f8 [HUDI-4279] Strength the remote fs view lagging check when latest commit refresh is enabled (#5917) 
Signed-off-by: LinMingQiang <1356469429@qq.com>
 2022-06-22 10:32:21 +08:00
Zhaojing Yu
c7e430bb46 Revert master (#5925) 
* Revert "udate"

This reverts commit 092e35c1e3.

* Revert "[HUDI-3475] Initialize hudi table management module."

This reverts commit 4640a3bbb8.
 2022-06-21 16:58:50 +08:00
喻兆靖
092e35c1e3 udate 2022-06-21 15:22:04 +08:00
喻兆靖
4640a3bbb8 [HUDI-3475] Initialize hudi table management module. 2022-06-21 15:21:30 +08:00
Bo Cui
7c4aaa9715 [HUDI-4270] Bootstrap op data loading missing (#5888) 2022-06-21 11:47:39 +08:00