lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,409 Commits 1 Branch 0 Tags
c0e8b03d93223777bd1e8e5c9dbb7edcb9e37e47
Commit Graph

2409 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
peanut-chenzhong
c0e8b03d93 [HUDI-1977] Fix Hudi CLI tempview query issue (#4626) 2022-01-29 10:39:08 +08:00
Sagar Sumit
e78b2f1b55 [HUDI-2943] Complete pending clustering before deltastreamer sync (#4572) 2022-01-28 21:28:04 -05:00
Sagar Sumit
2b52a56981 [HUDI-2688][RFC-40] A new Hudi connector for Trino (#3957) 
Add implementation details of a new hudi-trino connector
 2022-01-28 19:13:11 +05:30
Raymond Xu
0bd38f26ca [HUDI-2596] Make class names consistent in hudi-client (#4680) 2022-01-27 17:05:08 -08:00
Yann Byron
4a9f826382 [HUDI-3215] Solve UT for Spark 3.2 (#4565) 2022-01-26 14:48:26 -08:00
Sivabalan Narayanan
3f21e5f14c [MINOR] Fixing serializability of SerializableHoodieRollbackRequest (#4688) 2022-01-26 13:45:35 -08:00
Manoj Govindassamy
f87c47352a [HUDI-2763] Metadata table records - support for key deduplication based on hardcoded key field (#4449) 
* [HUDI-2763] Metadata table records - support for key deduplication and virtual keys
- The backing log format for the metadata table is HFile, a KeyValue type.
Since the key field in the metadata record payload is a duplicate of the
Key in the Cell, the redundant key field in the record can be emptied
to save on the cost.

- HoodieHFileWriter and HoodieHFileDataBlock will now serialize records
with the key field emptied by default. HFile writer tries to find if
the record has metadata payload schema field 'key' and if so it does
the key trimming from the record payload.

- HoodieHFileReader when reading the serialized records back from disk,
it materializes the missing keyFields if any. HFile reader tries to
find if the record has metadata payload schema fiels 'key' and if so
it does the key materialization in the record payload.

- Tests have been added to verify the default virtual keys and key
   deduplication support for the metadata table records.

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2022-01-26 13:34:04 -05:00
Sivabalan Narayanan
dd4ce1bdfd [HUDI-3328] Updating doap file for release 0.10.1 (#4689) 2022-01-26 08:45:57 -05:00
Sivabalan Narayanan
9363804b1d [MINOR] Fixing serializability with ListingBasedRollbackRequest (#4655) 2022-01-25 19:35:37 -05:00
Alexey Kudinkin
78e6ab0e67 [HUDI-3217] Claim the number for RFC-46 (#4687) 2022-01-25 14:58:34 -08:00
Sivabalan Narayanan
920f45926a [HUDI-1822] Rewriting rfc-27 for data skipping index (#4280) 2022-01-25 00:27:59 -05:00
xuzifu666
bf409e8423 [MINOR] Standardize HoodieSqlCommon.g4 file (#4582) 2022-01-25 10:09:08 +08:00
Yann Byron
26c3f797b0 [HUDI-3237] gracefully fail to change column data type (#4677) 2022-01-24 16:33:36 -08:00
Alexey Kudinkin
bc7882cbe9 [HUDI-2872][HUDI-2646] Refactoring layout optimization (clustering) flow to support linear ordering (#4606) 
Refactoring layout optimization (clustering) flow to
- Enable support for linear (lexicographic) ordering as one of the ordering strategies (along w/ Z-order, Hilbert)
- Reconcile Layout Optimization and Clustering configuration to be more congruent
 2022-01-24 16:53:54 -05:00
Satyam Raj
6f10107998 [HUDI-3306] Upgrade rocksdb version (#4663) 
Co-authored-by: Satyam Raj <satyam.raj@olacabs.com>
 2022-01-24 14:53:20 -05:00
董可伦
1f7b6b2154 [HUDI-2417] Add support allowDuplicateInserts in HoodieJavaClient (#3644) 2022-01-24 14:26:27 -05:00
Vinish Reddy
87db4ded42 [MINOR] Add default value as null for S3 Incremental source properties (#4674) 2022-01-24 14:24:43 -05:00
YueZhang
7bd389fb47 [MINOR] typo fix in BaseTableMetadata wrt spurious deletes handling (#4673) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-24 07:09:54 -05:00
Sivabalan Narayanan
e00a9042e9 [HUDI-3072] Fixing conflict resolution in transaction management code path for auto commit code path (#4588) 
* Fixing conflict resolution in transaction management code path for auto commit code path

* Addressing comments

* Fixing test failures
 2022-01-24 16:13:28 +05:30
董可伦
cfde45b548 [HUDI-3282] Fix delete exception for Spark SQL when sync Hive (#4644) 2022-01-23 11:32:57 -08:00
Sivabalan Narayanan
f7a77961e3 [HUDI-1850][HUDI-3234] Fixing read of a empty table but with failed write (#2903) 2022-01-23 14:23:21 -05:00
Sivabalan Narayanan
e72553accf [HUDI-3262] Fixing utilities and integ test suite bundle to include hudi spark datasource (#4670) 2022-01-23 08:46:37 -05:00
董可伦
56cd8ffae0 [HUDI-2837] Add support for using database name in incremental query (#4083) 2022-01-22 22:11:27 -08:00
Danny Chan
64b1426005 [minor] Fix hive-exec scope of flink bundle jar (#4664) 2022-01-23 10:28:41 +08:00
Y Ethan Guo
4b9085057a [HUDI-3268] Fix NPE while reading table with Spark datasource (#4630) 2022-01-21 08:46:07 -05:00
董可伦
8547f11752 [HUDI-3271] Code optimization and clean up unused code in HoodieSparkSqlWriter (#4631) 2022-01-20 18:49:04 -05:00
YueZhang
79bf6ab00b [HUDI-3281][Performance]Tuning performance of getAllPartitionPaths API in FileSystemBackedTableMetadata (#4643) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-20 18:47:02 -05:00
Sagar Sumit
2071e3bfda [HUDI-3250] Upgrade Presto docker image (#4646) 2022-01-20 09:30:25 -08:00
wangxianghu
a66004a340 [HUDI-3285] Drop unused method SparkBootstrapCommitActionExecutor#handleMetadataBootstrap (#4653) 2022-01-20 20:04:36 +04:00
wangxianghu
14d08bb64c [MINOR] Fix typo in the doc of BULK_INSERT_SORT_MODE (#4652) 2022-01-20 15:34:56 +04:00
wangxianghu
b7a79aa943 [HUDI-3283] Bootstrap support overwrite existing table (#4647) 2022-01-20 14:42:52 +04:00
Yann Byron
31b57a256f [HUDI-3236] use fields'comments persisted in catalog to fill in schema (#4587) 2022-01-19 21:44:35 -08:00
Y Ethan Guo
a08a2b7306 [MINOR] Add instructions to build and upload Docker Demo images (#4612) 
* [MINOR] Add instructions to build and upload Docker Demo images

* Add local test instruction
 2022-01-20 09:55:28 +05:30
wangxianghu
db93ad2f4b [HUDI-3277] Filter non-parquet files in bootstrap procedure (#4639) 2022-01-19 21:13:51 +04:00
YueZhang
7647562dad [HUDI-2833][Design] Merge small archive files instead of expanding indefinitely. (#4078) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-18 22:42:35 -08:00
Alexey Kudinkin
4bea758738 [HUDI-3191] Rebasing Hive's FileInputFormat onto AbstractHoodieTableFileIndex (#4531) 2022-01-18 14:54:51 -08:00
Thinking Chen
caeea946fb [HUDI-3245] Convert uppercase letters to lowercase in storage configs (#4602) 2022-01-18 14:51:09 -05:00
Yann Byron
a09c231911 [HUDI-2903] get table schema from the last commit with data written (#4180) 2022-01-18 10:50:30 -05:00
Danny Chan
45f054ffde [HUDI-3263] Do not nullify members in HoodieTableFileSystemView#resetViewState to avoid NPE (#4625) 2022-01-18 17:46:40 +08:00
EchoLee5
3b56320bd8 [HUDI-3261] Read rt table by hive cli throw NoSuchMethodError (#4624) 2022-01-18 16:58:08 +08:00
wangxianghu
3d93e857cc [MINOR] Minor improvement in JsonkafkaSource (#4620) 2022-01-18 11:13:05 +04:00
RexAn
f18447406d [HUDI-1558] Struct Stream Source Support Spark3 (#4586) 
Co-authored-by: Hui An <hui.an@shopee.com>
 2022-01-18 11:08:33 +08:00
董可伦
20e7983866 [HUDI-3252] Avoid creating empty requestedReplaceCommit in the startCommit method (#4515) 2022-01-17 17:28:18 -05:00
Yuwei XIAO
d36533735f [HUDI-3194] fix MOR snapshot query during compaction (#4540) 2022-01-17 17:24:24 -05:00
Danny Chan
36a9f63e45 [HUDI-3257] Excluding clustering instants from pending rollback info (#4616) 2022-01-17 18:18:45 +08:00
Alexey Kudinkin
75caa7d3d8 [HUDI-3179] Extracted common AbstractHoodieTableFileIndex to be shared across engines (#4520) 2022-01-16 22:46:20 -08:00
xiaotianzhang01
ed92c217ed [MINOR] Delete unused parameter in TablePathUtils (#4595) 
Co-authored-by: zhangxiaotian13 <zhangxiaotian13@jd.com>
 2022-01-16 22:24:43 -08:00
Yann Byron
d2dda55794 [HUDI-2968] add UT for update/delete on non-pk condition (#4568) 2022-01-16 12:02:12 -08:00
0x574C
28b3b6ad8f [MINOR] Remove org.apache.directory.api.util.Strings import (#4601) 2022-01-16 16:58:18 +08:00
董可伦
822230d9ea [MINOR] Optimize variable names and logs (#4581) 2022-01-16 16:09:22 +08:00