lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,418 Commits 1 Branch 0 Tags
72f7348830a5a2ca320720d34d12f8317e11ceff
Commit Graph

2418 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Manoj Govindassamy
72f7348830 [HUDI-2589] RFC-37: Metadata table based bloom index (#3989) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2022-02-01 18:38:20 -05:00
Sivabalan Narayanan
16138db4f2 [HUDI-3368] Revert "[HUDI-3306] Upgrade rocksdb version (#4663)" (#4733) 
This reverts commit 6f10107998.
 2022-02-01 14:18:38 -05:00
Sivabalan Narayanan
4e61e5c9ea [HUDI-3293] Fixing default value for clustering small file config to 300MB (#4662) 2022-02-01 08:22:37 -05:00
Sivabalan Narayanan
f140c58d9e [HUDI-3346] Fixing non existant marker dir handling in TwoToOnedowngrade (#4726) 2022-02-01 08:21:55 -05:00
jsbali
7ce0f4522b [HUDI-2711] Fallback to fulltable scan for IncrementalRelation if underlying files have been cleared or moved by cleaner (#3946) 
Co-authored-by: sivabalan <n.siva.b@gmail.com>
 2022-01-31 23:03:18 -05:00
Sivabalan Narayanan
4b388c104e [HUDI-3292] Enabling lazy read by default for log blocks during compaction (#4661) 2022-01-31 22:36:17 -05:00
Alexey Kudinkin
d3cfe07436 [HUDI-3318] [RFC-46] Optimize Record Payload handling (#4697) 2022-01-31 18:33:35 -08:00
Yann Byron
ecbad9526a [HUDI-3253] preferred to use the table's own location (#4608) 2022-01-29 00:39:42 -08:00
Harsha Teja Kanna
ed7aa138e8 [MINOR] Added log to debug checkpoint resumption when set to 0 (#4650) 2022-01-28 23:08:25 -05:00
peanut-chenzhong
c0e8b03d93 [HUDI-1977] Fix Hudi CLI tempview query issue (#4626) 2022-01-29 10:39:08 +08:00
Sagar Sumit
e78b2f1b55 [HUDI-2943] Complete pending clustering before deltastreamer sync (#4572) 2022-01-28 21:28:04 -05:00
Sagar Sumit
2b52a56981 [HUDI-2688][RFC-40] A new Hudi connector for Trino (#3957) 
Add implementation details of a new hudi-trino connector
 2022-01-28 19:13:11 +05:30
Raymond Xu
0bd38f26ca [HUDI-2596] Make class names consistent in hudi-client (#4680) 2022-01-27 17:05:08 -08:00
Yann Byron
4a9f826382 [HUDI-3215] Solve UT for Spark 3.2 (#4565) 2022-01-26 14:48:26 -08:00
Sivabalan Narayanan
3f21e5f14c [MINOR] Fixing serializability of SerializableHoodieRollbackRequest (#4688) 2022-01-26 13:45:35 -08:00
Manoj Govindassamy
f87c47352a [HUDI-2763] Metadata table records - support for key deduplication based on hardcoded key field (#4449) 
* [HUDI-2763] Metadata table records - support for key deduplication and virtual keys
- The backing log format for the metadata table is HFile, a KeyValue type.
Since the key field in the metadata record payload is a duplicate of the
Key in the Cell, the redundant key field in the record can be emptied
to save on the cost.

- HoodieHFileWriter and HoodieHFileDataBlock will now serialize records
with the key field emptied by default. HFile writer tries to find if
the record has metadata payload schema field 'key' and if so it does
the key trimming from the record payload.

- HoodieHFileReader when reading the serialized records back from disk,
it materializes the missing keyFields if any. HFile reader tries to
find if the record has metadata payload schema fiels 'key' and if so
it does the key materialization in the record payload.

- Tests have been added to verify the default virtual keys and key
   deduplication support for the metadata table records.

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2022-01-26 13:34:04 -05:00
Sivabalan Narayanan
dd4ce1bdfd [HUDI-3328] Updating doap file for release 0.10.1 (#4689) 2022-01-26 08:45:57 -05:00
Sivabalan Narayanan
9363804b1d [MINOR] Fixing serializability with ListingBasedRollbackRequest (#4655) 2022-01-25 19:35:37 -05:00
Alexey Kudinkin
78e6ab0e67 [HUDI-3217] Claim the number for RFC-46 (#4687) 2022-01-25 14:58:34 -08:00
Sivabalan Narayanan
920f45926a [HUDI-1822] Rewriting rfc-27 for data skipping index (#4280) 2022-01-25 00:27:59 -05:00
xuzifu666
bf409e8423 [MINOR] Standardize HoodieSqlCommon.g4 file (#4582) 2022-01-25 10:09:08 +08:00
Yann Byron
26c3f797b0 [HUDI-3237] gracefully fail to change column data type (#4677) 2022-01-24 16:33:36 -08:00
Alexey Kudinkin
bc7882cbe9 [HUDI-2872][HUDI-2646] Refactoring layout optimization (clustering) flow to support linear ordering (#4606) 
Refactoring layout optimization (clustering) flow to
- Enable support for linear (lexicographic) ordering as one of the ordering strategies (along w/ Z-order, Hilbert)
- Reconcile Layout Optimization and Clustering configuration to be more congruent
 2022-01-24 16:53:54 -05:00
Satyam Raj
6f10107998 [HUDI-3306] Upgrade rocksdb version (#4663) 
Co-authored-by: Satyam Raj <satyam.raj@olacabs.com>
 2022-01-24 14:53:20 -05:00
董可伦
1f7b6b2154 [HUDI-2417] Add support allowDuplicateInserts in HoodieJavaClient (#3644) 2022-01-24 14:26:27 -05:00
Vinish Reddy
87db4ded42 [MINOR] Add default value as null for S3 Incremental source properties (#4674) 2022-01-24 14:24:43 -05:00
YueZhang
7bd389fb47 [MINOR] typo fix in BaseTableMetadata wrt spurious deletes handling (#4673) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-24 07:09:54 -05:00
Sivabalan Narayanan
e00a9042e9 [HUDI-3072] Fixing conflict resolution in transaction management code path for auto commit code path (#4588) 
* Fixing conflict resolution in transaction management code path for auto commit code path

* Addressing comments

* Fixing test failures
 2022-01-24 16:13:28 +05:30
董可伦
cfde45b548 [HUDI-3282] Fix delete exception for Spark SQL when sync Hive (#4644) 2022-01-23 11:32:57 -08:00
Sivabalan Narayanan
f7a77961e3 [HUDI-1850][HUDI-3234] Fixing read of a empty table but with failed write (#2903) 2022-01-23 14:23:21 -05:00
Sivabalan Narayanan
e72553accf [HUDI-3262] Fixing utilities and integ test suite bundle to include hudi spark datasource (#4670) 2022-01-23 08:46:37 -05:00
董可伦
56cd8ffae0 [HUDI-2837] Add support for using database name in incremental query (#4083) 2022-01-22 22:11:27 -08:00
Danny Chan
64b1426005 [minor] Fix hive-exec scope of flink bundle jar (#4664) 2022-01-23 10:28:41 +08:00
Y Ethan Guo
4b9085057a [HUDI-3268] Fix NPE while reading table with Spark datasource (#4630) 2022-01-21 08:46:07 -05:00
董可伦
8547f11752 [HUDI-3271] Code optimization and clean up unused code in HoodieSparkSqlWriter (#4631) 2022-01-20 18:49:04 -05:00
YueZhang
79bf6ab00b [HUDI-3281][Performance]Tuning performance of getAllPartitionPaths API in FileSystemBackedTableMetadata (#4643) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-20 18:47:02 -05:00
Sagar Sumit
2071e3bfda [HUDI-3250] Upgrade Presto docker image (#4646) 2022-01-20 09:30:25 -08:00
wangxianghu
a66004a340 [HUDI-3285] Drop unused method SparkBootstrapCommitActionExecutor#handleMetadataBootstrap (#4653) 2022-01-20 20:04:36 +04:00
wangxianghu
14d08bb64c [MINOR] Fix typo in the doc of BULK_INSERT_SORT_MODE (#4652) 2022-01-20 15:34:56 +04:00
wangxianghu
b7a79aa943 [HUDI-3283] Bootstrap support overwrite existing table (#4647) 2022-01-20 14:42:52 +04:00
Yann Byron
31b57a256f [HUDI-3236] use fields'comments persisted in catalog to fill in schema (#4587) 2022-01-19 21:44:35 -08:00
Y Ethan Guo
a08a2b7306 [MINOR] Add instructions to build and upload Docker Demo images (#4612) 
* [MINOR] Add instructions to build and upload Docker Demo images

* Add local test instruction
 2022-01-20 09:55:28 +05:30
wangxianghu
db93ad2f4b [HUDI-3277] Filter non-parquet files in bootstrap procedure (#4639) 2022-01-19 21:13:51 +04:00
YueZhang
7647562dad [HUDI-2833][Design] Merge small archive files instead of expanding indefinitely. (#4078) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-18 22:42:35 -08:00
Alexey Kudinkin
4bea758738 [HUDI-3191] Rebasing Hive's FileInputFormat onto AbstractHoodieTableFileIndex (#4531) 2022-01-18 14:54:51 -08:00
Thinking Chen
caeea946fb [HUDI-3245] Convert uppercase letters to lowercase in storage configs (#4602) 2022-01-18 14:51:09 -05:00
Yann Byron
a09c231911 [HUDI-2903] get table schema from the last commit with data written (#4180) 2022-01-18 10:50:30 -05:00
Danny Chan
45f054ffde [HUDI-3263] Do not nullify members in HoodieTableFileSystemView#resetViewState to avoid NPE (#4625) 2022-01-18 17:46:40 +08:00
EchoLee5
3b56320bd8 [HUDI-3261] Read rt table by hive cli throw NoSuchMethodError (#4624) 2022-01-18 16:58:08 +08:00
wangxianghu
3d93e857cc [MINOR] Minor improvement in JsonkafkaSource (#4620) 2022-01-18 11:13:05 +04:00