lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,397 Commits 1 Branch 0 Tags
26c3f797b0fda964cf601836d6314ff6241e9d5d
Commit Graph

2397 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Byron
26c3f797b0 [HUDI-3237] gracefully fail to change column data type (#4677) 2022-01-24 16:33:36 -08:00
Alexey Kudinkin
bc7882cbe9 [HUDI-2872][HUDI-2646] Refactoring layout optimization (clustering) flow to support linear ordering (#4606) 
Refactoring layout optimization (clustering) flow to
- Enable support for linear (lexicographic) ordering as one of the ordering strategies (along w/ Z-order, Hilbert)
- Reconcile Layout Optimization and Clustering configuration to be more congruent
 2022-01-24 16:53:54 -05:00
Satyam Raj
6f10107998 [HUDI-3306] Upgrade rocksdb version (#4663) 
Co-authored-by: Satyam Raj <satyam.raj@olacabs.com>
 2022-01-24 14:53:20 -05:00
董可伦
1f7b6b2154 [HUDI-2417] Add support allowDuplicateInserts in HoodieJavaClient (#3644) 2022-01-24 14:26:27 -05:00
Vinish Reddy
87db4ded42 [MINOR] Add default value as null for S3 Incremental source properties (#4674) 2022-01-24 14:24:43 -05:00
YueZhang
7bd389fb47 [MINOR] typo fix in BaseTableMetadata wrt spurious deletes handling (#4673) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-24 07:09:54 -05:00
Sivabalan Narayanan
e00a9042e9 [HUDI-3072] Fixing conflict resolution in transaction management code path for auto commit code path (#4588) 
* Fixing conflict resolution in transaction management code path for auto commit code path

* Addressing comments

* Fixing test failures
 2022-01-24 16:13:28 +05:30
董可伦
cfde45b548 [HUDI-3282] Fix delete exception for Spark SQL when sync Hive (#4644) 2022-01-23 11:32:57 -08:00
Sivabalan Narayanan
f7a77961e3 [HUDI-1850][HUDI-3234] Fixing read of a empty table but with failed write (#2903) 2022-01-23 14:23:21 -05:00
Sivabalan Narayanan
e72553accf [HUDI-3262] Fixing utilities and integ test suite bundle to include hudi spark datasource (#4670) 2022-01-23 08:46:37 -05:00
董可伦
56cd8ffae0 [HUDI-2837] Add support for using database name in incremental query (#4083) 2022-01-22 22:11:27 -08:00
Danny Chan
64b1426005 [minor] Fix hive-exec scope of flink bundle jar (#4664) 2022-01-23 10:28:41 +08:00
Y Ethan Guo
4b9085057a [HUDI-3268] Fix NPE while reading table with Spark datasource (#4630) 2022-01-21 08:46:07 -05:00
董可伦
8547f11752 [HUDI-3271] Code optimization and clean up unused code in HoodieSparkSqlWriter (#4631) 2022-01-20 18:49:04 -05:00
YueZhang
79bf6ab00b [HUDI-3281][Performance]Tuning performance of getAllPartitionPaths API in FileSystemBackedTableMetadata (#4643) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-20 18:47:02 -05:00
Sagar Sumit
2071e3bfda [HUDI-3250] Upgrade Presto docker image (#4646) 2022-01-20 09:30:25 -08:00
wangxianghu
a66004a340 [HUDI-3285] Drop unused method SparkBootstrapCommitActionExecutor#handleMetadataBootstrap (#4653) 2022-01-20 20:04:36 +04:00
wangxianghu
14d08bb64c [MINOR] Fix typo in the doc of BULK_INSERT_SORT_MODE (#4652) 2022-01-20 15:34:56 +04:00
wangxianghu
b7a79aa943 [HUDI-3283] Bootstrap support overwrite existing table (#4647) 2022-01-20 14:42:52 +04:00
Yann Byron
31b57a256f [HUDI-3236] use fields'comments persisted in catalog to fill in schema (#4587) 2022-01-19 21:44:35 -08:00
Y Ethan Guo
a08a2b7306 [MINOR] Add instructions to build and upload Docker Demo images (#4612) 
* [MINOR] Add instructions to build and upload Docker Demo images

* Add local test instruction
 2022-01-20 09:55:28 +05:30
wangxianghu
db93ad2f4b [HUDI-3277] Filter non-parquet files in bootstrap procedure (#4639) 2022-01-19 21:13:51 +04:00
YueZhang
7647562dad [HUDI-2833][Design] Merge small archive files instead of expanding indefinitely. (#4078) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-18 22:42:35 -08:00
Alexey Kudinkin
4bea758738 [HUDI-3191] Rebasing Hive's FileInputFormat onto AbstractHoodieTableFileIndex (#4531) 2022-01-18 14:54:51 -08:00
Thinking Chen
caeea946fb [HUDI-3245] Convert uppercase letters to lowercase in storage configs (#4602) 2022-01-18 14:51:09 -05:00
Yann Byron
a09c231911 [HUDI-2903] get table schema from the last commit with data written (#4180) 2022-01-18 10:50:30 -05:00
Danny Chan
45f054ffde [HUDI-3263] Do not nullify members in HoodieTableFileSystemView#resetViewState to avoid NPE (#4625) 2022-01-18 17:46:40 +08:00
EchoLee5
3b56320bd8 [HUDI-3261] Read rt table by hive cli throw NoSuchMethodError (#4624) 2022-01-18 16:58:08 +08:00
wangxianghu
3d93e857cc [MINOR] Minor improvement in JsonkafkaSource (#4620) 2022-01-18 11:13:05 +04:00
RexAn
f18447406d [HUDI-1558] Struct Stream Source Support Spark3 (#4586) 
Co-authored-by: Hui An <hui.an@shopee.com>
 2022-01-18 11:08:33 +08:00
董可伦
20e7983866 [HUDI-3252] Avoid creating empty requestedReplaceCommit in the startCommit method (#4515) 2022-01-17 17:28:18 -05:00
Yuwei XIAO
d36533735f [HUDI-3194] fix MOR snapshot query during compaction (#4540) 2022-01-17 17:24:24 -05:00
Danny Chan
36a9f63e45 [HUDI-3257] Excluding clustering instants from pending rollback info (#4616) 2022-01-17 18:18:45 +08:00
Alexey Kudinkin
75caa7d3d8 [HUDI-3179] Extracted common AbstractHoodieTableFileIndex to be shared across engines (#4520) 2022-01-16 22:46:20 -08:00
xiaotianzhang01
ed92c217ed [MINOR] Delete unused parameter in TablePathUtils (#4595) 
Co-authored-by: zhangxiaotian13 <zhangxiaotian13@jd.com>
 2022-01-16 22:24:43 -08:00
Yann Byron
d2dda55794 [HUDI-2968] add UT for update/delete on non-pk condition (#4568) 2022-01-16 12:02:12 -08:00
0x574C
28b3b6ad8f [MINOR] Remove org.apache.directory.api.util.Strings import (#4601) 2022-01-16 16:58:18 +08:00
董可伦
822230d9ea [MINOR] Optimize variable names and logs (#4581) 2022-01-16 16:09:22 +08:00
Yann Byron
5e0171a5ee [HUDI-3198] Improve Spark SQL create table from existing hudi table (#4584) 
To modify SQL statement for creating hudi table based on an existing hudi path.

From:

```sql
create table hudi_tbl using hudi tblproperties (primaryKey='id', preCombineField='ts', type='cow') partitioned by (pt) location '/path/to/hudi'
```

To:
```sql
create table hudi_tbl using hudi location '/path/to/hudi'
```
 2022-01-14 10:15:29 -08:00
Y Ethan Guo
53f75f84b8 [HUDI-2785] Add Trino setup in Docker Demo (#4300) 
* [HUDI-2785] Add Trino setup in Docker Demo

* Update docker account and remove unnecessary configs

* Adjust sparkadhoc Dockerfile
 2022-01-14 22:08:55 +05:30
Y Ethan Guo
7d163ee3de [MINOR] Fix local flaky test in TestFSUtils (#4596) 
Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>
 2022-01-13 22:48:57 -08:00
leesf
5ce45c440b [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation (#4514) 
* Introduce hudi-spark3-common and hudi-spark2-common modules to place classes that would be reused in different spark versions, also introduce hudi-spark3.1.x to support spark 3.1.x.
* Introduce hudi format under hudi-spark2, hudi-spark3, hudi-spark3.1.x modules and change the hudi format in original hudi-spark module to hudi_v1 format.
* Manually tested on Spark 3.1.2 and Spark 3.2.0 SQL.
* Added a README.md file under hudi-spark-datasource module.
 2022-01-14 13:42:35 +08:00
Sagar Sumit
195dac90fa [MINOR] Disable flaky tests to unlock CI (#4592) 2022-01-13 19:43:27 -08:00
Sagar Sumit
209f91cb33 [HUDI-3010] Unbundle parquet-avro and shade other dependencies in prsto bundle (#4551) 2022-01-12 20:00:24 -08:00
Y Ethan Guo
397795c7d0 [HUDI-3007] Fix issues in HoodieRepairTool (#4564) 2022-01-12 09:03:27 -08:00
Sagar Sumit
12e95771ee [HUDI-3235] Fix ClassNotFoundException due to log4j-core dependency (#4574) 
- Move log4j-core to top level pom
 2022-01-12 11:53:43 -05:00
Sagar Sumit
8a40d95506 [HUDI-3225] Claim RFC-45 for async metadata indexing (#4569) 2022-01-12 11:53:01 -05:00
todd5167
2969fb3835 [HUDI-3233] Make metadata commit synchronous for flink batch 
close apache/hudi#4561
 2022-01-12 20:22:53 +08:00
YueZhang
9fe28e56b4 [HUDI-3045] New clustering regex match config to choose partitions when building clustering plan (#4346) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-11 23:23:55 -08:00
董可伦
017ddbbfac [MINOR] Fix typos (#4567) 2022-01-11 23:17:10 -08:00