lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,206 Commits 1 Branch 0 Tags
bd084704210ee0247a4b31a6a32e5de3afd7048c
Commit Graph

2206 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Danny Chan
bd08470421 [HUDI-2957] Shade kryo jar for flink bundle jar (#4251) 2021-12-09 10:16:42 +08:00
Sivabalan Narayanan
7c3f0777aa [HUDI-2964] Fixing aws lock configs to inherit from HoodieConfig (#4258) 2021-12-08 19:17:56 -05:00
Vinoth Govindarajan
082faa3851 [HUDI-2832][RFC-41] Proposal to integrate Hudi on Snowflake platform (#4074) 
* [HUDI-2832][RFC-40] Proposal to integrate Hudi on Snowflake platform

* rebased and addressed review comments
 2021-12-08 11:27:19 -08:00
Danny Chan
c56d93e7b8 [MINOR] Update DOAP with 0.10.0 Release (#4246) 2021-12-08 17:55:22 +08:00
xuzifu666
c9e18d1e7d [HUDI-2942] add error message log in HoodieCombineHiveInputFormat (#4224) 2021-12-07 22:05:39 -08:00
Danny Chan
e8473b9a2b [HUDI-2951] Disable remote view storage config for flink (#4237) 2021-12-07 18:04:15 +08:00
Sagar Sumit
6dab307e6f [MINOR] Remove redundant and conflicting spark-hive dependency (#4228) 
Disable TestHiveSchemaProvider
 2021-12-06 17:48:32 -08:00
wenningd
4a437f25d3 [MINOR] Use maven-shade-plugin version for hudi-timeline-server-bundle from main pom.xml (#4209) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2021-12-06 12:29:18 -08:00
Y Ethan Guo
2d66451a51 [MINOR] Fix partition path formatting in error log (#4168) 2021-12-06 11:11:44 -08:00
xiarixiaoyao
57c4bf8152 [HUDI-2876] for hive/presto hudi should remove the temp file which created by HoodieMergedLogRecordSanner when the query finished. (#4139) 2021-12-06 21:33:10 +08:00
Ron
84b531ae75 [HUDI-2900] Fix corrupt block end position (#4181) 
* [HUDI-2900] Fix corrupt block end position

* add a test
 2021-12-06 20:38:39 +08:00
leesf
f0e46bf522 [HUDI-2916] Add IssueNavigationLink for IDEA (#4192) 2021-12-05 22:53:54 -08:00
冯健
734c9f5f2d [HUDI-2418] Support HiveSchemaProvider (#3671) 
Co-authored-by: jian.feng <fengjian428@gmial.com>
 2021-12-05 00:10:13 -08:00
ForwardXu
63b15607ff [HUDI-2937] Introduce a pulsar implementation of hoodie write commit … (#4217) 
* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback
 2021-12-05 11:51:06 +04:00
Ron
a8fb69656f [HUDI-2877] Support flink catalog to help user use flink table conveniently (#4153) 
* [HUDI-2877] Support flink catalog to help user use flink table conveniently

* Fix comment

* fix comment2
 2021-12-05 10:14:29 +08:00
vinoth chandar
36b69d8033 [HUDI-2935] Remove special casing of clustering in deltastreamer checkpoint retrival (#4216) 
- We now seek backwards to find the checkpoint
 - No need to return empty anymore
 2021-12-04 17:16:11 +08:00
fengli
568181a3e7 [HUDI-2934] Optimize RequestHandler code style 
close apache/hudi#4215
 2021-12-04 15:30:52 +08:00
Sivabalan Narayanan
1d4fb827e7 [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight (#4206) 
* [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

* Fixing retry of pending compaction in metadata table and enhancing tests
 2021-12-03 21:44:50 -08:00
rmahindra123
94f45e928c [HUDI-2890] Kafka Connect: Fix failed writes and avoid table service concurrent operations (#4211) 
* Fix kafka connect readme

* Fix handling of errors in write records for kafka connect

* By default, ensure we skip error records and keep the pipeline alive

* Fix indentation

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-12-03 21:30:32 -08:00
vinoth chandar
0fd6b2d71e [HUDI-2933] DISABLE Metadata table by default (#4213) 2021-12-03 21:12:35 -08:00
Raymond Xu
a799fae316 [MINOR] Mitigate CI jobs timeout issues (#4173) 
* skip shutdown zookeeper in `@AfterAll` in TestHBaseIndex

* rebalance CI tests
 2021-12-03 21:08:32 -08:00
zhangyue19921010
5616830ae1 Revert "[HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests" 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-12-04 08:26:53 +05:30
Manoj Govindassamy
383d5edc16 [HUDI-2894][HUDI-2905] Metadata table - avoiding key lookup failures on base files over S3 (#4185) 
- Fetching partition files or all partitions from the metadata table is failing
   when run over S3. Metadata table uses HFile format for the base files and the
   record lookup uses HFile.Reader and HFileScanner interfaces to get records by
   partition keys. When the backing storage is S3, this record lookup from HFiles
   is failing with IOException, in turn failing the caller commit/update operations.

 - Metadata table looks up HFile records with positional read enabled so as to
   perform better for random lookups. But this positional read key lookup is
   returning with partial read sizes over S3 leading to HFile scanner throwing
   IOException. This doesn't happen over HDFS. Metadata table though uses the HFile
   for random key lookups, the positional read is not mandatory as we sort the keys
   when doing a lookup for multiple keys.

 - The fix is to disable HFile positional read for all HFile scanner based
   key lookups.
 2021-12-03 14:18:10 -05:00
Yann Byron
2f96f4300b Revert "[HUDI-2495] Resolve inconsistent key generation for timestamp types by GenericRecord and Row (#3944)" (#4201) 2021-12-03 11:13:38 -05:00
Alexey Kudinkin
bed7f9897a [HUDI-2911] Removing default value for PARTITIONPATH_FIELD_NAME resulting in incorrect KeyGenerator configuration (#4195) 2021-12-03 07:33:38 -05:00
Sivabalan Narayanan
e483f7c776 [HUDI-2902] Fixing populate meta fields with Hfile writers and Disabling virtual keys by default for metadata table (#4194) 2021-12-03 07:20:21 -05:00
Yann Byron
ca427240c0 [MINOR] use catalog schema if can not find table schema (#4182) 2021-12-03 00:37:13 -08:00
Danny Chan
0699521f83 [HUDI-2924] Refresh the fs view on successful checkpoints for write profile (#4199) 2021-12-03 16:12:59 +08:00
Danny Chan
f74b3d12aa [minor] Refactor write profile to always generate fs view (#4198) 2021-12-03 11:38:29 +08:00
Danny Chan
934fe54cc5 [HUDI-2914] Fix remote timeline server config for flink (#4191) 2021-12-03 08:59:10 +08:00
rmahindra123
91d2e61433 [HUDI-2904] Fix metadata table archival overstepping between regular writers and table services (#4186) 
- Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
- Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-12-02 13:32:26 -05:00
zzzhy
61a03bc072 [MINOR] Fix the wrong usage of timestamp length variable bug (#4179) 
Signed-off-by: zzzhy <candle_1667@163.com>
 2021-12-02 22:47:31 +08:00
Alexey Kudinkin
772f5ca24e Fixed partitions produced by layout optimization in case order-by key is composed of a single column (#4183) 2021-12-01 20:56:04 -08:00
Shawy Geng
5284730175 [HUDI-2881] Compact the file group with larger log files to reduce write amplification (#4152) 2021-12-02 09:41:04 +08:00
Sivabalan Narayanan
f4c25ba3fd [HUDI-2880] Fixing loading of props from default dir (#4167) 
* Fixing loading of props from default dir

* addressing comments
 2021-12-01 16:02:30 +08:00
Y Ethan Guo
9b254b6fc5 Revert "[HUDI-2856] Bit cask disk map delete modified (#4116)" (#4171) 
This reverts commit 257a6a7456.
 2021-12-01 14:08:44 +08:00
Alexey Kudinkin
24380c2060 Revert "[HUDI-2855] Change the default value of 'PAYLOAD_CLASS_NAME' to 'DefaultHoodieRecordPayload' (#4115)" (#4169) 
This reverts commit 88067f57a2.
 2021-11-30 17:47:16 -08:00
Y Ethan Guo
ea009b55a3 [HUDI-2891] Fix write configs for Java engine in Kafka Connect Sink (#4161) 2021-11-30 09:45:50 -05:00
董可伦
a398aad1fc [HUDI-2642] Add support ignoring case in update sql operation (#3882) 2021-11-29 22:36:36 -08:00
董可伦
3433f00cb5 [MINOR] Fix typo,rename 'getUrlEncodePartitoning' to 'getUrlEncodePartitioning' (#4130) 2021-11-29 18:31:22 -08:00
Danny Chan
536af4b954 [MINOR] Fix syntax error in create_source_release.sh (#4150) 2021-11-29 14:17:24 +08:00
Sivabalan Narayanan
38e75ea806 Removing rfc from release package and fixing release validation script (#4147) 2021-11-29 13:18:35 +08:00
Sivabalan Narayanan
52aae36b53 [MINOR] Fixing integ test suite for hudi-aws and archival validation (#4142) 2021-11-28 20:11:50 -05:00
vortual
eca1693288 [MINOR] fix typo (#4140) 2021-11-28 17:13:50 +08:00
yuzhao.cyz
a1d0ff4209 Moving to 0.11.0-SNAPSHOT on master branch. 2021-11-27 17:22:10 +08:00
xiarixiaoyao
780a2ac5b2 [HUDI-2102] Support hilbert curve for hudi (#3952) 
Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com>
 2021-11-26 23:20:19 -08:00
Manoj Govindassamy
2c7656c35f [HUDI-2475] [HUDI-2862] Metadata table creation and avoid bootstrapping race for write client & add locking for upgrade (#4114) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-11-26 23:19:26 -08:00
Raymond Xu
3a8d64e584 [HUDI-2868] Fix skipped HoodieSparkSqlWriterSuite (#4125) 
- Co-authored-by: Yann Byron <biyan900116@gmail.com>
 2021-11-26 22:59:20 -05:00
Sivabalan Narayanan
9c059ef8e5 [MINOR] Follow ups from HUDI-2861 (re-use same rollback instant for failed rollback) (#4133) 2021-11-26 19:22:53 -05:00
xuzifu666
257a6a7456 [HUDI-2856] Bit cask disk map delete modified (#4116) 
* modified BitCaskDiskMap_close_function

* change iterators location to finally

* Update BitCaskDiskMap.java
 2021-11-26 15:11:01 -08:00