lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,223 Commits 1 Branch 0 Tags
c48a2a125a9d41f4fb4bf7d6a1d01bf75ba1877a
Commit Graph

2223 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Manoj Govindassamy
c48a2a125a [HUDI-2527] Multi writer test with conflicting async table services (#4046) 2021-12-10 20:01:19 -05:00
Alexey Kudinkin
2d864f7524 [HUDI-2814] Make Z-index more generic Column-Stats Index (#4106) 2021-12-10 14:56:09 -08:00
Y Ethan Guo
72901a33a1 [HUDI-2784] Add a hudi-trino-bundle for Trino (#4279) 2021-12-10 14:27:22 -08:00
zhangyue19921010
3ba2909690 [HUDI-2892][BUG] Pending Clustering may stain the ActiveTimeLine and lead to incomplete query results (#4172) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-12-10 09:57:01 -08:00
Sivabalan Narayanan
3ce0526924 Adding verbose output for metadata validate files command (#4166) 2021-12-10 09:38:38 -08:00
yuzhaojing
3ad9b121f1 [HUDI-2912] Fix CompactionPlanOperator typo (#4187) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-12-10 09:32:53 -08:00
Sivabalan Narayanan
be368264f4 [HUDI-2952] Fixing metadata table for non-partitioned dataset (#4243) 2021-12-10 11:11:42 -05:00
Yuwei XIAO
f194566ed4 [HUDI-2849] Improve SparkUI job description for write path (#4222) 2021-12-10 23:22:37 +08:00
Sagar Sumit
c7473a7b0c [HUDI-2936] Add data count checks in async clustering tests (#4236) 2021-12-10 09:25:37 -05:00
xiarixiaoyao
456d74ce4e [HUDI-2901] Fixed the bug clustering jobs cannot running in parallel (#4178) 2021-12-09 22:39:35 -08:00
Sivabalan Narayanan
ea154bcb5d Revert "Claiming RFC for data skipping index for updated version (#4271)" (#4272) 
This reverts commit 8321d20c2c.
 2021-12-10 00:46:26 -05:00
Sivabalan Narayanan
8321d20c2c Claiming RFC for data skipping index for updated version (#4271) 2021-12-10 10:07:42 +05:30
arunkc
3fb2f974ca [MINOR] FAQ link in SUPPORT_REQUEST template (#4266) 2021-12-09 14:43:36 -08:00
xiarixiaoyao
68f8597b12 [HUDI-2966] Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScaner when the query finished. (#4265) 
* [HUDI-2966] Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScaner when the query finished.
 2021-12-09 19:51:49 +08:00
RexAn
f612a20815 [HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown (#4014) 2021-12-09 16:04:11 +05:30
leesf
5ac9ce7289 [MINOR] Fix Compile broken (#4263) 2021-12-09 13:12:18 +08:00
guanziyue
9c8ad0f0fa [HUDI-2665] Fix overflow of huge log file in HoodieLogFormatWriter (#3912) 
Co-authored-by: guanziyue.gzy <guanziyue.gzy@bytedance.com>
 2021-12-09 10:47:13 +08:00
Danny Chan
bd08470421 [HUDI-2957] Shade kryo jar for flink bundle jar (#4251) 2021-12-09 10:16:42 +08:00
Sivabalan Narayanan
7c3f0777aa [HUDI-2964] Fixing aws lock configs to inherit from HoodieConfig (#4258) 2021-12-08 19:17:56 -05:00
Vinoth Govindarajan
082faa3851 [HUDI-2832][RFC-41] Proposal to integrate Hudi on Snowflake platform (#4074) 
* [HUDI-2832][RFC-40] Proposal to integrate Hudi on Snowflake platform

* rebased and addressed review comments
 2021-12-08 11:27:19 -08:00
Danny Chan
c56d93e7b8 [MINOR] Update DOAP with 0.10.0 Release (#4246) 2021-12-08 17:55:22 +08:00
xuzifu666
c9e18d1e7d [HUDI-2942] add error message log in HoodieCombineHiveInputFormat (#4224) 2021-12-07 22:05:39 -08:00
Danny Chan
e8473b9a2b [HUDI-2951] Disable remote view storage config for flink (#4237) 2021-12-07 18:04:15 +08:00
Sagar Sumit
6dab307e6f [MINOR] Remove redundant and conflicting spark-hive dependency (#4228) 
Disable TestHiveSchemaProvider
 2021-12-06 17:48:32 -08:00
wenningd
4a437f25d3 [MINOR] Use maven-shade-plugin version for hudi-timeline-server-bundle from main pom.xml (#4209) 
Co-authored-by: Wenning Ding <wenningd@amazon.com>
 2021-12-06 12:29:18 -08:00
Y Ethan Guo
2d66451a51 [MINOR] Fix partition path formatting in error log (#4168) 2021-12-06 11:11:44 -08:00
xiarixiaoyao
57c4bf8152 [HUDI-2876] for hive/presto hudi should remove the temp file which created by HoodieMergedLogRecordSanner when the query finished. (#4139) 2021-12-06 21:33:10 +08:00
Ron
84b531ae75 [HUDI-2900] Fix corrupt block end position (#4181) 
* [HUDI-2900] Fix corrupt block end position

* add a test
 2021-12-06 20:38:39 +08:00
leesf
f0e46bf522 [HUDI-2916] Add IssueNavigationLink for IDEA (#4192) 2021-12-05 22:53:54 -08:00
冯健
734c9f5f2d [HUDI-2418] Support HiveSchemaProvider (#3671) 
Co-authored-by: jian.feng <fengjian428@gmial.com>
 2021-12-05 00:10:13 -08:00
ForwardXu
63b15607ff [HUDI-2937] Introduce a pulsar implementation of hoodie write commit … (#4217) 
* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback

* [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback
 2021-12-05 11:51:06 +04:00
Ron
a8fb69656f [HUDI-2877] Support flink catalog to help user use flink table conveniently (#4153) 
* [HUDI-2877] Support flink catalog to help user use flink table conveniently

* Fix comment

* fix comment2
 2021-12-05 10:14:29 +08:00
vinoth chandar
36b69d8033 [HUDI-2935] Remove special casing of clustering in deltastreamer checkpoint retrival (#4216) 
- We now seek backwards to find the checkpoint
 - No need to return empty anymore
 2021-12-04 17:16:11 +08:00
fengli
568181a3e7 [HUDI-2934] Optimize RequestHandler code style 
close apache/hudi#4215
 2021-12-04 15:30:52 +08:00
Sivabalan Narayanan
1d4fb827e7 [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight (#4206) 
* [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

* Fixing retry of pending compaction in metadata table and enhancing tests
 2021-12-03 21:44:50 -08:00
rmahindra123
94f45e928c [HUDI-2890] Kafka Connect: Fix failed writes and avoid table service concurrent operations (#4211) 
* Fix kafka connect readme

* Fix handling of errors in write records for kafka connect

* By default, ensure we skip error records and keep the pipeline alive

* Fix indentation

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-12-03 21:30:32 -08:00
vinoth chandar
0fd6b2d71e [HUDI-2933] DISABLE Metadata table by default (#4213) 2021-12-03 21:12:35 -08:00
Raymond Xu
a799fae316 [MINOR] Mitigate CI jobs timeout issues (#4173) 
* skip shutdown zookeeper in `@AfterAll` in TestHBaseIndex

* rebalance CI tests
 2021-12-03 21:08:32 -08:00
zhangyue19921010
5616830ae1 Revert "[HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests" 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-12-04 08:26:53 +05:30
Manoj Govindassamy
383d5edc16 [HUDI-2894][HUDI-2905] Metadata table - avoiding key lookup failures on base files over S3 (#4185) 
- Fetching partition files or all partitions from the metadata table is failing
   when run over S3. Metadata table uses HFile format for the base files and the
   record lookup uses HFile.Reader and HFileScanner interfaces to get records by
   partition keys. When the backing storage is S3, this record lookup from HFiles
   is failing with IOException, in turn failing the caller commit/update operations.

 - Metadata table looks up HFile records with positional read enabled so as to
   perform better for random lookups. But this positional read key lookup is
   returning with partial read sizes over S3 leading to HFile scanner throwing
   IOException. This doesn't happen over HDFS. Metadata table though uses the HFile
   for random key lookups, the positional read is not mandatory as we sort the keys
   when doing a lookup for multiple keys.

 - The fix is to disable HFile positional read for all HFile scanner based
   key lookups.
 2021-12-03 14:18:10 -05:00
Yann Byron
2f96f4300b Revert "[HUDI-2495] Resolve inconsistent key generation for timestamp types by GenericRecord and Row (#3944)" (#4201) 2021-12-03 11:13:38 -05:00
Alexey Kudinkin
bed7f9897a [HUDI-2911] Removing default value for PARTITIONPATH_FIELD_NAME resulting in incorrect KeyGenerator configuration (#4195) 2021-12-03 07:33:38 -05:00
Sivabalan Narayanan
e483f7c776 [HUDI-2902] Fixing populate meta fields with Hfile writers and Disabling virtual keys by default for metadata table (#4194) 2021-12-03 07:20:21 -05:00
Yann Byron
ca427240c0 [MINOR] use catalog schema if can not find table schema (#4182) 2021-12-03 00:37:13 -08:00
Danny Chan
0699521f83 [HUDI-2924] Refresh the fs view on successful checkpoints for write profile (#4199) 2021-12-03 16:12:59 +08:00
Danny Chan
f74b3d12aa [minor] Refactor write profile to always generate fs view (#4198) 2021-12-03 11:38:29 +08:00
Danny Chan
934fe54cc5 [HUDI-2914] Fix remote timeline server config for flink (#4191) 2021-12-03 08:59:10 +08:00
rmahindra123
91d2e61433 [HUDI-2904] Fix metadata table archival overstepping between regular writers and table services (#4186) 
- Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
- Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-12-02 13:32:26 -05:00
zzzhy
61a03bc072 [MINOR] Fix the wrong usage of timestamp length variable bug (#4179) 
Signed-off-by: zzzhy <candle_1667@163.com>
 2021-12-02 22:47:31 +08:00
Alexey Kudinkin
772f5ca24e Fixed partitions produced by layout optimization in case order-by key is composed of a single column (#4183) 2021-12-01 20:56:04 -08:00