lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
1,607 Commits 1 Branch 0 Tags
5ce64a81bdfaa94aefacf7b365e6fb044763827c
Commit Graph

1607 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
swuferhong
5ce64a81bd Fix the filter condition is missing in the judgment condition of compaction instance (#3025) 2021-06-16 14:28:53 -07:00
Wei
d519c74626 [HUDI-2008] Avoid the raw type usage in some classes under hudi-utilities module (#3076) 2021-06-16 22:37:29 +08:00
swuferhong
8b0a502c4f [HUDI-2014] Support flink hive sync in batch mode (#3081) 2021-06-16 14:29:16 +08:00
yuzhaojing
61efc6af79 [HUDI-2022] Release writer for append handle #close (#3087) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-16 09:18:38 +08:00
vinoth chandar
910fe4842c [MINOR] Rename broken codecov file (#3088) 
- Stop polluting PRs with wrong coverage info
- Retaining the file, so someone can try digging in
 2021-06-15 18:05:50 -07:00
Jintao Guan
b8fe5b91d5 [HUDI-764] [HUDI-765] ORC reader writer Implementation (#2999) 
Co-authored-by: Qingyun (Teresa) Kang <kteresa@uber.com>
 2021-06-15 15:21:43 -07:00
Danny Chan
cb642ceb75 [HUDI-1999] Refresh the base file view cache for WriteProfile (#3067) 
Refresh the view to discover new small files.
 2021-06-15 08:18:38 -07:00
Raymond Xu
f922837064 [HUDI-1950] Fix Azure CI failure in TestParquetUtils (#2984) 
* fix azure pipeline configs

* add pentaho.org in maven repositories

* Make sure file paths with scheme in TestParquetUtils

* add azure build status to README
 2021-06-15 03:45:17 -07:00
Prashant Wason
515ce8eb36 [MINOR] Fixed the log which should only be printed when the Metadata Table is disabled. (#3080) 2021-06-15 16:18:15 +08:00
Vinay Patil
769dd2d7c9 [HUDI-2004] Move CheckpointUtils test cases to independant class (#3072) 2021-06-14 17:14:59 +08:00
Sivabalan Narayanan
7d9f9d7d82 [HUDI-1991] Fixing drop dups exception in bulk insert row writer path (#3055) 2021-06-14 09:55:52 +08:00
yuzhaojing
6e78682cea [HUDI-2000] Release file writer for merge handle #close (#3068) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-13 18:09:48 +08:00
swuferhong
0c4f2fdc15 [HUDI-1984] Support independent flink hudi compaction function (#3046) 2021-06-13 15:04:46 +08:00
Wei
ba728d822f [HUDI-2002] Modify HiveIncrementalPuller log level to ERROR (#3070) 
Co-authored-by: wei.zhang2 <wei.zhang2@dmall.com>
 2021-06-12 10:21:43 -07:00
Xuedong Luan
673d62f3c3 [MINOR] Add Tencent Cloud HDFS storage support for hudi (#3064) 2021-06-11 09:16:51 +08:00
Vinoth Govindarajan
9e4114dd46 [HUDI-1790] Added SqlSource to fetch data from any partitions for backfill use case (#2896) 2021-06-10 18:03:07 -04:00
Danny Chan
125415a8b8 [HUDI-1994] Release the new records iterator for append handle #close (#3058) 2021-06-10 19:09:23 +08:00
JunZhang
e0108e972e [MINOR] Add Baidu BOS storage support for hudi (#3061) 
Co-authored-by: zhangjun30 <zhangjun30@baidu.com>
 2021-06-10 15:51:36 +08:00
Wei
a8b10e9067 [MINOR] Remove boxing (#3062) 2021-06-10 13:03:32 +08:00
Danny Chan
afbafe7046 [HUDI-1992] Release the new records map for merge handle #close (#3056) 2021-06-09 21:12:56 +08:00
yuzhaojing
728089a888 delete duplicate bootstrap function (#3052) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-09 19:29:57 +08:00
Danny Chan
e8fcf04b57 [HUDI-1987] Fix non partition table hive meta sync for flink writer (#3049) 2021-06-09 14:20:04 +08:00
Danny Chan
a6f5fc5967 [HUDI-1986] Skip creating marker files for flink merge handle (#3047) 2021-06-09 14:17:28 +08:00
Vinay Patil
11360f707e [HUDI-1892] Fix NPE when avro field value is null (#3051) 2021-06-08 18:12:18 -04:00
Wei
75d663f65d [HUDI-1980] Optimize the code to prevent other exceptions from causing resources not to be closed (#3038) 
Co-authored-by: wei.zhang2 <wei.zhang2@dmall.com>
 2021-06-08 21:58:34 +08:00
wangxianghu
7261f08507 [HUDI-1929] Support configure KeyGenerator by type (#2993) 2021-06-08 09:26:10 -04:00
pengzhiwei
f760ec543e [HUDI-1659] Basic Implement Of Spark Sql Support For Hoodie (#2645) 
Main functions:
Support create table for hoodie.
Support CTAS.
Support Insert for hoodie. Including dynamic partition and static partition insert.
Support MergeInto for hoodie.
Support DELETE
Support UPDATE
Both support spark2 & spark3 based on DataSourceV1.

Main changes:
Add sql parser for spark2.
Add HoodieAnalysis for sql resolve and logical plan rewrite.
Add commands implementation for CREATE TABLE、INSERT、MERGE INTO & CTAS.
In order to push down the update&insert logical to the HoodieRecordPayload for MergeInto, I make same change to the
HoodieWriteHandler and other related classes.
1、Add the inputSchema for parser the incoming record. This is because the inputSchema for MergeInto is different from writeSchema as there are some transforms in the update& insert expression.
2、Add WRITE_SCHEMA to HoodieWriteConfig to pass the write schema for merge into.
3、Pass properties to HoodieRecordPayload#getInsertValue to pass the insert expression and table schema.


Verify this pull request
Add TestCreateTable for test create hoodie tables and CTAS.
Add TestInsertTable for test insert hoodie tables.
Add TestMergeIntoTable for test merge hoodie tables.
Add TestUpdateTable for test update hoodie tables.
Add TestDeleteTable for test delete hoodie tables.
Add TestSqlStatement for test supported ddl/dml currently.
 2021-06-07 23:24:32 -07:00
yuzhaojing
cf83f10f5b add BootstrapFunction to support index bootstrap (#3024) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-08 13:55:25 +08:00
Vinoth Govindarajan
57611d10b5 [HUDI-1743] Added support for SqlFileBasedTransformer (#2747) 2021-06-07 21:48:27 -04:00
Sivabalan Narayanan
919590988a [HUDI-1914] Add fetching latest schema to table command in hudi-cli (#2964) 2021-06-07 16:04:35 -07:00
Raymond Xu
441076b2cc [HUDI-1950] Move TestHiveMetastoreBasedLockProvider to functional (#3043) 
HiveTestUtil static setup mini servers caused connection refused issue in Azure CI environment, as TestHiveSyncTool and TestHiveMetastoreBasedLockProvider share the same test facilities. Moving TestHiveMetastoreBasedLockProvider (the easier one) to functional test with a separate and improved mini server setup resolved the issue.

Also cleaned up dfs cluster from HiveTestUtil.

The next step is to move TestHiveSyncTool to functional as well.
 2021-06-07 15:38:59 -07:00
Vinay Patil
f3d7b49bfe [HUDI-1148] Remove Hadoop Conf Logs (#3040) 2021-06-07 14:49:55 -07:00
Vinay Patil
0d0dc6fb07 [HUDI-1909] Skip Commits with empty files (#3045) 2021-06-07 21:58:19 +08:00
Danny Chan
08464a6a5b [HUDI-1931] BucketAssignFunction use ValueState instead of MapState (#3026) 
Co-authored-by: 854194341@qq.com <loukey_7821>
 2021-06-06 10:40:15 +08:00
Vinay Patil
2a7e1e091e [HUDI-1942] Add Default value for HIVE_AUTO_CREATE_DATABASE_OPT_KEY in HoodieSparkSqlWriter (#3036) 2021-06-05 18:02:26 -04:00
Wei
dab13f7473 [HUDI-1979] Optimize logic to improve code readability (#3037) 
Co-authored-by: wei.zhang2 <wei.zhang2@dmall.com>
 2021-06-05 19:40:45 +08:00
Danny Chan
c2383ee904 [HUDI-1967] Fix the NPE for MOR Hive rt table query (#3032) 
The HoodieInputFormatUtils.getTableMetaClientByBasePath returns the map
with table base path as keys while the HoodieRealtimeInputFormatUtils
query it with the partition path.
 2021-06-05 01:06:34 -07:00
Vinay Patil
cf90f17732 [HUDI-1281] Add deltacommit to ActionType (#3018) 
Co-authored-by: veenaypatil <vinay18.patil@gmail.com>
 2021-06-04 22:30:48 -07:00
yuzhaojing
c4a2ad2702 [HUDI-1954] only reset bucket when flush bucket success (#3029) 
Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com>
 2021-06-04 20:48:08 -07:00
vinoth chandar
d02c0e5387 [MINOR] Resolve build issue arising from inaccessible pentaho jar (#3034) 
- Fixes #160 #2479
 2021-06-04 15:28:44 -04:00
Danny Chan
a658328001 [HUDI-1961] Add a debezium json integration test case for flink (#3030) 2021-06-04 15:15:32 +08:00
wangxianghu
870e97b5f8 [MINOR] Remove unused method in DataSourceUtils (#3031) 2021-06-03 10:24:51 -07:00
Wei
f6eee77636 [MINOR] Remove the implementation of Serializable from HoodieException (#3020) 2021-06-03 19:46:33 +08:00
swuferhong
ad72691d24 [HUDI-1957] Fix flink timeline service lack jetty dependency (#3028) 2021-06-03 19:39:31 +08:00
taylorliao
86007e9a13 [HUDI-1953] Fix NPE due to not set the output type of the operator (#3023) 
Co-authored-by: enter58xuan <enter58xuan@zto.com>
 2021-06-03 14:20:57 +08:00
swuferhong
05a9830e86 [HUDI-1952] Fix hive3 meta sync for flink writer (#3021) 2021-06-02 14:12:03 +08:00
Danny Chan
7fa2f8ea82 [HUDI-1921] Add target io option for flink compaction (#2980) 2021-06-02 10:10:35 +08:00
Danny Chan
bf1cfb5635 [HUDI-1949] Refactor BucketAssigner to make it more efficient (#3017) 
Add a process single class WriteProfile, the record and small files
profile re-construction can be more efficient if we reuse by same
checkpoint id.
 2021-06-02 09:12:35 +08:00
taylorliao
83c31e356f [HUDI-1927] Improve HoodieFlinkStreamer (#3019) 
Co-authored-by: enter58xuan <enter58xuan@zto.com>
 2021-06-01 18:35:14 +08:00
hk__lrzy
83b0301c1a [HUDI-1943] Lose properties when hoodieWriteConfig initializtion (#3006) 
* [hudi-flink]fix lose properties problem

Co-authored-by: haoke <haoke@bytedance.com>
 2021-06-01 16:09:48 +08:00