lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,041 Commits 1 Branch 0 Tags
e057a10499729301ebe96d7cd54902b113af1811
Commit Graph

2041 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Danny Chan
e057a10499 [HUDI-2715] The BitCaskDiskMap iterator may cause memory leak (#3951) 2021-11-09 15:40:00 +08:00
Sivabalan Narayanan
6d109c6de5 [HUDI-2595] Fixing metadata table updates such that only regular writes from data table can trigger table services in metadata table (#3900) 2021-11-08 22:12:32 -05:00
yuzhaojing
7aaf47e716 [HUDI-2698] Remove the table source options validation (#3940) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-11-08 16:56:03 +08:00
Danny Chan
c7bf2c7687 [HUDI-2709] Add more options when initializing table (#3939) 2021-11-08 15:08:49 +08:00
Shawy Geng
cf2ecd77ba [HUDI-2679] Fix the TestMergeIntoLogOnlyTable typo. (#3918) 2021-11-07 10:19:17 -08:00
manasaks
e0285800fb HUDI-1827 : Add ORC support in Bootstrap Op (#3457) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-11-06 12:23:20 -04:00
Genmao Yu
f41539a9cb [HUDI-313] bugfix: NPE when select count start from a realtime table with Tez(#3630) 
Co-authored-by: dylonyu <dylonyu@tencent.com>
 2021-11-06 12:16:13 -04:00
Danny Chan
9a8963d05e [HUDI-2702] Set up keygen class explicit for write config for flink table upgrade (#3931) 2021-11-06 12:23:15 +08:00
Sagar Sumit
08c35a55b3 [HUDI-2526] Make spark.sql.parquet.writeLegacyFormat configurable (#3917) 2021-11-05 13:03:41 -04:00
董可伦
844346c3ab [HUDI-2471] Add support ignoring case in merge into (#3700) 2021-11-05 07:50:16 -07:00
Prashant Wason
b7ee341e14 [HUDI-1794] Moved static COMMIT_FORMATTER to thread local variable as SimpleDateFormat is not thread safe. (#2819) 2021-11-05 09:31:42 -04:00
Danny Chan
3af6568d31 [HUDI-2696] Remove the aborted checkpoint notification from coordinator (#3926) 2021-11-05 16:37:23 +08:00
yuzhaojing
f67da0c7d0 [HUDI-2686] Proccess record after all bootstrap operator ready (#3925) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-11-05 14:36:22 +08:00
yuzhaojing
2c1e259329 [HUDI-2651] Sync all the missing sql options for HoodieFlinkStreamer (#3903) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-11-05 12:16:21 +08:00
Danny Chan
33436aa359 Revert "[HUDI-2677] Add DFS based message queue for flink writer (#3915)" (#3923) 
This reverts commit dbf8c44bdb.
 2021-11-04 20:48:57 +08:00
Danny Chan
dbf8c44bdb [HUDI-2677] Add DFS based message queue for flink writer (#3915) 2021-11-04 18:09:00 +08:00
Danny Chan
689020f303 [HUDI-2684] Use DefaultHoodieRecordPayload when precombine field is specified specifically (#3922) 2021-11-04 16:23:36 +08:00
Danny Chan
8932c4f7b5 [HUDI-2678] flink writer writes huge log file (#3916) 2021-11-03 22:12:49 +08:00
xiarixiaoyao
acc40625f5 [HUDI-2676] Hudi should synchronize owner information to hudi _rt/_ro table. (#3911) 2021-11-03 20:36:01 +08:00
peanut-chenzhong
f2fb1aac26 [HUDI-2509] OverwriteNonDefaultsWithLatestAvroPayload doesn`t work when upsert data with some null value column (#3761) 
Co-authored-by: 502395931@qq.com <lzyadam315>
 2021-11-03 08:23:40 -04:00
Danny Chan
7fc7e9b2bc [HUDI-2660] Delete the view storage properties first before creation (#3899) 2021-11-03 14:30:20 +08:00
xiarixiaoyao
5517d292f9 [HUDI-2674] hudi hive reader should not print read values. (#3910) 2021-11-02 23:10:18 -04:00
Alexey Kudinkin
b12a25b0b1 [MINOR] Fixed RAT config for "hudi-utilities-bundle" to ignore transient build-bound artifiacts (#3909) 2021-11-02 23:06:26 -04:00
Yann Byron
6351e5f4d0 [HUDI-2538] persist some configs to hoodie.properties when the first write (#3823) 2021-11-03 10:04:23 +08:00
Yann Byron
1f17467f73 [HUDI-1869] Upgrading Spark3 To 3.1 (#3844) 
Co-authored-by: pengzhiwei <pengzhiwei2015@icloud.com>
 2021-11-02 18:25:12 -07:00
Carl-Zhou-CN
dee3a14aae [HUDI-2582] Support concurrent key gen for different tables with row writer path (#3817) 
Co-authored-by: yao.zhou <yao.zhou@linkflowtech.com>
 2021-11-02 18:05:09 -04:00
xiarixiaoyao
d194643b49 [HUDI-2101][RFC-28] support z-order for hudi (#3330) 
* [HUDI-2101]support z-order for hudi

* Renaming some configs for consistency/simplicity.

* Minor code cleanups

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-11-02 09:31:57 -07:00
Sivabalan Narayanan
f9bc3e03e5 [MINOR] Adding a deprecated constructor to AbstractSyncHoodieClient (#3902) 2021-11-02 12:16:38 -04:00
董可伦
f239187da8 [HUDI-2515] Add close when producing records failed (#3746) 2021-11-02 07:43:20 -04:00
Manoj Govindassamy
2eaf0fd939 [HUDI-2472] Enabling Metadata table for some of TestCleaner unit tests (#3803) 
- Making use of HoodieTableMetadataWriter when constructing the HoodieMetadataTestTable
   instance for the test to enable metadata table usage.
 2021-11-02 06:54:36 -04:00
Sivabalan Narayanan
35111131c3 [HUDI-2005] Fixing partition path creation in AbstractTableFileSystemView (#3769) 2021-11-02 00:16:45 -04:00
Sagar Sumit
5302b9a4ef [HUDI-2662] Downloads from Nexus Pentaho repo taking too long (#3901) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-11-01 19:14:48 -04:00
vinoyang
13b637ddc3 [HUDI-2643] Remove duplicated hbase-common with tests classifier exists in bundles (#3886) 2021-11-01 20:11:00 +08:00
Danny Chan
87c6f9cd07 [HUDI-2654] Add compaction failed event(part2) (#3896) 2021-10-31 17:51:11 +08:00
Danny Chan
92a3c458bd [HUDI-2654] Schedules the compaction from earliest for flink (#3891) 2021-10-30 08:37:30 +08:00
Manoj Govindassamy
f6326693ef [HUDI-1295] Hash ID generator util for Hudi table columns, partition and files (#3884) 
* [HUDI-1295] Hash ID generator util for Hudi table columns, partition and files

- Adding a new utility class HashID to generate 32,64,128 bits hashes for any
  given message of string or byte array type. This class internally uses
  MessageDigest and xxhash libraries.

- Adding stateful hash holders for Hudi table columns, partition and files to
  pass around for metaindex and to convert to base64encoded strings whenever
  needed
 2021-10-29 19:19:38 -04:00
Sagar Sumit
5b1992a92d [HUDI-1500] Support replace commit in DeltaSync with commit metadata preserved (#3802) 2021-10-29 13:09:09 -04:00
Sivabalan Narayanan
29574af239 [HUDI-2573] Fixing double locking with multi-writers (#3827) 
- There are two code paths, where we are taking double locking. this was added as part of adding data table locks to update metadata table. Fixing those flows to avoid taking locks if a parent transaction already acquired a lock.
 2021-10-29 12:14:39 -04:00
Sivabalan Narayanan
69ee790a47 [HUDI-1294] Adding inline read and seek based read(batch get) for hfile log blocks in metadata table (#3762) 2021-10-29 12:12:44 -04:00
Y Ethan Guo
0223c442ec [HUDI-2502] Refactor index in hudi-client module (#3778) 
- Refactor Index to reduce Line of Code and re-use across engines.
 2021-10-28 04:16:00 -04:00
Danny Chan
e5b6b8602c [HUDI-2633] Make precombine field optional for flink (#3874) 2021-10-28 13:52:06 +08:00
vinoth chandar
af3ffe5e42 [MINOR] Add links to all the existing RFCs in rfc/README.md (#3876) 2021-10-27 05:25:19 -07:00
Danny Chan
909c3ba45e [HUDI-2632] Schema evolution for flink parquet reader (#3872) 2021-10-27 20:00:24 +08:00
Pratyaksh Sharma
ae000795d7 [HUDI-1475]: fixed java doc for precombine api (#3867) 2021-10-26 15:15:20 -07:00
Y Ethan Guo
643b1a4df2 [MINOR] Fix README for hudi-kafka-connect (#3858) 2021-10-26 14:45:52 -07:00
vinoyang
b1c4acf0ae [HUDI-2614] Remove duplicated hadoop-hdfs with tests classifier exists in bundles (#3864) 2021-10-26 22:36:10 +08:00
Sivabalan Narayanan
e3fc74668f [HUDI-2625] Revert "[HUDI-2005] Avoiding direct fs calls in HoodieLogFileReader (#3757)" (#3863) 
This reverts commit 1bb0532563.
 2021-10-25 21:43:15 -04:00
董可伦
4b5512e685 [MINOR] Fix typo,'deseralized' corrected to 'deserialized' & 'Kyro' corrected to 'Kryo' (#3846) 2021-10-25 09:56:47 -04:00
vinoyang
220bf6a7e6 [HUDI-2600] Remove duplicated hadoop-common with tests classifier exists in bundles (#3847) 2021-10-25 13:45:28 +08:00
Sivabalan Narayanan
1bb0532563 [HUDI-2005] Avoiding direct fs calls in HoodieLogFileReader (#3757) 2021-10-25 01:21:08 -04:00