lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,161 Commits 1 Branch 0 Tags
780a2ac5b208b06f53ee64186b5392ebc37eadbc
Commit Graph

2161 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Sivabalan Narayanan
35111131c3 [HUDI-2005] Fixing partition path creation in AbstractTableFileSystemView (#3769) 2021-11-02 00:16:45 -04:00
Sagar Sumit
5302b9a4ef [HUDI-2662] Downloads from Nexus Pentaho repo taking too long (#3901) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-11-01 19:14:48 -04:00
vinoyang
13b637ddc3 [HUDI-2643] Remove duplicated hbase-common with tests classifier exists in bundles (#3886) 2021-11-01 20:11:00 +08:00
Danny Chan
87c6f9cd07 [HUDI-2654] Add compaction failed event(part2) (#3896) 2021-10-31 17:51:11 +08:00
Danny Chan
92a3c458bd [HUDI-2654] Schedules the compaction from earliest for flink (#3891) 2021-10-30 08:37:30 +08:00
Manoj Govindassamy
f6326693ef [HUDI-1295] Hash ID generator util for Hudi table columns, partition and files (#3884) 
* [HUDI-1295] Hash ID generator util for Hudi table columns, partition and files

- Adding a new utility class HashID to generate 32,64,128 bits hashes for any
  given message of string or byte array type. This class internally uses
  MessageDigest and xxhash libraries.

- Adding stateful hash holders for Hudi table columns, partition and files to
  pass around for metaindex and to convert to base64encoded strings whenever
  needed
 2021-10-29 19:19:38 -04:00
Sagar Sumit
5b1992a92d [HUDI-1500] Support replace commit in DeltaSync with commit metadata preserved (#3802) 2021-10-29 13:09:09 -04:00
Sivabalan Narayanan
29574af239 [HUDI-2573] Fixing double locking with multi-writers (#3827) 
- There are two code paths, where we are taking double locking. this was added as part of adding data table locks to update metadata table. Fixing those flows to avoid taking locks if a parent transaction already acquired a lock.
 2021-10-29 12:14:39 -04:00
Sivabalan Narayanan
69ee790a47 [HUDI-1294] Adding inline read and seek based read(batch get) for hfile log blocks in metadata table (#3762) 2021-10-29 12:12:44 -04:00
Y Ethan Guo
0223c442ec [HUDI-2502] Refactor index in hudi-client module (#3778) 
- Refactor Index to reduce Line of Code and re-use across engines.
 2021-10-28 04:16:00 -04:00
Danny Chan
e5b6b8602c [HUDI-2633] Make precombine field optional for flink (#3874) 2021-10-28 13:52:06 +08:00
vinoth chandar
af3ffe5e42 [MINOR] Add links to all the existing RFCs in rfc/README.md (#3876) 2021-10-27 05:25:19 -07:00
Danny Chan
909c3ba45e [HUDI-2632] Schema evolution for flink parquet reader (#3872) 2021-10-27 20:00:24 +08:00
Pratyaksh Sharma
ae000795d7 [HUDI-1475]: fixed java doc for precombine api (#3867) 2021-10-26 15:15:20 -07:00
Y Ethan Guo
643b1a4df2 [MINOR] Fix README for hudi-kafka-connect (#3858) 2021-10-26 14:45:52 -07:00
vinoyang
b1c4acf0ae [HUDI-2614] Remove duplicated hadoop-hdfs with tests classifier exists in bundles (#3864) 2021-10-26 22:36:10 +08:00
Sivabalan Narayanan
e3fc74668f [HUDI-2625] Revert "[HUDI-2005] Avoiding direct fs calls in HoodieLogFileReader (#3757)" (#3863) 
This reverts commit 1bb0532563.
 2021-10-25 21:43:15 -04:00
董可伦
4b5512e685 [MINOR] Fix typo,'deseralized' corrected to 'deserialized' & 'Kyro' corrected to 'Kryo' (#3846) 2021-10-25 09:56:47 -04:00
vinoyang
220bf6a7e6 [HUDI-2600] Remove duplicated hadoop-common with tests classifier exists in bundles (#3847) 2021-10-25 13:45:28 +08:00
Sivabalan Narayanan
1bb0532563 [HUDI-2005] Avoiding direct fs calls in HoodieLogFileReader (#3757) 2021-10-25 01:21:08 -04:00
Raymond Xu
d8560377c3 [HUDI-2077] Fix TestHoodieDeltaStreamerWithMultiWriter (#3849) 
Remove the logic of using deltastreamer to prep test table. Use fixture (compressed test table) instead.
 2021-10-24 21:14:39 -07:00
mincwang
91845e241d [MINOR] Show source table operator details on the flink web when reading hudi table (#3842) 2021-10-24 23:18:01 +08:00
Manoj Govindassamy
c9d641cc30 [HUDI-2468] Metadata table support for rolling back the first commit (#3843) 
- Fix is to make Metadata table writer creation aware of the currently inflight action so that it can
  make some informed decision about whether bootstrapping is needed for the table and whether
  any pending action on the data timeline can be ignored.
 2021-10-23 10:07:09 -04:00
Y Ethan Guo
5ed35bff83 [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module (#3741) 2021-10-22 15:58:51 -04:00
zhangyue19921010
1e285dc399 [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests (#3719) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-10-22 12:03:58 -04:00
Matrix42
499af7c039 [HUDI-2592] Fix write empty array when write.precombine.field is decimal type (#3837) 2021-10-22 19:42:13 +08:00
Manoj Govindassamy
84ca981cd0 [HUDI-2553] Metadata table compaction trigger max delta commits (#3794) 
-  Setting the max delta commits default value from 24 to 10 to trigger the compaction in metadata table.
 2021-10-21 13:09:37 -04:00
vinoyang
b480294e79 [HUDI-2507] Generate more dependency list file for other bundles (#3773) 2021-10-21 14:10:01 +08:00
Danny Chan
aa3c4ecda5 [HUDI-2583] Refactor TestWriteCopyOnWrite test cases (#3832) 2021-10-21 12:36:41 +08:00
Raymond Xu
f5d7362ee8 [HUDI-2077] Fix flakiness in TestHoodieDeltaStreamer (#3829) 2021-10-20 23:57:12 -04:00
Manoj Govindassamy
b68c5a68f9 [HUDI-2472] Fix few Cleaner tests with metadata table enabled (#3825) 2021-10-20 18:57:00 -04:00
Danny Chan
e355ab52db [HUDI-2578] Support merging small files for flink insert operation (#3822) 2021-10-20 21:10:07 +08:00
rmahindra123
3686c25fae [HUDI-2469] [Kafka Connect] Replace json based payload with protobuf for Transaction protocol. (#3694) 
* Substitue Control Event with protobuf

* Fix tests

* Fix unit tests

* Add javadocs

* Add javadocs

* Address reviewer comments

Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
 2021-10-19 14:29:48 -07:00
董可伦
46f0496a08 [MINOR] Fix typo,'intance' corrected to 'instance' (#3788) 2021-10-19 23:16:48 +08:00
Yann Byron
1e2be85a0f [HUDI-2482] support 'drop partition' sql (#3754) 2021-10-19 22:09:53 +08:00
jaxonzhang
60d4cb505a [MINOR] Fix typo, 'upsert' corrected to 'insert' in java write example (#3809) 2021-10-19 20:04:18 +08:00
Danny Chan
3a78be9203 [HUDI-2572] Strength flink compaction rollback strategy (#3819) 
* make the events of commit task distinct by file id
* fix the existence check for inflight state file
* make the compaction task fail-safe
 2021-10-19 10:47:38 +08:00
Manoj Govindassamy
335e80ea1b [HUDI-2561] BitCaskDiskMap - avoiding hostname resolution when logging messages (#3811) 
- InetAddress.getLocalHost() can take up as much as 30+seconds if the network
   configurations are not done right. This might be due to local hostname
   missing IPv6 address mapping in /etc/hosts or network configs slowing down
   any IPv6 name resolutions. If this API is used for logging verbose messages
   and that too in the hot code path, it can lead to order of magnitude
   slowness in the overall task completion.
 2021-10-18 10:07:53 -07:00
Danny Chan
588a34aa95 [HUDI-2571] Remove include-flink-sql-connector-hive profile from flink bundle (#3818) 2021-10-18 17:34:49 +08:00
yiduwangkai
dfdfbbedae HUDI-2569 shaded hive (#3816) 
Co-authored-by: wangkai9 <wangkai9@tuhu.cn>
 2021-10-18 17:12:13 +08:00
Danny Chan
3025f4d796 [HUDI-2568] Simplify the view storage config properties (#3815) 2021-10-18 14:42:33 +08:00
yiduwangkai
5276850415 [HUDI-2557] Shade javax.servlet for flink bundle jar (#3807) 
Co-authored-by: wangkai9 <wangkai9@tuhu.cn>
 2021-10-18 11:26:21 +08:00
Danny Chan
2eda3de7f9 [HUDI-2562] Embedded timeline server on JobManager (#3812) 2021-10-18 10:45:39 +08:00
Jimmy.Zhou
9aa7cfb802 [MINOR] fix typo,'seprarated' corrected to 'separated' (#3789) 2021-10-15 13:26:16 -07:00
Danny Chan
2c370cbae0 [HUDI-2556] Tweak some default config options for flink (#3800) 
* rename write.insert.drop.duplicates to write.precombine and set it as true for COW table
* set index.global.enabled default as true
* set compaction.target_io default as 500GB
 2021-10-14 19:42:56 +08:00
Danny Chan
f897e6d73e [HUDI-2551] Support DefaultHoodieRecordPayload for flink (#3792) 2021-10-14 13:46:53 +08:00
Danny Chan
abf3e3fe71 [HUDI-2548] Flink streaming reader misses the rolling over file handles (#3787) 2021-10-14 10:36:18 +08:00
Sivabalan Narayanan
cff384d23f [HUDI-2552] Fixing some test failures to unblock broken CI master (#3793) 2021-10-13 18:44:43 -04:00
zhangyue19921010
e6711b171a [HUDI-2435][BUG]Fix clustering handle errors (#3666) 
* done

* remove unused imports

* code reviewed

* code reviewed

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-10-12 15:24:48 -07:00
Sivabalan Narayanan
8a487eafa7 [HUDI-2494] Fixing glob pattern to skip all hoodie meta paths (#3768) 2021-10-12 14:06:40 -04:00