lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,074 Commits 1 Branch 0 Tags
1ee12cfa6fa8d18546af11a7ae03cb0e4969460f
Commit Graph

2074 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
wenningd
1ee12cfa6f [HUDI-2314] Add support for DynamoDb based lock provider (#3486) 
- Co-authored-by: Wenning Ding <wenningd@amazon.com>
- Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-11-17 12:09:31 -05:00
卢波
826414cff5 [MINOR] Add the Schema for GooseFS to StorageSchemes (#3982) 
Co-authored-by: lubo <bollu@tencent.com>
 2021-11-17 22:47:52 +08:00
董可伦
4d884bdaa9 [MINOR] Fix typo,'Hooide' corrected to 'Hoodie' (#4007) 2021-11-17 16:50:04 +08:00
0x574C
aec5d11da2 Check --source-avro-schema-path parameter (#3987) 
Co-authored-by: 0x3E6 <dragon1996>
 2021-11-17 14:45:43 +08:00
Sivabalan Narayanan
ce7d233307 [HUDI-2151] Part3 Enabling marker based rollback as default rollback strategy (#3950) 
* Enabling timeline server based markers

* Enabling timeline server based markers and marker based rollback

* Removing constraint that timeline server can be enabled only for hdfs

* Fixing tests
 2021-11-17 11:51:28 +05:30
Sivabalan Narayanan
04eb5fdc65 [HUDI-2753] Ensure list based rollback strategy is used for restore (#3983) 2021-11-17 10:06:55 +05:30
Alexey Kudinkin
cbcbec4d38 [MINOR] Fixed checkstyle config to be based off Maven root-dir (requires Maven >=3.3.1 to work properly); (#4009) 
Updated README
 2021-11-16 21:30:16 -05:00
Danny Chan
6f5e661010 [HUDI-2769] Fix StreamerUtil#medianInstantTime for very near instant time (#4005) 2021-11-16 13:46:34 +08:00
Sivabalan Narayanan
bff8769ed4 [HUDI-2712] Fixing a bug with rollback of partially failed commit which has new partitions (#3947) 2021-11-15 22:36:03 -05:00
zhangyue19921010
38b6934352 [HUDI-2683] Parallelize deleting archived hoodie commits (#3920) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-11-15 22:36:54 +08:00
Sivabalan Narayanan
53d2d6ae24 [HUDI-2744] Fix parsing of metadadata table compaction timestamp when metrics are enabled (#3976) 2021-11-15 07:27:35 -05:00
dufeng1010
3c4319729c [MINOR] Fix typo in IntervalTreeBasedGlobalIndexFileFilter (#3993) 
Co-authored-by: 闫杜峰 <yandufeng@sinochem.com>
 2021-11-15 14:39:43 +08:00
xiarixiaoyao
a0dae41409 [HUDI-2758] remove redundant code in the hoodieRealtimeInputFormatUitls.getRealtimeSplits (#3994) 2021-11-15 11:29:40 +08:00
Manoj Govindassamy
a14d1040b9 [HUDI-2589] Claiming RFC-37 for Metadata based bloom index feature. (#3995) 2021-11-14 20:47:41 -05:00
Yann Byron
0bb6d8ff80 [HUDI-2706] refactor spark-sql to make consistent with DataFrame api (#3936) 2021-11-14 15:44:39 -08:00
Danny Chan
c2f9094b49 [HUDI-2756] Fix flink parquet writer decimal type conversion (#3988) 2021-11-14 08:51:54 +08:00
Manoj Govindassamy
994922a159 [HUDI-2472] Enabling metadata table in TestHoodieIndex and TestMergeOnReadRollbackActionExecutor (#3978) 
- With rollback after first commit support added to metadata table, these test cases are safe to have metadata table turned on.
 2021-11-13 19:37:30 -05:00
xiarixiaoyao
0e8461e9ab [HUDI-2697] Minor changes about hbase index config. (#3927) 2021-11-13 09:12:33 +08:00
Sivabalan Narayanan
93fd3517e3 [HUDI-2741] Fixing instantiating metadata table config in HoodieFileIndex (#3974) 2021-11-12 17:28:25 -05:00
Manoj Govindassamy
9720820975 [HUDI-2718] ExternalSpillableMap payload size re-estimation throws ArithmeticException (#3955) 
- ExternalSpillableMap does the payload/value size estimation on the first put to
  determine when to spill over to disk map. The payload size re-estimation also
  happens after a minimum threshold of puts. This size re-estimation goes my the
  current in-memory map size for calculating average payload size and does attempts
  divide by zero operation when the map is size is empty. Avoiding the
  ArithmeticException during the payload size re-estimate by checking the map size
  upfront.
 2021-11-12 08:18:40 -05:00
Sivabalan Narayanan
4f217fe718 [HUDI-2151] Part1 Setting default parallelism to 200 for some of write configs (#3948) 2021-11-12 07:29:37 -05:00
Danny Chan
bc511edc85 [HUDI-2746] Do not bootstrap for flink insert overwrite (#3980) 2021-11-12 12:17:58 +08:00
yuzhaojing
6b93ccca9b [HUDI-2738] Remove the bucketAssignFunction useless context (#3972) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-11-11 21:03:01 +08:00
Yann Byron
90529aa552 [HUDI-2495] Resolve inconsistent key generation for timestamp types by GenericRecord and Row (#3944) 2021-11-11 11:54:34 +08:00
Prashant Wason
77b0440eb4 [HUDI-2634] Improved the metadata table bootstrap for very large tables. (#3873) 
* [HUDI-2634] Improved the metadata table bootstrap for very large tables.

Following improvements are implemented:
1. Memory overhead reduction:
  - Existing code caches FileStatus for each file in memory.
  - Created a new class DirectoryInfo which is used to cache a director's file list with parts of the FileStatus (only filename and file len). This reduces the memory requirements.

2. Improved parallelism:
  - Existing code collects all the listing to the Driver and then creates HoodieRecord on the Driver.
  - This takes a long time for large tables (11million HoodieRecords to be created)
  - Created a new function in SparkRDDWriteClient specifically for bootstrap commit. In it, the HoodieRecord creation is parallelized across executors so it completes fast.

3. Fixed setting to limit the number of parallel listings:
  - Existing code had a bug wherein 1500 executors were hardcoded to perform listing. This leads to exception due to limit in the spark's result memory.
  - Corrected the use of the config.

Result:
Dataset has 1299 partitions and 12Million files.
file listing time=1.5mins
HoodieRecord creation time=13seconds
deltacommit duration=2.6mins

Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-11-10 22:37:48 -05:00
yuzhaojing
90f9b4562a [HUDI-2685] Support scheduling online compaction plan when there are no commit data (#3928) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-11-11 10:13:21 +08:00
yuzhaojing
2d362af00a [HUDI-2730] Move EventTimeAvroPayload into hudi-common module (#3959) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-11-10 20:22:24 +08:00
Sagar Sumit
187bedf795 [HUDI-2442] Change default values for certin clustering configs (#3875) 2021-11-10 14:23:24 +05:30
xiarixiaoyao
a40ac62e0c [HUDI-2086]redo the logical of mor_incremental_view for hive (#3203) 2021-11-10 15:41:07 +08:00
Shawy Geng
fd0f5df26d [HUDI-2297] Estimate available memory size for spillable map accurately. (#3455) 2021-11-10 14:05:12 +08:00
Sagar Sumit
bb6a19e7d7 [HUDI-1877] Support records staying in same fileId after clustering (#3833) 
* [HUDI-1877] Support records staying in same fileId after clustering

Add plan strategy

* Ensure same filegroup id and refactor based on comments
 2021-11-09 20:17:50 -08:00
davehagman
dfe3b84715 [HUDI-2579] Make deltastreamer checkpoint state merging more explicit (#3820) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-11-09 17:37:59 -05:00
Prashant Wason
2f95967dfe [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required. (#3836) 2021-11-09 10:26:20 -05:00
Danny Chan
e057a10499 [HUDI-2715] The BitCaskDiskMap iterator may cause memory leak (#3951) 2021-11-09 15:40:00 +08:00
Sivabalan Narayanan
6d109c6de5 [HUDI-2595] Fixing metadata table updates such that only regular writes from data table can trigger table services in metadata table (#3900) 2021-11-08 22:12:32 -05:00
yuzhaojing
7aaf47e716 [HUDI-2698] Remove the table source options validation (#3940) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-11-08 16:56:03 +08:00
Danny Chan
c7bf2c7687 [HUDI-2709] Add more options when initializing table (#3939) 2021-11-08 15:08:49 +08:00
Shawy Geng
cf2ecd77ba [HUDI-2679] Fix the TestMergeIntoLogOnlyTable typo. (#3918) 2021-11-07 10:19:17 -08:00
manasaks
e0285800fb HUDI-1827 : Add ORC support in Bootstrap Op (#3457) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-11-06 12:23:20 -04:00
Genmao Yu
f41539a9cb [HUDI-313] bugfix: NPE when select count start from a realtime table with Tez(#3630) 
Co-authored-by: dylonyu <dylonyu@tencent.com>
 2021-11-06 12:16:13 -04:00
Danny Chan
9a8963d05e [HUDI-2702] Set up keygen class explicit for write config for flink table upgrade (#3931) 2021-11-06 12:23:15 +08:00
Sagar Sumit
08c35a55b3 [HUDI-2526] Make spark.sql.parquet.writeLegacyFormat configurable (#3917) 2021-11-05 13:03:41 -04:00
董可伦
844346c3ab [HUDI-2471] Add support ignoring case in merge into (#3700) 2021-11-05 07:50:16 -07:00
Prashant Wason
b7ee341e14 [HUDI-1794] Moved static COMMIT_FORMATTER to thread local variable as SimpleDateFormat is not thread safe. (#2819) 2021-11-05 09:31:42 -04:00
Danny Chan
3af6568d31 [HUDI-2696] Remove the aborted checkpoint notification from coordinator (#3926) 2021-11-05 16:37:23 +08:00
yuzhaojing
f67da0c7d0 [HUDI-2686] Proccess record after all bootstrap operator ready (#3925) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-11-05 14:36:22 +08:00
yuzhaojing
2c1e259329 [HUDI-2651] Sync all the missing sql options for HoodieFlinkStreamer (#3903) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-11-05 12:16:21 +08:00
Danny Chan
33436aa359 Revert "[HUDI-2677] Add DFS based message queue for flink writer (#3915)" (#3923) 
This reverts commit dbf8c44bdb.
 2021-11-04 20:48:57 +08:00
Danny Chan
dbf8c44bdb [HUDI-2677] Add DFS based message queue for flink writer (#3915) 2021-11-04 18:09:00 +08:00
Danny Chan
689020f303 [HUDI-2684] Use DefaultHoodieRecordPayload when precombine field is specified specifically (#3922) 2021-11-04 16:23:36 +08:00