lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,071 Commits 1 Branch 0 Tags
aec5d11da2e8a025fb6c8659a648f9e2d36990d6
Commit Graph

618 Commits

Author SHA1 Message Date
Sivabalan Narayanan
ce7d233307 [HUDI-2151] Part3 Enabling marker based rollback as default rollback strategy (#3950) 
* Enabling timeline server based markers

* Enabling timeline server based markers and marker based rollback

* Removing constraint that timeline server can be enabled only for hdfs

* Fixing tests
 2021-11-17 11:51:28 +05:30
Sivabalan Narayanan
04eb5fdc65 [HUDI-2753] Ensure list based rollback strategy is used for restore (#3983) 2021-11-17 10:06:55 +05:30
Sivabalan Narayanan
bff8769ed4 [HUDI-2712] Fixing a bug with rollback of partially failed commit which has new partitions (#3947) 2021-11-15 22:36:03 -05:00
zhangyue19921010
38b6934352 [HUDI-2683] Parallelize deleting archived hoodie commits (#3920) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-11-15 22:36:54 +08:00
Sivabalan Narayanan
53d2d6ae24 [HUDI-2744] Fix parsing of metadadata table compaction timestamp when metrics are enabled (#3976) 2021-11-15 07:27:35 -05:00
dufeng1010
3c4319729c [MINOR] Fix typo in IntervalTreeBasedGlobalIndexFileFilter (#3993) 
Co-authored-by: 闫杜峰 <yandufeng@sinochem.com>
 2021-11-15 14:39:43 +08:00
Danny Chan
c2f9094b49 [HUDI-2756] Fix flink parquet writer decimal type conversion (#3988) 2021-11-14 08:51:54 +08:00
Manoj Govindassamy
994922a159 [HUDI-2472] Enabling metadata table in TestHoodieIndex and TestMergeOnReadRollbackActionExecutor (#3978) 
- With rollback after first commit support added to metadata table, these test cases are safe to have metadata table turned on.
 2021-11-13 19:37:30 -05:00
xiarixiaoyao
0e8461e9ab [HUDI-2697] Minor changes about hbase index config. (#3927) 2021-11-13 09:12:33 +08:00
Sivabalan Narayanan
4f217fe718 [HUDI-2151] Part1 Setting default parallelism to 200 for some of write configs (#3948) 2021-11-12 07:29:37 -05:00
Prashant Wason
77b0440eb4 [HUDI-2634] Improved the metadata table bootstrap for very large tables. (#3873) 
* [HUDI-2634] Improved the metadata table bootstrap for very large tables.

Following improvements are implemented:
1. Memory overhead reduction:
  - Existing code caches FileStatus for each file in memory.
  - Created a new class DirectoryInfo which is used to cache a director's file list with parts of the FileStatus (only filename and file len). This reduces the memory requirements.

2. Improved parallelism:
  - Existing code collects all the listing to the Driver and then creates HoodieRecord on the Driver.
  - This takes a long time for large tables (11million HoodieRecords to be created)
  - Created a new function in SparkRDDWriteClient specifically for bootstrap commit. In it, the HoodieRecord creation is parallelized across executors so it completes fast.

3. Fixed setting to limit the number of parallel listings:
  - Existing code had a bug wherein 1500 executors were hardcoded to perform listing. This leads to exception due to limit in the spark's result memory.
  - Corrected the use of the config.

Result:
Dataset has 1299 partitions and 12Million files.
file listing time=1.5mins
HoodieRecord creation time=13seconds
deltacommit duration=2.6mins

Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-11-10 22:37:48 -05:00
yuzhaojing
90f9b4562a [HUDI-2685] Support scheduling online compaction plan when there are no commit data (#3928) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-11-11 10:13:21 +08:00
yuzhaojing
2d362af00a [HUDI-2730] Move EventTimeAvroPayload into hudi-common module (#3959) 
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
 2021-11-10 20:22:24 +08:00
Sagar Sumit
187bedf795 [HUDI-2442] Change default values for certin clustering configs (#3875) 2021-11-10 14:23:24 +05:30
Shawy Geng
fd0f5df26d [HUDI-2297] Estimate available memory size for spillable map accurately. (#3455) 2021-11-10 14:05:12 +08:00
Sagar Sumit
bb6a19e7d7 [HUDI-1877] Support records staying in same fileId after clustering (#3833) 
* [HUDI-1877] Support records staying in same fileId after clustering

Add plan strategy

* Ensure same filegroup id and refactor based on comments
 2021-11-09 20:17:50 -08:00
davehagman
dfe3b84715 [HUDI-2579] Make deltastreamer checkpoint state merging more explicit (#3820) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-11-09 17:37:59 -05:00
Prashant Wason
2f95967dfe [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required. (#3836) 2021-11-09 10:26:20 -05:00
Danny Chan
e057a10499 [HUDI-2715] The BitCaskDiskMap iterator may cause memory leak (#3951) 2021-11-09 15:40:00 +08:00
Sivabalan Narayanan
6d109c6de5 [HUDI-2595] Fixing metadata table updates such that only regular writes from data table can trigger table services in metadata table (#3900) 2021-11-08 22:12:32 -05:00
manasaks
e0285800fb HUDI-1827 : Add ORC support in Bootstrap Op (#3457) 
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
 2021-11-06 12:23:20 -04:00
Sagar Sumit
08c35a55b3 [HUDI-2526] Make spark.sql.parquet.writeLegacyFormat configurable (#3917) 2021-11-05 13:03:41 -04:00
Prashant Wason
b7ee341e14 [HUDI-1794] Moved static COMMIT_FORMATTER to thread local variable as SimpleDateFormat is not thread safe. (#2819) 2021-11-05 09:31:42 -04:00
Danny Chan
689020f303 [HUDI-2684] Use DefaultHoodieRecordPayload when precombine field is specified specifically (#3922) 2021-11-04 16:23:36 +08:00
Danny Chan
8932c4f7b5 [HUDI-2678] flink writer writes huge log file (#3916) 2021-11-03 22:12:49 +08:00
Yann Byron
6351e5f4d0 [HUDI-2538] persist some configs to hoodie.properties when the first write (#3823) 2021-11-03 10:04:23 +08:00
Yann Byron
1f17467f73 [HUDI-1869] Upgrading Spark3 To 3.1 (#3844) 
Co-authored-by: pengzhiwei <pengzhiwei2015@icloud.com>
 2021-11-02 18:25:12 -07:00
xiarixiaoyao
d194643b49 [HUDI-2101][RFC-28] support z-order for hudi (#3330) 
* [HUDI-2101]support z-order for hudi

* Renaming some configs for consistency/simplicity.

* Minor code cleanups

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-11-02 09:31:57 -07:00
Manoj Govindassamy
2eaf0fd939 [HUDI-2472] Enabling Metadata table for some of TestCleaner unit tests (#3803) 
- Making use of HoodieTableMetadataWriter when constructing the HoodieMetadataTestTable
   instance for the test to enable metadata table usage.
 2021-11-02 06:54:36 -04:00
Sivabalan Narayanan
35111131c3 [HUDI-2005] Fixing partition path creation in AbstractTableFileSystemView (#3769) 2021-11-02 00:16:45 -04:00
Sagar Sumit
5b1992a92d [HUDI-1500] Support replace commit in DeltaSync with commit metadata preserved (#3802) 2021-10-29 13:09:09 -04:00
Sivabalan Narayanan
29574af239 [HUDI-2573] Fixing double locking with multi-writers (#3827) 
- There are two code paths, where we are taking double locking. this was added as part of adding data table locks to update metadata table. Fixing those flows to avoid taking locks if a parent transaction already acquired a lock.
 2021-10-29 12:14:39 -04:00
Sivabalan Narayanan
69ee790a47 [HUDI-1294] Adding inline read and seek based read(batch get) for hfile log blocks in metadata table (#3762) 2021-10-29 12:12:44 -04:00
Y Ethan Guo
0223c442ec [HUDI-2502] Refactor index in hudi-client module (#3778) 
- Refactor Index to reduce Line of Code and re-use across engines.
 2021-10-28 04:16:00 -04:00
vinoyang
b1c4acf0ae [HUDI-2614] Remove duplicated hadoop-hdfs with tests classifier exists in bundles (#3864) 2021-10-26 22:36:10 +08:00
vinoyang
220bf6a7e6 [HUDI-2600] Remove duplicated hadoop-common with tests classifier exists in bundles (#3847) 2021-10-25 13:45:28 +08:00
Raymond Xu
d8560377c3 [HUDI-2077] Fix TestHoodieDeltaStreamerWithMultiWriter (#3849) 
Remove the logic of using deltastreamer to prep test table. Use fixture (compressed test table) instead.
 2021-10-24 21:14:39 -07:00
Manoj Govindassamy
c9d641cc30 [HUDI-2468] Metadata table support for rolling back the first commit (#3843) 
- Fix is to make Metadata table writer creation aware of the currently inflight action so that it can
  make some informed decision about whether bootstrapping is needed for the table and whether
  any pending action on the data timeline can be ignored.
 2021-10-23 10:07:09 -04:00
Y Ethan Guo
5ed35bff83 [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module (#3741) 2021-10-22 15:58:51 -04:00
Raymond Xu
f5d7362ee8 [HUDI-2077] Fix flakiness in TestHoodieDeltaStreamer (#3829) 2021-10-20 23:57:12 -04:00
Manoj Govindassamy
b68c5a68f9 [HUDI-2472] Fix few Cleaner tests with metadata table enabled (#3825) 2021-10-20 18:57:00 -04:00
Danny Chan
e355ab52db [HUDI-2578] Support merging small files for flink insert operation (#3822) 2021-10-20 21:10:07 +08:00
董可伦
46f0496a08 [MINOR] Fix typo,'intance' corrected to 'instance' (#3788) 2021-10-19 23:16:48 +08:00
Danny Chan
2eda3de7f9 [HUDI-2562] Embedded timeline server on JobManager (#3812) 2021-10-18 10:45:39 +08:00
Danny Chan
f897e6d73e [HUDI-2551] Support DefaultHoodieRecordPayload for flink (#3792) 2021-10-14 13:46:53 +08:00
Danny Chan
abf3e3fe71 [HUDI-2548] Flink streaming reader misses the rolling over file handles (#3787) 2021-10-14 10:36:18 +08:00
Sivabalan Narayanan
cff384d23f [HUDI-2552] Fixing some test failures to unblock broken CI master (#3793) 2021-10-13 18:44:43 -04:00
zhangyue19921010
e6711b171a [HUDI-2435][BUG]Fix clustering handle errors (#3666) 
* done

* remove unused imports

* code reviewed

* code reviewed

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-10-12 15:24:48 -07:00
Sivabalan Narayanan
8a487eafa7 [HUDI-2494] Fixing glob pattern to skip all hoodie meta paths (#3768) 2021-10-12 14:06:40 -04:00
董可伦
48a3906ccc [MINOR] Fix typo,'paritition' corrected to 'partition' (#3764) 2021-10-11 14:07:34 -04:00