lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,391 Commits 1 Branch 0 Tags
e00a9042e9a4e97f6f8ccaed4ac51a9aa8c2da76
Commit Graph

528 Commits

Author SHA1 Message Date
董可伦
56cd8ffae0 [HUDI-2837] Add support for using database name in incremental query (#4083) 2022-01-22 22:11:27 -08:00
YueZhang
79bf6ab00b [HUDI-3281][Performance]Tuning performance of getAllPartitionPaths API in FileSystemBackedTableMetadata (#4643) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-20 18:47:02 -05:00
YueZhang
7647562dad [HUDI-2833][Design] Merge small archive files instead of expanding indefinitely. (#4078) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-18 22:42:35 -08:00
Alexey Kudinkin
4bea758738 [HUDI-3191] Rebasing Hive's FileInputFormat onto AbstractHoodieTableFileIndex (#4531) 2022-01-18 14:54:51 -08:00
Yann Byron
a09c231911 [HUDI-2903] get table schema from the last commit with data written (#4180) 2022-01-18 10:50:30 -05:00
Danny Chan
45f054ffde [HUDI-3263] Do not nullify members in HoodieTableFileSystemView#resetViewState to avoid NPE (#4625) 2022-01-18 17:46:40 +08:00
董可伦
20e7983866 [HUDI-3252] Avoid creating empty requestedReplaceCommit in the startCommit method (#4515) 2022-01-17 17:28:18 -05:00
Alexey Kudinkin
75caa7d3d8 [HUDI-3179] Extracted common AbstractHoodieTableFileIndex to be shared across engines (#4520) 2022-01-16 22:46:20 -08:00
xiaotianzhang01
ed92c217ed [MINOR] Delete unused parameter in TablePathUtils (#4595) 
Co-authored-by: zhangxiaotian13 <zhangxiaotian13@jd.com>
 2022-01-16 22:24:43 -08:00
0x574C
28b3b6ad8f [MINOR] Remove org.apache.directory.api.util.Strings import (#4601) 2022-01-16 16:58:18 +08:00
Y Ethan Guo
7d163ee3de [MINOR] Fix local flaky test in TestFSUtils (#4596) 
Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>
 2022-01-13 22:48:57 -08:00
Sagar Sumit
195dac90fa [MINOR] Disable flaky tests to unlock CI (#4592) 2022-01-13 19:43:27 -08:00
Y Ethan Guo
397795c7d0 [HUDI-3007] Fix issues in HoodieRepairTool (#4564) 2022-01-12 09:03:27 -08:00
xuzifu666
4b2fd37fb4 [MINOR] Remove unused static var in HoodieAvroWriteSupport (#4543) 2022-01-11 11:53:45 -08:00
Sivabalan Narayanan
67ad4992e1 Removing extraneous warn logs in ClusteringUtils (#4553) 2022-01-11 08:20:14 +05:30
Alexey Kudinkin
f1e3762a94 [HUDI-2950] Addressing performance traps in Bulk Insert/Layout Optimization (#4234) 
* Cleaned up Z-curve/Hilbert ordering seqs:
  - Streamlined flow
  - Removed unnecessary operations (double-mapping, boxing, etc)
Updated `CollectionUtils::combine` to avoid AL resizing

* Tidying up

* Reducing small objects churn due to Scala/Java conversions by re-using `RowFactory`, passing `Object[]`

* Fixing name resolution (disambiguation overloads)

* `lint`

* Replaced `OverwriteAvroPayloadRecord` w/ `RewriteRecordPayload` to avoid unnecessary Avro ser/de loop

* Added `PathCachingFileName` to avoid fetching substrings every time file-name is fetched;
Inject `PathCachingFileName` into `HoodieWrapperFileSystem.convertPathWithScheme`

* Drastically reducing size of the `ArrayDeque` allocated by `ObjectSizeCalculator`

* XXX

* Missing license

* Fixed refs (after rebase)

* Fixing compilation failure in Scala 2.11

* `PathCachingFileName` > `FileNameCachingPath`

* Tidying up
 2022-01-10 18:23:22 -08:00
Sivabalan Narayanan
7a8b94c82d [HUDI-3180] Include files from completed commits while bootstrapping metadata table (#4519) 2022-01-10 15:33:15 -05:00
Manoj Govindassamy
251d4eb3b6 [HUDI-3030] InProcessLockPovider as default when any async servcies enabled with no lock provider override (#4406) 
* [HUDI-3030] InProcessLockPovider as default when any async servcies enabled with no lock provider override

 - Making InProcessLockProvider as the default lock provider when
   any async services are enabled and when no lock provider is
   explicitly set.

 - This is the workaround for metadata table updates racing with
   async table serice operations

* [HUDI-3030] InProcessLockPovider as default when any async servcies enabled with no lock provider override

 - Renaming isAnyTableServicesInline/Async() to areAnyTableServicesInline/Async()

* [HUDI-3030] InProcessLockPovider as default when any async servcies enabled with no lock provider override

 - Additionally checking for write config properties when verifying
   the lock provider override. Updated the unit test for this case.
 2022-01-10 08:40:24 +05:30
YueZhang
cf362fb2d5 [MINOR] Fix some code style issues based on check-style plugin (#4532) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-09 01:14:56 -08:00
Sagar Sumit
827549949c [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator (#4203) 
* [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator

Timestampbased key generator was returning diff values for row writer and non row writer path. this patch fixes it and is guarded by a config flag (`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled`)
 2022-01-08 10:22:44 -05:00
Sagar Sumit
518488c633 [HUDI-3185] HoodieConfig#getBoolean should return false when default not set (#4536) 
Remove unnecessary config
 2022-01-07 16:20:11 -05:00
YueZhang
b2b23f5d3a [HUDI-3183] Wrong result of HoodieArchivedTimeline loadInstants with TimeRangeFilter (#4521) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-01-06 21:16:29 -05:00
Danny Chan
0e297c0c4c [HUDI-3171] Sync empty table to hive metastore (#4511) 2022-01-05 16:41:33 +08:00
Manoj Govindassamy
bf4e3d63e7 [HUDI-3141] Metadata merged log record reader - avoiding NullPointerException when records by keys (#4505) 
- HoodieMetadataMergedLogRecordReader#getRecordsByKeys() and its parent class methods
   are not thread safe. When multiple queries come in for gettting log records
   by keys, they all operate on the same log record reader instance provided by
   HoodieBackedTableMetadata#openReadersIfNeeded() and they trip over each other
   as they clear/put/get the same class memeber records.

 - The fix is to streamline the mutatation to class member records. Making
   HoodieMetadataMergedLogRecordReader#getRecordsByKeys() a synchronized method
to avoid concurrent log records readers getting into NPE.
 2022-01-04 16:41:33 -05:00
Sagar Sumit
aaf5727495 [HUDI-2774] Handle duplicate instants when fetching pending clustering plans (#4118) 2022-01-04 16:32:05 -05:00
Shawy Geng
a4e622ac61 [HUDI-1951] Add bucket hash index, compatible with the hive bucket (#3173) 
* [HUDI-2154] Add index key field to HoodieKey

* [HUDI-2157] Add the bucket index and its read/write implemention of Spark engine.
* revert HUDI-2154 add index key field to HoodieKey
* fix all comments and introduce a new tricky way to get index key at runtime
support double insert for bucket index
* revert spark read optimizer based on bucket index
* add the storage layout
* index tag, hash function and add ut
* fix ut
* address partial comments
* Code review feedback
* add layout config and docs
* fix ut
* rename hoodie.layout and rebase master

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2021-12-30 12:38:26 -08:00
董可伦
436becf3ea [HUDI-2675] Fix the exception 'Not an Avro data file' when archive and clean (#4016) 2021-12-29 22:53:17 -05:00
xuzifu666
a29b27c7ca [MINOR] HoodieInstantTimeGenerator improve method used (#4462) 2021-12-29 18:43:16 +08:00
ForwardXu
32505d5adb [HUDI-3106] Fix HiveSyncTool not sync schema (#4452) 2021-12-27 22:11:14 -08:00
Danny Chan
c81df99e50 [HUDI-3102] Do not store rollback plan in inflight instant (#4445) 2021-12-25 18:10:43 +08:00
Danny Chan
7b07aac286 [HUDI-3101] Excluding compaction instants from pending rollback info (#4443) 2021-12-25 14:10:45 +08:00
xuzifu666
4721073b43 [MINOR] Remove unused method in HoodieActiveTimeline (#4435) 2021-12-24 22:29:34 +08:00
xuzifu666
032b883bd1 [HUDI-3014] Add table option to set utc timezone (#4306) 2021-12-23 16:27:45 +08:00
Raymond Xu
32a44bbe06 [HUDI-2970] Add test for archiving replace commit (#4345) 2021-12-21 00:01:59 -05:00
xuzifu666
f166ddad12 [MINOR] Remove unused method in HoodieActiveTimeline (#4401) 2021-12-20 22:19:37 +08:00
Danny Chan
478f9f3695 [minor] fix NetworkUtils#getHostname (#4355) 2021-12-19 10:09:48 +08:00
Sivabalan Narayanan
77abb5ccb9 [HUDI-3054] Fixing default lock configs for FileSystemBasedLock and fixing a flaky test (#4374) 2021-12-18 16:15:48 -05:00
Danny Chan
d0087d4040 [HUDI-3037] Add back remote view storage config for flink (#4338) 2021-12-17 13:57:53 +08:00
Danny Chan
ea2eba1a55 [HUDI-3015] Implement #reset and #sync for metadata filesystem view (#4307) 2021-12-16 15:26:16 +08:00
Manoj Govindassamy
bc8bf043d5 [HUDI-2995] Enabling metadata table by default (#4295) 
- Enabling metadata table by default
 2021-12-14 13:11:19 -08:00
Danny Chan
c8d6bd8898 [HUDI-2997] Skip the corrupt meta file for pending rollback action (#4296) 2021-12-14 14:08:13 +08:00
WangMinChao
46de25d5fd [HUDI-2994] Add judgement to existed partitionPath in the catch code block for HU… (#4294) 
* [HUDI-2994] Add judgement to existed partition path in the catch code block for HUDI-2743

Co-authored-by: wangminchao <wangminchao@asinking.com>
 2021-12-13 20:41:03 +08:00
ForwardXu
dd96129191 [HUDI-2990] Sync to HMS when deleting partitions (#4291) 2021-12-13 20:40:06 +08:00
Manoj Govindassamy
b22c2c611b [HUDI-2938] Metadata table util to get latest file slices for reader/writers (#4218) 2021-12-11 20:42:36 -08:00
Danny Chan
8dd0444ef9 [HUDI-2984] Implement #close for AbstractTableFileSystemView (#4285) 2021-12-11 16:19:10 +08:00
Y Ethan Guo
b5f05fd153 [HUDI-2906] Add a repair util to clean up dangling data and log files (#4278) 2021-12-11 00:16:05 -08:00
Alexey Kudinkin
2d864f7524 [HUDI-2814] Make Z-index more generic Column-Stats Index (#4106) 2021-12-10 14:56:09 -08:00
zhangyue19921010
3ba2909690 [HUDI-2892][BUG] Pending Clustering may stain the ActiveTimeLine and lead to incomplete query results (#4172) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2021-12-10 09:57:01 -08:00
Sivabalan Narayanan
be368264f4 [HUDI-2952] Fixing metadata table for non-partitioned dataset (#4243) 2021-12-10 11:11:42 -05:00
xiarixiaoyao
456d74ce4e [HUDI-2901] Fixed the bug clustering jobs cannot running in parallel (#4178) 2021-12-09 22:39:35 -08:00