lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity
2,559 Commits 1 Branch 0 Tags
08fd80c913aec05f36042111ef82d78bc0f55ad8
Commit Graph

389 Commits

Author SHA1 Message Date
Sagar Sumit
575bc63468 [HUDI-3356][HUDI-3203] HoodieData for metadata index records; BloomFilter construction from index based on the type param (#4848) 
Rework of #4761 
This diff introduces following changes:

- Write stats are converted to metadata index records during the commit. Making them use the HoodieData type so that the record generation scales up with needs. 
- Metadata index init support for bloom filter and column stats partitions.
- When building the BloomFilter from the index records, using the type param stored in the payload instead of hardcoded type.
- Delta writes can change column ranges and the column stats index need to be properly updated with new ranges to be consistent with the table dataset. This fix add column stats index update support for the delta writes.

Co-authored-by: Manoj Govindassamy <manoj.govindassamy@gmail.com>
 2022-03-08 10:39:04 -05:00
Sivabalan Narayanan
3539578ccb [HUDI-3213] Making commit preserve metadata to true for compaction (#4811) 
* Making commit preserve metadata to true

* Fixing integ tests

* Fixing preserve commit metadata for metadata table

* fixed bootstrap tests

* temp diff

* Fixing merge handle

* renaming fallback record

* fixing build issue

* Fixing test failures
 2022-03-07 18:02:05 +05:30
wangxianghu
c9ffdc493e [HUDI-3525] Introduce JsonkafkaSourceProcessor to support data preprocess before it is transformed to DataSet (#4930) 2022-03-06 15:41:01 -05:00
wangxianghu
4b471772aa [HUDI-3520] Introduce DeleteSupportSchemaPostProcessor to support adding _hoodie_is_deleted column to schema (#4921) 2022-03-06 15:37:09 -05:00
Pratyaksh Sharma
907e60c252 [HUDI-3264]: made schema registry urls configurable with MTDS (#4779) 2022-03-02 15:30:41 -05:00
Alexey Kudinkin
85f47b53df [HUDI-3469] Refactor HoodieTestDataGenerator to provide for reproducible Builds (#4866) 2022-03-01 22:15:26 -08:00
Sivabalan Narayanan
f7088a957c [HUDI-3497] Adding Datatable validator tool (#4902) 2022-02-28 22:46:32 -05:00
Y Ethan Guo
257052a94d [HUDI-3465] Add validation of column stats and bloom filters in HoodieMetadataTableValidator (#4878) 2022-02-28 18:49:30 -08:00
Sivabalan Narayanan
2f99e8458a [HUDI-3521] Fixing kakfa key and value serializer value type from class to string (#4919) 2022-02-27 11:13:13 -05:00
Sivabalan Narayanan
9678c3fbcf [MINOR] Fixing checkpoint management in S3IncrSource (#4871) 2022-02-22 09:15:16 -05:00
wangxianghu
4d1f74ebea [HUDI-3464] Fix wrong exception thrown from HiveSchemaProvider (#4865) 2022-02-22 10:20:20 +04:00
Sivabalan Narayanan
14dbbdf4c7 [HUDI-2189] Adding delete partitions support to DeltaStreamer (#4787) 2022-02-22 00:01:30 -05:00
Sivabalan Narayanan
d36fe24c9e [HUDI-3455] Fixing checkpoint management in hoodie incr source (#4850) 2022-02-21 08:19:57 -05:00
Sivabalan Narayanan
66ac1446dd [MINOR] Moving spark scheduling configs out of DataSourceOptions (#4843) 2022-02-20 13:49:18 -05:00
Y Ethan Guo
fba5822ee3 [HUDI-3430] Fix Deltastreamer to properly shut down the services upon failure (#4824) 2022-02-18 08:44:56 -05:00
Y Ethan Guo
9a05940a74 [HUDI-3366] Remove hardcoded logic of disabling metadata table in tests (#4792) 2022-02-15 16:41:47 -05:00
YueZhang
76e2faa28d [HUDI-3370] The files recorded in the commit may not match the actual ones for MOR Compaction (#4753) 
* use HoodieCommitMetadata to replace writeStatuses computation

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
 2022-02-14 11:12:52 +08:00
冯健
55777fec05 [HUDI-2413] fix Sql source's checkpoint issue (#3648) 
* [HUDI-2413] fix Sql source's checkpoint

* Fixing sql source checkpoint handling

* Fixing docs

Co-authored-by: jian.feng <fengjian428@gmial.com>
Co-authored-by: sivabalan <n.siva.b@gmail.com>
 2022-02-14 08:07:48 +05:30
Y Ethan Guo
6aba00e84f [MINOR] Fix typos in Spark client related classes (#4781) 2022-02-13 06:41:58 -08:00
wangxianghu
ce9762d588 [MINOR] unused import (#4799) 2022-02-12 13:11:37 +04:00
zhangxiang17
9518f78610 [HUDI-3413]fix jackson parse error when empty message from JsonKafkaSource Using HoodieDeltaStreamer (#4794) 2022-02-12 11:37:29 +04:00
Sivabalan Narayanan
60831d6906 [HUDI-3361] Fixing missing begin checkpoint in HoodieIncremental pull (#4755) 2022-02-08 12:03:07 -05:00
Sivabalan Narayanan
ab73047958 Adding support for custom scheduler configs with streaming sink (#4762) 2022-02-08 14:44:10 +05:30
YueZhang
1636876e8a [HUDI-3320] Hoodie metadata table validator (#4721) 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com>
 2022-02-08 00:29:44 -08:00
Vinish Reddy
8ab6f17149 [HUDI-3373] Add zero value metrics for empty data source and PROMETHEUS_PUSHGATEWAY reporter (#4760) 2022-02-07 15:17:46 -05:00
ForwardXu
773b317983 [HUDI-2941] Show _hoodie_operation in spark sql results (#4649) 2022-02-07 06:28:13 -08:00
Sivabalan Narayanan
24f738fe68 [HUDI-3360] Adding retries to deltastreamer for source errors (#4744) 2022-02-07 08:10:06 -05:00
ehui
538db185ca [HUDI-2491] Expose HMS mode metastore uri config option for spark writer (#3962) 2022-02-07 18:13:51 +05:30
YueZhang
de206acbae [HUDI-3369] New ScheduleAndExecute mode for HoodieCompactor and hudi-cli (#4750) 
Schedule and execute compaction plan in one single mode.
 2022-02-07 15:01:34 +05:30
Y Ethan Guo
b8601a9f58 [HUDI-2656] Generalize HoodieIndex for flexible record data type (#3893) 
Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>
 2022-02-03 20:24:04 -08:00
Raymond Xu
caef3d5c58 [HUDI-3330] Remove fixture test tables for multi writer tests (#4704) 2022-02-02 17:50:10 +05:30
jsbali
7ce0f4522b [HUDI-2711] Fallback to fulltable scan for IncrementalRelation if underlying files have been cleared or moved by cleaner (#3946) 
Co-authored-by: sivabalan <n.siva.b@gmail.com>
 2022-01-31 23:03:18 -05:00
Harsha Teja Kanna
ed7aa138e8 [MINOR] Added log to debug checkpoint resumption when set to 0 (#4650) 2022-01-28 23:08:25 -05:00
Sagar Sumit
e78b2f1b55 [HUDI-2943] Complete pending clustering before deltastreamer sync (#4572) 2022-01-28 21:28:04 -05:00
Manoj Govindassamy
f87c47352a [HUDI-2763] Metadata table records - support for key deduplication based on hardcoded key field (#4449) 
* [HUDI-2763] Metadata table records - support for key deduplication and virtual keys
- The backing log format for the metadata table is HFile, a KeyValue type.
Since the key field in the metadata record payload is a duplicate of the
Key in the Cell, the redundant key field in the record can be emptied
to save on the cost.

- HoodieHFileWriter and HoodieHFileDataBlock will now serialize records
with the key field emptied by default. HFile writer tries to find if
the record has metadata payload schema field 'key' and if so it does
the key trimming from the record payload.

- HoodieHFileReader when reading the serialized records back from disk,
it materializes the missing keyFields if any. HFile reader tries to
find if the record has metadata payload schema fiels 'key' and if so
it does the key materialization in the record payload.

- Tests have been added to verify the default virtual keys and key
   deduplication support for the metadata table records.

Co-authored-by: Vinoth Chandar <vinoth@apache.org>
 2022-01-26 13:34:04 -05:00
Vinish Reddy
87db4ded42 [MINOR] Add default value as null for S3 Incremental source properties (#4674) 2022-01-24 14:24:43 -05:00
wangxianghu
b7a79aa943 [HUDI-3283] Bootstrap support overwrite existing table (#4647) 2022-01-20 14:42:52 +04:00
wangxianghu
3d93e857cc [MINOR] Minor improvement in JsonkafkaSource (#4620) 2022-01-18 11:13:05 +04:00
leesf
5ce45c440b [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation (#4514) 
* Introduce hudi-spark3-common and hudi-spark2-common modules to place classes that would be reused in different spark versions, also introduce hudi-spark3.1.x to support spark 3.1.x.
* Introduce hudi format under hudi-spark2, hudi-spark3, hudi-spark3.1.x modules and change the hudi format in original hudi-spark module to hudi_v1 format.
* Manually tested on Spark 3.1.2 and Spark 3.2.0 SQL.
* Added a README.md file under hudi-spark-datasource module.
 2022-01-14 13:42:35 +08:00
Y Ethan Guo
397795c7d0 [HUDI-3007] Fix issues in HoodieRepairTool (#4564) 2022-01-12 09:03:27 -08:00
Sagar Sumit
12e95771ee [HUDI-3235] Fix ClassNotFoundException due to log4j-core dependency (#4574) 
- Move log4j-core to top level pom
 2022-01-12 11:53:43 -05:00
董可伦
017ddbbfac [MINOR] Fix typos (#4567) 2022-01-11 23:17:10 -08:00
Pratyaksh Sharma
a392e9ba46 [HUDI-485] Corrected the check for incremental sql (#2768) 
* [HUDI-485]: corrected the check for incremental sql

* [HUDI-485]: added tests

* code review comments addressed

* [HUDI-485]: added happy flow test case
 2022-01-12 08:22:07 +05:30
Sivabalan Narayanan
604d9885f1 [HUDI-3009] making some fixes to S3 incremental source (#4517) 2022-01-09 12:46:52 -05:00
Sagar Sumit
827549949c [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator (#4203) 
* [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator

Timestampbased key generator was returning diff values for row writer and non row writer path. this patch fixes it and is guarded by a config flag (`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled`)
 2022-01-08 10:22:44 -05:00
Sivabalan Narayanan
2e561defe9 [HUDI-2947] Fixing checkpoint fetch in detlastreamer (#4485) 
* Fixing checkpoint fetch in detlastreamer

* Addressing comments
 2022-01-07 22:08:58 +05:30
Sivabalan Narayanan
8718c30324 [HUDI-3165] Enabling InProcessLockProvider for all multi-writer tests instead of FileSystemBasedLockProviderTestClass (#4427) 2022-01-06 13:04:10 -05:00
hehexiaoduantui
50fa5a6aa7 Update HiveIncrementalPuller to configure filesystem (#4431) 
* Update HiveIncrementalPuller.java

fix get FileSystem bug

* Update HiveIncrementalPuller.java

fix error

* Update HiveIncrementalPuller.java

fie error
 2022-01-06 13:19:30 +05:30
Vinish Reddy
eee715b3ff [HUDI-3168] Fixing null schema with empty commit in incremental relation (#4513) 2022-01-05 11:43:10 -05:00
harshal
6409fc733d [HUDI-2374] Fixing AvroDFSSource does not use the overridden schema to deserialize Avro binaries (#4353) 2021-12-27 23:01:21 -05:00