lanyuanxiaoyao/hudi-service
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(executor-task): 完成动态调整的pulsar扫描

Browse Source 
对于pulsar在时间段上资源分配不均，通过pulsar reader中重新分配split解决，保证整个读取的流程中都能尽可能利用多线程的优势
This commit is contained in:
v-zhangjc9
2024-06-05 17:37:04 +08:00
parent 4c528bca61
commit 0b4cec3955
10 changed files with 244 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
test/test.http
View File

@@ -130,7 +130,7 @@ hoodie.datasource.write.hive_style_partitioning=false
hoodie.table.checksum=989688289
### Test police
GET http://{{username}}:{{password}}@b12s10.hdp.dc:33535/task/law_enforcement?
GET http://{{username}}:{{password}}@b12s10.hdp.dc:31719/task/law_enforcement?
pulsar_url=pulsar://132.122.115.158:16650,132.122.115.159:16650,132.122.115.160:16650,132.122.115.161:16650,132.122.115.167:16650,132.122.115.168:16650&
pulsar_topic=persistent://odcp/acct_sz/acct_item_755&start_time=1716858000000&end_time=1716861600000&
primary_keys=ACCT_ITEM_ID&partition_keys=ACCT_ID