90 Commits

Author SHA1 Message Date
df6f1eb548 feat(web): 增加编辑侧边栏关闭时保存节点数据 2025-06-23 20:15:32 +08:00
73e6ff3c54 feat(web): 优化编辑界面 2025-06-21 10:31:11 +08:00
v-zhangjc9
ec37d04ae5 feat(web): 替换amis渲染,amis渲染太慢,导致卡顿 2025-06-21 00:08:38 +08:00
v-zhangjc9
172ef4c099 feat(web): 增加流程定义基本能力 2025-06-20 18:15:49 +08:00
v-zhangjc9
306c20aa7f fix(web): 修复删除按钮hover不是红色 2025-06-19 11:31:07 +08:00
v-zhangjc9
24d5d10ecb fix(web): 优化字号 2025-06-19 11:14:59 +08:00
v-zhangjc9
4a9a9ec238 fix(ai-web): 优化qa嵌入提示词 2025-06-19 11:10:14 +08:00
v-zhangjc9
08aa1d8382 fix(ai-web): 修复chartjs工具获取bean错误 2025-06-19 11:09:52 +08:00
v-zhangjc9
1b3045dfd4 feat(web): 统一字体展示 2025-06-19 11:09:05 +08:00
v-zhangjc9
0f5ae1c4d4 fix(ai-web): 修复超时时间设置过短导致反复重连 2025-06-18 16:09:37 +08:00
v-zhangjc9
48e42ee99a fix(ai-web): 升级部份依赖版本 2025-06-18 14:53:45 +08:00
v-zhangjc9
0914b458d3 fix(ai-web): 修复页面失去焦点的时候没有断开对话的连接 2025-06-18 10:34:56 +08:00
v-zhangjc9
368c30676e feat(ai-web): 尝试优化对话连接的稳定性 2025-06-18 10:33:44 +08:00
v-zhangjc9
60477f99f5 fix(ai-web): 修复错别字 2025-06-17 17:43:18 +08:00
v-zhangjc9
565c530dd5 feat(ai-web): 知识库增加描述 2025-06-17 17:19:04 +08:00
v-zhangjc9
5130885033 fix(ai-web): 改正包名 2025-06-17 16:18:38 +08:00
v-zhangjc9
8e6463845b feat(ai-web): 开启gzip 2025-06-17 16:16:01 +08:00
v-zhangjc9
e89bffe289 feat(ai-web): 增加Feedback详情展示和处理情况确认 2025-06-17 16:15:42 +08:00
v-zhangjc9
1dd00d329c refactor(ai-web): 优化流程编排 2025-06-17 10:35:39 +08:00
e470a87372 feat(web): 修复yarn页面查看资源队列错误 2025-06-16 23:51:36 +08:00
v-zhangjc9
45da452f18 fix(ai-web): 完成feedback AI流程 2025-06-16 20:37:14 +08:00
v-zhangjc9
e6a1bc5383 fix(ai-web): 修复取值错误 2025-06-16 17:13:16 +08:00
v-zhangjc9
c5916703cd fix(web): 修复base url错误 2025-06-16 16:12:39 +08:00
v-zhangjc9
807ddbe5cb feat(ai-web): 完成图片上传和显示 2025-06-16 13:38:42 +08:00
v-zhangjc9
13de694e37 fix(all): 修复配置错误 2025-06-16 12:13:42 +08:00
v-zhangjc9
1962dd586c feat(ai-web): 增加辅助插件 2025-06-16 11:07:56 +08:00
138ee140e1 feat(ai-web): 增加feedback 2025-06-15 23:42:26 +08:00
e2d69bc6e8 refactor(ai-web): 优化id的生成 2025-06-15 22:19:05 +08:00
b9d707dc8f refactor(gateway): 适配ai web的路由 2025-06-15 20:10:54 +08:00
44d1473c6b refactor(ai): 移除chat,合并chat和knowledge为web
以后有需要再拆分
2025-06-15 20:08:51 +08:00
9c658afbd7 refactor(ai): 迁移chat到知识库中 2025-06-15 19:59:35 +08:00
e3f86e6497 refactor(knowledge): 优化配置文件 2025-06-15 19:15:40 +08:00
256c8c6bd5 refactor(knowledge): 修改代码结构,增加多环境配置文件方便本地开发 2025-06-15 19:10:38 +08:00
b627c91acb refactor(knowledge): 重构大模型配置
Spring AI默认大模型配置不支持同时配置两个文本大模型,比如一个文本大模型和一个图像大模型,改用自定义的配置
2025-06-15 17:57:09 +08:00
7fb490778a refactor(knowledge): 优化接口结构,统一到一个路径下,为合并做准备 2025-06-15 17:02:53 +08:00
d4d5aede31 feat(ai): restClient和webClient提供给其他类使用 2025-06-14 17:45:56 +08:00
v-zhangjc9
f11f5e7656 feat(ai): 调整模型 2025-06-13 19:07:13 +08:00
v-zhangjc9
bc32a89fea fix(ai): 移除不支持的模型 2025-06-13 16:11:37 +08:00
v-zhangjc9
2e24bdb90b feat(ai): 增加大模型对话测试 2025-06-12 20:29:37 +08:00
v-zhangjc9
5160c59ab0 feat(ai): 增加llama-swap配置文件生成工具 2025-06-12 20:24:46 +08:00
v-zhangjc9
506e28c9f7 feat(chat): 优化提示词,增加mermaid图表生成 2025-06-11 19:45:57 +08:00
9076bd4c09 fix(web): 修复vite类型没有引入导致import出现问题 2025-06-08 19:17:32 +08:00
69f0bed9a1 fix(web): 提交遗漏的文件 2025-06-08 02:42:58 +08:00
c04269c3fa feat(web): AI对话框增加图表显示
支持Mermaid、EChart、Chart.js
2025-06-08 02:42:05 +08:00
4fe21f3d8b refactor(chat): 提示词汇总,方便使用 2025-06-08 00:13:17 +08:00
cb42376e46 feat(bin): 优化脚本 2025-06-06 22:34:11 +08:00
34bdb59501 feat(bin): 优化脚本 2025-06-06 22:32:14 +08:00
v-zhangjc9
72c23d916a feat(chat): 优化提示词,增加外部调用方法 2025-06-06 19:56:52 +08:00
v-zhangjc9
e01a883d37 feat(chat): 优化提示词,增加外部调用方法 2025-06-06 17:20:15 +08:00
v-zhangjc9
951075fc9f feat(chat): 优化提示词和知识库查询 2025-06-06 10:57:27 +08:00
v-zhangjc9
90fea22de5 feat(chat): 增加数据库SQL访问接口 2025-06-05 19:53:25 +08:00
v-zhangjc9
a35980a5f4 perf(knowledge): 暂时关闭rerank提升性能 2025-06-05 15:21:03 +08:00
v-zhangjc9
e359bed97c feat(knowledge): 实现外部直接插入知识库 2025-06-05 15:20:33 +08:00
v-zhangjc9
8b4827b164 feat(web): 增加代码混淆 2025-06-04 19:19:37 +08:00
v-zhangjc9
577834568b feat(ai): 优化UI 2025-06-04 18:52:28 +08:00
v-zhangjc9
c4d5a7b300 feat(knowledge): 增加rerank模型适配 2025-06-04 17:43:22 +08:00
v-zhangjc9
4124a8a851 feat(chat): 优化提示词,时间参数直接放在提示词中 2025-06-04 17:42:46 +08:00
v-zhangjc9
6d4dedc3f4 fix(web): 增加访问超时时间 2025-06-04 17:40:54 +08:00
v-zhangjc9
b8aea3bdf0 feat(web): 优化界面,移除思考开关 2025-06-04 17:40:21 +08:00
v-zhangjc9
d36ad95a85 fix(web): 修复知识库id没有及时添加到http请求中 2025-06-04 17:39:55 +08:00
v-zhangjc9
fdec62b56e fix(bin): 优化打包脚本 2025-06-04 10:02:25 +08:00
1217d114bd feat(bin): 尝试合并编译脚本 2025-06-03 23:53:19 +08:00
2d7b30bb7a fix(bin): 适配在Windows下的打包 2025-06-03 23:22:01 +08:00
v-zhangjc9
c2af2d6365 feat(chat): 尝试在对话中加入知识库 2025-06-03 20:23:52 +08:00
v-zhangjc9
536c4e9cab feat(bin): 增加forest发布脚本 2025-06-03 20:22:29 +08:00
v-zhangjc9
c9a1ea2be5 feat(web): 用markdown显示思考过程 2025-06-03 16:12:23 +08:00
v-zhangjc9
602a337923 fix(chat): 修复上传文件选择错误 2025-05-30 17:35:59 +08:00
v-zhangjc9
fe9e185a9a feat(web): 增加根据环境切换debug状态 2025-05-30 17:35:39 +08:00
v-zhangjc9
3901a47da0 feat(web): 增加知识库选择 2025-05-30 17:35:24 +08:00
v-zhangjc9
dc5998cf72 feat(web): 增加命中测试按钮 2025-05-30 17:35:15 +08:00
v-zhangjc9
993940e810 feat(web): 升级前端依赖 2025-05-30 17:34:41 +08:00
v-zhangjc9
b8cc8fee67 fix(knowledge): 修复文件上传失败 2025-05-30 11:03:26 +08:00
v-zhangjc9
2cac589b0f fix(web): 优化代码 2025-05-30 10:31:32 +08:00
v-zhangjc9
29859664e3 fix(ai): 优化cors配置 2025-05-30 10:30:27 +08:00
v-zhangjc9
0ceb5d7fc3 feat(bin): 增加上传文件md5显示 2025-05-30 10:29:59 +08:00
v-zhangjc9
947c831609 fix(all): 优化忽略名单 2025-05-30 10:28:43 +08:00
v-zhangjc9
ce95ec7444 fix(gateway): 优化cors配置,网关统一处理 2025-05-30 10:28:16 +08:00
v-zhangjc9
95214f7af3 fix(gateway): 优化security配置 2025-05-29 18:36:50 +08:00
v-zhangjc9
0f49c91fde feat(bin): 优化上传命令 2025-05-29 11:15:04 +08:00
v-zhangjc9
0262c573ae fix(bin): 修复编译脚本获取jar包的匹配式 2025-05-29 10:36:14 +08:00
v-zhangjc9
8c9cb6f21d feat(knowledge): 完成知识库部署相关脚本调整 2025-05-28 18:46:20 +08:00
v-zhangjc9
e6e24dff27 fix(knowledge): 修复没有指定数据库名称导致无法查询 2025-05-28 17:11:13 +08:00
v-zhangjc9
fc2ea107d2 feat(knowledge): 补充建表语句和脚本 2025-05-28 16:06:07 +08:00
v-zhangjc9
6f9c898d51 fix(knowledge): 移除多余的日志打印 2025-05-28 15:41:36 +08:00
v-zhangjc9
7fd484eeab fix(knowledge): 优化多文件上传体验 2025-05-28 15:40:57 +08:00
v-zhangjc9
3ee6303cf5 feat(knowledge): 完成知识库基本功能开发 2025-05-28 15:06:30 +08:00
v-zhangjc9
f7ed3bd270 feat(core): 改用amis推荐的返回结构 2025-05-27 09:35:11 +08:00
v-zhangjc9
e57c81ce75 feat(knowledge): 初步完成知识库分片预览 2025-05-23 19:12:41 +08:00
v-zhangjc9
fce4816880 feat(all): 移除远程日志集中 2025-05-23 09:49:37 +08:00
v-zhangjc9
79f792b6cf feat(web): 增加预览 2025-05-23 09:48:16 +08:00
148 changed files with 8028 additions and 3274 deletions

1
.gitignore vendored
View File

@@ -109,3 +109,4 @@ Icon
Network Trash Folder
Temporary Items
.apdisk
**/temp/

View File

@@ -1,6 +1,6 @@
import {cd, path} from 'zx'
import {trim} from "licia";
import {run_deploy_batch, run_deploy_root, run_package, run_upload_normal} from "./library.js";
import {run_deploy_batch, run_deploy_root, run_package, run_upload} from "./library.js";
// 切换目录
cd(trim(path.dirname(import.meta.dirname)))
@@ -9,7 +9,7 @@ try {
await run_deploy_root()
await run_deploy_batch(['service-cli', 'service-cli/service-cli-core'])
await run_package('service-cli/service-cli-runner')
await run_upload_normal('service-cli-runner', 'service-cli')
await run_upload('**/service-cli-runner-1.0.0-SNAPSHOT.jar')
} catch (e) {
console.error(e)
}

View File

@@ -1,6 +1,6 @@
import {cd, path} from 'zx'
import {trim} from "licia";
import {run_deploy_batch, run_deploy_root, run_package, run_upload_normal} from "./library.js";
import {run_deploy_batch, run_deploy_root, run_package, run_upload} from "./library.js";
// 切换目录
cd(trim(path.dirname(import.meta.dirname)))
@@ -9,7 +9,7 @@ try {
await run_deploy_root()
await run_deploy_batch(['service-common', 'service-dependencies', 'service-configuration', 'service-forest', 'service-executor', 'service-executor/service-executor-core'])
await run_package('service-executor/service-executor-manager')
await run_upload_normal('service-executor-manager', 'service-executor')
await run_upload('**/service-executor-manager-1.0.0-SNAPSHOT.jar')
} catch (e) {
console.error(e)
}

View File

@@ -1,6 +1,6 @@
import {cd, path} from 'zx'
import {trim} from "licia";
import {run_deploy_batch, run_deploy_root, run_package, run_upload_normal} from "./library.js";
import {run_deploy_batch, run_deploy_root, run_package, run_upload} from "./library.js";
// 切换目录
cd(trim(path.dirname(import.meta.dirname)))
@@ -9,7 +9,7 @@ try {
await run_deploy_root()
await run_deploy_batch(['service-common', 'service-dependencies', 'service-configuration', 'service-forest', 'service-executor', 'service-executor/service-executor-core'])
await run_package('service-executor/service-executor-task')
await run_upload_normal('service-executor-task', 'service-executor')
await run_upload('**/service-executor-task-1.0.0-SNAPSHOT.jar')
} catch (e) {
console.error(e)
}

13
bin/build-forest.js Normal file
View File

@@ -0,0 +1,13 @@
import {cd, path} from 'zx'
import {trim} from "licia";
import {run_deploy_batch, run_deploy_root} from "./library.js";
// 切换目录
cd(trim(path.dirname(import.meta.dirname)))
// 执行流程
try {
await run_deploy_root()
await run_deploy_batch(['service-common', 'service-dependencies', 'service-configuration', 'service-forest'])
} catch (e) {
console.error(e)
}

View File

@@ -1,6 +1,6 @@
import {cd, path} from 'zx'
import {trim} from "licia";
import {run_deploy_batch, run_deploy_root, run_package, run_upload_normal} from "./library.js";
import {run_deploy_batch, run_deploy_root, run_package, run_upload} from "./library.js";
// 切换目录
cd(trim(path.dirname(import.meta.dirname)))
@@ -9,7 +9,7 @@ try {
await run_deploy_root()
await run_deploy_batch(['service-common'])
await run_package('utils/patch')
await run_upload_normal('**/patch-1.0.0-SNAPSHOT.jar')
await run_upload('**/patch-1.0.0-SNAPSHOT.jar')
} catch (e) {
console.error(e)
}

View File

@@ -1,6 +1,6 @@
import {cd, path} from 'zx'
import {trim} from "licia";
import {run_deploy_batch, run_deploy_root, run_package, run_upload, run_upload_normal} from "./library.js";
import {run_deploy_batch, run_deploy_root, run_package, run_upload_normal} from "./library.js";
// 切换目录
cd(trim(path.dirname(import.meta.dirname)))

View File

@@ -1,6 +1,6 @@
import {cd, path} from 'zx'
import {trim} from "licia";
import {run_deploy_batch, run_deploy_root, run_package, run_upload, run_upload_normal} from "./library.js";
import {run_deploy_batch, run_deploy_root, run_package, run_upload_normal} from "./library.js";
// 切换目录
cd(trim(path.dirname(import.meta.dirname)))

View File

@@ -1,6 +1,6 @@
import {cd, path} from 'zx'
import {trim} from "licia";
import {run_deploy_batch, run_deploy_root, run_package, run_upload, run_upload_normal} from "./library.js";
import {run_deploy_batch, run_deploy_root, run_package, run_upload_normal} from "./library.js";
// 切换目录
cd(trim(path.dirname(import.meta.dirname)))

View File

@@ -1,6 +1,6 @@
import {cd, path} from 'zx'
import {trim} from "licia";
import {run_deploy_batch, run_deploy_root, run_package, run_upload, run_upload_normal} from "./library.js";
import {run_deploy_batch, run_deploy_root, run_package, run_upload_normal} from "./library.js";
// 切换目录
cd(trim(path.dirname(import.meta.dirname)))

View File

@@ -4,7 +4,7 @@ jars_path=/data/datalake/jars
jdk_path=/opt/jdk1.8.0_162/bin/java
arguments=$@
# 手动上传jar包则注释掉这行显神通吧反正是
# 手动上传jar包则注释掉这行显神通吧反正是
curl http://AxhEbscwsJDbYMH2:cYxg3b4PtWoVD5SjFayWxtnSVsjzRsg4@132.126.207.124:36800/file/download/service-cli-runner-1.0.0-SNAPSHOT.jar -o ${jars_path}/service-cli-runner.jar
${jdk_path} -jar ${jars_path}/service-cli-runner.jar \
--spring.profiles.active=b12 \

View File

@@ -1,17 +1,24 @@
import {$, fetch, fs, glob, os, path, spinner, syncProcessCwd, usePowerShell} from 'zx'
import {isEqual, trim} from "licia";
import {isEqual, trim, fileSize} from "licia";
import md5file from 'md5-file'
syncProcessCwd(true)
if (isEqual(os.platform(), 'win32')) {
usePowerShell()
}
const maven_setting = '/Users/lanyuanxiaoyao/.m2/settings-nas.xml'
const maven_setting = path.join(os.homedir(), '.m2', 'settings-nas.xml')
const upload_url = 'http://132.126.207.124:36800'
const upload_username = 'AxhEbscwsJDbYMH2'
const upload_password = 'cYxg3b4PtWoVD5SjFayWxtnSVsjzRsg4'
/**
* 时间戳转自然语言
*
* @param timestamp 时间戳
* @returns {string} 自然语言描述的时间
*/
const millisecondToString = (timestamp) => {
const totalSeconds = Math.floor(parseFloat(timestamp) / 1000)
if (isNaN(totalSeconds) || totalSeconds < 0) {
@@ -39,7 +46,7 @@ export const run_deploy = async (project) => {
`Deploying project ${project}`,
() => $`mvn -pl ${project} clean deploy -D skipTests -s ${maven_setting}`
)
console.log(`✅ Finish deploy ${project} (${millisecondToString(output['duration'])})`)
console.log(`✅ Finished deploy ${project} (${millisecondToString(output['duration'])})`)
}
export const run_deploy_root = async () => {
@@ -47,7 +54,7 @@ export const run_deploy_root = async () => {
`Deploying root`,
() => $`mvn clean deploy -N -D skipTests -s ${maven_setting}`
)
console.log(`✅ Finish deploy root (${millisecondToString(output['duration'])})`)
console.log(`✅ Finished deploy root (${millisecondToString(output['duration'])})`)
}
export const run_deploy_batch = async (projects) => {
@@ -61,7 +68,7 @@ export const run_package = async (project, profile = 'b2b12') => {
`Packaging project ${project}${isEqual(profile, 'b2b12') ? '' : ` ${profile}`}`,
() => $`mvn -pl ${project} clean package -D skipTests -P ${profile} -s ${maven_setting}`
)
console.log(`✅ Finish package ${project}${isEqual(profile, 'b2b12') ? '' : ` ${profile}`} (${millisecondToString(output['duration'])})`)
console.log(`✅ Finished package ${project}${isEqual(profile, 'b2b12') ? '' : ` ${profile}`} (${millisecondToString(output['duration'])})`)
}
export const run_package_batch = async (projects) => {
@@ -70,10 +77,12 @@ export const run_package_batch = async (projects) => {
}
}
const upload = async (file_path) => {
export const upload = async (file_path) => {
let start = new Date().getTime()
let basename = path.basename(file_path)
let response = await spinner(
`Uploading project ${file_path}`,
() => fetch(`${upload_url}/file/upload/${path.basename(file_path)}`, {
() => fetch(`${upload_url}/file/upload/${basename}`, {
method: 'POST',
headers: {
'Content-Type': 'application/octet-stream',
@@ -87,7 +96,10 @@ const upload = async (file_path) => {
if (!isEqual(response.status, 200)) {
throw response
}
console.log(`✅ Finish upload ${file_path}`)
console.log(`✅ Finished upload ${file_path} (${millisecondToString((new Date().getTime()) - start)})`)
console.log(`📘 Uploaded ${fileSize(fs.statSync(file_path).size)}`)
console.log(`📘 MD5 ${md5file.sync(file_path)}`)
console.log(`📘 Download curl http://AxhEbscwsJDbYMH2:cYxg3b4PtWoVD5SjFayWxtnSVsjzRsg4@132.126.207.124:36800/file/download/${basename} -o ${basename}`)
fs.rmSync(file_path)
}

View File

@@ -1,42 +0,0 @@
#!/bin/bash
maven_setting=/Users/lanyuanxiaoyao/.m2/settings-nas.xml
build_profile=b2b12
upload_username=AxhEbscwsJDbYMH2
upload_password=cYxg3b4PtWoVD5SjFayWxtnSVsjzRsg4
upload_url=http://$upload_username:$upload_password@132.126.207.124:36800
root_path=$(dirname $(cd $(dirname $0);pwd))
function upload() {
source_file_path=$(realpath $1)
file_name=$(basename $source_file_path)
echo "↪ Source md5: $(md5sum $source_file_path | awk '{print $1}')"
echo "↪ Uploading $source_file_path"
curl $upload_url/file/upload/$file_name -T $source_file_path
echo "↪ Upload ytp success"
echo "↪ Download: curl $upload_url/file/download/$file_name -o $file_name"
echo "↪ Delete source"
rm $source_file_path
}
function joining {
local d=${1-} f=${2-}
if shift 2; then
printf %s "$f" "${@/#/$d}"
fi
}
function deploy_root() {
mvn deploy -N -D skipTests -s $maven_setting
}
function deploy() {
mvn -pl $(joining , $@) clean deploy -D skipTests -s $maven_setting
}
function package() {
mvn -pl $(joining , $@) clean package -D skipTests -P $build_profile -s $maven_setting
}

View File

@@ -9,6 +9,7 @@
"fs-extra": "^11.3.0",
"globby": "^14.1.0",
"licia": "^1.48.0",
"md5-file": "^5.0.0",
"minimist": "^1.2.8",
"node-fetch-native": "^1.6.6",
"which": "^5.0.0",

10
bin/pnpm-lock.yaml generated
View File

@@ -26,6 +26,9 @@ importers:
licia:
specifier: ^1.48.0
version: 1.48.0
md5-file:
specifier: ^5.0.0
version: 5.0.0
minimist:
specifier: ^1.2.8
version: 1.2.8
@@ -129,6 +132,11 @@ packages:
licia@1.48.0:
resolution: {integrity: sha512-bBWiT5CSdEtwuAHiYTJ74yItCjIFdHi4xiFk6BRDfKa+sdCpkUHp69YKb5udNOJlHDzFjNjcMgNZ/+wQIHrB8A==}
md5-file@5.0.0:
resolution: {integrity: sha512-xbEFXCYVWrSx/gEKS1VPlg84h/4L20znVIulKw6kMfmBUAZNAnF00eczz9ICMl+/hjQGo5KSXRxbL/47X3rmMw==}
engines: {node: '>=10.13.0'}
hasBin: true
merge2@1.4.1:
resolution: {integrity: sha512-8q7VEgMJW4J8tcfVPy8g09NcQwZdbwFEqhe/WZkoIzjn/3TGDwtOCYtXGxA3O8tPzpczCCDgv+P2P5y00ZJOOg==}
engines: {node: '>= 8'}
@@ -283,6 +291,8 @@ snapshots:
licia@1.48.0: {}
md5-file@5.0.0: {}
merge2@1.4.1: {}
micromatch@4.0.8:

10
pom.xml
View File

@@ -417,6 +417,16 @@
<artifactId>maven-shade-plugin</artifactId>
<version>3.3.0</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.4.2</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-invoker-plugin</artifactId>
<version>3.9.0</version>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>

View File

@@ -1,14 +1,12 @@
import {cd, path} from 'zx'
import {trim} from "licia";
import {run_deploy, run_package, run_upload} from '../../bin/library.js'
import {run_deploy} from '../../bin/library.js'
// 切换目录
cd(trim(path.dirname(import.meta.dirname)))
// 执行流程
try {
await run_deploy('service-ai-core')
await run_package('service-ai-chat')
await run_upload('service-ai-chat')
} catch (e) {
console.error(e)
}

View File

@@ -0,0 +1,21 @@
import {
cd,
path,
} from 'zx'
import {trim} from "licia";
import {
run_deploy,
run_package,
run_upload_normal,
} from '../../bin/library.js'
// 切换目录
cd(trim(path.dirname(import.meta.dirname)))
// 执行流程
try {
await run_deploy('service-ai-core')
await run_package('service-ai-web')
await run_upload_normal('service-ai-web')
} catch (e) {
console.error(e)
}

View File

@@ -9,6 +9,7 @@
"fs-extra": "^11.3.0",
"globby": "^14.1.0",
"licia": "^1.48.0",
"md5-file": "^5.0.0",
"minimist": "^1.2.8",
"node-fetch-native": "^1.6.6",
"which": "^5.0.0",

View File

@@ -26,6 +26,9 @@ importers:
licia:
specifier: ^1.48.0
version: 1.48.0
md5-file:
specifier: ^5.0.0
version: 5.0.0
minimist:
specifier: ^1.2.8
version: 1.2.8
@@ -129,6 +132,11 @@ packages:
licia@1.48.0:
resolution: {integrity: sha512-bBWiT5CSdEtwuAHiYTJ74yItCjIFdHi4xiFk6BRDfKa+sdCpkUHp69YKb5udNOJlHDzFjNjcMgNZ/+wQIHrB8A==}
md5-file@5.0.0:
resolution: {integrity: sha512-xbEFXCYVWrSx/gEKS1VPlg84h/4L20znVIulKw6kMfmBUAZNAnF00eczz9ICMl+/hjQGo5KSXRxbL/47X3rmMw==}
engines: {node: '>=10.13.0'}
hasBin: true
merge2@1.4.1:
resolution: {integrity: sha512-8q7VEgMJW4J8tcfVPy8g09NcQwZdbwFEqhe/WZkoIzjn/3TGDwtOCYtXGxA3O8tPzpczCCDgv+P2P5y00ZJOOg==}
engines: {node: '>= 8'}
@@ -283,6 +291,8 @@ snapshots:
licia@1.48.0: {}
md5-file@5.0.0: {}
merge2@1.4.1: {}
micromatch@4.0.8:

View File

@@ -0,0 +1,11 @@
CREATE TABLE `service_ai_feedback`
(
`id` bigint NOT NULL,
`source` longtext NOT NULL,
`conclusion` longtext,
`analysis` longtext,
`pictures` longtext,
`status` varchar(50) NOT NULL DEFAULT 'ANALYSIS_PROCESSING',
`created_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`modified_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) DEFAULT CHARSET = utf8mb4;

View File

@@ -0,0 +1,11 @@
CREATE TABLE `service_ai_file`
(
`id` bigint NOT NULL,
`filename` varchar(500) DEFAULT NULL,
`size` bigint DEFAULT NULL,
`md5` varchar(100) DEFAULT NULL,
`path` varchar(500) DEFAULT NULL,
`type` varchar(50) DEFAULT NULL,
`created_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) DEFAULT CHARSET = utf8mb4;

View File

@@ -0,0 +1,9 @@
CREATE TABLE `service_ai_group`
(
`id` bigint NOT NULL,
`knowledge_id` bigint NOT NULL,
`name` varchar(100) NOT NULL,
`status` varchar(10) NOT NULL DEFAULT 'RUNNING',
`created_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`modified_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) DEFAULT CHARSET = utf8mb4;

View File

@@ -0,0 +1,11 @@
CREATE TABLE `service_ai_knowledge`
(
`id` bigint NOT NULL,
`vector_source_id` varchar(100) NOT NULL,
`name` varchar(100) NOT NULL,
`description` longtext NOT NULL,
`strategy` varchar(10) NOT NULL,
`created_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`modified_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) DEFAULT CHARSET = utf8mb4;

View File

@@ -11,8 +11,8 @@
<description>Hudi AI服务集合</description>
<modules>
<module>service-ai-core</module>
<module>service-ai-chat</module>
<module>service-ai-knowledge</module>
<module>service-ai-web</module>
<module>service-ai-cli</module>
</modules>
<properties>
@@ -24,12 +24,26 @@
<spring-boot.version>3.4.3</spring-boot.version>
<spring-cloud.version>2024.0.1</spring-cloud.version>
<spring-ai.version>1.0.0-RC1</spring-ai.version>
<spring-ai.version>1.0.0</spring-ai.version>
<solon-ai.version>3.3.1</solon-ai.version>
<eclipse-collections.version>11.1.0</eclipse-collections.version>
<curator.version>5.1.0</curator.version>
<hutool.version>5.8.27</hutool.version>
<mapstruct.version>1.6.3</mapstruct.version>
</properties>
<dependencies>
<dependency>
<groupId>org.mapstruct</groupId>
<artifactId>mapstruct</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<!-- 当前项目依赖 -->
@@ -136,12 +150,32 @@
<artifactId>liteflow-spring-boot-starter</artifactId>
<version>2.13.2</version>
</dependency>
<dependency>
<groupId>org.noear</groupId>
<artifactId>solon-ai</artifactId>
<version>${solon-ai.version}</version>
</dependency>
<dependency>
<groupId>org.noear</groupId>
<artifactId>solon-ai-dialect-openai</artifactId>
<version>${solon-ai.version}</version>
</dependency>
<dependency>
<groupId>org.mapstruct</groupId>
<artifactId>mapstruct</artifactId>
<version>${mapstruct.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.14.0</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
@@ -150,7 +184,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.3.0</version>
<version>3.6.0</version>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>

View File

@@ -1,23 +0,0 @@
package com.lanyuanxiaoyao.service.ai.chat;
import com.ulisesbocchio.jasyptspringboot.annotation.EnableEncryptableProperties;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.retry.annotation.EnableRetry;
/**
* @author lanyuanxiaoyao
* @version 20250514
*/
@SpringBootApplication(scanBasePackages = "com.lanyuanxiaoyao.service")
@EnableDiscoveryClient
@EnableConfigurationProperties
@EnableEncryptableProperties
@EnableRetry
public class AiChatApplication {
public static void main(String[] args) {
SpringApplication.run(AiChatApplication.class, args);
}
}

View File

@@ -1,87 +0,0 @@
package com.lanyuanxiaoyao.service.ai.chat.controller;
import cn.hutool.core.util.StrUtil;
import com.lanyuanxiaoyao.service.ai.chat.entity.MessageVO;
import com.lanyuanxiaoyao.service.ai.chat.tools.DatetimeTools;
import java.io.IOException;
import org.eclipse.collections.api.list.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.messages.Message;
import org.springframework.ai.chat.messages.UserMessage;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.servlet.mvc.method.annotation.SseEmitter;
/**
* 聊天
*
* @author lanyuanxiaoyao
* @version 20250514
*/
@Controller
@RequestMapping("chat")
public class ChatController {
private static final Logger logger = LoggerFactory.getLogger(ChatController.class);
private final ChatClient chatClient;
public ChatController(ChatClient.Builder builder) {
this.chatClient = builder.build();
}
private ChatClient.ChatClientRequestSpec buildRequest(ImmutableList<MessageVO> messages) {
return chatClient.prompt()
.system("""
你是一名专业的AI运维助手负责“Hudi数据同步服务平台”的运维工作
你将会友好地帮助用户解答关于该平台运维工作的问题,你会尽可能通过各种方式获取知识和数据来解答;
对于无法通过已有知识回答的问题,你会提示用户你无法解答该问题,而不是虚构不存在的数据或答案;
对于与该平台无关的问题,你会委婉地拒绝用户,并提示无法回答;
你将始终在中文语境下进行对话。
""")
.tools(new DatetimeTools())
.messages(
messages
.collect(message -> StrUtil.equals(message.getRole(), "assistant")
? new AssistantMessage(message.getContent())
: new UserMessage(message.getContent()))
.collect(message -> (Message) message)
.toList()
);
}
@PostMapping("sync")
@ResponseBody
public String chatSync(@RequestBody ImmutableList<MessageVO> messages) {
String content = buildRequest(messages)
.call()
.content();
return StrUtil.trimToEmpty(content);
}
@PostMapping("async")
public SseEmitter chatAsync(@RequestBody ImmutableList<MessageVO> messages) {
SseEmitter emitter = new SseEmitter();
buildRequest(messages)
.stream()
.content()
.subscribe(
content -> {
try {
emitter.send(content);
} catch (IOException e) {
emitter.completeWithError(e);
throw new RuntimeException(e);
}
},
emitter::completeWithError,
emitter::complete
);
return emitter;
}
}

View File

@@ -1,34 +0,0 @@
package com.lanyuanxiaoyao.service.ai.chat.entity;
/**
* @author lanyuanxiaoyao
* @version 20250516
*/
public class MessageVO {
private String role;
private String content;
public String getRole() {
return role;
}
public void setRole(String role) {
this.role = role;
}
public String getContent() {
return content;
}
public void setContent(String content) {
this.content = content;
}
@Override
public String toString() {
return "MessageVO{" +
"role='" + role + '\'' +
", content='" + content + '\'' +
'}';
}
}

View File

@@ -1,18 +0,0 @@
package com.lanyuanxiaoyao.service.ai.chat.tools;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import org.springframework.ai.tool.annotation.Tool;
/**
* @author lanyuanxiaoyao
* @version 20250516
*/
public class DatetimeTools {
private final static DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");
@Tool(description = "获取当前日期和时间")
public String getCurrentDateTime() {
return LocalDateTime.now().format(formatter);
}
}

View File

@@ -1,15 +0,0 @@
spring:
application:
name: service-ai-chat
profiles:
include: random-port,common,discovery,metrics,forest
ai:
openai:
base-url: http://132.121.206.65:10086
api-key: ENC(K+Hff9QGC+fcyi510VIDd9CaeK/IN5WBJ9rlkUsHEdDgIidW+stHHJlsK0lLPUXXREha+ToQZqqDXJrqSE+GUKCXklFhelD8bRHFXBIeP/ZzT2cxhzgKUXgjw3S0Qw2R)
chat:
options:
model: 'Qwen3-1.7-vllm'
mvc:
async:
request-timeout: 300000

View File

@@ -1,37 +0,0 @@
package com.lanyuanxiaoyao.service.ai.chat;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.openai.OpenAiChatModel;
import org.springframework.ai.openai.OpenAiChatOptions;
import org.springframework.ai.openai.api.OpenAiApi;
import reactor.core.Disposable;
/**
* @author lanyuanxiaoyao
* @version 20250514
*/
public class TestChat {
public static void main(String[] args) {
ChatClient client = ChatClient.builder(
OpenAiChatModel.builder()
.openAiApi(
OpenAiApi.builder()
.baseUrl("http://132.121.206.65:10086")
.apiKey("*XMySqV%>hR&v>>g*NwCs3tpQ5FVMFEF2VHVTj<MYQd$&@$sY7CgqNyea4giJi4")
.build()
)
.defaultOptions(
OpenAiChatOptions.builder()
.model("Qwen3-1.7")
.build()
)
.build()
)
.build();
String content = client.prompt()
.user("你好")
.call()
.content();
System.out.println(content);
}
}

View File

@@ -9,16 +9,20 @@
<version>1.0.0-SNAPSHOT</version>
</parent>
<artifactId>service-ai-chat</artifactId>
<artifactId>service-ai-cli</artifactId>
<dependencies>
<dependency>
<groupId>com.lanyuanxiaoyao</groupId>
<artifactId>service-ai-core</artifactId>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-model-openai</artifactId>
<groupId>cn.hutool</groupId>
<artifactId>hutool-all</artifactId>
</dependency>
<dependency>
<groupId>org.freemarker</groupId>
<artifactId>freemarker</artifactId>
</dependency>
</dependencies>
@@ -38,4 +42,5 @@
</plugin>
</plugins>
</build>
</project>

View File

@@ -0,0 +1,22 @@
package com.lanyuanxiaoyao.service.ai.cli;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
/**
* @author lanyuanxiaoyao
* @version 20250612
*/
@SpringBootApplication
public class Generator implements ApplicationRunner {
public static void main(String[] args) {
SpringApplication.run(Generator.class, args);
}
@Override
public void run(ApplicationArguments args) throws Exception {
new LlamaSwapTool().generate();
}
}

View File

@@ -0,0 +1,26 @@
package com.lanyuanxiaoyao.service.ai.cli;
import cn.hutool.core.io.FileUtil;
import cn.hutool.extra.template.Template;
import cn.hutool.extra.template.TemplateConfig;
import cn.hutool.extra.template.TemplateEngine;
import cn.hutool.extra.template.TemplateUtil;
import java.nio.charset.StandardCharsets;
import java.util.Map;
/**
* @author lanyuanxiaoyao
* @version 20250612
*/
public abstract class GeneratorTool {
private final TemplateEngine engine = TemplateUtil.createEngine(new TemplateConfig("template", TemplateConfig.ResourceMode.CLASSPATH));
protected void generateTemplate(String templatePath, Map<?, ?> data, String targetScriptPath) {
Template template = engine.getTemplate(templatePath);
String script = template.render(data);
FileUtil.del(targetScriptPath);
FileUtil.writeString(script, targetScriptPath, StandardCharsets.UTF_8);
}
public abstract void generate() throws Exception;
}

View File

@@ -0,0 +1,244 @@
package com.lanyuanxiaoyao.service.ai.cli;
import cn.hutool.core.collection.ListUtil;
import cn.hutool.core.util.StrUtil;
import java.util.List;
import java.util.Map;
/**
* @author lanyuanxiaoyao
* @version 20250612
*/
public class LlamaSwapTool extends GeneratorTool {
private static final String API_KEY = "*XMySqV%>hR&v>>g*NwCs3tpQ5FVMFEF2VHVTj<MYQd$&@$sY7CgqNyea4giJi4";
public static String displayName(String name) {
return name.replaceAll("\\s+", "_")
.replaceAll("\\.", "_")
.toLowerCase();
}
@Override
public void generate() {
generateTemplate(
"llama-swap.ftl",
Map.of(
"models", List.of(
llamaCppEmbeddingCmd("BGE/bge-m3-q4km", "bge-m3-Q4_K_M.gguf", 20),
vllmEmbeddingCmd("BGE/bge-m3", "bge-m3", 5),
llamaCppRerankerCmd("BGE/beg-reranker-v2-q4km", "bge-reranker-v2-m3-Q4_K_M.gguf", 20),
vllmRerankerCmd("BGE/beg-reranker-v2", "bge-reranker-v2-m3", 5),
vllmCmd("Qwen3/qwen3-0.6b", "Qwen3-0.6B", 5, true),
vllmCmd("Qwen3/qwen3-1.7b", "Qwen3-1.7B", 5, true),
vllmCmd("Qwen3/qwen3-4b", "Qwen3-4B", 8, true),
llamaCppCmd("Qwen3/qwen3-4b-q4km", "Qwen3-4B-Q4_K_M.gguf", 35),
llamaCppCmd("Qwen3/qwen3-8b-q4km", "Qwen3-8B-Q4_K_M.gguf", 35),
vllmEmbeddingCmd("Qwen3/qwen3-embedding-0.6b", "Qwen3-Embedding-0.6B", 5),
vllmEmbeddingCmd("Qwen3/qwen3-embedding-4b", "Qwen3-Embedding-4B", 8),
llamaCppEmbeddingCmd("Qwen3/qwen3-embedding-4b-q4km", "Qwen3-Embedding-4B-Q4_K_M.gguf", 35),
llamaCppEmbeddingCmd("Qwen3/qwen3-embedding-8b-q4km", "Qwen3-Embedding-8B-Q4_K_M.gguf", 35),
// 0.9.1 vllm还未支持
// vllmRerankerCmd("Qwen3/qwen3-reranker-0.6b", "Qwen3-Reranker-0.6B", 5),
// vllmRerankerCmd("Qwen3/qwen3-reranker-4b", "Qwen3-Reranker-4B", 8),
llamaCppVisualCmd("Qwen2.5/qwen2.5-vl-7b", "Qwen2.5-VL-7B-Instruct-BF16.gguf", 35),
llamaCppVisualCmd("Qwen2.5/qwen2.5-vl-7b-q4km", "Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf", 35),
vllmCmd("Qwen2.5/qwen2.5-vl-3b-instruct", "Qwen2.5-VL-3B-Instruct", 8, false),
vllmCmd("Qwen2.5/qwen2.5-vl-7b-instruct", "Qwen2.5-VL-7B-Instruct", 8, false),
llamaCppVisualCmd("MiniCPM/minicpm-o-2.6-7.6b-q4km", "MiniCPM-o-2_6-7.6B-Q4_K_M.gguf", 35),
vllmCmd("MiniCPM/minicpm-o-2.6-7.6b", "MiniCPM-o-2_6", 10, false)
)
),
"config.yaml"
);
}
private DockerCmd llamaCppCmd(String name, String model, Integer thread) {
return llamaCppCmd(name, model, thread, false, false, false);
}
private DockerCmd llamaCppEmbeddingCmd(String name, String model, Integer thread) {
return llamaCppCmd(name, model, thread, true, false, false);
}
private DockerCmd llamaCppRerankerCmd(String name, String model, Integer thread) {
return llamaCppCmd(name, model, thread, false, true, false);
}
private DockerCmd llamaCppVisualCmd(String name, String model, Integer thread) {
return llamaCppCmd(name, model, thread, false, false, true);
}
private DockerCmd llamaCppCmd(String name, String model, Integer thread, Boolean isEmbedding, Boolean isReranker, Boolean isVisual) {
List<String> arguments = ListUtil.list(
false,
StrUtil.format("-m /models/{}", model),
"--port ${PORT}",
StrUtil.format("--api-key {}", API_KEY),
"-c 0",
"-b 4096",
StrUtil.format("-t {}", thread),
"-np 5",
"--log-disable",
"--no-webui"
);
if (isEmbedding) {
arguments.add("--embedding");
arguments.add("-ub 8192");
arguments.add("--pooling mean");
} else if (isReranker) {
arguments.add("--reranking");
} else if (isVisual) {
arguments.add(StrUtil.format("--mmproj /models/{}.mmproj", model));
} else {
arguments.add("--jinja");
}
return new DockerCmd(
"ghcr.io/ggml-org/llama.cpp:server",
name,
model,
StrUtil.format("http://llamacpp-{}:${PORT}", displayName(model)),
List.of(StrUtil.format("--name llamacpp-{}", displayName(model))),
arguments
);
}
private DockerCmd vllmCmd(String name, String model, Integer cache, Boolean isReasonable) {
return vllmCmd(name, model, cache, false, false, isReasonable);
}
private DockerCmd vllmEmbeddingCmd(String name, String model, Integer cache) {
return vllmCmd(name, model, cache, true, false, false);
}
private DockerCmd vllmRerankerCmd(String name, String model, Integer cache) {
return vllmCmd(name, model, cache, false, true, false);
}
private DockerCmd vllmVisualCmd(String name, String model, Integer cache, Boolean isReasonable) {
return vllmCmd(name, model, cache, false, false, isReasonable);
}
private DockerCmd vllmCmd(String name, String model, Integer cache, Boolean isEmbedding, Boolean isReranker, Boolean isReasonable) {
List<String> arguments = ListUtil.list(
false,
StrUtil.format("--model /models/{}", model),
StrUtil.format("--served-model-name {}", name),
"--port ${PORT}",
StrUtil.format("--api-key {}", API_KEY),
"--disable-log-requests",
"--uvicorn-log-level error",
"--trust-remote-code"
);
if (isEmbedding) {
arguments.add("--task embedding");
} else if (isReranker) {
arguments.add("--task score");
} else if (isReasonable) {
arguments.add("--enable-auto-tool-choice");
arguments.add("--tool-call-parser hermes");
arguments.add("--enable-reasoning");
arguments.add("--reasoning-parser deepseek_r1");
}
return new DockerCmd(
"vllm-server-cpu:0.8.5.post1",
name,
model,
StrUtil.format("http://vllm-{}:${PORT}", displayName(model)),
List.of(
StrUtil.format("--name vllm-{}", displayName(model)),
"--privileged=true",
"--shm-size=4g",
StrUtil.format("-e VLLM_CPU_KVCACHE_SPACE={}", cache)
),
arguments
);
}
public static class DockerCmd {
private String image;
private String name;
private String model;
private String proxy;
private List<String> options = ListUtil.list(
false,
"--rm",
"--network llama",
"-v /data/models:/models"
);
private List<String> arguments = ListUtil.list(false);
public DockerCmd(String image, String name, String model, String proxy, List<String> options, List<String> arguments) {
this.image = image;
this.name = name;
this.model = model;
this.proxy = proxy;
this.options.addAll(options);
this.arguments.addAll(arguments);
}
public String getImage() {
return image;
}
public void setImage(String image) {
this.image = image;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getModel() {
return model;
}
public void setModel(String model) {
this.model = model;
}
public String getProxy() {
return proxy;
}
public void setProxy(String proxy) {
this.proxy = proxy;
}
public List<String> getOptions() {
return options;
}
public void setOptions(List<String> options) {
this.options = options;
}
public List<String> getArguments() {
return arguments;
}
public void setArguments(List<String> arguments) {
this.arguments = arguments;
}
@Override
public String toString() {
return "DockerCmd{" +
"image='" + image + '\'' +
", name='" + name + '\'' +
", model='" + model + '\'' +
", proxy='" + proxy + '\'' +
", options=" + options +
", arguments=" + arguments +
'}';
}
}
}

View File

@@ -0,0 +1,25 @@
healthCheckTimeout: 600
logLevel: warn
models:
<#list models as model>
"${model.name}":
proxy: ${model.proxy}
ttl: 86400
cmd: |
docker run
<#list model.options as option>
${option}
</#list>
${model.image}
<#list model.arguments as arg>
${arg}
</#list>
</#list>
groups:
"persistent":
swap: false
exclusive: false
members:
<#list models as model>
- "${model.name}"
</#list>

View File

@@ -0,0 +1,74 @@
package com.lanyuanxiaoyao.service.ai.core.configuration;
import java.time.Instant;
/**
* 使用雪花算法作为ID生成器
*
* @author lanyuanxiaoyao
* @date 2024-11-14
*/
public class SnowflakeId {
/**
* 起始的时间戳
*/
private final static long START_TIMESTAMP = 1;
/**
* 序列号占用的位数
*/
private final static long SEQUENCE_BIT = 11;
/**
* 序列号最大值
*/
private final static long MAX_SEQUENCE_BIT = ~(-1 << SEQUENCE_BIT);
/**
* 时间戳值向左位移
*/
private final static long TIMESTAMP_OFFSET = SEQUENCE_BIT;
/**
* 序列号
*/
private static long sequence = 0;
/**
* 上一次时间戳
*/
private static long lastTimestamp = -1;
public static synchronized long next() {
long currentTimestamp = nowTimestamp();
if (currentTimestamp < lastTimestamp) {
throw new RuntimeException("Clock have moved backwards.");
}
if (currentTimestamp == lastTimestamp) {
// 相同毫秒内, 序列号自增
sequence = (sequence + 1) & MAX_SEQUENCE_BIT;
// 同一毫秒的序列数已经达到最大
if (sequence == 0) {
currentTimestamp = nextTimestamp();
}
} else {
// 不同毫秒内, 序列号置为0
sequence = 0;
}
lastTimestamp = currentTimestamp;
return (currentTimestamp - START_TIMESTAMP) << TIMESTAMP_OFFSET | sequence;
}
private static long nextTimestamp() {
long milli = nowTimestamp();
while (milli <= lastTimestamp) {
milli = nowTimestamp();
}
return milli;
}
private static long nowTimestamp() {
return Instant.now().toEpochMilli();
}
}

View File

@@ -17,22 +17,30 @@ import org.springframework.web.reactive.function.client.WebClient;
*/
@Configuration
public class WebClientConfiguration {
private HttpClient httpClient() {
return HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_1_1)
.build();
}
@Bean
@Primary
public RestClient.Builder restClientBuilder() {
return RestClient.builder()
.requestFactory(new JdkClientHttpRequestFactory(httpClient()));
return generateRestClientBuilder();
}
@Bean
@Primary
public WebClient.Builder webClientBuilder() {
return generateWebClientBuilder();
}
private static HttpClient httpClient() {
return HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_1_1)
.build();
}
public static RestClient.Builder generateRestClientBuilder() {
return RestClient.builder()
.requestFactory(new JdkClientHttpRequestFactory(httpClient()));
}
public static WebClient.Builder generateWebClientBuilder() {
return WebClient.builder()
.clientConnector(new JdkClientHttpConnector(httpClient()));
}

View File

@@ -0,0 +1,30 @@
package com.lanyuanxiaoyao.service.ai.core.entity.amis;
/**
* Crud 响应
*
* @author lanyuanxiaoyao
* @date 2023-07-06
*/
public class AmisCrudResponse extends AmisMapResponse {
public void setData(Iterable<?> list) {
getData().put("items", list);
}
public void setTotal(Long total) {
getData().put("total", total);
}
public void setTotal(Integer total) {
setTotal(total.longValue());
}
public void setData(Iterable<?> list, Long total) {
setData(list);
setTotal(total);
}
public void setData(Iterable<?> list, Integer total) {
setData(list, total.longValue());
}
}

View File

@@ -0,0 +1,13 @@
package com.lanyuanxiaoyao.service.ai.core.entity.amis;
/**
* Crud 响应
*
* @author lanyuanxiaoyao
* @date 2023-07-06
*/
public class AmisDetailResponse extends AmisMapResponse {
public void setDetail(Object detail) {
getData().put("detail", detail);
}
}

View File

@@ -0,0 +1,21 @@
package com.lanyuanxiaoyao.service.ai.core.entity.amis;
import java.util.HashMap;
import java.util.Map;
/**
* Map 响应
*
* @author lanyuanxiaoyao
* @date 2023-07-06
*/
public class AmisMapResponse extends AmisResponse<Map<String, Object>> {
public AmisMapResponse() {
setData(new HashMap<>());
}
public AmisMapResponse setData(String key, Object value) {
getData().put(key, value);
return this;
}
}

View File

@@ -0,0 +1,138 @@
package com.lanyuanxiaoyao.service.ai.core.entity.amis;
import java.util.Map;
/**
* Amis 组件结构化返回值
*
* @author lanyuanxiaoyao
* @date 2022-09-21
*/
public class AmisResponse<T> {
private static final int SUCCESS_STATUS = 0;
private static final int ERROR_STATUS = 500;
private static final String SUCCESS_MESSAGE = "OK";
private static final String ERROR_MESSAGE = "ERROR";
private Integer status;
private String message;
private T data;
public static AmisResponse<Object> responseError() {
return responseError(ERROR_MESSAGE);
}
public static AmisResponse<Object> responseError(String message) {
AmisResponse<Object> response = new AmisResponse<>();
response.setStatus(ERROR_STATUS);
response.setMessage(message);
return response;
}
public static AmisResponse<Object> responseSuccess() {
AmisResponse<Object> response = new AmisResponse<>();
response.setStatus(SUCCESS_STATUS);
response.setMessage(SUCCESS_MESSAGE);
return response;
}
public static AmisResponse<Object> responseSuccess(String message) {
AmisResponse<Object> response = new AmisResponse<>();
response.setStatus(SUCCESS_STATUS);
response.setMessage(message);
return response;
}
public static <E> AmisResponse<E> responseSuccess(String message, E data) {
AmisResponse<E> response = new AmisResponse<>();
response.setStatus(SUCCESS_STATUS);
response.setMessage(message);
response.setData(data);
return response;
}
public static <E> AmisResponse<E> responseSuccess(E data) {
AmisResponse<E> response = new AmisResponse<>();
response.setStatus(SUCCESS_STATUS);
response.setMessage(SUCCESS_MESSAGE);
response.setData(data);
return response;
}
public static AmisMapResponse responseMapData() {
AmisMapResponse response = new AmisMapResponse();
response.setStatus(SUCCESS_STATUS);
response.setMessage(SUCCESS_MESSAGE);
return response;
}
public static AmisMapResponse responseMapData(Map<String, Object> data) {
AmisMapResponse response = responseMapData();
response.setData(data);
return response;
}
public static AmisMapResponse responseMapData(String key, Object value) {
AmisMapResponse response = responseMapData();
response.setData(key, value);
return response;
}
public static AmisCrudResponse responseCrudData(Iterable<?> data) {
AmisCrudResponse response = new AmisCrudResponse();
response.setStatus(SUCCESS_STATUS);
response.setMessage(SUCCESS_MESSAGE);
response.setData(data);
return response;
}
public static AmisCrudResponse responseCrudData(Iterable<?> data, Integer total) {
AmisCrudResponse response = responseCrudData(data);
response.setTotal(total);
return response;
}
public static AmisCrudResponse responseCrudData(Iterable<?> data, Long total) {
AmisCrudResponse response = responseCrudData(data);
response.setTotal(total);
return response;
}
public static AmisDetailResponse responseDetailData(Object detail) {
AmisDetailResponse response = new AmisDetailResponse();
response.setDetail(detail);
return response;
}
public Integer getStatus() {
return status;
}
public void setStatus(Integer status) {
this.status = status;
}
public String getMessage() {
return message;
}
public void setMessage(String message) {
this.message = message;
}
public T getData() {
return data;
}
public void setData(T data) {
this.data = data;
}
@Override
public String toString() {
return "AmisResponse{" +
"status=" + status +
", message='" + message + '\'' +
", data=" + data +
'}';
}
}

View File

@@ -4,6 +4,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.HttpMethod;
import org.springframework.security.config.Customizer;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
@@ -12,9 +13,6 @@ import org.springframework.security.core.userdetails.User;
import org.springframework.security.core.userdetails.UserDetails;
import org.springframework.security.provisioning.InMemoryUserDetailsManager;
import org.springframework.security.web.SecurityFilterChain;
import org.springframework.web.cors.CorsConfiguration;
import org.springframework.web.cors.CorsConfigurationSource;
import org.springframework.web.cors.UrlBasedCorsConfigurationSource;
/**
* @author lanyuanxiaoyao
@@ -23,29 +21,22 @@ import org.springframework.web.cors.UrlBasedCorsConfigurationSource;
@Configuration
@EnableWebSecurity
public class SecurityConfig {
private static final Logger logger = LoggerFactory.getLogger(SecurityConfig.class);
@Bean
public SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
return http.authorizeHttpRequests(registry -> registry.anyRequest().authenticated())
return http.authorizeHttpRequests(
registry -> registry
.requestMatchers(HttpMethod.OPTIONS, "/**")
.permitAll()
.anyRequest()
.authenticated()
)
.httpBasic(Customizer.withDefaults())
.cors(AbstractHttpConfigurer::disable)
.csrf(AbstractHttpConfigurer::disable)
.cors(configurer -> configurer.configurationSource(corsConfigurationSource()))
.formLogin(AbstractHttpConfigurer::disable)
.build();
}
private CorsConfigurationSource corsConfigurationSource() {
CorsConfiguration configuration = new CorsConfiguration();
configuration.setAllowCredentials(true);
configuration.addAllowedHeader("*");
configuration.addAllowedMethod("*");
configuration.addAllowedOriginPattern("*");
UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
source.registerCorsConfiguration("/**", configuration);
return source;
}
@Bean
public InMemoryUserDetailsManager userDetailsService(SecurityProperties securityProperties) {
UserDetails user = User.builder()

View File

@@ -1,129 +0,0 @@
package com.lanyuanxiaoyao.service.ai.knowledge.controller;
import com.lanyuanxiaoyao.service.ai.knowledge.entity.vo.KnowledgeVO;
import com.lanyuanxiaoyao.service.ai.knowledge.entity.vo.PointVO;
import com.lanyuanxiaoyao.service.ai.knowledge.reader.TextLineReader;
import com.lanyuanxiaoyao.service.ai.knowledge.service.KnowledgeService;
import io.qdrant.client.QdrantClient;
import io.qdrant.client.grpc.Points;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.ExecutionException;
import java.util.stream.Collectors;
import org.eclipse.collections.api.factory.Lists;
import org.eclipse.collections.api.list.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.reader.TextReader;
import org.springframework.ai.reader.markdown.MarkdownDocumentReader;
import org.springframework.ai.reader.markdown.config.MarkdownDocumentReaderConfig;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.vectorstore.qdrant.QdrantVectorStore;
import org.springframework.core.io.ByteArrayResource;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
/**
* @author lanyuanxiaoyao
* @version 20250515
*/
@RestController
@RequestMapping("knowledge")
public class KnowledgeController {
private static final Logger logger = LoggerFactory.getLogger(KnowledgeController.class);
private final KnowledgeService knowledgeService;
private final QdrantClient client;
private final EmbeddingModel embeddingModel;
public KnowledgeController(KnowledgeService knowledgeService, VectorStore vectorStore, EmbeddingModel embeddingModel) {
this.knowledgeService = knowledgeService;
client = (QdrantClient) vectorStore.getNativeClient().orElseThrow();
this.embeddingModel = embeddingModel;
}
@PostMapping("add")
public void add(
@RequestParam("name") String name,
@RequestParam("strategy") String strategy
) throws ExecutionException, InterruptedException {
knowledgeService.add(name, strategy);
}
@GetMapping("list")
public ImmutableList<KnowledgeVO> list() {
return knowledgeService.list();
}
@GetMapping("list_points")
public ImmutableList<PointVO> listPoints(@RequestParam("name") String name) throws ExecutionException, InterruptedException {
Points.ScrollResponse response = client.scrollAsync(
Points.ScrollPoints.newBuilder()
.setCollectionName(name)
// .setLimit(2)
.setWithPayload(Points.WithPayloadSelector.newBuilder().setEnable(true).build())
.setWithVectors(Points.WithVectorsSelector.newBuilder().setEnable(false).build())
.build()
)
.get();
return response.getResultList()
.stream()
.collect(Collectors.toCollection(Lists.mutable::empty))
.collect(point -> {
PointVO vo = new PointVO();
vo.setId(point.getId().getUuid());
vo.setText(point.getPayloadMap().get("doc_content").getStringValue());
return vo;
})
.toImmutable();
}
@GetMapping("delete")
public void delete(@RequestParam("name") String name) throws ExecutionException, InterruptedException {
knowledgeService.remove(name);
}
@PostMapping("preview_text")
public ImmutableList<PointVO> previewText(
@RequestParam("name") String name,
@RequestParam(value = "mode", defaultValue = "normal") String mode,
@RequestParam(value = "type", defaultValue = "text") String type,
@RequestParam("content") String content
) {
TextReader reader = new TextLineReader(new ByteArrayResource(content.getBytes(StandardCharsets.UTF_8)));
return reader.get()
.stream()
.collect(Collectors.toCollection(Lists.mutable::empty))
.collect(doc -> {
PointVO vo = new PointVO();
vo.setId(doc.getId());
vo.setText(doc.getText());
return vo;
})
.toImmutable();
}
@PostMapping(value = "process_text", consumes = "text/plain;charset=utf-8")
public void processText(
@RequestParam("name") String name,
@RequestBody String text
) {
VectorStore source = QdrantVectorStore.builder(client, embeddingModel)
.collectionName(name)
.initializeSchema(true)
.build();
MarkdownDocumentReader reader = new MarkdownDocumentReader(
new ByteArrayResource(text.getBytes(StandardCharsets.UTF_8)),
MarkdownDocumentReaderConfig.builder()
.withHorizontalRuleCreateDocument(true)
.withIncludeCodeBlock(false)
.withIncludeBlockquote(false)
.build()
);
source.add(reader.get());
}
}

View File

@@ -1,54 +0,0 @@
package com.lanyuanxiaoyao.service.ai.knowledge.entity;
/**
* @author lanyuanxiaoyao
* @version 20250522
*/
public class Knowledge {
private Long id;
private Long vectorSourceId;
private String name;
private String strategy;
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public Long getVectorSourceId() {
return vectorSourceId;
}
public void setVectorSourceId(Long vectorSourceId) {
this.vectorSourceId = vectorSourceId;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getStrategy() {
return strategy;
}
public void setStrategy(String strategy) {
this.strategy = strategy;
}
@Override
public String toString() {
return "Knowledge{" +
"id=" + id +
", vectorSourceId=" + vectorSourceId +
", name='" + name + '\'' +
", strategy='" + strategy + '\'' +
'}';
}
}

View File

@@ -1,74 +0,0 @@
package com.lanyuanxiaoyao.service.ai.knowledge.entity.vo;
/**
* @author lanyuanxiaoyao
* @version 20250516
*/
public class KnowledgeVO {
private String name;
private String strategy;
private Long size;
private Long points;
private Long segments;
private String status;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getStrategy() {
return strategy;
}
public void setStrategy(String strategy) {
this.strategy = strategy;
}
public Long getSize() {
return size;
}
public void setSize(Long size) {
this.size = size;
}
public Long getPoints() {
return points;
}
public void setPoints(Long points) {
this.points = points;
}
public Long getSegments() {
return segments;
}
public void setSegments(Long segments) {
this.segments = segments;
}
public String getStatus() {
return status;
}
public void setStatus(String status) {
this.status = status;
}
@Override
public String toString() {
return "CollectionVO{" +
"name='" + name + '\'' +
", strategy='" + strategy + '\'' +
", size=" + size +
", points=" + points +
", segments=" + segments +
", status='" + status + '\'' +
'}';
}
}

View File

@@ -1,34 +0,0 @@
package com.lanyuanxiaoyao.service.ai.knowledge.entity.vo;
/**
* @author lanyuanxiaoyao
* @version 20250516
*/
public class PointVO {
private String id;
private String text;
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getText() {
return text;
}
public void setText(String text) {
this.text = text;
}
@Override
public String toString() {
return "PointVO{" +
"id='" + id + '\'' +
", text='" + text + '\'' +
'}';
}
}

View File

@@ -1,34 +0,0 @@
package com.lanyuanxiaoyao.service.ai.knowledge.reader;
import cn.hutool.core.util.StrUtil;
import java.util.List;
import java.util.stream.Stream;
import org.springframework.ai.document.Document;
import org.springframework.ai.reader.TextReader;
import org.springframework.core.io.Resource;
/**
* @author lanyuanxiaoyao
* @version 20250522
*/
public class TextLineReader extends TextReader {
public TextLineReader(Resource resource) {
super(resource);
}
@Override
public List<Document> get() {
return super.get()
.stream()
.flatMap(doc -> {
String text = doc.getText();
if (StrUtil.isBlank(text)) {
return Stream.of(doc);
}
return Stream.of(text.split("\n\n"))
.filter(StrUtil::isNotBlank)
.map(line -> new Document(line, doc.getMetadata()));
})
.toList();
}
}

View File

@@ -1,16 +0,0 @@
package com.lanyuanxiaoyao.service.ai.knowledge.service;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
/**
* @author lanyuanxiaoyao
* @version 20250522
*/
@Service
public class EmbeddingService {
private static final Logger logger = LoggerFactory.getLogger(EmbeddingService.class);
}

View File

@@ -1,58 +0,0 @@
package com.lanyuanxiaoyao.service.ai.knowledge.service;
import club.kingon.sql.builder.SqlBuilder;
import cn.hutool.core.util.IdUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
/**
* @author lanyuanxiaoyao
* @version 20250522
*/
@Service
public class KnowledgeGroupService {
private static final Logger logger = LoggerFactory.getLogger(KnowledgeGroupService.class);
private static final String GROUP_TABLE_NAME = "service_ai_group";
private final JdbcTemplate template;
public KnowledgeGroupService(JdbcTemplate template) {
this.template = template;
}
@Transactional(rollbackFor = Exception.class)
public void add(Long knowledgeId, String name) {
template.update(
SqlBuilder.insertInto(GROUP_TABLE_NAME, "id", "knowledge_id", "name")
.values()
.addValue("?", "?", "?")
.precompileSql(),
IdUtil.getSnowflakeNextId(),
knowledgeId,
name
);
}
@Transactional(rollbackFor = Exception.class)
public void remove(Long groupId) {
template.update(
SqlBuilder.delete(GROUP_TABLE_NAME)
.whereEq("id", "?")
.precompileSql(),
groupId
);
}
@Transactional(rollbackFor = Exception.class)
public void removeByKnowledgeId(Long knowledgeId) {
template.update(
SqlBuilder.delete(GROUP_TABLE_NAME)
.whereEq("knowledge_id", "?")
.precompileSql(),
knowledgeId
);
}
}

View File

@@ -1,151 +0,0 @@
package com.lanyuanxiaoyao.service.ai.knowledge.service;
import club.kingon.sql.builder.SqlBuilder;
import cn.hutool.core.util.IdUtil;
import cn.hutool.core.util.ObjectUtil;
import cn.hutool.core.util.StrUtil;
import com.lanyuanxiaoyao.service.ai.knowledge.entity.Knowledge;
import com.lanyuanxiaoyao.service.ai.knowledge.entity.vo.KnowledgeVO;
import io.qdrant.client.QdrantClient;
import io.qdrant.client.grpc.Collections;
import java.util.concurrent.ExecutionException;
import java.util.stream.Collectors;
import org.eclipse.collections.api.factory.Lists;
import org.eclipse.collections.api.list.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
/**
* @author lanyuanxiaoyao
* @version 20250522
*/
@Service
public class KnowledgeService {
private static final Logger logger = LoggerFactory.getLogger(KnowledgeService.class);
private static final String KNOWLEDGE_TABLE_NAME = "service_ai_knowledge";
private final JdbcTemplate template;
private final EmbeddingModel embeddingModel;
private final QdrantClient client;
private final KnowledgeGroupService knowledgeGroupService;
public KnowledgeService(JdbcTemplate template, EmbeddingModel embeddingModel, VectorStore vectorStore, KnowledgeGroupService knowledgeGroupService) {
this.template = template;
this.embeddingModel = embeddingModel;
this.client = (QdrantClient) vectorStore.getNativeClient().orElseThrow();
this.knowledgeGroupService = knowledgeGroupService;
}
public Knowledge get(Long id) {
return template.queryForObject(
SqlBuilder.select("id", "vector_source_id", "name", "strategy")
.from(KNOWLEDGE_TABLE_NAME)
.whereEq("id", "?")
.precompileSql(),
Knowledge.class,
id
);
}
public Knowledge get(String name) {
return template.queryForObject(
SqlBuilder.select("id", "vector_source_id", "name", "strategy")
.from(KNOWLEDGE_TABLE_NAME)
.whereEq("name", "?")
.precompileSql(),
Knowledge.class,
name
);
}
@Transactional(rollbackFor = Exception.class)
public void add(String name, String strategy) throws ExecutionException, InterruptedException {
Integer count = template.queryForObject(
SqlBuilder.select("count(*)")
.from(KNOWLEDGE_TABLE_NAME)
.whereEq("name", "?")
.precompileSql(),
Integer.class,
name
);
if (count > 0) {
throw new RuntimeException("名称已存在");
}
long id = IdUtil.getSnowflakeNextId();
long vectorSourceId = IdUtil.getSnowflakeNextId();
template.update(
SqlBuilder.insertInto(KNOWLEDGE_TABLE_NAME, "id", "vector_source_id", "name", "strategy")
.values()
.addValue("?", "?", "?", "?")
.precompileSql(),
id,
vectorSourceId,
name,
strategy
);
client.createCollectionAsync(
String.valueOf(vectorSourceId),
Collections.VectorParams.newBuilder()
.setDistance(Collections.Distance.valueOf(strategy))
.setSize(embeddingModel.dimensions())
.build()
).get();
}
public ImmutableList<KnowledgeVO> list() {
return template.query(
SqlBuilder.select("id", "vector_source_id", "name", "strategy")
.from(KNOWLEDGE_TABLE_NAME)
.build(),
(rs, index) -> {
Knowledge knowledge = new Knowledge();
knowledge.setId(rs.getLong(1));
knowledge.setVectorSourceId(rs.getLong(2));
knowledge.setName(rs.getString(3));
knowledge.setStrategy(rs.getString(4));
return knowledge;
}
)
.stream()
.map(knowledge -> {
try {
Collections.CollectionInfo info = client.getCollectionInfoAsync(String.valueOf(knowledge.getVectorSourceId())).get();
KnowledgeVO vo = new KnowledgeVO();
vo.setName(knowledge.getName());
vo.setPoints(info.getPointsCount());
vo.setSegments(info.getSegmentsCount());
vo.setStatus(info.getStatus().name());
Collections.VectorParams vectorParams = info.getConfig().getParams().getVectorsConfig().getParams();
vo.setStrategy(vectorParams.getDistance().name());
vo.setSize(vectorParams.getSize());
return vo;
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
})
.collect(Collectors.toCollection(Lists.mutable::empty))
.toImmutable();
}
@Transactional(rollbackFor = Exception.class)
public void remove(String name) throws ExecutionException, InterruptedException {
Knowledge knowledge = get(name);
if (ObjectUtil.isNull(knowledge)) {
throw new RuntimeException(StrUtil.format("{} 不存在"));
}
template.update(
SqlBuilder.delete(KNOWLEDGE_TABLE_NAME)
.whereEq("id", "?")
.precompileSql(),
knowledge.getId()
);
knowledgeGroupService.removeByKnowledgeId(knowledge.getId());
client.deleteCollectionAsync(String.valueOf(knowledge.getVectorSourceId())).get();
}
}

View File

@@ -1,51 +0,0 @@
spring:
application:
name: service-ai-knowledge
profiles:
include: common,metrics,forest
cloud:
zookeeper:
enabled: true
connect-string: b1m2.hdp.dc:2181,b1m3.hdp.dc:2181,b1m4.hdp.dc:2181,b1m5.hdp.dc:2181,b1m6.hdp.dc:2181
discovery:
enabled: ${spring.cloud.zookeeper.enabled}
root: /hudi-services
instance-id: ${spring.application.name}-127.0.0.1-${random.uuid}-20250514
metadata:
discovery: zookeeper
ip: 127.0.0.1
hostname: localhost
hostname_full: localhost
start_time: 20250514112750
datasource:
url: jdbc:mysql://localhost:3307/ai?useSSL=false
username: test
password: test
driver-class-name: com.mysql.cj.jdbc.Driver
security:
meta:
authority: ENC(GXKnbq1LS11U2HaONspvH+D/TkIx13aWTaokdkzaF7HSvq6Z0Rv1+JUWFnYopVXu)
username: ENC(moIO5mO39V1Z+RDwROK9JXY4GfM8ZjDgM6Si7wRZ1MPVjbhTpmLz3lz28rAiw7c2LeCmizfJzHkEXIwGlB280g==)
darkcode: ENC(0jzpQ7T6S+P7bZrENgYsUoLhlqGvw7DA2MN3BRqEOwq7plhtg72vuuiPQNnr3DaYz0CpyTvxInhpx11W3VZ1trD6NINh7O3LN70ZqO5pWXk=)
ai:
openai:
base-url: http://132.121.206.65:10086
api-key: ENC(K+Hff9QGC+fcyi510VIDd9CaeK/IN5WBJ9rlkUsHEdDgIidW+stHHJlsK0lLPUXXREha+ToQZqqDXJrqSE+GUKCXklFhelD8bRHFXBIeP/ZzT2cxhzgKUXgjw3S0Qw2R)
chat:
options:
model: 'Qwen3-1.7'
embedding:
options:
model: 'Bge-m3'
vectorstore:
qdrant:
api-key: lanyuanxiaoyao
jasypt:
encryptor:
password: 'r#(R,P"Dp^A47>WSn:Wn].gs/+"v:q_Q*An~zF*g-@j@jtSTv5H/,S-3:R?r9R}.'
server:
port: 8080
liteflow:
rule-source: config/flow.xml
print-banner: false
check-node-exists: false

View File

@@ -1,14 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<flow>
<chain name="embedding">
SER(
embedding_start,
SWITCH(embedding_mode_switch).TO(
normal_embedding,
llm_embedding,
qa_embedding
),
embedding_finish
);
</chain>
</flow>

View File

@@ -1,34 +0,0 @@
<configuration>
<conversionRule conversionWord="clr" converterClass="org.springframework.boot.logging.logback.ColorConverter" />
<conversionRule conversionWord="wex" converterClass="org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter" />
<conversionRule conversionWord="wEx" converterClass="org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter" />
<springProperty scope="context" name="LOKI_PUSH_URL" source="loki.url"/>
<springProperty scope="context" name="LOGGING_PARENT" source="logging.parent"/>
<springProperty scope="context" name="APP_NAME" source="spring.application.name"/>
<appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %clr(%5p) %clr([${HOSTNAME}]){yellow} %clr([%t]){magenta} %clr(%logger{40}){cyan} #@# %m%n%wEx</pattern>
</encoder>
</appender>
<appender name="RollingFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOGGING_PARENT:-.}/${APP_NAME:-run}.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOGGING_PARENT:-.}/archive/${APP_NAME:-run}-%d{yyyy-MM-dd}.gz</fileNamePattern>
<MaxHistory>7</MaxHistory>
</rollingPolicy>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %p [${HOSTNAME}] [%t] %logger #@# %m%n%wEx</pattern>
</encoder>
</appender>
<logger name="com.zaxxer.hikari" level="ERROR"/>
<logger name="com.netflix.discovery.shared.resolver.aws.ConfigClusterResolver" level="WARN"/>
<root level="INFO">
<appender-ref ref="Console"/>
<!-- <appender-ref ref="RollingFile"/>-->
</root>
</configuration>

View File

@@ -9,7 +9,7 @@
<version>1.0.0-SNAPSHOT</version>
</parent>
<artifactId>service-ai-knowledge</artifactId>
<artifactId>service-ai-web</artifactId>
<dependencies>
<dependency>
@@ -26,6 +26,10 @@
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-model-deepseek</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-vector-store-qdrant</artifactId>
@@ -46,10 +50,52 @@
<groupId>com.yomahub</groupId>
<artifactId>liteflow-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-tika-document-reader</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-pdf-document-reader</artifactId>
</dependency>
<dependency>
<groupId>org.noear</groupId>
<artifactId>solon-ai</artifactId>
</dependency>
<dependency>
<groupId>org.noear</groupId>
<artifactId>solon-ai-dialect-openai</artifactId>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<annotationProcessorPaths>
<path>
<groupId>org.mapstruct</groupId>
<artifactId>mapstruct-processor</artifactId>
<version>${mapstruct.version}</version>
</path>
<path>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</path>
<path>
<groupId>org.projectlombok</groupId>
<artifactId>lombok-mapstruct-binding</artifactId>
<version>0.2.0</version>
</path>
</annotationProcessorPaths>
<compilerArgs>
<arg>-Amapstruct.defaultComponentModel=spring</arg>
<arg>-Amapstruct.defaultInjectionStrategy=constructor</arg>
</compilerArgs>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>

View File

@@ -1,13 +1,17 @@
package com.lanyuanxiaoyao.service.ai.knowledge;
package com.lanyuanxiaoyao.service.ai.web;
import com.ulisesbocchio.jasyptspringboot.annotation.EnableEncryptableProperties;
import org.springframework.beans.BeansException;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.retry.annotation.EnableRetry;
import org.springframework.scheduling.annotation.EnableScheduling;
/**
* @author lanyuanxiaoyao
@@ -18,12 +22,28 @@ import org.springframework.retry.annotation.EnableRetry;
@EnableConfigurationProperties
@EnableEncryptableProperties
@EnableRetry
public class KnowledgeApplication implements ApplicationRunner {
@EnableScheduling
public class WebApplication implements ApplicationRunner, ApplicationContextAware {
private static ApplicationContext context;
public static void main(String[] args) {
SpringApplication.run(KnowledgeApplication.class, args);
SpringApplication.run(WebApplication.class, args);
}
public static <T> T getBean(Class<T> clazz) {
return context.getBean(clazz);
}
public static <T> T getBean(String name, Class<T> clazz) {
return context.getBean(name, clazz);
}
@Override
public void run(ApplicationArguments args) {
}
@Override
public void setApplicationContext(ApplicationContext context) throws BeansException {
WebApplication.context = context;
}
}

View File

@@ -0,0 +1,17 @@
package com.lanyuanxiaoyao.service.ai.web.configuration;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
/**
* @author lanyuanxiaoyao
* @version 20250527
*/
@Data
@Configuration
@ConfigurationProperties(prefix = "file-store")
public class FileStoreProperties {
private String downloadPrefix;
private String uploadPath;
}

View File

@@ -0,0 +1,111 @@
package com.lanyuanxiaoyao.service.ai.web.configuration;
import cn.hutool.core.lang.Assert;
import cn.hutool.core.util.StrUtil;
import java.time.Duration;
import lombok.extern.slf4j.Slf4j;
import org.noear.solon.ai.reranking.RerankingModel;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.deepseek.DeepSeekChatModel;
import org.springframework.ai.deepseek.DeepSeekChatOptions;
import org.springframework.ai.deepseek.api.DeepSeekApi;
import org.springframework.ai.document.MetadataMode;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.openai.OpenAiChatModel;
import org.springframework.ai.openai.OpenAiChatOptions;
import org.springframework.ai.openai.OpenAiEmbeddingModel;
import org.springframework.ai.openai.OpenAiEmbeddingOptions;
import org.springframework.ai.openai.api.OpenAiApi;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestClient;
import org.springframework.web.reactive.function.client.WebClient;
@Slf4j
@Configuration
public class LlmConfiguration {
@Bean("chat")
public ChatClient.Builder chatClientBuilder(LlmProperties llmProperties, WebClient.Builder webClientBuilder, RestClient.Builder restClientBuilder) {
Assert.notNull(llmProperties.getChat(), "chat properties is null");
DeepSeekApi.Builder apiBuilder = DeepSeekApi.builder()
.baseUrl(StrUtil.firstNonBlank(llmProperties.getChat().getBaseUrl(), llmProperties.getBaseUrl()))
.apiKey(StrUtil.firstNonBlank(llmProperties.getChat().getApiKey(), llmProperties.getApiKey()))
.webClientBuilder(webClientBuilder)
.restClientBuilder(restClientBuilder);
if (StrUtil.isNotBlank(llmProperties.getChat().getEndpoint())) {
apiBuilder.completionsPath(llmProperties.getChat().getEndpoint());
}
return ChatClient.builder(
DeepSeekChatModel.builder()
.deepSeekApi(apiBuilder.build())
.defaultOptions(
DeepSeekChatOptions.builder()
.model(llmProperties.getChat().getModel())
.build()
)
.build()
);
}
@Bean("visual")
public ChatClient.Builder visualClientBuilder(LlmProperties llmProperties, WebClient.Builder webClientBuilder, RestClient.Builder restClientBuilder) {
Assert.notNull(llmProperties.getVisual(), "visual properties is null");
OpenAiApi.Builder apiBuilder = OpenAiApi.builder()
.baseUrl(StrUtil.firstNonBlank(llmProperties.getVisual().getBaseUrl(), llmProperties.getBaseUrl()))
.apiKey(StrUtil.firstNonBlank(llmProperties.getVisual().getApiKey(), llmProperties.getApiKey()))
.webClientBuilder(webClientBuilder)
.restClientBuilder(restClientBuilder);
if (StrUtil.isNotBlank(llmProperties.getVisual().getEndpoint())) {
apiBuilder.completionsPath(llmProperties.getVisual().getEndpoint());
}
return ChatClient.builder(
OpenAiChatModel.builder()
.openAiApi(apiBuilder.build())
.defaultOptions(
OpenAiChatOptions.builder()
.model(llmProperties.getVisual().getModel())
.build()
)
.build()
);
}
@Bean
public EmbeddingModel embeddingModel(LlmProperties llmProperties, WebClient.Builder webClientBuilder, RestClient.Builder restClientBuilder) {
Assert.notNull(llmProperties.getEmbedding(), "embedding properties is null");
OpenAiApi.Builder apiBuilder = OpenAiApi.builder()
.baseUrl(StrUtil.firstNonBlank(llmProperties.getEmbedding().getBaseUrl(), llmProperties.getBaseUrl()))
.apiKey(StrUtil.firstNonBlank(llmProperties.getEmbedding().getApiKey(), llmProperties.getApiKey()))
.webClientBuilder(webClientBuilder)
.restClientBuilder(restClientBuilder);
if (StrUtil.isNotBlank(llmProperties.getEmbedding().getEndpoint())) {
apiBuilder.embeddingsPath(llmProperties.getEmbedding().getEndpoint());
}
return new OpenAiEmbeddingModel(
apiBuilder.build(),
MetadataMode.EMBED,
OpenAiEmbeddingOptions.builder()
.model(llmProperties.getEmbedding().getModel())
.build()
);
}
@Bean
public RerankingModel rerankingModel(LlmProperties llmProperties) {
Assert.notNull(llmProperties.getReranker(), "reranker properties is null");
String url = llmProperties.getBaseUrl();
if (StrUtil.isNotBlank(llmProperties.getReranker().getBaseUrl())) {
url = llmProperties.getReranker().getBaseUrl();
}
if (StrUtil.isNotBlank(llmProperties.getReranker().getEndpoint())) {
url += llmProperties.getReranker().getEndpoint();
} else {
url += "/v1/rerank";
}
return RerankingModel.of(url)
.apiKey(StrUtil.firstNonBlank(llmProperties.getReranker().getApiKey(), llmProperties.getApiKey()))
.model(llmProperties.getReranker().getModel())
.timeout(Duration.ofMinutes(10))
.build();
}
}

View File

@@ -0,0 +1,25 @@
package com.lanyuanxiaoyao.service.ai.web.configuration;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
@Data
@Configuration
@ConfigurationProperties(prefix = "spring.llm")
public class LlmProperties {
private String baseUrl;
private String apiKey;
private ChatProperties chat;
private ChatProperties visual;
private ChatProperties embedding;
private ChatProperties reranker;
@Data
public static class ChatProperties {
private String baseUrl;
private String apiKey;
private String model;
private String endpoint;
}
}

View File

@@ -0,0 +1,110 @@
package com.lanyuanxiaoyao.service.ai.web.configuration;
import cn.hutool.core.util.StrUtil;
public interface Prompts {
String hudiBase = """
Hudi数据同步服务
实现从源端数据库TeleDB、TelePG到Hudi表的数据实时同步并通过Hudi-Hive插件将Hudi表封为Hive外表即目标端供外部系统进行SQL查询。
数据同步任务基于Flink开发运行在Hadoop Yarn集群上该任务常驻运行在集群上每个任务负责一张或多张表的据同步同步任务从指定的Pulsar队列中实时读取数据经过数据转换和业务处理后写入Hudi表。
数据压缩任务基于Flink开发运行在Hadoop Yarn集群上该任务根据指定的调度策略周期性定时启动每个任务将张指定的Hudi表中的log数据压缩为parquet数据当压缩完成时上一次压缩到当前压缩之间被写入的数据才能被外部统查询。
服务管理平台使用Spring Cloud框架开发的微服务集群用于管理同步任务和压缩任务的运行、停止和状态监控等息提供友好的前端交互网页。
源端上游类似TeleDB、TelePG这些提供原始数据的数据库。
目标端下游数据同步到Hudi表后提供给外部系统查询用的Hive表。
TeleDB中国电信自研分布式MySQL集群组件逻辑上的一张表在物理上被水平分片Sharding为多个“set”表存储在个MySQL节点支持弹性扩容和容灾。
TelePG中国电信自研分布式PostgreSQL集群组件其架构和功能特性与TeleDB高度相似。
逻辑表一张逻辑表对应一张TeleDB或TelePG上的业务表逻辑表中包含广东所有地市的业务数据数据量大的表通常按地市进行分片分区
Hudi表逻辑表数据经同步任务处理后实际存储的位置根据逻辑表的数据量数据量大的会根据地市分为A、B表B也被称为大表通常包含广深东佛广州、深圳、东莞、佛山四个大区的数据A表包含广东除了广深东佛外的其他市特大表如acct_item会按一个地市对应一个Hudi表Hudi表统一配置为MOR表使用Bucket Index索引。
Hive表通过Hudi-Hive插件创建的Hive外表作为下游系统的唯一查询入口该表逻辑上对应源端的一张逻辑表它Hudi服务内部可能存在的多个物理Hudi表如A表、B表或地市分表聚合封装成一个逻辑视图透明地对外提供完整的辑表数据查询。
重点表根据业务系统的要求对于有的表及时性和数据完整性有更高的要求这些表被称为重点表在tb_app_collect_table_info表中的tags字段包含“FOCUS”字符的是重点表。
Flink 任务:即数据同步任务,根据逻辑表的数据量通常有如下规则:
大数据量:采用 1逻辑表:1 Flink任务 模式。该Flink任务内聚合处理该逻辑表对应的所有Hudi表如A表 + B表或多地市表的同步子任务。
小数据量:采用 N逻辑表:1 Flink任务 模式。一个Flink任务内聚合处理多张低数据量逻辑表对应的所有Hudi表同步自务。
Pulsar队列消息队列源端TeleDB逻辑表增量日志包含新增-I、更新-U、删除-D、DDL操作类型由Canal同步组件实时写入Pulsar队列。TelePG逻辑表增量日志由PGSync组件以相同逻辑同步到对应队列。
压缩调度服务service-scheduler(单实例)按策略将压缩任务放入预调度队列定时将预调度任务转移至各集群对应的压缩任务队列service-launcher(与集群一一对应)定时轮询对应集群的压缩任务队列发现任务即调度执行service-queue提供队列机制scheduler与launcher均基于集群资源状态动态调节任务产生与执行速率利用队列缓冲避免资源超限。
压缩调度压缩任务耗时长、资源大。为平衡效率与资源调度服务通常从凌晨2点起每3小时调度一次全部Hudi表的压缩任务。
跨天调度为确保关键表数据在0点后及时可用此时Hive最新数据常未达0点0点至2点间对重点表进行更高频压缩调度。此高频率调度持续直至目标表被标记为“已跨天”
跨天判断按照先后次序1. 源端同步组件(Canal/PGSync)判断源表数据是否跨天。若跨天向队列写入跨天消息2.同步任务接收跨天消息在tb_app_collect_table_version表插入记录3.独立程序判断Hudi表数据是否已跨天。若跨天更新tb_app_collect_table_version对应记录状态。此时逻辑表标记为“已跨天”一张表必须先接收到跨天标记然后才能完成跨天。
集群指Hadoop Yarn集群同步任务仅在b12运行压缩任务主要在b12运行部分重点表在b1或a4运行调度服务根据资源动态在多个集群间分配压缩任务其中b12集群使用default队列b1集群使用datalake队列a4集群使用ten_iap.datalake队列。
""";
String hudiDatabase = """
Hudi数据同步服务使用的数据表详情如下
hudi_collect_build_b12.tb_app_collect_table_info表记录同步任务配置信息
id主键
alias表别名同样可以唯一标识该记录
flink_job_idtb_app_flink_job_config表主键对应的Flink任务
hudi_job_idhuditb_app_hudi_job_config表主键对应Hudi表的配置
sync_yarn_job_idtb_app_yarn_job_config表主键同步任务的yarn资源配置
compaction_yarn_job_idtb_app_yarn_job_config表主键压缩任务的yarn资源配置
src_db源端数据库名称
src_type源端数据库类型取值有teledb、telepg
src_schema源端数据库schema名称
src_table源端表名
src_pulsar_addrpulsar地址
src_topicpulsar队列名称
tgt_hdfs_pathHudi表对应的hdfs路径
status逻辑删除状态y为正常n为删除
filter_field过滤字段用于指定源端消息中的某个字段进行消息过滤如B表同步任务会使用CITY_ID字段并过滤出该字段为广深东佛编码的消息
filter_values过滤值
filter_type过滤类型INCLUDE表示包含指定过滤值的记录保留EXCLUDE表示包含指定过滤值的记录丢弃
bucket_numberHudi bucket index的参数值
partition_fieldHudi表分区字段通常使用CITY_ID
priority优先级整数数字越大优先级越高优先级高该表对应在压缩调度队列中会排在更靠前的位置
hive_db目标端hive库名称
hive_table目标端hive表名
tags标签使用英文逗号分隔用于标识表的特殊属性
hudi_collect_build_b12.tb_app_collect_table_version表记录跨天版本
flink_job_idtb_app_flink_job_config表主键对应的Flink任务
alias表别名 tb_app_collect_table_info表alias字段对应唯一同步任务
version版本格式为yyyyMMdd如2025年6月6日跨天版本则该值为20250605
op_ts操作时间接收到跨天标记的时间
create_time记录创建时间
update_time记录创建时间
scheduled是否已跨天1为已跨天0为未跨天
hudi_collect_build_b12.tb_app_flink_job_config表Flink 任务配置
id主键
nameFlink任务名称
status逻辑删除状态y为正常n为删除
application_idflink任务对应的yarn任务的application id
hudi_collect_build_b12.tb_app_hudi_job_config表Hudi表原生配置
id主键
source_tasks读取Pulsar消息的Flink算子并行度
write_tasks写Hudi表的Flink算子并行度
keep_file_version保留数据文件版本
keep_commit_version保留时间线版本
hudi_collect_build_b12.tb_app_hudi_sync_state表同步、压缩任务运行状态
id主键由flink_job_id和alias使用短横线连接而成格式为flink_job_id-alias
message_id最新消费到的Pulsar消息的message id
source_start_time同步任务启动时间
source_receive_time同步任务最新接收到消息队列消息的时间
source_checkpoint_time同步任务对应Flink任务最近一次执行checkpoint的时间这个时间每15分钟更新一次可以用来判断flink任务是否还在运行
source_publish_time同步任务接收到的最新一条消息队列的消息被发布到队列中的时间
source_op_time同步任务接收到的最新一条消息队列的消息在源端产生的时间
source_application_id同步任务对应Flink任务对应的yarn任务的application id
source_cluster同步任务运行的yarn集群
compaction_start_time最近一次压缩任务启动时间
compaction_finish_time最近一次压缩任务完成时间
compaction_application_id压缩任务对应的Flink任务对应的yarn任务的application id
compaction_cluster压缩任务运行所在的yarn集群
hudi_collect_build_b12.tb_app_yarn_job_config表同步、压缩任务对应的yarn任务资源配置
id主键
job_manager_memoryJob Manager内存MB
task_manager_memoryTask Manager内存MB
""";
String hudi = StrUtil.format(
"""
{}
{}
""",
hudiBase,
hudiDatabase
);
}

View File

@@ -0,0 +1,217 @@
package com.lanyuanxiaoyao.service.ai.web.controller;
import cn.hutool.core.io.FileUtil;
import cn.hutool.core.io.IoUtil;
import cn.hutool.core.util.StrUtil;
import cn.hutool.core.util.URLUtil;
import cn.hutool.crypto.SecureUtil;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.lanyuanxiaoyao.service.ai.core.entity.amis.AmisResponse;
import com.lanyuanxiaoyao.service.ai.web.configuration.FileStoreProperties;
import com.lanyuanxiaoyao.service.ai.web.entity.vo.DataFileVO;
import com.lanyuanxiaoyao.service.ai.web.service.DataFileService;
import jakarta.servlet.http.HttpServletResponse;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.channels.FileChannel;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.eclipse.collections.api.list.ImmutableList;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;
/**
* 文件上传接口
*
* @author lanyuanxiaoyao
* @date 2024-11-21
*/
@Slf4j
@RestController
@RequestMapping("/upload")
public class DataFileController {
private final FileStoreProperties fileStoreProperties;
private final DataFileService dataFileService;
private final String uploadFolderPath;
private final String cacheFolderPath;
private final String sliceFolderPath;
public DataFileController(FileStoreProperties fileStoreProperties, DataFileService dataFileService) {
this.fileStoreProperties = fileStoreProperties;
this.dataFileService = dataFileService;
this.uploadFolderPath = fileStoreProperties.getUploadPath();
this.cacheFolderPath = StrUtil.format("{}/cache", uploadFolderPath);
this.sliceFolderPath = StrUtil.format("{}/slice", uploadFolderPath);
}
@PostMapping("")
public AmisResponse<FinishResponse> upload(@RequestParam("file") MultipartFile file) throws IOException {
String filename = file.getOriginalFilename();
Long id = dataFileService.initialDataFile(filename);
String url = StrUtil.format("{}/upload/download/{}", fileStoreProperties.getDownloadPrefix(), id);
byte[] bytes = file.getBytes();
String originMd5 = SecureUtil.md5(new ByteArrayInputStream(bytes));
File targetFile = new File(StrUtil.format("{}/{}", uploadFolderPath, originMd5));
if (targetFile.exists()) {
dataFileService.updateDataFile(id, FileUtil.getAbsolutePath(targetFile), FileUtil.size(targetFile), originMd5, file.getContentType());
return AmisResponse.responseSuccess(new FinishResponse(id, filename, String.valueOf(id), url));
}
File cacheFile = new File(StrUtil.format("{}/{}", cacheFolderPath, id));
cacheFile = FileUtil.writeBytes(bytes, cacheFile);
String targetMd5 = SecureUtil.md5(cacheFile);
if (!StrUtil.equals(originMd5, targetMd5)) {
throw new RuntimeException("文件上传失败,校验不匹配");
}
FileUtil.move(cacheFile, targetFile, true);
dataFileService.updateDataFile(id, FileUtil.getAbsolutePath(targetFile), FileUtil.size(targetFile), targetMd5, file.getContentType());
return AmisResponse.responseSuccess(new FinishResponse(id, filename, String.valueOf(id), url));
}
@GetMapping("/download/{id}")
public void download(@PathVariable("id") Long id, HttpServletResponse response) throws IOException {
DataFileVO dataFile = dataFileService.downloadFile(id);
File targetFile = new File(dataFile.getPath());
response.setHeader("Content-Type", dataFile.getType());
response.setHeader("Access-Control-Expose-Headers", "Content-Disposition");
response.setHeader("Content-Disposition", StrUtil.format("attachment; filename={}", URLUtil.encodeAll(dataFile.getFilename())));
IoUtil.copy(new FileInputStream(targetFile), response.getOutputStream());
}
@PostMapping("/start")
public AmisResponse<StartResponse> start(@RequestBody StartRequest request) {
Long id = dataFileService.initialDataFile(request.filename);
return AmisResponse.responseSuccess(new StartResponse(id.toString()));
}
@PostMapping("/slice")
public AmisResponse<SliceResponse> slice(
@RequestParam("uploadId")
Long uploadId,
@RequestParam("partNumber")
Integer sequence,
@RequestParam("partSize")
Long size,
@RequestParam("file")
MultipartFile file
) throws IOException {
byte[] bytes = file.getBytes();
String md5 = SecureUtil.md5(new ByteArrayInputStream(bytes));
String targetFilename = StrUtil.format("{}-{}", sequence, md5);
String targetFilePath = sliceFilePath(uploadId, targetFilename);
FileUtil.mkParentDirs(targetFilePath);
FileUtil.writeBytes(bytes, targetFilePath);
return AmisResponse.responseSuccess(new SliceResponse(targetFilename));
}
private String sliceFilePath(Long uploadId, String sliceFilename) {
return StrUtil.format("{}/{}/{}", sliceFolderPath, uploadId, sliceFilename);
}
@PostMapping("finish")
public AmisResponse<FinishResponse> finish(@RequestBody FinishRequest request) {
if (request.partList.anySatisfy(part -> !FileUtil.exist(sliceFilePath(request.uploadId, part.eTag)))) {
throw new RuntimeException("文件校验失败,请重新上传");
}
try {
File cacheFile = new File(StrUtil.format("{}/{}", cacheFolderPath, request.uploadId));
FileUtil.mkParentDirs(cacheFile);
if (cacheFile.createNewFile()) {
try (FileOutputStream fos = new FileOutputStream(cacheFile)) {
try (FileChannel fosChannel = fos.getChannel()) {
for (FinishRequest.Part part : request.partList) {
File sliceFile = new File(sliceFilePath(request.uploadId, part.eTag));
try (FileInputStream fis = new FileInputStream(sliceFile)) {
try (FileChannel fisChannel = fis.getChannel()) {
fisChannel.transferTo(0, fisChannel.size(), fosChannel);
}
}
}
}
}
String md5 = SecureUtil.md5(cacheFile);
File targetFile = new File(StrUtil.format("{}/{}", uploadFolderPath, md5));
if (!targetFile.exists()) {
FileUtil.move(cacheFile, targetFile, true);
}
String absolutePath = FileUtil.getAbsolutePath(targetFile);
dataFileService.updateDataFile(
request.uploadId,
absolutePath,
FileUtil.size(targetFile),
SecureUtil.md5(targetFile),
FileUtil.getMimeType(absolutePath)
);
return AmisResponse.responseSuccess(new FinishResponse(
request.uploadId,
request.filename,
request.uploadId.toString(),
StrUtil.format("{}/upload/download/{}", fileStoreProperties.getDownloadPrefix(), request.uploadId)
));
} else {
throw new RuntimeException("合并文件失败");
}
} catch (Throwable throwable) {
throw new RuntimeException(throwable);
} finally {
FileUtil.del(StrUtil.format("{}/{}", cacheFolderPath, request.uploadId));
FileUtil.del(StrUtil.format("{}/{}", sliceFolderPath, request.uploadId));
}
}
@Data
public static final class StartRequest {
private String name;
private String filename;
}
@Data
@NoArgsConstructor
@AllArgsConstructor
public static final class StartResponse {
private String uploadId;
}
@Data
@NoArgsConstructor
@AllArgsConstructor
public static final class SliceResponse {
@JsonProperty("eTag")
private String eTag;
}
@Data
public static final class FinishRequest {
private String filename;
private Long uploadId;
private ImmutableList<Part> partList;
@Data
public static final class Part {
private Integer partNumber;
@JsonProperty("eTag")
private String eTag;
}
}
@Data
@NoArgsConstructor
@AllArgsConstructor
public static final class FinishResponse {
private Long id;
private String filename;
private String value;
private String url;
}
}

View File

@@ -0,0 +1,148 @@
package com.lanyuanxiaoyao.service.ai.web.controller.chat;
import cn.hutool.core.util.StrUtil;
import com.lanyuanxiaoyao.service.ai.web.configuration.Prompts;
import com.lanyuanxiaoyao.service.ai.web.entity.vo.MessageVO;
import com.lanyuanxiaoyao.service.ai.web.tools.ChartTool;
import com.lanyuanxiaoyao.service.ai.web.tools.TableTool;
import com.lanyuanxiaoyao.service.ai.web.tools.YarnTool;
import com.lanyuanxiaoyao.service.configuration.ExecutorProvider;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.Optional;
import java.util.concurrent.TimeoutException;
import org.eclipse.collections.api.list.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.messages.Message;
import org.springframework.ai.chat.messages.UserMessage;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.ai.deepseek.DeepSeekAssistantMessage;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.servlet.mvc.method.annotation.SseEmitter;
/**
* 聊天
*
* @author lanyuanxiaoyao
* @version 20250514
*/
@Controller
@RequestMapping("chat")
public class ChatController {
private static final Logger logger = LoggerFactory.getLogger(ChatController.class);
private final static DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");
private static final String ROLE_ASSISTANT = "assistant";
private static final String ROLE_USER = "user";
private final ChatClient chatClient;
public ChatController(@Qualifier("chat") ChatClient.Builder builder) {
this.chatClient = builder.build();
}
private ChatClient.ChatClientRequestSpec buildRequest(ImmutableList<MessageVO> messages) {
ChatClient.ChatClientRequestSpec spec = chatClient.prompt()
.system(
StrUtil.format("""
你是一名专业的AI运维助手专职负责“Hudi数据同步服务”的平台运维工作。你的核心职责是
1.友好解答:积极、专业地解答用户(通常是平台管理员或用户)关于该平台运维工作的疑问。
2.知识驱动:在解答时,应尽可能通过各种方式(知识库、上下文、外部工具等)全面获取准确知识和数据来支持回答。
3.诚实守界:
对于无法通过已有知识或数据确认的问题,必须明确告知用户你无法解答,切勿捏造信息或提供不确定的答案。
对于与该Hudi数据同步服务平台运维工作无关的问题需委婉拒绝用户并明确说明超出你的职责和能力范围。
对话语言:中文
{}
当前时间为:{}
""",
Prompts.hudi,
LocalDateTime.now().format(formatter)
))
.messages(
messages
.collect(message -> StrUtil.equals(message.getRole(), ROLE_ASSISTANT)
? new AssistantMessage(message.getContent())
: new UserMessage(message.getContent()))
.collect(message -> (Message) message)
.toList()
);
spec.tools(
new TableTool(),
new YarnTool(),
new ChartTool()
);
return spec;
}
@PostMapping("sync")
@ResponseBody
public MessageVO chatSync(
@RequestBody ImmutableList<MessageVO> messages
) {
ChatResponse response = buildRequest(messages)
.call()
.chatResponse();
return toMessage(response);
}
@PostMapping("async")
public SseEmitter chatAsync(
@RequestBody ImmutableList<MessageVO> messages,
HttpServletResponse httpResponse
) {
httpResponse.setHeader("X-Accel-Buffering", "no");
SseEmitter emitter = new SseEmitter(20 * 60 * 1000L);
ExecutorProvider.EXECUTORS.submit(() -> {
buildRequest(messages)
.stream()
.chatResponse()
.subscribe(
response -> {
try {
emitter.send(
SseEmitter.event()
.data(toMessage(response))
.reconnectTime(5 * 1000L)
.build()
);
} catch (IOException e) {
emitter.completeWithError(e);
}
},
emitter::completeWithError,
emitter::complete
);
});
emitter.onTimeout(() -> emitter.completeWithError(new TimeoutException("SseEmitter Timeout")));
return emitter;
}
private MessageVO toMessage(ChatResponse response) {
AssistantMessage message = Optional.ofNullable(response)
.map(ChatResponse::getResult)
.map(Generation::getOutput)
.orElseThrow(() -> new RuntimeException("ChatResponse is null"));
MessageVO vo = new MessageVO();
vo.setRole(ROLE_ASSISTANT);
vo.setContent(message.getText());
if (message instanceof DeepSeekAssistantMessage deepseekMessage) {
vo.setReason(deepseekMessage.getReasoningContent());
}
return vo;
}
}

View File

@@ -0,0 +1,89 @@
package com.lanyuanxiaoyao.service.ai.web.controller.feedback;
import cn.hutool.core.util.ObjectUtil;
import cn.hutool.core.util.StrUtil;
import com.lanyuanxiaoyao.service.ai.core.entity.amis.AmisCrudResponse;
import com.lanyuanxiaoyao.service.ai.core.entity.amis.AmisResponse;
import com.lanyuanxiaoyao.service.ai.web.configuration.FileStoreProperties;
import com.lanyuanxiaoyao.service.ai.web.entity.Feedback;
import com.lanyuanxiaoyao.service.ai.web.service.feedback.FeedbackService;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.eclipse.collections.api.factory.Lists;
import org.eclipse.collections.api.list.ImmutableList;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
@Slf4j
@RestController
@RequestMapping("feedback")
public class FeedbackController {
private final FileStoreProperties fileStoreProperties;
private final FeedbackService feedbackService;
public FeedbackController(FileStoreProperties fileStoreProperties, FeedbackService feedbackService) {
this.fileStoreProperties = fileStoreProperties;
this.feedbackService = feedbackService;
}
@PostMapping("add")
public void add(@RequestBody CreateItem item) {
feedbackService.add(item.source, ObjectUtil.defaultIfNull(item.pictures, Lists.immutable.empty()));
}
@GetMapping("list")
public AmisCrudResponse list() {
return AmisResponse.responseCrudData(feedbackService.list().collect(feedback -> new ListItem(fileStoreProperties, feedback)));
}
@GetMapping("delete")
public void delete(@RequestParam("id") Long id) {
feedbackService.remove(id);
}
@GetMapping("reanalysis")
public void reanalysis(@RequestParam("id") Long id) {
feedbackService.reanalysis(id);
}
@PostMapping("conclude")
public void conclude(@RequestBody ConcludeItem item) {
feedbackService.updateConclusion(item.getId(), item.getConclusion());
}
@Data
public static final class CreateItem {
private String source;
private ImmutableList<Long> pictures;
}
@Data
public static final class ConcludeItem {
private Long id;
private String conclusion;
}
@Data
public static final class ListItem {
private Long id;
private String source;
private ImmutableList<String> pictures;
private Feedback.Status status;
private String analysis;
private String conclusion;
public ListItem(FileStoreProperties fileStoreProperties, Feedback feedback) {
this.id = feedback.getId();
this.source = feedback.getSource();
this.pictures = feedback.getPictureIds()
.collect(id -> StrUtil.format("{}/upload/download/{}", fileStoreProperties.getDownloadPrefix(), id));
this.status = feedback.getStatus();
this.analysis = feedback.getAnalysis();
this.conclusion = feedback.getConclusion();
}
}
}

View File

@@ -0,0 +1,36 @@
package com.lanyuanxiaoyao.service.ai.web.controller.knowledge;
import com.lanyuanxiaoyao.service.ai.core.entity.amis.AmisResponse;
import com.lanyuanxiaoyao.service.ai.web.service.knowledge.GroupService;
import java.util.concurrent.ExecutionException;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
/**
* @author lanyuanxiaoyao
* @version 20250528
*/
@Slf4j
@RestController
@RequestMapping("knowledge/group")
public class GroupController {
private final GroupService groupService;
public GroupController(GroupService groupService) {
this.groupService = groupService;
}
@GetMapping("list")
public AmisResponse<?> list(@RequestParam("knowledge_id") Long knowledgeId) {
return AmisResponse.responseCrudData(groupService.list(knowledgeId));
}
@GetMapping("delete")
public AmisResponse<?> delete(@RequestParam("id") Long id) throws ExecutionException, InterruptedException {
groupService.remove(id);
return AmisResponse.responseSuccess();
}
}

View File

@@ -0,0 +1,138 @@
package com.lanyuanxiaoyao.service.ai.web.controller.knowledge;
import cn.hutool.core.util.StrUtil;
import com.lanyuanxiaoyao.service.ai.core.entity.amis.AmisMapResponse;
import com.lanyuanxiaoyao.service.ai.core.entity.amis.AmisResponse;
import com.lanyuanxiaoyao.service.ai.web.entity.vo.SegmentVO;
import com.lanyuanxiaoyao.service.ai.web.service.EmbeddingService;
import com.lanyuanxiaoyao.service.ai.web.service.knowledge.KnowledgeBaseService;
import java.io.IOException;
import java.util.concurrent.ExecutionException;
import lombok.extern.slf4j.Slf4j;
import org.eclipse.collections.api.factory.Lists;
import org.eclipse.collections.api.list.ImmutableList;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
/**
* @author lanyuanxiaoyao
* @version 20250515
*/
@Slf4j
@RestController
@RequestMapping("knowledge")
public class KnowledgeBaseController {
private final KnowledgeBaseService knowledgeBaseService;
private final EmbeddingService embeddingService;
public KnowledgeBaseController(KnowledgeBaseService knowledgeBaseService, EmbeddingService embeddingService) {
this.knowledgeBaseService = knowledgeBaseService;
this.embeddingService = embeddingService;
}
@PostMapping("add")
public void add(
@RequestParam("name") String name,
@RequestParam("description") String description,
@RequestParam("strategy") String strategy
) throws ExecutionException, InterruptedException {
knowledgeBaseService.add(name, description, strategy);
}
@PostMapping("update_description")
public void updateDescription(
@RequestParam("id") Long id,
@RequestParam("description") String description
) {
knowledgeBaseService.updateDescription(id, description);
}
@GetMapping("name")
public AmisMapResponse name(@RequestParam("id") Long id) {
return AmisResponse.responseMapData()
.setData("name", knowledgeBaseService.getName(id));
}
@GetMapping("list")
public AmisResponse<?> list() {
return AmisResponse.responseCrudData(knowledgeBaseService.list());
}
@GetMapping("delete")
public void delete(@RequestParam("id") Long id) throws ExecutionException, InterruptedException {
knowledgeBaseService.remove(id);
}
@PostMapping("preview_text")
public AmisResponse<?> previewText(
@RequestParam(value = "mode", defaultValue = "NORMAL") String mode,
@RequestParam(value = "type", defaultValue = "text") String type,
@RequestParam(value = "content", required = false) String content,
@RequestParam(value = "files", required = false) String files
) {
if (StrUtil.equals("text", type)) {
return AmisResponse.responseCrudData(
embeddingService.preview(mode, content)
.collect(doc -> {
SegmentVO vo = new SegmentVO();
vo.setId(doc.getId());
vo.setText(doc.getText());
return vo;
})
);
} else if (StrUtil.equals("file", type)) {
return AmisResponse.responseCrudData(
embeddingService.preview(mode, Lists.immutable.of(files.split(",")))
.collect(doc -> {
SegmentVO vo = new SegmentVO();
vo.setId(doc.getId());
vo.setText(doc.getText());
return vo;
})
);
} else {
throw new IllegalArgumentException("Unsupported type: " + type);
}
}
@PostMapping("submit_text")
public void submitText(
@RequestParam("id") Long id,
@RequestParam(value = "mode", defaultValue = "NORMAL") String mode,
@RequestParam(value = "type", defaultValue = "text") String type,
@RequestParam(value = "content", required = false) String content,
@RequestParam(value = "files", required = false) String files
) {
if (StrUtil.equals("text", type)) {
embeddingService.submit(id, mode, content);
} else if (StrUtil.equals("file", type)) {
embeddingService.submit(id, mode, Lists.immutable.of(files.split(",")));
} else {
throw new IllegalArgumentException("Unsupported type: " + type);
}
}
@PostMapping("submit_text_directly")
public void submitDirectly(
@RequestParam("id") Long id,
@RequestParam(value = "name", required = false) String name,
@RequestParam(value = "split_key", defaultValue = "\n\n") String splitKey,
@RequestBody String content
) {
embeddingService.submitDirectly(id, name, Lists.immutable.of(content.split(splitKey)));
}
@PostMapping("query")
public ImmutableList<String> query(
@RequestParam("id") Long id,
@RequestParam(value = "limit", defaultValue = "5") Integer limit,
@RequestParam(value = "threshold", defaultValue = "0.6") Double threshold,
@RequestBody String text
) throws ExecutionException, InterruptedException, IOException {
return knowledgeBaseService.query(id, text, limit, threshold);
}
}

View File

@@ -0,0 +1,36 @@
package com.lanyuanxiaoyao.service.ai.web.controller.knowledge;
import com.lanyuanxiaoyao.service.ai.core.entity.amis.AmisResponse;
import com.lanyuanxiaoyao.service.ai.web.service.knowledge.SegmentService;
import java.util.concurrent.ExecutionException;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
/**
* @author lanyuanxiaoyao
* @version 20250528
*/
@Slf4j
@RestController
@RequestMapping("knowledge/segment")
public class SegmentController {
private final SegmentService segmentService;
public SegmentController(SegmentService segmentService) {
this.segmentService = segmentService;
}
@GetMapping("list")
public AmisResponse<?> list(@RequestParam("knowledge_id") Long knowledgeId, @RequestParam("group_id") Long groupId) throws ExecutionException, InterruptedException {
return AmisResponse.responseCrudData(segmentService.list(knowledgeId, groupId));
}
@GetMapping("delete")
public AmisResponse<?> delete(@RequestParam("knowledge_id") Long knowledgeId, @RequestParam("segment_id") Long segmentId) throws ExecutionException, InterruptedException {
segmentService.remove(knowledgeId, segmentId);
return AmisResponse.responseSuccess();
}
}

View File

@@ -0,0 +1,22 @@
package com.lanyuanxiaoyao.service.ai.web.entity;
import lombok.Data;
import org.eclipse.collections.api.list.ImmutableList;
@Data
public class Feedback {
private Long id;
private String source;
private ImmutableList<Long> pictureIds;
private String analysis;
private String conclusion;
private Status status;
private Long createdTime;
private Long modifiedTime;
public enum Status {
ANALYSIS_PROCESSING,
ANALYSIS_SUCCESS,
FINISHED,
}
}

View File

@@ -0,0 +1,16 @@
package com.lanyuanxiaoyao.service.ai.web.entity;
import lombok.Data;
/**
* @author lanyuanxiaoyao
* @version 20250527
*/
@Data
public class Group {
private Long id;
private String name;
private String status;
private Long createdTime;
private Long modifiedTime;
}

View File

@@ -0,0 +1,18 @@
package com.lanyuanxiaoyao.service.ai.web.entity;
import lombok.Data;
/**
* @author lanyuanxiaoyao
* @version 20250522
*/
@Data
public class Knowledge {
private Long id;
private Long vectorSourceId;
private String name;
private String description;
private String strategy;
private Long createdTime;
private Long modifiedTime;
}

View File

@@ -0,0 +1,39 @@
package com.lanyuanxiaoyao.service.ai.web.entity.context;
import java.util.List;
import java.util.Map;
import lombok.Builder;
import lombok.Data;
import org.eclipse.collections.api.factory.Lists;
import org.eclipse.collections.api.factory.Maps;
import org.springframework.ai.document.Document;
/**
* @author lanyuanxiaoyao
* @version 20250523
*/
@Data
@Builder
public class EmbeddingContext {
private Long vectorSourceId;
private Long groupId;
private Config config;
private String content;
private String file;
private String fileFormat;
@Builder.Default
private List<Document> documents = Lists.mutable.empty();
@Builder.Default
private Map<String, Object> metadata = Maps.mutable.empty();
@Data
@Builder
public static final class Config {
@Builder.Default
private SplitStrategy splitStrategy = SplitStrategy.NORMAL;
public enum SplitStrategy {
NORMAL, LLM, QA
}
}
}

View File

@@ -0,0 +1,17 @@
package com.lanyuanxiaoyao.service.ai.web.entity.context;
import com.lanyuanxiaoyao.service.ai.web.entity.Feedback;
import java.util.List;
import lombok.Data;
import org.eclipse.collections.api.factory.Lists;
/**
* @author lanyuanxiaoyao
* @version 20250616
*/
@Data
public class FeedbackContext {
private Feedback feedback;
private String optimizedSource;
private List<String> pictureDescriptions = Lists.mutable.empty();
}

View File

@@ -0,0 +1,17 @@
package com.lanyuanxiaoyao.service.ai.web.entity.vo;
import lombok.Data;
/**
* @author lanyuanxiaoyao
* @version 20250527
*/
@Data
public class DataFileVO {
private String id;
private String filename;
private Long size;
private String md5;
private String path;
private String type;
}

View File

@@ -0,0 +1,22 @@
package com.lanyuanxiaoyao.service.ai.web.entity.vo;
import lombok.Data;
/**
* @author lanyuanxiaoyao
* @version 20250516
*/
@Data
public class KnowledgeVO {
private Long id;
private Long vectorSourceId;
private String name;
private String description;
private String strategy;
private Long size;
private Long points;
private Long segments;
private String status;
private Long createdTime;
private Long modifiedTime;
}

View File

@@ -0,0 +1,14 @@
package com.lanyuanxiaoyao.service.ai.web.entity.vo;
import lombok.Data;
/**
* @author lanyuanxiaoyao
* @version 20250516
*/
@Data
public class MessageVO {
private String role;
private String content;
private String reason;
}

View File

@@ -0,0 +1,13 @@
package com.lanyuanxiaoyao.service.ai.web.entity.vo;
import lombok.Data;
/**
* @author lanyuanxiaoyao
* @version 20250516
*/
@Data
public class SegmentVO {
private String id;
private String text;
}

View File

@@ -0,0 +1,76 @@
package com.lanyuanxiaoyao.service.ai.web.service;
import club.kingon.sql.builder.SqlBuilder;
import com.lanyuanxiaoyao.service.ai.core.configuration.SnowflakeId;
import com.lanyuanxiaoyao.service.ai.web.entity.vo.DataFileVO;
import com.lanyuanxiaoyao.service.common.Constants;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
/**
* @author lanyuanxiaoyao
* @version 20250527
*/
@Service
public class DataFileService {
private static final String DATA_FILE_TABLE_NAME = Constants.DATABASE_NAME + ".service_ai_file";
private final JdbcTemplate template;
public DataFileService(JdbcTemplate template) {
this.template = template;
}
public DataFileVO downloadFile(Long id) {
return template.queryForObject(
SqlBuilder.select("id", "filename", "size", "md5", "path", "type")
.from(DATA_FILE_TABLE_NAME)
.whereEq("id", "?")
.precompileSql(),
(rs, row) -> {
DataFileVO vo = new DataFileVO();
vo.setId(String.valueOf(rs.getLong(1)));
vo.setFilename(rs.getString(2));
vo.setSize(rs.getLong(3));
vo.setMd5(rs.getString(4));
vo.setPath(rs.getString(5));
vo.setType(rs.getString(6));
return vo;
},
id
);
}
@Transactional(rollbackFor = Exception.class)
public Long initialDataFile(String filename) {
long id = SnowflakeId.next();
template.update(
SqlBuilder.insertInto(DATA_FILE_TABLE_NAME, "id", "filename")
.values()
.addValue("?", "?")
.precompileSql(),
id,
filename
);
return id;
}
@Transactional(rollbackFor = Exception.class)
public void updateDataFile(Long id, String path, Long size, String md5, String type) {
template.update(
SqlBuilder.update(DATA_FILE_TABLE_NAME)
.set("size", "?")
.addSet("md5", "?")
.addSet("path", "?")
.addSet("type", "?")
.whereEq("id", "?")
.precompileSql(),
size,
md5,
path,
type,
id
);
}
}

View File

@@ -0,0 +1,134 @@
package com.lanyuanxiaoyao.service.ai.web.service;
import cn.hutool.core.io.FileUtil;
import cn.hutool.core.lang.Pair;
import cn.hutool.core.util.IdUtil;
import cn.hutool.core.util.StrUtil;
import com.lanyuanxiaoyao.service.ai.web.entity.Knowledge;
import com.lanyuanxiaoyao.service.ai.web.entity.context.EmbeddingContext;
import com.lanyuanxiaoyao.service.ai.web.entity.vo.DataFileVO;
import com.lanyuanxiaoyao.service.ai.web.service.knowledge.GroupService;
import com.lanyuanxiaoyao.service.ai.web.service.knowledge.KnowledgeBaseService;
import com.yomahub.liteflow.core.FlowExecutor;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.eclipse.collections.api.factory.Lists;
import org.eclipse.collections.api.list.ImmutableList;
import org.springframework.ai.document.Document;
import org.springframework.stereotype.Service;
/**
* @author lanyuanxiaoyao
* @version 20250522
*/
@Service
public class EmbeddingService {
private final DataFileService dataFileService;
private final FlowExecutor executor;
private final KnowledgeBaseService knowledgeBaseService;
private final GroupService groupService;
private final ExecutorService executors = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
@SuppressWarnings("SpringJavaInjectionPointsAutowiringInspection")
public EmbeddingService(DataFileService dataFileService, FlowExecutor executor, KnowledgeBaseService knowledgeBaseService, GroupService groupService) {
this.dataFileService = dataFileService;
this.executor = executor;
this.knowledgeBaseService = knowledgeBaseService;
this.groupService = groupService;
}
public ImmutableList<Document> preview(String mode, String content) {
if (content.length() > 2000) {
content = content.substring(0, 2000);
}
EmbeddingContext context = EmbeddingContext.builder()
.content(content)
.config(EmbeddingContext.Config.builder()
.splitStrategy(EmbeddingContext.Config.SplitStrategy.valueOf(mode))
.build())
.build();
executor.execute2Resp("embedding_preview", null, context);
return Lists.immutable.ofAll(context.getDocuments());
}
public ImmutableList<Document> preview(String mode, ImmutableList<String> ids) {
DataFileVO vo = dataFileService.downloadFile(Long.parseLong(ids.get(0)));
String content = FileUtil.readString(vo.getPath(), StandardCharsets.UTF_8);
return preview(mode, content);
}
public void submit(Long id, String mode, String content) {
executors.submit(() -> {
Knowledge knowledge = knowledgeBaseService.get(id);
Long groupId = groupService.add(knowledge.getId(), StrUtil.format("文本-{}", IdUtil.nanoId(10)));
EmbeddingContext context = EmbeddingContext.builder()
.vectorSourceId(knowledge.getVectorSourceId())
.groupId(groupId)
.content(content)
.config(EmbeddingContext.Config.builder()
.splitStrategy(EmbeddingContext.Config.SplitStrategy.valueOf(mode))
.build())
.build();
executor.execute2Resp("embedding_submit", null, context);
groupService.finish(groupId);
});
}
public void submit(Long id, String mode, ImmutableList<String> ids) {
executors.submit(() -> {
Knowledge knowledge = knowledgeBaseService.get(id);
List<Pair<Long, DataFileVO>> vos = Lists.mutable.empty();
for (String fileId : ids) {
DataFileVO vo = dataFileService.downloadFile(Long.parseLong(fileId));
Long groupId = groupService.add(id, vo.getFilename());
vos.add(Pair.of(groupId, vo));
}
for (Pair<Long, DataFileVO> pair : vos) {
Long groupId = pair.getKey();
DataFileVO vo = pair.getValue();
EmbeddingContext context = EmbeddingContext.builder()
.vectorSourceId(knowledge.getVectorSourceId())
.groupId(groupId)
.file(vo.getPath())
.fileFormat(vo.getFilename())
.config(EmbeddingContext.Config.builder()
.splitStrategy(EmbeddingContext.Config.SplitStrategy.valueOf(mode))
.build())
.build();
executor.execute2Resp("embedding_submit", null, context);
groupService.finish(groupId);
}
});
}
public void submitDirectly(Long id, String name, ImmutableList<String> contents) {
executors.submit(() -> {
Knowledge knowledge = knowledgeBaseService.get(id);
String groupName = name;
if (StrUtil.isBlank(groupName)) {
groupName = StrUtil.format("外部-{}", IdUtil.nanoId(10));
}
Long groupId = groupService.add(knowledge.getId(), groupName);
EmbeddingContext context = EmbeddingContext.builder()
.vectorSourceId(knowledge.getVectorSourceId())
.groupId(groupId)
.build();
context.setDocuments(
contents
.collect(StrUtil::trim)
.collect(content ->
Document.builder()
.text(content)
.metadata(new HashMap<>())
.build()
)
.toList()
);
executor.execute2Resp("embedding_submit_directly", null, context);
groupService.finish(groupId);
});
}
}

View File

@@ -0,0 +1,160 @@
package com.lanyuanxiaoyao.service.ai.web.service.feedback;
import club.kingon.sql.builder.SqlBuilder;
import cn.hutool.core.lang.Assert;
import cn.hutool.core.util.EnumUtil;
import cn.hutool.core.util.ObjectUtil;
import cn.hutool.core.util.StrUtil;
import com.lanyuanxiaoyao.service.ai.core.configuration.SnowflakeId;
import com.lanyuanxiaoyao.service.ai.web.entity.Feedback;
import com.lanyuanxiaoyao.service.ai.web.entity.context.FeedbackContext;
import com.lanyuanxiaoyao.service.common.Constants;
import com.yomahub.liteflow.core.FlowExecutor;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.eclipse.collections.api.factory.Lists;
import org.eclipse.collections.api.list.ImmutableList;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.RowMapper;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Slf4j
@Service
public class FeedbackService {
public static final String FEEDBACK_TABLE_NAME = Constants.DATABASE_NAME + ".service_ai_feedback";
public static final String[] FEEDBACK_COLUMNS = new String[]{"id", "source", "conclusion", "analysis", "pictures", "status", "created_time", "modified_time"};
private static final RowMapper<Feedback> feedbackMapper = (rs, row) -> {
Feedback feedback = new Feedback();
feedback.setId(rs.getLong(1));
feedback.setSource(rs.getString(2));
feedback.setConclusion(rs.getString(3));
feedback.setAnalysis(rs.getString(4));
feedback.setPictureIds(
StrUtil.isBlank(rs.getString(5))
? Lists.immutable.empty()
: Lists.immutable.ofAll(StrUtil.split(rs.getString(5), ",")).collect(Long::parseLong)
);
feedback.setStatus(EnumUtil.fromString(Feedback.Status.class, rs.getString(6)));
feedback.setCreatedTime(rs.getTimestamp(7).getTime());
feedback.setModifiedTime(rs.getTimestamp(8).getTime());
return feedback;
};
private final JdbcTemplate template;
private final FlowExecutor executor;
@SuppressWarnings("SpringJavaInjectionPointsAutowiringInspection")
public FeedbackService(JdbcTemplate template, FlowExecutor executor) {
this.template = template;
this.executor = executor;
}
@Scheduled(initialDelay = 1, fixedDelay = 1, timeUnit = TimeUnit.MINUTES)
public void analysis() {
List<Feedback> feedbacks = template.query(
SqlBuilder.select(FEEDBACK_COLUMNS)
.from(FEEDBACK_TABLE_NAME)
.whereEq("status", Feedback.Status.ANALYSIS_PROCESSING.name())
.build(),
feedbackMapper
);
for (Feedback feedback : feedbacks) {
FeedbackContext context = new FeedbackContext();
context.setFeedback(feedback);
executor.execute2Resp("feedback_analysis", null, context);
}
}
public Feedback get(Long id) {
return template.queryForObject(
SqlBuilder.select(FEEDBACK_COLUMNS)
.from(FEEDBACK_TABLE_NAME)
.whereEq("id", id)
.build(),
feedbackMapper
);
}
@Transactional(rollbackFor = Exception.class)
public void add(String source, ImmutableList<Long> pictureIds) {
template.update(
SqlBuilder.insertInto(FEEDBACK_TABLE_NAME, "id", "source", "pictures")
.values()
.addValue("?", "?", "?")
.precompileSql(),
SnowflakeId.next(),
source,
ObjectUtil.isEmpty(pictureIds) ? null : pictureIds.makeString(",")
);
}
@Transactional(rollbackFor = Exception.class)
public void updateAnalysis(Long id, String analysis) {
Assert.notNull(id, "ID cannot be null");
template.update(
SqlBuilder.update(FEEDBACK_TABLE_NAME)
.set("analysis", "?")
.whereEq("id", "?")
.precompileSql(),
analysis,
id
);
}
@Transactional(rollbackFor = Exception.class)
public void updateConclusion(Long id, String conclusion) {
Assert.notNull(id, "ID cannot be null");
template.update(
SqlBuilder.update(FEEDBACK_TABLE_NAME)
.set("conclusion", "?")
.whereEq("id", "?")
.precompileSql(),
conclusion,
id
);
updateStatus(id, Feedback.Status.FINISHED);
}
@Transactional(rollbackFor = Exception.class)
public void updateStatus(Long id, Feedback.Status status) {
Assert.notNull(id, "ID cannot be null");
template.update(
SqlBuilder.update(FEEDBACK_TABLE_NAME)
.set("status", "?")
.whereEq("id", "?")
.precompileSql(),
status.name(),
id
);
}
public ImmutableList<Feedback> list() {
return template.query(
SqlBuilder.select(FEEDBACK_COLUMNS)
.from(FEEDBACK_TABLE_NAME)
.orderByDesc("created_time")
.build(),
feedbackMapper
)
.stream()
.collect(Collectors.toCollection(Lists.mutable::empty))
.toImmutable();
}
@Transactional(rollbackFor = Exception.class)
public void remove(Long id) {
template.update(
SqlBuilder.delete(FEEDBACK_TABLE_NAME)
.whereEq("id", id)
.build()
);
}
@Transactional(rollbackFor = Exception.class)
public void reanalysis(Long id) {
updateStatus(id, Feedback.Status.ANALYSIS_PROCESSING);
}
}

View File

@@ -0,0 +1,132 @@
package com.lanyuanxiaoyao.service.ai.web.service.knowledge;
import club.kingon.sql.builder.SqlBuilder;
import club.kingon.sql.builder.entry.Alias;
import club.kingon.sql.builder.entry.Column;
import com.lanyuanxiaoyao.service.ai.core.configuration.SnowflakeId;
import com.lanyuanxiaoyao.service.ai.web.entity.Group;
import com.lanyuanxiaoyao.service.common.Constants;
import io.qdrant.client.ConditionFactory;
import io.qdrant.client.QdrantClient;
import io.qdrant.client.grpc.Points;
import java.util.concurrent.ExecutionException;
import java.util.stream.Collectors;
import org.eclipse.collections.api.factory.Lists;
import org.eclipse.collections.api.list.ImmutableList;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.RowMapper;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
/**
* @author lanyuanxiaoyao
* @version 20250522
*/
@Service
public class GroupService {
public static final String GROUP_TABLE_NAME = Constants.DATABASE_NAME + ".service_ai_group";
private static final RowMapper<Group> groupMapper = (rs, row) -> {
Group vo = new Group();
vo.setId(rs.getLong(1));
vo.setName(rs.getString(2));
vo.setStatus(rs.getString(3));
vo.setCreatedTime(rs.getTimestamp(4).getTime());
vo.setModifiedTime(rs.getTimestamp(5).getTime());
return vo;
};
private final JdbcTemplate template;
private final QdrantClient client;
public GroupService(JdbcTemplate template, VectorStore vectorStore) {
this.template = template;
this.client = (QdrantClient) vectorStore.getNativeClient().orElseThrow();
}
public Group get(Long id) {
return template.queryForObject(
SqlBuilder.select("id", "name", "status", "created_time", "modified_time")
.from(GROUP_TABLE_NAME)
.whereEq("id", id)
.orderByDesc("created_time")
.build(),
groupMapper
);
}
@Transactional(rollbackFor = Exception.class)
public Long add(Long knowledgeId, String name) {
long id = SnowflakeId.next();
template.update(
SqlBuilder.insertInto(GROUP_TABLE_NAME, "id", "knowledge_id", "name", "status")
.values()
.addValue("?", "?", "?", "?")
.precompileSql(),
id,
knowledgeId,
name,
"RUNNING"
);
return id;
}
public ImmutableList<Group> list(Long knowledgeId) {
return template.query(
SqlBuilder.select("id", "name", "status", "created_time", "modified_time")
.from(GROUP_TABLE_NAME)
.whereEq("knowledge_id", knowledgeId)
.orderByDesc("created_time")
.build(),
groupMapper
)
.stream()
.collect(Collectors.toCollection(Lists.mutable::empty))
.toImmutable();
}
@Transactional(rollbackFor = Exception.class)
public void finish(Long groupId) {
template.update(
SqlBuilder.update(GROUP_TABLE_NAME)
.set("status", "FINISHED")
.whereEq("id", groupId)
.build()
);
}
@Transactional(rollbackFor = Exception.class)
public void remove(Long groupId) throws ExecutionException, InterruptedException {
Long vectorSourceId = template.queryForObject(
SqlBuilder.select("k.vector_source_id")
.from(Alias.of(GROUP_TABLE_NAME, "g"), Alias.of(KnowledgeBaseService.KNOWLEDGE_TABLE_NAME, "k"))
.whereEq("g.knowledge_id", Column.as("k.id"))
.andEq("g.id", groupId)
.precompileSql(),
Long.class,
groupId
);
client.deleteAsync(
String.valueOf(vectorSourceId),
Points.Filter.newBuilder()
.addMust(ConditionFactory.matchKeyword("vector_source_id", String.valueOf(vectorSourceId)))
.addMust(ConditionFactory.matchKeyword("group_id", String.valueOf(groupId)))
.build()
).get();
template.update(
SqlBuilder.delete(GROUP_TABLE_NAME)
.whereEq("id", groupId)
.build()
);
}
@Transactional(rollbackFor = Exception.class)
public void removeByKnowledgeId(Long knowledgeId) {
template.update(
SqlBuilder.delete(GROUP_TABLE_NAME)
.whereEq("knowledge_id", "?")
.precompileSql(),
knowledgeId
);
}
}

View File

@@ -0,0 +1,203 @@
package com.lanyuanxiaoyao.service.ai.web.service.knowledge;
import club.kingon.sql.builder.SqlBuilder;
import cn.hutool.core.util.ObjectUtil;
import cn.hutool.core.util.StrUtil;
import com.lanyuanxiaoyao.service.ai.core.configuration.SnowflakeId;
import com.lanyuanxiaoyao.service.ai.web.entity.Knowledge;
import com.lanyuanxiaoyao.service.ai.web.entity.vo.KnowledgeVO;
import com.lanyuanxiaoyao.service.common.Constants;
import io.qdrant.client.QdrantClient;
import io.qdrant.client.grpc.Collections;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.stream.Collectors;
import org.eclipse.collections.api.factory.Lists;
import org.eclipse.collections.api.list.ImmutableList;
import org.springframework.ai.document.Document;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.vectorstore.qdrant.QdrantVectorStore;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.RowMapper;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
/**
* @author lanyuanxiaoyao
* @version 20250522
*/
@Service
public class KnowledgeBaseService {
public static final String KNOWLEDGE_TABLE_NAME = Constants.DATABASE_NAME + ".service_ai_knowledge";
public static final String[] KNOWLEDGE_COLUMNS = new String[]{"id", "vector_source_id", "name", "description", "strategy", "created_time", "modified_time"};
private static final RowMapper<Knowledge> knowledgeMapper = (rs, row) -> {
Knowledge knowledge = new Knowledge();
knowledge.setId(rs.getLong(1));
knowledge.setVectorSourceId(rs.getLong(2));
knowledge.setName(rs.getString(3));
knowledge.setDescription(rs.getString(4));
knowledge.setStrategy(rs.getString(5));
knowledge.setCreatedTime(rs.getTimestamp(6).getTime());
knowledge.setModifiedTime(rs.getTimestamp(7).getTime());
return knowledge;
};
private final JdbcTemplate template;
private final EmbeddingModel model;
private final QdrantClient client;
private final GroupService groupService;
public KnowledgeBaseService(JdbcTemplate template, EmbeddingModel model, VectorStore vectorStore, GroupService groupService) {
this.template = template;
this.model = model;
this.client = (QdrantClient) vectorStore.getNativeClient().orElseThrow();
this.groupService = groupService;
}
public Knowledge get(Long id) {
return template.queryForObject(
SqlBuilder.select(KNOWLEDGE_COLUMNS)
.from(KNOWLEDGE_TABLE_NAME)
.whereEq("id", "?")
.precompileSql(),
knowledgeMapper,
id
);
}
@Transactional(rollbackFor = Exception.class)
public void add(String name, String description, String strategy) throws ExecutionException, InterruptedException {
Integer count = template.queryForObject(
SqlBuilder.select("count(*)")
.from(KNOWLEDGE_TABLE_NAME)
.whereEq("name", "?")
.precompileSql(),
Integer.class,
name
);
if (count > 0) {
throw new RuntimeException("名称已存在");
}
long id = SnowflakeId.next();
long vectorSourceId = SnowflakeId.next();
template.update(
SqlBuilder.insertInto(KNOWLEDGE_TABLE_NAME, "id", "vector_source_id", "name", "description", "strategy")
.values()
.addValue("?", "?", "?", "?", "?")
.precompileSql(),
id,
vectorSourceId,
name,
description,
strategy
);
client.createCollectionAsync(
String.valueOf(vectorSourceId),
Collections.VectorParams.newBuilder()
.setDistance(Collections.Distance.valueOf(strategy))
.setSize(model.dimensions())
.build()
).get();
}
@Transactional(rollbackFor = Exception.class)
public void updateDescription(Long id, String description) {
template.update(
SqlBuilder.update(KNOWLEDGE_TABLE_NAME)
.set("description", "?")
.whereEq("id", "?")
.precompileSql(),
description,
id
);
}
public String getName(Long id) {
return template.queryForObject(
SqlBuilder.select("name")
.from(KNOWLEDGE_TABLE_NAME)
.whereEq("id", id)
.orderByDesc("created_time")
.build(),
String.class
);
}
public ImmutableList<KnowledgeVO> list() {
return template.query(
SqlBuilder.select(KNOWLEDGE_COLUMNS)
.from(KNOWLEDGE_TABLE_NAME)
.orderByDesc("created_time")
.build(),
knowledgeMapper
)
.stream()
.map(knowledge -> {
try {
Collections.CollectionInfo info = client.getCollectionInfoAsync(String.valueOf(knowledge.getVectorSourceId())).get();
KnowledgeVO vo = new KnowledgeVO();
vo.setId(knowledge.getId());
vo.setVectorSourceId(knowledge.getVectorSourceId());
vo.setName(knowledge.getName());
vo.setDescription(knowledge.getDescription());
vo.setPoints(info.getPointsCount());
vo.setSegments(info.getSegmentsCount());
vo.setStatus(info.getStatus().name());
Collections.VectorParams vectorParams = info.getConfig().getParams().getVectorsConfig().getParams();
vo.setStrategy(vectorParams.getDistance().name());
vo.setSize(vectorParams.getSize());
vo.setCreatedTime(vo.getCreatedTime());
vo.setModifiedTime(vo.getModifiedTime());
return vo;
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
})
.collect(Collectors.toCollection(Lists.mutable::empty))
.toImmutable();
}
@Transactional(rollbackFor = Exception.class)
public void remove(Long id) throws ExecutionException, InterruptedException {
Knowledge knowledge = get(id);
if (ObjectUtil.isNull(knowledge)) {
throw new RuntimeException(StrUtil.format("{} 不存在"));
}
template.update(
SqlBuilder.delete(KNOWLEDGE_TABLE_NAME)
.whereEq("id", "?")
.precompileSql(),
knowledge.getId()
);
groupService.removeByKnowledgeId(knowledge.getId());
client.deleteCollectionAsync(String.valueOf(knowledge.getVectorSourceId())).get();
}
public ImmutableList<String> query(
Long id,
String text,
Integer limit,
Double threshold
) throws ExecutionException, InterruptedException {
Knowledge knowledge = get(id);
Boolean exists = client.collectionExistsAsync(String.valueOf(knowledge.getVectorSourceId())).get();
if (!exists) {
throw new RuntimeException(StrUtil.format("{} not exists", id));
}
VectorStore vs = QdrantVectorStore.builder(client, model)
.collectionName(String.valueOf(knowledge.getVectorSourceId()))
.initializeSchema(false)
.build();
List<Document> documents = vs.similaritySearch(
SearchRequest.builder()
.query(text)
.topK(limit)
.similarityThreshold(threshold)
.build()
);
return Lists.immutable.ofAll(documents)
.collect(Document::getText);
}
}

View File

@@ -0,0 +1,67 @@
package com.lanyuanxiaoyao.service.ai.web.service.knowledge;
import com.lanyuanxiaoyao.service.ai.web.entity.Knowledge;
import com.lanyuanxiaoyao.service.ai.web.entity.vo.SegmentVO;
import io.qdrant.client.ConditionFactory;
import io.qdrant.client.QdrantClient;
import io.qdrant.client.grpc.Points;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.stream.Collectors;
import org.eclipse.collections.api.factory.Lists;
import org.eclipse.collections.api.list.ImmutableList;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.stereotype.Service;
/**
* @author lanyuanxiaoyao
* @version 20250528
*/
@Service
public class SegmentService {
private final KnowledgeBaseService knowledgeBaseService;
private final QdrantClient client;
public SegmentService(KnowledgeBaseService knowledgeBaseService, VectorStore vectorStore) {
this.knowledgeBaseService = knowledgeBaseService;
this.client = (QdrantClient) vectorStore.getNativeClient().orElseThrow();
}
public ImmutableList<SegmentVO> list(Long id, Long groupId) throws ExecutionException, InterruptedException {
Knowledge knowledge = knowledgeBaseService.get(id);
Points.ScrollResponse response = client.scrollAsync(
Points.ScrollPoints.newBuilder()
.setCollectionName(String.valueOf(knowledge.getVectorSourceId()))
.setWithPayload(Points.WithPayloadSelector.newBuilder().setEnable(true).build())
.setWithVectors(Points.WithVectorsSelector.newBuilder().setEnable(false).build())
.setFilter(
Points.Filter.newBuilder()
.addMust(ConditionFactory.matchKeyword("group_id", String.valueOf(groupId)))
.build()
)
.build()
)
.get();
return response.getResultList()
.stream()
.collect(Collectors.toCollection(Lists.mutable::empty))
.collect(point -> {
SegmentVO vo = new SegmentVO();
vo.setId(point.getId().getUuid());
vo.setText(point.getPayloadMap().get("doc_content").getStringValue());
return vo;
})
.toImmutable();
}
public void remove(Long knowledgeId, Long segmentId) throws ExecutionException, InterruptedException {
Knowledge knowledge = knowledgeBaseService.get(knowledgeId);
client.deletePayloadAsync(
String.valueOf(knowledgeId),
List.of(String.valueOf(segmentId)),
null,
null,
null
).get();
}
}

View File

@@ -0,0 +1,234 @@
package com.lanyuanxiaoyao.service.ai.web.service.node;
import cn.hutool.core.io.FileUtil;
import cn.hutool.core.lang.Assert;
import cn.hutool.core.util.ObjectUtil;
import cn.hutool.core.util.StrUtil;
import com.lanyuanxiaoyao.service.ai.web.entity.context.EmbeddingContext;
import com.yomahub.liteflow.annotation.LiteflowComponent;
import com.yomahub.liteflow.annotation.LiteflowMethod;
import com.yomahub.liteflow.core.NodeComponent;
import com.yomahub.liteflow.enums.LiteFlowMethodEnum;
import com.yomahub.liteflow.enums.NodeTypeEnum;
import io.qdrant.client.QdrantClient;
import java.nio.charset.Charset;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.document.Document;
import org.springframework.ai.document.DocumentReader;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.reader.ExtractedTextFormatter;
import org.springframework.ai.reader.pdf.PagePdfDocumentReader;
import org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig;
import org.springframework.ai.reader.tika.TikaDocumentReader;
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.vectorstore.qdrant.QdrantVectorStore;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.core.io.PathResource;
/**
* @author lanyuanxiaoyao
* @version 20250523
*/
@Slf4j
@LiteflowComponent
public class EmbeddingNodes {
private final ChatClient.Builder chatClientBuilder;
private final QdrantClient qdrantClient;
private final EmbeddingModel embeddingModel;
public EmbeddingNodes(@Qualifier("chat") ChatClient.Builder builder, VectorStore vectorStore, EmbeddingModel embeddingModel) {
this.chatClientBuilder = builder;
this.qdrantClient = (QdrantClient) vectorStore.getNativeClient().orElseThrow();
this.embeddingModel = embeddingModel;
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS_BOOLEAN, nodeId = "embedding_check_if_file_needed", nodeName = "判断是否需要读取文件", nodeType = NodeTypeEnum.BOOLEAN)
public boolean checkIfFileReadNeeded(NodeComponent node) {
EmbeddingContext context = node.getContextBean(EmbeddingContext.class);
Assert.notNull(context, "EmbeddingContext is null");
if (StrUtil.isNotBlank(context.getFile())) {
Assert.isTrue(FileUtil.exist(context.getFile()), "File [{}] not exist", context.getFile());
return true;
}
Assert.notBlank(context.getContent(), "Contents is empty");
return false;
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS_SWITCH, nodeId = "file_reader_switch", nodeName = "判断文件格式", nodeType = NodeTypeEnum.SWITCH)
public String fileReaderSwitch(NodeComponent node) {
EmbeddingContext context = node.getContextBean(EmbeddingContext.class);
String extName = FileUtil.extName(context.getFileFormat());
return switch (extName.toLowerCase()) {
case "txt", "md", "markdown" -> "txt_file_reader";
case "pdf" -> "pdf_file_reader";
case "doc", "docx", "xls", "xlsx", "ppt", "pptx", "html", "xml", "wps", "et", "dpt" -> "any_file_reader";
default -> throw new IllegalStateException("Unsupported ext: " + extName);
};
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS, nodeId = "txt_file_reader", nodeName = "读取文本文件", nodeType = NodeTypeEnum.COMMON)
public void txtFileReader(NodeComponent node) {
EmbeddingContext context = node.getContextBean(EmbeddingContext.class);
context.setContent(FileUtil.readString(context.getFile(), Charset.defaultCharset()));
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS, nodeId = "pdf_file_reader", nodeName = "读取pdf文件", nodeType = NodeTypeEnum.COMMON)
public void pdfFileReader(NodeComponent node) {
EmbeddingContext context = node.getContextBean(EmbeddingContext.class);
PagePdfDocumentReader reader = new PagePdfDocumentReader(
new PathResource(context.getFile()),
PdfDocumentReaderConfig.builder()
.withPageTopMargin(0)
.withPageExtractedTextFormatter(ExtractedTextFormatter.builder()
.withNumberOfTopTextLinesToDelete(0)
.build())
.build());
context.setContent(readBySpringAiReader(reader));
}
/**
* <a href="https://tika.apache.org/3.1.0/formats.html#Database_formats">Tika支持的格式</a>
*/
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS, nodeId = "any_file_reader", nodeName = "使用Tika尝试读取文件", nodeType = NodeTypeEnum.COMMON)
public void anyFileReader(NodeComponent node) {
EmbeddingContext context = node.getContextBean(EmbeddingContext.class);
context.setContent(readBySpringAiReader(new TikaDocumentReader(new PathResource(context.getFile()))));
}
private String readBySpringAiReader(DocumentReader reader) {
return reader.get()
.stream()
.map(Document::getText)
.collect(Collectors.joining("\n"))
.trim();
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS_SWITCH, nodeId = "split_switch", nodeName = "判断使用什么分段方法", nodeType = NodeTypeEnum.SWITCH)
public String splitSwitch(NodeComponent node) {
EmbeddingContext context = node.getContextBean(EmbeddingContext.class);
return switch (context.getConfig().getSplitStrategy()) {
case NORMAL -> "normal_split";
case LLM -> "llm_split";
case QA -> "qa_split";
};
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS, nodeId = "normal_split", nodeName = "使用普通分段", nodeType = NodeTypeEnum.COMMON)
public void normalSplit(NodeComponent node) {
EmbeddingContext context = node.getContextBean(EmbeddingContext.class);
TokenTextSplitter splitter = new TokenTextSplitter(200, 100, 5, 200, true);
Document document = Document.builder()
.text(context.getContent())
.build();
context.setDocuments(splitter.split(document));
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS, nodeId = "llm_split", nodeName = "使用大模型分段", nodeType = NodeTypeEnum.COMMON)
public void llmSplit(NodeComponent node) {
EmbeddingContext context = node.getContextBean(EmbeddingContext.class);
context.getDocuments().addAll(llmSplit(
"""
请你将用户输入的文本进行语义切分,生成用于知识库检索的文本段。
每个文本段要尽可能多地覆盖用户输入文本的各方面知识和细节,包括但不限于主题、概念、关键信息等。对于关键的数字、理论、细节等,要严格遵循原文,不能进行任何虚构和捏造不存在的知识,确保输出内容准确、真实且全面。
输出格式为纯文本段,分段之间使用“---”作为分割,方便后续使用代码进行切分。
输出文本避免添加markdown格式保持文本格式紧凑。
切分过程中,要注重保持文本的完整性和逻辑性,确保每个文本段都能独立地表达出清晰、准确的信息,以便更好地进行知识库检索。
""",
context.getContent(),
context.getMetadata()
));
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS, nodeId = "qa_split", nodeName = "使用Q/A格式分段", nodeType = NodeTypeEnum.COMMON)
public void qaSplit(NodeComponent node) {
EmbeddingContext context = node.getContextBean(EmbeddingContext.class);
// language=TEXT
context.getDocuments().addAll(llmSplit(
"""
对用户输入的文本,生成多组高质量的问答对。请遵循以下指南:
1. 问题部分:
为同一个主题创建尽可能多的不同表述的问题,确保问题的多样性。
每个问题应考虑用户可能的多种问法,例如:
直接询问(如“什么是...?”)
请求确认(如“是否可以说...?”)
寻求解释(如“请解释一下...的含义。”)
假设性问题(如“如果...会怎样?”)
例子请求(如“能否举个例子说明...?”)
问题应涵盖文本中的关键信息、主要概念和细节,确保不遗漏重要内容。
2. 答案部分:
提供一个全面、信息丰富的答案,涵盖问题的所有可能角度,确保逻辑连贯。
答案应直接基于给定文本,确保准确性和一致性。
包含相关的细节,如日期、名称、职位等具体信息,必要时提供背景信息以增强理解。
3. 格式:
使用"问:"标记问题的开始,问题文本应在一个段落内完成。
使用"答:"标记答案的开始,答案应清晰分段,便于阅读。
问答对之间用“---”分隔,以提高可读性。
4. 内容要求:
确保问答对紧密围绕文本主题,避免偏离主题。
避免添加文本中未提及的信息,确保信息的真实性。
一个问题搭配一个答案,避免一组问答对中同时涉及多个问题。
如果文本信息不足以回答某个方面,可以在答案中说明 "根据给定信息无法确定",并尽量提供相关的上下文。
除了问答对本身,避免输出任何与问答对无关的提示性、引导性、解释性的文本。
格式样例:
问:苹果通常是什么颜色的?
答:红色。
---
问:苹果长在树上还是地上?
答:苹果长在树上。
""",
context.getContent(),
context.getMetadata()
));
}
private List<Document> llmSplit(String prompt, String content, Map<String, Object> metadata) {
ChatClient client = chatClientBuilder.build();
String response = client.prompt()
.system(prompt)
.user(content)
.call()
.content();
Assert.notBlank(response, "LLM response is empty");
// noinspection DataFlowIssue
return Arrays.stream(StrUtil.trim(response).split("---"))
.map(text -> text.replaceAll("(?!^.+) +$", ""))
.map(StrUtil::trim)
.map(text -> Document.builder()
.text(text)
.metadata(Optional.ofNullable(metadata).orElse(new HashMap<>()))
.build())
.toList();
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS, nodeId = "import_vector_source", nodeName = "导入向量库", nodeType = NodeTypeEnum.COMMON)
public void importVectorSource(NodeComponent node) {
EmbeddingContext context = node.getContextBean(EmbeddingContext.class);
Assert.notNull(context.getVectorSourceId(), "VectorSourceId is null");
Assert.notNull(context.getGroupId(), "GroupId is null");
if (ObjectUtil.isNotEmpty(context.getDocuments())) {
VectorStore vs = QdrantVectorStore.builder(qdrantClient, embeddingModel)
.collectionName(String.valueOf(context.getVectorSourceId()))
.build();
for (Document document : context.getDocuments()) {
Map<String, Object> metadata = document.getMetadata();
if (StrUtil.isNotBlank(context.getFileFormat()))
metadata.put("filename", context.getFileFormat());
if (StrUtil.isNotBlank(context.getFile()))
metadata.put("filepath", context.getFile());
metadata.put("group_id", String.valueOf(context.getGroupId()));
metadata.put("vector_source_id", String.valueOf(context.getVectorSourceId()));
}
vs.add(context.getDocuments());
}
}
}

View File

@@ -0,0 +1,203 @@
package com.lanyuanxiaoyao.service.ai.web.service.node;
import cn.hutool.core.lang.Assert;
import cn.hutool.core.util.ObjectUtil;
import cn.hutool.core.util.StrUtil;
import com.lanyuanxiaoyao.service.ai.web.entity.Feedback;
import com.lanyuanxiaoyao.service.ai.web.entity.context.FeedbackContext;
import com.lanyuanxiaoyao.service.ai.web.entity.vo.DataFileVO;
import com.lanyuanxiaoyao.service.ai.web.service.DataFileService;
import com.lanyuanxiaoyao.service.ai.web.service.feedback.FeedbackService;
import com.yomahub.liteflow.annotation.LiteflowComponent;
import com.yomahub.liteflow.annotation.LiteflowMethod;
import com.yomahub.liteflow.core.NodeComponent;
import com.yomahub.liteflow.enums.LiteFlowMethodEnum;
import com.yomahub.liteflow.enums.NodeTypeEnum;
import lombok.extern.slf4j.Slf4j;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.core.io.FileSystemResource;
import org.springframework.util.MimeType;
import org.springframework.util.MimeTypeUtils;
@Slf4j
@LiteflowComponent
public class FeedbackNodes {
private final ChatClient.Builder chatClientBuilder;
private final ChatClient.Builder visualChatClientBuilder;
private final DataFileService dataFileService;
private final FeedbackService feedbackService;
public FeedbackNodes(
@Qualifier("chat") ChatClient.Builder chatClientBuilder,
@Qualifier("visual") ChatClient.Builder visualClientBuilder, DataFileService dataFileService, FeedbackService feedbackService
) {
this.chatClientBuilder = chatClientBuilder;
this.visualChatClientBuilder = visualClientBuilder;
this.dataFileService = dataFileService;
this.feedbackService = feedbackService;
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS_BOOLEAN, nodeId = "feedback_check_if_picture_needed", nodeName = "判断有图片进行识别", nodeType = NodeTypeEnum.BOOLEAN)
public boolean checkIfPictureReadNeeded(NodeComponent node) {
FeedbackContext context = node.getContextBean(FeedbackContext.class);
Feedback feedback = context.getFeedback();
return ObjectUtil.isNotEmpty(feedback.getPictureIds());
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS, nodeId = "image_read", nodeName = "读取图片", nodeType = NodeTypeEnum.COMMON)
public void imageRead(NodeComponent node) {
FeedbackContext context = node.getContextBean(FeedbackContext.class);
Feedback feedback = context.getFeedback();
ChatClient client = visualChatClientBuilder
// language=TEXT
.defaultSystem("""
你是一个专业的OCR解析助手。请严格按以下步骤处理用户上传的图片
1. 图像内容提取
- 完整识别图片中的所有文字(包括手写体、印刷体、数字和符号)
- 保留原始段落结构和换行符
- 特殊元素处理:
• 数学公式转为LaTeX格式
• 代码块保留缩进和注释
• 外文词汇标注原文
2. 表格解析优化
- 识别所有表格区域
- 转换为Markdown表格格式对齐表头与单元格
- 补充缺失的表格线
- 用▲标注合并单元格▲跨3列▲
3. 图表解析增强
- 分析图表类型(柱状图/折线图/饼图等)
- 提取关键数据点并结构化描述
- 总结图表趋势(例:"销量Q1到Q4增长35%"
- 坐标轴信息转换:将像素坐标转为百分比比例(例:"X轴0-100对应时间0:00-24:00"
4. 输出规范
- 按[文本][表格][图表]分区块输出
- 表格/图表区域标注原始位置(例:"[左上区域表格]"
- 模糊内容用[?]标注并给出备选(例:"年收[?]入(可能为'入'或'人'"
- 保持原始数据精度(不四舍五入)
立即开始处理用户图片,无需确认步骤。
""")
.build();
for (Long pictureId : feedback.getPictureIds()) {
DataFileVO file = dataFileService.downloadFile(pictureId);
log.info("Parse picture: {} {}", file.getFilename(), file.getPath());
MimeType type = switch (StrUtil.blankToDefault(file.getType(), "").toLowerCase()) {
case "jpg", "jpeg" -> MimeTypeUtils.IMAGE_JPEG;
default -> MimeTypeUtils.IMAGE_PNG;
};
String content = client.prompt()
.user(spec -> spec
.text("输出图片内容")
.media(type, new FileSystemResource(file.getPath())))
.call()
.content();
log.info("Picture: {}", content);
context.getPictureDescriptions().add(content);
}
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS, nodeId = "optimize_source", nodeName = "报障信息优化", nodeType = NodeTypeEnum.COMMON)
public void optimizeSource(NodeComponent node) {
FeedbackContext context = node.getContextBean(FeedbackContext.class);
Feedback feedback = context.getFeedback();
String optimizedSource = chatClientBuilder.build()
.prompt()
// language=TEXT
.system("""
你是一名专业的IT系统运维工程师对于用户输入的关于系统的报障信息你会严格遵循以下步骤进行处理
1.输入
[故障描述]
(这里是用户遇到的系统故障的详细描述)
[相关截图]
(这里是用户遇到的系统故障相关的截图的文字描述,如果没有相关截图,这里会写“无”;如果有多张图片,图片和图片之间会使用“---”分隔)
2.处理逻辑
解析输入
读取并解析用户提供的故障描述和相关截图描述。
识别关键元素:包括故障类型(如硬件故障、软件错误)、受影响系统组件(如服务器、网络设备)、错误消息、发生时间、重现步骤、影响范围等。
如果截图描述存在,提取关键细节(如错误弹窗文本、系统状态截图),并将其作为辅助证据;如果无截图,则忽略此部分。
分析与重写故障描述
专业化改写使用标准IT术语替换非专业用语例如“电脑死机”改为“系统无响应”“连不上网”改为“网络连接中断”并确保描述符合行业规范。
排除歧义澄清模糊描述如添加具体时间戳、系统版本、IP地址或错误代码移除主观语言如“我觉得”或“可能”并添加必要上下文如操作系统环境、相关服务运行状态
结构化组织:将故障描述重写为逻辑段落,格式包括:
问题概述:简明总结故障本质(例如,“数据库服务异常导致应用无法访问”)。
详细症状描述具体现象包括错误消息、发生频率和影响范围如“影响用户登录功能错误代码500”
重现步骤列出可复现故障的操作序列如“1. 访问URL X2. 触发操作 Y”
相关环境添加系统细节如“运行在Linux Ubuntu 20.04, Java 11环境”
整合截图信息如果截图描述存在将其嵌入重写中作为证据例如“根据截图错误弹窗显示Connection timeout
质量校验
检查重写后的内容是否完整、一致且无歧义:确保所有用户输入细节都被涵盖,添加缺失信息(如建议的故障分类),并验证专业术语的准确性。
如果输入信息不足(如缺少时间戳或系统版本),在输出中添加注释提示用户补充。
3.输出
输出一个专业、结构化的故障报告,格式清晰,可直接用于运维团队诊断。
重写的故障描述,以结构化段落呈现,涵盖问题概述、详细症状、重现步骤和相关环境。
输出将使用中性、客观语言,避免任何个人意见或建议,以确保报告专注于事实描述。
""")
.user(StrUtil.format(
"""
[故障描述]
{}
[相关截图]
{}
""",
feedback.getSource(),
ObjectUtil.isEmpty(context.getPictureDescriptions()) ? "" : StrUtil.join(",", context.getPictureDescriptions())
))
.call()
.content();
if (StrUtil.isBlank(optimizedSource)) {
log.warn("Optimized source is blank");
}
context.setOptimizedSource(StrUtil.trim(optimizedSource));
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS, nodeId = "feedback_suggest", nodeName = "大模型建议", nodeType = NodeTypeEnum.COMMON)
public void suggestFeedback(NodeComponent node) {
FeedbackContext context = node.getContextBean(FeedbackContext.class);
Feedback feedback = context.getFeedback();
ChatClient client = chatClientBuilder.build();
String analysis = client.prompt()
// language=TEXT
.system("""
你是一名专业的IT系统运维工程师。针对用户输入的报障信息严格遵循以下步骤
1. 提取关键要素: 精准识别报障信息中的关键要素(如系统/设备名称、故障现象、错误信息、时间地点、影响范围、用户操作步骤等)。
2. 分析诊断: 基于提取的要素,运用运维专业知识,分析潜在原因,并提出优先级高、可操作性强的排障建议或初步诊断方向。
3. 输出专业意见:
内容要求: 意见必须包含明确的行动步骤如检查XX日志、验证XX配置、重启XX服务、联系XX团队、潜在风险提示及预期效果。
表达要求: 使用清晰、简洁、专业的技术术语,避免使用推测性语言或不确定词汇(如“可能”、“大概”、“也许”)。
格式要求: 直接输出意见内容,避免添加引导语(如“我的建议是”、“根据报障信息”等)。
""")
.user(context.getOptimizedSource())
.call()
.content();
feedback.setAnalysis(StrUtil.trim(analysis));
Assert.notBlank(analysis, "Analysis cannot be blank");
String conclusion = client.prompt()
// language=TEXT
.system("""
你是一名专业的文字编辑。严格按以下要求处理用户输入:
核心逻辑: 用一段话精准总结输入内容的核心信息,确保涵盖所有关键点。禁止虚构、扩展或添加原文不存在的信息。
输出规范: 仅输出一段总结文本。禁止分点、分段、使用Markdown格式如着重、- 或 1. 等列表符号)或添加任何无关引导语(如“总结如下”、“该总结指出”、“本文总结”等)。直接给出总结文本。""")
.user(analysis)
.call()
.content();
feedback.setConclusion(StrUtil.trim(conclusion));
}
@LiteflowMethod(value = LiteFlowMethodEnum.PROCESS, nodeId = "feedback_save", nodeName = "保存分析内容", nodeType = NodeTypeEnum.COMMON)
public void saveFeedback(NodeComponent node) {
FeedbackContext context = node.getContextBean(FeedbackContext.class);
Feedback feedback = context.getFeedback();
feedbackService.updateAnalysis(feedback.getId(), feedback.getAnalysis());
feedbackService.updateConclusion(feedback.getId(), feedback.getConclusion());
feedbackService.updateStatus(feedback.getId(), Feedback.Status.ANALYSIS_SUCCESS);
}
}

View File

@@ -0,0 +1,64 @@
package com.lanyuanxiaoyao.service.ai.web.tools;
import com.lanyuanxiaoyao.service.ai.web.WebApplication;
import lombok.extern.slf4j.Slf4j;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
/**
* 图表工具
*
* @author lanyuanxiaoyao
* @version 20250611
*/
@Slf4j
public class ChartTool {
@Tool(description = """
根据需求生成mermaid图表代码
""")
public String mermaid(
// language=TEXT
@ToolParam(description = """
请按以下结构描述图表需求:
1.图表类型:[必填] 明确指定图表类型,例如:流程图(flowchart TD)、序列图(sequenceDiagram)、类图(classDiagram)、甘特图(gantt)、状态图(stateDiagram-v2)、用户旅程图(journey)、饼图(pie)、思维导图(mindmap)、实体关系图(erDiagram)等。
2.图表标题:[可选] 简述图表的核心主题,例如:“用户登录流程”,“订单处理系统时序交互”,“项目开发计划甘特图”。
3.核心内容:[必填] 描述图表逻辑:
实体清单:列出所有节点/角色/对象(如“用户、支付系统、数据库”)
示例 (流程图)`用户, 登录界面, 认证服务, 数据库, 主界面, 错误提示`
示例 (序列图)`用户 (User), 客户端 (ClientApp), 认证服务器 (AuthServer), 数据库 (DB)`
示例 (甘特图)`需求分析, 设计, 开发, 测试, 部署`
关系描述:说明实体间交互/流程顺序(如“用户提交订单 → 支付系统验证 → 数据库更新记录”)
示例 (流程图)`用户 输入 -> 登录界面 -> 提交 -> 认证服务 -> 验证通过? --> 是 --> 主界面; 否 --> 错误提示`
示例 (序列图)`用户 -> 客户端:输入凭据; 客户端 -> 认证服务器:发送认证请求; 认证服务器 -> 数据库:查询用户信息; 数据库 --> 认证服务器:返回结果; 认证服务器 --> 客户端:返回认证结果; 客户端 --> 用户:显示结果`
示例 (甘特图)`需求分析 开始于 2023-10-01, 持续 5天 设计 开始于 需求分析结束, 持续 7天 ...`
示例(饼图):`成功15.25; 失败20.22`
关键分支:如有条件判断,说明条件及路径(如“验证失败 → 显示错误信息”)
示例 (流程图)`认证服务 --> |验证通过| 主界面; 认证服务 --> |验证失败| 错误提示`
4.样式规范:[选填] 指定细节要求:
主题指定预定义主题例如default, forest, dark, neutral]` (默认是 default)
颜色:实体颜色映射(如“用户节点用#3498db”
形状:特殊节点形状(如“决策节点用菱形”)
布局:指定图表方向,例如:从上到下 (TB/TD) 从左到右 (LR) 从右到左 (RL) 从下到上 (BT)]` (默认通常是 TB/TD)
其他:箭头类型(实线/虚线)、注释文本等
""") String request
) {
log.info("Enter method: mermaid[request]. request:{}", request);
ChatClient.Builder builder = WebApplication.getBean("chat", ChatClient.Builder.class);
ChatClient client = builder.build();
return client.prompt()
// language=TEXT
.system("""
你是一位专业的Mermaid图表专家精通各类图表语法流程图、序列图、类图、甘特图、状态图、饼图等并能根据需求生成清晰、准确、符合最佳实践的Mermaid代码。
请根据我提供的详细描述为我生成可直接用于渲染的Mermaid图表代码。确保代码语法正确、布局合理、易于理解。
输出要求:
仅输出纯净的、可立即渲染的 Mermaid 代码。
不要包含任何解释性文字、Markdown 标记(除了代码块标识)或额外的说明。
代码格式清晰,使用适当的缩进(如果适用)。
确保代码语法完全符合 Mermaid 官方文档规范。
""")
.user(request)
.call()
.content();
}
}

View File

@@ -0,0 +1,40 @@
package com.lanyuanxiaoyao.service.ai.web.tools;
import cn.hutool.core.util.ObjectUtil;
import cn.hutool.core.util.StrUtil;
import com.lanyuanxiaoyao.service.ai.web.WebApplication;
import com.lanyuanxiaoyao.service.forest.service.KnowledgeService;
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
/**
* @author lanyuanxiaoyao
* @version 20250606
*/
public class KnowledgeTool {
private final Long knowledgeId;
public KnowledgeTool(Long knowledgeId) {
this.knowledgeId = knowledgeId;
}
@Tool(description = """
从知识库中检索相关外部知识
""")
public String queryKnowledge(
@ToolParam(description = """
精炼准确的知识库关键词,使用逗号分隔,尽可能全面地覆盖需要获取的知识
""")
String query
) {
KnowledgeService knowledgeService = WebApplication.getBean(KnowledgeService.class);
var documents = knowledgeService.query(knowledgeId, query, 10, 0.5);
if (ObjectUtil.isNotEmpty(documents)) {
return StrUtil.format("""
以下是与用户问题有关的外部知识,优先结合该知识回答用户的提问:
{}
""", documents.makeString("\n"));
}
return "";
}
}

View File

@@ -0,0 +1,103 @@
package com.lanyuanxiaoyao.service.ai.web.tools;
import cn.hutool.core.util.StrUtil;
import com.lanyuanxiaoyao.service.ai.web.WebApplication;
import com.lanyuanxiaoyao.service.forest.service.InfoService;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import lombok.extern.slf4j.Slf4j;
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
/**
* @author lanyuanxiaoyao
* @version 20250605
*/
@Slf4j
public class TableTool {
private static final DateTimeFormatter FORMATTER = DateTimeFormatter.ofPattern("yyyyMMdd");
@Tool(description = """
执行SQL语句获取查询结果结果可能是一行或多行行内以逗号分隔字段。
""")
public String executeJdbc(
@ToolParam(description = """
完整的MySQL查询语句禁止使用除select外的任何语句。
""") String sql
) {
log.info("Enter method: executeJdbc[sql]. sql:{}", sql);
InfoService infoService = WebApplication.getBean(InfoService.class);
String result = infoService.jdbc(sql)
.collect(map -> map.valuesView().makeString(","))
.makeString("\n");
log.info("SQL result: \n{}", result);
return result;
}
@Tool(description = """
查询表Hudi表、Hive表、逻辑表等数量
""")
private String tableCount(
@ToolParam(description = """
查询类型,取值如下
Hudi表hudi
Hive表hive
逻辑表logic
一次调用只能传一个类型,不支持多个类型同时查询
""") String type
) {
log.info("Enter method: tableCount[type]. type:{}", type);
var infoService = WebApplication.getBean(InfoService.class);
return switch (type) {
case "logic" -> StrUtil.format("""
逻辑表共{}张,其中重点表{}张
""", infoService.tableCount(), infoService.tableFocusCount());
case "hive" -> StrUtil.format("""
Hive表共{}张,其中重点表{}张
""", infoService.hiveCount(), infoService.hiveFocusCount());
case "hudi" -> StrUtil.format("""
Hudi表共{}张,其中重点表{}张
""", infoService.hudiCount(), infoService.hudiFocusCount());
default -> throw new IllegalStateException("Unexpected value: " + type);
};
}
@Tool(description = """
查询Hudi表跨天情况
返回结果包含未接收到跨天标记的表数量和接收到跨天标记但未完成跨天的表数量
一张表必须先接收到跨天标记才能完成跨天两者数量均为0才算所有表完成跨天
""")
public String version(
@ToolParam(description = """
查询指定日期的跨天情况格式为yyyyMMdd如2025年6月6日取值为20250606
一次调用只能传一个日期,不支持多个日期同时查询
""")
String date,
@ToolParam(description = """
表类型,取值如下
普通表normal
重点表focus
一次调用只能传一个类型,不支持多个类型同时查询
""")
String type
) {
log.info("Enter method: version[date, type]. date:{},type:{}", date, type);
InfoService infoService = WebApplication.getBean(InfoService.class);
String version = date;
if (StrUtil.isBlank(version)) {
version = LocalDateTime.now().minusDays(1).format(FORMATTER);
} else {
version = LocalDate.parse(version, FORMATTER).minusDays(1).format(FORMATTER);
}
return switch (type) {
case "normal" -> StrUtil.format("""
未接收到跨天标记的表数量:{},接收到跨天标记但未完成跨天的表数量:{}
""", infoService.unReceiveVersionNormalTableCount(version), infoService.unScheduledNormalTableCount(version));
case "focus" -> StrUtil.format("""
未接收到跨天标记的表数量:{},接收到跨天标记但未完成跨天的表数量:{}
""", infoService.unReceiveVersionFocusTableCount(version), infoService.unScheduledFocusTableCount(version));
default -> throw new IllegalStateException("Unexpected value: " + type);
};
}
}

View File

@@ -0,0 +1,79 @@
package com.lanyuanxiaoyao.service.ai.web.tools;
import cn.hutool.core.util.StrUtil;
import com.lanyuanxiaoyao.service.ai.web.WebApplication;
import com.lanyuanxiaoyao.service.configuration.entity.yarn.YarnApplication;
import com.lanyuanxiaoyao.service.configuration.entity.yarn.YarnQueue;
import com.lanyuanxiaoyao.service.configuration.entity.yarn.YarnRootQueue;
import com.lanyuanxiaoyao.service.forest.service.YarnService;
import lombok.extern.slf4j.Slf4j;
import org.eclipse.collections.api.list.ImmutableList;
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
/**
* @author lanyuanxiaoyao
* @version 20250606
*/
@Slf4j
public class YarnTool {
@Tool(description = """
查询yarn集群整体资源情况返回值为资源占用率%
""")
public Double yarnStatus(
@ToolParam(description = """
yarn集群名称取值有b12、b1和a4
一次调用只能查询一个集群,不支持多个集群同时查询
""") String cluster
) {
log.info("Enter method: yarnStatus[cluster]. cluster:{}", cluster);
YarnService yarnService = WebApplication.getBean(YarnService.class);
YarnRootQueue status = yarnService.cluster(cluster);
return (status.getUsedCapacity() * 100.0) / status.getCapacity();
}
@Tool(description = """
查询yarn集群中指定队列的资源情况返回值为资源占用率%
""")
public Double yarnQueueStatus(
@ToolParam(description = """
yarn集群名称取值有b12、b1和a4
一次调用只能查询一个集群,不支持多个集群同时查询
""") String cluster,
@ToolParam(description = """
yarn队列名称
""") String queue
) {
log.info("Enter method: yarnQueueStatus[cluster, queue]. cluster:{},queue:{}", cluster, queue);
YarnService yarnService = WebApplication.getBean(YarnService.class);
YarnQueue status = yarnService.queueDetail(cluster, queue);
return (status.getAbsoluteCapacity() * 100.0) / status.getAbsoluteMaxCapacity();
}
@Tool(description = """
查询指定集群上同步任务或压缩任务的运行情况
""")
public String yarnTaskStatus(
@ToolParam(description = """
yarn集群名称取值有b12、b1和a4
一次调用只能查询一个集群,不支持多个集群同时查询
""") String cluster,
@ToolParam(description = """
查询任务种类,取值如下
同步任务Sync
压缩任务Compaction
一次调用只能传一个类型,不支持多个类型同时查询
""") String type
) {
log.info("Enter method: yarnTaskStatus[cluster, type]. cluster:{},type:{}", cluster, type);
YarnService yarnService = WebApplication.getBean(YarnService.class);
ImmutableList<YarnApplication> applications = yarnService.jobList(cluster).select(app -> StrUtil.isNotBlank(type) && StrUtil.contains(app.getName(), type));
return StrUtil.format(
"""
运行中:{},调度中:{}
""",
applications.count(app -> StrUtil.equals(app.getState(), "RUNNING")),
applications.count(app -> StrUtil.equals(app.getState(), "ACCEPTED"))
);
}
}

View File

@@ -0,0 +1,42 @@
server:
compression:
enabled: true
spring:
application:
name: service-ai-web
profiles:
include: random-port,common,discovery,metrics,forest
mvc:
async:
request-timeout: 3600000
autoconfigure:
exclude: |
org.springframework.ai.model.openai.autoconfigure.OpenAiChatAutoConfiguration,
org.springframework.ai.model.openai.autoconfigure.OpenAiAudioSpeechAutoConfiguration,
org.springframework.ai.model.openai.autoconfigure.OpenAiAudioTranscriptionAutoConfiguration,
org.springframework.ai.model.openai.autoconfigure.OpenAiImageAutoConfiguration,
org.springframework.ai.model.openai.autoconfigure.OpenAiEmbeddingAutoConfiguration,
org.springframework.ai.model.openai.autoconfigure.OpenAiModerationAutoConfiguration,
org.springframework.ai.model.deepseek.autoconfigure.DeepSeekChatAutoConfiguration
ai:
vectorstore:
qdrant:
host: 132.121.206.65
port: 29463
api-key: ENC(0/0UkIKeAvyV17yNqSU3v04wmm8CdWKe4BYSSJa2FuBtK12TcZRJPdwk+ZpYnpISv+KmVTUrrmFBzAYrDR3ysA==)
llm:
base-url: http://132.121.206.65:10086
api-key: ENC(K+Hff9QGC+fcyi510VIDd9CaeK/IN5WBJ9rlkUsHEdDgIidW+stHHJlsK0lLPUXXREha+ToQZqqDXJrqSE+GUKCXklFhelD8bRHFXBIeP/ZzT2cxhzgKUXgjw3S0Qw2R)
chat:
base-url: ${spring.llm.base-url}/v1
model: 'Qwen3/qwen3-1.7b'
visual:
model: 'Qwen2.5/qwen2.5-vl-7b-q4km'
embedding:
model: 'Qwen3/qwen3-embedding-4b'
reranker:
model: 'BGE/beg-reranker-v2'
liteflow:
rule-source: liteflow.xml
print-banner: false
check-node-exists: false

View File

@@ -0,0 +1,37 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE flow PUBLIC "liteflow" "https://liteflow.cc/liteflow.dtd">
<flow>
<chain id="embedding_preview">
SER(
IF(
embedding_check_if_file_needed,
SWITCH(file_reader_switch).TO(
txt_file_reader,
pdf_file_reader
).DEFAULT(any_file_reader)
),
SWITCH(split_switch).TO(
normal_split,
llm_split,
qa_split
)
)
</chain>
<chain id="embedding_submit">
SER(embedding_preview, import_vector_source)
</chain>
<chain id="embedding_submit_directly">
SER(import_vector_source)
</chain>
<chain id="feedback_analysis">
SER(
IF(
feedback_check_if_picture_needed,
image_read
),
optimize_source,
feedback_suggest,
feedback_save
)
</chain>
</flow>

View File

@@ -0,0 +1,70 @@
package com.lanyuanxiaoyao.service.ai.web;
import io.qdrant.client.QdrantClient;
import io.qdrant.client.QdrantGrpcClient;
import java.net.http.HttpClient;
import java.util.List;
import java.util.concurrent.ExecutionException;
import org.springframework.ai.document.Document;
import org.springframework.ai.document.MetadataMode;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.openai.OpenAiEmbeddingModel;
import org.springframework.ai.openai.OpenAiEmbeddingOptions;
import org.springframework.ai.openai.api.OpenAiApi;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.vectorstore.qdrant.QdrantVectorStore;
import org.springframework.http.client.JdkClientHttpRequestFactory;
import org.springframework.http.client.reactive.JdkClientHttpConnector;
import org.springframework.web.client.RestClient;
import org.springframework.web.reactive.function.client.WebClient;
/**
* @author lanyuanxiaoyao
* @version 20250527
*/
public class TestEmbedding {
public static void main(String[] args) throws ExecutionException, InterruptedException {
HttpClient httpClient = HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_1_1)
.build();
EmbeddingModel model = new OpenAiEmbeddingModel(
OpenAiApi.builder()
.baseUrl("http://132.121.206.65:10086")
.apiKey("*XMySqV%>hR&v>>g*NwCs3tpQ5FVMFEF2VHVTj<MYQd$&@$sY7CgqNyea4giJi4")
.webClientBuilder(WebClient.builder().clientConnector(new JdkClientHttpConnector(httpClient)))
.restClientBuilder(RestClient.builder().requestFactory(new JdkClientHttpRequestFactory(httpClient)))
.build(),
MetadataMode.ALL,
OpenAiEmbeddingOptions.builder()
.model("Bge-m3-vllm")
.build()
);
QdrantClient client = new QdrantClient(
QdrantGrpcClient.newBuilder("132.121.206.65", 29463, false)
.withApiKey("jdHyKdp9qxNqCK3c")
.build()
);
/* client.createCollectionAsync(
"1927659521614176256",
Collections.VectorParams.newBuilder()
.setDistance(Collections.Distance.Cosine)
.setSize(1024)
.build()
).get(); */
VectorStore store = QdrantVectorStore.builder(client, model)
.initializeSchema(false)
.collectionName("1929833382422159361")
.build();
List<Document> documents = store.similaritySearch(
SearchRequest.builder()
.query("Hudi是什么")
.topK(5)
.similarityThreshold(0.2)
.build()
);
for (Document document : documents) {
System.out.println(document.getText());
}
}
}

View File

@@ -0,0 +1,70 @@
package com.lanyuanxiaoyao.service.ai.web;
import com.lanyuanxiaoyao.service.ai.core.configuration.WebClientConfiguration;
import com.lanyuanxiaoyao.service.ai.web.configuration.LlmConfiguration;
import com.lanyuanxiaoyao.service.ai.web.configuration.LlmProperties;
import lombok.extern.slf4j.Slf4j;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.core.io.FileSystemResource;
import org.springframework.util.MimeTypeUtils;
/**
* @author lanyuanxiaoyao
* @version 20250526
*/
@Slf4j
public class TestLlm {
public static void main(String[] args) {
LlmConfiguration configuration = new LlmConfiguration();
LlmProperties properties = new LlmProperties();
LlmProperties.ChatProperties chatProperties = new LlmProperties.ChatProperties();
chatProperties.setBaseUrl("http://132.121.206.65:10086");
chatProperties.setApiKey("*XMySqV%>hR&v>>g*NwCs3tpQ5FVMFEF2VHVTj<MYQd$&@$sY7CgqNyea4giJi4");
chatProperties.setModel("Qwen2.5/qwen2.5-vl-7b");
properties.setVisual(chatProperties);
ChatClient client = configuration.visualClientBuilder(
properties,
WebClientConfiguration.generateWebClientBuilder(),
WebClientConfiguration.generateRestClientBuilder()
).build();
String content = client.prompt()
// language=TEXT
.system("""
你是一个专业的OCR解析助手。请严格按以下步骤处理用户上传的图片
1. 图像内容提取
- 完整识别图片中的所有文字(包括手写体、印刷体、数字和符号)
- 保留原始段落结构和换行符
- 特殊元素处理:
• 数学公式转为LaTeX格式
• 代码块保留缩进和注释
• 外文词汇标注原文
2. 表格解析优化
- 识别所有表格区域
- 转换为Markdown表格格式对齐表头与单元格
- 补充缺失的表格线
- 用▲标注合并单元格▲跨3列▲
3. 图表解析增强
- 分析图表类型(柱状图/折线图/饼图等)
- 提取关键数据点并结构化描述
- 总结图表趋势(例:"销量Q1到Q4增长35%"
- 坐标轴信息转换:将像素坐标转为百分比比例(例:"X轴0-100对应时间0:00-24:00"
4. 输出规范
- 按[文本][表格][图表]分区块输出
- 表格/图表区域标注原始位置(例:"[左上区域表格]"
- 模糊内容用[?]标注并给出备选(例:"年收[?]入(可能为'入'或'人'"
- 保持原始数据精度(不四舍五入)
立即开始处理用户图片,无需确认步骤。
""")
.user(spec -> spec
// language=TEXT
.text("输出图片内容")
.media(MimeTypeUtils.IMAGE_PNG, new FileSystemResource("/Users/lanyuanxiaoyao/Downloads/图片_002_07.png")))
.call()
.content();
log.info(content);
}
}

View File

@@ -0,0 +1,99 @@
package com.lanyuanxiaoyao.service.ai.web;
import cn.hutool.http.HttpResponse;
import cn.hutool.http.HttpUtil;
/**
* @author lanyuanxiaoyao
* @version 20250605
*/
public class UploadDirectly {
public static void main(String[] args) {
try (HttpResponse response = HttpUtil.createPost("http://132.126.207.130:35690/hudi_services/ai_knowledge/knowledge/submit_text_directly?id=1930517588999553024&name=Hadoop")
.basicAuth("AxhEbscwsJDbYMH2", "cYxg3b4PtWoVD5SjFayWxtnSVsjzRsg4")
.contentType("plain/text")
// language=TEXT
.body("""
什么是Apache Hadoop的核心组件 Hadoop由哪几个主要部分构成
Apache Hadoop的核心组件包括三部分
HDFS (Hadoop Distributed File System)负责分布式存储将数据分割为多个块默认128MB或256MB并在集群节点间分布存储通过多副本机制实现高容错性。
YARN (Yet Another Resource Negotiator)作为资源管理器分配集群的计算资源CPU、内存支持多种数据处理框架运行。
MapReduce分布式计算模型通过Map任务分解并行处理和Reduce结果聚合两阶段处理海量数据。
三者协作流程为HDFS存储数据 → YARN分配资源 → MapReduce执行计算。
Hadoop适合哪些应用场景能否举例说明
Hadoop的典型应用场景包括
大数据分析处理PB级结构化/非结构化数据,如日志分析(用户行为追踪)、社交媒体数据挖掘。
数据湖构建以HDFS为基础存储混合格式数据如传感器数据、文本
ETL流程通过Hive、Pig等工具清洗转换数据构建数据仓库。
机器学习利用Mahout或Spark MLlib训练分布式模型。
案例百度、淘宝使用Hadoop处理每日TB级用户数据。
Hadoop与Spark的主要区别是什么两者如何协同工作
答:核心区别如下:
维度 Hadoop MapReduce Apache Spark
计算模型 基于磁盘的批处理 基于内存的迭代/流处理
延迟 高(分钟级) 低(毫秒级)
适用场景 离线分析、ETL 实时分析、机器学习
成本 低(依赖廉价磁盘) 高(依赖大内存)
协同机制:
Spark可运行在YARN上共享Hadoop集群资源。
HDFS为Spark提供存储底座Spark替代MapReduce作为计算引擎例如批处理用MapReduce实时分析用Spark
Hadoop的优势和局限性有哪些
答:
优势:
高容错性:数据多副本机制,节点故障时自动恢复。
横向扩展:支持数千节点线性扩容。
低成本:可在廉价硬件上部署。
批处理优化:适合海量离线数据处理(如日志分析)。
局限性:
高延迟MapReduce依赖磁盘I/O无法实时处理。
小文件效率低大量小文件增加NameNode负担。
编程复杂需手动编写MapReduce代码。
安全缺陷:缺乏存储/网络级加密。
Hadoop的历史发展中有哪些关键里程碑
答:关键里程碑包括:
2002年起源于Doug Cutting的Apache Nutch项目。
2004年受Google GFS/MapReduce论文启发开始实现类似技术。
2006年HDFS和MapReduce从Nutch独立为Hadoop项目。
2008年成为Apache顶级项目同年以209秒完成1TB数据排序910节点集群创世界纪录。
2012年Hadoop 2.0发布引入YARN取代MapReduce的资源管理角色。
2014年Spark成为默认执行引擎。
2017年Hadoop 3.0支持纠删码降低存储开销。
Hadoop生态系统包含哪些重要工具
答:生态系统分为四层:
数据接入Flume日志收集、Sqoop关系数据库同步、Kafka消息队列
数据存储HBase列式数据库、HDFS核心存储
计算引擎:
批处理MapReduce、Tez
交互式HiveSQL查询、Pig脚本
实时Spark Streaming、Storm
机器学习Mahout、Spark MLlib。
管理运维Ambari监控、Oozie工作流调度、Kerberos认证
是否可以说Hadoop是实时数据处理框架
不能。Hadoop的核心组件MapReduce设计为批处理模型依赖磁盘I/O导致高延迟分钟级。实时处理需依赖其生态系统中的其他工具
Spark Streaming支持毫秒级流处理。
Storm专攻实时计算。
因此Hadoop本身并非实时框架但可通过集成扩展实时能力。
Hadoop如何处理节点故障
答:通过双重容错机制:
HDFS多副本每个数据块默认存储3个副本可配置当节点故障时自动从其他节点读取副本数据。
任务自动重启MapReduce或YARN管理的任务失败时YARN会将其重新分配到健康节点执行确保任务完成。
""")
.execute()) {
System.out.println(response.getStatus());
}
}
}

View File

@@ -7,23 +7,6 @@
<springProperty scope="context" name="LOGGING_PARENT" source="logging.parent"/>
<springProperty scope="context" name="APP_NAME" source="spring.application.name"/>
<appender name="Loki" class="com.github.loki4j.logback.Loki4jAppender">
<metricsEnabled>true</metricsEnabled>
<http class="com.github.loki4j.logback.ApacheHttpSender">
<url>${LOKI_PUSH_URL:-http://localhost/loki/api/v1/push}</url>
</http>
<format>
<label>
<pattern>app=${APP_NAME:-none},host=${HOSTNAME:-none},level=%level</pattern>
<readMarkers>true</readMarkers>
</label>
<message>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %p [${HOSTNAME}] [%t] %logger #@# %m%n%wEx</pattern>
</message>
<sortByTime>true</sortByTime>
</format>
</appender>
<appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %clr(%5p) %clr([${HOSTNAME}]){yellow} %clr([%t]){magenta} %clr(%logger{40}){cyan} #@# %m%n%wEx</pattern>
@@ -45,7 +28,6 @@
<logger name="com.netflix.discovery.shared.resolver.aws.ConfigClusterResolver" level="WARN"/>
<root level="INFO">
<appender-ref ref="Loki"/>
<appender-ref ref="Console"/>
<appender-ref ref="RollingFile"/>
</root>

View File

@@ -161,10 +161,17 @@ deploy:
- "service"
source-jar: service-monitor-1.0.0-SNAPSHOT.jar
replicas: 1
service-ai-chat:
service-ai-web:
order: 6
groups:
- "ai"
source-jar: service-ai-chat-1.0.0-SNAPSHOT.jar
source-jar: service-ai-web-1.0.0-SNAPSHOT.jar
jdk: "jdk17"
replicas: 1
arguments:
"[file-store.download-prefix]": 'http://132.126.207.130:35690/hudi_services/service_ai_web'
"[file-store.upload-path]": ${deploy.runtime.data-path}/knowledge
"[spring.datasource.url]": ${deploy.runtime.database.config.url}
"[spring.datasource.username]": ${deploy.runtime.database.config.username}
"[spring.datasource.password]": ${deploy.runtime.database.config.password}
"[spring.datasource.driver-class-name]": ${deploy.runtime.database.config.driver-class-name}

View File

@@ -7,23 +7,6 @@
<springProperty scope="context" name="LOGGING_PARENT" source="logging.parent"/>
<springProperty scope="context" name="APP_NAME" source="spring.application.name"/>
<appender name="Loki" class="com.github.loki4j.logback.Loki4jAppender">
<metricsEnabled>true</metricsEnabled>
<http class="com.github.loki4j.logback.ApacheHttpSender">
<url>${LOKI_PUSH_URL:-http://localhost/loki/api/v1/push}</url>
</http>
<format>
<label>
<pattern>app=${APP_NAME:-none},host=${HOSTNAME:-none},level=%level</pattern>
<readMarkers>true</readMarkers>
</label>
<message>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %p [${HOSTNAME}] [%t] %logger #@# %m%n%wEx</pattern>
</message>
<sortByTime>true</sortByTime>
</format>
</appender>
<appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %clr(%5p) %clr([${HOSTNAME}]){yellow} %clr([%t]){magenta} %clr(%logger{40}){cyan} #@# %m%n%wEx</pattern>
@@ -45,7 +28,6 @@
<logger name="com.netflix.discovery.shared.resolver.aws.ConfigClusterResolver" level="WARN"/>
<root level="INFO">
<appender-ref ref="Loki"/>
<appender-ref ref="Console"/>
<appender-ref ref="RollingFile"/>
</root>

View File

@@ -21,7 +21,6 @@ public class SecurityConfig extends WebSecurityConfigurerAdapter {
private final SecurityProperties securityProperties;
@SuppressWarnings("SpringJavaInjectionPointsAutowiringInspection")
public SecurityConfig(SecurityProperties securityProperties) {
this.securityProperties = securityProperties;
}
@@ -36,6 +35,8 @@ public class SecurityConfig extends WebSecurityConfigurerAdapter {
.and()
.csrf()
.disable()
.cors()
.disable()
.formLogin()
.disable();
}

View File

@@ -21,6 +21,6 @@ public class TestSecurityDecrypt {
config.setStringOutputType("base64");
encryptor.setConfig(config);
System.out.println(encryptor.decrypt("GXKnbq1LS11U2HaONspvH+D/TkIx13aWTaokdkzaF7HSvq6Z0Rv1+JUWFnYopVXu"));
System.out.println(encryptor.encrypt("jdHyKdp9qxNqCK3c"));
}
}

View File

@@ -7,23 +7,6 @@
<springProperty scope="context" name="LOGGING_PARENT" source="logging.parent"/>
<springProperty scope="context" name="APP_NAME" source="spring.application.name"/>
<appender name="Loki" class="com.github.loki4j.logback.Loki4jAppender">
<metricsEnabled>true</metricsEnabled>
<http class="com.github.loki4j.logback.ApacheHttpSender">
<url>${LOKI_PUSH_URL:-http://localhost/loki/api/v1/push}</url>
</http>
<format>
<label>
<pattern>app=${APP_NAME:-none},host=${HOSTNAME:-none},level=%level</pattern>
<readMarkers>true</readMarkers>
</label>
<message>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %p [${HOSTNAME}] [%t] %logger #@# %m%n%wEx</pattern>
</message>
<sortByTime>true</sortByTime>
</format>
</appender>
<appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %clr(%5p) %clr([${HOSTNAME}]){yellow} %clr([%t]){magenta} %clr(%logger{40}){cyan} #@# %m%n%wEx</pattern>
@@ -42,7 +25,6 @@
</appender>
<root level="INFO">
<appender-ref ref="Loki"/>
<appender-ref ref="Console"/>
<appender-ref ref="RollingFile"/>
</root>

View File

@@ -1,43 +1,4 @@
<configuration>
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
<url>${loki_push_url:- }</url>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [${HOSTNAME}] %-5level ${PID:- } --- [%t] %-40.40logger{39} #@# : %m%n</pattern>
</encoder>
<label>
<name>app</name>
<value>service-task</value>
</label>
<label>
<name>host</name>
<value>${HOSTNAME}</value>
</label>
<label>
<name>task_id</name>
<value>${task_id:- }</value>
</label>
<label>
<name>hdfs</name>
<value>${hdfs:- }</value>
</label>
<label>
<name>key</name>
<value>${key:- }</value>
</label>
<label>
<name>app_id</name>
<value>${_APP_ID:- }</value>
</label>
<label>
<name>container_id</name>
<value>${CONTAINER_ID:- }</value>
</label>
<logLevelLabel>level</logLevelLabel>
</appender>
<appender name="File" class="ch.qos.logback.core.FileAppender">
<file>run.log</file>
<append>false</append>

Some files were not shown because too many files have changed in this diff Show More