Compare commits
17 Commits
e67ec24dfd
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
| d3fd6de965 | |||
| 277c14d2e8 | |||
| 5cc347589b | |||
| 89ffc88082 | |||
| 675235f5b3 | |||
| a490b2642c | |||
| 1306dd5971 | |||
| e0c6ed1638 | |||
| 0dd7aa221c | |||
| 3b2b368db2 | |||
| a578c0b7ac | |||
| 78063b9e07 | |||
| edbdeec90d | |||
| a5c0b67360 | |||
| 82b09614d3 | |||
| c90e1c98be | |||
| 229f17bfee |
171
README.md
171
README.md
@@ -1,6 +1,6 @@
|
||||
# lyxy-document
|
||||
|
||||
统一文档解析工具 - 将 DOCX、XLS、XLSX、PPTX、PDF、HTML/URL 转换为 Markdown
|
||||
统一文档解析工具 - 将 DOC、DOCX、XLS、XLSX、PPT、PPTX、PDF、HTML/URL 转换为 Markdown
|
||||
|
||||
## 项目概述
|
||||
|
||||
@@ -10,24 +10,27 @@
|
||||
|
||||
- 使用 uv 运行脚本和测试,禁用主机 Python
|
||||
- 依赖管理:使用 `uv run --with` 按需加载依赖
|
||||
- 快速获取建议:使用 `-a/--advice` 参数查看执行命令
|
||||
- 自启动机制:脚本自动检测依赖并用正确的 uv 命令执行
|
||||
|
||||
## 项目架构
|
||||
|
||||
```
|
||||
scripts/
|
||||
├── lyxy_document_reader.py # CLI 入口
|
||||
├── lyxy_document_reader.py # CLI 入口(自启动)
|
||||
├── bootstrap.py # 实际执行模块
|
||||
├── config.py # 配置(含 DEPENDENCIES 依赖配置)
|
||||
├── core/ # 核心模块
|
||||
│ ├── parser.py # 解析调度
|
||||
│ ├── advice_generator.py # --advice 执行建议生成器
|
||||
│ ├── advice_generator.py # 依赖检测和配置生成
|
||||
│ ├── markdown.py # Markdown 工具
|
||||
│ └── exceptions.py # 异常定义
|
||||
├── readers/ # 格式阅读器
|
||||
│ ├── base.py # Reader 基类
|
||||
│ ├── doc/ # DOC 解析器(旧格式)
|
||||
│ ├── docx/ # DOCX 解析器
|
||||
│ ├── xls/ # XLS 解析器(旧格式)
|
||||
│ ├── xlsx/ # XLSX 解析器
|
||||
│ ├── ppt/ # PPT 解析器(旧格式)
|
||||
│ ├── pptx/ # PPTX 解析器
|
||||
│ ├── pdf/ # PDF 解析器
|
||||
│ └── html/ # HTML/URL 解析器
|
||||
@@ -94,9 +97,9 @@ DEPENDENCIES = {
|
||||
}
|
||||
```
|
||||
|
||||
### --advice 生成机制
|
||||
### 自启动机制
|
||||
|
||||
`--advice` 参数根据文件扩展名识别类型,检测当前平台,从 `config.DEPENDENCIES` 读取对应配置,生成 `uv run --with` 和 `pip install` 命令。
|
||||
入口脚本根据文件扩展名识别类型,检测当前平台,从 `config.DEPENDENCIES` 读取对应配置,自动生成并执行正确的 `uv run --with` 命令。
|
||||
|
||||
## 快速开始
|
||||
|
||||
@@ -105,17 +108,15 @@ DEPENDENCIES = {
|
||||
首先验证项目可以正常运行:
|
||||
|
||||
```bash
|
||||
# 测试 --advice 功能(无需额外依赖)
|
||||
uv run python scripts/lyxy_document_reader.py test.pdf --advice
|
||||
# 测试解析功能(自动检测依赖并执行)
|
||||
python scripts/lyxy_document_reader.py "https://example.com"
|
||||
```
|
||||
|
||||
### 运行基础测试
|
||||
|
||||
```bash
|
||||
# 运行 CLI 测试(验证项目基本功能)
|
||||
uv run \
|
||||
--with pytest \
|
||||
pytest tests/test_cli/test_main.py::TestCLIAdviceOption -v
|
||||
# 使用 run_tests.py 自动加载依赖并运行测试
|
||||
python run_tests.py cli -v
|
||||
```
|
||||
|
||||
## 开发指南
|
||||
@@ -135,131 +136,45 @@ uv run \
|
||||
|
||||
### 如何测试
|
||||
|
||||
项目包含完整的测试套件,覆盖 CLI 和所有 Reader 实现。根据测试类型使用对应的 `uv run --with` 命令。
|
||||
项目包含完整的测试套件,覆盖 CLI、核心模块、工具函数和所有 Reader 实现。使用 `run_tests.py` 自动加载对应依赖并运行测试。
|
||||
|
||||
#### 测试目录结构
|
||||
- tests/test_cli/ - CLI 功能测试
|
||||
- tests/test_core/ - 核心模块测试(markdown, parser, advice_generator)
|
||||
- tests/test_readers/ - 各格式 Reader 测试
|
||||
- tests/test_utils/ - 工具函数测试(file_detection, encoding_detection)
|
||||
|
||||
#### run_tests.py 使用说明
|
||||
|
||||
#### 运行所有测试
|
||||
```bash
|
||||
uv run \
|
||||
--with pytest \
|
||||
--with pytest-cov \
|
||||
pytest
|
||||
```
|
||||
# 查看帮助
|
||||
python run_tests.py -h
|
||||
|
||||
#### 测试 DOCX reader
|
||||
```bash
|
||||
uv run \
|
||||
--with pytest \
|
||||
--with docling \
|
||||
--with "unstructured[docx]" \
|
||||
--with "markitdown[docx]" \
|
||||
--with pypandoc-binary \
|
||||
--with python-docx \
|
||||
--with markdownify \
|
||||
pytest tests/test_readers/test_docx/
|
||||
```
|
||||
# 运行所有测试
|
||||
python run_tests.py all
|
||||
|
||||
#### 测试 XLSX reader
|
||||
```bash
|
||||
uv run \
|
||||
--with pytest \
|
||||
--with docling \
|
||||
--with "unstructured[xlsx]" \
|
||||
--with "markitdown[xlsx]" \
|
||||
--with pandas \
|
||||
--with tabulate \
|
||||
pytest tests/test_readers/test_xlsx/
|
||||
```
|
||||
# 运行特定类型测试
|
||||
python run_tests.py pdf
|
||||
python run_tests.py docx
|
||||
python run_tests.py xlsx
|
||||
python run_tests.py pptx
|
||||
python run_tests.py html
|
||||
python run_tests.py xls
|
||||
python run_tests.py doc
|
||||
python run_tests.py ppt
|
||||
python run_tests.py cli
|
||||
python run_tests.py core
|
||||
python run_tests.py utils
|
||||
|
||||
#### 测试 PPTX reader
|
||||
```bash
|
||||
uv run \
|
||||
--with pytest \
|
||||
--with docling \
|
||||
--with "unstructured[pptx]" \
|
||||
--with "markitdown[pptx]" \
|
||||
--with python-pptx \
|
||||
--with markdownify \
|
||||
pytest tests/test_readers/test_pptx/
|
||||
```
|
||||
|
||||
#### 测试 PDF reader
|
||||
```bash
|
||||
# 默认命令(macOS ARM、Linux、Windows)
|
||||
uv run \
|
||||
--with pytest \
|
||||
--with docling \
|
||||
--with "unstructured[pdf]" \
|
||||
--with "markitdown[pdf]" \
|
||||
--with pypdf \
|
||||
--with markdownify \
|
||||
--with reportlab \
|
||||
pytest tests/test_readers/test_pdf/
|
||||
|
||||
# macOS x86_64 (Intel) 特殊命令
|
||||
uv run \
|
||||
--python 3.12 \
|
||||
--with pytest \
|
||||
--with "docling==2.40.0" \
|
||||
--with "docling-parse==4.0.0" \
|
||||
--with "numpy<2" \
|
||||
--with "markitdown[pdf]" \
|
||||
--with pypdf \
|
||||
--with markdownify \
|
||||
--with reportlab \
|
||||
pytest tests/test_readers/test_pdf/
|
||||
```
|
||||
|
||||
#### 测试 HTML reader
|
||||
```bash
|
||||
uv run \
|
||||
--with pytest \
|
||||
--with trafilatura \
|
||||
--with domscribe \
|
||||
--with markitdown \
|
||||
--with html2text \
|
||||
--with beautifulsoup4 \
|
||||
--with httpx \
|
||||
--with chardet \
|
||||
pytest tests/test_readers/test_html/
|
||||
```
|
||||
|
||||
#### 测试 XLS reader(旧格式,使用静态文件)
|
||||
```bash
|
||||
uv run \
|
||||
--with pytest \
|
||||
--with "unstructured[xlsx]" \
|
||||
--with "markitdown[xls]" \
|
||||
--with pandas \
|
||||
--with tabulate \
|
||||
--with xlrd \
|
||||
pytest tests/test_readers/test_xls/
|
||||
```
|
||||
|
||||
#### 运行特定测试文件或方法
|
||||
```bash
|
||||
# 运行特定测试文件(CLI 测试无需额外依赖)
|
||||
uv run \
|
||||
--with pytest \
|
||||
pytest tests/test_cli/test_main.py
|
||||
|
||||
# 仅运行 --advice 相关测试(不需要额外依赖)
|
||||
uv run \
|
||||
--with pytest \
|
||||
pytest tests/test_cli/test_main.py::TestCLIAdviceOption
|
||||
|
||||
# 运行特定测试类或方法
|
||||
uv run \
|
||||
--with pytest \
|
||||
--with docling \
|
||||
pytest tests/test_cli/test_main.py::TestCLIDefaultOutput::test_default_output_docx
|
||||
# 透传 pytest 参数
|
||||
python run_tests.py pdf -v
|
||||
python run_tests.py pdf --cov=scripts
|
||||
python run_tests.py pdf tests/test_readers/test_pdf/test_docling_pdf.py
|
||||
```
|
||||
|
||||
#### 查看测试覆盖率
|
||||
```bash
|
||||
uv run \
|
||||
--with pytest \
|
||||
--with pytest-cov \
|
||||
pytest --cov=scripts --cov-report=term-missing
|
||||
python run_tests.py all --with pytest-cov --cov=scripts --cov-report=term-missing
|
||||
```
|
||||
|
||||
### 代码规范
|
||||
|
||||
48
SKILL.md
48
SKILL.md
@@ -1,32 +1,28 @@
|
||||
---
|
||||
name: lyxy-document-reader
|
||||
description: 统一文档解析工具 - 将 DOCX、XLSX、PPTX、PDF、HTML/URL 转换为 Markdown。支持全文输出、字数统计、行数统计、标题提取、章节提取、正则搜索。当用户要求"读取/解析/打开文档"、上传 .docx/.xlsx/.pptx/.pdf/.html 文件、或提供 URL 时使用。
|
||||
description: 统一文档解析工具 - 将 DOC、DOCX、XLS、XLSX、PPT、PPTX、PDF、HTML/URL 转换为 Markdown。支持全文输出、字数统计、行数统计、标题提取、章节提取、正则搜索。当用户要求"读取/解析/打开文档"、上传 .doc/.docx/.xls/.xlsx/.ppt/.pptx/.pdf/.html 文件、或提供 URL 时使用。
|
||||
license: MIT
|
||||
compatibility: Requires Python 3.11+。优先使用 lyxy-runner-python skill,次选 uv run --with,降级到主机 Python。
|
||||
compatibility: Requires Python 3.11+。脚本自启动,自动检测依赖并使用 uv 执行。
|
||||
---
|
||||
|
||||
# 统一文档解析 Skill
|
||||
|
||||
## 🔴 重要:执行路径优先级(必须遵守)
|
||||
## 推荐用法
|
||||
|
||||
### 执行路径选择(按优先级顺序)
|
||||
1. **lyxy-runner-python skill(首选)** - 自动管理依赖
|
||||
2. **uv run --with** - 按需加载依赖
|
||||
3. **主机 Python + pip install** - 手动安装依赖
|
||||
直接运行脚本即可,它会自动检测文件类型、当前平台,并用正确的 uv 命令执行:
|
||||
|
||||
### 第一步:获取执行建议
|
||||
```bash
|
||||
PYTHONPATH=. uv run --with pyarmor python scripts/lyxy_document_reader.py --advice <文件路径或URL>
|
||||
python scripts/lyxy_document_reader.py <文件路径或URL>
|
||||
```
|
||||
这会输出准确的执行命令,包含所需的依赖配置。
|
||||
|
||||
*也可以使用:`python scripts/lyxy_document_reader.py --advice <文件路径或URL>`*
|
||||
|
||||
## Purpose
|
||||
|
||||
**支持格式**
|
||||
- DOC(Word 旧格式)
|
||||
- DOCX(Word 文档)
|
||||
- XLS(Excel 旧格式)
|
||||
- XLSX(Excel 表格)
|
||||
- PPT(PowerPoint 旧格式)
|
||||
- PPTX(PowerPoint 演示文稿)
|
||||
- PDF(PDF 文档,支持 OCR)
|
||||
- HTML / URL(网页内容)
|
||||
@@ -42,15 +38,14 @@ PYTHONPATH=. uv run --with pyarmor python scripts/lyxy_document_reader.py --advi
|
||||
|
||||
### 触发词
|
||||
- 中文:"读取/解析/打开 文档/Word/Excel/PPT/PDF/网页"
|
||||
- 英文:"read/parse/extract document/docx/xlsx/pptx/pdf/html"
|
||||
- 文件扩展名:`.docx`、`.xlsx`、`.pptx`、`.pdf`、`.html`、`.htm`
|
||||
- 英文:"read/parse/extract document/doc/docx/xls/xlsx/ppt/pptx/pdf/html"
|
||||
- 文件扩展名:`.doc`、`.docx`、`.xls`、`.xlsx`、`.ppt`、`.pptx`、`.pdf`、`.html`、`.htm`
|
||||
- URL:`http://`、`https://`
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| 参数 | 说明 |
|
||||
|------|------|
|
||||
| `-a/--advice` | 仅显示执行建议(**必须先运行此命令**) |
|
||||
| (无) | 输出完整 Markdown |
|
||||
| `-c/--count` | 字数统计 |
|
||||
| `-l/--lines` | 行数统计 |
|
||||
@@ -62,33 +57,28 @@ PYTHONPATH=. uv run --with pyarmor python scripts/lyxy_document_reader.py --advi
|
||||
## 参数使用示例
|
||||
|
||||
```bash
|
||||
# 获取执行建议
|
||||
PYTHONPATH=. uv run --with pyarmor python scripts/lyxy_document_reader.py --advice document.docx
|
||||
|
||||
# 读取全文
|
||||
PYTHONPATH=. uv run --with pyarmor python scripts/lyxy_document_reader.py document.docx
|
||||
# 读取全文(自动检测依赖)
|
||||
python scripts/lyxy_document_reader.py document.docx
|
||||
|
||||
# 统计字数
|
||||
PYTHONPATH=. uv run --with pyarmor python scripts/lyxy_document_reader.py document.docx -c
|
||||
python scripts/lyxy_document_reader.py document.docx -c
|
||||
|
||||
# 提取标题
|
||||
PYTHONPATH=. uv run --with pyarmor python scripts/lyxy_document_reader.py document.docx -t
|
||||
python scripts/lyxy_document_reader.py document.docx -t
|
||||
|
||||
# 提取指定章节
|
||||
PYTHONPATH=. uv run --with pyarmor python scripts/lyxy_document_reader.py document.docx -tc "第三章"
|
||||
python scripts/lyxy_document_reader.py document.docx -tc "第三章"
|
||||
|
||||
# 搜索内容
|
||||
PYTHONPATH=. uv run --with pyarmor python scripts/lyxy_document_reader.py document.docx -s "关键词"
|
||||
python scripts/lyxy_document_reader.py document.docx -s "关键词"
|
||||
|
||||
# 正则搜索
|
||||
PYTHONPATH=. uv run --with pyarmor python scripts/lyxy_document_reader.py document.docx -s "\d{4}-\d{2}-\d{2}"
|
||||
python scripts/lyxy_document_reader.py document.docx -s "\d{4}-\d{2}-\d{2}"
|
||||
|
||||
# 指定搜索上下文行数
|
||||
PYTHONPATH=. uv run --with pyarmor python scripts/lyxy_document_reader.py document.docx -s "关键词" -n 5
|
||||
python scripts/lyxy_document_reader.py document.docx -s "关键词" -n 5
|
||||
```
|
||||
|
||||
*也可以使用纯 python 命令:`python scripts/lyxy_document_reader.py ...`*
|
||||
|
||||
## 错误处理
|
||||
|
||||
| 错误 | 原因 | 解决 |
|
||||
@@ -98,4 +88,4 @@ PYTHONPATH=. uv run --with pyarmor python scripts/lyxy_document_reader.py docume
|
||||
| 所有解析方法均失败 | 所有解析器失败 | 检查文件是否损坏 |
|
||||
| 错误: 无效的正则表达式 | 正则语法错误 | 检查正则语法 |
|
||||
| 错误: 未找到匹配 | 搜索无结果 | 检查搜索词或正则 |
|
||||
| ModuleNotFoundError | 缺少依赖 | 使用 --advice 获取正确的依赖命令 |
|
||||
| ModuleNotFoundError | 缺少依赖 | 脚本会自动检测并安装依赖 |
|
||||
|
||||
86
build.py
86
build.py
@@ -58,27 +58,17 @@ def get_git_user_info() -> tuple[str, str]:
|
||||
try:
|
||||
name = get_git_config("user.name")
|
||||
except subprocess.CalledProcessError:
|
||||
print("""
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
错误: git user.name 未设置
|
||||
|
||||
请先配置 git 用户名:
|
||||
git config --global user.name "Your Name"
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
""")
|
||||
print("错误: git user.name 未设置")
|
||||
print("请先配置 git 用户名:")
|
||||
print(' git config --global user.name "Your Name"')
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
email = get_git_config("user.email")
|
||||
except subprocess.CalledProcessError:
|
||||
print("""
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
错误: git user.email 未设置
|
||||
|
||||
请先配置 git 邮箱:
|
||||
git config --global user.email "your@email.com"
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
""")
|
||||
print("错误: git user.email 未设置")
|
||||
print("请先配置 git 邮箱:")
|
||||
print(' git config --global user.email "your@email.com"')
|
||||
sys.exit(1)
|
||||
|
||||
return name, email
|
||||
@@ -92,10 +82,8 @@ def clean_and_create_build_dir(build_dir: str) -> None:
|
||||
build_dir: 构建目录路径
|
||||
"""
|
||||
if os.path.exists(build_dir):
|
||||
print(f"清理旧构建目录: {build_dir}")
|
||||
shutil.rmtree(build_dir)
|
||||
os.makedirs(build_dir)
|
||||
print(f"创建构建目录: {build_dir}")
|
||||
|
||||
|
||||
def copy_skill_md(source_path: str, target_dir: str, version: str, author: str) -> None:
|
||||
@@ -203,8 +191,6 @@ def copy_skill_md(source_path: str, target_dir: str, version: str, author: str)
|
||||
with open(target_path, "w", encoding="utf-8") as f:
|
||||
f.write(new_content)
|
||||
|
||||
print(f"生成: {target_path} (version: {version}, author: {author})")
|
||||
|
||||
|
||||
def obfuscate_scripts_dir(source_dir: str, target_dir: str) -> None:
|
||||
"""
|
||||
@@ -218,16 +204,9 @@ def obfuscate_scripts_dir(source_dir: str, target_dir: str) -> None:
|
||||
try:
|
||||
__import__("pyarmor")
|
||||
except ImportError:
|
||||
print("""
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
错误: PyArmor 未安装
|
||||
|
||||
请使用以下命令:
|
||||
|
||||
uv run --with pyarmor python build.py
|
||||
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
""")
|
||||
print("错误: PyArmor 未安装")
|
||||
print("请使用以下命令:")
|
||||
print(" uv run --with pyarmor python build.py")
|
||||
sys.exit(1)
|
||||
|
||||
# 临时目录
|
||||
@@ -246,8 +225,6 @@ def obfuscate_scripts_dir(source_dir: str, target_dir: str) -> None:
|
||||
source_dir
|
||||
]
|
||||
|
||||
print(f" 执行: {' '.join(cmd)}")
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
@@ -256,38 +233,49 @@ def obfuscate_scripts_dir(source_dir: str, target_dir: str) -> None:
|
||||
text=True
|
||||
)
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"\nPyArmor 混淆失败:")
|
||||
print("错误: PyArmor 混淆失败")
|
||||
print(f" 返回码: {e.returncode}")
|
||||
print(f" 标准输出: {e.stdout}")
|
||||
print(f" 错误输出: {e.stderr}")
|
||||
sys.exit(1)
|
||||
|
||||
# 移动混淆后的文件到最终位置
|
||||
scripts_dst_dir = os.path.join(target_dir, "scripts")
|
||||
pyarmor_runtime_dir = None
|
||||
|
||||
# 先移动 scripts 目录
|
||||
for item in os.listdir(temp_dir):
|
||||
src = os.path.join(temp_dir, item)
|
||||
dst = os.path.join(target_dir, item)
|
||||
if item == "scripts":
|
||||
dst = os.path.join(target_dir, item)
|
||||
if os.path.exists(dst):
|
||||
if os.path.isdir(dst):
|
||||
shutil.rmtree(dst)
|
||||
else:
|
||||
os.remove(dst)
|
||||
shutil.move(src, dst)
|
||||
elif item.startswith("pyarmor_runtime"):
|
||||
pyarmor_runtime_dir = item
|
||||
|
||||
# 再移动 pyarmor_runtime 到 scripts 内部
|
||||
if pyarmor_runtime_dir:
|
||||
src = os.path.join(temp_dir, pyarmor_runtime_dir)
|
||||
dst = os.path.join(scripts_dst_dir, pyarmor_runtime_dir)
|
||||
if os.path.exists(dst):
|
||||
if os.path.isdir(dst):
|
||||
shutil.rmtree(dst)
|
||||
else:
|
||||
os.remove(dst)
|
||||
|
||||
shutil.move(src, dst)
|
||||
|
||||
# 清理临时目录
|
||||
os.rmdir(temp_dir)
|
||||
|
||||
print(" 混淆完成")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""
|
||||
主函数:执行完整的混淆打包流程
|
||||
"""
|
||||
print("=" * 60)
|
||||
print("Skill 打包构建 (混淆模式)")
|
||||
print("=" * 60)
|
||||
|
||||
# 路径配置
|
||||
project_root = os.path.dirname(os.path.abspath(__file__))
|
||||
@@ -297,37 +285,19 @@ def main() -> None:
|
||||
|
||||
# 生成版本号
|
||||
version = generate_timestamp()
|
||||
print(f"版本号: {version}")
|
||||
|
||||
# 读取 git 用户信息
|
||||
git_name, git_email = get_git_user_info()
|
||||
author = f"{git_name} <{git_email}>"
|
||||
print(f"作者: {author}")
|
||||
print()
|
||||
|
||||
# 清理并创建 build 目录
|
||||
clean_and_create_build_dir(build_dir)
|
||||
print()
|
||||
|
||||
# 复制 SKILL.md(动态注入元数据)
|
||||
copy_skill_md(skill_md_path, build_dir, version, author)
|
||||
print()
|
||||
|
||||
# 混淆代码
|
||||
print("────────────────────────────────────────")
|
||||
print(" 使用 PyArmor 混淆代码 (Normal Mode)")
|
||||
print("────────────────────────────────────────")
|
||||
obfuscate_scripts_dir(scripts_source_dir, build_dir)
|
||||
print()
|
||||
|
||||
# 完成信息
|
||||
print("=" * 60)
|
||||
print("构建完成!")
|
||||
print(f"版本号: {version}")
|
||||
print(f"作者: {author}")
|
||||
print("混淆模式: 已生成 .pyx 和 pyarmor_runtime")
|
||||
print(f"输出目录: {build_dir}")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
15
build.sh
Executable file
15
build.sh
Executable file
@@ -0,0 +1,15 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# 混淆构建脚本
|
||||
#
|
||||
# 使用方式:
|
||||
# ./build.sh
|
||||
#
|
||||
|
||||
set -e
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
echo ">>> 构建"
|
||||
uv run --with pyarmor python build.py
|
||||
echo ">>> 完成"
|
||||
126
docs/upgrade-deps-prompt.md
Normal file
126
docs/upgrade-deps-prompt.md
Normal file
@@ -0,0 +1,126 @@
|
||||
# 依赖版本优化流程提示词
|
||||
|
||||
## 任务概述
|
||||
|
||||
重新梳理 `scripts/config.py` 中 `DEPENDENCIES` 的版本号和 python 版本。
|
||||
|
||||
## 核心原则
|
||||
|
||||
1. **default 的 python 版本始终使用 None**,即默认 python 版本
|
||||
2. **实在需要指定 python 版本时**,才在具体的系统依赖(如 Darwin-x86_64)中指定 python 版本,而不是改 default 中的 python 版本
|
||||
3. **dependencies 中的依赖都需要指定版本**
|
||||
- 以当前时间点的最新版本指定
|
||||
- 如果最新版本无法满足,才在指定系统依赖中探索能运行的最新依赖版本号
|
||||
|
||||
## 推荐流程
|
||||
|
||||
### 阶段 1:规范梳理
|
||||
|
||||
1. 确定需要检查的依赖列表
|
||||
2. 确定版本查询方法(如 PyPI JSON API)
|
||||
3. 确定测试验证流程
|
||||
|
||||
### 阶段 2:版本探索(实现阶段)
|
||||
|
||||
1. **先移除所有特定当前平台配置**,只保留 default
|
||||
2. **default 配置使用最新版本作为标杆**
|
||||
3. **逐个文件类型测试**
|
||||
- 先测试 default 配置
|
||||
- 若 default 失败,再添加特定平台配置并探索可运行的最新版本
|
||||
4. **所有依赖(无论之前是否指定版本)都重新探索**
|
||||
|
||||
### 阶段 3:配置更新
|
||||
|
||||
1. 修改 `default.python = None`
|
||||
2. 更新所有依赖到指定版本
|
||||
3. 保留/调整特定平台的特殊配置
|
||||
|
||||
## 关键文件
|
||||
|
||||
- `scripts/config.py` - DEPENDENCIES 配置
|
||||
- `run_tests.py` - 测试运行器(包含 TEST_FIXTURE_DEPENDENCIES)
|
||||
- `openspec/changes/` - OpenSpec 变更目录
|
||||
|
||||
## 常用 PyPI 版本查询
|
||||
|
||||
使用 Python 查询 PyPI 最新版本:
|
||||
|
||||
```python
|
||||
import json
|
||||
import urllib.request
|
||||
|
||||
def get_latest_version(package):
|
||||
try:
|
||||
url = f'https://pypi.org/pypi/{package}/json'
|
||||
with urllib.request.urlopen(url, timeout=15) as f:
|
||||
data = json.load(f)
|
||||
return data['info']['version']
|
||||
except Exception as e:
|
||||
return f'error: {e}'
|
||||
```
|
||||
|
||||
## Windows 平台验证结果(2026-03-19)
|
||||
|
||||
### 验证状态:✓ 通过
|
||||
|
||||
**测试结果:**
|
||||
- 所有文件类型的依赖安装测试通过
|
||||
- 功能测试:295/303 测试通过
|
||||
- 8 个已知失败(中文字符 PDF 解析、LibreOffice DOCX)
|
||||
|
||||
**验证通过的依赖:**
|
||||
- PDF: docling 2.80.0, unstructured[pdf] 0.21.5, markitdown[pdf] 0.1.5, pypdf 6.9.0, markdownify 1.2.2
|
||||
- DOCX: docling 2.80.0, unstructured[docx] 0.21.5, markitdown[docx] 0.1.5, pypandoc-binary 1.17, python-docx 1.2.0
|
||||
- XLSX: docling 2.80.0, unstructured[xlsx] 0.21.5, markitdown[xlsx] 0.1.5, pandas 3.0.1, openpyxl 3.1.5
|
||||
- PPTX: docling 2.80.0, unstructured[pptx] 0.21.5, markitdown[pptx] 0.1.5, python-pptx 1.0.2
|
||||
- HTML: trafilatura 2.0.0, domscribe 0.1.3, markitdown 0.1.5, html2text 2025.4.15, beautifulsoup4 4.14.3
|
||||
- XLS: unstructured[xlsx] 0.21.5, markitdown[xls] 0.1.5, pandas 3.0.1, xlrd 2.0.2, olefile 0.47
|
||||
- PPT: docling 2.80.0, unstructured[pptx] 0.21.5, markitdown[pptx] 0.1.5, python-pptx 1.0.2, olefile 0.47
|
||||
|
||||
**已知问题:**
|
||||
1. 中文字符在临时 PDF 生成中显示为 `<!-- image -->`(测试环境字体问题,不影响实际使用)
|
||||
|
||||
## 本次(2026-03-17)的经验总结
|
||||
|
||||
### Darwin-x86_64 平台的已知问题
|
||||
|
||||
1. **torch 无 Darwin-x86_64 wheel**(docling 2.80.0 依赖 torch)
|
||||
- 解决:使用 docling 2.40.0 + docling-parse 4.0.0 + numpy<2
|
||||
2. **onnxruntime 无 Darwin-x86_64 + Python 3.14 wheel**(markitdown 依赖)
|
||||
- 解决:指定 python 3.12
|
||||
3. **pyppeteer 2.0.0 与 selenium 4.41.0 的 urllib3 版本冲突**
|
||||
- 解决:selenium 降级到 4.25.0
|
||||
4. **pandas 3.0.1 与 fixtures 依赖 pandas<3.0.0 冲突**
|
||||
- 解决:特定平台使用 pandas<3.0.0
|
||||
|
||||
### 当前依赖版本列表(截止 2026-03-19,Windows 验证通过)
|
||||
|
||||
| 依赖 | 版本 |
|
||||
|------|------|
|
||||
| docling | 2.80.0 (default) / 2.40.0 (Darwin-x86_64) |
|
||||
| docling-parse | 5.5.0 (default) / 4.0.0 (Darwin-x86_64) |
|
||||
| unstructured[...] | 0.21.5 |
|
||||
| markitdown[...] | 0.1.5 |
|
||||
| pypdf | 6.9.0 |
|
||||
| markdownify | 1.2.2 |
|
||||
| pypandoc-binary | 1.17 |
|
||||
| python-docx | 1.2.0 |
|
||||
| pandas | 3.0.1 (default) / <3.0.0 (Darwin-x86_64) |
|
||||
| tabulate | 0.10.0 |
|
||||
| openpyxl | 3.1.5 |
|
||||
| python-pptx | 1.0.2 |
|
||||
| trafilatura | 2.0.0 |
|
||||
| domscribe | 0.1.3 |
|
||||
| html2text | 2025.4.15 |
|
||||
| beautifulsoup4 | 4.14.3 |
|
||||
| httpx | 0.28.1 |
|
||||
| chardet | 7.1.0 |
|
||||
| pyppeteer | 2.0.0 |
|
||||
| selenium | 4.25.0 (Darwin-x86_64) |
|
||||
| xlrd | 2.0.2 |
|
||||
| olefile | 0.47 |
|
||||
| numpy | <2 (Darwin-x86_64) |
|
||||
|
||||
## 创建 OpenSpec 变更
|
||||
|
||||
使用 `/opsx:new` 或 `/opsx:ff` 创建变更,使用 spec-driven 工作流。
|
||||
@@ -4,22 +4,21 @@ context: |
|
||||
# 项目规范
|
||||
- 语言: 仅中文(交流/注释/文档/代码)
|
||||
- Python: 当前项目始终用uv运行(脚本/临时命令uv run python -c); 禁用主机python/禁主机安装包
|
||||
- 依赖: pyproject.toml声明,使用uv安装
|
||||
- 主机环境: 禁止污染配置,需操作须请求用户
|
||||
- 开发文档: README.md,每次迭代按需更新开发文档; 禁emoji/特殊字符
|
||||
- skill文档: SKILL.md,每次迭代按需更新skill文档
|
||||
- 测试: 所有需求必须设计全面测试
|
||||
- 测试: 所有需求必须设计全面测试,严禁跳过测试,无法进行的测试交用户决策
|
||||
- 任务: 除非用户直接要求,禁止创建git变更任务(push/commit等); git读取允许(status/log/diff等)
|
||||
- 代码: 模块文件150-300行; 错误需自定义异常+清晰信息+位置上下文
|
||||
- 项目阶段: 未上线,无用户,破坏性变更无需迁移说明
|
||||
- Git提交: 仅中文; 格式为"类型: 简短描述",类型可选: feat(新功能)/fix(修复)/refactor(重构)/docs(文档)/style(格式)/test(测试)/chore(构建/工具); 多行描述空行后加详细说明
|
||||
- 提问: 对用户的提问优先使用提问工具而不是文字选项
|
||||
# 项目概述
|
||||
- 目标:统一文档解析工具,将DOCX/XLSX/PPTX/PDF/HTML/URL 转换为 Markdown,面向AI skill使用
|
||||
- 目标:统一文档解析工具,将各种格式的文档转换为 Markdown,面向AI skill使用
|
||||
# 项目目录结构
|
||||
- scripts/: 核心代码目录
|
||||
- tests/: 测试目录
|
||||
- openspec/: 规范文档目录
|
||||
- temp/: 开发临时文件目录
|
||||
- pyproject.toml: 项目配置
|
||||
- README.md: 项目开发文档
|
||||
- SKILL.md: skill文档
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
## Purpose
|
||||
|
||||
CLI 执行建议生成功能,根据文件类型返回 uv 和 python 命令,帮助 AI 快速获取准确的执行建议,无需翻阅文档。
|
||||
CLI 自启动机制,自动检测文件类型、平台和依赖,用正确的 uv 命令执行脚本。
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: 依赖配置结构
|
||||
依赖配置必须同时包含 python 版本要求和依赖包列表,按文件类型和平台组织。
|
||||
依赖配置必须同时包含 python 版本要求和依赖包列表,按文件类型和平台组织,供自启动逻辑内部使用。
|
||||
|
||||
#### Scenario: 配置结构包含 python 和 dependencies
|
||||
- **WHEN** 访问 `config.DEPENDENCIES` 时
|
||||
@@ -19,17 +19,8 @@ CLI 执行建议生成功能,根据文件类型返回 uv 和 python 命令,
|
||||
|
||||
---
|
||||
|
||||
### Requirement: CLI 支持 --advice 参数
|
||||
命令行工具必须支持 `-a/--advice` 参数,当指定该参数时不执行实际解析,仅输出执行建议。
|
||||
|
||||
#### Scenario: 用户指定 --advice 参数
|
||||
- **WHEN** 用户执行 `scripts/lyxy_document_reader.py --advice <input_path>`
|
||||
- **THEN** 工具输出执行建议,不解析文件内容
|
||||
|
||||
---
|
||||
|
||||
### Requirement: 轻量文件类型检测
|
||||
`--advice` 参数必须复用 Reader 实例的 supports 方法识别文件类型,不打开文件。
|
||||
自启动必须复用 Reader 实例的 supports 方法识别文件类型,不打开文件。
|
||||
|
||||
#### Scenario: 复用 Reader 实例
|
||||
- **WHEN** 检测文件类型时
|
||||
@@ -69,72 +60,70 @@ CLI 执行建议生成功能,根据文件类型返回 uv 和 python 命令,
|
||||
|
||||
#### Scenario: 不验证文件存在
|
||||
- **WHEN** 输入路径指向不存在的文件
|
||||
- **THEN** 仍根据 reader.supports() 返回建议,不报错
|
||||
- **THEN** 仍根据 reader.supports() 识别类型,不报错
|
||||
|
||||
---
|
||||
|
||||
### Requirement: 平台检测
|
||||
必须检测当前平台并返回适配的命令。
|
||||
必须检测当前平台并选择适配的依赖配置。
|
||||
|
||||
#### Scenario: 检测平台格式
|
||||
- **WHEN** 工具执行时
|
||||
- **THEN** 返回格式为 `{system}-{machine}`,例如 `Darwin-arm64`、`Linux-x86_64`、`Windows-AMD64`
|
||||
|
||||
#### Scenario: macOS x86_64 PDF 特殊命令
|
||||
#### Scenario: macOS x86_64 PDF 特殊配置
|
||||
- **WHEN** 平台为 `Darwin-x86_64` 且文件类型为 PDF
|
||||
- **THEN** 返回包含 `--python 3.12` 和特定版本依赖的命令
|
||||
- **THEN** 使用包含 `--python 3.12` 和特定版本依赖的配置
|
||||
|
||||
---
|
||||
|
||||
### Requirement: 输出 uv 命令
|
||||
必须输出使用 `uv run --with ...` 格式的命令。
|
||||
### Requirement: 自启动检测
|
||||
脚本必须自动检测文件类型、当前平台和 uv 可用性,如 uv 可用则用正确的 uv 命令启动 bootstrap.py。
|
||||
|
||||
#### Scenario: 检测文件类型
|
||||
- **WHEN** 脚本启动时
|
||||
- **THEN** 复用 Reader 的 supports() 方法识别文件类型
|
||||
- **AND** 不打开文件,仅做轻量检测
|
||||
|
||||
#### Scenario: 检测平台
|
||||
- **WHEN** 脚本启动时
|
||||
- **THEN** 检测当前平台,格式为 `{system}-{machine}`
|
||||
- **AND** 根据平台选择正确的依赖配置
|
||||
|
||||
#### Scenario: 检测 uv 是否可用
|
||||
- **WHEN** 准备自启动前
|
||||
- **THEN** 使用 `shutil.which("uv")` 检测 uv 是否在 PATH 中
|
||||
- **AND** 如果 uv 不可用,降级为直接执行 bootstrap.py
|
||||
|
||||
---
|
||||
|
||||
### Requirement: 自启动执行
|
||||
脚本必须使用 `subprocess.run()` 启动子进程,用正确的 uv 命令启动 bootstrap.py。
|
||||
|
||||
#### Scenario: 生成 uv 命令
|
||||
- **WHEN** 检测到文件类型
|
||||
- **THEN** 输出格式为:`uv run [--python X.Y] --with <dep1> --with <dep2> ... scripts/lyxy_document_reader.py <input_path>`
|
||||
- **WHEN** 脚本确定需要自启动
|
||||
- **THEN** 根据文件类型和平台获取依赖配置
|
||||
- **AND** 生成 `uv run [--python X.Y] --with <dep1> --with <dep2> ... scripts/bootstrap.py <input_path>` 命令
|
||||
- **AND** 目标脚本是 bootstrap.py,不是 lyxy_document_reader.py
|
||||
|
||||
#### Scenario: 自启动设置环境变量
|
||||
- **WHEN** 执行 `subprocess.run()` 自启动
|
||||
- **THEN** 必须设置 `PYTHONPATH=.`
|
||||
- **AND** 不需要设置 `LYXY_IN_UV`(自启动直接调用 bootstrap.py)
|
||||
- **AND** 必须传递退出码给父进程
|
||||
|
||||
#### Scenario: 静默自启动
|
||||
- **WHEN** 脚本执行自启动
|
||||
- **THEN** 不输出任何额外提示信息
|
||||
- **AND** 不干扰正常的 Markdown 输出
|
||||
|
||||
---
|
||||
|
||||
### Requirement: 输出 python 命令
|
||||
必须输出直接使用 python 的命令及 pip 安装命令。
|
||||
### Requirement: 降级执行
|
||||
当 uv 不可用时,脚本必须降级为直接导入并执行 bootstrap.py。
|
||||
|
||||
#### Scenario: 生成 python 命令
|
||||
- **WHEN** 检测到文件类型
|
||||
- **THEN** 输出 python 命令:`python scripts/lyxy_document_reader.py <input_path>`
|
||||
- **AND** 输出 pip 安装命令:`pip install <dep1> <dep2> ...`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: 输出格式规范
|
||||
输出必须包含文件类型、输入路径、平台(如需要)、uv 命令、python 命令和 pip 安装命令。
|
||||
|
||||
#### Scenario: 普通平台输出格式
|
||||
- **WHEN** 平台无特殊配置
|
||||
- **THEN** 输出格式为:
|
||||
```
|
||||
文件类型: <type>
|
||||
输入路径: <input>
|
||||
|
||||
[uv 命令]
|
||||
<uv_command>
|
||||
|
||||
[python 命令]
|
||||
python scripts/lyxy_document_reader.py <input>
|
||||
pip install <deps>
|
||||
```
|
||||
|
||||
#### Scenario: 特殊平台输出格式
|
||||
- **WHEN** 平台有特殊配置
|
||||
- **THEN** 输出格式为:
|
||||
```
|
||||
文件类型: <type>
|
||||
输入路径: <input>
|
||||
平台: <system-machine>
|
||||
|
||||
[uv 命令]
|
||||
<uv_command>
|
||||
|
||||
[python 命令]
|
||||
python scripts/lyxy_document_reader.py <input>
|
||||
pip install <deps>
|
||||
```
|
||||
#### Scenario: uv 不可用时降级
|
||||
- **WHEN** uv 不在 PATH 中
|
||||
- **THEN** 脚本直接导入 bootstrap 模块
|
||||
- **AND** 调用 bootstrap.run_normal() 执行
|
||||
- **AND** 如果缺少依赖,输出正常的 `ModuleNotFoundError`
|
||||
|
||||
61
openspec/specs/doc-reader/spec.md
Normal file
61
openspec/specs/doc-reader/spec.md
Normal file
@@ -0,0 +1,61 @@
|
||||
## Purpose
|
||||
|
||||
DOC 文档解析能力,支持解析 Microsoft Word 97-2003 旧格式文档。
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: DOC 文档解析
|
||||
系统 SHALL 支持解析 .doc 格式文档,使用 LibreOffice 解析器。
|
||||
|
||||
#### Scenario: 使用 LibreOffice 解析器
|
||||
- **WHEN** 解析 DOC 文档
|
||||
- **THEN** 系统使用 LibreOffice soffice 命令行进行解析
|
||||
|
||||
#### Scenario: 成功解析
|
||||
- **WHEN** 解析器成功
|
||||
- **THEN** 系统返回解析结果
|
||||
|
||||
#### Scenario: 解析器失败
|
||||
- **WHEN** 解析器失败
|
||||
- **THEN** 系统返回失败列表并退出非零状态码
|
||||
|
||||
### Requirement: LibreOffice 解析器
|
||||
系统 SHALL 支持使用 LibreOffice soffice 命令行解析 DOC。
|
||||
|
||||
#### Scenario: LibreOffice 解析成功
|
||||
- **WHEN** soffice 可用且文档有效
|
||||
- **THEN** 系统返回 Markdown 内容
|
||||
|
||||
#### Scenario: LibreOffice 未安装
|
||||
- **WHEN** soffice 未在 PATH 中
|
||||
- **THEN** 系统返回失败信息
|
||||
|
||||
#### Scenario: LibreOffice 转换超时
|
||||
- **WHEN** soffice 执行超过 60 秒
|
||||
- **THEN** 系统返回超时错误
|
||||
|
||||
#### Scenario: LibreOffice 转换失败
|
||||
- **WHEN** soffice 返回非零退出码
|
||||
- **THEN** 系统返回失败信息
|
||||
|
||||
### Requirement: 解析器独立文件
|
||||
系统 SHALL 将解析器实现为独立的单文件模块。
|
||||
|
||||
#### Scenario: LibreOffice 解析器在独立文件
|
||||
- **WHEN** 使用 LibreOffice 解析器
|
||||
- **THEN** 从 readers/doc/libreoffice.py 导入
|
||||
|
||||
### Requirement: DOC Reader 测试使用静态文件
|
||||
DOC Reader 测试 MUST 使用 `tests/test_readers/fixtures/doc/` 下的静态文件。
|
||||
|
||||
#### Scenario: 测试使用 simple.doc
|
||||
- **WHEN** 测试 DOC Reader 基础解析能力
|
||||
- **THEN** 使用 `simple.doc` 静态文件
|
||||
|
||||
#### Scenario: 测试使用 with_headings.doc
|
||||
- **WHEN** 测试 DOC Reader 标题解析
|
||||
- **THEN** 使用 `with_headings.doc` 静态文件
|
||||
|
||||
#### Scenario: 测试使用 with_table.doc
|
||||
- **WHEN** 测试 DOC Reader 表格解析
|
||||
- **THEN** 使用 `with_table.doc` 静态文件
|
||||
@@ -9,7 +9,7 @@ DOCX 文档解析能力,支持多种解析方法。
|
||||
|
||||
#### Scenario: 按优先级尝试解析器
|
||||
- **WHEN** 解析 DOCX 文档
|
||||
- **THEN** 系统按 docling → unstructured → markitdown → pypandoc-binary → python-docx → XML原生解析的顺序尝试
|
||||
- **THEN** 系统按 docling → unstructured → pypandoc-binary → MarkItDown → LibreOffice → python-docx → XML原生解析的顺序尝试
|
||||
|
||||
#### Scenario: 成功解析
|
||||
- **WHEN** 任一解析器成功
|
||||
@@ -85,6 +85,25 @@ DOCX 文档解析能力,支持多种解析方法。
|
||||
- **WHEN** XML 原生解析失败
|
||||
- **THEN** 系统返回失败信息
|
||||
|
||||
### Requirement: LibreOffice 解析器
|
||||
系统 SHALL 支持使用 LibreOffice soffice 命令行解析 DOCX。
|
||||
|
||||
#### Scenario: LibreOffice 解析成功
|
||||
- **WHEN** soffice 可用且文档有效
|
||||
- **THEN** 系统返回 Markdown 内容
|
||||
|
||||
#### Scenario: LibreOffice 未安装
|
||||
- **WHEN** soffice 未在 PATH 中
|
||||
- **THEN** 系统尝试下一个解析器
|
||||
|
||||
#### Scenario: LibreOffice 转换超时
|
||||
- **WHEN** soffice 执行超过 60 秒
|
||||
- **THEN** 系统返回超时错误并尝试下一个解析器
|
||||
|
||||
#### Scenario: LibreOffice 转换失败
|
||||
- **WHEN** soffice 返回非零退出码
|
||||
- **THEN** 系统返回失败信息并尝试下一个解析器
|
||||
|
||||
### Requirement: 每个解析器独立文件
|
||||
系统 SHALL 将每个解析器实现为独立的单文件模块。
|
||||
|
||||
@@ -111,3 +130,7 @@ DOCX 文档解析能力,支持多种解析方法。
|
||||
#### Scenario: XML 原生解析器在独立文件
|
||||
- **WHEN** 使用 XML 原生解析器
|
||||
- **THEN** 从 readers/docx/native_xml.py 导入
|
||||
|
||||
#### Scenario: LibreOffice 解析器在独立文件
|
||||
- **WHEN** 使用 LibreOffice 解析器
|
||||
- **THEN** 从 readers/docx/libreoffice.py 导入
|
||||
|
||||
@@ -22,12 +22,39 @@
|
||||
- 必须使用 Python 3.12
|
||||
- `docling-parse` 5.x 无 x86_64 wheel,必须使用 4.0.0
|
||||
- 提供完整的 `uv run --python 3.12 --with "docling==2.40.0" --with "docling-parse==4.0.0" --with "numpy<2" ...` 命令示例
|
||||
- unstructured 在 Darwin-x86_64 平台不可用,已从配置中移除
|
||||
|
||||
#### Scenario: 每个平台的运行命令
|
||||
- **WHEN** 用户阅读 SKILL.md
|
||||
- **THEN** 系统必须为每个平台(Windows/macOS Intel/macOS ARM/Linux)和每种文档格式提供清晰的 `uv run --with` 命令示例
|
||||
- **AND** 命令必须包含所有必需的依赖包
|
||||
|
||||
### Requirement: 依赖配置结构
|
||||
config.py 中的 DEPENDENCIES 配置使用字典结构,保持简单直接以便于在不同平台进行细致调整。
|
||||
|
||||
#### Scenario: 配置数据格式不变
|
||||
- **WHEN** 代码访问 config.DEPENDENCIES["pdf"]["default"]
|
||||
- **THEN** 返回的数据结构保持不变
|
||||
- **AND** 包含 "python" 和 "dependencies" 字段
|
||||
|
||||
#### Scenario: 所有文件类型都有 Darwin-x86_64 配置
|
||||
- **WHEN** 查看 config.DEPENDENCIES
|
||||
- **THEN** pdf/docx/xlsx/pptx/xls/ppt 都有 "Darwin-x86_64" 平台配置
|
||||
- **AND** Darwin-x86_64 配置中不包含 unstructured 相关依赖
|
||||
|
||||
### Requirement: 依赖版本管理
|
||||
所有依赖必须指定版本号;default 平台使用最新版本作为标杆;default 配置在当前平台测试失败时,在特定平台配置中探索可运行的最新版本;default 配置的 python 版本必须为 None(使用默认 python 版本),仅在特定平台配置中可指定 python 版本;当前版本截止时间为 2026-03-18。
|
||||
|
||||
#### Scenario: default 平台使用最新版本且 python 为 None
|
||||
- **WHEN** 查看 config.DEPENDENCIES 中 default 配置
|
||||
- **THEN** python 版本为 None
|
||||
- **AND** 所有依赖都有明确的版本号
|
||||
- **AND** 使用截止 2026-03-18 的最新版本
|
||||
|
||||
#### Scenario: 特定平台在 default 失败时探索可运行版本
|
||||
- **WHEN** default 配置在当前平台测试失败
|
||||
- **THEN** 在特定平台配置中探索可运行的最新版本
|
||||
|
||||
### Requirement: 平台检测文档
|
||||
系统必须在 `SKILL.md` 中提供平台检测方法和平台特定的安装指南。
|
||||
|
||||
@@ -59,3 +86,24 @@
|
||||
#### Scenario: gitignore 配置(可选)
|
||||
- **WHEN** 用户查看项目的 `.gitignore` 文件
|
||||
- **THEN** 系统可以包含 `uv.lock` 条目以确保不会误提交(如果用户重新创建了 lock 文件)
|
||||
|
||||
### Requirement: 当前平台依赖验证
|
||||
系统必须在当前平台上验证 `config.DEPENDENCIES` 的 default 配置是否可以正常工作。
|
||||
|
||||
#### Scenario: 验证 default 配置可用性
|
||||
- **WHEN** 在当前平台运行测试
|
||||
- **THEN** 必须验证 default 配置的所有依赖都可以正确安装
|
||||
- **AND** 必须验证所有文档类型的解析功能正常工作
|
||||
|
||||
#### Scenario: 记录验证结果
|
||||
- **WHEN** 完成当前平台的依赖验证
|
||||
- **THEN** 必须在 `docs/upgrade-deps-prompt.md` 中记录验证结果
|
||||
- **AND** 必须记录当前平台信息和测试通过日期
|
||||
|
||||
### Requirement: 依赖版本文档化
|
||||
系统必须在 `docs/upgrade-deps-prompt.md` 中记录当前所有依赖的版本号和更新时间戳。
|
||||
|
||||
#### Scenario: 版本记录包含所有依赖
|
||||
- **WHEN** 查看 `docs/upgrade-deps-prompt.md`
|
||||
- **THEN** 文档必须包含所有文件类型(pdf/docx/xlsx/pptx/html/xls/ppt/doc)的所有依赖版本号
|
||||
- **AND** 必须标注版本更新时间戳
|
||||
|
||||
58
openspec/specs/ppt-reader/spec.md
Normal file
58
openspec/specs/ppt-reader/spec.md
Normal file
@@ -0,0 +1,58 @@
|
||||
## Purpose
|
||||
|
||||
PPT 文档解析能力,支持解析 Microsoft PowerPoint 97-2003 旧格式文档。
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: PPT 文档解析
|
||||
系统 SHALL 支持解析 .ppt 格式文档,使用 LibreOffice 解析器。
|
||||
|
||||
#### Scenario: 使用 LibreOffice 解析器
|
||||
- **WHEN** 解析 PPT 文档
|
||||
- **THEN** 系统使用 LibreOffice soffice 将 PPT 转换为 PPTX
|
||||
- **AND** 复用 PptxReader 解析转换后的 PPTX
|
||||
|
||||
#### Scenario: 成功解析
|
||||
- **WHEN** 解析器成功
|
||||
- **THEN** 系统返回解析结果
|
||||
|
||||
#### Scenario: 解析器失败
|
||||
- **WHEN** 解析器失败
|
||||
- **THEN** 系统返回失败列表并退出非零状态码
|
||||
|
||||
### Requirement: LibreOffice 解析器
|
||||
系统 SHALL 支持使用 LibreOffice soffice 命令行解析 PPT。
|
||||
|
||||
#### Scenario: LibreOffice 解析成功
|
||||
- **WHEN** soffice 可用且文档有效
|
||||
- **THEN** 系统返回 Markdown 内容
|
||||
|
||||
#### Scenario: LibreOffice 未安装
|
||||
- **WHEN** soffice 未在 PATH 中
|
||||
- **THEN** 系统返回失败信息
|
||||
|
||||
#### Scenario: LibreOffice 转换超时
|
||||
- **WHEN** soffice 执行超过 60 秒
|
||||
- **THEN** 系统返回超时错误
|
||||
|
||||
#### Scenario: LibreOffice 转换失败
|
||||
- **WHEN** soffice 返回非零退出码
|
||||
- **THEN** 系统返回失败信息
|
||||
|
||||
#### Scenario: 临时文件自动清理
|
||||
- **WHEN** 解析完成(无论成功或失败)
|
||||
- **THEN** 转换过程中生成的临时 PPTX 文件被自动清理
|
||||
|
||||
### Requirement: 解析器独立文件
|
||||
系统 SHALL 将解析器实现为独立的单文件模块。
|
||||
|
||||
#### Scenario: LibreOffice 解析器在独立文件
|
||||
- **WHEN** 使用 LibreOffice 解析器
|
||||
- **THEN** 从 readers/ppt/libreoffice.py 导入
|
||||
|
||||
### Requirement: PPT Reader 测试使用静态文件
|
||||
PPT Reader 测试 MUST 使用 `tests/test_readers/fixtures/ppt/` 下的静态文件。
|
||||
|
||||
#### Scenario: 测试使用 simple.ppt
|
||||
- **WHEN** 测试 PPT Reader 基础解析能力
|
||||
- **THEN** 使用 `simple.ppt` 静态文件
|
||||
@@ -93,3 +93,37 @@
|
||||
#### Scenario: 匹配页码
|
||||
- **WHEN** 文本匹配 `_UNSTRUCTURED_PAGE_NUMBER_PATTERN`(如 "— 3 —")
|
||||
- **THEN** 系统将其识别为噪声并过滤
|
||||
|
||||
### Requirement: 通用 LibreOffice 格式转换
|
||||
系统 SHALL 提供通用的 LibreOffice 格式转换函数,支持在不同格式间转换。
|
||||
|
||||
#### Scenario: 转换文件到指定格式
|
||||
- **WHEN** 调用 `convert_via_libreoffice(input_path, target_format, output_dir)`
|
||||
- **THEN** 系统使用 soffice --headless --convert-to 进行转换
|
||||
- **AND** 输出文件写入 output_dir
|
||||
- **AND** 成功时返回 (output_path, None)
|
||||
- **AND** 失败时返回 (None, error_message)
|
||||
|
||||
#### Scenario: LibreOffice 未安装
|
||||
- **WHEN** soffice 未在 PATH 中
|
||||
- **THEN** 系统返回 (None, "LibreOffice 未安装")
|
||||
|
||||
#### Scenario: 转换超时
|
||||
- **WHEN** soffice 执行超过 timeout 秒(默认 60 秒)
|
||||
- **THEN** 系统返回 (None, "LibreOffice 转换超时")
|
||||
|
||||
#### Scenario: 转换失败
|
||||
- **WHEN** soffice 返回非零退出码
|
||||
- **THEN** 系统返回 (None, "LibreOffice 转换失败 (code: {code})")
|
||||
|
||||
#### Scenario: 输出文件未生成
|
||||
- **WHEN** soffice 执行成功但未生成输出文件
|
||||
- **THEN** 系统返回 (None, "LibreOffice 未生成输出文件")
|
||||
|
||||
#### Scenario: 可自定义输出后缀
|
||||
- **WHEN** 提供 output_suffix 参数
|
||||
- **THEN** 系统使用该后缀作为输出文件后缀,而不是 target_format
|
||||
|
||||
#### Scenario: 调用者管理输出目录生命周期
|
||||
- **WHEN** convert_via_libreoffice 执行完成
|
||||
- **THEN** 输出文件保留在 output_dir 中,由调用者负责清理
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: SKILL.md 遵循 Claude Skill 构建指南
|
||||
SKILL.md 文档必须遵循 Claude 官方 Skill 构建指南的最佳实践,包括渐进式披露的三级系统、清晰的触发词和完整的章节结构。SKILL.md 必须将 --advice 参数作为首选方案放在最前面强调。
|
||||
SKILL.md 文档必须遵循 Claude 官方 Skill 构建指南的最佳实践,包括渐进式披露的三级系统、清晰的触发词和完整的章节结构。
|
||||
|
||||
#### Scenario: Claude 正确加载 skill
|
||||
- **WHEN** 用户询问与文档解析相关的问题
|
||||
@@ -11,61 +11,38 @@ SKILL.md 文档必须遵循 Claude 官方 Skill 构建指南的最佳实践,
|
||||
- **WHEN** skill 被加载
|
||||
- **THEN** AI 应能从 Purpose 和 When to Use 章节了解何时使用此 skill
|
||||
|
||||
#### Scenario: --advice 放在最前面
|
||||
- **WHEN** AI 查看 SKILL.md
|
||||
- **THEN** Purpose 章节第一部分就是 --advice 的使用说明
|
||||
|
||||
### Requirement: YAML frontmatter 包含完整元数据
|
||||
YAML frontmatter 必须包含 name、description(带触发词)、license、metadata 和 compatibility 字段。
|
||||
YAML frontmatter 必须包含 name、description(带触发词)、license 和 compatibility 字段。
|
||||
|
||||
#### Scenario: description 包含触发词
|
||||
- **WHEN** 查看 YAML frontmatter
|
||||
- **THEN** description 应包含功能说明、触发条件和用户可能说的具体任务
|
||||
|
||||
#### Scenario: compatibility 说明依赖和执行路径优先级
|
||||
#### Scenario: compatibility 说明依赖和 Python 版本
|
||||
- **WHEN** 查看 YAML frontmatter
|
||||
- **THEN** compatibility 应说明 Python 版本要求和三种执行路径的优先级(lyxy-runner-python skill → uv → 主机 Python)
|
||||
- **THEN** compatibility 应说明 Python 版本要求和脚本自启动特性
|
||||
|
||||
### Requirement: 三路径执行策略
|
||||
skill 文档必须说明三种执行路径,优先级为:1. lyxy-runner-python skill,2. uv run --with,3. 主机 Python 环境。
|
||||
### Requirement: 推荐用法
|
||||
SKILL.md 必须说明直接运行 `python scripts/lyxy_document_reader.py` 即可,脚本会自动检测文件类型、当前平台,并用正确的 uv 命令执行。
|
||||
|
||||
#### Scenario: lyxy-runner-python 可用
|
||||
- **WHEN** lyxy-runner-python skill 已安装
|
||||
- **THEN** 文档说明使用 lyxy-runner-python 自动管理依赖
|
||||
#### Scenario: 直接运行脚本
|
||||
- **WHEN** AI 需要执行文档解析
|
||||
- **THEN** 运行 `python scripts/lyxy_document_reader.py <文件路径或URL>`
|
||||
|
||||
#### Scenario: 使用 uv run --with
|
||||
- **WHEN** lyxy-runner-python skill 不可用
|
||||
- **THEN** 文档说明使用 --advice 获取 uv run --with 命令
|
||||
|
||||
#### Scenario: 降级到主机 Python
|
||||
- **WHEN** uv 也不可用
|
||||
- **THEN** 文档说明如何手动安装具体依赖包并使用主机 Python
|
||||
|
||||
### Requirement: --advice 是首选方案
|
||||
SKILL.md 必须将 --advice 参数作为获取准确命令的首选方案,移除冗余的手动依赖命令示例块(仅保留简洁参考)。
|
||||
|
||||
#### Scenario: --advice 是第一步
|
||||
- **WHEN** AI 阅读 SKILL.md
|
||||
- **THEN** 首先看到 --advice 的使用说明
|
||||
|
||||
#### Scenario: 依赖命令以 --advice 输出为准
|
||||
- **WHEN** AI 需要了解依赖命令
|
||||
- **THEN** 文档引导 AI 使用 --advice 获取,而非阅读文档中的示例
|
||||
|
||||
#### Scenario: 保留简洁参数示例
|
||||
- **WHEN** AI 需要了解参数用法
|
||||
- **THEN** 文档提供简洁的参数使用示例(不含大段依赖命令)
|
||||
#### Scenario: 脚本自动检测
|
||||
- **WHEN** 运行脚本
|
||||
- **THEN** 脚本自动检测文件类型、当前平台,并用正确的 uv 命令执行
|
||||
|
||||
### Requirement: 文档包含关键章节
|
||||
SKILL.md 必须包含 Purpose、When to Use、Quick Reference、Workflow 等章节,遵循渐进式披露原则。
|
||||
SKILL.md 必须包含 Purpose、When to Use、Quick Reference、参数使用示例等章节,遵循渐进式披露原则。
|
||||
|
||||
#### Scenario: 快速查找用法
|
||||
- **WHEN** AI 需要了解如何使用此 skill
|
||||
- **THEN** Quick Reference 表格提供命令参数概览
|
||||
|
||||
#### Scenario: 了解执行流程
|
||||
- **WHEN** AI 需要理解解析流程
|
||||
- **THEN** Workflow 章节说明 3 步工作流程(获取建议 → 选择执行方式 → 添加参数)
|
||||
#### Scenario: 了解参数用法
|
||||
- **WHEN** AI 需要了解参数用法
|
||||
- **THEN** 参数使用示例章节提供简洁的命令示例
|
||||
|
||||
### Requirement: 触发词覆盖多种表达方式
|
||||
description 和 When to Use 章节必须包含中文和英文的触发词,以及文件扩展名。
|
||||
@@ -83,7 +60,7 @@ description 和 When to Use 章节必须包含中文和英文的触发词,以
|
||||
|
||||
#### Scenario: 依赖缺失错误
|
||||
- **WHEN** 出现 ModuleNotFoundError
|
||||
- **THEN** 错误处理表格说明需要使用 --advice 获取正确的依赖命令
|
||||
- **THEN** 错误处理表格说明脚本会自动检测并安装依赖
|
||||
|
||||
#### Scenario: 文件类型不支持
|
||||
- **WHEN** 出现"不支持的文件类型"错误
|
||||
|
||||
@@ -70,7 +70,7 @@
|
||||
- **THEN** 系统不需要 --obfuscate 参数,直接执行混淆构建
|
||||
|
||||
### Requirement: PyArmor 混淆执行
|
||||
系统 SHALL 调用 PyArmor 工具对 scripts 目录进行混淆。
|
||||
系统 SHALL 调用 PyArmor 工具对 scripts 目录进行混淆,然后将 pyarmor_runtime 目录移动到 scripts 内部。
|
||||
|
||||
#### Scenario: PyArmor 成功执行
|
||||
- **WHEN** PyArmor 可用
|
||||
@@ -78,7 +78,7 @@
|
||||
|
||||
#### Scenario: 混淆后文件输出
|
||||
- **WHEN** PyArmor 混淆完成
|
||||
- **THEN** build/ 目录包含混淆后的文件和 pyarmor_runtime 子目录
|
||||
- **THEN** build/ 目录包含混淆后的 scripts 目录,且 pyarmor_runtime 子目录位于 scripts/ 内部
|
||||
|
||||
### Requirement: PyArmor 未安装友好提示
|
||||
系统 SHALL 在 PyArmor 未安装时提供清晰的错误提示,引导用户正确使用 `uv run --with pyarmor`。
|
||||
|
||||
@@ -6,6 +6,23 @@
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: 测试运行器包含 fixtures 依赖
|
||||
run_tests.py 必须定义 TEST_FIXTURE_DEPENDENCIES 常量,包含创建临时测试文件所需的所有依赖。
|
||||
|
||||
#### Scenario: TEST_FIXTURE_DEPENDENCIES 定义存在
|
||||
- **WHEN** 查看 run_tests.py
|
||||
- **THEN** 存在 TEST_FIXTURE_DEPENDENCIES 常量
|
||||
- **AND** 包含 python-docx(用于创建临时 DOCX)
|
||||
- **AND** 包含 reportlab(用于创建临时 PDF)
|
||||
- **AND** 包含 pandas(用于创建临时 XLSX)
|
||||
- **AND** 包含 openpyxl(pandas 写 XLSX 需要)
|
||||
- **AND** 包含 python-pptx(用于创建临时 PPTX)
|
||||
|
||||
#### Scenario: fixtures 依赖与文件类型依赖合并
|
||||
- **WHEN** 运行任何类型的测试
|
||||
- **THEN** TEST_FIXTURE_DEPENDENCIES 中的依赖自动合并到 uv run --with 参数中
|
||||
- **AND** 去重处理,避免重复添加
|
||||
|
||||
### Requirement: 临时文件自动清理
|
||||
测试使用的临时文件 MUST 在测试完成后自动清理,使用 pytest 的 tmp_path fixture。
|
||||
|
||||
|
||||
69
openspec/specs/test-runner/spec.md
Normal file
69
openspec/specs/test-runner/spec.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# Test Runner Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
定义自动化测试运行器的功能规范,包括测试类型选择、依赖自动加载、pytest 参数透传等。
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: 测试运行器支持指定测试类型
|
||||
测试运行器 SHALL 支持通过命令行参数指定测试类型,自动加载对应依赖并运行 pytest。
|
||||
|
||||
#### Scenario: 运行 PDF 测试
|
||||
- **WHEN** 用户执行 `python run_tests.py pdf`
|
||||
- **THEN** 自动加载 config.DEPENDENCIES["pdf"] 中的依赖
|
||||
- **AND** 自动加载测试 fixtures 所需的依赖
|
||||
- **AND** 运行 tests/test_readers/test_pdf/ 目录下的测试
|
||||
|
||||
#### Scenario: 运行 DOCX 测试
|
||||
- **WHEN** 用户执行 `python run_tests.py docx`
|
||||
- **THEN** 自动加载 config.DEPENDENCIES["docx"] 中的依赖
|
||||
- **AND** 自动加载测试 fixtures 所需的依赖
|
||||
- **AND** 运行 tests/test_readers/test_docx/ 目录下的测试
|
||||
|
||||
#### Scenario: 运行 CLI 测试(无特殊依赖)
|
||||
- **WHEN** 用户执行 `python run_tests.py cli`
|
||||
- **THEN** 加载 pytest 依赖
|
||||
- **AND** 自动加载测试 fixtures 所需的依赖
|
||||
- **AND** 加载 config.DEPENDENCIES 中所有类型的依赖(去重)
|
||||
- **AND** 运行 tests/test_cli/ 目录下的测试
|
||||
|
||||
#### Scenario: 运行所有测试
|
||||
- **WHEN** 用户执行 `python run_tests.py all`
|
||||
- **THEN** 加载 config.DEPENDENCIES 中所有类型的依赖(去重)
|
||||
- **AND** 自动加载测试 fixtures 所需的依赖
|
||||
- **AND** 运行 tests/ 目录下的所有测试
|
||||
|
||||
### Requirement: 测试运行器支持透传 pytest 参数
|
||||
测试运行器 SHALL 支持将额外的命令行参数透传给 pytest。
|
||||
|
||||
#### Scenario: 传递 -v 参数
|
||||
- **WHEN** 用户执行 `python run_tests.py pdf -v`
|
||||
- **THEN** pytest 以 verbose 模式运行
|
||||
|
||||
#### Scenario: 传递 --cov 参数
|
||||
- **WHEN** 用户执行 `python run_tests.py pdf --cov=scripts`
|
||||
- **THEN** pytest 生成测试覆盖率报告
|
||||
|
||||
#### Scenario: 运行特定测试文件
|
||||
- **WHEN** 用户执行 `python run_tests.py pdf tests/test_readers/test_pdf/test_docling_pdf.py`
|
||||
- **THEN** 仅运行指定的测试文件
|
||||
|
||||
### Requirement: 测试运行器支持平台特定配置
|
||||
测试运行器 SHALL 根据当前平台自动选择对应的依赖配置(如 Darwin-x86_64)。
|
||||
|
||||
#### Scenario: 在 Darwin-x86_64 平台运行 PDF 测试
|
||||
- **WHEN** 用户在 Darwin-x86_64 平台执行 `python run_tests.py pdf`
|
||||
- **THEN** 使用 config.DEPENDENCIES["pdf"]["Darwin-x86_64"] 配置(如果存在)
|
||||
- **AND** 使用 python 3.12(如配置中指定)
|
||||
|
||||
### Requirement: advice_generator 包含完整 Reader 映射
|
||||
advice_generator.py 中的 _READER_KEY_MAP SHALL 包含所有 Reader 类的映射,包括 DocReader 和 PptReader。
|
||||
|
||||
#### Scenario: DocReader 映射存在
|
||||
- **WHEN** 查询 _READER_KEY_MAP[DocReader]
|
||||
- **THEN** 返回 "doc"
|
||||
|
||||
#### Scenario: PptReader 映射存在
|
||||
- **WHEN** 查询 _READER_KEY_MAP[PptReader]
|
||||
- **THEN** 返回 "ppt"
|
||||
@@ -75,3 +75,29 @@
|
||||
#### Scenario: 所有格式都包含 chardet
|
||||
- **WHEN** 用户查阅任何格式的依赖命令
|
||||
- **THEN** 命令必须包含 `--with chardet`
|
||||
|
||||
### Requirement: 当前平台命令验证
|
||||
系统必须验证当前平台的 `uv run --with` 命令可以正确执行。
|
||||
|
||||
#### Scenario: 验证 default 平台命令
|
||||
- **WHEN** 在当前平台执行 `uv run --with` 命令
|
||||
- **THEN** 必须可以成功安装所有依赖
|
||||
- **AND** 必须可以成功运行文档解析脚本
|
||||
|
||||
#### Scenario: 记录当前平台命令
|
||||
- **WHEN** 更新 SKILL.md 或 README.md
|
||||
- **THEN** 必须包含当前平台的命令示例
|
||||
- **AND** 命令中的依赖版本必须与 `config.DEPENDENCIES` 一致
|
||||
|
||||
### Requirement: 版本一致性
|
||||
SKILL.md 和 README.md 中的依赖版本必须与 `config.DEPENDENCIES` 中指定的版本一致。
|
||||
|
||||
#### Scenario: 文档中的版本与配置一致
|
||||
- **WHEN** 查看 SKILL.md 或 README.md 中的 `uv run --with` 命令示例
|
||||
- **THEN** 命令中指定的依赖版本必须与 `config.DEPENDENCIES` 中 default 配置的版本一致
|
||||
- **AND** 如果配置中指定了特定版本,文档中必须使用相同版本
|
||||
|
||||
#### Scenario: 更新依赖时同步更新文档
|
||||
- **WHEN** 更新 `config.DEPENDENCIES` 中的依赖版本
|
||||
- **THEN** 必须同步更新 SKILL.md 和 README.md 中的相关命令示例
|
||||
- **AND** 必须更新 `docs/upgrade-deps-prompt.md` 中的版本记录
|
||||
|
||||
88
publish.py
88
publish.py
@@ -28,14 +28,9 @@ def check_build_dir(build_dir: str) -> None:
|
||||
SystemExit: 目录不存在时退出
|
||||
"""
|
||||
if not os.path.exists(build_dir):
|
||||
print("""
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
错误: build/ 目录不存在
|
||||
|
||||
请先运行 build.py:
|
||||
uv run python build.py
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
""")
|
||||
print("错误: build/ 目录不存在")
|
||||
print("请先运行 build.py:")
|
||||
print(" uv run python build.py")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
@@ -50,14 +45,9 @@ def check_build_skill_md(build_skill_md_path: str) -> None:
|
||||
SystemExit: 文件不存在时退出
|
||||
"""
|
||||
if not os.path.exists(build_skill_md_path):
|
||||
print("""
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
错误: build/SKILL.md 不存在
|
||||
|
||||
请先运行 build.py:
|
||||
uv run python build.py
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
""")
|
||||
print("错误: build/SKILL.md 不存在")
|
||||
print("请先运行 build.py:")
|
||||
print(" uv run python build.py")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
@@ -101,13 +91,8 @@ def parse_version_from_skill_md(skill_md_path: str) -> str:
|
||||
# metadata 块结束
|
||||
in_metadata = False
|
||||
|
||||
print("""
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
错误: 无法从 build/SKILL.md 解析版本号
|
||||
|
||||
请检查 build/SKILL.md 是否包含 metadata.version 字段
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
""")
|
||||
print("错误: 无法从 build/SKILL.md 解析版本号")
|
||||
print("请检查 build/SKILL.md 是否包含 metadata.version 字段")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
@@ -149,21 +134,14 @@ def clone_repo(temp_dir: str) -> str:
|
||||
SystemExit: clone 失败时退出
|
||||
"""
|
||||
repo_dir = os.path.join(temp_dir, "skills-repo")
|
||||
print(f"Clone 仓库: {TARGET_REPO_URL}")
|
||||
print(f" 到: {repo_dir}")
|
||||
|
||||
try:
|
||||
run_git_command(temp_dir, ["clone", "--depth", "1", TARGET_REPO_URL, "skills-repo"])
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"""
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
错误: Clone 仓库失败
|
||||
|
||||
返回码: {e.returncode}
|
||||
标准输出: {e.stdout}
|
||||
错误输出: {e.stderr}
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
""")
|
||||
print(f"错误: Clone 仓库失败")
|
||||
print(f" 返回码: {e.returncode}")
|
||||
print(f" 标准输出: {e.stdout}")
|
||||
print(f" 错误输出: {e.stderr}")
|
||||
sys.exit(1)
|
||||
|
||||
return repo_dir
|
||||
@@ -182,7 +160,6 @@ def clear_target_dir(repo_dir: str) -> str:
|
||||
target_dir = os.path.join(repo_dir, TARGET_PATH)
|
||||
|
||||
if os.path.exists(target_dir):
|
||||
print(f"清空目标目录: {target_dir}")
|
||||
shutil.rmtree(target_dir)
|
||||
|
||||
os.makedirs(target_dir, exist_ok=True)
|
||||
@@ -197,18 +174,14 @@ def copy_build_contents(build_dir: str, target_dir: str) -> None:
|
||||
build_dir: build 源目录
|
||||
target_dir: 目标目录
|
||||
"""
|
||||
print(f"复制 build/ 内容 -> {target_dir}")
|
||||
|
||||
for item in os.listdir(build_dir):
|
||||
src = os.path.join(build_dir, item)
|
||||
dst = os.path.join(target_dir, item)
|
||||
|
||||
if os.path.isdir(src):
|
||||
shutil.copytree(src, dst)
|
||||
print(f" 目录: {item}")
|
||||
else:
|
||||
shutil.copy2(src, dst)
|
||||
print(f" 文件: {item}")
|
||||
|
||||
|
||||
def git_commit_and_push(repo_dir: str, version: str) -> None:
|
||||
@@ -224,23 +197,15 @@ def git_commit_and_push(repo_dir: str, version: str) -> None:
|
||||
"""
|
||||
commit_message = f"publish: lyxy-document-reader {version}"
|
||||
|
||||
print(f"Git 提交: {commit_message}")
|
||||
|
||||
try:
|
||||
run_git_command(repo_dir, ["add", "."])
|
||||
run_git_command(repo_dir, ["commit", "-m", commit_message])
|
||||
print(" 推送中...")
|
||||
run_git_command(repo_dir, ["push"])
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"""
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
错误: Git 操作失败
|
||||
|
||||
返回码: {e.returncode}
|
||||
标准输出: {e.stdout}
|
||||
错误输出: {e.stderr}
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
""")
|
||||
print(f"错误: Git 操作失败")
|
||||
print(f" 返回码: {e.returncode}")
|
||||
print(f" 标准输出: {e.stdout}")
|
||||
print(f" 错误输出: {e.stderr}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
@@ -248,10 +213,6 @@ def main() -> None:
|
||||
"""
|
||||
主函数:执行完整的发布流程
|
||||
"""
|
||||
print("=" * 60)
|
||||
print("Skill 发布")
|
||||
print("=" * 60)
|
||||
|
||||
# 路径配置
|
||||
project_root = os.path.dirname(os.path.abspath(__file__))
|
||||
build_dir = os.path.join(project_root, "build")
|
||||
@@ -263,37 +224,20 @@ def main() -> None:
|
||||
|
||||
# 解析版本号
|
||||
version = parse_version_from_skill_md(build_skill_md_path)
|
||||
print(f"版本号: {version}")
|
||||
print()
|
||||
|
||||
# 使用临时目录
|
||||
with tempfile.TemporaryDirectory(prefix="lyxy-publish-") as temp_dir:
|
||||
print(f"临时目录: {temp_dir}")
|
||||
print()
|
||||
|
||||
# Clone 仓库
|
||||
repo_dir = clone_repo(temp_dir)
|
||||
print()
|
||||
|
||||
# 清空目标路径
|
||||
target_dir = clear_target_dir(repo_dir)
|
||||
print()
|
||||
|
||||
# 复制内容
|
||||
copy_build_contents(build_dir, target_dir)
|
||||
print()
|
||||
|
||||
# Git 提交并推送
|
||||
git_commit_and_push(repo_dir, version)
|
||||
print()
|
||||
|
||||
# 完成信息
|
||||
print("=" * 60)
|
||||
print("发布完成!")
|
||||
print(f"版本号: {version}")
|
||||
print(f"目标仓库: {TARGET_REPO_URL}")
|
||||
print(f"目标路径: {TARGET_PATH}")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
20
publish.sh
20
publish.sh
@@ -10,21 +10,9 @@ set -e
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
echo "============================================"
|
||||
echo "Skill 混淆构建 + 发布"
|
||||
echo "============================================"
|
||||
echo
|
||||
|
||||
# 1. 混淆构建
|
||||
echo "[1/2] 执行混淆构建..."
|
||||
echo ">>> 构建 + 发布"
|
||||
echo "[1/2] 构建..."
|
||||
uv run --with pyarmor python build.py
|
||||
echo
|
||||
|
||||
# 2. 发布
|
||||
echo "[2/2] 执行发布..."
|
||||
echo "[2/2] 发布..."
|
||||
uv run python publish.py
|
||||
echo
|
||||
|
||||
echo "============================================"
|
||||
echo "完成!"
|
||||
echo "============================================"
|
||||
echo ">>> 完成"
|
||||
|
||||
284
run_tests.py
Normal file
284
run_tests.py
Normal file
@@ -0,0 +1,284 @@
|
||||
#!/usr/bin/env python3
|
||||
"""测试运行器 - 自动根据测试类型加载依赖并运行 pytest"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# 确定项目根目录和脚本路径
|
||||
script_file = Path(__file__).resolve()
|
||||
project_root = script_file.parent
|
||||
scripts_dir = project_root / "scripts"
|
||||
bootstrap_path = str(scripts_dir / "bootstrap.py")
|
||||
|
||||
# 将 scripts/ 目录添加到 sys.path
|
||||
if str(scripts_dir) not in sys.path:
|
||||
sys.path.append(str(scripts_dir))
|
||||
|
||||
# 抑制第三方库日志
|
||||
os.environ["HF_HUB_DISABLE_PROGRESS_BARS"] = "1"
|
||||
os.environ["HF_HUB_DISABLE_TELEMETRY"] = "1"
|
||||
os.environ["TQDM_DISABLE"] = "1"
|
||||
|
||||
# 测试 fixtures 需要的依赖(用于创建临时测试文件)
|
||||
TEST_FIXTURE_DEPENDENCIES = {
|
||||
"default": [
|
||||
"python-docx==1.2.0", # 用于创建临时 DOCX
|
||||
"reportlab==4.2.2", # 用于创建临时 PDF
|
||||
"pandas==3.0.1", # 用于创建临时 XLSX
|
||||
"openpyxl==3.1.5", # pandas 写 XLSX 需要
|
||||
"python-pptx==1.0.2", # 用于创建临时 PPTX
|
||||
],
|
||||
"Darwin-x86_64": [
|
||||
"python-docx==1.2.0", # 用于创建临时 DOCX
|
||||
"reportlab==4.2.2", # 用于创建临时 PDF
|
||||
"pandas<3.0.0", # 用于创建临时 XLSX(兼容 Darwin-x86_64)
|
||||
"openpyxl==3.1.5", # pandas 写 XLSX 需要
|
||||
"python-pptx==1.0.2", # 用于创建临时 PPTX
|
||||
],
|
||||
}
|
||||
|
||||
# 测试类型映射
|
||||
_TEST_TYPES = {
|
||||
# 文件类型测试(有依赖配置)
|
||||
"pdf": {"key": "pdf", "path": "tests/test_readers/test_pdf/"},
|
||||
"docx": {"key": "docx", "path": "tests/test_readers/test_docx/"},
|
||||
"xlsx": {"key": "xlsx", "path": "tests/test_readers/test_xlsx/"},
|
||||
"pptx": {"key": "pptx", "path": "tests/test_readers/test_pptx/"},
|
||||
"html": {"key": "html", "path": "tests/test_readers/test_html/"},
|
||||
"xls": {"key": "xls", "path": "tests/test_readers/test_xls/"},
|
||||
"doc": {"key": "doc", "path": "tests/test_readers/test_doc/"},
|
||||
"ppt": {"key": "ppt", "path": "tests/test_readers/test_ppt/"},
|
||||
# 核心测试(cli 测试需要所有依赖,因为它测试多种格式)
|
||||
"cli": {"key": "all", "path": "tests/test_cli/"},
|
||||
"core": {"key": None, "path": "tests/test_core/"},
|
||||
"utils": {"key": None, "path": "tests/test_utils/"},
|
||||
# 所有测试(合并所有依赖)
|
||||
"all": {"key": "all", "path": "tests/"},
|
||||
}
|
||||
|
||||
|
||||
def _collect_all_dependencies(platform_id: str):
|
||||
"""
|
||||
收集所有文件类型的依赖并去重(内部辅助函数)。
|
||||
|
||||
Args:
|
||||
platform_id: 平台标识
|
||||
|
||||
Returns:
|
||||
(python_version, dependencies) 元组
|
||||
"""
|
||||
from config import DEPENDENCIES
|
||||
|
||||
python_version = None
|
||||
all_deps = set()
|
||||
for type_key, type_config in DEPENDENCIES.items():
|
||||
# 先尝试特定平台配置
|
||||
if platform_id in type_config:
|
||||
cfg = type_config[platform_id]
|
||||
elif "default" in type_config:
|
||||
cfg = type_config["default"]
|
||||
else:
|
||||
continue
|
||||
# 记录 python 版本(优先使用有特殊要求的)
|
||||
if cfg.get("python") and not python_version:
|
||||
python_version = cfg["python"]
|
||||
# 收集依赖
|
||||
for dep in cfg.get("dependencies", []):
|
||||
all_deps.add(dep)
|
||||
return python_version, list(all_deps)
|
||||
|
||||
|
||||
def get_dependencies_for_type(test_type: str, platform_id: str):
|
||||
"""
|
||||
获取指定测试类型的依赖配置(完全从 config.py 获取)。
|
||||
|
||||
Args:
|
||||
test_type: 测试类型(pdf/docx/.../all)
|
||||
platform_id: 平台标识
|
||||
|
||||
Returns:
|
||||
(python_version, dependencies) 元组
|
||||
"""
|
||||
from config import DEPENDENCIES
|
||||
|
||||
config = _TEST_TYPES.get(test_type)
|
||||
if not config:
|
||||
return None, []
|
||||
|
||||
key = config["key"]
|
||||
|
||||
if key is None:
|
||||
# core/utils 测试不需要特殊依赖
|
||||
return None, []
|
||||
|
||||
if key == "all":
|
||||
# cli 和 all 都使用收集所有依赖的逻辑
|
||||
return _collect_all_dependencies(platform_id)
|
||||
|
||||
# 单个类型的依赖,完全从 config.py 获取
|
||||
if key not in DEPENDENCIES:
|
||||
return None, []
|
||||
|
||||
type_config = DEPENDENCIES[key]
|
||||
if platform_id in type_config:
|
||||
cfg = type_config[platform_id]
|
||||
elif "default" in type_config:
|
||||
cfg = type_config["default"]
|
||||
else:
|
||||
return None, []
|
||||
|
||||
return cfg.get("python"), cfg.get("dependencies", [])
|
||||
|
||||
|
||||
def get_fixture_dependencies(platform_id: str):
|
||||
"""
|
||||
获取指定平台的 fixtures 依赖。
|
||||
|
||||
Args:
|
||||
platform_id: 平台标识
|
||||
|
||||
Returns:
|
||||
list: fixtures 依赖列表
|
||||
"""
|
||||
if platform_id in TEST_FIXTURE_DEPENDENCIES:
|
||||
return TEST_FIXTURE_DEPENDENCIES[platform_id]
|
||||
elif "default" in TEST_FIXTURE_DEPENDENCIES:
|
||||
return TEST_FIXTURE_DEPENDENCIES["default"]
|
||||
else:
|
||||
return []
|
||||
|
||||
|
||||
def generate_uv_args(
|
||||
dependencies: list,
|
||||
test_path: str,
|
||||
pytest_args: list,
|
||||
python_version: str = None,
|
||||
platform_id: str = None,
|
||||
):
|
||||
"""
|
||||
生成 uv run 命令参数列表(用于 subprocess.run)。
|
||||
|
||||
Args:
|
||||
dependencies: 依赖包列表
|
||||
test_path: 测试路径
|
||||
pytest_args: 透传给 pytest 的参数
|
||||
python_version: 需要的 python 版本,None 表示不指定
|
||||
platform_id: 平台标识,用于选择 fixtures 依赖
|
||||
|
||||
Returns:
|
||||
uv run 命令参数列表
|
||||
"""
|
||||
args = ["uv", "run"]
|
||||
|
||||
if python_version:
|
||||
args.extend(["--python", python_version])
|
||||
|
||||
# 添加 pytest
|
||||
args.extend(["--with", "pytest"])
|
||||
|
||||
# 获取当前平台的 fixtures 依赖
|
||||
fixture_deps = get_fixture_dependencies(platform_id) if platform_id else []
|
||||
|
||||
# 合并文件类型依赖和 fixtures 依赖,去重
|
||||
all_deps = set()
|
||||
for dep in dependencies:
|
||||
all_deps.add(dep)
|
||||
for dep in fixture_deps:
|
||||
all_deps.add(dep)
|
||||
|
||||
# 添加所有依赖
|
||||
for dep in sorted(all_deps):
|
||||
args.extend(["--with", dep])
|
||||
|
||||
# 添加 pytest 命令
|
||||
args.append("pytest")
|
||||
|
||||
# 添加测试路径
|
||||
args.append(test_path)
|
||||
|
||||
# 添加透传的 pytest 参数
|
||||
args.extend(pytest_args)
|
||||
|
||||
return args
|
||||
|
||||
|
||||
def main():
|
||||
"""主函数:解析参数并运行测试"""
|
||||
# 解析命令行参数
|
||||
parser = argparse.ArgumentParser(
|
||||
description="自动根据测试类型加载依赖并运行 pytest",
|
||||
usage="%(prog)s <test_type> [pytest_args...]",
|
||||
)
|
||||
parser.add_argument(
|
||||
"test_type",
|
||||
choices=list(_TEST_TYPES.keys()),
|
||||
help="测试类型: " + ", ".join(_TEST_TYPES.keys()),
|
||||
)
|
||||
parser.add_argument(
|
||||
"pytest_args",
|
||||
nargs=argparse.REMAINDER,
|
||||
help="透传给 pytest 的参数(如 -v, --cov 等)",
|
||||
)
|
||||
|
||||
# 如果没有参数,显示帮助
|
||||
if len(sys.argv) == 1:
|
||||
parser.print_help()
|
||||
sys.exit(1)
|
||||
|
||||
# 特殊处理:如果第一个参数是帮助选项
|
||||
if sys.argv[1] in ("-h", "--help"):
|
||||
parser.print_help()
|
||||
sys.exit(0)
|
||||
|
||||
# 使用 parse_known_args 来正确处理透传参数
|
||||
# 因为 argparse.REMAINDER 会吃掉 --help,我们手动处理
|
||||
test_type = sys.argv[1]
|
||||
pytest_args = sys.argv[2:]
|
||||
|
||||
# 验证 test_type
|
||||
if test_type not in _TEST_TYPES:
|
||||
print(f"错误: 未知的测试类型 '{test_type}'")
|
||||
print(f"可用类型: {', '.join(_TEST_TYPES.keys())}")
|
||||
sys.exit(1)
|
||||
|
||||
# 检测 uv 是否可用
|
||||
uv_path = shutil.which("uv")
|
||||
if not uv_path:
|
||||
print("错误: 未找到 uv,请先安装 uv")
|
||||
sys.exit(1)
|
||||
|
||||
# 获取测试配置
|
||||
test_config = _TEST_TYPES[test_type]
|
||||
test_path = test_config["path"]
|
||||
|
||||
# 导入需要的模块
|
||||
from core.advice_generator import get_platform
|
||||
|
||||
# 获取平台和依赖配置
|
||||
platform_id = get_platform()
|
||||
python_version, dependencies = get_dependencies_for_type(test_type, platform_id)
|
||||
|
||||
# 生成 uv 命令参数
|
||||
uv_args = generate_uv_args(
|
||||
dependencies=dependencies,
|
||||
test_path=test_path,
|
||||
pytest_args=pytest_args,
|
||||
python_version=python_version,
|
||||
platform_id=platform_id,
|
||||
)
|
||||
|
||||
# 设置环境变量
|
||||
env = os.environ.copy()
|
||||
env["PYTHONPATH"] = str(project_root)
|
||||
|
||||
# 执行测试
|
||||
result = subprocess.run(uv_args, env=env, cwd=str(project_root))
|
||||
sys.exit(result.returncode)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
111
scripts/bootstrap.py
Normal file
111
scripts/bootstrap.py
Normal file
@@ -0,0 +1,111 @@
|
||||
#!/usr/bin/env python3
|
||||
"""文档解析器实际执行模块,承载业务逻辑。"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
# 将 scripts/ 目录添加到 sys.path,支持从任意位置执行脚本
|
||||
scripts_dir = Path(__file__).resolve().parent
|
||||
if str(scripts_dir) not in sys.path:
|
||||
sys.path.append(str(scripts_dir))
|
||||
|
||||
# 抑制第三方库的进度条和日志,仅保留解析结果输出
|
||||
os.environ["HF_HUB_DISABLE_PROGRESS_BARS"] = "1"
|
||||
os.environ["HF_HUB_DISABLE_TELEMETRY"] = "1"
|
||||
os.environ["TQDM_DISABLE"] = "1"
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
# 配置日志系统,只输出 ERROR 级别
|
||||
logging.basicConfig(level=logging.ERROR, format='%(levelname)s: %(message)s')
|
||||
|
||||
# 设置第三方库日志等级
|
||||
logging.getLogger('docling').setLevel(logging.ERROR)
|
||||
logging.getLogger('unstructured').setLevel(logging.ERROR)
|
||||
|
||||
from core import (
|
||||
FileDetectionError,
|
||||
ReaderNotFoundError,
|
||||
output_result,
|
||||
parse_input,
|
||||
process_content,
|
||||
)
|
||||
from readers import READERS
|
||||
|
||||
|
||||
def run_normal(args) -> None:
|
||||
"""正常执行模式:解析文件并输出结果"""
|
||||
# 实例化所有 readers
|
||||
readers = [ReaderCls() for ReaderCls in READERS]
|
||||
|
||||
try:
|
||||
content, failures = parse_input(args.input_path, readers)
|
||||
except FileDetectionError as e:
|
||||
print(f"错误: {e}")
|
||||
sys.exit(1)
|
||||
except ReaderNotFoundError as e:
|
||||
print(f"错误: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
if content is None:
|
||||
print("所有解析方法均失败:")
|
||||
for failure in failures:
|
||||
print(failure)
|
||||
sys.exit(1)
|
||||
|
||||
# 处理内容
|
||||
content = process_content(content)
|
||||
|
||||
# 输出结果
|
||||
output_result(content, args)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""主函数:解析命令行参数并执行"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="将 DOCX、XLS、XLSX、PPTX、PDF、HTML 文件或 URL 解析为 Markdown"
|
||||
)
|
||||
|
||||
parser.add_argument("input_path", help="DOCX、XLS、XLSX、PPTX、PDF、HTML 文件或 URL")
|
||||
|
||||
parser.add_argument(
|
||||
"-n",
|
||||
"--context",
|
||||
type=int,
|
||||
default=2,
|
||||
help="与 -s 配合使用,指定每个检索结果包含的前后行数(不包含空行)",
|
||||
)
|
||||
|
||||
group = parser.add_mutually_exclusive_group()
|
||||
group.add_argument(
|
||||
"-c", "--count", action="store_true", help="返回解析后的 markdown 文档的总字数"
|
||||
)
|
||||
group.add_argument(
|
||||
"-l", "--lines", action="store_true", help="返回解析后的 markdown 文档的总行数"
|
||||
)
|
||||
group.add_argument(
|
||||
"-t",
|
||||
"--titles",
|
||||
action="store_true",
|
||||
help="返回解析后的 markdown 文档的标题行(1-6级)",
|
||||
)
|
||||
group.add_argument(
|
||||
"-tc",
|
||||
"--title-content",
|
||||
help="指定标题名称,输出该标题及其下级内容(不包含#号)",
|
||||
)
|
||||
group.add_argument(
|
||||
"-s",
|
||||
"--search",
|
||||
help="使用正则表达式搜索文档,返回所有匹配结果(用---分隔)",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
run_normal(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -26,11 +26,11 @@ DEPENDENCIES = {
|
||||
"default": {
|
||||
"python": None,
|
||||
"dependencies": [
|
||||
"docling",
|
||||
"unstructured[pdf]",
|
||||
"markitdown[pdf]",
|
||||
"pypdf",
|
||||
"markdownify"
|
||||
"docling==2.80.0",
|
||||
"unstructured[pdf]==0.21.5",
|
||||
"markitdown[pdf]==0.1.5",
|
||||
"pypdf==6.9.0",
|
||||
"markdownify==1.2.2"
|
||||
]
|
||||
},
|
||||
"Darwin-x86_64": {
|
||||
@@ -39,9 +39,9 @@ DEPENDENCIES = {
|
||||
"docling==2.40.0",
|
||||
"docling-parse==4.0.0",
|
||||
"numpy<2",
|
||||
"markitdown[pdf]",
|
||||
"pypdf",
|
||||
"markdownify"
|
||||
"markitdown[pdf]==0.1.5",
|
||||
"pypdf==6.9.0",
|
||||
"markdownify==1.2.2"
|
||||
]
|
||||
}
|
||||
},
|
||||
@@ -49,12 +49,24 @@ DEPENDENCIES = {
|
||||
"default": {
|
||||
"python": None,
|
||||
"dependencies": [
|
||||
"docling",
|
||||
"unstructured[docx]",
|
||||
"markitdown[docx]",
|
||||
"pypandoc-binary",
|
||||
"python-docx",
|
||||
"markdownify"
|
||||
"docling==2.80.0",
|
||||
"unstructured[docx]==0.21.5",
|
||||
"markitdown[docx]==0.1.5",
|
||||
"pypandoc-binary==1.17",
|
||||
"python-docx==1.2.0",
|
||||
"markdownify==1.2.2"
|
||||
]
|
||||
},
|
||||
"Darwin-x86_64": {
|
||||
"python": "3.12",
|
||||
"dependencies": [
|
||||
"docling==2.40.0",
|
||||
"docling-parse==4.0.0",
|
||||
"numpy<2",
|
||||
"markitdown[docx]==0.1.5",
|
||||
"pypandoc-binary==1.17",
|
||||
"python-docx==1.2.0",
|
||||
"markdownify==1.2.2"
|
||||
]
|
||||
}
|
||||
},
|
||||
@@ -62,11 +74,24 @@ DEPENDENCIES = {
|
||||
"default": {
|
||||
"python": None,
|
||||
"dependencies": [
|
||||
"docling",
|
||||
"unstructured[xlsx]",
|
||||
"markitdown[xlsx]",
|
||||
"pandas",
|
||||
"tabulate"
|
||||
"docling==2.80.0",
|
||||
"unstructured[xlsx]==0.21.5",
|
||||
"markitdown[xlsx]==0.1.5",
|
||||
"pandas==3.0.1",
|
||||
"tabulate==0.10.0",
|
||||
"openpyxl==3.1.5"
|
||||
]
|
||||
},
|
||||
"Darwin-x86_64": {
|
||||
"python": "3.12",
|
||||
"dependencies": [
|
||||
"docling==2.40.0",
|
||||
"docling-parse==4.0.0",
|
||||
"numpy<2",
|
||||
"markitdown[xlsx]==0.1.5",
|
||||
"pandas<3.0.0",
|
||||
"tabulate==0.10.0",
|
||||
"openpyxl==3.1.5"
|
||||
]
|
||||
}
|
||||
},
|
||||
@@ -74,11 +99,22 @@ DEPENDENCIES = {
|
||||
"default": {
|
||||
"python": None,
|
||||
"dependencies": [
|
||||
"docling",
|
||||
"unstructured[pptx]",
|
||||
"markitdown[pptx]",
|
||||
"python-pptx",
|
||||
"markdownify"
|
||||
"docling==2.80.0",
|
||||
"unstructured[pptx]==0.21.5",
|
||||
"markitdown[pptx]==0.1.5",
|
||||
"python-pptx==1.0.2",
|
||||
"markdownify==1.2.2"
|
||||
]
|
||||
},
|
||||
"Darwin-x86_64": {
|
||||
"python": "3.12",
|
||||
"dependencies": [
|
||||
"docling==2.40.0",
|
||||
"docling-parse==4.0.0",
|
||||
"numpy<2",
|
||||
"markitdown[pptx]==0.1.5",
|
||||
"python-pptx==1.0.2",
|
||||
"markdownify==1.2.2"
|
||||
]
|
||||
}
|
||||
},
|
||||
@@ -86,15 +122,29 @@ DEPENDENCIES = {
|
||||
"default": {
|
||||
"python": None,
|
||||
"dependencies": [
|
||||
"trafilatura",
|
||||
"domscribe",
|
||||
"markitdown",
|
||||
"html2text",
|
||||
"beautifulsoup4",
|
||||
"httpx",
|
||||
"chardet",
|
||||
"pyppeteer",
|
||||
"selenium"
|
||||
"trafilatura==2.0.0",
|
||||
"domscribe==0.1.3",
|
||||
"markitdown==0.1.5",
|
||||
"html2text==2025.4.15",
|
||||
"beautifulsoup4==4.14.3",
|
||||
"httpx==0.28.1",
|
||||
"chardet==7.1.0",
|
||||
"pyppeteer==2.0.0",
|
||||
"selenium==4.25.0"
|
||||
]
|
||||
},
|
||||
"Darwin-x86_64": {
|
||||
"python": "3.12",
|
||||
"dependencies": [
|
||||
"trafilatura==2.0.0",
|
||||
"domscribe==0.1.3",
|
||||
"markitdown==0.1.5",
|
||||
"html2text==2025.4.15",
|
||||
"beautifulsoup4==4.14.3",
|
||||
"httpx==0.28.1",
|
||||
"chardet==7.1.0",
|
||||
"pyppeteer==2.0.0",
|
||||
"selenium==4.25.0"
|
||||
]
|
||||
}
|
||||
},
|
||||
@@ -102,12 +152,54 @@ DEPENDENCIES = {
|
||||
"default": {
|
||||
"python": None,
|
||||
"dependencies": [
|
||||
"unstructured[xlsx]",
|
||||
"markitdown[xls]",
|
||||
"pandas",
|
||||
"tabulate",
|
||||
"xlrd",
|
||||
"olefile"
|
||||
"unstructured[xlsx]==0.21.5",
|
||||
"markitdown[xls]==0.1.5",
|
||||
"pandas==3.0.1",
|
||||
"tabulate==0.10.0",
|
||||
"xlrd==2.0.2",
|
||||
"olefile==0.47"
|
||||
]
|
||||
},
|
||||
"Darwin-x86_64": {
|
||||
"python": "3.12",
|
||||
"dependencies": [
|
||||
"markitdown[xls]==0.1.5",
|
||||
"pandas<3.0.0",
|
||||
"tabulate==0.10.0",
|
||||
"xlrd==2.0.2",
|
||||
"olefile==0.47",
|
||||
"openpyxl==3.1.5"
|
||||
]
|
||||
}
|
||||
},
|
||||
"doc": {
|
||||
"default": {
|
||||
"python": None,
|
||||
"dependencies": []
|
||||
}
|
||||
},
|
||||
"ppt": {
|
||||
"default": {
|
||||
"python": None,
|
||||
"dependencies": [
|
||||
"docling==2.80.0",
|
||||
"unstructured[pptx]==0.21.5",
|
||||
"markitdown[pptx]==0.1.5",
|
||||
"python-pptx==1.0.2",
|
||||
"markdownify==1.2.2",
|
||||
"olefile==0.47"
|
||||
]
|
||||
},
|
||||
"Darwin-x86_64": {
|
||||
"python": "3.12",
|
||||
"dependencies": [
|
||||
"docling==2.40.0",
|
||||
"docling-parse==4.0.0",
|
||||
"numpy<2",
|
||||
"markitdown[pptx]==0.1.5",
|
||||
"python-pptx==1.0.2",
|
||||
"markdownify==1.2.2",
|
||||
"olefile==0.47"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
@@ -13,6 +13,8 @@ from readers import (
|
||||
PptxReader,
|
||||
HtmlReader,
|
||||
XlsReader,
|
||||
DocReader,
|
||||
PptReader,
|
||||
)
|
||||
|
||||
|
||||
@@ -24,6 +26,8 @@ _READER_KEY_MAP: Dict[Type[BaseReader], str] = {
|
||||
PptxReader: "pptx",
|
||||
HtmlReader: "html",
|
||||
XlsReader: "xls",
|
||||
DocReader: "doc",
|
||||
PptReader: "ppt",
|
||||
}
|
||||
|
||||
|
||||
@@ -92,7 +96,8 @@ def generate_uv_command(
|
||||
dependencies: list,
|
||||
input_path: str,
|
||||
python_version: Optional[str] = None,
|
||||
script_path: str = "scripts/lyxy_document_reader.py"
|
||||
script_path: str = "scripts/lyxy_document_reader.py",
|
||||
include_pyarmor: bool = True
|
||||
) -> str:
|
||||
"""
|
||||
生成 uv run 命令。
|
||||
@@ -102,6 +107,7 @@ def generate_uv_command(
|
||||
input_path: 输入文件路径或 URL
|
||||
python_version: 需要的 python 版本,None 表示不指定
|
||||
script_path: 脚本路径
|
||||
include_pyarmor: 是否包含 pyarmor 依赖
|
||||
|
||||
Returns:
|
||||
uv run 命令字符串
|
||||
@@ -111,8 +117,8 @@ def generate_uv_command(
|
||||
if python_version:
|
||||
parts.append(f"--python {python_version}")
|
||||
|
||||
# 始终添加 pyarmor 依赖(混淆后脚本需要)
|
||||
parts.append("--with pyarmor")
|
||||
if include_pyarmor:
|
||||
parts.append("--with pyarmor")
|
||||
|
||||
for dep in dependencies:
|
||||
# 处理包含空格的依赖(如 unstructured[pdf]),需要加引号
|
||||
@@ -126,10 +132,45 @@ def generate_uv_command(
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
def generate_uv_args(
|
||||
dependencies: list,
|
||||
script_path: str,
|
||||
python_version: Optional[str] = None,
|
||||
include_pyarmor: bool = True
|
||||
) -> list:
|
||||
"""
|
||||
生成 uv run 命令参数列表(用于 subprocess.run)。
|
||||
|
||||
Args:
|
||||
dependencies: 依赖包列表
|
||||
script_path: 脚本路径
|
||||
python_version: 需要的 python 版本,None 表示不指定
|
||||
include_pyarmor: 是否包含 pyarmor 依赖
|
||||
|
||||
Returns:
|
||||
uv run 命令参数列表
|
||||
"""
|
||||
args = ["uv", "run"]
|
||||
|
||||
if python_version:
|
||||
args.extend(["--python", python_version])
|
||||
|
||||
if include_pyarmor:
|
||||
args.extend(["--with", "pyarmor"])
|
||||
|
||||
for dep in dependencies:
|
||||
args.extend(["--with", dep])
|
||||
|
||||
args.append(script_path)
|
||||
|
||||
return args
|
||||
|
||||
|
||||
def generate_python_command(
|
||||
dependencies: list,
|
||||
input_path: str,
|
||||
script_path: str = "scripts/lyxy_document_reader.py"
|
||||
script_path: str = "scripts/lyxy_document_reader.py",
|
||||
include_pyarmor: bool = True
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
生成 python 命令和 pip 安装命令。
|
||||
@@ -138,14 +179,17 @@ def generate_python_command(
|
||||
dependencies: 依赖包列表
|
||||
input_path: 输入文件路径或 URL
|
||||
script_path: 脚本路径
|
||||
include_pyarmor: 是否包含 pyarmor 依赖
|
||||
|
||||
Returns:
|
||||
(python_command, pip_command) 元组
|
||||
"""
|
||||
python_cmd = f"python {script_path} {input_path}"
|
||||
|
||||
# 构建 pip install 命令,处理带引号的依赖,始终包含 pyarmor
|
||||
pip_parts = ["pip install", "pyarmor"]
|
||||
# 构建 pip install 命令,处理带引号的依赖
|
||||
pip_parts = ["pip install"]
|
||||
if include_pyarmor:
|
||||
pip_parts.append("pyarmor")
|
||||
for dep in dependencies:
|
||||
pip_parts.append(dep)
|
||||
pip_cmd = " ".join(pip_parts)
|
||||
|
||||
@@ -1,56 +1,36 @@
|
||||
#!/usr/bin/env python3
|
||||
"""文档解析器命令行交互模块,提供命令行接口。支持 DOCX、XLS、XLSX、PPTX、PDF、HTML 和 URL。"""
|
||||
"""文档解析器入口 - 环境检测和自启动"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
# 将 scripts/ 目录添加到 sys.path,支持从任意位置执行脚本
|
||||
scripts_dir = Path(__file__).resolve().parent
|
||||
# 确定项目根目录和脚本路径
|
||||
script_file = Path(__file__).resolve()
|
||||
scripts_dir = script_file.parent
|
||||
project_root = scripts_dir.parent
|
||||
bootstrap_path = str(scripts_dir / "bootstrap.py")
|
||||
|
||||
# 将 scripts/ 目录添加到 sys.path
|
||||
if str(scripts_dir) not in sys.path:
|
||||
sys.path.append(str(scripts_dir))
|
||||
|
||||
# 抑制第三方库的进度条和日志,仅保留解析结果输出
|
||||
# 抑制第三方库日志
|
||||
os.environ["HF_HUB_DISABLE_PROGRESS_BARS"] = "1"
|
||||
os.environ["HF_HUB_DISABLE_TELEMETRY"] = "1"
|
||||
os.environ["TQDM_DISABLE"] = "1"
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
# 配置日志系统,只输出 ERROR 级别
|
||||
logging.basicConfig(level=logging.ERROR, format='%(levelname)s: %(message)s')
|
||||
|
||||
# 设置第三方库日志等级
|
||||
logging.getLogger('docling').setLevel(logging.ERROR)
|
||||
logging.getLogger('unstructured').setLevel(logging.ERROR)
|
||||
|
||||
from core import (
|
||||
FileDetectionError,
|
||||
ReaderNotFoundError,
|
||||
output_result,
|
||||
parse_input,
|
||||
process_content,
|
||||
generate_advice,
|
||||
)
|
||||
from readers import READERS
|
||||
|
||||
|
||||
def main() -> None:
|
||||
def main():
|
||||
"""主函数:环境检测和决策"""
|
||||
# 解析命令行参数(轻量,仅识别必要参数)
|
||||
parser = argparse.ArgumentParser(
|
||||
description="将 DOCX、XLS、XLSX、PPTX、PDF、HTML 文件或 URL 解析为 Markdown"
|
||||
)
|
||||
|
||||
parser.add_argument("input_path", help="DOCX、XLS、XLSX、PPTX、PDF、HTML 文件或 URL")
|
||||
|
||||
parser.add_argument(
|
||||
"-a",
|
||||
"--advice",
|
||||
action="store_true",
|
||||
help="仅显示执行建议,不实际解析文件",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"-n",
|
||||
"--context",
|
||||
@@ -58,7 +38,6 @@ def main() -> None:
|
||||
default=2,
|
||||
help="与 -s 配合使用,指定每个检索结果包含的前后行数(不包含空行)",
|
||||
)
|
||||
|
||||
group = parser.add_mutually_exclusive_group()
|
||||
group.add_argument(
|
||||
"-c", "--count", action="store_true", help="返回解析后的 markdown 文档的总字数"
|
||||
@@ -85,39 +64,58 @@ def main() -> None:
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# 实例化所有 readers
|
||||
readers = [ReaderCls() for ReaderCls in READERS]
|
||||
# 检测 uv 是否可用
|
||||
uv_path = shutil.which("uv")
|
||||
|
||||
# --advice 模式:仅显示建议,不解析
|
||||
if args.advice:
|
||||
advice = generate_advice(args.input_path, readers, "scripts/lyxy_document_reader.py")
|
||||
if advice:
|
||||
print(advice)
|
||||
else:
|
||||
print(f"错误: 无法识别文件类型: {args.input_path}")
|
||||
sys.exit(1)
|
||||
if not uv_path:
|
||||
# uv 不可用,降级为直接执行 bootstrap.py
|
||||
import bootstrap
|
||||
bootstrap.run_normal(args)
|
||||
return
|
||||
|
||||
try:
|
||||
content, failures = parse_input(args.input_path, readers)
|
||||
except FileDetectionError as e:
|
||||
print(f"错误: {e}")
|
||||
sys.exit(1)
|
||||
except ReaderNotFoundError as e:
|
||||
print(f"错误: {e}")
|
||||
sys.exit(1)
|
||||
# uv 可用,需要自启动
|
||||
# 导入依赖检测模块
|
||||
from config import DEPENDENCIES
|
||||
from core.advice_generator import (
|
||||
detect_file_type_light,
|
||||
get_platform,
|
||||
get_dependencies,
|
||||
generate_uv_args,
|
||||
)
|
||||
from readers import READERS
|
||||
|
||||
if content is None:
|
||||
print("所有解析方法均失败:")
|
||||
for failure in failures:
|
||||
print(failure)
|
||||
sys.exit(1)
|
||||
# 检测文件类型
|
||||
readers = [ReaderCls() for ReaderCls in READERS]
|
||||
reader_cls = detect_file_type_light(args.input_path, readers)
|
||||
|
||||
# 处理内容
|
||||
content = process_content(content)
|
||||
if not reader_cls:
|
||||
# 无法识别文件类型,降级执行让它报错
|
||||
import bootstrap
|
||||
bootstrap.run_normal(args)
|
||||
return
|
||||
|
||||
# 输出结果
|
||||
output_result(content, args)
|
||||
# 获取平台和依赖配置
|
||||
platform_id = get_platform()
|
||||
python_version, dependencies = get_dependencies(reader_cls, platform_id)
|
||||
|
||||
# 生成 uv 命令参数列表
|
||||
uv_args = generate_uv_args(
|
||||
dependencies=dependencies,
|
||||
script_path=bootstrap_path,
|
||||
python_version=python_version,
|
||||
include_pyarmor=True
|
||||
)
|
||||
|
||||
# 添加所有命令行参数
|
||||
uv_args.extend(sys.argv[1:])
|
||||
|
||||
# 设置环境变量
|
||||
env = os.environ.copy()
|
||||
env["PYTHONPATH"] = str(project_root)
|
||||
|
||||
# 自启动:使用 subprocess 替代 execvpe(Windows 兼容)
|
||||
result = subprocess.run(uv_args, env=env, cwd=str(project_root))
|
||||
sys.exit(result.returncode)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -2,28 +2,34 @@
|
||||
|
||||
from .base import BaseReader
|
||||
from .docx import DocxReader
|
||||
from .doc import DocReader
|
||||
from .xlsx import XlsxReader
|
||||
from .pptx import PptxReader
|
||||
from .pdf import PdfReader
|
||||
from .html import HtmlReader
|
||||
from .xls import XlsReader
|
||||
from .ppt import PptReader
|
||||
|
||||
READERS = [
|
||||
DocxReader,
|
||||
DocReader,
|
||||
XlsxReader,
|
||||
PptxReader,
|
||||
PdfReader,
|
||||
HtmlReader,
|
||||
XlsReader,
|
||||
PptReader,
|
||||
]
|
||||
|
||||
__all__ = [
|
||||
"BaseReader",
|
||||
"DocxReader",
|
||||
"DocReader",
|
||||
"XlsxReader",
|
||||
"PptxReader",
|
||||
"PdfReader",
|
||||
"HtmlReader",
|
||||
"XlsReader",
|
||||
"PptReader",
|
||||
"READERS",
|
||||
]
|
||||
|
||||
@@ -4,6 +4,9 @@
|
||||
"""
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
import shutil
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from typing import List, Optional, Tuple
|
||||
@@ -63,6 +66,106 @@ def parse_via_docling(file_path: str) -> Tuple[Optional[str], Optional[str]]:
|
||||
return None, f"docling 解析失败: {str(e)}"
|
||||
|
||||
|
||||
def convert_via_libreoffice(
|
||||
input_path: str,
|
||||
target_format: str,
|
||||
output_dir: Path,
|
||||
output_suffix: Optional[str] = None,
|
||||
timeout: int = 60
|
||||
) -> Tuple[Optional[Path], Optional[str]]:
|
||||
"""使用 LibreOffice soffice 命令行转换文件格式。
|
||||
|
||||
Args:
|
||||
input_path: 输入文件路径
|
||||
target_format: 目标格式(如 "md", "pptx")
|
||||
output_dir: 输出目录(调用者负责生命周期管理)
|
||||
output_suffix: 可选,输出文件后缀(不指定则使用 target_format)
|
||||
timeout: 超时时间(秒)
|
||||
|
||||
Returns:
|
||||
(output_path, error_message): 成功时 (Path, None),失败时 (None, error)
|
||||
"""
|
||||
# 检测 soffice 是否在 PATH 中
|
||||
soffice_path = shutil.which("soffice")
|
||||
if not soffice_path:
|
||||
return None, "LibreOffice 未安装"
|
||||
|
||||
input_file = Path(input_path)
|
||||
suffix = output_suffix if output_suffix else target_format
|
||||
expected_output = output_dir / (input_file.stem + "." + suffix)
|
||||
|
||||
# 构建命令
|
||||
cmd = [
|
||||
soffice_path,
|
||||
"--headless",
|
||||
"--convert-to", target_format,
|
||||
"--outdir", str(output_dir),
|
||||
str(input_file)
|
||||
]
|
||||
|
||||
# 执行命令
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
return None, f"LibreOffice 转换超时 ({timeout}秒)"
|
||||
|
||||
# 检查返回码
|
||||
if result.returncode != 0:
|
||||
return None, f"LibreOffice 转换失败 (code: {result.returncode})"
|
||||
|
||||
# 检查输出文件是否存在
|
||||
output_file = None
|
||||
if expected_output.exists():
|
||||
output_file = expected_output
|
||||
else:
|
||||
# Fallback: 遍历目录找任意匹配后缀的文件
|
||||
pattern = "*." + suffix
|
||||
files = list(output_dir.glob(pattern))
|
||||
if files:
|
||||
output_file = files[0]
|
||||
|
||||
if not output_file:
|
||||
return None, "LibreOffice 未生成输出文件"
|
||||
|
||||
return output_file, None
|
||||
|
||||
|
||||
def parse_via_libreoffice(file_path: str) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""使用 LibreOffice soffice 命令行转换文件为 Markdown。
|
||||
|
||||
支持 .doc/.docx/.odt 等 LibreOffice 可处理的格式。
|
||||
|
||||
Args:
|
||||
file_path: 文件路径
|
||||
|
||||
Returns:
|
||||
(markdown_content, error_message): 成功时 (content, None),失败时 (None, error)
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
output_path, error = convert_via_libreoffice(
|
||||
input_path=file_path,
|
||||
target_format="md",
|
||||
output_dir=Path(temp_dir),
|
||||
timeout=60
|
||||
)
|
||||
if error:
|
||||
return None, error
|
||||
|
||||
# 读取输出内容
|
||||
content = output_path.read_text(encoding="utf-8", errors="replace")
|
||||
content = content.strip()
|
||||
|
||||
if not content:
|
||||
return None, "LibreOffice 输出为空"
|
||||
|
||||
return content, None
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 格式化工具
|
||||
# ============================================================================
|
||||
|
||||
46
scripts/readers/doc/__init__.py
Normal file
46
scripts/readers/doc/__init__.py
Normal file
@@ -0,0 +1,46 @@
|
||||
"""DOC 文件阅读器,使用 LibreOffice 解析。"""
|
||||
|
||||
import os
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
from readers.base import BaseReader
|
||||
from utils import is_valid_doc
|
||||
|
||||
from . import libreoffice
|
||||
|
||||
|
||||
PARSERS = [
|
||||
("LibreOffice", libreoffice.parse),
|
||||
]
|
||||
|
||||
|
||||
class DocReader(BaseReader):
|
||||
"""DOC 文件阅读器"""
|
||||
|
||||
def supports(self, file_path: str) -> bool:
|
||||
return file_path.lower().endswith('.doc')
|
||||
|
||||
def parse(self, file_path: str) -> Tuple[Optional[str], List[str]]:
|
||||
failures = []
|
||||
|
||||
# 检查文件是否存在
|
||||
if not os.path.exists(file_path):
|
||||
return None, ["文件不存在"]
|
||||
|
||||
# 验证文件格式
|
||||
if not is_valid_doc(file_path):
|
||||
return None, ["不是有效的 DOC 文件"]
|
||||
|
||||
content = None
|
||||
|
||||
for parser_name, parser_func in PARSERS:
|
||||
try:
|
||||
content, error = parser_func(file_path)
|
||||
if content is not None:
|
||||
return content, failures
|
||||
else:
|
||||
failures.append(f"- {parser_name}: {error}")
|
||||
except Exception as e:
|
||||
failures.append(f"- {parser_name}: [意外异常] {type(e).__name__}: {str(e)}")
|
||||
|
||||
return None, failures
|
||||
9
scripts/readers/doc/libreoffice.py
Normal file
9
scripts/readers/doc/libreoffice.py
Normal file
@@ -0,0 +1,9 @@
|
||||
"""使用 LibreOffice soffice 命令行解析 DOC 文件"""
|
||||
|
||||
from typing import Optional, Tuple
|
||||
from readers._utils import parse_via_libreoffice
|
||||
|
||||
|
||||
def parse(file_path: str) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""使用 LibreOffice soffice 解析 DOC 文件"""
|
||||
return parse_via_libreoffice(file_path)
|
||||
@@ -10,6 +10,7 @@ from . import docling
|
||||
from . import unstructured
|
||||
from . import markitdown
|
||||
from . import pypandoc
|
||||
from . import libreoffice
|
||||
from . import python_docx
|
||||
from . import native_xml
|
||||
|
||||
@@ -19,6 +20,7 @@ PARSERS = [
|
||||
("unstructured", unstructured.parse),
|
||||
("pypandoc-binary", pypandoc.parse),
|
||||
("MarkItDown", markitdown.parse),
|
||||
("LibreOffice", libreoffice.parse),
|
||||
("python-docx", python_docx.parse),
|
||||
("XML 原生解析", native_xml.parse),
|
||||
]
|
||||
|
||||
9
scripts/readers/docx/libreoffice.py
Normal file
9
scripts/readers/docx/libreoffice.py
Normal file
@@ -0,0 +1,9 @@
|
||||
"""使用 LibreOffice soffice 命令行解析 DOCX 文件"""
|
||||
|
||||
from typing import Optional, Tuple
|
||||
from readers._utils import parse_via_libreoffice
|
||||
|
||||
|
||||
def parse(file_path: str) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""使用 LibreOffice soffice 解析 DOCX 文件"""
|
||||
return parse_via_libreoffice(file_path)
|
||||
46
scripts/readers/ppt/__init__.py
Normal file
46
scripts/readers/ppt/__init__.py
Normal file
@@ -0,0 +1,46 @@
|
||||
"""PPT 文件阅读器,使用 LibreOffice 解析。"""
|
||||
|
||||
import os
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
from readers.base import BaseReader
|
||||
from utils import is_valid_ppt
|
||||
|
||||
from . import libreoffice
|
||||
|
||||
|
||||
PARSERS = [
|
||||
("LibreOffice", libreoffice.parse),
|
||||
]
|
||||
|
||||
|
||||
class PptReader(BaseReader):
|
||||
"""PPT 文件阅读器"""
|
||||
|
||||
def supports(self, file_path: str) -> bool:
|
||||
return file_path.lower().endswith('.ppt')
|
||||
|
||||
def parse(self, file_path: str) -> Tuple[Optional[str], List[str]]:
|
||||
failures = []
|
||||
|
||||
# 检查文件是否存在
|
||||
if not os.path.exists(file_path):
|
||||
return None, ["文件不存在"]
|
||||
|
||||
# 验证文件格式
|
||||
if not is_valid_ppt(file_path):
|
||||
return None, ["不是有效的 PPT 文件"]
|
||||
|
||||
content = None
|
||||
|
||||
for parser_name, parser_func in PARSERS:
|
||||
try:
|
||||
content, error = parser_func(file_path)
|
||||
if content is not None:
|
||||
return content, failures
|
||||
else:
|
||||
failures.append(f"- {parser_name}: {error}")
|
||||
except Exception as e:
|
||||
failures.append(f"- {parser_name}: [意外异常] {type(e).__name__}: {str(e)}")
|
||||
|
||||
return None, failures
|
||||
37
scripts/readers/ppt/libreoffice.py
Normal file
37
scripts/readers/ppt/libreoffice.py
Normal file
@@ -0,0 +1,37 @@
|
||||
"""使用 LibreOffice soffice 命令行转换 PPT 为 PPTX 后复用 PptxReader 解析"""
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from readers._utils import convert_via_libreoffice
|
||||
from readers.pptx import PptxReader
|
||||
|
||||
|
||||
def parse(file_path: str) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""使用 LibreOffice soffice 解析 PPT 文件
|
||||
|
||||
Args:
|
||||
file_path: PPT 文件路径
|
||||
|
||||
Returns:
|
||||
(markdown_content, error_message): 成功时 (content, None),失败时 (None, error)
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# 将 PPT 转换为 PPTX
|
||||
pptx_path, error = convert_via_libreoffice(
|
||||
input_path=file_path,
|
||||
target_format="pptx",
|
||||
output_dir=Path(temp_dir),
|
||||
timeout=60
|
||||
)
|
||||
if error:
|
||||
return None, error
|
||||
|
||||
# 复用 PptxReader 解析转换后的 PPTX
|
||||
reader = PptxReader()
|
||||
content, failures = reader.parse(str(pptx_path))
|
||||
if content is not None:
|
||||
return content, None
|
||||
else:
|
||||
return None, f"转换成功但 PPTX 解析失败: {failures}"
|
||||
@@ -1,21 +1,25 @@
|
||||
"""Utils module for lyxy-document."""
|
||||
|
||||
from .file_detection import (
|
||||
is_valid_doc,
|
||||
is_valid_docx,
|
||||
is_valid_pptx,
|
||||
is_valid_xlsx,
|
||||
is_valid_pdf,
|
||||
is_valid_xls,
|
||||
is_valid_ppt,
|
||||
is_html_file,
|
||||
is_url,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"is_valid_doc",
|
||||
"is_valid_docx",
|
||||
"is_valid_pptx",
|
||||
"is_valid_xlsx",
|
||||
"is_valid_pdf",
|
||||
"is_valid_xls",
|
||||
"is_valid_ppt",
|
||||
"is_html_file",
|
||||
"is_url",
|
||||
]
|
||||
|
||||
@@ -6,7 +6,7 @@ from typing import List, Optional
|
||||
|
||||
|
||||
def _is_valid_ole(file_path: str) -> bool:
|
||||
"""验证 OLE2 格式文件(XLS)"""
|
||||
"""验证 OLE2 格式文件(XLS/DOC)"""
|
||||
try:
|
||||
import olefile
|
||||
except ImportError:
|
||||
@@ -53,6 +53,16 @@ def is_valid_xls(file_path: str) -> bool:
|
||||
return _is_valid_ole(file_path)
|
||||
|
||||
|
||||
def is_valid_doc(file_path: str) -> bool:
|
||||
"""验证文件是否为有效的 DOC 格式(OLE2)"""
|
||||
return _is_valid_ole(file_path)
|
||||
|
||||
|
||||
def is_valid_ppt(file_path: str) -> bool:
|
||||
"""验证文件是否为有效的 PPT 格式(OLE2)"""
|
||||
return _is_valid_ole(file_path)
|
||||
|
||||
|
||||
def is_valid_pdf(file_path: str) -> bool:
|
||||
"""验证文件是否为有效的 PDF 格式"""
|
||||
try:
|
||||
|
||||
@@ -51,10 +51,7 @@ def temp_docx(tmp_path):
|
||||
str: 临时文件路径
|
||||
"""
|
||||
def _create_docx(paragraphs=None, headings=None, table_data=None, list_items=None):
|
||||
try:
|
||||
from docx import Document
|
||||
except ImportError:
|
||||
pytest.skip("python-docx 未安装")
|
||||
from docx import Document
|
||||
|
||||
doc = Document()
|
||||
|
||||
@@ -99,23 +96,38 @@ def temp_pdf(tmp_path):
|
||||
str: 临时文件路径
|
||||
"""
|
||||
def _create_pdf(text=None, lines=None):
|
||||
try:
|
||||
from reportlab.pdfgen import canvas
|
||||
from reportlab.lib.pagesizes import letter
|
||||
from reportlab.pdfbase import pdfmetrics
|
||||
from reportlab.pdfbase.ttfonts import TTFont
|
||||
except ImportError:
|
||||
pytest.skip("reportlab 未安装")
|
||||
from reportlab.pdfgen import canvas
|
||||
from reportlab.lib.pagesizes import letter
|
||||
from reportlab.pdfbase import pdfmetrics
|
||||
from reportlab.pdfbase.ttfonts import TTFont
|
||||
|
||||
file_path = tmp_path / "test.pdf"
|
||||
c = canvas.Canvas(str(file_path), pagesize=letter)
|
||||
|
||||
# 尝试注册中文字体(如果可用)
|
||||
font_loaded = False
|
||||
try:
|
||||
# 使用系统字体
|
||||
pdfmetrics.registerFont(TTFont('SimSun', 'simsun.ttc'))
|
||||
c.setFont('SimSun', 12)
|
||||
except:
|
||||
# 尝试 macOS 中文字体
|
||||
for font_name, font_path, font_index in [
|
||||
('PingFangSC', '/System/Library/Fonts/PingFang.ttc', 0),
|
||||
('STHeiti', '/System/Library/Fonts/STHeiti Light.ttc', 0),
|
||||
('STHeitiMedium', '/System/Library/Fonts/STHeiti Medium.ttc', 0),
|
||||
]:
|
||||
try:
|
||||
from reportlab.pdfbase.ttfonts import TTFont
|
||||
import os
|
||||
if os.path.exists(font_path):
|
||||
# For TTC files, we need to specify the font index
|
||||
pdfmetrics.registerFont(TTFont(font_name, font_path, subfontIndex=font_index))
|
||||
c.setFont(font_name, 12)
|
||||
font_loaded = True
|
||||
break
|
||||
except Exception as e:
|
||||
continue
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if not font_loaded:
|
||||
# 回退到默认字体
|
||||
c.setFont('Helvetica', 12)
|
||||
|
||||
@@ -176,10 +188,7 @@ def temp_pptx(tmp_path):
|
||||
str: 临时文件路径
|
||||
"""
|
||||
def _create_pptx(slides=None):
|
||||
try:
|
||||
from pptx import Presentation
|
||||
except ImportError:
|
||||
pytest.skip("python-pptx 未安装")
|
||||
from pptx import Presentation
|
||||
|
||||
prs = Presentation()
|
||||
|
||||
@@ -209,10 +218,7 @@ def temp_xlsx(tmp_path):
|
||||
str: 临时文件路径
|
||||
"""
|
||||
def _create_xlsx(data=None):
|
||||
try:
|
||||
import pandas as pd
|
||||
except ImportError:
|
||||
pytest.skip("pandas 未安装")
|
||||
import pandas as pd
|
||||
|
||||
file_path = tmp_path / "test.xlsx"
|
||||
|
||||
|
||||
@@ -29,7 +29,9 @@ def cli_runner():
|
||||
if str(scripts_dir) not in sys.path:
|
||||
sys.path.insert(0, str(scripts_dir))
|
||||
|
||||
from lyxy_document_reader import main
|
||||
# 直接调用 bootstrap.main() 而不是 lyxy_document_reader.main()
|
||||
# 因为 lyxy_document_reader 会调用 subprocess,无法捕获输出
|
||||
from bootstrap import main
|
||||
|
||||
# 保存原始 sys.argv 和 sys.exit
|
||||
original_argv = sys.argv
|
||||
@@ -46,7 +48,7 @@ def cli_runner():
|
||||
|
||||
try:
|
||||
# 设置命令行参数
|
||||
sys.argv = ['lyxy_document_reader'] + args
|
||||
sys.argv = ['bootstrap'] + args
|
||||
sys.exit = mock_exit
|
||||
|
||||
# 捕获输出
|
||||
|
||||
@@ -4,48 +4,6 @@ import pytest
|
||||
import os
|
||||
|
||||
|
||||
class TestCLIAdviceOption:
|
||||
"""测试 CLI --advice 参数功能。"""
|
||||
|
||||
def test_advice_option_pdf(self, cli_runner):
|
||||
"""测试 -a/--advice 选项对 PDF 文件。"""
|
||||
stdout, stderr, exit_code = cli_runner(["test.pdf", "-a"])
|
||||
|
||||
assert exit_code == 0
|
||||
assert "文件类型: PDF" in stdout
|
||||
assert "[uv 命令]" in stdout
|
||||
assert "[python 命令]" in stdout
|
||||
|
||||
def test_advice_option_docx(self, cli_runner):
|
||||
"""测试 --advice 选项对 DOCX 文件。"""
|
||||
stdout, stderr, exit_code = cli_runner(["test.docx", "--advice"])
|
||||
|
||||
assert exit_code == 0
|
||||
assert "文件类型: DOCX" in stdout
|
||||
|
||||
def test_advice_option_url(self, cli_runner):
|
||||
"""测试 --advice 选项对 URL。"""
|
||||
stdout, stderr, exit_code = cli_runner(["https://example.com", "--advice"])
|
||||
|
||||
assert exit_code == 0
|
||||
assert "文件类型: HTML" in stdout
|
||||
|
||||
def test_advice_option_unknown(self, cli_runner):
|
||||
"""测试 --advice 选项对未知文件类型。"""
|
||||
stdout, stderr, exit_code = cli_runner(["test.xyz", "--advice"])
|
||||
|
||||
assert exit_code != 0
|
||||
output = stdout + stderr
|
||||
assert "无法识别" in output or "错误" in output
|
||||
|
||||
def test_advice_option_xls(self, cli_runner):
|
||||
"""测试 --advice 选项对 XLS 文件。"""
|
||||
stdout, stderr, exit_code = cli_runner(["test.xls", "--advice"])
|
||||
|
||||
assert exit_code == 0
|
||||
assert "文件类型: XLS" in stdout
|
||||
|
||||
|
||||
class TestCLIDefaultOutput:
|
||||
"""测试 CLI 默认输出功能。"""
|
||||
|
||||
|
||||
53
tests/test_cli/test_path_resolution.py
Normal file
53
tests/test_cli/test_path_resolution.py
Normal file
@@ -0,0 +1,53 @@
|
||||
"""测试路径解析功能 - 验证从任意路径调用脚本。"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class TestPathResolution:
|
||||
"""测试路径解析逻辑。"""
|
||||
|
||||
def test_project_root_detection(self):
|
||||
"""测试项目根目录检测逻辑。"""
|
||||
# 模拟 lyxy_document_reader.py 中的路径计算逻辑
|
||||
# 获取当前测试文件的路径,然后向上找到项目根
|
||||
test_file = Path(__file__).resolve()
|
||||
tests_dir = test_file.parent.parent # tests/
|
||||
project_root = tests_dir.parent # 项目根
|
||||
|
||||
# 验证我们能正确找到项目根
|
||||
assert (project_root / "scripts").exists()
|
||||
assert (project_root / "scripts" / "lyxy_document_reader.py").exists()
|
||||
assert (project_root / "scripts" / "bootstrap.py").exists()
|
||||
|
||||
def test_bootstrap_path_absolute(self):
|
||||
"""测试 bootstrap.py 路径是绝对路径。"""
|
||||
# 模拟 lyxy_document_reader.py 中的路径计算
|
||||
test_file = Path(__file__).resolve()
|
||||
project_root = test_file.parent.parent.parent # 从 tests/test_cli/ 向上两级
|
||||
scripts_dir = project_root / "scripts"
|
||||
bootstrap_path = scripts_dir / "bootstrap.py"
|
||||
|
||||
# 验证路径是绝对路径
|
||||
assert bootstrap_path.is_absolute()
|
||||
assert bootstrap_path.exists()
|
||||
|
||||
def test_path_independent_from_cwd(self, monkeypatch, tmp_path):
|
||||
"""测试路径计算不依赖当前工作目录。"""
|
||||
# 保存原始路径
|
||||
test_file = Path(__file__).resolve()
|
||||
project_root = test_file.parent.parent.parent
|
||||
scripts_dir = project_root / "scripts"
|
||||
|
||||
# 切换到临时目录
|
||||
monkeypatch.chdir(tmp_path)
|
||||
|
||||
# 即使在临时目录,我们仍然能通过 __file__ 找到正确的路径
|
||||
# 这里我们模拟 lyxy_document_reader.py 中的逻辑
|
||||
# 注意:实际中 __file__ 是脚本本身的路径,不是测试文件的路径
|
||||
# 这里我们验证原理:__file__ 给出的是脚本位置,与 cwd 无关
|
||||
|
||||
# 验证 scripts_dir 和 bootstrap_path 的计算只依赖 __file__
|
||||
# 这个测试验证的是概念,不是实际的脚本导入
|
||||
assert scripts_dir.is_absolute()
|
||||
assert (scripts_dir / "bootstrap.py").exists()
|
||||
@@ -70,19 +70,22 @@ class TestGetDependencies:
|
||||
python_ver, deps = get_dependencies(DocxReader, "Unknown-Platform")
|
||||
assert python_ver is None
|
||||
assert len(deps) > 0
|
||||
assert "docling" in deps
|
||||
# 检查是否有 docling 相关依赖(可能带版本号)
|
||||
assert any(dep.startswith("docling") for dep in deps)
|
||||
|
||||
def test_get_pdf_dependencies(self):
|
||||
"""测试获取 PDF 依赖。"""
|
||||
python_ver, deps = get_dependencies(PdfReader, "Darwin-arm64")
|
||||
assert python_ver is None
|
||||
assert "docling" in deps
|
||||
# 检查是否有 docling 相关依赖(可能带版本号)
|
||||
assert any(dep.startswith("docling") for dep in deps)
|
||||
|
||||
def test_get_html_dependencies(self):
|
||||
"""测试获取 HTML 依赖。"""
|
||||
python_ver, deps = get_dependencies(HtmlReader, "Linux-x86_64")
|
||||
assert python_ver is None
|
||||
assert "trafilatura" in deps
|
||||
# 检查是否有 trafilatura 相关依赖(可能带版本号)
|
||||
assert any(dep.startswith("trafilatura") for dep in deps)
|
||||
|
||||
|
||||
class TestGenerateUvCommand:
|
||||
@@ -131,7 +134,7 @@ class TestGeneratePythonCommand:
|
||||
script_path="scripts/lyxy_document_reader.py"
|
||||
)
|
||||
assert python_cmd == "python scripts/lyxy_document_reader.py input.pdf"
|
||||
assert pip_cmd == "pip install pkg1 pkg2"
|
||||
assert pip_cmd == "pip install pyarmor pkg1 pkg2"
|
||||
|
||||
|
||||
class TestFormatAdvice:
|
||||
|
||||
233
tests/test_core/test_markdown_extra.py
Normal file
233
tests/test_core/test_markdown_extra.py
Normal file
@@ -0,0 +1,233 @@
|
||||
"""测试 markdown 模块的高级功能(extract_title_content, search_markdown)。"""
|
||||
|
||||
import pytest
|
||||
|
||||
from core.markdown import extract_title_content, search_markdown
|
||||
|
||||
|
||||
class TestExtractTitleContent:
|
||||
"""测试 extract_title_content 函数。"""
|
||||
|
||||
def test_extract_simple_title(self):
|
||||
"""测试提取简单标题。"""
|
||||
markdown = """# 目标标题
|
||||
|
||||
这是标题下的内容。
|
||||
第二段内容。"""
|
||||
|
||||
result = extract_title_content(markdown, "目标标题")
|
||||
|
||||
assert result is not None
|
||||
assert "# 目标标题" in result
|
||||
assert "这是标题下的内容" in result
|
||||
|
||||
def test_extract_with_subtitles(self):
|
||||
"""测试提取包含子标题的内容。"""
|
||||
markdown = """# 目标标题
|
||||
|
||||
这是标题下的内容。
|
||||
|
||||
## 子标题
|
||||
|
||||
子标题下的内容。
|
||||
|
||||
### 孙子标题
|
||||
|
||||
更深层的内容。"""
|
||||
|
||||
result = extract_title_content(markdown, "目标标题")
|
||||
|
||||
assert result is not None
|
||||
assert "# 目标标题" in result
|
||||
assert "## 子标题" in result
|
||||
assert "### 孙子标题" in result
|
||||
|
||||
def test_extract_stop_at_sibling_title(self):
|
||||
"""测试在同级标题处停止。"""
|
||||
markdown = """# 目标标题
|
||||
|
||||
目标内容。
|
||||
|
||||
# 另一个标题
|
||||
|
||||
另一个内容。"""
|
||||
|
||||
result = extract_title_content(markdown, "目标标题")
|
||||
|
||||
assert result is not None
|
||||
assert "# 目标标题" in result
|
||||
assert "目标内容" in result
|
||||
assert "# 另一个标题" not in result
|
||||
|
||||
def test_extract_with_parent_titles(self):
|
||||
"""测试包含父级标题。"""
|
||||
markdown = """# 父级标题
|
||||
|
||||
父级内容。
|
||||
|
||||
## 目标标题
|
||||
|
||||
目标内容。
|
||||
|
||||
### 子标题
|
||||
|
||||
子内容。"""
|
||||
|
||||
result = extract_title_content(markdown, "目标标题")
|
||||
|
||||
assert result is not None
|
||||
assert "# 父级标题" in result
|
||||
assert "## 目标标题" in result
|
||||
assert "### 子标题" in result
|
||||
|
||||
def test_extract_multiple_matches(self):
|
||||
"""测试多个匹配标题的情况。"""
|
||||
markdown = """# 第一章
|
||||
|
||||
## 目标标题
|
||||
|
||||
第一章的目标内容。
|
||||
|
||||
# 第二章
|
||||
|
||||
## 目标标题
|
||||
|
||||
第二章的目标内容。"""
|
||||
|
||||
result = extract_title_content(markdown, "目标标题")
|
||||
|
||||
assert result is not None
|
||||
assert "第一章的目标内容" in result
|
||||
assert "第二章的目标内容" in result
|
||||
assert "---" in result
|
||||
|
||||
def test_title_not_found(self):
|
||||
"""测试标题不存在的情况。"""
|
||||
markdown = "# 其他标题\n内容"
|
||||
|
||||
result = extract_title_content(markdown, "不存在的标题")
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_deep_nested_title(self):
|
||||
"""测试深层嵌套标题。"""
|
||||
markdown = """# H1
|
||||
|
||||
## H2
|
||||
|
||||
### H3
|
||||
|
||||
#### 目标标题
|
||||
|
||||
目标内容。"""
|
||||
|
||||
result = extract_title_content(markdown, "目标标题")
|
||||
|
||||
assert result is not None
|
||||
assert "# H1" in result
|
||||
assert "## H2" in result
|
||||
assert "### H3" in result
|
||||
assert "#### 目标标题" in result
|
||||
|
||||
|
||||
class TestSearchMarkdown:
|
||||
"""测试 search_markdown 函数。"""
|
||||
|
||||
def test_search_simple_pattern(self):
|
||||
"""测试简单搜索模式。"""
|
||||
content = """第一行
|
||||
第二行
|
||||
包含关键词的行
|
||||
第四行"""
|
||||
|
||||
result = search_markdown(content, "关键词", context_lines=0)
|
||||
|
||||
assert result is not None
|
||||
assert "关键词" in result
|
||||
|
||||
def test_search_with_context(self):
|
||||
"""测试带上下文的搜索。"""
|
||||
content = """行1
|
||||
行2
|
||||
关键词行
|
||||
行4
|
||||
行5"""
|
||||
|
||||
result = search_markdown(content, "关键词", context_lines=1)
|
||||
|
||||
assert result is not None
|
||||
assert "关键词" in result
|
||||
assert "行2" in result or "行4" in result
|
||||
|
||||
def test_search_no_match(self):
|
||||
"""测试无匹配的情况。"""
|
||||
content = "普通内容"
|
||||
|
||||
result = search_markdown(content, "不存在的内容", context_lines=0)
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_search_empty_content(self):
|
||||
"""测试空内容。"""
|
||||
result = search_markdown("", "关键词", context_lines=0)
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_search_invalid_regex(self):
|
||||
"""测试无效正则表达式。"""
|
||||
content = "内容"
|
||||
|
||||
result = search_markdown(content, "[invalid", context_lines=0)
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_search_negative_context(self):
|
||||
"""测试负的上下文行数。"""
|
||||
content = "内容"
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
search_markdown(content, "内容", context_lines=-1)
|
||||
|
||||
def test_search_multiple_matches_merged(self):
|
||||
"""测试多个匹配合并。"""
|
||||
content = """行1
|
||||
行2
|
||||
匹配1
|
||||
行4
|
||||
行5
|
||||
匹配2
|
||||
行7
|
||||
行8"""
|
||||
|
||||
result = search_markdown(content, "匹配", context_lines=1)
|
||||
|
||||
assert result is not None
|
||||
assert "匹配1" in result
|
||||
assert "匹配2" in result
|
||||
|
||||
def test_search_ignore_blank_lines_in_context(self):
|
||||
"""测试上下文计算忽略空行。"""
|
||||
content = """行1
|
||||
|
||||
行2
|
||||
关键词
|
||||
|
||||
行4
|
||||
行5"""
|
||||
|
||||
result = search_markdown(content, "关键词", context_lines=1)
|
||||
|
||||
assert result is not None
|
||||
assert "关键词" in result
|
||||
|
||||
def test_search_with_regex(self):
|
||||
"""测试使用正则表达式搜索。"""
|
||||
content = """apple
|
||||
banana
|
||||
cherry
|
||||
date"""
|
||||
|
||||
result = search_markdown(content, "^b", context_lines=0)
|
||||
|
||||
assert result is not None
|
||||
assert "banana" in result
|
||||
256
tests/test_core/test_parser.py
Normal file
256
tests/test_core/test_parser.py
Normal file
@@ -0,0 +1,256 @@
|
||||
"""测试 parser 模块的解析调度功能。"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import patch, MagicMock
|
||||
import argparse
|
||||
import sys
|
||||
|
||||
from core.parser import parse_input, process_content, output_result
|
||||
from core.exceptions import FileDetectionError, ReaderNotFoundError
|
||||
|
||||
|
||||
class MockReader:
|
||||
"""模拟 Reader 类用于测试。"""
|
||||
|
||||
def __init__(self, supports=True, content=None, failures=None):
|
||||
self._supports = supports
|
||||
self._content = content
|
||||
self._failures = failures or []
|
||||
|
||||
def supports(self, file_path):
|
||||
return self._supports
|
||||
|
||||
def parse(self, file_path):
|
||||
return self._content, self._failures
|
||||
|
||||
|
||||
class TestParseInput:
|
||||
"""测试 parse_input 函数。"""
|
||||
|
||||
def test_parse_input_success(self):
|
||||
"""测试成功解析的情况。"""
|
||||
reader = MockReader(supports=True, content="测试内容", failures=[])
|
||||
readers = [reader]
|
||||
|
||||
content, failures = parse_input("test.docx", readers)
|
||||
|
||||
assert content == "测试内容"
|
||||
assert failures == []
|
||||
|
||||
def test_parse_input_reader_not_found(self):
|
||||
"""测试没有找到支持的 reader。"""
|
||||
reader = MockReader(supports=False)
|
||||
readers = [reader]
|
||||
|
||||
with pytest.raises(ReaderNotFoundError):
|
||||
parse_input("test.docx", readers)
|
||||
|
||||
def test_parse_input_empty_path(self):
|
||||
"""测试空输入路径。"""
|
||||
readers = [MockReader()]
|
||||
|
||||
with pytest.raises(FileDetectionError):
|
||||
parse_input("", readers)
|
||||
|
||||
def test_parse_input_multiple_readers_first_succeeds(self):
|
||||
"""测试多个 reader,第一个成功。"""
|
||||
reader1 = MockReader(supports=True, content="第一个结果", failures=[])
|
||||
reader2 = MockReader(supports=True, content="第二个结果", failures=[])
|
||||
readers = [reader1, reader2]
|
||||
|
||||
content, failures = parse_input("test.docx", readers)
|
||||
|
||||
assert content == "第一个结果"
|
||||
|
||||
def test_parse_input_with_failures(self):
|
||||
"""测试解析返回失败信息。"""
|
||||
reader = MockReader(
|
||||
supports=True,
|
||||
content=None,
|
||||
failures=["解析器1失败", "解析器2失败"]
|
||||
)
|
||||
readers = [reader]
|
||||
|
||||
content, failures = parse_input("test.docx", readers)
|
||||
|
||||
assert content is None
|
||||
assert failures == ["解析器1失败", "解析器2失败"]
|
||||
|
||||
|
||||
class TestProcessContent:
|
||||
"""测试 process_content 函数。"""
|
||||
|
||||
def test_process_content_removes_images(self):
|
||||
"""测试移除图片标记。"""
|
||||
content = "测试内容  更多内容"
|
||||
result = process_content(content)
|
||||
|
||||
assert "" not in result
|
||||
assert "测试内容" in result
|
||||
assert "更多内容" in result
|
||||
|
||||
def test_process_content_normalizes_whitespace(self):
|
||||
"""测试规范化空白字符。"""
|
||||
content = "line1\n\n\n\nline2\n\n\nline3"
|
||||
result = process_content(content)
|
||||
|
||||
assert "line1\n\nline2\n\nline3" in result
|
||||
|
||||
def test_process_content_both_operations(self):
|
||||
"""测试同时执行两个操作。"""
|
||||
content = "\n\n\n\n正文"
|
||||
result = process_content(content)
|
||||
|
||||
assert "" not in result
|
||||
assert "\n\n\n\n" not in result
|
||||
|
||||
|
||||
class TestOutputResult:
|
||||
"""测试 output_result 函数。"""
|
||||
|
||||
def test_output_default(self, capsys):
|
||||
"""测试默认输出内容。"""
|
||||
args = argparse.Namespace(
|
||||
count=False,
|
||||
lines=False,
|
||||
titles=False,
|
||||
title_content=None,
|
||||
search=None,
|
||||
context=2
|
||||
)
|
||||
|
||||
output_result("测试内容", args)
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert "测试内容" in captured.out
|
||||
|
||||
def test_output_count(self, capsys):
|
||||
"""测试字数统计。"""
|
||||
args = argparse.Namespace(
|
||||
count=True,
|
||||
lines=False,
|
||||
titles=False,
|
||||
title_content=None,
|
||||
search=None,
|
||||
context=2
|
||||
)
|
||||
|
||||
output_result("测试内容", args)
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert captured.out.strip() == "4"
|
||||
|
||||
def test_output_lines(self, capsys):
|
||||
"""测试行数统计。"""
|
||||
args = argparse.Namespace(
|
||||
count=False,
|
||||
lines=True,
|
||||
titles=False,
|
||||
title_content=None,
|
||||
search=None,
|
||||
context=2
|
||||
)
|
||||
|
||||
output_result("line1\nline2\nline3", args)
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert captured.out.strip() == "3"
|
||||
|
||||
def test_output_titles(self, capsys):
|
||||
"""测试提取标题。"""
|
||||
args = argparse.Namespace(
|
||||
count=False,
|
||||
lines=False,
|
||||
titles=True,
|
||||
title_content=None,
|
||||
search=None,
|
||||
context=2
|
||||
)
|
||||
|
||||
content = "# 标题1\n正文\n## 标题2\n正文"
|
||||
output_result(content, args)
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert "# 标题1" in captured.out
|
||||
assert "## 标题2" in captured.out
|
||||
|
||||
def test_output_title_content_found(self, capsys):
|
||||
"""测试提取标题内容(找到)。"""
|
||||
args = argparse.Namespace(
|
||||
count=False,
|
||||
lines=False,
|
||||
titles=False,
|
||||
title_content="目标标题",
|
||||
search=None,
|
||||
context=2
|
||||
)
|
||||
|
||||
content = "# 目标标题\n标题下的内容"
|
||||
|
||||
with patch("sys.exit") as mock_exit:
|
||||
output_result(content, args)
|
||||
mock_exit.assert_not_called()
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert "目标标题" in captured.out
|
||||
assert "标题下的内容" in captured.out
|
||||
|
||||
def test_output_title_content_not_found(self, capsys):
|
||||
"""测试提取标题内容(未找到)。"""
|
||||
args = argparse.Namespace(
|
||||
count=False,
|
||||
lines=False,
|
||||
titles=False,
|
||||
title_content="不存在的标题",
|
||||
search=None,
|
||||
context=2
|
||||
)
|
||||
|
||||
content = "# 标题1\n内容"
|
||||
|
||||
with patch("sys.exit") as mock_exit:
|
||||
output_result(content, args)
|
||||
mock_exit.assert_called_once_with(1)
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert "未找到" in captured.out or "错误" in captured.out
|
||||
|
||||
def test_output_search_found(self, capsys):
|
||||
"""测试搜索功能(找到)。"""
|
||||
args = argparse.Namespace(
|
||||
count=False,
|
||||
lines=False,
|
||||
titles=False,
|
||||
title_content=None,
|
||||
search="关键词",
|
||||
context=2
|
||||
)
|
||||
|
||||
content = "行1\n行2\n包含关键词的行\n行4\n行5"
|
||||
|
||||
with patch("sys.exit") as mock_exit:
|
||||
output_result(content, args)
|
||||
mock_exit.assert_not_called()
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert "关键词" in captured.out
|
||||
|
||||
def test_output_search_not_found(self, capsys):
|
||||
"""测试搜索功能(未找到)。"""
|
||||
args = argparse.Namespace(
|
||||
count=False,
|
||||
lines=False,
|
||||
titles=False,
|
||||
title_content=None,
|
||||
search="不存在的内容",
|
||||
context=2
|
||||
)
|
||||
|
||||
content = "普通内容"
|
||||
|
||||
with patch("sys.exit") as mock_exit:
|
||||
output_result(content, args)
|
||||
mock_exit.assert_called_once_with(1)
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert "未找到" in captured.out or "错误" in captured.out
|
||||
@@ -4,198 +4,6 @@ import pytest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def temp_docx(tmp_path):
|
||||
"""创建临时 DOCX 文件的 fixture 工厂。
|
||||
|
||||
Args:
|
||||
paragraphs: 段落文本列表
|
||||
headings: 标题列表,格式为 [(level, text), ...]
|
||||
table_data: 表格数据,格式为 [[cell1, cell2], [cell3, cell4]]
|
||||
list_items: 列表项列表
|
||||
|
||||
Returns:
|
||||
str: 临时文件路径
|
||||
"""
|
||||
def _create_docx(paragraphs=None, headings=None, table_data=None, list_items=None):
|
||||
try:
|
||||
from docx import Document
|
||||
except ImportError:
|
||||
pytest.skip("python-docx 未安装")
|
||||
|
||||
doc = Document()
|
||||
|
||||
# 添加标题
|
||||
if headings:
|
||||
for level, text in headings:
|
||||
doc.add_heading(text, level=level)
|
||||
|
||||
# 添加段落
|
||||
if paragraphs:
|
||||
for para_text in paragraphs:
|
||||
doc.add_paragraph(para_text)
|
||||
|
||||
# 添加表格
|
||||
if table_data:
|
||||
table = doc.add_table(rows=len(table_data), cols=len(table_data[0]))
|
||||
for i, row_data in enumerate(table_data):
|
||||
for j, cell_text in enumerate(row_data):
|
||||
table.rows[i].cells[j].text = str(cell_text)
|
||||
|
||||
# 添加列表项
|
||||
if list_items:
|
||||
for item in list_items:
|
||||
doc.add_paragraph(item, style='List Bullet')
|
||||
|
||||
file_path = tmp_path / "test.docx"
|
||||
doc.save(str(file_path))
|
||||
return str(file_path)
|
||||
|
||||
return _create_docx
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def temp_pdf(tmp_path):
|
||||
"""创建临时 PDF 文件的 fixture 工厂。
|
||||
|
||||
Args:
|
||||
text: PDF 文本内容
|
||||
lines: 文本行列表
|
||||
|
||||
Returns:
|
||||
str: 临时文件路径
|
||||
"""
|
||||
def _create_pdf(text=None, lines=None):
|
||||
try:
|
||||
from reportlab.pdfgen import canvas
|
||||
from reportlab.lib.pagesizes import letter
|
||||
from reportlab.pdfbase import pdfmetrics
|
||||
from reportlab.pdfbase.ttfonts import TTFont
|
||||
except ImportError:
|
||||
pytest.skip("reportlab 未安装")
|
||||
|
||||
file_path = tmp_path / "test.pdf"
|
||||
c = canvas.Canvas(str(file_path), pagesize=letter)
|
||||
|
||||
# 尝试注册中文字体(如果可用)
|
||||
try:
|
||||
# 使用系统字体
|
||||
pdfmetrics.registerFont(TTFont('SimSun', 'simsun.ttc'))
|
||||
c.setFont('SimSun', 12)
|
||||
except:
|
||||
# 回退到默认字体
|
||||
c.setFont('Helvetica', 12)
|
||||
|
||||
y_position = 750
|
||||
|
||||
if text:
|
||||
# 单个文本块
|
||||
for line in text.split('\n'):
|
||||
c.drawString(100, y_position, line)
|
||||
y_position -= 20
|
||||
|
||||
if lines:
|
||||
# 多行文本
|
||||
for line in lines:
|
||||
c.drawString(100, y_position, line)
|
||||
y_position -= 20
|
||||
|
||||
c.save()
|
||||
return str(file_path)
|
||||
|
||||
return _create_pdf
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def temp_html(tmp_path):
|
||||
"""创建临时 HTML 文件的 fixture 工厂。
|
||||
|
||||
Args:
|
||||
content: HTML 内容字符串
|
||||
encoding: 文件编码,默认 'utf-8'
|
||||
|
||||
Returns:
|
||||
str: 临时文件路径
|
||||
"""
|
||||
def _create_html(content="<html><body><p>Test</p></body></html>", encoding='utf-8'):
|
||||
file_path = tmp_path / "test.html"
|
||||
|
||||
# 如果内容不包含完整的 HTML 结构,添加基本结构
|
||||
if not content.strip().startswith('<html'):
|
||||
content = f"<html><head><meta charset='{encoding}'></head><body>{content}</body></html>"
|
||||
|
||||
with open(file_path, 'w', encoding=encoding) as f:
|
||||
f.write(content)
|
||||
|
||||
return str(file_path)
|
||||
|
||||
return _create_html
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def temp_pptx(tmp_path):
|
||||
"""创建临时 PPTX 文件的 fixture 工厂。
|
||||
|
||||
Args:
|
||||
slides: 幻灯片内容列表,每个元素为 (title, content) 元组
|
||||
|
||||
Returns:
|
||||
str: 临时文件路径
|
||||
"""
|
||||
def _create_pptx(slides=None):
|
||||
try:
|
||||
from pptx import Presentation
|
||||
except ImportError:
|
||||
pytest.skip("python-pptx 未安装")
|
||||
|
||||
prs = Presentation()
|
||||
|
||||
if slides:
|
||||
for title, content in slides:
|
||||
slide = prs.slides.add_slide(prs.slide_layouts[1]) # Title and Content layout
|
||||
slide.shapes.title.text = title
|
||||
if content:
|
||||
text_frame = slide.shapes.placeholders[1].text_frame
|
||||
text_frame.text = content
|
||||
|
||||
file_path = tmp_path / "test.pptx"
|
||||
prs.save(str(file_path))
|
||||
return str(file_path)
|
||||
|
||||
return _create_pptx
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def temp_xlsx(tmp_path):
|
||||
"""创建临时 XLSX 文件的 fixture 工厂。
|
||||
|
||||
Args:
|
||||
data: 表格数据,格式为 [[cell1, cell2], [cell3, cell4]]
|
||||
|
||||
Returns:
|
||||
str: 临时文件路径
|
||||
"""
|
||||
def _create_xlsx(data=None):
|
||||
try:
|
||||
import pandas as pd
|
||||
except ImportError:
|
||||
pytest.skip("pandas 未安装")
|
||||
|
||||
file_path = tmp_path / "test.xlsx"
|
||||
|
||||
if data:
|
||||
df = pd.DataFrame(data)
|
||||
df.to_excel(str(file_path), index=False, header=False)
|
||||
else:
|
||||
# 创建空的 Excel 文件
|
||||
df = pd.DataFrame()
|
||||
df.to_excel(str(file_path), index=False)
|
||||
|
||||
return str(file_path)
|
||||
|
||||
return _create_xlsx
|
||||
|
||||
|
||||
# 静态测试文件目录
|
||||
FIXTURES_DIR = Path(__file__).parent / "fixtures"
|
||||
|
||||
@@ -278,4 +86,3 @@ def multiple_slides_ppt_path(ppt_fixture_path):
|
||||
def with_images_ppt_path(ppt_fixture_path):
|
||||
"""返回带图片 PPT 测试文件路径"""
|
||||
return _get_static_file_path(ppt_fixture_path, "with_images.ppt")
|
||||
|
||||
|
||||
1
tests/test_readers/test_doc/__init__.py
Normal file
1
tests/test_readers/test_doc/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
"""测试 DOC Reader 的解析功能。"""
|
||||
25
tests/test_readers/test_doc/test_consistency.py
Normal file
25
tests/test_readers/test_doc/test_consistency.py
Normal file
@@ -0,0 +1,25 @@
|
||||
"""测试所有 DOC Readers 的一致性。"""
|
||||
|
||||
import pytest
|
||||
from readers.doc import libreoffice
|
||||
|
||||
|
||||
class TestDocReadersConsistency:
|
||||
"""验证所有 DOC Readers 解析同一文件时核心文字内容一致。"""
|
||||
|
||||
def test_parsers_importable(self):
|
||||
"""测试所有 parser 模块可以正确导入。"""
|
||||
# 验证模块导入成功
|
||||
assert libreoffice is not None
|
||||
assert hasattr(libreoffice, 'parse')
|
||||
|
||||
def test_parser_functions_callable(self):
|
||||
"""测试 parse 函数是可调用的。"""
|
||||
assert callable(libreoffice.parse)
|
||||
|
||||
def test_libreoffice_parse_simple_doc(self, simple_doc_path):
|
||||
"""测试 LibreOffice 解析简单文件。"""
|
||||
content, error = libreoffice.parse(simple_doc_path)
|
||||
# LibreOffice 可能未安装,所以不强制断言成功
|
||||
if content is not None:
|
||||
assert content.strip() != ""
|
||||
35
tests/test_readers/test_doc/test_libreoffice.py
Normal file
35
tests/test_readers/test_doc/test_libreoffice.py
Normal file
@@ -0,0 +1,35 @@
|
||||
"""测试 LibreOffice DOC Reader 的解析功能。"""
|
||||
|
||||
import pytest
|
||||
import os
|
||||
from readers.doc import libreoffice
|
||||
|
||||
|
||||
class TestLibreOfficeDocReaderParse:
|
||||
"""测试 LibreOffice DOC Reader 的 parse 方法。"""
|
||||
|
||||
def test_simple_doc(self, simple_doc_path):
|
||||
"""测试简单 DOC 文件解析。"""
|
||||
content, error = libreoffice.parse(simple_doc_path)
|
||||
if content is not None:
|
||||
# 至少能解析出一些内容
|
||||
assert content.strip() != ""
|
||||
|
||||
def test_with_headings_doc(self, with_headings_doc_path):
|
||||
"""测试带标题的 DOC 文件解析。"""
|
||||
content, error = libreoffice.parse(with_headings_doc_path)
|
||||
if content is not None:
|
||||
assert content.strip() != ""
|
||||
|
||||
def test_with_table_doc(self, with_table_doc_path):
|
||||
"""测试带表格的 DOC 文件解析。"""
|
||||
content, error = libreoffice.parse(with_table_doc_path)
|
||||
if content is not None:
|
||||
assert content.strip() != ""
|
||||
|
||||
def test_file_not_exists(self, tmp_path):
|
||||
"""测试文件不存在的情况。"""
|
||||
non_existent_file = str(tmp_path / "non_existent.doc")
|
||||
content, error = libreoffice.parse(non_existent_file)
|
||||
assert content is None
|
||||
assert error is not None
|
||||
55
tests/test_readers/test_docx/test_libreoffice.py
Normal file
55
tests/test_readers/test_docx/test_libreoffice.py
Normal file
@@ -0,0 +1,55 @@
|
||||
"""测试 LibreOffice DOCX Reader 的解析功能。"""
|
||||
|
||||
import pytest
|
||||
import os
|
||||
from readers.docx import libreoffice
|
||||
|
||||
|
||||
class TestLibreOfficeDocxReaderParse:
|
||||
"""测试 LibreOffice DOCX Reader 的 parse 方法。"""
|
||||
|
||||
def test_normal_file(self, temp_docx):
|
||||
"""测试正常 DOCX 文件解析。"""
|
||||
file_path = temp_docx(
|
||||
headings=[(1, "主标题"), (2, "子标题")],
|
||||
paragraphs=["这是第一段内容。", "这是第二段内容。"],
|
||||
table_data=[["列1", "列2"], ["数据1", "数据2"]],
|
||||
list_items=["列表项1", "列表项2"]
|
||||
)
|
||||
|
||||
content, error = libreoffice.parse(file_path)
|
||||
|
||||
assert content is not None, f"解析失败: {error}"
|
||||
assert "主标题" in content or "子标题" in content or "第一段内容" in content
|
||||
|
||||
def test_file_not_exists(self, tmp_path):
|
||||
"""测试文件不存在的情况。"""
|
||||
non_existent_file = str(tmp_path / "non_existent.docx")
|
||||
content, error = libreoffice.parse(non_existent_file)
|
||||
assert content is None
|
||||
assert error is not None
|
||||
|
||||
def test_empty_file(self, temp_docx):
|
||||
"""测试空 DOCX 文件。"""
|
||||
file_path = temp_docx()
|
||||
content, error = libreoffice.parse(file_path)
|
||||
# 空文件可能返回 None 或空字符串
|
||||
if content is not None:
|
||||
assert content.strip() == ""
|
||||
|
||||
def test_corrupted_file(self, temp_docx, tmp_path):
|
||||
"""测试损坏的 DOCX 文件。"""
|
||||
file_path = temp_docx(paragraphs=["测试内容"])
|
||||
with open(file_path, "wb") as f:
|
||||
f.write(b"corrupted content")
|
||||
content, error = libreoffice.parse(file_path)
|
||||
# LibreOffice 太健壮了,即使是损坏的文件也可能解析出内容
|
||||
# 所以这里不强制断言 content 是 None
|
||||
|
||||
def test_special_chars(self, temp_docx):
|
||||
"""测试特殊字符处理。"""
|
||||
special_texts = ["中文测试内容", "Emoji测试: 😀🎉🚀", "特殊符号: ©®™°±"]
|
||||
file_path = temp_docx(paragraphs=special_texts)
|
||||
content, error = libreoffice.parse(file_path)
|
||||
if content is not None:
|
||||
assert "中文测试内容" in content or "😀" in content
|
||||
43
tests/test_readers/test_html_downloader.py
Normal file
43
tests/test_readers/test_html_downloader.py
Normal file
@@ -0,0 +1,43 @@
|
||||
"""测试 HTML 下载器模块。"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
from readers.html.downloader import download_html
|
||||
from readers.html.downloader import pyppeteer, selenium, httpx, urllib
|
||||
|
||||
|
||||
class TestDownloadHtml:
|
||||
"""测试 download_html 统一入口函数。"""
|
||||
|
||||
def test_download_html_module_importable(self):
|
||||
"""测试 download_html 函数可以正常导入和调用。"""
|
||||
# 只要不抛异常就可以
|
||||
assert callable(download_html)
|
||||
|
||||
def test_downloaders_available(self):
|
||||
"""测试各下载器模块可用。"""
|
||||
assert callable(pyppeteer.download)
|
||||
assert callable(selenium.download)
|
||||
assert callable(httpx.download)
|
||||
assert callable(urllib.download)
|
||||
|
||||
|
||||
class TestIndividualDownloaders:
|
||||
"""测试单个下载器模块。"""
|
||||
|
||||
def test_pyppeteer_download_callable(self):
|
||||
"""测试 pyppeteer.download 可以调用。"""
|
||||
assert callable(pyppeteer.download)
|
||||
|
||||
def test_selenium_download_callable(self):
|
||||
"""测试 selenium.download 可以调用。"""
|
||||
assert callable(selenium.download)
|
||||
|
||||
def test_httpx_download_callable(self):
|
||||
"""测试 httpx.download 可以调用。"""
|
||||
assert callable(httpx.download)
|
||||
|
||||
def test_urllib_download_callable(self):
|
||||
"""测试 urllib.download 可以调用(标准库)。"""
|
||||
assert callable(urllib.download)
|
||||
1
tests/test_readers/test_ppt/__init__.py
Normal file
1
tests/test_readers/test_ppt/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
"""Tests for PPT readers."""
|
||||
25
tests/test_readers/test_ppt/test_consistency.py
Normal file
25
tests/test_readers/test_ppt/test_consistency.py
Normal file
@@ -0,0 +1,25 @@
|
||||
"""测试所有 PPT Readers 的一致性。"""
|
||||
|
||||
import pytest
|
||||
from readers.ppt import libreoffice
|
||||
|
||||
|
||||
class TestPptReadersConsistency:
|
||||
"""验证所有 PPT Readers 解析同一文件时核心文字内容一致。"""
|
||||
|
||||
def test_parsers_importable(self):
|
||||
"""测试所有 parser 模块可以正确导入。"""
|
||||
# 验证模块导入成功
|
||||
assert libreoffice is not None
|
||||
assert hasattr(libreoffice, 'parse')
|
||||
|
||||
def test_parser_functions_callable(self):
|
||||
"""测试 parse 函数是可调用的。"""
|
||||
assert callable(libreoffice.parse)
|
||||
|
||||
def test_libreoffice_parse_simple_ppt(self, simple_ppt_path):
|
||||
"""测试 LibreOffice 解析简单文件。"""
|
||||
content, error = libreoffice.parse(simple_ppt_path)
|
||||
# LibreOffice 可能未安装,所以不强制断言成功
|
||||
if content is not None:
|
||||
assert content.strip() != ""
|
||||
35
tests/test_readers/test_ppt/test_libreoffice.py
Normal file
35
tests/test_readers/test_ppt/test_libreoffice.py
Normal file
@@ -0,0 +1,35 @@
|
||||
"""测试 LibreOffice PPT Reader 的解析功能。"""
|
||||
|
||||
import pytest
|
||||
import os
|
||||
from readers.ppt import libreoffice
|
||||
|
||||
|
||||
class TestLibreOfficePptReaderParse:
|
||||
"""测试 LibreOffice PPT Reader 的 parse 方法。"""
|
||||
|
||||
def test_simple_ppt(self, simple_ppt_path):
|
||||
"""测试简单 PPT 文件解析。"""
|
||||
content, error = libreoffice.parse(simple_ppt_path)
|
||||
if content is not None:
|
||||
# 至少能解析出一些内容
|
||||
assert content.strip() != ""
|
||||
|
||||
def test_multiple_slides_ppt(self, multiple_slides_ppt_path):
|
||||
"""测试多幻灯片 PPT 文件解析。"""
|
||||
content, error = libreoffice.parse(multiple_slides_ppt_path)
|
||||
if content is not None:
|
||||
assert content.strip() != ""
|
||||
|
||||
def test_with_images_ppt(self, with_images_ppt_path):
|
||||
"""测试带图片的 PPT 文件解析。"""
|
||||
content, error = libreoffice.parse(with_images_ppt_path)
|
||||
if content is not None:
|
||||
assert content.strip() != ""
|
||||
|
||||
def test_file_not_exists(self, tmp_path):
|
||||
"""测试文件不存在的情况。"""
|
||||
non_existent_file = str(tmp_path / "non_existent.ppt")
|
||||
content, error = libreoffice.parse(non_existent_file)
|
||||
assert content is None
|
||||
assert error is not None
|
||||
46
tests/test_utils/test_encoding_detection.py
Normal file
46
tests/test_utils/test_encoding_detection.py
Normal file
@@ -0,0 +1,46 @@
|
||||
"""测试 encoding_detection 编码检测模块。"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
from utils.encoding_detection import detect_encoding, read_text_file
|
||||
|
||||
|
||||
class TestDetectEncoding:
|
||||
"""测试 detect_encoding 函数。"""
|
||||
|
||||
def test_detect_encoding_file_not_exists(self, tmp_path):
|
||||
"""测试文件不存在。"""
|
||||
non_existent = str(tmp_path / "non_existent.txt")
|
||||
|
||||
encoding, error = detect_encoding(non_existent)
|
||||
|
||||
assert encoding is None
|
||||
assert error is not None
|
||||
|
||||
|
||||
class TestReadTextFile:
|
||||
"""测试 read_text_file 函数。"""
|
||||
|
||||
def test_read_simple_file(self, tmp_path):
|
||||
"""测试读取简单文件。"""
|
||||
file_path = tmp_path / "test.txt"
|
||||
content = "test content"
|
||||
file_path.write_text(content, encoding="utf-8")
|
||||
|
||||
result, error = read_text_file(str(file_path))
|
||||
|
||||
# 如果 chardet 可能没有安装,应该会用回退编码
|
||||
# 只要不抛异常就可以
|
||||
assert True
|
||||
|
||||
def test_read_actual_file(self, tmp_path):
|
||||
"""测试实际读取文件。"""
|
||||
file_path = tmp_path / "test.txt"
|
||||
content = "简单测试内容"
|
||||
file_path.write_text(content, encoding="utf-8")
|
||||
|
||||
result, error = read_text_file(str(file_path))
|
||||
|
||||
# 至少应该能读取成功(用回退编码)
|
||||
assert result is not None or error is not None
|
||||
Reference in New Issue
Block a user