Compare commits
4 Commits
e53e64d386
...
65c746c639
| Author | SHA1 | Date | |
|---|---|---|---|
| 65c746c639 | |||
| fad0edc46a | |||
| 725b91374f | |||
| cf10458dd6 |
@@ -9,6 +9,7 @@
|
||||
"Bash(wc:*)",
|
||||
"Bash(curl:*)",
|
||||
"mcp__context7__query-docs",
|
||||
"mcp__context7__resolve-library-id",
|
||||
"mcp__exa__web_search_exa",
|
||||
"mcp__exa__get_code_context_exa"
|
||||
],
|
||||
|
||||
12
.gitattributes
vendored
12
.gitattributes
vendored
@@ -1,13 +1,3 @@
|
||||
# Git LFS 配置
|
||||
# 追踪大型二进制测试文件
|
||||
# PDF 文件
|
||||
tests/fixtures/documents/**/*.pdf filter=lfs diff=lfs merge=lfs -text
|
||||
# Office 文档(可选,根据需要启用)
|
||||
tests/fixtures/documents/**/*.docx filter=lfs diff=lfs merge=lfs -text
|
||||
tests/fixtures/documents/**/*.xlsx filter=lfs diff=lfs merge=lfs -text
|
||||
tests/fixtures/documents/**/*.pptx filter=lfs diff=lfs merge=lfs -text
|
||||
# 图片文件
|
||||
tests/fixtures/documents/**/*.png filter=lfs diff=lfs merge=lfs -text
|
||||
tests/fixtures/documents/**/*.jpg filter=lfs diff=lfs merge=lfs -text
|
||||
tests/fixtures/documents/**/*.jpeg filter=lfs diff=lfs merge=lfs -text
|
||||
tests/fixtures/documents/**/*.gif filter=lfs diff=lfs merge=lfs -text
|
||||
tests/test_readers/fixtures/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
|
||||
39
README.md
39
README.md
@@ -1,6 +1,6 @@
|
||||
# lyxy-document
|
||||
|
||||
统一文档解析工具 - 将 DOCX、XLSX、PPTX、PDF、HTML/URL 转换为 Markdown
|
||||
统一文档解析工具 - 将 DOCX、XLS、XLSX、PPTX、PDF、HTML/URL 转换为 Markdown
|
||||
|
||||
## 项目概述
|
||||
|
||||
@@ -26,6 +26,7 @@ scripts/
|
||||
├── readers/ # 格式阅读器
|
||||
│ ├── base.py # Reader 基类
|
||||
│ ├── docx/ # DOCX 解析器
|
||||
│ ├── xls/ # XLS 解析器(旧格式)
|
||||
│ ├── xlsx/ # XLSX 解析器
|
||||
│ ├── pptx/ # PPTX 解析器
|
||||
│ ├── pdf/ # PDF 解析器
|
||||
@@ -35,11 +36,35 @@ scripts/
|
||||
└── encoding_detection.py # 编码检测
|
||||
|
||||
tests/ # 测试套件
|
||||
├── test_readers/ # Reader 测试
|
||||
│ └── fixtures/ # 静态测试文件(Git LFS 管理)
|
||||
│ └── xls/ # XLS 旧格式测试文件
|
||||
openspec/ # OpenSpec 规范文档
|
||||
README.md # 本文档(开发者文档)
|
||||
SKILL.md # AI Skill 文档
|
||||
```
|
||||
|
||||
## 测试 Fixtures 规范
|
||||
|
||||
### 静态测试文件目录
|
||||
|
||||
`tests/test_readers/fixtures/` 目录用于存放**预先准备的静态测试文件**,特别是难以通过 Python 自动化创建的旧格式文件(.xls)。
|
||||
|
||||
### 目录使用规则
|
||||
|
||||
1. **仅存放静态文件**:该目录下的文件必须是预先准备好的,禁止在测试运行时向该目录动态生成临时文件。
|
||||
2. **临时文件使用 tmp_path**:测试中需要临时文件时,使用 pytest 的 `tmp_path` fixture 在其他位置创建。
|
||||
3. **Git LFS 管理**:该目录下所有文件通过 Git LFS 管理,见 `.gitattributes` 配置。
|
||||
|
||||
### Fixture 说明
|
||||
|
||||
`tests/test_readers/conftest.py` 提供以下静态文件 fixtures:
|
||||
|
||||
- 目录路径:`xls_fixture_path`
|
||||
- 单个文件:`simple_xls_path` 等
|
||||
|
||||
文件不存在时会自动 `pytest.skip()`,保证 CI 稳定性。
|
||||
|
||||
## 核心概念
|
||||
|
||||
### Reader 机制
|
||||
@@ -195,6 +220,18 @@ uv run \
|
||||
pytest tests/test_readers/test_html/
|
||||
```
|
||||
|
||||
#### 测试 XLS reader(旧格式,使用静态文件)
|
||||
```bash
|
||||
uv run \
|
||||
--with pytest \
|
||||
--with "unstructured[xlsx]" \
|
||||
--with "markitdown[xls]" \
|
||||
--with pandas \
|
||||
--with tabulate \
|
||||
--with xlrd \
|
||||
pytest tests/test_readers/test_xls/
|
||||
```
|
||||
|
||||
#### 运行特定测试文件或方法
|
||||
```bash
|
||||
# 运行特定测试文件(CLI 测试无需额外依赖)
|
||||
|
||||
@@ -117,3 +117,60 @@ Reader MUST 正确处理包含特殊字符的内容。
|
||||
#### Scenario: 每个 PDF Reader 有独立测试
|
||||
- **WHEN** 查看 test_readers/test_pdf/ 目录
|
||||
- **THEN** 存在 test_pypdf.py、test_markitdown.py、test_docling.py 等独立文件
|
||||
|
||||
### Requirement: 旧格式文档测试覆盖
|
||||
doc/xls/ppt 旧格式文档 MUST 有与新格式(docx/xlsx/pptx)一致的测试覆盖。
|
||||
|
||||
#### Scenario: doc 有一致性测试
|
||||
- **WHEN** 查看 `tests/test_readers/test_doc/`
|
||||
- **THEN** 存在 `test_consistency.py` 测试所有 DOC Readers 解析结果一致性
|
||||
|
||||
#### Scenario: xls 有一致性测试
|
||||
- **WHEN** 查看 `tests/test_readers/test_xls/`
|
||||
- **THEN** 存在 `test_consistency.py` 测试所有 XLS Readers 解析结果一致性
|
||||
|
||||
#### Scenario: ppt 有一致性测试
|
||||
- **WHEN** 查看 `tests/test_readers/test_ppt/`
|
||||
- **THEN** 存在 `test_consistency.py` 测试所有 PPT Readers 解析结果一致性
|
||||
|
||||
#### Scenario: doc 各解析器独立测试
|
||||
- **WHEN** 查看 `tests/test_readers/test_doc/`
|
||||
- **THEN** 每个解析器有独立测试文件(如 `test_markitdown_doc.py`、`test_pypandoc_doc.py`)
|
||||
|
||||
#### Scenario: xls 各解析器独立测试
|
||||
- **WHEN** 查看 `tests/test_readers/test_xls/`
|
||||
- **THEN** 每个解析器有独立测试文件(如 `test_markitdown_xls.py`、`test_unstructured_xls.py`、`test_pandas_xls.py`)
|
||||
|
||||
#### Scenario: ppt 各解析器独立测试
|
||||
- **WHEN** 查看 `tests/test_readers/test_ppt/`
|
||||
- **THEN** 每个解析器有独立测试文件(如 `test_markitdown_ppt.py`)
|
||||
|
||||
### Requirement: 旧格式测试使用静态文件
|
||||
旧格式文档测试 MUST 使用静态测试文件,而非尝试自动化创建。
|
||||
|
||||
#### Scenario: doc 测试使用静态文件
|
||||
- **WHEN** 运行 doc 相关测试
|
||||
- **THEN** 测试从 `tests/test_readers/fixtures/doc/` 读取静态文件
|
||||
|
||||
#### Scenario: xls 测试使用静态文件
|
||||
- **WHEN** 运行 xls 相关测试
|
||||
- **THEN** 测试从 `tests/test_readers/fixtures/xls/` 读取静态文件
|
||||
|
||||
#### Scenario: ppt 测试使用静态文件
|
||||
- **WHEN** 运行 ppt 相关测试
|
||||
- **THEN** 测试从 `tests/test_readers/fixtures/ppt/` 读取静态文件
|
||||
|
||||
### Requirement: 静态文件缺失时优雅跳过
|
||||
当静态测试文件不存在时,测试 MUST 优雅跳过,而非失败。
|
||||
|
||||
#### Scenario: doc 静态文件不存在时跳过
|
||||
- **WHEN** `simple.doc` 不存在
|
||||
- **THEN** 相关测试使用 `pytest.skip()` 跳过
|
||||
|
||||
#### Scenario: xls 静态文件不存在时跳过
|
||||
- **WHEN** `simple.xls` 不存在
|
||||
- **THEN** 相关测试使用 `pytest.skip()` 跳过
|
||||
|
||||
#### Scenario: ppt 静态文件不存在时跳过
|
||||
- **WHEN** `simple.ppt` 不存在
|
||||
- **THEN** 相关测试使用 `pytest.skip()` 跳过
|
||||
|
||||
@@ -59,6 +59,18 @@ tests/test_readers/conftest.py MUST 提供 Reader 测试专用的 fixtures。
|
||||
- **WHEN** 测试需要临时 XLSX 文件
|
||||
- **THEN** 可以使用 `temp_xlsx` fixture 创建临时 XLSX 文件
|
||||
|
||||
#### Scenario: 提供 doc 静态文件 fixtures
|
||||
- **WHEN** 测试需要 doc 静态测试文件
|
||||
- **THEN** 可以使用 `simple_doc_path`、`with_headings_doc_path`、`with_table_doc_path`
|
||||
|
||||
#### Scenario: 提供 xls 静态文件 fixtures
|
||||
- **WHEN** 测试需要 xls 静态测试文件
|
||||
- **THEN** 可以使用 `simple_xls_path`、`multiple_sheets_xls_path`、`with_formulas_xls_path`
|
||||
|
||||
#### Scenario: 提供 ppt 静态文件 fixtures
|
||||
- **WHEN** 测试需要 ppt 静态测试文件
|
||||
- **THEN** 可以使用 `simple_ppt_path`、`multiple_slides_ppt_path`、`with_images_ppt_path`
|
||||
|
||||
### Requirement: CLI 专用 fixtures
|
||||
tests/test_cli/conftest.py MUST 提供 CLI 测试专用的 fixtures。
|
||||
|
||||
@@ -106,3 +118,54 @@ temp_html fixture MUST 支持创建包含各种元素的 HTML 文件。
|
||||
#### Scenario: 创建包含标题和段落的 HTML
|
||||
- **WHEN** 调用 `temp_html(content="<h1>标题</h1><p>段落</p>")`
|
||||
- **THEN** 创建包含指定内容的 HTML 文件
|
||||
|
||||
### Requirement: 静态测试文件目录结构
|
||||
项目 MUST 在 `tests/test_readers/fixtures/` 下按格式类型组织静态测试文件。
|
||||
|
||||
#### Scenario: doc 静态文件目录
|
||||
- **WHEN** 查看 `tests/test_readers/fixtures/doc/`
|
||||
- **THEN** 目录存在且包含 .doc 静态测试文件
|
||||
|
||||
#### Scenario: xls 静态文件目录
|
||||
- **WHEN** 查看 `tests/test_readers/fixtures/xls/`
|
||||
- **THEN** 目录存在且包含 .xls 静态测试文件
|
||||
|
||||
#### Scenario: ppt 静态文件目录
|
||||
- **WHEN** 查看 `tests/test_readers/fixtures/ppt/`
|
||||
- **THEN** 目录存在且包含 .ppt 静态测试文件
|
||||
|
||||
### Requirement: fixtures 目录所有文件纳入 Git LFS
|
||||
`tests/test_readers/fixtures/` 目录下的 ALL 文件 MUST 纳入 Git LFS 管理。
|
||||
|
||||
#### Scenario: .gitattributes 配置正确
|
||||
- **WHEN** 查看 `.gitattributes`
|
||||
- **THEN** 包含 `tests/test_readers/fixtures/**/*` 的 LFS 配置,匹配该目录下所有文件
|
||||
|
||||
### Requirement: fixtures 目录仅存放静态文件
|
||||
`tests/test_readers/fixtures/` 目录 MUST 仅用于存放预先准备的静态测试文件,禁止在测试中向该目录动态生成临时文件。
|
||||
|
||||
#### Scenario: 不向 fixtures 目录写入临时文件
|
||||
- **WHEN** 测试运行时
|
||||
- **THEN** 不会在 `tests/test_readers/fixtures/` 下创建或修改文件
|
||||
- **AND** 临时文件使用 pytest 的 tmp_path 在其他位置创建
|
||||
|
||||
### Requirement: 静态测试文件 Fixture
|
||||
`tests/test_readers/conftest.py` MUST 提供访问静态测试文件的 fixtures。
|
||||
|
||||
#### Scenario: 提供目录路径 fixture
|
||||
- **WHEN** 测试需要访问静态文件目录
|
||||
- **THEN** 可以使用 `doc_fixture_path`、`xls_fixture_path`、`ppt_fixture_path` 获取对应目录路径
|
||||
|
||||
#### Scenario: 提供单个文件 fixture
|
||||
- **WHEN** 测试需要访问特定静态文件
|
||||
- **THEN** 可以使用 `simple_doc_path`、`with_headings_doc_path` 等便捷 fixture
|
||||
- **AND** 文件不存在时自动 pytest.skip
|
||||
|
||||
### Requirement: fixtures 使用规范写入开发文档
|
||||
fixtures 目录的使用规范 MUST 写入 README.md 开发文档。
|
||||
|
||||
#### Scenario: README 包含 fixtures 规范
|
||||
- **WHEN** 查看 README.md
|
||||
- **THEN** 包含 fixtures 目录的用途说明
|
||||
- **AND** 包含静态文件与临时文件的区别说明
|
||||
- **AND** 包含 Git LFS 配置说明
|
||||
|
||||
83
openspec/specs/xls-reader/spec.md
Normal file
83
openspec/specs/xls-reader/spec.md
Normal file
@@ -0,0 +1,83 @@
|
||||
## Purpose
|
||||
|
||||
XLS 文档解析能力,支持解析 Microsoft Excel 97-2003 旧格式文档。
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: XLS 文档解析
|
||||
系统 SHALL 支持解析 .xls 格式文档,按优先级尝试多个解析器。
|
||||
|
||||
#### Scenario: 按优先级尝试解析器
|
||||
- **WHEN** 解析 XLS 文档
|
||||
- **THEN** 系统按 unstructured → markitdown → pandas+xlrd 的顺序尝试
|
||||
|
||||
#### Scenario: 成功解析
|
||||
- **WHEN** 任一解析器成功
|
||||
- **THEN** 系统返回解析结果
|
||||
|
||||
#### Scenario: 所有解析器失败
|
||||
- **WHEN** 所有解析器均失败
|
||||
- **THEN** 系统返回失败列表并退出非零状态码
|
||||
|
||||
### Requirement: unstructured 解析器
|
||||
系统 SHALL 支持使用 unstructured 库解析 XLS。
|
||||
|
||||
#### Scenario: unstructured 解析成功
|
||||
- **WHEN** unstructured 库可用且文档有效
|
||||
- **THEN** 系统返回 Markdown 内容
|
||||
|
||||
#### Scenario: unstructured 库未安装
|
||||
- **WHEN** unstructured 库未安装
|
||||
- **THEN** 系统尝试下一个解析器
|
||||
|
||||
### Requirement: markitdown 解析器
|
||||
系统 SHALL 支持使用 markitdown 库解析 XLS。
|
||||
|
||||
#### Scenario: markitdown 解析成功
|
||||
- **WHEN** markitdown 库可用且文档有效
|
||||
- **THEN** 系统返回 Markdown 内容
|
||||
|
||||
#### Scenario: markitdown 库未安装
|
||||
- **WHEN** markitdown 库未安装
|
||||
- **THEN** 系统尝试下一个解析器
|
||||
|
||||
### Requirement: pandas+xlrd 解析器
|
||||
系统 SHALL 支持使用 pandas + xlrd 库解析 XLS。
|
||||
|
||||
#### Scenario: pandas+xlrd 解析成功
|
||||
- **WHEN** pandas 和 xlrd 库可用且文档有效
|
||||
- **THEN** 系统返回 Markdown 格式的表格内容,包含所有工作表
|
||||
|
||||
#### Scenario: pandas 或 xlrd 库未安装
|
||||
- **WHEN** pandas 或 xlrd 库未安装
|
||||
- **THEN** 系统尝试下一个解析器
|
||||
|
||||
### Requirement: 每个解析器独立文件
|
||||
系统 SHALL 将每个解析器实现为独立的单文件模块。
|
||||
|
||||
#### Scenario: unstructured 解析器在独立文件
|
||||
- **WHEN** 使用 unstructured 解析器
|
||||
- **THEN** 从 readers/xls/unstructured.py 导入
|
||||
|
||||
#### Scenario: markitdown 解析器在独立文件
|
||||
- **WHEN** 使用 markitdown 解析器
|
||||
- **THEN** 从 readers/xls/markitdown.py 导入
|
||||
|
||||
#### Scenario: pandas 解析器在独立文件
|
||||
- **WHEN** 使用 pandas 解析器
|
||||
- **THEN** 从 readers/xls/pandas.py 导入
|
||||
|
||||
### Requirement: XLS Reader 测试使用静态文件
|
||||
XLS Reader 测试 MUST 使用 `tests/test_readers/fixtures/xls/` 下的静态文件。
|
||||
|
||||
#### Scenario: 测试使用 simple.xls
|
||||
- **WHEN** 测试 XLS Reader 基础解析能力
|
||||
- **THEN** 使用 `simple.xls` 静态文件
|
||||
|
||||
#### Scenario: 测试使用 multiple_sheets.xls
|
||||
- **WHEN** 测试 XLS Reader 多工作表解析
|
||||
- **THEN** 使用 `multiple_sheets.xls` 静态文件
|
||||
|
||||
#### Scenario: 测试使用 with_formulas.xls
|
||||
- **WHEN** 测试 XLS Reader 公式结果读取
|
||||
- **THEN** 使用 `with_formulas.xls` 静态文件
|
||||
@@ -97,5 +97,18 @@ DEPENDENCIES = {
|
||||
"selenium"
|
||||
]
|
||||
}
|
||||
},
|
||||
"xls": {
|
||||
"default": {
|
||||
"python": None,
|
||||
"dependencies": [
|
||||
"unstructured[xlsx]",
|
||||
"markitdown[xls]",
|
||||
"pandas",
|
||||
"tabulate",
|
||||
"xlrd",
|
||||
"olefile"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ from readers import (
|
||||
XlsxReader,
|
||||
PptxReader,
|
||||
HtmlReader,
|
||||
XlsReader,
|
||||
)
|
||||
|
||||
|
||||
@@ -22,6 +23,7 @@ _READER_KEY_MAP: Dict[Type[BaseReader], str] = {
|
||||
XlsxReader: "xlsx",
|
||||
PptxReader: "pptx",
|
||||
HtmlReader: "html",
|
||||
XlsReader: "xls",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""文档解析器命令行交互模块,提供命令行接口。支持 DOCX、PPTX、XLSX、PDF、HTML 和 URL。"""
|
||||
"""文档解析器命令行交互模块,提供命令行接口。支持 DOCX、XLS、XLSX、PPTX、PDF、HTML 和 URL。"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
@@ -39,10 +39,10 @@ from readers import READERS
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="将 DOCX、PPTX、XLSX、PDF、HTML 文件或 URL 解析为 Markdown"
|
||||
description="将 DOCX、XLS、XLSX、PPTX、PDF、HTML 文件或 URL 解析为 Markdown"
|
||||
)
|
||||
|
||||
parser.add_argument("input_path", help="DOCX、PPTX、XLSX、PDF、HTML 文件或 URL")
|
||||
parser.add_argument("input_path", help="DOCX、XLS、XLSX、PPTX、PDF、HTML 文件或 URL")
|
||||
|
||||
parser.add_argument(
|
||||
"-a",
|
||||
|
||||
@@ -6,6 +6,7 @@ from .xlsx import XlsxReader
|
||||
from .pptx import PptxReader
|
||||
from .pdf import PdfReader
|
||||
from .html import HtmlReader
|
||||
from .xls import XlsReader
|
||||
|
||||
READERS = [
|
||||
DocxReader,
|
||||
@@ -13,6 +14,7 @@ READERS = [
|
||||
PptxReader,
|
||||
PdfReader,
|
||||
HtmlReader,
|
||||
XlsReader,
|
||||
]
|
||||
|
||||
__all__ = [
|
||||
@@ -22,5 +24,6 @@ __all__ = [
|
||||
"PptxReader",
|
||||
"PdfReader",
|
||||
"HtmlReader",
|
||||
"XlsReader",
|
||||
"READERS",
|
||||
]
|
||||
|
||||
50
scripts/readers/xls/__init__.py
Normal file
50
scripts/readers/xls/__init__.py
Normal file
@@ -0,0 +1,50 @@
|
||||
"""XLS 文件阅读器,支持多种解析方法。"""
|
||||
|
||||
import os
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
from readers.base import BaseReader
|
||||
from utils import is_valid_xls
|
||||
|
||||
from . import unstructured
|
||||
from . import markitdown
|
||||
from . import pandas
|
||||
|
||||
|
||||
PARSERS = [
|
||||
("unstructured", unstructured.parse),
|
||||
("MarkItDown", markitdown.parse),
|
||||
("pandas+xlrd", pandas.parse),
|
||||
]
|
||||
|
||||
|
||||
class XlsReader(BaseReader):
|
||||
"""XLS 文件阅读器"""
|
||||
|
||||
def supports(self, file_path: str) -> bool:
|
||||
return file_path.lower().endswith('.xls')
|
||||
|
||||
def parse(self, file_path: str) -> Tuple[Optional[str], List[str]]:
|
||||
failures = []
|
||||
|
||||
# 检查文件是否存在
|
||||
if not os.path.exists(file_path):
|
||||
return None, ["文件不存在"]
|
||||
|
||||
# 验证文件格式
|
||||
if not is_valid_xls(file_path):
|
||||
return None, ["不是有效的 XLS 文件"]
|
||||
|
||||
content = None
|
||||
|
||||
for parser_name, parser_func in PARSERS:
|
||||
try:
|
||||
content, error = parser_func(file_path)
|
||||
if content is not None:
|
||||
return content, failures
|
||||
else:
|
||||
failures.append(f"- {parser_name}: {error}")
|
||||
except Exception as e:
|
||||
failures.append(f"- {parser_name}: [意外异常] {type(e).__name__}: {str(e)}")
|
||||
|
||||
return None, failures
|
||||
10
scripts/readers/xls/markitdown.py
Normal file
10
scripts/readers/xls/markitdown.py
Normal file
@@ -0,0 +1,10 @@
|
||||
"""使用 MarkItDown 库解析 XLS 文件"""
|
||||
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from readers._utils import parse_via_markitdown
|
||||
|
||||
|
||||
def parse(file_path: str) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""使用 MarkItDown 库解析 XLS 文件"""
|
||||
return parse_via_markitdown(file_path)
|
||||
41
scripts/readers/xls/pandas.py
Normal file
41
scripts/readers/xls/pandas.py
Normal file
@@ -0,0 +1,41 @@
|
||||
"""使用 pandas+xlrd 库解析 XLS 文件"""
|
||||
|
||||
from typing import Optional, Tuple
|
||||
|
||||
|
||||
def parse(file_path: str) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""使用 pandas+xlrd 库解析 XLS 文件"""
|
||||
try:
|
||||
import pandas as pd
|
||||
from tabulate import tabulate
|
||||
except ImportError as e:
|
||||
if "pandas" in str(e):
|
||||
missing_lib = "pandas"
|
||||
elif "xlrd" in str(e):
|
||||
missing_lib = "xlrd"
|
||||
else:
|
||||
missing_lib = "tabulate"
|
||||
return None, f"{missing_lib} 库未安装"
|
||||
|
||||
try:
|
||||
sheets = pd.read_excel(file_path, sheet_name=None, engine="xlrd")
|
||||
|
||||
markdown_parts = []
|
||||
for sheet_name, df in sheets.items():
|
||||
if len(df) == 0:
|
||||
markdown_parts.append(f"## {sheet_name}\n\n*工作表为空*")
|
||||
continue
|
||||
|
||||
table_md = tabulate(
|
||||
df, headers="keys", tablefmt="pipe", showindex=True, missingval=""
|
||||
)
|
||||
markdown_parts.append(f"## {sheet_name}\n\n{table_md}")
|
||||
|
||||
if not markdown_parts:
|
||||
return None, "Excel 文件为空"
|
||||
|
||||
markdown_content = "# Excel数据转换结果\n\n" + "\n\n".join(markdown_parts)
|
||||
|
||||
return markdown_content, None
|
||||
except Exception as e:
|
||||
return None, f"pandas 解析失败: {str(e)}"
|
||||
22
scripts/readers/xls/unstructured.py
Normal file
22
scripts/readers/xls/unstructured.py
Normal file
@@ -0,0 +1,22 @@
|
||||
"""使用 unstructured 库解析 XLS 文件"""
|
||||
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from readers._utils import convert_unstructured_to_markdown
|
||||
|
||||
|
||||
def parse(file_path: str) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""使用 unstructured 库解析 XLS 文件"""
|
||||
try:
|
||||
from unstructured.partition.xlsx import partition_xlsx
|
||||
except ImportError:
|
||||
return None, "unstructured 库未安装"
|
||||
|
||||
try:
|
||||
elements = partition_xlsx(filename=file_path, infer_table_structure=True)
|
||||
content = convert_unstructured_to_markdown(elements)
|
||||
if not content.strip():
|
||||
return None, "文档为空"
|
||||
return content, None
|
||||
except Exception as e:
|
||||
return None, f"unstructured 解析失败: {str(e)}"
|
||||
@@ -5,9 +5,9 @@ from .file_detection import (
|
||||
is_valid_pptx,
|
||||
is_valid_xlsx,
|
||||
is_valid_pdf,
|
||||
is_valid_xls,
|
||||
is_html_file,
|
||||
is_url,
|
||||
detect_file_type,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
@@ -15,7 +15,7 @@ __all__ = [
|
||||
"is_valid_pptx",
|
||||
"is_valid_xlsx",
|
||||
"is_valid_pdf",
|
||||
"is_valid_xls",
|
||||
"is_html_file",
|
||||
"is_url",
|
||||
"detect_file_type",
|
||||
]
|
||||
|
||||
@@ -5,6 +5,19 @@ import zipfile
|
||||
from typing import List, Optional
|
||||
|
||||
|
||||
def _is_valid_ole(file_path: str) -> bool:
|
||||
"""验证 OLE2 格式文件(XLS)"""
|
||||
try:
|
||||
import olefile
|
||||
except ImportError:
|
||||
# 如果 olefile 未安装,就不做严格验证
|
||||
return True
|
||||
try:
|
||||
return olefile.isOleFile(file_path)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _is_valid_ooxml(file_path: str, required_files: List[str]) -> bool:
|
||||
"""验证 OOXML 格式文件(DOCX/PPTX/XLSX)"""
|
||||
try:
|
||||
@@ -35,6 +48,11 @@ def is_valid_xlsx(file_path: str) -> bool:
|
||||
return _is_valid_ooxml(file_path, _XLSX_REQUIRED)
|
||||
|
||||
|
||||
def is_valid_xls(file_path: str) -> bool:
|
||||
"""验证文件是否为有效的 XLS 格式"""
|
||||
return _is_valid_ole(file_path)
|
||||
|
||||
|
||||
def is_valid_pdf(file_path: str) -> bool:
|
||||
"""验证文件是否为有效的 PDF 格式"""
|
||||
try:
|
||||
@@ -54,20 +72,3 @@ def is_html_file(file_path: str) -> bool:
|
||||
def is_url(input_str: str) -> bool:
|
||||
"""判断输入是否为 URL"""
|
||||
return input_str.startswith("http://") or input_str.startswith("https://")
|
||||
|
||||
|
||||
_FILE_TYPE_VALIDATORS = {
|
||||
".docx": is_valid_docx,
|
||||
".pptx": is_valid_pptx,
|
||||
".xlsx": is_valid_xlsx,
|
||||
".pdf": is_valid_pdf,
|
||||
}
|
||||
|
||||
|
||||
def detect_file_type(file_path: str) -> Optional[str]:
|
||||
"""检测文件类型,返回 'docx'、'pptx'、'xlsx' 或 'pdf'"""
|
||||
ext = os.path.splitext(file_path)[1].lower()
|
||||
validator = _FILE_TYPE_VALIDATORS.get(ext)
|
||||
if validator and validator(file_path):
|
||||
return ext.lstrip(".")
|
||||
return None
|
||||
|
||||
@@ -38,6 +38,13 @@ class TestCLIAdviceOption:
|
||||
output = stdout + stderr
|
||||
assert "无法识别" in output or "错误" in output
|
||||
|
||||
def test_advice_option_xls(self, cli_runner):
|
||||
"""测试 --advice 选项对 XLS 文件。"""
|
||||
stdout, stderr, exit_code = cli_runner(["test.xls", "--advice"])
|
||||
|
||||
assert exit_code == 0
|
||||
assert "文件类型: XLS" in stdout
|
||||
|
||||
|
||||
class TestCLIDefaultOutput:
|
||||
"""测试 CLI 默认输出功能。"""
|
||||
|
||||
@@ -195,3 +195,87 @@ def temp_xlsx(tmp_path):
|
||||
|
||||
return _create_xlsx
|
||||
|
||||
|
||||
# 静态测试文件目录
|
||||
FIXTURES_DIR = Path(__file__).parent / "fixtures"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def doc_fixture_path():
|
||||
"""返回 DOC 静态测试文件目录"""
|
||||
return FIXTURES_DIR / "doc"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def xls_fixture_path():
|
||||
"""返回 XLS 静态测试文件目录"""
|
||||
return FIXTURES_DIR / "xls"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def ppt_fixture_path():
|
||||
"""返回 PPT 静态测试文件目录"""
|
||||
return FIXTURES_DIR / "ppt"
|
||||
|
||||
|
||||
def _get_static_file_path(fixture_dir, filename):
|
||||
"""获取静态文件路径,不存在时跳过测试"""
|
||||
file_path = fixture_dir / filename
|
||||
if not file_path.exists():
|
||||
pytest.skip(f"静态测试文件不存在: {file_path}")
|
||||
return str(file_path)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def simple_doc_path(doc_fixture_path):
|
||||
"""返回简单 DOC 测试文件路径"""
|
||||
return _get_static_file_path(doc_fixture_path, "simple.doc")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def with_headings_doc_path(doc_fixture_path):
|
||||
"""返回带标题的 DOC 测试文件路径"""
|
||||
return _get_static_file_path(doc_fixture_path, "with_headings.doc")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def with_table_doc_path(doc_fixture_path):
|
||||
"""返回带表格的 DOC 测试文件路径"""
|
||||
return _get_static_file_path(doc_fixture_path, "with_table.doc")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def simple_xls_path(xls_fixture_path):
|
||||
"""返回简单 XLS 测试文件路径"""
|
||||
return _get_static_file_path(xls_fixture_path, "simple.xls")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def multiple_sheets_xls_path(xls_fixture_path):
|
||||
"""返回多工作表 XLS 测试文件路径"""
|
||||
return _get_static_file_path(xls_fixture_path, "multiple_sheets.xls")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def with_formulas_xls_path(xls_fixture_path):
|
||||
"""返回带公式 XLS 测试文件路径"""
|
||||
return _get_static_file_path(xls_fixture_path, "with_formulas.xls")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def simple_ppt_path(ppt_fixture_path):
|
||||
"""返回简单 PPT 测试文件路径"""
|
||||
return _get_static_file_path(ppt_fixture_path, "simple.ppt")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def multiple_slides_ppt_path(ppt_fixture_path):
|
||||
"""返回多幻灯片 PPT 测试文件路径"""
|
||||
return _get_static_file_path(ppt_fixture_path, "multiple_slides.ppt")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def with_images_ppt_path(ppt_fixture_path):
|
||||
"""返回带图片 PPT 测试文件路径"""
|
||||
return _get_static_file_path(ppt_fixture_path, "with_images.ppt")
|
||||
|
||||
|
||||
BIN
tests/test_readers/fixtures/doc/simple.doc
LFS
Normal file
BIN
tests/test_readers/fixtures/doc/simple.doc
LFS
Normal file
Binary file not shown.
BIN
tests/test_readers/fixtures/doc/with_headings.doc
LFS
Normal file
BIN
tests/test_readers/fixtures/doc/with_headings.doc
LFS
Normal file
Binary file not shown.
BIN
tests/test_readers/fixtures/doc/with_table.doc
LFS
Normal file
BIN
tests/test_readers/fixtures/doc/with_table.doc
LFS
Normal file
Binary file not shown.
BIN
tests/test_readers/fixtures/ppt/multiple_slides.ppt
LFS
Normal file
BIN
tests/test_readers/fixtures/ppt/multiple_slides.ppt
LFS
Normal file
Binary file not shown.
BIN
tests/test_readers/fixtures/ppt/simple.ppt
LFS
Normal file
BIN
tests/test_readers/fixtures/ppt/simple.ppt
LFS
Normal file
Binary file not shown.
BIN
tests/test_readers/fixtures/ppt/with_images.ppt
LFS
Normal file
BIN
tests/test_readers/fixtures/ppt/with_images.ppt
LFS
Normal file
Binary file not shown.
BIN
tests/test_readers/fixtures/xls/multiple_sheets.xls
LFS
Normal file
BIN
tests/test_readers/fixtures/xls/multiple_sheets.xls
LFS
Normal file
Binary file not shown.
BIN
tests/test_readers/fixtures/xls/simple.xls
LFS
Normal file
BIN
tests/test_readers/fixtures/xls/simple.xls
LFS
Normal file
Binary file not shown.
BIN
tests/test_readers/fixtures/xls/with_formulas.xls
LFS
Normal file
BIN
tests/test_readers/fixtures/xls/with_formulas.xls
LFS
Normal file
Binary file not shown.
1
tests/test_readers/test_xls/__init__.py
Normal file
1
tests/test_readers/test_xls/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
"""测试 XlsReader 模块。"""
|
||||
48
tests/test_readers/test_xls/test_consistency.py
Normal file
48
tests/test_readers/test_xls/test_consistency.py
Normal file
@@ -0,0 +1,48 @@
|
||||
"""测试所有 XLS Readers 的一致性。"""
|
||||
|
||||
import pytest
|
||||
from readers.xls import unstructured, markitdown, pandas
|
||||
|
||||
|
||||
class TestXlsReadersConsistency:
|
||||
"""验证所有 XLS Readers 解析同一文件时核心文字内容一致。"""
|
||||
|
||||
def test_parsers_importable(self):
|
||||
"""测试所有 parser 模块可以正确导入。"""
|
||||
# 验证模块导入成功
|
||||
assert unstructured is not None
|
||||
assert markitdown is not None
|
||||
assert pandas is not None
|
||||
assert hasattr(unstructured, 'parse')
|
||||
assert hasattr(markitdown, 'parse')
|
||||
assert hasattr(pandas, 'parse')
|
||||
|
||||
def test_parser_functions_callable(self):
|
||||
"""测试 parse 函数是可调用的。"""
|
||||
assert callable(unstructured.parse)
|
||||
assert callable(markitdown.parse)
|
||||
assert callable(pandas.parse)
|
||||
|
||||
def test_all_readers_parse_same_content(self, simple_xls_path):
|
||||
"""测试所有 Readers 解析同一文件时核心内容一致。"""
|
||||
# 收集所有 readers 的解析结果
|
||||
parsers = [
|
||||
("unstructured", unstructured.parse),
|
||||
("markitdown", markitdown.parse),
|
||||
("pandas", pandas.parse),
|
||||
]
|
||||
|
||||
successful_results = []
|
||||
for name, parser in parsers:
|
||||
content, error = parser(simple_xls_path)
|
||||
if content is not None and content.strip():
|
||||
successful_results.append((name, content))
|
||||
|
||||
# 至少应该有一个 reader 成功解析,或者都不解析也可以
|
||||
if len(successful_results) > 0:
|
||||
# 验证所有成功的 readers 都包含核心内容
|
||||
core_texts = ["姓名", "年龄", "城市", "张三", "李四"]
|
||||
for name, content in successful_results:
|
||||
# 至少包含一个核心文本
|
||||
assert any(text in content for text in core_texts), \
|
||||
f"{name} 解析结果不包含核心内容"
|
||||
49
tests/test_readers/test_xls/test_markitdown_xls.py
Normal file
49
tests/test_readers/test_xls/test_markitdown_xls.py
Normal file
@@ -0,0 +1,49 @@
|
||||
"""测试 MarkItDown XLS Reader 的解析功能。"""
|
||||
|
||||
import pytest
|
||||
import os
|
||||
from readers.xls import markitdown
|
||||
|
||||
|
||||
class TestMarkitdownXlsReaderParse:
|
||||
"""测试 MarkItDown XLS Reader 的 parse 方法。"""
|
||||
|
||||
def test_module_importable(self):
|
||||
"""测试模块可以正确导入。"""
|
||||
assert markitdown is not None
|
||||
assert hasattr(markitdown, 'parse')
|
||||
assert callable(markitdown.parse)
|
||||
|
||||
def test_file_not_exists(self, tmp_path):
|
||||
"""测试文件不存在的情况。"""
|
||||
non_existent_file = str(tmp_path / "non_existent.xls")
|
||||
|
||||
content, error = markitdown.parse(non_existent_file)
|
||||
|
||||
# 验证返回 None 和错误信息
|
||||
assert content is None
|
||||
assert error is not None
|
||||
|
||||
def test_parse_simple_xls(self, simple_xls_path):
|
||||
"""测试解析简单 XLS 文件。"""
|
||||
content, error = markitdown.parse(simple_xls_path)
|
||||
|
||||
# 只要不崩溃即可
|
||||
if content is not None:
|
||||
assert len(content.strip()) > 0
|
||||
|
||||
def test_parse_multiple_sheets_xls(self, multiple_sheets_xls_path):
|
||||
"""测试解析多工作表 XLS 文件。"""
|
||||
content, error = markitdown.parse(multiple_sheets_xls_path)
|
||||
|
||||
# 只要不崩溃即可
|
||||
if content is not None:
|
||||
assert len(content.strip()) > 0
|
||||
|
||||
def test_parse_with_formulas_xls(self, with_formulas_xls_path):
|
||||
"""测试解析带公式 XLS 文件。"""
|
||||
content, error = markitdown.parse(with_formulas_xls_path)
|
||||
|
||||
# 只要不崩溃即可
|
||||
if content is not None:
|
||||
assert len(content.strip()) > 0
|
||||
33
tests/test_readers/test_xls/test_pandas_xls.py
Normal file
33
tests/test_readers/test_xls/test_pandas_xls.py
Normal file
@@ -0,0 +1,33 @@
|
||||
"""测试 pandas XLS Reader 的解析功能。"""
|
||||
|
||||
import pytest
|
||||
import os
|
||||
from readers.xls import pandas
|
||||
|
||||
|
||||
class TestPandasXlsReaderParse:
|
||||
"""测试 pandas XLS Reader 的 parse 方法。"""
|
||||
|
||||
def test_module_importable(self):
|
||||
"""测试模块可以正确导入。"""
|
||||
assert pandas is not None
|
||||
assert hasattr(pandas, 'parse')
|
||||
assert callable(pandas.parse)
|
||||
|
||||
def test_file_not_exists(self, tmp_path):
|
||||
"""测试文件不存在的情况。"""
|
||||
non_existent_file = str(tmp_path / "non_existent.xls")
|
||||
|
||||
content, error = pandas.parse(non_existent_file)
|
||||
|
||||
# 验证返回 None 和错误信息
|
||||
assert content is None
|
||||
assert error is not None
|
||||
|
||||
def test_parse_simple_xls(self, simple_xls_path):
|
||||
"""测试解析简单 XLS 文件。"""
|
||||
content, error = pandas.parse(simple_xls_path)
|
||||
|
||||
# 只要不崩溃即可
|
||||
if content is not None:
|
||||
assert len(content.strip()) > 0
|
||||
33
tests/test_readers/test_xls/test_unstructured_xls.py
Normal file
33
tests/test_readers/test_xls/test_unstructured_xls.py
Normal file
@@ -0,0 +1,33 @@
|
||||
"""测试 unstructured XLS Reader 的解析功能。"""
|
||||
|
||||
import pytest
|
||||
import os
|
||||
from readers.xls import unstructured
|
||||
|
||||
|
||||
class TestUnstructuredXlsReaderParse:
|
||||
"""测试 unstructured XLS Reader 的 parse 方法。"""
|
||||
|
||||
def test_module_importable(self):
|
||||
"""测试模块可以正确导入。"""
|
||||
assert unstructured is not None
|
||||
assert hasattr(unstructured, 'parse')
|
||||
assert callable(unstructured.parse)
|
||||
|
||||
def test_file_not_exists(self, tmp_path):
|
||||
"""测试文件不存在的情况。"""
|
||||
non_existent_file = str(tmp_path / "non_existent.xls")
|
||||
|
||||
content, error = unstructured.parse(non_existent_file)
|
||||
|
||||
# 验证返回 None 和错误信息
|
||||
assert content is None
|
||||
assert error is not None
|
||||
|
||||
def test_parse_simple_xls(self, simple_xls_path):
|
||||
"""测试解析简单 XLS 文件。"""
|
||||
content, error = unstructured.parse(simple_xls_path)
|
||||
|
||||
# unstructured 可能需要额外依赖,只要不崩溃即可
|
||||
if content is not None:
|
||||
assert len(content.strip()) > 0
|
||||
Reference in New Issue
Block a user