giteahh/llm_wiki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 0-2: Schema cleanup, typed relations, event-driven automation

Browse Source 
- Phase 0: AGENTS.md cleanup (dedup quotes, renumber sections, merge qmd)
- Phase 1: typed relations (manage-relations.py, graph-search.py, check-staleness.py, detect-conflicts.py)
- Phase 2: frontmatter validator, weekly lint, knowledge promotion, git hooks
- Fix .gitignore to track tools/ and .githooks/
- Fix git remote URL (remove plaintext token)
- New wiki pages: 504 pages, 34 raw sources
This commit is contained in:
hehaiguang1123
2026-07-01 08:05:43 +08:00
parent e544d6e04a
commit a6f05ab2d5
1067 changed files with 522992 additions and 819 deletions
Download Patch File Download Diff File Expand all files Collapse all files
raw/教育AI研究/高等教育AI专题/tools/README_论文下载工具.md
+221
View File
@@ -0,0 +1,221 @@
---
created: 2026-04-14
title: 学术论文下载工具 (Academic Paper Downloader)
tags: [文献, 论文]
category: 文献库
---
# 学术论文下载工具 (Academic Paper Downloader)
> 为教育AI研究项目创建的学术论文批量下载工具，支持HTTP(S)/HTTPS重试、大文件分块下载、进度显示、日志记录
---
## 📋 功能特性
- ✅ **跨平台支持**：Windows/Linux/macOS
- ✅ **自动重试**：默认3次，可自定义
- ✅ **超时控制**：默认300秒，可自定义
- ✅ **分块下载**：支持大文件（默认8KB/块）
- ✅ **进度显示**：实时显示下载进度和百分比
- ✅ **日志系统**：记录每次下载结果（成功/失败）
- ✅ **批量下载**：支持从配置文件批量下载
- ✅ **UTF-8编码**：强制UTF-8输出，避免中文路径问题
- ✅ **统计功能**：显示下载成功率、平均大小等
---
## 🚀 快速开始
### 方式1：单篇论文下载
```bash
python paper_downloader.py download https://cs.harvard.edu/malan/publications/V1fp0567-liu.pdf
```
**功能**：
- 自动从URL提取文件名
- 自动创建目标目录
- 实时进度显示
- 完整错误处理
---
### 方式2：批量下载
1. **创建配置文件** `my_papers.json`：
```json
{
"name": "我的论文集合",
"papers": [
{
"name": "Teaching CS50 with AI (SIGCSE 2024)",
"url": "https://cs.harvard.edu/malan/publications/V1fp0567-liu.pdf",
"filename": "Teaching_CS50_with_AI_SIGCSE2024.pdf",
"enabled": true
},
{
"name": "Improving AI in CS50 (SIGCSE 2025)",
"url": "https://cs.harvard.edu/malan/publications/fp0627-liu.pdf",
"filename": "Improving_AI_in_CS50_SIGCSE2025.pdf",
"enabled": true
}
]
}
```
2. **执行批量下载**：
```bash
python paper_downloader.py config my_papers.json
```
**功能**：
- 读取JSON配置
- 下载所有`enabled: true`的论文
- 显示下载统计
- 自动跳过`enabled: false`的论文
---
### 方式3：查看下载日志
```bash
python paper_downloader.py logs
```
**统计信息**：
- 总下载次数
- 成功次数
- 失败次数
- 成功率
- 总下载大小
- 平均大小
---
## ⚙️ 高级选项
```bash
# 自定义超时时间（默认300秒）
python paper_downloader.py config my_papers.json --timeout 600
# 自定义重试次数（默认3次）
python paper_downloader.py config my_papers.json --retry 5
# 自定义分块大小（默认8192字节）
python paper_downloader.py config my_papers.json --chunk-size 16384
```
---
## 📂 配置文件结构
### 完整示例
```json
{
"name": "学术论文集合",
"description": "教育AI研究相关的学术论文下载配置",
"papers": [
{
"name": "论文标题",
"url": "https://example.com/paper.pdf",
"filename": "自定义文件名.pdf",
"enabled": true
}
]
}
```
### 配置文件管理
**创建示例配置**：
```bash
python paper_downloader.py create-sample
```
**说明**：工具会在`configs/`目录下生成示例配置文件，可以直接修改使用。
---
## 🔧 优化建议
### 1. 网络优化
- 使用稳定的网络连接
- 避免高峰时段下载
- 启用下载加速（如果支持）
### 2. 批量下载策略
- **分批下载**：不要一次性下载太多论文，每次5-10篇
- **添加延迟**：每批之间添加10-30秒延迟
- **优先级排序**：重要论文优先下载
### 3. 存储管理
- **定期清理日志**：日志文件会持续增长，建议每月清理
- **分类存储**：按主题/机构创建子目录
- **文件验证**：下载后检查文件完整性
### 4. 错误处理
**常见错误及解决方案**：
| 错误 | 原因 | 解决方案 |
|------|--------|----------|
| `URLError: <urlopen error>` | 网络连接问题 | 检查网络、重试 |
| `HTTPError: 404 Not Found` | 论文URL失效 | 更新URL或手动下载 |
| `HTTPError: 403 Forbidden` | 访问被拒绝 | 检查User-Agent、添加延迟 |
| `UnicodeEncodeError` | 文件名编码问题 | 使用UTF-8编码 |
---
## 📊 成功案例
已成功下载论文：
1. ✅ **Teaching CS50 with AI (SIGCSE 2024)** - 1,688,180 字节（~1.6 MB）
2. ✅ **Improving AI in CS50 (SIGCSE 2025)** - 823,527 字节（~823 KB）
下载位置：`D:\TC_UP\2023card\projects\openclaw\教育AI研究\高等教育AI专题\文献库\`
---
## 🎯 最佳实践
### 学术诚信
⚠️ **重要提醒**：
- 下载的论文仅限个人学习和研究使用
- 请勿用于商业用途或大规模分发
- 遵守论文的版权协议
- 引用时请注明作者和来源
### 工作流建议
```
1. 下载论文 → 文献库/
2. 阅读论文 → 提取关键信息
3. 创建读书笔记 → Knowledge/
4. 更新Obsidian笔记链接
```
---
## 🔗 相关资源
- **论文下载工具**：`D:\TC_UP\2023card\projects\openclaw\教育AI研究\高等教育AI专题\tools\paper_downloader.py`
- **文献库目录**：`D:\TC_UP\2023card\projects\openclaw\教育AI研究\高等教育AI专题\文献库\`
- **示例配置**：`configs/sample_config.json`
---
**版本**：1.0.0
**更新时间**：2026-04-14
**作者**：Sisyphus (AI助手)
**许可证**：MIT
raw/教育AI研究/高等教育AI专题/tools/paper_downloader.py
+530
View File
@@ -0,0 +1,530 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
学术论文批量下载工具 (Academic Paper Downloader)
支持HTTP(S)/HTTPS重试、大文件分块下载、进度显示、日志记录
创建时间：2026-04-14
最后更新：2026-04-14
"""
import urllib.request
import ssl
import os
import sys
import io
import time
import json
from pathlib import Path
from urllib.error import URLError, HTTPError
from datetime import datetime
# Force UTF-8 encoding for console output
sys.stdout = io.TextIOWrapper(sys.stdout, encoding="utf-8")
# =============================================================================
# 配置区域
# =============================================================================
# 基础目录
BASE_DIR = r"D:\TC_UP\2023card\projects\openclaw\教育AI研究\高等教育AI专题\文献库"
LOG_DIR = os.path.join(BASE_DIR, "logs")
CONFIG_DIR = os.path.join(BASE_DIR, "configs")
os.makedirs(LOG_DIR, exist_ok=True)
os.makedirs(CONFIG_DIR, exist_ok=True)
# 下载配置
class DownloadConfig:
"""下载配置"""
def __init__(self):
self.timeout = 300 # 5分钟
self.retry_count = 3
self.retry_delay = 5 # 秒
self.chunk_size = 8192 # 8KB per chunk
self.user_agent = "Mozilla/5.0 (compatible; AcademicPaperDownloader/1.0)"
def set_timeout(self, seconds):
"""设置超时时间"""
self.timeout = seconds
def set_retry(self, count, delay):
"""设置重试次数和延迟"""
self.retry_count = count
self.retry_delay = delay
# SSL上下文
SSL_CONTEXT = ssl._create_default_https_context()
# =============================================================================
# 日志系统
# =============================================================================
class DownloadLogger:
"""下载日志记录器"""
def __init__(self, log_file):
self.log_file = os.path.join(LOG_DIR, log_file)
self.records = []
self._load_existing_logs()
def _load_existing_logs(self):
"""加载已有日志"""
if os.path.exists(self.log_file):
try:
with open(self.log_file, "r", encoding="utf-8") as f:
for line in f:
if line.strip():
try:
self.records.append(json.loads(line))
except:
pass
except:
pass
def log(self, paper_name, url, status, size_mb, error=None):
"""记录下载日志"""
record = {
"timestamp": datetime.now().isoformat(),
"paper": paper_name,
"url": url,
"status": status, # 'success' or 'failed'
"size_mb": size_mb,
"error": error,
}
self.records.append(record)
self._save_log()
def _save_log(self):
"""保存日志到文件"""
with open(self.log_file, "w", encoding="utf-8") as f:
for record in self.records:
f.write(json.dumps(record, ensure_ascii=False) + "\n")
def get_stats(self):
"""获取统计信息"""
total = len(self.records)
success = len([r for r in self.records if r["status"] == "success"])
failed = len([r for r in self.records if r["status"] == "failed"])
total_size = sum(
[r.get("size_mb", 0) for r in self.records if r["status"] == "success"]
)
return {
"total": total,
"success": success,
"failed": failed,
"success_rate": f"{(success / total * 100):.1f}%",
"total_size_mb": f"{total_size:.2f}",
"avg_size_mb": f"{(total_size / success if success > 0 else 0):.2f}",
}
# =============================================================================
# 核心下载器
# =============================================================================
class PaperDownloader:
"""学术论文下载器"""
def __init__(self, config):
self.config = config
self.logger = DownloadLogger("paper_downloads.log")
def download_with_retry(self, url, filepath, paper_name=""):
"""带重试机制的下载"""
filename = os.path.basename(filepath)
for attempt in range(self.config.retry_count):
try:
print(f"\n📥 尝试 {attempt + 1}/{self.config.retry_count}: {filename}")
return self._download(url, filepath, paper_name)
except (URLError, HTTPError) as e:
print(f"❌ 尝试 {attempt + 1}/{self.config.retry_count} 失败: {e}")
if attempt < self.config.retry_count - 1:
print(f"⏳ {self.config.retry_delay} 秒后重试...")
time.sleep(self.config.retry_delay)
else:
# 最后一次失败，记录日志
self.logger.log(
paper_name or filename, url, "failed", 0, error=str(e)
)
return False
def _download(self, url, filepath, paper_name=""):
"""实际下载方法，支持分块下载"""
try:
print(f"📥 开始下载: {paper_name or os.path.basename(filepath)}")
req = urllib.request.Request(
url, headers={"User-Agent": self.config.user_agent}
)
with urllib.request.urlopen(
req, context=SSL_CONTEXT, timeout=self.config.timeout
) as response:
total_size = int(response.headers.get("content-length", 0))
total_mb = total_size / (1024 * 1024)
print(f"📊 文件大小: {total_mb:.2f} MB")
# 创建目录
os.makedirs(os.path.dirname(filepath), exist_ok=True)
downloaded = 0
start_time = time.time()
with open(filepath, "wb") as f:
while downloaded < total_size:
chunk = response.read(self.config.chunk_size)
if not chunk:
break
f.write(chunk)
downloaded += len(chunk)
# 进度显示
percent = (downloaded / total_size) * 100
downloaded_mb = downloaded / (1024 * 1024)
elapsed = time.time() - start_time
# 进度条
progress_bar = "█" * int(percent / 5)
print(
f"\r进度: {percent:5.1f}% [{progress_bar:20}] {downloaded_mb:.2f}/{total_mb:.2f}MB {elapsed:.0f}s",
end="",
)
print() # 换行
# 记录成功日志
self.logger.log(
paper_name or os.path.basename(filepath), url, "success", total_mb
)
print(
f"\n✅ 下载成功: {paper_name or os.path.basename(filepath)} ({total_mb:.2f} MB)"
)
return True
except Exception as e:
print(f"\n❌ 下载失败: {e}")
return False
# =============================================================================
# 配置文件管理
# =============================================================================
def load_config(config_file):
"""从JSON文件加载配置"""
config_path = os.path.join(CONFIG_DIR, config_file)
if not os.path.exists(config_path):
print(f"⚠️ 配置文件不存在: {config_file}")
return None
try:
with open(config_path, "r", encoding="utf-8") as f:
config_data = json.load(f)
print(f"✅ 已加载配置: {config_file}")
print(f"📋 论文数量: {len(config_data.get('papers', []))}")
return config_data
except Exception as e:
print(f"❌ 加载配置失败: {e}")
return None
def create_sample_config():
"""创建示例配置文件（使用英文以避免编码问题）"""
sample_config = {
"name": "Academic Paper Downloader Configuration Sample",
"description": "Configuration file for batch downloading academic papers",
"papers": [
{
"name": "Teaching CS50 with AI (SIGCSE 2024)",
"url": "https://cs.harvard.edu/malan/publications/V1fp0567-liu.pdf",
"filename": "Teaching_CS50_with_AI_SIGCSE2024.pdf",
"enabled": True,
},
{
"name": "Improving AI in CS50 (SIGCSE 2025)",
"url": "https://cs.harvard.edu/malan/publications/fp0627-liu.pdf",
"filename": "Improving_AI_in_CS50_SIGCSE2025.pdf",
"enabled": True,
},
],
}
sample_path = os.path.join(CONFIG_DIR, "sample_config.json")
try:
with open(sample_path, "w", encoding="utf-8") as f:
json.dump(sample_data, f, ensure_ascii=False, indent=2)
print("Sample config created successfully: " + sample_path)
return sample_path
except Exception as e:
print(f"❌ 创建示例配置失败: {e}")
return None
# =============================================================================
# 主命令
# =============================================================================
def download_from_config(config_file):
"""从配置文件批量下载"""
config_data = load_config(config_file)
if not config_data:
return
downloader = PaperDownloader(DownloadConfig())
papers = config_data.get("papers", [])
enabled_papers = [p for p in papers if p.get("enabled", True)]
if not enabled_papers:
print("⚠️ 没有启用的论文可下载")
return
print(f"\n{'=' * 50}")
print(f"准备下载 {len(enabled_papers)} 篇论文...\n")
success_count = 0
failed_count = 0
for idx, paper in enumerate(enabled_papers, 1):
print(
f"\n[{'=' * 40}/{len(enabled_papers)}] {paper.get('name', 'Unknown Paper')}"
)
url = paper.get("url")
filename = paper.get("filename", f"paper_{idx}.pdf")
filepath = os.path.join(BASE_DIR, filename)
name = paper.get("name", "Unknown Paper")
if downloader.download_with_retry(url, filepath, name):
success_count += 1
else:
failed_count += 1
print(f"\n{'=' * 50}")
print(f"✅ 成功: {success_count}")
print(f"❌ 失败: {failed_count}")
# 显示统计
stats = downloader.logger.get_stats()
print(f"\n📊 下载统计:")
print(f"总论文数: {stats['total']}")
print(f"成功率: {stats['success_rate']}")
print(f"总下载大小: {stats['total_size_mb']} MB")
print(f"平均大小: {stats['avg_size_mb']} MB")
def download_single(url, filename=None):
"""单篇论文下载"""
if not filename:
# 从URL提取文件名
filename = url.split("/")[-1]
if not filename.endswith(".pdf"):
filename = filename + ".pdf"
filepath = os.path.join(BASE_DIR, filename)
paper_name = os.path.splitext(filename)[0]
downloader = PaperDownloader(DownloadConfig())
downloader.download_with_retry(url, filepath, paper_name)
# 显示统计
stats = downloader.logger.get_stats()
print(f"\n📊 当前下载统计:")
print(f"总下载次数: {stats['total']}")
print(f"成功率: {stats['success_rate']}")
print(f"总下载大小: {stats['total_size_mb']} MB")
def show_logs():
"""显示下载日志"""
stats = DownloadLogger("paper_downloads.log").get_stats()
print(f"\n📊 下载日志统计:")
print(f"{'=' * 40}")
print(f"总下载次数: {stats['total']}")
print(f"成功次数: {stats['success']}")
print(f"失败次数: {stats['failed']}")
print(f"成功率: {stats['success_rate']}")
print(f"总下载大小: {stats['total_size_mb']} MB")
print(f"平均大小: {stats['avg_size_mb']} MB")
print(f"日志文件: {os.path.join(LOG_DIR, 'paper_downloads.log')}")
def clear_logs():
"""清除日志"""
log_file = os.path.join(LOG_DIR, "paper_downloads.log")
try:
if os.path.exists(log_file):
os.remove(log_file)
print(f"✅ 日志已清除: {log_file}")
else:
print("⚠️ 日志文件不存在")
except Exception as e:
print(f"❌ 清除日志失败: {e}")
# =============================================================================
# 主程序
# =============================================================================
def print_help():
"""显示帮助信息"""
help_text = """
学术论文批量下载工具 (Academic Paper Downloader)
使用方法:
python paper_downloader.py <command> [options]
命令:
download <url> 下载单篇论文
config <config.json> 从配置文件批量下载
logs 显示下载日志统计
clear 清除下载日志
create-sample 创建示例配置文件
help 显示此帮助信息
选项:
--timeout <seconds> 设置超时时间（默认: 300秒）
--retry <count> 设置重试次数（默认: 3次）
--chunk-size <bytes> 设置分块大小（默认: 8192字节）
--no-progress 不显示进度条
示例:
# 下载单篇论文
python paper_downloader.py download https://cs.harvard.edu/malan/publications/V1fp0567-liu.pdf
# 从配置文件批量下载
python paper_downloader.py config my_papers.json
# 创建示例配置文件
python paper_downloader.py create-sample
# 查看下载日志
python paper_downloader.py logs
# 清除下载日志
python paper_downloader.py clear
配置文件格式:
{
"name": "论文集合名称",
"papers": [
{
"name": "论文标题",
"url": "论文URL",
"filename": "保存的文件名.pdf",
"enabled": true
}
]
}
作者: Sisyphus
版本: 1.0.0
更新: 2026-04-14
"""
print(help_text)
def main():
"""主函数"""
if len(sys.argv) < 2:
print_help()
return
command = sys.argv[1].lower()
# 解析选项
options = {"timeout": 300, "retry": 3, "chunk_size": 8192, "no_progress": False}
args = sys.argv[2:]
i = 0
while i < len(args):
if args[i] == "--timeout":
if i + 1 < len(args):
options["timeout"] = int(args[i + 1])
i += 2
elif args[i] == "--retry":
if i + 1 < len(args):
options["retry"] = int(args[i + 1])
i += 2
elif args[i] == "--chunk-size":
if i + 1 < len(args):
options["chunk_size"] = int(args[i + 1])
i += 2
elif args[i] == "--no-progress":
options["no_progress"] = True
i += 1
else:
print(f"⚠️ 未知选项: {args[i]}")
i += 1
# 应用配置
config = DownloadConfig()
if options["timeout"]:
config.set_timeout(options["timeout"])
if options["retry"]:
config.set_retry(
options["retry"], options.get("retry_delay", config.retry_delay)
)
if options["chunk_size"]:
config.chunk_size = options["chunk_size"]
if options["no_progress"]:
config.user_agent = (
"Mozilla/5.0 (compatible; AcademicPaperDownloader/1.0) (No Progress)"
)
# 执行命令
try:
if command == "download":
if len(args) > i:
download_single(args[i])
else:
print("❌ 请提供论文URL")
elif command == "config":
if len(args) > i:
download_from_config(args[i])
else:
print("❌ 请提供配置文件路径")
elif command == "logs":
show_logs()
elif command == "clear":
clear_logs()
elif command == "create-sample":
create_sample_config()
elif command == "help":
print_help()
else:
print(f"❌ 未知命令: {command}")
print_help()
except KeyboardInterrupt:
print("\n\n⚠️ 下载已中断")
except Exception as e:
print(f"\n❌ 发生错误: {e}")
if __name__ == "__main__":
main()