feat(p1): add daily snapshots and fan tracking
Implement P1: seven-metric view snapshots, relation/stat fan snapshots, failure placeholders, resume by daily completion markers, Cookie login warnings, daily SQLite backups, Shanghai daily scheduling, and CLI snapshot/auth-check commands. Real read-only run captured 795/795 videos successfully over 1124s with a verified daily backup.
This commit is contained in:
1 parent
ee289bfa56
commit
5c2ab7b187
20 files changed
+1680
-54
No files matched your search
+29
-3
@@ -1,6 +1,6 @@
|
|||||||
# 咻咻满 Reaction 视频数据看板开发文档
|
# 咻咻满 Reaction 视频数据看板开发文档
|
||||||
|
|
||||||
> **状态:** P0(数据底座与收藏夹同步)已实施并通过只读联调;P1 待开始
|
> **状态:** P0(数据底座与收藏夹同步)、P1(每日快照与粉丝追踪)已实施并通过只读联调;P2 待开始
|
||||||
>
|
>
|
||||||
> **技术基线:** Python 3.14、Python 标准库、SQLite、原生 HTML/CSS/JavaScript、`uv`
|
> **技术基线:** Python 3.14、Python 标准库、SQLite、原生 HTML/CSS/JavaScript、`uv`
|
||||||
>
|
>
|
||||||
@@ -95,6 +95,13 @@ uv run python app.py sync
|
|||||||
uv run python app.py run
|
uv run python app.py run
|
||||||
uv run python app.py run --once
|
uv run python app.py run --once
|
||||||
|
|
||||||
|
# 每日七项快照与粉丝数(P1)
|
||||||
|
uv run python app.py snapshot
|
||||||
|
uv run python app.py snapshot --force
|
||||||
|
|
||||||
|
# 检查 Cookie 登录态(P1)
|
||||||
|
uv run python app.py auth-check
|
||||||
|
|
||||||
# 查看数据库状态(离线,P0)
|
# 查看数据库状态(离线,P0)
|
||||||
uv run python app.py status
|
uv run python app.py status
|
||||||
|
|
||||||
@@ -388,7 +395,7 @@ score < 0:丢弃
|
|||||||
|
|
||||||
- 对 `status IN ('active', 'unfaved')` 的视频逐条请求 `view?aid=`;
|
- 对 `status IN ('active', 'unfaved')` 的视频逐条请求 `view?aid=`;
|
||||||
- 成功写入播放、点赞、收藏、投币、转发、评论、弹幕七项数据;
|
- 成功写入播放、点赞、收藏、投币、转发、评论、弹幕七项数据;
|
||||||
- `-404` 时标记 `removed`;
|
- `-404` / `62002` / `62012` 时标记 `removed`(62002/62012 为当前实测返回);
|
||||||
- 其他错误最多重试 3 次,仍失败则写入 `ok=0` 与错误信息,不写伪造的零值;
|
- 其他错误最多重试 3 次,仍失败则写入 `ok=0` 与错误信息,不写伪造的零值;
|
||||||
- 每条视频请求之间随机等待 1.0~1.6 秒;
|
- 每条视频请求之间随机等待 1.0~1.6 秒;
|
||||||
- 对 `track_fans` 中的每个 mid 请求一次 `relation/stat` 并写入 `up_stat`;
|
- 对 `track_fans` 中的每个 mid 请求一次 `relation/stat` 并写入 `up_stat`;
|
||||||
@@ -396,6 +403,15 @@ score < 0:丢弃
|
|||||||
|
|
||||||
服务启动后若当天快照尚未成功完成,应补跑当日任务;已成功的 `(stat_date, aid)` 不重复请求。
|
服务启动后若当天快照尚未成功完成,应补跑当日任务;已成功的 `(stat_date, aid)` 不重复请求。
|
||||||
|
|
||||||
|
P1 实现说明:
|
||||||
|
|
||||||
|
- 时区使用固定 UTC+8(中国无夏令时),避免 Windows 缺少 `tzdata`;
|
||||||
|
- 当日全部视频与粉丝采集成功后在 `kv` 写入 `snapshot:completed:<date>=1`;启动时未完成会补跑;
|
||||||
|
- `ok=0` 的当日记录会在下次补跑时重试;`ok=1` 直接跳过;
|
||||||
|
- 每次采集后生成 `data/backup/app-YYYYMMDD.db` 并清理超过 `storage.backup_keep_days` 的日备份;
|
||||||
|
- Cookie 登录态通过 `x/web-interface/nav` 检查,失效只告警,不阻塞匿名 `view`;
|
||||||
|
- Telegram 摘要推迟到 P3,P1 先写结构化 JSON 日志和 `kv` 状态。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 8. Telegram 与收藏操作
|
## 8. Telegram 与收藏操作
|
||||||
@@ -555,12 +571,22 @@ uv run python tests/test_moves.py
|
|||||||
- 黑名单 9 个 mid 已写入 `up_blacklist`,且不参与收藏夹同步过滤;
|
- 黑名单 9 个 mid 已写入 `up_blacklist`,且不参与收藏夹同步过滤;
|
||||||
- 自动化测试见 `tests/test_config.py`、`test_migrations.py`、`test_database.py`、`test_repository.py`、`test_bilibili_client.py`、`test_folder_sync.py`、`test_scheduler.py`、`test_logging.py`、`test_cli.py`。
|
- 自动化测试见 `tests/test_config.py`、`test_migrations.py`、`test_database.py`、`test_repository.py`、`test_bilibili_client.py`、`test_folder_sync.py`、`test_scheduler.py`、`test_logging.py`、`test_cli.py`。
|
||||||
|
|
||||||
### P1:快照与粉丝追踪
|
### P1:快照与粉丝追踪 ✅
|
||||||
|
|
||||||
- 每日七项视频快照;
|
- 每日七项视频快照;
|
||||||
- 粉丝快照;
|
- 粉丝快照;
|
||||||
- 失败占位、断点续跑、Cookie 失效告警、SQLite 备份。
|
- 失败占位、断点续跑、Cookie 失效告警、SQLite 备份。
|
||||||
|
|
||||||
|
完成情况(2026-10-31):
|
||||||
|
|
||||||
|
- 已实现 `services/snapshot.py`:`view` 七项指标、`relation/stat` 粉丝数、`ok=0` 失败占位、`ok=1` 断点跳过;
|
||||||
|
- 实测不可访问视频返回 `62012`,已和旧口径 `-404`、`62002` 一起按 `removed` 处理;
|
||||||
|
- `run` 调度器同时注册收藏夹同步与每日 `snapshot_at` 任务,启动时立即检查当日是否完成以支持补跑;
|
||||||
|
- 每次快照后生成日备份并清理超过 14 天的日备份;
|
||||||
|
- `auth-check` 与快照启动时检查 Cookie 登录态,失效写 `kv` 并告警;
|
||||||
|
- 真实全量联调:795 条视频全部 `ok=1`,0 失败、0 移除,耗时 1124 秒;粉丝 `follower=2767270`;日备份 `data/backup/app-20261031.db` 含 795 条快照;二次执行正确命中完成标记并跳过;
|
||||||
|
- 自动化测试见 `tests/test_snapshot.py`、`test_maintenance.py`、`test_timeutil.py`,并扩展了 `test_bilibili_client.py` 与 `test_scheduler.py`。
|
||||||
|
|
||||||
### P2:本地看板
|
### P2:本地看板
|
||||||
|
|
||||||
- 总览、榜单、详情、UP、粉丝、候选和系统状态页面;
|
- 总览、榜单、详情、UP、粉丝、候选和系统状态页面;
|
||||||
|
|||||||
@@ -4,9 +4,9 @@
|
|||||||
|
|
||||||
项目以 B 站年度收藏夹为可信视频来源,补充关键词发现和 Telegram 人工确认,持续采集 reaction 视频的七项数据与咻咻满粉丝数,并提供本地数据看板。现有的收藏夹排序工具仍保留并作为一键加入后的排序能力基础。
|
项目以 B 站年度收藏夹为可信视频来源,补充关键词发现和 Telegram 人工确认,持续采集 reaction 视频的七项数据与咻咻满粉丝数,并提供本地数据看板。现有的收藏夹排序工具仍保留并作为一键加入后的排序能力基础。
|
||||||
|
|
||||||
> 当前状态:**P0(数据底座与收藏夹同步)已实施,P1 待开始。**
|
> 当前状态:**P0(数据底座与收藏夹同步)、P1(快照与粉丝追踪)已实施;P2 待开始。**
|
||||||
>
|
>
|
||||||
> 当前已可用:扫码授权、收藏夹读取、排序计划生成、执行与校验;SQLite 建库迁移、收藏夹首次导入与增量同步、失效识别、黑名单种子。
|
> 当前已可用:扫码授权、收藏夹读取、排序计划生成、执行与校验;SQLite 建库迁移、收藏夹首次导入与增量同步、失效识别、黑名单种子;每日七项视频快照、粉丝数快照、失败占位、断点续跑与日备份。
|
||||||
>
|
>
|
||||||
> 详细业务口径、接口验证和风险记录见 [project.md](project.md);开发架构与实施计划见 [DEVELOPMENT.md](DEVELOPMENT.md)。
|
> 详细业务口径、接口验证和风险记录见 [project.md](project.md);开发架构与实施计划见 [DEVELOPMENT.md](DEVELOPMENT.md)。
|
||||||
|
|
||||||
@@ -17,8 +17,8 @@
|
|||||||
| 收藏夹排序 | 可用 | 同一 UP 连续,组内按发布时间倒序,组间按该 UP 最早视频发布时间倒序。 |
|
| 收藏夹排序 | 可用 | 同一 UP 连续,组内按发布时间倒序,组间按该 UP 最早视频发布时间倒序。 |
|
||||||
| 扫码授权与 Cookie 续期 | 可用 | 本地生成二维码;更新身份 Cookie 时保留设备指纹。 |
|
| 扫码授权与 Cookie 续期 | 可用 | 本地生成二维码;更新身份 Cookie 时保留设备指纹。 |
|
||||||
| 收藏夹同步与首次导入 | 可用(P0) | 监控已配置年度夹,自动发现人工新收藏的视频与失效资源。 |
|
| 收藏夹同步与首次导入 | 可用(P0) | 监控已配置年度夹,自动发现人工新收藏的视频与失效资源。 |
|
||||||
| 视频日快照 | 规划(P1) | 逐条采集播放、点赞、投币、收藏、转发、评论、弹幕。 |
|
| 视频日快照 | 可用(P1) | 逐条采集播放、点赞、投币、收藏、转发、评论、弹幕。 |
|
||||||
| 粉丝追踪 | 规划(P1) | 每日记录咻咻满等配置 UP 的粉丝数。 |
|
| 粉丝追踪 | 可用(P1) | 每日记录咻咻满等配置 UP 的粉丝数。 |
|
||||||
| 本地数据看板 | 规划(P2) | 总览、榜单、视频/UP 详情、粉丝曲线、候选池和系统状态。 |
|
| 本地数据看板 | 规划(P2) | 总览、榜单、视频/UP 详情、粉丝曲线、候选池和系统状态。 |
|
||||||
| Telegram 发现与一键加入 | 规划(P3) | 每小时发现候选,支持收藏、忽略、拉黑和收藏夹内定位。 |
|
| Telegram 发现与一键加入 | 规划(P3) | 每小时发现候选,支持收藏、忽略、拉黑和收藏夹内定位。 |
|
||||||
|
|
||||||
@@ -68,7 +68,7 @@ uv run python app.py init
|
|||||||
# 首次导入或增量同步配置中的年度收藏夹(只读联网)
|
# 首次导入或增量同步配置中的年度收藏夹(只读联网)
|
||||||
uv run python app.py sync
|
uv run python app.py sync
|
||||||
|
|
||||||
# 按 folder_sync_minutes 常驻周期同步;--once 只跑一次
|
# 按配置周期常驻运行收藏夹同步 + 每日快照;--once 只跑一次到期任务
|
||||||
uv run python app.py run
|
uv run python app.py run
|
||||||
uv run python app.py run --once
|
uv run python app.py run --once
|
||||||
|
|
||||||
@@ -86,6 +86,32 @@ uv run python app.py status
|
|||||||
- 黑名单只作为发现阶段的种子写入,**不会过滤收藏夹同步**;
|
- 黑名单只作为发现阶段的种子写入,**不会过滤收藏夹同步**;
|
||||||
- 收藏夹达到 950 条时预警,系统绝不自动换夹。
|
- 收藏夹达到 950 条时预警,系统绝不自动换夹。
|
||||||
|
|
||||||
|
## P1:每日快照与粉丝追踪
|
||||||
|
|
||||||
|
P1 已实现每日七项视频指标快照、咻咻满粉丝数快照、失败占位、断点续跑、Cookie 失效告警和 SQLite 日备份。
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
# 采集当日快照与粉丝数(只读联网,约 18~20 分钟 / 795 条)
|
||||||
|
uv run python app.py snapshot
|
||||||
|
|
||||||
|
# 忽略当日完成标记,重新采集
|
||||||
|
uv run python app.py snapshot --force
|
||||||
|
|
||||||
|
# 检查 Cookie 登录态
|
||||||
|
uv run python app.py auth-check
|
||||||
|
```
|
||||||
|
|
||||||
|
- `snapshot` 对 `active` / `unfaved` 视频逐条请求 `view`,每条请求间隔 1.0~1.6 秒;
|
||||||
|
- 成功写入 `snapshot` 的七项指标;同日 `ok=1` 的记录直接跳过,可断点续跑;
|
||||||
|
- 视频不可访问(`-404` / `62002` / `62012`)标记为 `removed`;
|
||||||
|
- 其他错误重试 3 次,仍失败写入 `ok=0` 和错误信息,不写伪造的零值;
|
||||||
|
- 粉丝数写入 `up_stat`,失败同样写 `ok=0`;
|
||||||
|
- 当日全部视频与粉丝采集成功后写入完成标记;服务启动时若当天未完成会自动补跑;
|
||||||
|
- 每次采集后生成 `data/backup/app-YYYYMMDD.db`,按 `storage.backup_keep_days` 清理旧日备份;
|
||||||
|
- Cookie 失效只告警,不阻塞匿名 `view` 采集;`auth-check` 可单独检查。
|
||||||
|
|
||||||
|
首次真实全量采集结果:795 条视频全部成功、0 失败、0 移除,耗时约 19 分钟;粉丝 `follower=2767270`;日备份为 `data/backup/app-20261031.db`。
|
||||||
|
|
||||||
## 配置
|
## 配置
|
||||||
|
|
||||||
复制样例配置,并填写 B 站账号信息:
|
复制样例配置,并填写 B 站账号信息:
|
||||||
@@ -198,8 +224,11 @@ uv run python tests/test_config.py # 配置加载、校验、ENV 覆盖
|
|||||||
uv run python tests/test_migrations.py # 迁移、备份、回滚与保留期
|
uv run python tests/test_migrations.py # 迁移、备份、回滚与保留期
|
||||||
uv run python tests/test_database.py # 单写队列、事务与并发
|
uv run python tests/test_database.py # 单写队列、事务与并发
|
||||||
uv run python tests/test_repository.py # 收藏夹/视频状态机与黑名单
|
uv run python tests/test_repository.py # 收藏夹/视频状态机与黑名单
|
||||||
uv run python tests/test_bilibili_client.py # ids/infos 契约、分块、缺失与重试
|
uv run python tests/test_bilibili_client.py # ids/infos/view/fans/nav 契约与重试
|
||||||
uv run python tests/test_folder_sync.py # 首次导入、差集、失效识别、容量预警
|
uv run python tests/test_folder_sync.py # 首次导入、差集、失效识别、容量预警
|
||||||
|
uv run python tests/test_snapshot.py # 七项快照、断点续跑、失败占位、粉丝
|
||||||
|
uv run python tests/test_maintenance.py # 日备份与保留期
|
||||||
|
uv run python tests/test_timeutil.py # 上海时区与每日任务时间
|
||||||
uv run python tests/test_scheduler.py # 内置调度器到期/失败隔离
|
uv run python tests/test_scheduler.py # 内置调度器到期/失败隔离
|
||||||
uv run python tests/test_logging.py # JSON 日志脱敏
|
uv run python tests/test_logging.py # JSON 日志脱敏
|
||||||
uv run python tests/test_cli.py # CLI 离线路径
|
uv run python tests/test_cli.py # CLI 离线路径
|
||||||
@@ -211,8 +240,9 @@ uv run python tests/test_cli.py # CLI 离线路径
|
|||||||
config.json.bak 最近一次 auth 更新前的配置备份
|
config.json.bak 最近一次 auth 更新前的配置备份
|
||||||
data/plan-<media_id>.json 排序计划、顺序快照和元数据缓存
|
data/plan-<media_id>.json 排序计划、顺序快照和元数据缓存
|
||||||
data/journal-<media_id>.jsonl 排序执行日志
|
data/journal-<media_id>.jsonl 排序执行日志
|
||||||
data/app.db P0 起的 SQLite 数据库(收藏夹、视频、黑名单)
|
data/app.db SQLite 数据库(收藏夹、视频、快照、粉丝、黑名单)
|
||||||
data/backup/ 数据库迁移前的自动备份
|
data/backup/app-YYYYMMDD.db 每日快照后的 SQLite 备份
|
||||||
|
data/backup/app-pre-v*.db 数据库迁移前的自动备份
|
||||||
```
|
```
|
||||||
|
|
||||||
## 文档索引
|
## 文档索引
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""看板服务入口。
|
"""看板服务入口。
|
||||||
|
|
||||||
P0 提供 ``init`` / ``sync`` / ``run`` / ``status`` 子命令;后续阶段会在同一入口启动
|
P1 提供 ``init`` / ``sync`` / ``snapshot`` / ``auth-check`` / ``run`` / ``status``
|
||||||
本地 Web 服务。
|
子命令;后续阶段会在同一入口启动本地 Web 服务。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import sys
|
import sys
|
||||||
|
|||||||
+7
-2
@@ -1,7 +1,7 @@
|
|||||||
# 咻咻满 Reaction 视频数据看板 —— 可行性分析与技术路线
|
# 咻咻满 Reaction 视频数据看板 —— 可行性分析与技术路线
|
||||||
|
|
||||||
> 状态:**设计定稿;P0(数据底座与收藏夹同步)已实施并完成只读联调**
|
> 状态:**设计定稿;P0(数据底座与收藏夹同步)、P1(每日快照与粉丝追踪)已实施**
|
||||||
> 最后更新:2026-10-31(第 5 轮:P0 落地)
|
> 最后更新:2026-10-31(第 6 轮:P1 落地)
|
||||||
> 全部探测均为只读(GET),未调用 `fav/resource/deal`、`fav/resource/sort`
|
> 全部探测均为只读(GET),未调用 `fav/resource/deal`、`fav/resource/sort`
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -770,6 +770,11 @@ storage:SQLite WAL,单写线程
|
|||||||
| 21 | **搜索 item 的 duration 格式** | `分钟:秒`(`105:5`、`336:31`),超 60 分钟不转小时 |
|
| 21 | **搜索 item 的 duration 格式** | `分钟:秒`(`105:5`、`336:31`),超 60 分钟不转小时 |
|
||||||
| 22 | **搜索 item 自带统计字段** | `play/like/favorites/danmaku/review/duration/cover/pubdate/mid/uname` 齐全,通知可用索引值 |
|
| 22 | **搜索 item 自带统计字段** | `play/like/favorites/danmaku/review/duration/cover/pubdate/mid/uname` 齐全,通知可用索引值 |
|
||||||
| 23 | **初始黑名单 9 个 mid 核对** | `x/web-interface/card` 全部 `code=0`,昵称/粉丝数已记录(§1.6) |
|
| 23 | **初始黑名单 9 个 mid 核对** | `x/web-interface/card` 全部 `code=0`,昵称/粉丝数已记录(§1.6) |
|
||||||
|
| 24 | `x/web-interface/view` 七项 stat | `code=0`,view/like/favorite/coin/share/reply/danmaku 齐全 |
|
||||||
|
| 25 | `view` 不可访问返回码 | 实测为 `62012`;代码同时兼容 `-404`、`62002`、HTTP 404 |
|
||||||
|
| 26 | `x/relation/stat` 粉丝数 | `code=0`,匿名可用,实测 `follower=2767268` |
|
||||||
|
| 27 | `x/web-interface/nav` 登录态 | `code=0`、`isLogin=true`、mid=28802351 |
|
||||||
|
| 28 | **P1 真实全量快照** | 795 条全部 `ok=1`、0 失败、0 移除;耗时 1124s;粉丝 `follower=2767270`;日备份 `app-20261031.db` 含 795 条快照 |
|
||||||
|
|
||||||
**未验证(需授权或外部配合)**:
|
**未验证(需授权或外部配合)**:
|
||||||
|
|
||||||
|
|||||||
@@ -18,11 +18,21 @@ from dataclasses import dataclass
|
|||||||
from typing import Any, Mapping, Protocol, Sequence
|
from typing import Any, Mapping, Protocol, Sequence
|
||||||
|
|
||||||
from .auth import make_headers
|
from .auth import make_headers
|
||||||
from .models import FavId, FavResource, FolderInfo, InfosResult
|
from .models import (
|
||||||
|
FanStat,
|
||||||
|
FavId,
|
||||||
|
FavResource,
|
||||||
|
FolderInfo,
|
||||||
|
InfosResult,
|
||||||
|
LoginState,
|
||||||
|
ViewStat,
|
||||||
|
)
|
||||||
|
|
||||||
BASE = "https://api.bilibili.com"
|
BASE = "https://api.bilibili.com"
|
||||||
DEFAULT_CHUNK = 200
|
DEFAULT_CHUNK = 200
|
||||||
MAX_CHUNK = 400
|
MAX_CHUNK = 400
|
||||||
|
# 视频不可访问/已删除:-404 为旧口径,62002/62012 为当前实测返回。
|
||||||
|
VIDEO_UNAVAILABLE_CODES = frozenset({-404, 62002, 62012})
|
||||||
|
|
||||||
|
|
||||||
class BilibiliError(RuntimeError):
|
class BilibiliError(RuntimeError):
|
||||||
@@ -360,3 +370,64 @@ class BilibiliClient:
|
|||||||
media_count=count,
|
media_count=count,
|
||||||
))
|
))
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
# ------------------------------------------------------------ view / fan
|
||||||
|
|
||||||
|
def view(self, aid: int) -> ViewStat:
|
||||||
|
url = f"{BASE}/x/web-interface/view?aid={int(aid)}"
|
||||||
|
payload = self.get_json(url)
|
||||||
|
data = payload.get("data")
|
||||||
|
if not isinstance(data, Mapping):
|
||||||
|
raise BilibiliProtocolError("view data 不是对象")
|
||||||
|
try:
|
||||||
|
return ViewStat.from_api(data)
|
||||||
|
except (TypeError, ValueError) as exc:
|
||||||
|
raise BilibiliProtocolError(f"view 解析失败: {exc}") from exc
|
||||||
|
|
||||||
|
def relation_stat(self, mid: int) -> FanStat:
|
||||||
|
url = f"{BASE}/x/relation/stat?vmid={int(mid)}"
|
||||||
|
payload = self.get_json(url)
|
||||||
|
data = payload.get("data")
|
||||||
|
if not isinstance(data, Mapping):
|
||||||
|
raise BilibiliProtocolError("relation/stat data 不是对象")
|
||||||
|
try:
|
||||||
|
return FanStat.from_api(data)
|
||||||
|
except (TypeError, ValueError) as exc:
|
||||||
|
raise BilibiliProtocolError(f"relation/stat 解析失败: {exc}") from exc
|
||||||
|
|
||||||
|
def check_login(self) -> LoginState:
|
||||||
|
"""检查 Cookie 登录态;未配置 Cookie 时不发请求。"""
|
||||||
|
if not self.cookie:
|
||||||
|
return LoginState(
|
||||||
|
is_login=False,
|
||||||
|
mid=None,
|
||||||
|
uname=None,
|
||||||
|
code=0,
|
||||||
|
message="未配置 cookie",
|
||||||
|
)
|
||||||
|
url = f"{BASE}/x/web-interface/nav"
|
||||||
|
try:
|
||||||
|
payload = self.get_json(url, auth=True)
|
||||||
|
except BilibiliAPIError as exc:
|
||||||
|
return LoginState(
|
||||||
|
is_login=False, mid=None, uname=None,
|
||||||
|
code=exc.code, message=exc.message,
|
||||||
|
)
|
||||||
|
except BilibiliError as exc:
|
||||||
|
return LoginState(
|
||||||
|
is_login=False, mid=None, uname=None,
|
||||||
|
code=-1, message=str(exc),
|
||||||
|
)
|
||||||
|
data = payload.get("data") or {}
|
||||||
|
if not isinstance(data, Mapping):
|
||||||
|
raise BilibiliProtocolError("nav data 不是对象")
|
||||||
|
raw_mid = data.get("mid")
|
||||||
|
mid = raw_mid if isinstance(raw_mid, int) and not isinstance(raw_mid, bool) else None
|
||||||
|
uname = data.get("uname")
|
||||||
|
return LoginState(
|
||||||
|
is_login=bool(data.get("isLogin")),
|
||||||
|
mid=mid,
|
||||||
|
uname=str(uname) if isinstance(uname, str) else None,
|
||||||
|
code=0,
|
||||||
|
message=str(payload.get("message") or ""),
|
||||||
|
)
|
||||||
@@ -85,3 +85,83 @@ class InfosResult:
|
|||||||
requested: tuple[int, ...]
|
requested: tuple[int, ...]
|
||||||
items: dict[int, FavResource]
|
items: dict[int, FavResource]
|
||||||
missing: tuple[int, ...]
|
missing: tuple[int, ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ViewStat:
|
||||||
|
"""``x/web-interface/view`` 的七项统计。"""
|
||||||
|
|
||||||
|
aid: int
|
||||||
|
bvid: str | None
|
||||||
|
title: str
|
||||||
|
view: int
|
||||||
|
like: int
|
||||||
|
favorite: int
|
||||||
|
coin: int
|
||||||
|
share: int
|
||||||
|
reply: int
|
||||||
|
danmaku: int
|
||||||
|
|
||||||
|
def metrics(self) -> dict[str, int]:
|
||||||
|
return {
|
||||||
|
"view": self.view,
|
||||||
|
"like": self.like,
|
||||||
|
"favorite": self.favorite,
|
||||||
|
"coin": self.coin,
|
||||||
|
"share": self.share,
|
||||||
|
"reply": self.reply,
|
||||||
|
"danmaku": self.danmaku,
|
||||||
|
}
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_api(cls, item: Mapping[str, Any]) -> "ViewStat":
|
||||||
|
aid = item.get("aid")
|
||||||
|
if isinstance(aid, bool) or not isinstance(aid, int):
|
||||||
|
raise ValueError("view data.aid 不是整数")
|
||||||
|
stat = item.get("stat")
|
||||||
|
if not isinstance(stat, Mapping):
|
||||||
|
raise ValueError("view data.stat 不是对象")
|
||||||
|
metrics: dict[str, int] = {}
|
||||||
|
for key in ("view", "like", "favorite", "coin", "share", "reply", "danmaku"):
|
||||||
|
value = stat.get(key)
|
||||||
|
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
|
||||||
|
raise ValueError(f"view stat.{key} 不是非负整数")
|
||||||
|
metrics[key] = value
|
||||||
|
bvid = item.get("bvid")
|
||||||
|
if not isinstance(bvid, str) or not bvid:
|
||||||
|
bvid = None
|
||||||
|
return cls(
|
||||||
|
aid=aid,
|
||||||
|
bvid=bvid,
|
||||||
|
title=str(item.get("title") or ""),
|
||||||
|
**metrics,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class FanStat:
|
||||||
|
mid: int
|
||||||
|
follower: int
|
||||||
|
following: int
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_api(cls, item: Mapping[str, Any]) -> "FanStat":
|
||||||
|
mid = item.get("mid")
|
||||||
|
follower = item.get("follower")
|
||||||
|
following = item.get("following", 0)
|
||||||
|
if isinstance(mid, bool) or not isinstance(mid, int):
|
||||||
|
raise ValueError("relation/stat data.mid 不是整数")
|
||||||
|
if isinstance(follower, bool) or not isinstance(follower, int) or follower < 0:
|
||||||
|
raise ValueError("relation/stat data.follower 不是非负整数")
|
||||||
|
if isinstance(following, bool) or not isinstance(following, int) or following < 0:
|
||||||
|
raise ValueError("relation/stat data.following 不是非负整数")
|
||||||
|
return cls(mid=mid, follower=follower, following=following)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class LoginState:
|
||||||
|
is_login: bool
|
||||||
|
mid: int | None
|
||||||
|
uname: str | None
|
||||||
|
code: int
|
||||||
|
message: str
|
||||||
+121
-31
@@ -1,10 +1,12 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""P0 命令行入口。
|
"""看板命令行入口。
|
||||||
|
|
||||||
python app.py init # 建库 / 迁移 / 写入黑名单种子(离线)
|
python app.py init # 建库 / 迁移 / 写入黑名单种子(离线)
|
||||||
python app.py sync # 收藏夹只读同步(首次导入或增量)
|
python app.py sync # 收藏夹只读同步(首次导入或增量)
|
||||||
python app.py run # 按 folder_sync_minutes 常驻周期同步(--once 只跑一次)
|
python app.py snapshot # 采集当日七项快照与粉丝数
|
||||||
python app.py status # 查看库内状态(离线)
|
python app.py auth-check # 检查 Cookie 登录态
|
||||||
|
python app.py run # 按配置周期常驻运行(--once 只跑一次)
|
||||||
|
python app.py status # 查看库内状态(离线)
|
||||||
|
|
||||||
P2 起 ``python app.py`` 将在此基础上启动本地看板服务。
|
P2 起 ``python app.py`` 将在此基础上启动本地看板服务。
|
||||||
"""
|
"""
|
||||||
@@ -17,13 +19,15 @@ import sys
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Sequence
|
from typing import Sequence
|
||||||
|
|
||||||
from .bilibili.client import BilibiliClient
|
from .bilibili.client import BilibiliClient, Pacer
|
||||||
from .config import ConfigError, ConfigStore
|
from .config import ConfigError, ConfigStore
|
||||||
from .database import Database
|
from .database import Database
|
||||||
from .logging_json import JsonLogger
|
from .logging_json import JsonLogger
|
||||||
from .repository import Repository
|
from .repository import Repository
|
||||||
from .scheduler import ScheduledTask, Scheduler
|
from .scheduler import ScheduledTask, Scheduler
|
||||||
from .services.folder_sync import FolderSyncService
|
from .services.folder_sync import FolderSyncService
|
||||||
|
from .services.snapshot import SnapshotService
|
||||||
|
from .timeutil import next_daily_run, parse_hhmm, shanghai_date
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parent.parent
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
DEFAULT_CONFIG = ROOT / "config.json"
|
DEFAULT_CONFIG = ROOT / "config.json"
|
||||||
@@ -39,7 +43,7 @@ def _build_parser() -> argparse.ArgumentParser:
|
|||||||
|
|
||||||
parser = argparse.ArgumentParser(
|
parser = argparse.ArgumentParser(
|
||||||
prog="app.py",
|
prog="app.py",
|
||||||
description="咻咻满 Reaction 数据看板 —— P0 数据底座",
|
description="咻咻满 Reaction 数据看板 —— 数据底座、同步与每日快照",
|
||||||
)
|
)
|
||||||
sub = parser.add_subparsers(dest="command", required=True)
|
sub = parser.add_subparsers(dest="command", required=True)
|
||||||
|
|
||||||
@@ -48,7 +52,14 @@ def _build_parser() -> argparse.ArgumentParser:
|
|||||||
sync.add_argument(
|
sync.add_argument(
|
||||||
"--full", action="store_true", help="强制全量刷新元数据(忽略 6 小时周期)"
|
"--full", action="store_true", help="强制全量刷新元数据(忽略 6 小时周期)"
|
||||||
)
|
)
|
||||||
run = sub.add_parser("run", parents=[common], help="按配置周期常驻运行收藏夹同步")
|
snapshot = sub.add_parser(
|
||||||
|
"snapshot", parents=[common], help="采集当日七项快照与粉丝数"
|
||||||
|
)
|
||||||
|
snapshot.add_argument(
|
||||||
|
"--force", action="store_true", help="忽略当日完成标记,重新采集"
|
||||||
|
)
|
||||||
|
sub.add_parser("auth-check", parents=[common], help="检查 Cookie 登录态")
|
||||||
|
run = sub.add_parser("run", parents=[common], help="按配置周期常驻运行同步与快照")
|
||||||
run.add_argument(
|
run.add_argument(
|
||||||
"--once", action="store_true", help="只执行一次到期任务后退出"
|
"--once", action="store_true", help="只执行一次到期任务后退出"
|
||||||
)
|
)
|
||||||
@@ -71,6 +82,17 @@ def _print(payload) -> None:
|
|||||||
print(json.dumps(payload, ensure_ascii=False, indent=2), flush=True)
|
print(json.dumps(payload, ensure_ascii=False, indent=2), flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _client(cfg, logger: JsonLogger, *, pacer=None, retries: int = 3) -> BilibiliClient:
|
||||||
|
return BilibiliClient(
|
||||||
|
cookie=cfg.bilibili.cookie,
|
||||||
|
csrf=cfg.bilibili.csrf,
|
||||||
|
mid=cfg.bilibili.mid,
|
||||||
|
pacer=pacer,
|
||||||
|
retries=retries,
|
||||||
|
logger=logger,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def cmd_init(args, logger: JsonLogger) -> int:
|
def cmd_init(args, logger: JsonLogger) -> int:
|
||||||
store = ConfigStore(args.config, require_auth=False)
|
store = ConfigStore(args.config, require_auth=False)
|
||||||
cfg = store.get()
|
cfg = store.get()
|
||||||
@@ -98,12 +120,7 @@ def cmd_sync(args, logger: JsonLogger) -> int:
|
|||||||
db = _open_db(cfg, logger)
|
db = _open_db(cfg, logger)
|
||||||
try:
|
try:
|
||||||
repo = Repository(db)
|
repo = Repository(db)
|
||||||
client = BilibiliClient(
|
client = _client(cfg, logger)
|
||||||
cookie=cfg.bilibili.cookie,
|
|
||||||
csrf=cfg.bilibili.csrf,
|
|
||||||
mid=cfg.bilibili.mid,
|
|
||||||
logger=logger,
|
|
||||||
)
|
|
||||||
service = FolderSyncService(store, repo, client, logger=logger)
|
service = FolderSyncService(store, repo, client, logger=logger)
|
||||||
summary = service.run(full_refresh=args.full)
|
summary = service.run(full_refresh=args.full)
|
||||||
payload = summary.as_dict()
|
payload = summary.as_dict()
|
||||||
@@ -115,6 +132,46 @@ def cmd_sync(args, logger: JsonLogger) -> int:
|
|||||||
db.close()
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_snapshot(args, logger: JsonLogger) -> int:
|
||||||
|
store = ConfigStore(args.config, require_auth=False)
|
||||||
|
cfg = store.get()
|
||||||
|
db = _open_db(cfg, logger)
|
||||||
|
try:
|
||||||
|
repo = Repository(db)
|
||||||
|
client = _client(
|
||||||
|
cfg, logger, pacer=Pacer(1.0, 1.6), retries=1
|
||||||
|
)
|
||||||
|
service = SnapshotService(store, repo, client, logger=logger)
|
||||||
|
summary = service.run_if_needed(force=args.force)
|
||||||
|
payload = summary.as_dict()
|
||||||
|
payload["database"] = str(cfg.storage.database_path)
|
||||||
|
payload["snapshot_counts"] = repo.snapshot_counts(summary.stat_date)
|
||||||
|
payload["counts"] = repo.counts()
|
||||||
|
_print(payload)
|
||||||
|
return 0 if summary.completed else 1
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_auth_check(args, logger: JsonLogger) -> int:
|
||||||
|
store = ConfigStore(args.config, require_auth=False)
|
||||||
|
cfg = store.get()
|
||||||
|
if not cfg.bilibili.cookie:
|
||||||
|
_print({"ok": False, "is_login": False, "message": "未配置 cookie"})
|
||||||
|
return 1
|
||||||
|
client = _client(cfg, logger, retries=1)
|
||||||
|
state = client.check_login()
|
||||||
|
_print({
|
||||||
|
"ok": state.is_login,
|
||||||
|
"is_login": state.is_login,
|
||||||
|
"mid": state.mid,
|
||||||
|
"uname": state.uname,
|
||||||
|
"code": state.code,
|
||||||
|
"message": state.message,
|
||||||
|
})
|
||||||
|
return 0 if state.is_login else 1
|
||||||
|
|
||||||
|
|
||||||
def cmd_run(args, logger: JsonLogger) -> int:
|
def cmd_run(args, logger: JsonLogger) -> int:
|
||||||
store = ConfigStore(args.config, require_auth=True)
|
store = ConfigStore(args.config, require_auth=True)
|
||||||
cfg = store.get()
|
cfg = store.get()
|
||||||
@@ -122,26 +179,40 @@ def cmd_run(args, logger: JsonLogger) -> int:
|
|||||||
last: dict[str, object] = {}
|
last: dict[str, object] = {}
|
||||||
try:
|
try:
|
||||||
repo = Repository(db)
|
repo = Repository(db)
|
||||||
client = BilibiliClient(
|
folder_client = _client(cfg, logger)
|
||||||
cookie=cfg.bilibili.cookie,
|
folder_service = FolderSyncService(store, repo, folder_client, logger=logger)
|
||||||
csrf=cfg.bilibili.csrf,
|
snapshot_client = _client(cfg, logger, pacer=Pacer(1.0, 1.6), retries=1)
|
||||||
mid=cfg.bilibili.mid,
|
snapshot_service = SnapshotService(store, repo, snapshot_client, logger=logger)
|
||||||
logger=logger,
|
|
||||||
)
|
|
||||||
service = FolderSyncService(store, repo, client, logger=logger)
|
|
||||||
|
|
||||||
def run_sync():
|
def run_sync():
|
||||||
summary = service.run()
|
summary = folder_service.run()
|
||||||
last["summary"] = summary
|
last["folder"] = summary
|
||||||
logger.info("folder_sync.summary", **summary.as_dict())
|
logger.info("folder_sync.summary", **summary.as_dict())
|
||||||
return summary
|
return summary
|
||||||
|
|
||||||
|
def run_snapshot():
|
||||||
|
summary = snapshot_service.run_if_needed()
|
||||||
|
last["snapshot"] = summary
|
||||||
|
logger.info("snapshot.summary", **summary.as_dict())
|
||||||
|
return summary
|
||||||
|
|
||||||
|
hh, mm = parse_hhmm(cfg.schedule.snapshot_at)
|
||||||
|
snapshot_at = f"{hh:02d}:{mm:02d}"
|
||||||
scheduler = Scheduler(
|
scheduler = Scheduler(
|
||||||
[ScheduledTask(
|
[
|
||||||
"folder_sync",
|
ScheduledTask(
|
||||||
float(cfg.schedule.folder_sync_minutes * 60),
|
"folder_sync",
|
||||||
run_sync,
|
float(cfg.schedule.folder_sync_minutes * 60),
|
||||||
)],
|
run_sync,
|
||||||
|
),
|
||||||
|
ScheduledTask(
|
||||||
|
"snapshot",
|
||||||
|
86400.0,
|
||||||
|
run_snapshot,
|
||||||
|
next_run=0.0,
|
||||||
|
next_run_fn=lambda started: next_daily_run(started, snapshot_at),
|
||||||
|
),
|
||||||
|
],
|
||||||
logger=logger,
|
logger=logger,
|
||||||
)
|
)
|
||||||
if args.once:
|
if args.once:
|
||||||
@@ -150,11 +221,16 @@ def cmd_run(args, logger: JsonLogger) -> int:
|
|||||||
else:
|
else:
|
||||||
logger.info(
|
logger.info(
|
||||||
"scheduler.start",
|
"scheduler.start",
|
||||||
interval_seconds=cfg.schedule.folder_sync_minutes * 60,
|
folder_sync_seconds=cfg.schedule.folder_sync_minutes * 60,
|
||||||
|
snapshot_at=snapshot_at,
|
||||||
)
|
)
|
||||||
scheduler.run_forever()
|
scheduler.run_forever()
|
||||||
summary = last.get("summary")
|
|
||||||
return 0 if summary is None or getattr(summary, "ok", True) else 1
|
folder_summary = last.get("folder")
|
||||||
|
snapshot_summary = last.get("snapshot")
|
||||||
|
folder_ok = folder_summary is None or getattr(folder_summary, "ok", True)
|
||||||
|
snapshot_ok = snapshot_summary is None or getattr(snapshot_summary, "completed", True)
|
||||||
|
return 0 if folder_ok and snapshot_ok else 1
|
||||||
finally:
|
finally:
|
||||||
db.close()
|
db.close()
|
||||||
|
|
||||||
@@ -165,6 +241,15 @@ def cmd_status(args, logger: JsonLogger) -> int:
|
|||||||
db = _open_db(cfg, logger)
|
db = _open_db(cfg, logger)
|
||||||
try:
|
try:
|
||||||
repo = Repository(db)
|
repo = Repository(db)
|
||||||
|
today = shanghai_date()
|
||||||
|
snapshot = repo.snapshot_counts(today)
|
||||||
|
snapshot.update({
|
||||||
|
"date": today,
|
||||||
|
"last_attempt": repo.kv_get("snapshot:last_attempt"),
|
||||||
|
"last_completed": repo.kv_get("snapshot:last_completed"),
|
||||||
|
"auth_ok": repo.kv_get("auth:ok"),
|
||||||
|
"auth_message": repo.kv_get("auth:message"),
|
||||||
|
})
|
||||||
_print({
|
_print({
|
||||||
"ok": True,
|
"ok": True,
|
||||||
"schema_version": db.schema_version,
|
"schema_version": db.schema_version,
|
||||||
@@ -181,6 +266,7 @@ def cmd_status(args, logger: JsonLogger) -> int:
|
|||||||
}
|
}
|
||||||
for row in repo.list_folders()
|
for row in repo.list_folders()
|
||||||
],
|
],
|
||||||
|
"snapshot": snapshot,
|
||||||
**repo.counts(),
|
**repo.counts(),
|
||||||
})
|
})
|
||||||
finally:
|
finally:
|
||||||
@@ -196,6 +282,10 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
return cmd_init(args, logger)
|
return cmd_init(args, logger)
|
||||||
if args.command == "sync":
|
if args.command == "sync":
|
||||||
return cmd_sync(args, logger)
|
return cmd_sync(args, logger)
|
||||||
|
if args.command == "snapshot":
|
||||||
|
return cmd_snapshot(args, logger)
|
||||||
|
if args.command == "auth-check":
|
||||||
|
return cmd_auth_check(args, logger)
|
||||||
if args.command == "run":
|
if args.command == "run":
|
||||||
return cmd_run(args, logger)
|
return cmd_run(args, logger)
|
||||||
if args.command == "status":
|
if args.command == "status":
|
||||||
|
|||||||
@@ -7,6 +7,7 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
import queue
|
import queue
|
||||||
import sqlite3
|
import sqlite3
|
||||||
import threading
|
import threading
|
||||||
@@ -165,6 +166,25 @@ class Database:
|
|||||||
finally:
|
finally:
|
||||||
conn.close()
|
conn.close()
|
||||||
|
|
||||||
|
def backup_to(self, target: str | Path) -> Path:
|
||||||
|
"""用 SQLite backup API 生成一致性备份;先写临时文件再原子替换。"""
|
||||||
|
target = Path(target)
|
||||||
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
tmp = target.with_name(target.name + ".tmp")
|
||||||
|
if tmp.exists():
|
||||||
|
tmp.unlink()
|
||||||
|
|
||||||
|
def run(conn: sqlite3.Connection) -> None:
|
||||||
|
dst = sqlite3.connect(tmp)
|
||||||
|
try:
|
||||||
|
conn.backup(dst)
|
||||||
|
finally:
|
||||||
|
dst.close()
|
||||||
|
|
||||||
|
self.write(run)
|
||||||
|
os.replace(tmp, target)
|
||||||
|
return target
|
||||||
|
|
||||||
def close(self) -> None:
|
def close(self) -> None:
|
||||||
if self._closed:
|
if self._closed:
|
||||||
return
|
return
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ from __future__ import annotations
|
|||||||
import sqlite3
|
import sqlite3
|
||||||
import time
|
import time
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Any, Iterable
|
from typing import Any, Iterable, Mapping
|
||||||
|
|
||||||
from .config import FolderConfig
|
from .config import FolderConfig
|
||||||
from .database import Database
|
from .database import Database
|
||||||
@@ -368,6 +368,136 @@ class Repository:
|
|||||||
return [dict(row) for row in rows]
|
return [dict(row) for row in rows]
|
||||||
return self.db.read(run)
|
return self.db.read(run)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------ snapshot
|
||||||
|
|
||||||
|
def video_aids_for_snapshot(self) -> list[int]:
|
||||||
|
def run(conn: sqlite3.Connection) -> list[int]:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT aid FROM video WHERE status IN ('active', 'unfaved') "
|
||||||
|
"ORDER BY aid"
|
||||||
|
).fetchall()
|
||||||
|
return [int(row["aid"]) for row in rows]
|
||||||
|
return self.db.read(run)
|
||||||
|
|
||||||
|
def get_snapshot(self, stat_date: str, aid: int) -> dict[str, Any] | None:
|
||||||
|
def run(conn: sqlite3.Connection) -> dict[str, Any] | None:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT * FROM snapshot WHERE stat_date = ? AND aid = ?",
|
||||||
|
(stat_date, int(aid)),
|
||||||
|
).fetchone()
|
||||||
|
return dict(row) if row else None
|
||||||
|
return self.db.read(run)
|
||||||
|
|
||||||
|
def upsert_snapshot(
|
||||||
|
self,
|
||||||
|
stat_date: str,
|
||||||
|
aid: int,
|
||||||
|
*,
|
||||||
|
metrics: Mapping[str, int] | None,
|
||||||
|
ok: bool,
|
||||||
|
err: str | None,
|
||||||
|
captured_at: int,
|
||||||
|
src: str = "view",
|
||||||
|
) -> None:
|
||||||
|
values = metrics or {}
|
||||||
|
def run(conn: sqlite3.Connection) -> None:
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO snapshot(stat_date, aid, "view", "like", favorite, coin,
|
||||||
|
share, reply, danmaku, captured_at, ok, err, src)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||||
|
ON CONFLICT(stat_date, aid) DO UPDATE SET
|
||||||
|
"view" = excluded."view",
|
||||||
|
"like" = excluded."like",
|
||||||
|
favorite = excluded.favorite,
|
||||||
|
coin = excluded.coin,
|
||||||
|
share = excluded.share,
|
||||||
|
reply = excluded.reply,
|
||||||
|
danmaku = excluded.danmaku,
|
||||||
|
captured_at = excluded.captured_at,
|
||||||
|
ok = excluded.ok,
|
||||||
|
err = excluded.err,
|
||||||
|
src = excluded.src
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
stat_date,
|
||||||
|
int(aid),
|
||||||
|
values.get("view"),
|
||||||
|
values.get("like"),
|
||||||
|
values.get("favorite"),
|
||||||
|
values.get("coin"),
|
||||||
|
values.get("share"),
|
||||||
|
values.get("reply"),
|
||||||
|
values.get("danmaku"),
|
||||||
|
int(captured_at),
|
||||||
|
1 if ok else 0,
|
||||||
|
err,
|
||||||
|
src,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self.db.write(run)
|
||||||
|
|
||||||
|
def snapshot_counts(self, stat_date: str) -> dict[str, int]:
|
||||||
|
def run(conn: sqlite3.Connection) -> dict[str, int]:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT ok, COUNT(*) AS n FROM snapshot WHERE stat_date = ? GROUP BY ok",
|
||||||
|
(stat_date,),
|
||||||
|
).fetchall()
|
||||||
|
by_ok = {int(row["ok"]): int(row["n"]) for row in rows}
|
||||||
|
return {
|
||||||
|
"total": sum(by_ok.values()),
|
||||||
|
"ok": by_ok.get(1, 0),
|
||||||
|
"failed": by_ok.get(0, 0),
|
||||||
|
}
|
||||||
|
return self.db.read(run)
|
||||||
|
|
||||||
|
def upsert_up_stat(
|
||||||
|
self,
|
||||||
|
stat_date: str,
|
||||||
|
mid: int,
|
||||||
|
*,
|
||||||
|
follower: int | None,
|
||||||
|
following: int | None,
|
||||||
|
ok: bool,
|
||||||
|
captured_at: int,
|
||||||
|
) -> None:
|
||||||
|
self.db.write(lambda conn: conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO up_stat(stat_date, mid, follower, following, captured_at, ok)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?)
|
||||||
|
ON CONFLICT(stat_date, mid) DO UPDATE SET
|
||||||
|
follower = excluded.follower,
|
||||||
|
following = excluded.following,
|
||||||
|
captured_at = excluded.captured_at,
|
||||||
|
ok = excluded.ok
|
||||||
|
""",
|
||||||
|
(stat_date, int(mid), follower, following, int(captured_at), 1 if ok else 0),
|
||||||
|
))
|
||||||
|
|
||||||
|
def get_up_stat(self, stat_date: str, mid: int) -> dict[str, Any] | None:
|
||||||
|
def run(conn: sqlite3.Connection) -> dict[str, Any] | None:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT * FROM up_stat WHERE stat_date = ? AND mid = ?",
|
||||||
|
(stat_date, int(mid)),
|
||||||
|
).fetchone()
|
||||||
|
return dict(row) if row else None
|
||||||
|
return self.db.read(run)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------ kv
|
||||||
|
|
||||||
|
def kv_get(self, key: str) -> str | None:
|
||||||
|
def run(conn: sqlite3.Connection) -> str | None:
|
||||||
|
row = conn.execute("SELECT v FROM kv WHERE k = ?", (key,)).fetchone()
|
||||||
|
return None if row is None else str(row["v"])
|
||||||
|
return self.db.read(run)
|
||||||
|
|
||||||
|
def kv_set(self, key: str, value: str) -> None:
|
||||||
|
self.db.write(lambda conn: conn.execute(
|
||||||
|
"INSERT INTO kv(k, v) VALUES (?, ?) "
|
||||||
|
"ON CONFLICT(k) DO UPDATE SET v = excluded.v",
|
||||||
|
(key, str(value)),
|
||||||
|
))
|
||||||
|
|
||||||
# ------------------------------------------------------------ blacklist
|
# ------------------------------------------------------------ blacklist
|
||||||
|
|
||||||
def seed_blacklist(self, mids: Iterable[int], *, now: int | None = None) -> int:
|
def seed_blacklist(self, mids: Iterable[int], *, now: int | None = None) -> int:
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ class ScheduledTask:
|
|||||||
interval: float
|
interval: float
|
||||||
fn: Callable[[], Any]
|
fn: Callable[[], Any]
|
||||||
next_run: float = 0.0
|
next_run: float = 0.0
|
||||||
|
next_run_fn: Callable[[float], float] | None = None
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -77,7 +78,10 @@ class Scheduler:
|
|||||||
else:
|
else:
|
||||||
self._log("info", "scheduler.task_ok", task=task.name)
|
self._log("info", "scheduler.task_ok", task=task.name)
|
||||||
finished = self._clock()
|
finished = self._clock()
|
||||||
task.next_run = started + task.interval
|
if task.next_run_fn is not None:
|
||||||
|
task.next_run = task.next_run_fn(started)
|
||||||
|
else:
|
||||||
|
task.next_run = started + task.interval
|
||||||
result = TaskResult(
|
result = TaskResult(
|
||||||
name=task.name,
|
name=task.name,
|
||||||
started_at=started,
|
started_at=started,
|
||||||
|
|||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""备份与保留期清理。"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from ..database import Database
|
||||||
|
|
||||||
|
|
||||||
|
def daily_backup_path(backup_dir: str | Path, stat_date: str) -> Path:
|
||||||
|
compact = stat_date.replace("-", "")
|
||||||
|
return Path(backup_dir) / f"app-{compact}.db"
|
||||||
|
|
||||||
|
|
||||||
|
def backup_daily(
|
||||||
|
db: Database,
|
||||||
|
backup_dir: str | Path,
|
||||||
|
stat_date: str,
|
||||||
|
*,
|
||||||
|
logger=None,
|
||||||
|
) -> Path:
|
||||||
|
target = daily_backup_path(backup_dir, stat_date)
|
||||||
|
db.backup_to(target)
|
||||||
|
if logger is not None:
|
||||||
|
logger.info("backup.daily", path=str(target), stat_date=stat_date)
|
||||||
|
return target
|
||||||
|
|
||||||
|
|
||||||
|
def prune_daily_backups(
|
||||||
|
backup_dir: str | Path,
|
||||||
|
*,
|
||||||
|
keep_days: int,
|
||||||
|
now: float | None = None,
|
||||||
|
) -> list[Path]:
|
||||||
|
"""只清理 ``app-YYYYMMDD.db`` 日备份,不动迁移备份。"""
|
||||||
|
target_dir = Path(backup_dir)
|
||||||
|
if not target_dir.is_dir():
|
||||||
|
return []
|
||||||
|
cutoff = (time.time() if now is None else now) - keep_days * 86400
|
||||||
|
removed: list[Path] = []
|
||||||
|
for path in target_dir.glob("app-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].db"):
|
||||||
|
try:
|
||||||
|
if path.stat().st_mtime < cutoff:
|
||||||
|
path.unlink()
|
||||||
|
removed.append(path)
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
return removed
|
||||||
@@ -0,0 +1,313 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""每日视频七项快照与粉丝数采集。
|
||||||
|
|
||||||
|
规则(DEVELOPMENT.md §7.4):
|
||||||
|
|
||||||
|
* 对 ``active`` / ``unfaved`` 视频逐条请求 ``view``;
|
||||||
|
* 成功写入七项指标;``ok=1`` 的同日快照直接跳过(断点续跑);
|
||||||
|
* 视频不可访问时标记 ``removed``;
|
||||||
|
* 其他错误最多重试 3 次,仍失败写 ``ok=0`` 占位,不写假零值;
|
||||||
|
* 对 ``track_fans`` 逐个请求 ``relation/stat``;
|
||||||
|
* 完成后写入当日完成标记并生成日备份。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from ..bilibili.client import (
|
||||||
|
VIDEO_UNAVAILABLE_CODES,
|
||||||
|
BilibiliAPIError,
|
||||||
|
BilibiliClient,
|
||||||
|
BilibiliError,
|
||||||
|
BilibiliHTTPError,
|
||||||
|
)
|
||||||
|
from ..config import AppConfig, ConfigStore
|
||||||
|
from ..repository import Repository
|
||||||
|
from ..timeutil import is_date, shanghai_date
|
||||||
|
from .maintenance import backup_daily, prune_daily_backups
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class SnapshotSummary:
|
||||||
|
stat_date: str
|
||||||
|
started_at: int
|
||||||
|
total: int = 0
|
||||||
|
success: int = 0
|
||||||
|
failed: int = 0
|
||||||
|
skipped: int = 0
|
||||||
|
removed: int = 0
|
||||||
|
fans_ok: int = 0
|
||||||
|
fans_failed: int = 0
|
||||||
|
auth_ok: bool | None = None
|
||||||
|
auth_message: str = ""
|
||||||
|
backup_path: str | None = None
|
||||||
|
backup_error: str | None = None
|
||||||
|
completed: bool = False
|
||||||
|
already_completed: bool = False
|
||||||
|
finished_at: int = 0
|
||||||
|
errors: list[str] = field(default_factory=list)
|
||||||
|
|
||||||
|
def as_dict(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"stat_date": self.stat_date,
|
||||||
|
"started_at": self.started_at,
|
||||||
|
"finished_at": self.finished_at,
|
||||||
|
"elapsed": max(0, self.finished_at - self.started_at),
|
||||||
|
"total": self.total,
|
||||||
|
"success": self.success,
|
||||||
|
"failed": self.failed,
|
||||||
|
"skipped": self.skipped,
|
||||||
|
"removed": self.removed,
|
||||||
|
"fans_ok": self.fans_ok,
|
||||||
|
"fans_failed": self.fans_failed,
|
||||||
|
"auth_ok": self.auth_ok,
|
||||||
|
"auth_message": self.auth_message,
|
||||||
|
"backup_path": self.backup_path,
|
||||||
|
"backup_error": self.backup_error,
|
||||||
|
"completed": self.completed,
|
||||||
|
"already_completed": self.already_completed,
|
||||||
|
"errors": self.errors,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class SnapshotService:
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
config_store: ConfigStore,
|
||||||
|
repository: Repository,
|
||||||
|
client: BilibiliClient,
|
||||||
|
*,
|
||||||
|
clock=time.time,
|
||||||
|
sleeper=time.sleep,
|
||||||
|
logger=None,
|
||||||
|
retries: int = 3,
|
||||||
|
backoff: float = 5.0,
|
||||||
|
progress_every: int = 50,
|
||||||
|
) -> None:
|
||||||
|
if retries < 1:
|
||||||
|
raise ValueError("retries 必须 >= 1")
|
||||||
|
self.config_store = config_store
|
||||||
|
self.repo = repository
|
||||||
|
self.client = client
|
||||||
|
self._clock = clock
|
||||||
|
self._sleep = sleeper
|
||||||
|
self.logger = logger
|
||||||
|
self.retries = retries
|
||||||
|
self.backoff = backoff
|
||||||
|
self.progress_every = progress_every
|
||||||
|
|
||||||
|
def _now(self) -> int:
|
||||||
|
return int(self._clock())
|
||||||
|
|
||||||
|
def _log(self, level: str, event: str, **fields) -> None:
|
||||||
|
if self.logger is not None:
|
||||||
|
getattr(self.logger, level)(event, **fields)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def completed_key(stat_date: str) -> str:
|
||||||
|
return f"snapshot:completed:{stat_date}"
|
||||||
|
|
||||||
|
def run_if_needed(
|
||||||
|
self, *, date: str | None = None, force: bool = False
|
||||||
|
) -> SnapshotSummary:
|
||||||
|
now = self._now()
|
||||||
|
stat_date = date or shanghai_date(now)
|
||||||
|
if not is_date(stat_date):
|
||||||
|
raise ValueError(f"日期格式必须是 YYYY-MM-DD: {stat_date!r}")
|
||||||
|
if not force and self.repo.kv_get(self.completed_key(stat_date)) == "1":
|
||||||
|
self._log("info", "snapshot.skip", stat_date=stat_date, reason="already_completed")
|
||||||
|
return SnapshotSummary(
|
||||||
|
stat_date=stat_date,
|
||||||
|
started_at=now,
|
||||||
|
finished_at=now,
|
||||||
|
completed=True,
|
||||||
|
already_completed=True,
|
||||||
|
)
|
||||||
|
return self.run(date=stat_date, force=force)
|
||||||
|
|
||||||
|
def run(self, *, date: str | None = None, force: bool = False) -> SnapshotSummary:
|
||||||
|
cfg = self.config_store.get()
|
||||||
|
now = self._now()
|
||||||
|
stat_date = date or shanghai_date(now)
|
||||||
|
if not is_date(stat_date):
|
||||||
|
raise ValueError(f"日期格式必须是 YYYY-MM-DD: {stat_date!r}")
|
||||||
|
|
||||||
|
summary = SnapshotSummary(stat_date=stat_date, started_at=now)
|
||||||
|
self._log("info", "snapshot.start", stat_date=stat_date, force=force)
|
||||||
|
|
||||||
|
summary.auth_ok, summary.auth_message = self._check_auth(cfg, now)
|
||||||
|
|
||||||
|
videos = self.repo.video_aids_for_snapshot()
|
||||||
|
summary.total = len(videos)
|
||||||
|
for index, aid in enumerate(videos, 1):
|
||||||
|
existing = self.repo.get_snapshot(stat_date, aid)
|
||||||
|
if existing is not None and int(existing.get("ok") or 0) == 1 and not force:
|
||||||
|
summary.skipped += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
stat, error, unavailable = self._fetch_view(aid)
|
||||||
|
if unavailable:
|
||||||
|
summary.removed += 1
|
||||||
|
self.repo.set_video_status(aid, "removed", now=now)
|
||||||
|
self._log(
|
||||||
|
"warning", "snapshot.removed",
|
||||||
|
stat_date=stat_date, aid=aid, error=error,
|
||||||
|
)
|
||||||
|
elif stat is not None:
|
||||||
|
self.repo.upsert_snapshot(
|
||||||
|
stat_date,
|
||||||
|
aid,
|
||||||
|
metrics=stat.metrics(),
|
||||||
|
ok=True,
|
||||||
|
err=None,
|
||||||
|
captured_at=self._now(),
|
||||||
|
)
|
||||||
|
summary.success += 1
|
||||||
|
else:
|
||||||
|
summary.failed += 1
|
||||||
|
self.repo.upsert_snapshot(
|
||||||
|
stat_date,
|
||||||
|
aid,
|
||||||
|
metrics=None,
|
||||||
|
ok=False,
|
||||||
|
err=error,
|
||||||
|
captured_at=self._now(),
|
||||||
|
)
|
||||||
|
summary.errors.append(f"aid {aid}: {error}")
|
||||||
|
self._log(
|
||||||
|
"error", "snapshot.failed",
|
||||||
|
stat_date=stat_date, aid=aid, error=error,
|
||||||
|
)
|
||||||
|
|
||||||
|
if self.progress_every > 0 and (
|
||||||
|
index % self.progress_every == 0 or index == summary.total
|
||||||
|
):
|
||||||
|
self._log(
|
||||||
|
"info", "snapshot.progress",
|
||||||
|
stat_date=stat_date, done=index, total=summary.total,
|
||||||
|
success=summary.success, failed=summary.failed,
|
||||||
|
skipped=summary.skipped, removed=summary.removed,
|
||||||
|
)
|
||||||
|
|
||||||
|
for mid in cfg.track_fans:
|
||||||
|
stat, error = self._fetch_fan(mid)
|
||||||
|
if stat is not None:
|
||||||
|
self.repo.upsert_up_stat(
|
||||||
|
stat_date,
|
||||||
|
mid,
|
||||||
|
follower=stat.follower,
|
||||||
|
following=stat.following,
|
||||||
|
ok=True,
|
||||||
|
captured_at=self._now(),
|
||||||
|
)
|
||||||
|
summary.fans_ok += 1
|
||||||
|
else:
|
||||||
|
summary.fans_failed += 1
|
||||||
|
self.repo.upsert_up_stat(
|
||||||
|
stat_date,
|
||||||
|
mid,
|
||||||
|
follower=None,
|
||||||
|
following=None,
|
||||||
|
ok=False,
|
||||||
|
captured_at=self._now(),
|
||||||
|
)
|
||||||
|
summary.errors.append(f"fans mid {mid}: {error}")
|
||||||
|
self._log(
|
||||||
|
"error", "snapshot.fan_failed",
|
||||||
|
stat_date=stat_date, mid=mid, error=error,
|
||||||
|
)
|
||||||
|
|
||||||
|
summary.completed = summary.failed == 0 and summary.fans_failed == 0
|
||||||
|
self.repo.kv_set(self.completed_key(stat_date), "1" if summary.completed else "0")
|
||||||
|
self.repo.kv_set("snapshot:last_attempt", stat_date)
|
||||||
|
if summary.completed:
|
||||||
|
self.repo.kv_set("snapshot:last_completed", stat_date)
|
||||||
|
elif self.repo.kv_get("snapshot:last_completed") == stat_date:
|
||||||
|
self.repo.kv_set("snapshot:last_completed", "")
|
||||||
|
|
||||||
|
try:
|
||||||
|
path = backup_daily(
|
||||||
|
self.repo.db,
|
||||||
|
cfg.storage.backup_dir,
|
||||||
|
stat_date,
|
||||||
|
logger=self.logger,
|
||||||
|
)
|
||||||
|
prune_daily_backups(
|
||||||
|
cfg.storage.backup_dir,
|
||||||
|
keep_days=cfg.storage.backup_keep_days,
|
||||||
|
)
|
||||||
|
summary.backup_path = str(path)
|
||||||
|
except Exception as exc: # noqa: BLE001 备份失败不推翻已采集数据
|
||||||
|
summary.backup_error = str(exc)
|
||||||
|
self._log("error", "backup.failed", error=str(exc), stat_date=stat_date)
|
||||||
|
|
||||||
|
summary.finished_at = self._now()
|
||||||
|
self._log(
|
||||||
|
"info" if summary.completed else "warning",
|
||||||
|
"snapshot.done",
|
||||||
|
**summary.as_dict(),
|
||||||
|
)
|
||||||
|
return summary
|
||||||
|
|
||||||
|
# ------------------------------------------------------------ internals
|
||||||
|
|
||||||
|
def _check_auth(self, cfg: AppConfig, now: int) -> tuple[bool, str]:
|
||||||
|
if not cfg.bilibili.cookie:
|
||||||
|
self.repo.kv_set("auth:last_check", str(now))
|
||||||
|
self.repo.kv_set("auth:ok", "0")
|
||||||
|
self.repo.kv_set("auth:message", "未配置 cookie")
|
||||||
|
return False, "未配置 cookie"
|
||||||
|
try:
|
||||||
|
state = self.client.check_login()
|
||||||
|
except BilibiliError as exc:
|
||||||
|
message = str(exc)
|
||||||
|
self.repo.kv_set("auth:last_check", str(now))
|
||||||
|
self.repo.kv_set("auth:ok", "0")
|
||||||
|
self.repo.kv_set("auth:message", message)
|
||||||
|
self._log("warning", "auth.invalid", message=message)
|
||||||
|
return False, message
|
||||||
|
|
||||||
|
self.repo.kv_set("auth:last_check", str(now))
|
||||||
|
self.repo.kv_set("auth:ok", "1" if state.is_login else "0")
|
||||||
|
self.repo.kv_set("auth:message", state.message)
|
||||||
|
if state.mid is not None:
|
||||||
|
self.repo.kv_set("auth:mid", str(state.mid))
|
||||||
|
if not state.is_login:
|
||||||
|
self._log(
|
||||||
|
"warning", "auth.invalid",
|
||||||
|
code=state.code, message=state.message,
|
||||||
|
)
|
||||||
|
return state.is_login, state.message
|
||||||
|
|
||||||
|
def _fetch_view(self, aid: int):
|
||||||
|
last_error = ""
|
||||||
|
for attempt in range(1, self.retries + 1):
|
||||||
|
try:
|
||||||
|
return self.client.view(aid), None, False
|
||||||
|
except BilibiliAPIError as exc:
|
||||||
|
last_error = str(exc)
|
||||||
|
if exc.code in VIDEO_UNAVAILABLE_CODES:
|
||||||
|
return None, last_error, True
|
||||||
|
except BilibiliHTTPError as exc:
|
||||||
|
last_error = str(exc)
|
||||||
|
if exc.status == 404:
|
||||||
|
return None, last_error, True
|
||||||
|
except BilibiliError as exc:
|
||||||
|
last_error = str(exc)
|
||||||
|
if attempt < self.retries:
|
||||||
|
self._sleep(self.backoff * (2 ** (attempt - 1)))
|
||||||
|
return None, last_error or "未知错误", False
|
||||||
|
|
||||||
|
def _fetch_fan(self, mid: int):
|
||||||
|
last_error = ""
|
||||||
|
for attempt in range(1, self.retries + 1):
|
||||||
|
try:
|
||||||
|
return self.client.relation_stat(mid), None
|
||||||
|
except BilibiliError as exc:
|
||||||
|
last_error = str(exc)
|
||||||
|
if attempt < self.retries:
|
||||||
|
self._sleep(self.backoff * (2 ** (attempt - 1)))
|
||||||
|
return None, last_error or "未知错误"
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""上海时区与每日任务时间计算。
|
||||||
|
|
||||||
|
B 站数据口径使用 ``Asia/Shanghai``。中国自 1991 年起不再使用夏令时,因此这里
|
||||||
|
用固定 UTC+8,避免 Windows 上依赖未安装的 ``tzdata`` 包。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
import time
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
|
SHANGHAI_TZ = timezone(timedelta(hours=8))
|
||||||
|
_HHMM_RE = re.compile(r"^([01]\d|2[0-3]):([0-5]\d)$")
|
||||||
|
_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
|
||||||
|
|
||||||
|
|
||||||
|
def shanghai_datetime(ts: float | None = None) -> datetime:
|
||||||
|
return datetime.fromtimestamp(time.time() if ts is None else ts, SHANGHAI_TZ)
|
||||||
|
|
||||||
|
|
||||||
|
def shanghai_date(ts: float | None = None) -> str:
|
||||||
|
return shanghai_datetime(ts).strftime("%Y-%m-%d")
|
||||||
|
|
||||||
|
|
||||||
|
def parse_hhmm(value: str) -> tuple[int, int]:
|
||||||
|
match = _HHMM_RE.match(value or "")
|
||||||
|
if not match:
|
||||||
|
raise ValueError(f"时间必须是 HH:MM(24 小时制): {value!r}")
|
||||||
|
return int(match.group(1)), int(match.group(2))
|
||||||
|
|
||||||
|
|
||||||
|
def next_daily_run(now: float, hhmm: str) -> float:
|
||||||
|
"""返回 ``now`` 之后(严格大于)最近的 ``hh:mm`` 上海时间时间戳。"""
|
||||||
|
hour, minute = parse_hhmm(hhmm)
|
||||||
|
current = shanghai_datetime(now)
|
||||||
|
target = current.replace(hour=hour, minute=minute, second=0, microsecond=0)
|
||||||
|
if target.timestamp() <= now:
|
||||||
|
target += timedelta(days=1)
|
||||||
|
return target.timestamp()
|
||||||
|
|
||||||
|
|
||||||
|
def is_date(value: str) -> bool:
|
||||||
|
if not _DATE_RE.match(value or ""):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
datetime.strptime(value, "%Y-%m-%d")
|
||||||
|
except ValueError:
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def day_bounds(date: str) -> tuple[float, float]:
|
||||||
|
"""返回上海时区某一天的 [start, end) 时间戳。"""
|
||||||
|
if not is_date(date):
|
||||||
|
raise ValueError(f"日期格式必须是 YYYY-MM-DD: {date!r}")
|
||||||
|
start = datetime.strptime(date, "%Y-%m-%d").replace(tzinfo=SHANGHAI_TZ)
|
||||||
|
return start.timestamp(), (start + timedelta(days=1)).timestamp()
|
||||||
@@ -161,3 +161,108 @@ def info_for(
|
|||||||
f"bilibili://music/playlist/playpage/3728825651?page_type=3&oid={aid}&otype=2"
|
f"bilibili://music/playlist/playpage/3728825651?page_type=3&oid={aid}&otype=2"
|
||||||
)
|
)
|
||||||
return item
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
# GET x/web-interface/view?aid=117530286889638(真实七项 stat)
|
||||||
|
VIEW_PAYLOAD: dict[str, Any] = {
|
||||||
|
"code": 0,
|
||||||
|
"message": "0",
|
||||||
|
"ttl": 1,
|
||||||
|
"data": {
|
||||||
|
"aid": 117530286889638,
|
||||||
|
"bvid": "BV1xvxi6LEZD",
|
||||||
|
"cid": 43184885058,
|
||||||
|
"title": "【咻咻满充电专场①】百变音色机能怪,还有什么风格是你驾驭不了的?!",
|
||||||
|
"pubdate": 1793370433,
|
||||||
|
"ctime": 1793370433,
|
||||||
|
"duration": 9504,
|
||||||
|
"owner": {"mid": 434442072, "name": "_-抱抱-_", "face": ""},
|
||||||
|
"stat": {
|
||||||
|
"aid": 117530286889638,
|
||||||
|
"view": 819,
|
||||||
|
"danmaku": 7,
|
||||||
|
"reply": 16,
|
||||||
|
"favorite": 43,
|
||||||
|
"coin": 9,
|
||||||
|
"share": 1,
|
||||||
|
"now_rank": 0,
|
||||||
|
"his_rank": 0,
|
||||||
|
"like": 44,
|
||||||
|
"dislike": 0,
|
||||||
|
"evaluation": "",
|
||||||
|
"vt": 0,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
# 当前实测:收藏夹里已不可访问的 aid 返回 62012,而不是旧文档里的 -404
|
||||||
|
VIEW_UNAVAILABLE: dict[str, Any] = {
|
||||||
|
"code": 62012,
|
||||||
|
"message": "62012",
|
||||||
|
"ttl": 1,
|
||||||
|
"data": None,
|
||||||
|
}
|
||||||
|
|
||||||
|
# GET x/relation/stat?vmid=37754047
|
||||||
|
RELATION_PAYLOAD: dict[str, Any] = {
|
||||||
|
"code": 0,
|
||||||
|
"message": "0",
|
||||||
|
"ttl": 1,
|
||||||
|
"data": {
|
||||||
|
"mid": 37754047,
|
||||||
|
"following": 95,
|
||||||
|
"whisper": 0,
|
||||||
|
"black": 0,
|
||||||
|
"follower": 2767268,
|
||||||
|
"fans_medal_toast": None,
|
||||||
|
"fans_effect": None,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
# GET x/web-interface/nav(已登录)
|
||||||
|
NAV_PAYLOAD: dict[str, Any] = {
|
||||||
|
"code": 0,
|
||||||
|
"message": "0",
|
||||||
|
"ttl": 1,
|
||||||
|
"data": {
|
||||||
|
"isLogin": True,
|
||||||
|
"mid": 28802351,
|
||||||
|
"uname": "冷萃编译器",
|
||||||
|
"face": "https://i0.hdslb.com/bfs/face/example.jpg",
|
||||||
|
"level_info": {"current_level": 5},
|
||||||
|
"vipStatus": 0,
|
||||||
|
"wbi_img": {
|
||||||
|
"img_url": "https://i0.hdslb.com/bfs/wbi/example.png",
|
||||||
|
"sub_url": "https://i0.hdslb.com/bfs/wbi/example-sub.png",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def view_for(
|
||||||
|
aid: int,
|
||||||
|
*,
|
||||||
|
view: int = 819,
|
||||||
|
like: int = 44,
|
||||||
|
coin: int = 9,
|
||||||
|
favorite: int = 43,
|
||||||
|
share: int = 1,
|
||||||
|
reply: int = 16,
|
||||||
|
danmaku: int = 7,
|
||||||
|
bvid: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
item = copy.deepcopy(VIEW_PAYLOAD)
|
||||||
|
data = item["data"]
|
||||||
|
data["aid"] = aid
|
||||||
|
data["bvid"] = bvid or f"BV{aid}"
|
||||||
|
data["stat"].update({
|
||||||
|
"aid": aid,
|
||||||
|
"view": view,
|
||||||
|
"like": like,
|
||||||
|
"coin": coin,
|
||||||
|
"favorite": favorite,
|
||||||
|
"share": share,
|
||||||
|
"reply": reply,
|
||||||
|
"danmaku": danmaku,
|
||||||
|
})
|
||||||
|
return item
|
||||||
@@ -25,8 +25,13 @@ from fixtures import ( # noqa: E402
|
|||||||
FOLDERS_PAYLOAD,
|
FOLDERS_PAYLOAD,
|
||||||
IDS_PAYLOAD,
|
IDS_PAYLOAD,
|
||||||
INVALID_INFO,
|
INVALID_INFO,
|
||||||
|
NAV_PAYLOAD,
|
||||||
|
RELATION_PAYLOAD,
|
||||||
VALID_INFO,
|
VALID_INFO,
|
||||||
|
VIEW_PAYLOAD,
|
||||||
|
VIEW_UNAVAILABLE,
|
||||||
info_for,
|
info_for,
|
||||||
|
view_for,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -274,6 +279,88 @@ def test_timeout_and_method_forwarded():
|
|||||||
assert call["timeout"] == 30
|
assert call["timeout"] == 30
|
||||||
|
|
||||||
|
|
||||||
|
def test_view_seven_metrics():
|
||||||
|
transport = QueueTransport([http_json(VIEW_PAYLOAD)])
|
||||||
|
client, _ = make_client(transport)
|
||||||
|
stat = client.view(117530286889638)
|
||||||
|
assert stat.aid == 117530286889638
|
||||||
|
assert stat.bvid == "BV1xvxi6LEZD"
|
||||||
|
assert stat.metrics() == {
|
||||||
|
"view": 819, "like": 44, "favorite": 43, "coin": 9,
|
||||||
|
"share": 1, "reply": 16, "danmaku": 7,
|
||||||
|
}
|
||||||
|
assert "aid=117530286889638" in transport.calls[0]["url"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_view_unavailable_and_malformed():
|
||||||
|
transport = QueueTransport([
|
||||||
|
http_json(VIEW_UNAVAILABLE), http_json(VIEW_UNAVAILABLE),
|
||||||
|
])
|
||||||
|
client, _ = make_client(transport)
|
||||||
|
try:
|
||||||
|
client.view(1)
|
||||||
|
except BilibiliAPIError as exc:
|
||||||
|
assert exc.code == 62012
|
||||||
|
else:
|
||||||
|
raise AssertionError("不可访问视频应抛业务错误")
|
||||||
|
|
||||||
|
payload = view_for(1)
|
||||||
|
payload["data"]["stat"].pop("coin")
|
||||||
|
bad = http_json(payload)
|
||||||
|
transport = QueueTransport([bad, bad])
|
||||||
|
client, _ = make_client(transport)
|
||||||
|
try:
|
||||||
|
client.view(1)
|
||||||
|
except BilibiliProtocolError as exc:
|
||||||
|
assert "coin" in str(exc)
|
||||||
|
else:
|
||||||
|
raise AssertionError("缺少 stat 字段应报协议错误")
|
||||||
|
|
||||||
|
|
||||||
|
def test_relation_stat():
|
||||||
|
transport = QueueTransport([http_json(RELATION_PAYLOAD)])
|
||||||
|
client, _ = make_client(transport)
|
||||||
|
fan = client.relation_stat(37754047)
|
||||||
|
assert fan.mid == 37754047
|
||||||
|
assert fan.follower == 2767268
|
||||||
|
assert fan.following == 95
|
||||||
|
assert "vmid=37754047" in transport.calls[0]["url"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_check_login():
|
||||||
|
transport = QueueTransport([http_json(NAV_PAYLOAD)])
|
||||||
|
client, _ = make_client(transport)
|
||||||
|
state = client.check_login()
|
||||||
|
assert state.is_login is True
|
||||||
|
assert state.mid == 28802351
|
||||||
|
assert state.uname == "冷萃编译器"
|
||||||
|
assert "SESSDATA=secret" in transport.calls[0]["headers"]["cookie"]
|
||||||
|
|
||||||
|
transport = QueueTransport([http_json({
|
||||||
|
"code": 0, "message": "0", "ttl": 1,
|
||||||
|
"data": {"isLogin": False, "mid": 0, "uname": ""},
|
||||||
|
})])
|
||||||
|
client, _ = make_client(transport)
|
||||||
|
state = client.check_login()
|
||||||
|
assert state.is_login is False and state.mid == 0
|
||||||
|
|
||||||
|
transport = QueueTransport([http_json({"code": -101, "message": "账号未登录", "data": None})])
|
||||||
|
client, _ = make_client(transport)
|
||||||
|
state = client.check_login()
|
||||||
|
assert state.is_login is False
|
||||||
|
assert state.code == -101
|
||||||
|
assert "账号未登录" in state.message
|
||||||
|
|
||||||
|
|
||||||
|
def test_check_login_without_cookie_makes_no_request():
|
||||||
|
transport = QueueTransport([])
|
||||||
|
client, _ = make_client(transport, cookie="")
|
||||||
|
state = client.check_login()
|
||||||
|
assert state.is_login is False
|
||||||
|
assert state.message == "未配置 cookie"
|
||||||
|
assert transport.calls == []
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
test_fav_ids_parse_and_anonymous_header()
|
test_fav_ids_parse_and_anonymous_header()
|
||||||
test_anonymous_failure_falls_back_to_full_cookie()
|
test_anonymous_failure_falls_back_to_full_cookie()
|
||||||
@@ -288,6 +375,11 @@ def main():
|
|||||||
test_fav_infos_chunk_size_validation()
|
test_fav_infos_chunk_size_validation()
|
||||||
test_pacer_delays_after_first_request()
|
test_pacer_delays_after_first_request()
|
||||||
test_timeout_and_method_forwarded()
|
test_timeout_and_method_forwarded()
|
||||||
|
test_view_seven_metrics()
|
||||||
|
test_view_unavailable_and_malformed()
|
||||||
|
test_relation_stat()
|
||||||
|
test_check_login()
|
||||||
|
test_check_login_without_cookie_makes_no_request()
|
||||||
print("OK: B 站只读客户端契约与重试测试全部通过")
|
print("OK: B 站只读客户端契约与重试测试全部通过")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+44
-5
@@ -12,9 +12,11 @@ from pathlib import Path
|
|||||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
sys.stdout.reconfigure(encoding="utf-8")
|
sys.stdout.reconfigure(encoding="utf-8")
|
||||||
|
|
||||||
|
from reaction_board.bilibili.models import LoginState # noqa: E402
|
||||||
from reaction_board.cli import main # noqa: E402
|
from reaction_board.cli import main # noqa: E402
|
||||||
from reaction_board.database import Database # noqa: E402
|
from reaction_board.database import Database # noqa: E402
|
||||||
from reaction_board.repository import Repository # noqa: E402
|
from reaction_board.repository import Repository # noqa: E402
|
||||||
|
import reaction_board.cli as cli # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
def write_config(tmp, *, cookie="SESSDATA=x; buvid3=y", csrf="csrf"):
|
def write_config(tmp, *, cookie="SESSDATA=x; buvid3=y", csrf="csrf"):
|
||||||
@@ -86,24 +88,61 @@ def test_sync_requires_auth_without_network():
|
|||||||
assert not (Path(tmp) / "app.db").exists()
|
assert not (Path(tmp) / "app.db").exists()
|
||||||
|
|
||||||
|
|
||||||
def test_run_once_without_folders():
|
def test_snapshot_offline_empty_db():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
path = write_config(tmp, cookie="", csrf="")
|
||||||
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
data["folders"] = []
|
||||||
|
data["track_fans"] = []
|
||||||
|
path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
|
||||||
|
code, output = run_cli(["snapshot", "--config", str(path)])
|
||||||
|
assert code == 0
|
||||||
|
assert '"completed": true' in output
|
||||||
|
assert '"total": 0' in output
|
||||||
|
assert len(list((Path(tmp) / "backup").glob("app-*.db"))) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_auth_check_without_cookie():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
path = write_config(tmp, cookie="", csrf="")
|
||||||
|
code, output = run_cli(["auth-check", "--config", str(path)])
|
||||||
|
assert code == 1
|
||||||
|
payload = json.loads(output)
|
||||||
|
assert payload["is_login"] is False
|
||||||
|
assert "未配置 cookie" in payload["message"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_once_offline():
|
||||||
|
class FakeClient:
|
||||||
|
def check_login(self):
|
||||||
|
return LoginState(True, 1, "tester", 0, "OK")
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as tmp:
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
path = write_config(tmp)
|
path = write_config(tmp)
|
||||||
data = json.loads(path.read_text(encoding="utf-8"))
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
data["folders"] = []
|
data["folders"] = []
|
||||||
|
data["track_fans"] = []
|
||||||
path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
|
path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
|
||||||
code, output = run_cli(["run", "--once", "--config", str(path)])
|
|
||||||
|
original = cli._client
|
||||||
|
cli._client = lambda *args, **kwargs: FakeClient()
|
||||||
|
try:
|
||||||
|
code, output = run_cli(["run", "--once", "--config", str(path)])
|
||||||
|
finally:
|
||||||
|
cli._client = original
|
||||||
assert code == 0
|
assert code == 0
|
||||||
assert '"name": "folder_sync"' in output
|
assert '"name": "folder_sync"' in output
|
||||||
assert '"ok": true' in output
|
assert '"name": "snapshot"' in output
|
||||||
|
|
||||||
|
|
||||||
def main_test():
|
def main_test():
|
||||||
test_init_and_status()
|
test_init_and_status()
|
||||||
test_missing_config()
|
test_missing_config()
|
||||||
test_sync_requires_auth_without_network()
|
test_sync_requires_auth_without_network()
|
||||||
test_run_once_without_folders()
|
test_snapshot_offline_empty_db()
|
||||||
print("OK: CLI init/sync/run/status/错误路径测试全部通过")
|
test_auth_check_without_cookie()
|
||||||
|
test_run_once_offline()
|
||||||
|
print("OK: CLI init/sync/snapshot/auth-check/run/status/错误路径测试全部通过")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""每日备份与保留期测试: uv run python tests/test_maintenance.py"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sqlite3
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
|
sys.stdout.reconfigure(encoding="utf-8")
|
||||||
|
|
||||||
|
from reaction_board.database import Database # noqa: E402
|
||||||
|
from reaction_board.services.maintenance import ( # noqa: E402
|
||||||
|
backup_daily,
|
||||||
|
daily_backup_path,
|
||||||
|
prune_daily_backups,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_daily_backup_and_restore():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
db = Database(Path(tmp) / "app.db")
|
||||||
|
db.initialize()
|
||||||
|
try:
|
||||||
|
db.write(lambda c: c.execute("INSERT INTO kv(k, v) VALUES ('x', '42')"))
|
||||||
|
target = backup_daily(db, Path(tmp) / "backup", "2026-10-31")
|
||||||
|
assert target == daily_backup_path(Path(tmp) / "backup", "2026-10-31")
|
||||||
|
assert target.name == "app-20261031.db"
|
||||||
|
assert target.exists()
|
||||||
|
|
||||||
|
check = sqlite3.connect(target)
|
||||||
|
try:
|
||||||
|
assert check.execute("SELECT v FROM kv WHERE k='x'").fetchone()[0] == "42"
|
||||||
|
finally:
|
||||||
|
check.close()
|
||||||
|
|
||||||
|
# 覆盖同日备份也安全
|
||||||
|
db.write(lambda c: c.execute("UPDATE kv SET v='43' WHERE k='x'"))
|
||||||
|
backup_daily(db, Path(tmp) / "backup", "2026-10-31")
|
||||||
|
check = sqlite3.connect(target)
|
||||||
|
try:
|
||||||
|
assert check.execute("SELECT v FROM kv WHERE k='x'").fetchone()[0] == "43"
|
||||||
|
finally:
|
||||||
|
check.close()
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_prune_daily_backups_only():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
backup_dir = Path(tmp)
|
||||||
|
old_daily = backup_dir / "app-20260101.db"
|
||||||
|
fresh_daily = backup_dir / "app-20261031.db"
|
||||||
|
old_migration = backup_dir / "app-pre-v1-20260101.db"
|
||||||
|
other = backup_dir / "keep.txt"
|
||||||
|
for path in (old_daily, fresh_daily, old_migration, other):
|
||||||
|
path.write_bytes(b"x")
|
||||||
|
now = 1_800_000_000
|
||||||
|
old_time = now - 20 * 86400
|
||||||
|
for path in (old_daily, old_migration):
|
||||||
|
os.utime(path, (old_time, old_time))
|
||||||
|
os.utime(fresh_daily, (now, now))
|
||||||
|
removed = prune_daily_backups(backup_dir, keep_days=14, now=now)
|
||||||
|
assert removed == [old_daily]
|
||||||
|
assert fresh_daily.exists()
|
||||||
|
assert old_migration.exists() # 迁移备份不属于日备份清理范围
|
||||||
|
assert other.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
test_daily_backup_and_restore()
|
||||||
|
test_prune_daily_backups_only()
|
||||||
|
print("OK: 每日备份与保留期测试全部通过")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -110,8 +110,27 @@ def test_max_iterations_and_validation():
|
|||||||
raise AssertionError(f"应当失败: {tasks}, {kwargs}")
|
raise AssertionError(f"应当失败: {tasks}, {kwargs}")
|
||||||
|
|
||||||
|
|
||||||
|
def test_next_run_fn_controls_reschedule():
|
||||||
|
clock = [1_000.0]
|
||||||
|
calls = []
|
||||||
|
task = ScheduledTask(
|
||||||
|
"daily", 86400, lambda: calls.append(clock[0]),
|
||||||
|
next_run_fn=lambda started: started + 123,
|
||||||
|
)
|
||||||
|
scheduler = Scheduler([task], clock=lambda: clock[0])
|
||||||
|
scheduler.run_due()
|
||||||
|
assert calls == [1_000.0]
|
||||||
|
assert task.next_run == 1_123.0
|
||||||
|
clock[0] = 1_122.0
|
||||||
|
assert scheduler.run_due() == []
|
||||||
|
clock[0] = 1_123.0
|
||||||
|
scheduler.run_due()
|
||||||
|
assert calls == [1_000.0, 1_123.0]
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
test_run_when_due_and_reschedule()
|
test_run_when_due_and_reschedule()
|
||||||
|
test_next_run_fn_controls_reschedule()
|
||||||
test_task_error_does_not_block_other_tasks()
|
test_task_error_does_not_block_other_tasks()
|
||||||
test_run_forever_with_stop_event()
|
test_run_forever_with_stop_event()
|
||||||
test_max_iterations_and_validation()
|
test_max_iterations_and_validation()
|
||||||
|
|||||||
@@ -0,0 +1,331 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""每日快照与粉丝采集测试(transport 边界 Mock,不联网):
|
||||||
|
uv run python tests/test_snapshot.py
|
||||||
|
"""
|
||||||
|
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import urllib.parse
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
|
sys.stdout.reconfigure(encoding="utf-8")
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
|
from fixtures import ( # noqa: E402
|
||||||
|
NAV_PAYLOAD,
|
||||||
|
RELATION_PAYLOAD,
|
||||||
|
VIEW_UNAVAILABLE,
|
||||||
|
view_for,
|
||||||
|
)
|
||||||
|
from reaction_board.bilibili.client import ( # noqa: E402
|
||||||
|
BilibiliClient,
|
||||||
|
HTTPResponse,
|
||||||
|
Pacer,
|
||||||
|
)
|
||||||
|
from reaction_board.bilibili.models import FavResource # noqa: E402
|
||||||
|
from reaction_board.config import ConfigStore, FolderConfig # noqa: E402
|
||||||
|
from reaction_board.database import Database # noqa: E402
|
||||||
|
from reaction_board.logging_json import JsonLogger # noqa: E402
|
||||||
|
from reaction_board.repository import Repository # noqa: E402
|
||||||
|
from reaction_board.services.snapshot import SnapshotService # noqa: E402
|
||||||
|
|
||||||
|
DATE = "2026-10-31"
|
||||||
|
FOLDER = FolderConfig(year=2026, media_id=3700023351, title="2026")
|
||||||
|
|
||||||
|
|
||||||
|
def http_json(payload):
|
||||||
|
return HTTPResponse(
|
||||||
|
200, {"content-type": "application/json"},
|
||||||
|
json.dumps(payload, ensure_ascii=False).encode("utf-8"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class SnapshotTransport:
|
||||||
|
def __init__(self, views, *, nav=None, fan=None):
|
||||||
|
self.views = views
|
||||||
|
self.nav = NAV_PAYLOAD if nav is None else nav
|
||||||
|
self.fan = RELATION_PAYLOAD if fan is None else fan
|
||||||
|
self.calls = []
|
||||||
|
|
||||||
|
def request(self, method, url, headers, data=None, timeout=30):
|
||||||
|
self.calls.append(url)
|
||||||
|
if "/x/web-interface/nav" in url:
|
||||||
|
return http_json(self.nav)
|
||||||
|
if "/x/web-interface/view" in url:
|
||||||
|
aid = int(urllib.parse.parse_qs(
|
||||||
|
urllib.parse.urlparse(url).query
|
||||||
|
)["aid"][0])
|
||||||
|
payload = self.views[aid]
|
||||||
|
if isinstance(payload, HTTPResponse):
|
||||||
|
return payload
|
||||||
|
return http_json(payload)
|
||||||
|
if "/x/relation/stat" in url:
|
||||||
|
return http_json(self.fan)
|
||||||
|
raise AssertionError(f"未预期的 URL: {url}")
|
||||||
|
|
||||||
|
def count(self, part):
|
||||||
|
return sum(1 for url in self.calls if part in url)
|
||||||
|
|
||||||
|
|
||||||
|
def make_service(tmp, transport, *, track_fans=(37754047,), logger=None,
|
||||||
|
retries=3, cookie="SESSDATA=secret; buvid3=device"):
|
||||||
|
path = Path(tmp) / "config.json"
|
||||||
|
path.write_text(json.dumps({
|
||||||
|
"mid": 28802351,
|
||||||
|
"csrf": "csrf",
|
||||||
|
"cookie": cookie,
|
||||||
|
"storage": {
|
||||||
|
"database_path": str(Path(tmp) / "app.db"),
|
||||||
|
"backup_dir": str(Path(tmp) / "backup"),
|
||||||
|
"backup_keep_days": 14,
|
||||||
|
},
|
||||||
|
"folders": [],
|
||||||
|
"track_fans": list(track_fans),
|
||||||
|
}, ensure_ascii=False), encoding="utf-8")
|
||||||
|
store = ConfigStore(path, env={}, require_auth=False)
|
||||||
|
db = Database(Path(tmp) / "app.db", backup_dir=Path(tmp) / "backup")
|
||||||
|
db.initialize()
|
||||||
|
repo = Repository(db, clock=lambda: 1_800_000_000)
|
||||||
|
repo.ensure_folder(FOLDER)
|
||||||
|
client = BilibiliClient(
|
||||||
|
cookie=cookie,
|
||||||
|
csrf="csrf",
|
||||||
|
mid=28802351,
|
||||||
|
transport=transport,
|
||||||
|
pacer=Pacer(0, 0),
|
||||||
|
retries=1,
|
||||||
|
)
|
||||||
|
service = SnapshotService(
|
||||||
|
store, repo, client,
|
||||||
|
clock=lambda: 1_800_000_000,
|
||||||
|
sleeper=lambda seconds: None,
|
||||||
|
logger=logger,
|
||||||
|
retries=retries,
|
||||||
|
backoff=0,
|
||||||
|
)
|
||||||
|
return db, repo, service
|
||||||
|
|
||||||
|
|
||||||
|
def add_video(repo, aid, *, status="active"):
|
||||||
|
repo.upsert_from_fav(
|
||||||
|
FavResource(
|
||||||
|
aid=aid, bvid=f"BV{aid}", title=f"视频{aid}", up_mid=100,
|
||||||
|
up_name="UP", pubdate=1_700_000_000, duration=600,
|
||||||
|
cover=None, attr=0, fav_time=None,
|
||||||
|
),
|
||||||
|
FOLDER.media_id,
|
||||||
|
was_added=True,
|
||||||
|
)
|
||||||
|
if status != "active":
|
||||||
|
repo.set_video_status(aid, status)
|
||||||
|
|
||||||
|
|
||||||
|
def test_snapshot_success_resume_and_force():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
transport = SnapshotTransport({
|
||||||
|
1: view_for(1, view=100, like=10, coin=2, favorite=3,
|
||||||
|
share=1, reply=4, danmaku=5),
|
||||||
|
2: view_for(2, view=200, like=20, coin=4, favorite=6,
|
||||||
|
share=2, reply=8, danmaku=10),
|
||||||
|
})
|
||||||
|
db, repo, service = make_service(tmp, transport)
|
||||||
|
try:
|
||||||
|
add_video(repo, 1)
|
||||||
|
add_video(repo, 2)
|
||||||
|
result = service.run(date=DATE)
|
||||||
|
assert result.completed is True
|
||||||
|
assert result.total == 2
|
||||||
|
assert result.success == 2
|
||||||
|
assert result.failed == 0
|
||||||
|
assert result.fans_ok == 1
|
||||||
|
assert result.fans_failed == 0
|
||||||
|
assert result.auth_ok is True
|
||||||
|
assert Path(result.backup_path or "").exists()
|
||||||
|
|
||||||
|
row = repo.get_snapshot(DATE, 1)
|
||||||
|
assert (row["view"], row["like"], row["favorite"], row["coin"],
|
||||||
|
row["share"], row["reply"], row["danmaku"]) == (100, 10, 3, 2, 1, 4, 5)
|
||||||
|
assert row["ok"] == 1 and row["src"] == "view"
|
||||||
|
fan = repo.get_up_stat(DATE, 37754047)
|
||||||
|
assert fan["follower"] == 2767268 and fan["ok"] == 1
|
||||||
|
assert repo.kv_get(service.completed_key(DATE)) == "1"
|
||||||
|
assert repo.kv_get("snapshot:last_completed") == DATE
|
||||||
|
assert repo.snapshot_counts(DATE) == {"total": 2, "ok": 2, "failed": 0}
|
||||||
|
|
||||||
|
# 已完成 -> 不再请求
|
||||||
|
view_before = transport.count("/web-interface/view")
|
||||||
|
again = service.run_if_needed(date=DATE)
|
||||||
|
assert again.already_completed is True
|
||||||
|
assert transport.count("/web-interface/view") == view_before
|
||||||
|
|
||||||
|
# force 重新采集
|
||||||
|
forced = service.run(date=DATE, force=True)
|
||||||
|
assert forced.success == 2
|
||||||
|
assert transport.count("/web-interface/view") == view_before + 2
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_resume_skips_ok_and_retries_failed():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
transport = SnapshotTransport({
|
||||||
|
1: view_for(1, view=111),
|
||||||
|
2: view_for(2, view=222),
|
||||||
|
})
|
||||||
|
db, repo, service = make_service(tmp, transport)
|
||||||
|
try:
|
||||||
|
add_video(repo, 1)
|
||||||
|
add_video(repo, 2)
|
||||||
|
repo.upsert_snapshot(
|
||||||
|
DATE, 1, metrics={"view": 1, "like": 1, "favorite": 1,
|
||||||
|
"coin": 1, "share": 1, "reply": 1, "danmaku": 1},
|
||||||
|
ok=True, err=None, captured_at=1,
|
||||||
|
)
|
||||||
|
repo.upsert_snapshot(
|
||||||
|
DATE, 2, metrics=None, ok=False, err="previous error",
|
||||||
|
captured_at=1,
|
||||||
|
)
|
||||||
|
result = service.run(date=DATE)
|
||||||
|
assert result.skipped == 1
|
||||||
|
assert result.success == 1
|
||||||
|
assert result.completed is True
|
||||||
|
assert transport.count("/web-interface/view") == 1
|
||||||
|
assert repo.get_snapshot(DATE, 1)["view"] == 1 # 未被覆盖
|
||||||
|
assert repo.get_snapshot(DATE, 2)["view"] == 222
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_unavailable_marks_removed():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
transport = SnapshotTransport({
|
||||||
|
1: VIEW_UNAVAILABLE,
|
||||||
|
2: HTTPResponse(404, {}, b"not found"),
|
||||||
|
})
|
||||||
|
db, repo, service = make_service(tmp, transport, track_fans=())
|
||||||
|
try:
|
||||||
|
add_video(repo, 1)
|
||||||
|
add_video(repo, 2)
|
||||||
|
result = service.run(date=DATE)
|
||||||
|
assert result.removed == 2
|
||||||
|
assert result.failed == 0
|
||||||
|
assert result.completed is True
|
||||||
|
assert repo.get_video(1)["status"] == "removed"
|
||||||
|
assert repo.get_video(2)["status"] == "removed"
|
||||||
|
assert repo.get_snapshot(DATE, 1) is None
|
||||||
|
assert repo.get_snapshot(DATE, 2) is None
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_other_error_writes_failed_placeholder():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
transport = SnapshotTransport({
|
||||||
|
1: {"code": -500, "message": "服务异常", "data": None},
|
||||||
|
})
|
||||||
|
db, repo, service = make_service(tmp, transport, track_fans=(), retries=3)
|
||||||
|
try:
|
||||||
|
add_video(repo, 1)
|
||||||
|
result = service.run(date=DATE)
|
||||||
|
assert result.failed == 1
|
||||||
|
assert result.success == 0
|
||||||
|
assert result.completed is False
|
||||||
|
row = repo.get_snapshot(DATE, 1)
|
||||||
|
assert row["ok"] == 0
|
||||||
|
assert row["view"] is None and row["like"] is None
|
||||||
|
assert "code=-500" in row["err"]
|
||||||
|
assert repo.kv_get(service.completed_key(DATE)) == "0"
|
||||||
|
# 每次业务失败会走匿名+完整 Cookie 两条请求,服务层重试 3 次
|
||||||
|
assert transport.count("/web-interface/view") == 6
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_fan_failure_placeholder():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
transport = SnapshotTransport({}, fan={"code": -500, "message": "x", "data": None})
|
||||||
|
db, repo, service = make_service(tmp, transport, retries=2)
|
||||||
|
try:
|
||||||
|
result = service.run(date=DATE)
|
||||||
|
assert result.total == 0
|
||||||
|
assert result.fans_failed == 1
|
||||||
|
assert result.completed is False
|
||||||
|
row = repo.get_up_stat(DATE, 37754047)
|
||||||
|
assert row["ok"] == 0
|
||||||
|
assert row["follower"] is None and row["following"] is None
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_auth_invalid_is_warned_but_snapshot_continues():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
stream = io.StringIO()
|
||||||
|
logger = JsonLogger(stream)
|
||||||
|
transport = SnapshotTransport(
|
||||||
|
{}, nav={"code": 0, "message": "0", "ttl": 1,
|
||||||
|
"data": {"isLogin": False, "mid": 0, "uname": ""}},
|
||||||
|
fan=RELATION_PAYLOAD,
|
||||||
|
)
|
||||||
|
db, repo, service = make_service(tmp, transport, logger=logger)
|
||||||
|
try:
|
||||||
|
result = service.run(date=DATE)
|
||||||
|
assert result.auth_ok is False
|
||||||
|
assert result.completed is True
|
||||||
|
assert repo.kv_get("auth:ok") == "0"
|
||||||
|
events = [json.loads(line) for line in stream.getvalue().splitlines()]
|
||||||
|
assert any(e["event"] == "auth.invalid" for e in events)
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_cookie_skips_nav_request():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
transport = SnapshotTransport({}, fan=RELATION_PAYLOAD)
|
||||||
|
db, repo, service = make_service(tmp, transport, cookie="")
|
||||||
|
try:
|
||||||
|
result = service.run(date=DATE)
|
||||||
|
assert result.auth_ok is False
|
||||||
|
assert result.auth_message == "未配置 cookie"
|
||||||
|
assert transport.count("/web-interface/nav") == 0
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_force_failure_clears_last_completed():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
transport = SnapshotTransport({1: view_for(1, view=100)})
|
||||||
|
db, repo, service = make_service(tmp, transport, track_fans=(), retries=1)
|
||||||
|
try:
|
||||||
|
add_video(repo, 1)
|
||||||
|
first = service.run(date=DATE)
|
||||||
|
assert first.completed is True
|
||||||
|
assert repo.kv_get("snapshot:last_completed") == DATE
|
||||||
|
|
||||||
|
transport.views[1] = {"code": -500, "message": "x", "data": None}
|
||||||
|
second = service.run(date=DATE, force=True)
|
||||||
|
assert second.failed == 1
|
||||||
|
assert second.completed is False
|
||||||
|
assert repo.kv_get("snapshot:last_completed") == ""
|
||||||
|
assert repo.get_snapshot(DATE, 1)["ok"] == 0
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
test_snapshot_success_resume_and_force()
|
||||||
|
test_resume_skips_ok_and_retries_failed()
|
||||||
|
test_unavailable_marks_removed()
|
||||||
|
test_other_error_writes_failed_placeholder()
|
||||||
|
test_fan_failure_placeholder()
|
||||||
|
test_auth_invalid_is_warned_but_snapshot_continues()
|
||||||
|
test_no_cookie_skips_nav_request()
|
||||||
|
test_force_failure_clears_last_completed()
|
||||||
|
print("OK: 每日快照、断点续跑、失败占位与粉丝采集测试全部通过")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""上海时区与每日时间计算测试: uv run python tests/test_timeutil.py"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
|
sys.stdout.reconfigure(encoding="utf-8")
|
||||||
|
|
||||||
|
from reaction_board.timeutil import ( # noqa: E402
|
||||||
|
SHANGHAI_TZ,
|
||||||
|
day_bounds,
|
||||||
|
is_date,
|
||||||
|
next_daily_run,
|
||||||
|
parse_hhmm,
|
||||||
|
shanghai_date,
|
||||||
|
shanghai_datetime,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def ts(text):
|
||||||
|
return datetime.strptime(text, "%Y-%m-%d %H:%M:%S").replace(tzinfo=SHANGHAI_TZ).timestamp()
|
||||||
|
|
||||||
|
|
||||||
|
def test_shanghai_date_boundary():
|
||||||
|
# UTC 2026-10-31 15:59:59 = 上海 2026-10-31 23:59:59
|
||||||
|
assert shanghai_date(ts("2026-10-31 23:59:59")) == "2026-10-31"
|
||||||
|
# UTC 2026-10-31 16:00:00 = 上海 2026-11-01 00:00:00
|
||||||
|
assert shanghai_date(ts("2026-11-01 00:00:00")) == "2026-11-01"
|
||||||
|
assert shanghai_datetime().tzinfo is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_hhmm():
|
||||||
|
assert parse_hhmm("00:01") == (0, 1)
|
||||||
|
assert parse_hhmm("23:59") == (23, 59)
|
||||||
|
for bad in ("24:00", "1:01", "00:60", "", "abc"):
|
||||||
|
try:
|
||||||
|
parse_hhmm(bad)
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
raise AssertionError(f"{bad!r} 应失败")
|
||||||
|
|
||||||
|
|
||||||
|
def test_next_daily_run():
|
||||||
|
# 当天 00:01 之前 -> 当天 00:01
|
||||||
|
now = ts("2026-11-01 00:00:10")
|
||||||
|
nxt = next_daily_run(now, "00:01")
|
||||||
|
assert nxt == ts("2026-11-01 00:01:00")
|
||||||
|
# 正好 00:01:00 -> 下一天(严格大于)
|
||||||
|
assert next_daily_run(ts("2026-11-01 00:01:00"), "00:01") == ts("2026-11-02 00:01:00")
|
||||||
|
# 当天 00:01 之后 -> 次日
|
||||||
|
assert next_daily_run(ts("2026-11-01 23:59:59"), "00:01") == ts("2026-11-02 00:01:00")
|
||||||
|
# 跨月
|
||||||
|
assert next_daily_run(ts("2026-10-31 12:00:00"), "00:01") == ts("2026-11-01 00:01:00")
|
||||||
|
|
||||||
|
|
||||||
|
def test_day_bounds_and_date_validation():
|
||||||
|
start, end = day_bounds("2026-11-01")
|
||||||
|
assert start == ts("2026-11-01 00:00:00")
|
||||||
|
assert end == ts("2026-11-02 00:00:00")
|
||||||
|
assert end - start == 86400
|
||||||
|
assert is_date("2026-11-01")
|
||||||
|
for bad in ("2026-13-01", "2026-02-30", "2026/11/01", "20261101", ""):
|
||||||
|
assert not is_date(bad), bad
|
||||||
|
try:
|
||||||
|
day_bounds("bad")
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
raise AssertionError("非法日期应失败")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
test_shanghai_date_boundary()
|
||||||
|
test_parse_hhmm()
|
||||||
|
test_next_daily_run()
|
||||||
|
test_day_bounds_and_date_validation()
|
||||||
|
print("OK: 上海时区与每日时间计算测试全部通过")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Reference in new issue
Block a user