feat(p1): add daily snapshots and fan tracking

Implement P1: seven-metric view snapshots, relation/stat fan snapshots, failure placeholders, resume by daily completion markers, Cookie login warnings, daily SQLite backups, Shanghai daily scheduling, and CLI snapshot/auth-check commands. Real read-only run captured 795/795 videos successfully over 1124s with a verified daily backup.
This commit is contained in:
kairee committed 2026-10-31 19:19:24 +08:00
1 parent ee289bfa56
commit 5c2ab7b187
20 files changed
+1680 -54

No files matched your search

+29 -3
View File
@@ -1,6 +1,6 @@
# 咻咻满 Reaction 视频数据看板开发文档 # 咻咻满 Reaction 视频数据看板开发文档
> **状态:** P0(数据底座与收藏夹同步)已实施并通过只读联调;P1 待开始 > **状态:** P0(数据底座与收藏夹同步)、P1(每日快照与粉丝追踪)已实施并通过只读联调;P2 待开始
> >
> **技术基线:** Python 3.14、Python 标准库、SQLite、原生 HTML/CSS/JavaScript、`uv` > **技术基线:** Python 3.14、Python 标准库、SQLite、原生 HTML/CSS/JavaScript、`uv`
> >
@@ -95,6 +95,13 @@ uv run python app.py sync
uv run python app.py run uv run python app.py run
uv run python app.py run --once uv run python app.py run --once
# 每日七项快照与粉丝数(P1)
uv run python app.py snapshot
uv run python app.py snapshot --force
# 检查 Cookie 登录态(P1)
uv run python app.py auth-check
# 查看数据库状态(离线,P0) # 查看数据库状态(离线,P0)
uv run python app.py status uv run python app.py status
@@ -388,7 +395,7 @@ score < 0:丢弃
- 对 `status IN ('active', 'unfaved')` 的视频逐条请求 `view?aid=`; - 对 `status IN ('active', 'unfaved')` 的视频逐条请求 `view?aid=`;
- 成功写入播放、点赞、收藏、投币、转发、评论、弹幕七项数据; - 成功写入播放、点赞、收藏、投币、转发、评论、弹幕七项数据;
- `-404` 时标记 `removed`; - `-404` / `62002` / `62012` 时标记 `removed`(62002/62012 为当前实测返回);
- 其他错误最多重试 3 次,仍失败则写入 `ok=0` 与错误信息,不写伪造的零值; - 其他错误最多重试 3 次,仍失败则写入 `ok=0` 与错误信息,不写伪造的零值;
- 每条视频请求之间随机等待 1.0~1.6 秒; - 每条视频请求之间随机等待 1.0~1.6 秒;
- 对 `track_fans` 中的每个 mid 请求一次 `relation/stat` 并写入 `up_stat`; - 对 `track_fans` 中的每个 mid 请求一次 `relation/stat` 并写入 `up_stat`;
@@ -396,6 +403,15 @@ score < 0:丢弃
服务启动后若当天快照尚未成功完成,应补跑当日任务;已成功的 `(stat_date, aid)` 不重复请求。 服务启动后若当天快照尚未成功完成,应补跑当日任务;已成功的 `(stat_date, aid)` 不重复请求。
P1 实现说明:
- 时区使用固定 UTC+8(中国无夏令时),避免 Windows 缺少 `tzdata`;
- 当日全部视频与粉丝采集成功后在 `kv` 写入 `snapshot:completed:<date>=1`;启动时未完成会补跑;
- `ok=0` 的当日记录会在下次补跑时重试;`ok=1` 直接跳过;
- 每次采集后生成 `data/backup/app-YYYYMMDD.db` 并清理超过 `storage.backup_keep_days` 的日备份;
- Cookie 登录态通过 `x/web-interface/nav` 检查,失效只告警,不阻塞匿名 `view`;
- Telegram 摘要推迟到 P3,P1 先写结构化 JSON 日志和 `kv` 状态。
--- ---
## 8. Telegram 与收藏操作 ## 8. Telegram 与收藏操作
@@ -555,12 +571,22 @@ uv run python tests/test_moves.py
- 黑名单 9 个 mid 已写入 `up_blacklist`,且不参与收藏夹同步过滤; - 黑名单 9 个 mid 已写入 `up_blacklist`,且不参与收藏夹同步过滤;
- 自动化测试见 `tests/test_config.py`、`test_migrations.py`、`test_database.py`、`test_repository.py`、`test_bilibili_client.py`、`test_folder_sync.py`、`test_scheduler.py`、`test_logging.py`、`test_cli.py`。 - 自动化测试见 `tests/test_config.py`、`test_migrations.py`、`test_database.py`、`test_repository.py`、`test_bilibili_client.py`、`test_folder_sync.py`、`test_scheduler.py`、`test_logging.py`、`test_cli.py`。
### P1:快照与粉丝追踪 ### P1:快照与粉丝追踪 ✅
- 每日七项视频快照; - 每日七项视频快照;
- 粉丝快照; - 粉丝快照;
- 失败占位、断点续跑、Cookie 失效告警、SQLite 备份。 - 失败占位、断点续跑、Cookie 失效告警、SQLite 备份。
完成情况(2026-10-31):
- 已实现 `services/snapshot.py`:`view` 七项指标、`relation/stat` 粉丝数、`ok=0` 失败占位、`ok=1` 断点跳过;
- 实测不可访问视频返回 `62012`,已和旧口径 `-404`、`62002` 一起按 `removed` 处理;
- `run` 调度器同时注册收藏夹同步与每日 `snapshot_at` 任务,启动时立即检查当日是否完成以支持补跑;
- 每次快照后生成日备份并清理超过 14 天的日备份;
- `auth-check` 与快照启动时检查 Cookie 登录态,失效写 `kv` 并告警;
- 真实全量联调:795 条视频全部 `ok=1`,0 失败、0 移除,耗时 1124 秒;粉丝 `follower=2767270`;日备份 `data/backup/app-20261031.db` 含 795 条快照;二次执行正确命中完成标记并跳过;
- 自动化测试见 `tests/test_snapshot.py`、`test_maintenance.py`、`test_timeutil.py`,并扩展了 `test_bilibili_client.py` 与 `test_scheduler.py`。
### P2:本地看板 ### P2:本地看板
- 总览、榜单、详情、UP、粉丝、候选和系统状态页面; - 总览、榜单、详情、UP、粉丝、候选和系统状态页面;
+38 -8
View File
@@ -4,9 +4,9 @@
项目以 B 站年度收藏夹为可信视频来源,补充关键词发现和 Telegram 人工确认,持续采集 reaction 视频的七项数据与咻咻满粉丝数,并提供本地数据看板。现有的收藏夹排序工具仍保留并作为一键加入后的排序能力基础。 项目以 B 站年度收藏夹为可信视频来源,补充关键词发现和 Telegram 人工确认,持续采集 reaction 视频的七项数据与咻咻满粉丝数,并提供本地数据看板。现有的收藏夹排序工具仍保留并作为一键加入后的排序能力基础。
> 当前状态:**P0(数据底座与收藏夹同步)已实施,P1 待开始。** > 当前状态:**P0(数据底座与收藏夹同步)、P1(快照与粉丝追踪)已实施;P2 待开始。**
> >
> 当前已可用:扫码授权、收藏夹读取、排序计划生成、执行与校验;SQLite 建库迁移、收藏夹首次导入与增量同步、失效识别、黑名单种子。 > 当前已可用:扫码授权、收藏夹读取、排序计划生成、执行与校验;SQLite 建库迁移、收藏夹首次导入与增量同步、失效识别、黑名单种子;每日七项视频快照、粉丝数快照、失败占位、断点续跑与日备份。
> >
> 详细业务口径、接口验证和风险记录见 [project.md](project.md);开发架构与实施计划见 [DEVELOPMENT.md](DEVELOPMENT.md)。 > 详细业务口径、接口验证和风险记录见 [project.md](project.md);开发架构与实施计划见 [DEVELOPMENT.md](DEVELOPMENT.md)。
@@ -17,8 +17,8 @@
| 收藏夹排序 | 可用 | 同一 UP 连续,组内按发布时间倒序,组间按该 UP 最早视频发布时间倒序。 | | 收藏夹排序 | 可用 | 同一 UP 连续,组内按发布时间倒序,组间按该 UP 最早视频发布时间倒序。 |
| 扫码授权与 Cookie 续期 | 可用 | 本地生成二维码;更新身份 Cookie 时保留设备指纹。 | | 扫码授权与 Cookie 续期 | 可用 | 本地生成二维码;更新身份 Cookie 时保留设备指纹。 |
| 收藏夹同步与首次导入 | 可用(P0) | 监控已配置年度夹,自动发现人工新收藏的视频与失效资源。 | | 收藏夹同步与首次导入 | 可用(P0) | 监控已配置年度夹,自动发现人工新收藏的视频与失效资源。 |
| 视频日快照 | 规划(P1) | 逐条采集播放、点赞、投币、收藏、转发、评论、弹幕。 | | 视频日快照 | 可用(P1) | 逐条采集播放、点赞、投币、收藏、转发、评论、弹幕。 |
| 粉丝追踪 | 规划(P1) | 每日记录咻咻满等配置 UP 的粉丝数。 | | 粉丝追踪 | 可用(P1) | 每日记录咻咻满等配置 UP 的粉丝数。 |
| 本地数据看板 | 规划(P2) | 总览、榜单、视频/UP 详情、粉丝曲线、候选池和系统状态。 | | 本地数据看板 | 规划(P2) | 总览、榜单、视频/UP 详情、粉丝曲线、候选池和系统状态。 |
| Telegram 发现与一键加入 | 规划(P3) | 每小时发现候选,支持收藏、忽略、拉黑和收藏夹内定位。 | | Telegram 发现与一键加入 | 规划(P3) | 每小时发现候选,支持收藏、忽略、拉黑和收藏夹内定位。 |
@@ -68,7 +68,7 @@ uv run python app.py init
# 首次导入或增量同步配置中的年度收藏夹(只读联网) # 首次导入或增量同步配置中的年度收藏夹(只读联网)
uv run python app.py sync uv run python app.py sync
# 按 folder_sync_minutes 常驻周期同步;--once 只跑一次 # 按配置周期常驻运行收藏夹同步 + 每日快照;--once 只跑一次到期任务
uv run python app.py run uv run python app.py run
uv run python app.py run --once uv run python app.py run --once
@@ -86,6 +86,32 @@ uv run python app.py status
- 黑名单只作为发现阶段的种子写入,**不会过滤收藏夹同步**; - 黑名单只作为发现阶段的种子写入,**不会过滤收藏夹同步**;
- 收藏夹达到 950 条时预警,系统绝不自动换夹。 - 收藏夹达到 950 条时预警,系统绝不自动换夹。
## P1:每日快照与粉丝追踪
P1 已实现每日七项视频指标快照、咻咻满粉丝数快照、失败占位、断点续跑、Cookie 失效告警和 SQLite 日备份。
```powershell
# 采集当日快照与粉丝数(只读联网,约 18~20 分钟 / 795 条)
uv run python app.py snapshot
# 忽略当日完成标记,重新采集
uv run python app.py snapshot --force
# 检查 Cookie 登录态
uv run python app.py auth-check
```
- `snapshot` 对 `active` / `unfaved` 视频逐条请求 `view`,每条请求间隔 1.0~1.6 秒;
- 成功写入 `snapshot` 的七项指标;同日 `ok=1` 的记录直接跳过,可断点续跑;
- 视频不可访问(`-404` / `62002` / `62012`)标记为 `removed`;
- 其他错误重试 3 次,仍失败写入 `ok=0` 和错误信息,不写伪造的零值;
- 粉丝数写入 `up_stat`,失败同样写 `ok=0`;
- 当日全部视频与粉丝采集成功后写入完成标记;服务启动时若当天未完成会自动补跑;
- 每次采集后生成 `data/backup/app-YYYYMMDD.db`,按 `storage.backup_keep_days` 清理旧日备份;
- Cookie 失效只告警,不阻塞匿名 `view` 采集;`auth-check` 可单独检查。
首次真实全量采集结果:795 条视频全部成功、0 失败、0 移除,耗时约 19 分钟;粉丝 `follower=2767270`;日备份为 `data/backup/app-20261031.db`。
## 配置 ## 配置
复制样例配置,并填写 B 站账号信息: 复制样例配置,并填写 B 站账号信息:
@@ -198,8 +224,11 @@ uv run python tests/test_config.py # 配置加载、校验、ENV 覆盖
uv run python tests/test_migrations.py # 迁移、备份、回滚与保留期 uv run python tests/test_migrations.py # 迁移、备份、回滚与保留期
uv run python tests/test_database.py # 单写队列、事务与并发 uv run python tests/test_database.py # 单写队列、事务与并发
uv run python tests/test_repository.py # 收藏夹/视频状态机与黑名单 uv run python tests/test_repository.py # 收藏夹/视频状态机与黑名单
uv run python tests/test_bilibili_client.py # ids/infos 契约、分块、缺失与重试 uv run python tests/test_bilibili_client.py # ids/infos/view/fans/nav 契约与重试
uv run python tests/test_folder_sync.py # 首次导入、差集、失效识别、容量预警 uv run python tests/test_folder_sync.py # 首次导入、差集、失效识别、容量预警
uv run python tests/test_snapshot.py # 七项快照、断点续跑、失败占位、粉丝
uv run python tests/test_maintenance.py # 日备份与保留期
uv run python tests/test_timeutil.py # 上海时区与每日任务时间
uv run python tests/test_scheduler.py # 内置调度器到期/失败隔离 uv run python tests/test_scheduler.py # 内置调度器到期/失败隔离
uv run python tests/test_logging.py # JSON 日志脱敏 uv run python tests/test_logging.py # JSON 日志脱敏
uv run python tests/test_cli.py # CLI 离线路径 uv run python tests/test_cli.py # CLI 离线路径
@@ -211,8 +240,9 @@ uv run python tests/test_cli.py # CLI 离线路径
config.json.bak 最近一次 auth 更新前的配置备份 config.json.bak 最近一次 auth 更新前的配置备份
data/plan-<media_id>.json 排序计划、顺序快照和元数据缓存 data/plan-<media_id>.json 排序计划、顺序快照和元数据缓存
data/journal-<media_id>.jsonl 排序执行日志 data/journal-<media_id>.jsonl 排序执行日志
data/app.db P0 起的 SQLite 数据库(收藏夹、视频、黑名单) data/app.db SQLite 数据库(收藏夹、视频、快照、粉丝、黑名单)
data/backup/ 数据库迁移前的自动备份 data/backup/app-YYYYMMDD.db 每日快照后的 SQLite 备份
data/backup/app-pre-v*.db 数据库迁移前的自动备份
``` ```
## 文档索引 ## 文档索引
+2 -2
View File
@@ -1,8 +1,8 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""看板服务入口。 """看板服务入口。
P0 提供 ``init`` / ``sync`` / ``run`` / ``status`` 子命令;后续阶段会在同一入口启动 P1 提供 ``init`` / ``sync`` / ``snapshot`` / ``auth-check`` / ``run`` / ``status``
本地 Web 服务。 子命令;后续阶段会在同一入口启动本地 Web 服务。
""" """
import sys import sys
+7 -2
View File
@@ -1,7 +1,7 @@
# 咻咻满 Reaction 视频数据看板 —— 可行性分析与技术路线 # 咻咻满 Reaction 视频数据看板 —— 可行性分析与技术路线
> 状态:**设计定稿;P0(数据底座与收藏夹同步)已实施并完成只读联调** > 状态:**设计定稿;P0(数据底座与收藏夹同步)、P1(每日快照与粉丝追踪)已实施**
> 最后更新:2026-10-31(第 5 轮:P0 落地) > 最后更新:2026-10-31(第 6 轮:P1 落地)
> 全部探测均为只读(GET),未调用 `fav/resource/deal`、`fav/resource/sort` > 全部探测均为只读(GET),未调用 `fav/resource/deal`、`fav/resource/sort`
--- ---
@@ -770,6 +770,11 @@ storage:SQLite WAL,单写线程
| 21 | **搜索 item 的 duration 格式** | `分钟:秒`(`105:5`、`336:31`),超 60 分钟不转小时 | | 21 | **搜索 item 的 duration 格式** | `分钟:秒`(`105:5`、`336:31`),超 60 分钟不转小时 |
| 22 | **搜索 item 自带统计字段** | `play/like/favorites/danmaku/review/duration/cover/pubdate/mid/uname` 齐全,通知可用索引值 | | 22 | **搜索 item 自带统计字段** | `play/like/favorites/danmaku/review/duration/cover/pubdate/mid/uname` 齐全,通知可用索引值 |
| 23 | **初始黑名单 9 个 mid 核对** | `x/web-interface/card` 全部 `code=0`,昵称/粉丝数已记录(§1.6) | | 23 | **初始黑名单 9 个 mid 核对** | `x/web-interface/card` 全部 `code=0`,昵称/粉丝数已记录(§1.6) |
| 24 | `x/web-interface/view` 七项 stat | `code=0`,view/like/favorite/coin/share/reply/danmaku 齐全 |
| 25 | `view` 不可访问返回码 | 实测为 `62012`;代码同时兼容 `-404`、`62002`、HTTP 404 |
| 26 | `x/relation/stat` 粉丝数 | `code=0`,匿名可用,实测 `follower=2767268` |
| 27 | `x/web-interface/nav` 登录态 | `code=0`、`isLogin=true`、mid=28802351 |
| 28 | **P1 真实全量快照** | 795 条全部 `ok=1`、0 失败、0 移除;耗时 1124s;粉丝 `follower=2767270`;日备份 `app-20261031.db` 含 795 条快照 |
**未验证(需授权或外部配合)**: **未验证(需授权或外部配合)**:
+72 -1
View File
@@ -18,11 +18,21 @@ from dataclasses import dataclass
from typing import Any, Mapping, Protocol, Sequence from typing import Any, Mapping, Protocol, Sequence
from .auth import make_headers from .auth import make_headers
from .models import FavId, FavResource, FolderInfo, InfosResult from .models import (
FanStat,
FavId,
FavResource,
FolderInfo,
InfosResult,
LoginState,
ViewStat,
)
BASE = "https://api.bilibili.com" BASE = "https://api.bilibili.com"
DEFAULT_CHUNK = 200 DEFAULT_CHUNK = 200
MAX_CHUNK = 400 MAX_CHUNK = 400
# 视频不可访问/已删除:-404 为旧口径,62002/62012 为当前实测返回。
VIDEO_UNAVAILABLE_CODES = frozenset({-404, 62002, 62012})
class BilibiliError(RuntimeError): class BilibiliError(RuntimeError):
@@ -360,3 +370,64 @@ class BilibiliClient:
media_count=count, media_count=count,
)) ))
return out return out
# ------------------------------------------------------------ view / fan
def view(self, aid: int) -> ViewStat:
url = f"{BASE}/x/web-interface/view?aid={int(aid)}"
payload = self.get_json(url)
data = payload.get("data")
if not isinstance(data, Mapping):
raise BilibiliProtocolError("view data 不是对象")
try:
return ViewStat.from_api(data)
except (TypeError, ValueError) as exc:
raise BilibiliProtocolError(f"view 解析失败: {exc}") from exc
def relation_stat(self, mid: int) -> FanStat:
url = f"{BASE}/x/relation/stat?vmid={int(mid)}"
payload = self.get_json(url)
data = payload.get("data")
if not isinstance(data, Mapping):
raise BilibiliProtocolError("relation/stat data 不是对象")
try:
return FanStat.from_api(data)
except (TypeError, ValueError) as exc:
raise BilibiliProtocolError(f"relation/stat 解析失败: {exc}") from exc
def check_login(self) -> LoginState:
"""检查 Cookie 登录态;未配置 Cookie 时不发请求。"""
if not self.cookie:
return LoginState(
is_login=False,
mid=None,
uname=None,
code=0,
message="未配置 cookie",
)
url = f"{BASE}/x/web-interface/nav"
try:
payload = self.get_json(url, auth=True)
except BilibiliAPIError as exc:
return LoginState(
is_login=False, mid=None, uname=None,
code=exc.code, message=exc.message,
)
except BilibiliError as exc:
return LoginState(
is_login=False, mid=None, uname=None,
code=-1, message=str(exc),
)
data = payload.get("data") or {}
if not isinstance(data, Mapping):
raise BilibiliProtocolError("nav data 不是对象")
raw_mid = data.get("mid")
mid = raw_mid if isinstance(raw_mid, int) and not isinstance(raw_mid, bool) else None
uname = data.get("uname")
return LoginState(
is_login=bool(data.get("isLogin")),
mid=mid,
uname=str(uname) if isinstance(uname, str) else None,
code=0,
message=str(payload.get("message") or ""),
)
+80
View File
@@ -85,3 +85,83 @@ class InfosResult:
requested: tuple[int, ...] requested: tuple[int, ...]
items: dict[int, FavResource] items: dict[int, FavResource]
missing: tuple[int, ...] missing: tuple[int, ...]
@dataclass(frozen=True)
class ViewStat:
"""``x/web-interface/view`` 的七项统计。"""
aid: int
bvid: str | None
title: str
view: int
like: int
favorite: int
coin: int
share: int
reply: int
danmaku: int
def metrics(self) -> dict[str, int]:
return {
"view": self.view,
"like": self.like,
"favorite": self.favorite,
"coin": self.coin,
"share": self.share,
"reply": self.reply,
"danmaku": self.danmaku,
}
@classmethod
def from_api(cls, item: Mapping[str, Any]) -> "ViewStat":
aid = item.get("aid")
if isinstance(aid, bool) or not isinstance(aid, int):
raise ValueError("view data.aid 不是整数")
stat = item.get("stat")
if not isinstance(stat, Mapping):
raise ValueError("view data.stat 不是对象")
metrics: dict[str, int] = {}
for key in ("view", "like", "favorite", "coin", "share", "reply", "danmaku"):
value = stat.get(key)
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
raise ValueError(f"view stat.{key} 不是非负整数")
metrics[key] = value
bvid = item.get("bvid")
if not isinstance(bvid, str) or not bvid:
bvid = None
return cls(
aid=aid,
bvid=bvid,
title=str(item.get("title") or ""),
**metrics,
)
@dataclass(frozen=True)
class FanStat:
mid: int
follower: int
following: int
@classmethod
def from_api(cls, item: Mapping[str, Any]) -> "FanStat":
mid = item.get("mid")
follower = item.get("follower")
following = item.get("following", 0)
if isinstance(mid, bool) or not isinstance(mid, int):
raise ValueError("relation/stat data.mid 不是整数")
if isinstance(follower, bool) or not isinstance(follower, int) or follower < 0:
raise ValueError("relation/stat data.follower 不是非负整数")
if isinstance(following, bool) or not isinstance(following, int) or following < 0:
raise ValueError("relation/stat data.following 不是非负整数")
return cls(mid=mid, follower=follower, following=following)
@dataclass(frozen=True)
class LoginState:
is_login: bool
mid: int | None
uname: str | None
code: int
message: str
+121 -31
View File
@@ -1,10 +1,12 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""P0 命令行入口。 """看板命令行入口。
python app.py init # 建库 / 迁移 / 写入黑名单种子(离线) python app.py init # 建库 / 迁移 / 写入黑名单种子(离线)
python app.py sync # 收藏夹只读同步(首次导入或增量) python app.py sync # 收藏夹只读同步(首次导入或增量)
python app.py run # 按 folder_sync_minutes 常驻周期同步(--once 只跑一次) python app.py snapshot # 采集当日七项快照与粉丝数
python app.py status # 查看库内状态(离线) python app.py auth-check # 检查 Cookie 登录态
python app.py run # 按配置周期常驻运行(--once 只跑一次)
python app.py status # 查看库内状态(离线)
P2 起 ``python app.py`` 将在此基础上启动本地看板服务。 P2 起 ``python app.py`` 将在此基础上启动本地看板服务。
""" """
@@ -17,13 +19,15 @@ import sys
from pathlib import Path from pathlib import Path
from typing import Sequence from typing import Sequence
from .bilibili.client import BilibiliClient from .bilibili.client import BilibiliClient, Pacer
from .config import ConfigError, ConfigStore from .config import ConfigError, ConfigStore
from .database import Database from .database import Database
from .logging_json import JsonLogger from .logging_json import JsonLogger
from .repository import Repository from .repository import Repository
from .scheduler import ScheduledTask, Scheduler from .scheduler import ScheduledTask, Scheduler
from .services.folder_sync import FolderSyncService from .services.folder_sync import FolderSyncService
from .services.snapshot import SnapshotService
from .timeutil import next_daily_run, parse_hhmm, shanghai_date
ROOT = Path(__file__).resolve().parent.parent ROOT = Path(__file__).resolve().parent.parent
DEFAULT_CONFIG = ROOT / "config.json" DEFAULT_CONFIG = ROOT / "config.json"
@@ -39,7 +43,7 @@ def _build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
prog="app.py", prog="app.py",
description="咻咻满 Reaction 数据看板 —— P0 数据底座", description="咻咻满 Reaction 数据看板 —— 数据底座、同步与每日快照",
) )
sub = parser.add_subparsers(dest="command", required=True) sub = parser.add_subparsers(dest="command", required=True)
@@ -48,7 +52,14 @@ def _build_parser() -> argparse.ArgumentParser:
sync.add_argument( sync.add_argument(
"--full", action="store_true", help="强制全量刷新元数据(忽略 6 小时周期)" "--full", action="store_true", help="强制全量刷新元数据(忽略 6 小时周期)"
) )
run = sub.add_parser("run", parents=[common], help="按配置周期常驻运行收藏夹同步") snapshot = sub.add_parser(
"snapshot", parents=[common], help="采集当日七项快照与粉丝数"
)
snapshot.add_argument(
"--force", action="store_true", help="忽略当日完成标记,重新采集"
)
sub.add_parser("auth-check", parents=[common], help="检查 Cookie 登录态")
run = sub.add_parser("run", parents=[common], help="按配置周期常驻运行同步与快照")
run.add_argument( run.add_argument(
"--once", action="store_true", help="只执行一次到期任务后退出" "--once", action="store_true", help="只执行一次到期任务后退出"
) )
@@ -71,6 +82,17 @@ def _print(payload) -> None:
print(json.dumps(payload, ensure_ascii=False, indent=2), flush=True) print(json.dumps(payload, ensure_ascii=False, indent=2), flush=True)
def _client(cfg, logger: JsonLogger, *, pacer=None, retries: int = 3) -> BilibiliClient:
return BilibiliClient(
cookie=cfg.bilibili.cookie,
csrf=cfg.bilibili.csrf,
mid=cfg.bilibili.mid,
pacer=pacer,
retries=retries,
logger=logger,
)
def cmd_init(args, logger: JsonLogger) -> int: def cmd_init(args, logger: JsonLogger) -> int:
store = ConfigStore(args.config, require_auth=False) store = ConfigStore(args.config, require_auth=False)
cfg = store.get() cfg = store.get()
@@ -98,12 +120,7 @@ def cmd_sync(args, logger: JsonLogger) -> int:
db = _open_db(cfg, logger) db = _open_db(cfg, logger)
try: try:
repo = Repository(db) repo = Repository(db)
client = BilibiliClient( client = _client(cfg, logger)
cookie=cfg.bilibili.cookie,
csrf=cfg.bilibili.csrf,
mid=cfg.bilibili.mid,
logger=logger,
)
service = FolderSyncService(store, repo, client, logger=logger) service = FolderSyncService(store, repo, client, logger=logger)
summary = service.run(full_refresh=args.full) summary = service.run(full_refresh=args.full)
payload = summary.as_dict() payload = summary.as_dict()
@@ -115,6 +132,46 @@ def cmd_sync(args, logger: JsonLogger) -> int:
db.close() db.close()
def cmd_snapshot(args, logger: JsonLogger) -> int:
store = ConfigStore(args.config, require_auth=False)
cfg = store.get()
db = _open_db(cfg, logger)
try:
repo = Repository(db)
client = _client(
cfg, logger, pacer=Pacer(1.0, 1.6), retries=1
)
service = SnapshotService(store, repo, client, logger=logger)
summary = service.run_if_needed(force=args.force)
payload = summary.as_dict()
payload["database"] = str(cfg.storage.database_path)
payload["snapshot_counts"] = repo.snapshot_counts(summary.stat_date)
payload["counts"] = repo.counts()
_print(payload)
return 0 if summary.completed else 1
finally:
db.close()
def cmd_auth_check(args, logger: JsonLogger) -> int:
store = ConfigStore(args.config, require_auth=False)
cfg = store.get()
if not cfg.bilibili.cookie:
_print({"ok": False, "is_login": False, "message": "未配置 cookie"})
return 1
client = _client(cfg, logger, retries=1)
state = client.check_login()
_print({
"ok": state.is_login,
"is_login": state.is_login,
"mid": state.mid,
"uname": state.uname,
"code": state.code,
"message": state.message,
})
return 0 if state.is_login else 1
def cmd_run(args, logger: JsonLogger) -> int: def cmd_run(args, logger: JsonLogger) -> int:
store = ConfigStore(args.config, require_auth=True) store = ConfigStore(args.config, require_auth=True)
cfg = store.get() cfg = store.get()
@@ -122,26 +179,40 @@ def cmd_run(args, logger: JsonLogger) -> int:
last: dict[str, object] = {} last: dict[str, object] = {}
try: try:
repo = Repository(db) repo = Repository(db)
client = BilibiliClient( folder_client = _client(cfg, logger)
cookie=cfg.bilibili.cookie, folder_service = FolderSyncService(store, repo, folder_client, logger=logger)
csrf=cfg.bilibili.csrf, snapshot_client = _client(cfg, logger, pacer=Pacer(1.0, 1.6), retries=1)
mid=cfg.bilibili.mid, snapshot_service = SnapshotService(store, repo, snapshot_client, logger=logger)
logger=logger,
)
service = FolderSyncService(store, repo, client, logger=logger)
def run_sync(): def run_sync():
summary = service.run() summary = folder_service.run()
last["summary"] = summary last["folder"] = summary
logger.info("folder_sync.summary", **summary.as_dict()) logger.info("folder_sync.summary", **summary.as_dict())
return summary return summary
def run_snapshot():
summary = snapshot_service.run_if_needed()
last["snapshot"] = summary
logger.info("snapshot.summary", **summary.as_dict())
return summary
hh, mm = parse_hhmm(cfg.schedule.snapshot_at)
snapshot_at = f"{hh:02d}:{mm:02d}"
scheduler = Scheduler( scheduler = Scheduler(
[ScheduledTask( [
"folder_sync", ScheduledTask(
float(cfg.schedule.folder_sync_minutes * 60), "folder_sync",
run_sync, float(cfg.schedule.folder_sync_minutes * 60),
)], run_sync,
),
ScheduledTask(
"snapshot",
86400.0,
run_snapshot,
next_run=0.0,
next_run_fn=lambda started: next_daily_run(started, snapshot_at),
),
],
logger=logger, logger=logger,
) )
if args.once: if args.once:
@@ -150,11 +221,16 @@ def cmd_run(args, logger: JsonLogger) -> int:
else: else:
logger.info( logger.info(
"scheduler.start", "scheduler.start",
interval_seconds=cfg.schedule.folder_sync_minutes * 60, folder_sync_seconds=cfg.schedule.folder_sync_minutes * 60,
snapshot_at=snapshot_at,
) )
scheduler.run_forever() scheduler.run_forever()
summary = last.get("summary")
return 0 if summary is None or getattr(summary, "ok", True) else 1 folder_summary = last.get("folder")
snapshot_summary = last.get("snapshot")
folder_ok = folder_summary is None or getattr(folder_summary, "ok", True)
snapshot_ok = snapshot_summary is None or getattr(snapshot_summary, "completed", True)
return 0 if folder_ok and snapshot_ok else 1
finally: finally:
db.close() db.close()
@@ -165,6 +241,15 @@ def cmd_status(args, logger: JsonLogger) -> int:
db = _open_db(cfg, logger) db = _open_db(cfg, logger)
try: try:
repo = Repository(db) repo = Repository(db)
today = shanghai_date()
snapshot = repo.snapshot_counts(today)
snapshot.update({
"date": today,
"last_attempt": repo.kv_get("snapshot:last_attempt"),
"last_completed": repo.kv_get("snapshot:last_completed"),
"auth_ok": repo.kv_get("auth:ok"),
"auth_message": repo.kv_get("auth:message"),
})
_print({ _print({
"ok": True, "ok": True,
"schema_version": db.schema_version, "schema_version": db.schema_version,
@@ -181,6 +266,7 @@ def cmd_status(args, logger: JsonLogger) -> int:
} }
for row in repo.list_folders() for row in repo.list_folders()
], ],
"snapshot": snapshot,
**repo.counts(), **repo.counts(),
}) })
finally: finally:
@@ -196,6 +282,10 @@ def main(argv: Sequence[str] | None = None) -> int:
return cmd_init(args, logger) return cmd_init(args, logger)
if args.command == "sync": if args.command == "sync":
return cmd_sync(args, logger) return cmd_sync(args, logger)
if args.command == "snapshot":
return cmd_snapshot(args, logger)
if args.command == "auth-check":
return cmd_auth_check(args, logger)
if args.command == "run": if args.command == "run":
return cmd_run(args, logger) return cmd_run(args, logger)
if args.command == "status": if args.command == "status":
+20
View File
@@ -7,6 +7,7 @@
from __future__ import annotations from __future__ import annotations
import os
import queue import queue
import sqlite3 import sqlite3
import threading import threading
@@ -165,6 +166,25 @@ class Database:
finally: finally:
conn.close() conn.close()
def backup_to(self, target: str | Path) -> Path:
"""用 SQLite backup API 生成一致性备份;先写临时文件再原子替换。"""
target = Path(target)
target.parent.mkdir(parents=True, exist_ok=True)
tmp = target.with_name(target.name + ".tmp")
if tmp.exists():
tmp.unlink()
def run(conn: sqlite3.Connection) -> None:
dst = sqlite3.connect(tmp)
try:
conn.backup(dst)
finally:
dst.close()
self.write(run)
os.replace(tmp, target)
return target
def close(self) -> None: def close(self) -> None:
if self._closed: if self._closed:
return return
+131 -1
View File
@@ -10,7 +10,7 @@ from __future__ import annotations
import sqlite3 import sqlite3
import time import time
from dataclasses import dataclass from dataclasses import dataclass
from typing import Any, Iterable from typing import Any, Iterable, Mapping
from .config import FolderConfig from .config import FolderConfig
from .database import Database from .database import Database
@@ -368,6 +368,136 @@ class Repository:
return [dict(row) for row in rows] return [dict(row) for row in rows]
return self.db.read(run) return self.db.read(run)
# ------------------------------------------------------------ snapshot
def video_aids_for_snapshot(self) -> list[int]:
def run(conn: sqlite3.Connection) -> list[int]:
rows = conn.execute(
"SELECT aid FROM video WHERE status IN ('active', 'unfaved') "
"ORDER BY aid"
).fetchall()
return [int(row["aid"]) for row in rows]
return self.db.read(run)
def get_snapshot(self, stat_date: str, aid: int) -> dict[str, Any] | None:
def run(conn: sqlite3.Connection) -> dict[str, Any] | None:
row = conn.execute(
"SELECT * FROM snapshot WHERE stat_date = ? AND aid = ?",
(stat_date, int(aid)),
).fetchone()
return dict(row) if row else None
return self.db.read(run)
def upsert_snapshot(
self,
stat_date: str,
aid: int,
*,
metrics: Mapping[str, int] | None,
ok: bool,
err: str | None,
captured_at: int,
src: str = "view",
) -> None:
values = metrics or {}
def run(conn: sqlite3.Connection) -> None:
conn.execute(
"""
INSERT INTO snapshot(stat_date, aid, "view", "like", favorite, coin,
share, reply, danmaku, captured_at, ok, err, src)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(stat_date, aid) DO UPDATE SET
"view" = excluded."view",
"like" = excluded."like",
favorite = excluded.favorite,
coin = excluded.coin,
share = excluded.share,
reply = excluded.reply,
danmaku = excluded.danmaku,
captured_at = excluded.captured_at,
ok = excluded.ok,
err = excluded.err,
src = excluded.src
""",
(
stat_date,
int(aid),
values.get("view"),
values.get("like"),
values.get("favorite"),
values.get("coin"),
values.get("share"),
values.get("reply"),
values.get("danmaku"),
int(captured_at),
1 if ok else 0,
err,
src,
),
)
self.db.write(run)
def snapshot_counts(self, stat_date: str) -> dict[str, int]:
def run(conn: sqlite3.Connection) -> dict[str, int]:
rows = conn.execute(
"SELECT ok, COUNT(*) AS n FROM snapshot WHERE stat_date = ? GROUP BY ok",
(stat_date,),
).fetchall()
by_ok = {int(row["ok"]): int(row["n"]) for row in rows}
return {
"total": sum(by_ok.values()),
"ok": by_ok.get(1, 0),
"failed": by_ok.get(0, 0),
}
return self.db.read(run)
def upsert_up_stat(
self,
stat_date: str,
mid: int,
*,
follower: int | None,
following: int | None,
ok: bool,
captured_at: int,
) -> None:
self.db.write(lambda conn: conn.execute(
"""
INSERT INTO up_stat(stat_date, mid, follower, following, captured_at, ok)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(stat_date, mid) DO UPDATE SET
follower = excluded.follower,
following = excluded.following,
captured_at = excluded.captured_at,
ok = excluded.ok
""",
(stat_date, int(mid), follower, following, int(captured_at), 1 if ok else 0),
))
def get_up_stat(self, stat_date: str, mid: int) -> dict[str, Any] | None:
def run(conn: sqlite3.Connection) -> dict[str, Any] | None:
row = conn.execute(
"SELECT * FROM up_stat WHERE stat_date = ? AND mid = ?",
(stat_date, int(mid)),
).fetchone()
return dict(row) if row else None
return self.db.read(run)
# ------------------------------------------------------------ kv
def kv_get(self, key: str) -> str | None:
def run(conn: sqlite3.Connection) -> str | None:
row = conn.execute("SELECT v FROM kv WHERE k = ?", (key,)).fetchone()
return None if row is None else str(row["v"])
return self.db.read(run)
def kv_set(self, key: str, value: str) -> None:
self.db.write(lambda conn: conn.execute(
"INSERT INTO kv(k, v) VALUES (?, ?) "
"ON CONFLICT(k) DO UPDATE SET v = excluded.v",
(key, str(value)),
))
# ------------------------------------------------------------ blacklist # ------------------------------------------------------------ blacklist
def seed_blacklist(self, mids: Iterable[int], *, now: int | None = None) -> int: def seed_blacklist(self, mids: Iterable[int], *, now: int | None = None) -> int:
+5 -1
View File
@@ -18,6 +18,7 @@ class ScheduledTask:
interval: float interval: float
fn: Callable[[], Any] fn: Callable[[], Any]
next_run: float = 0.0 next_run: float = 0.0
next_run_fn: Callable[[float], float] | None = None
@dataclass @dataclass
@@ -77,7 +78,10 @@ class Scheduler:
else: else:
self._log("info", "scheduler.task_ok", task=task.name) self._log("info", "scheduler.task_ok", task=task.name)
finished = self._clock() finished = self._clock()
task.next_run = started + task.interval if task.next_run_fn is not None:
task.next_run = task.next_run_fn(started)
else:
task.next_run = started + task.interval
result = TaskResult( result = TaskResult(
name=task.name, name=task.name,
started_at=started, started_at=started,
+50
View File
@@ -0,0 +1,50 @@
# -*- coding: utf-8 -*-
"""备份与保留期清理。"""
from __future__ import annotations
import time
from pathlib import Path
from ..database import Database
def daily_backup_path(backup_dir: str | Path, stat_date: str) -> Path:
compact = stat_date.replace("-", "")
return Path(backup_dir) / f"app-{compact}.db"
def backup_daily(
db: Database,
backup_dir: str | Path,
stat_date: str,
*,
logger=None,
) -> Path:
target = daily_backup_path(backup_dir, stat_date)
db.backup_to(target)
if logger is not None:
logger.info("backup.daily", path=str(target), stat_date=stat_date)
return target
def prune_daily_backups(
backup_dir: str | Path,
*,
keep_days: int,
now: float | None = None,
) -> list[Path]:
"""只清理 ``app-YYYYMMDD.db`` 日备份,不动迁移备份。"""
target_dir = Path(backup_dir)
if not target_dir.is_dir():
return []
cutoff = (time.time() if now is None else now) - keep_days * 86400
removed: list[Path] = []
for path in target_dir.glob("app-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].db"):
try:
if path.stat().st_mtime < cutoff:
path.unlink()
removed.append(path)
except OSError:
continue
return removed
+313
View File
@@ -0,0 +1,313 @@
# -*- coding: utf-8 -*-
"""每日视频七项快照与粉丝数采集。
规则(DEVELOPMENT.md §7.4):
* 对 ``active`` / ``unfaved`` 视频逐条请求 ``view``;
* 成功写入七项指标;``ok=1`` 的同日快照直接跳过(断点续跑);
* 视频不可访问时标记 ``removed``;
* 其他错误最多重试 3 次,仍失败写 ``ok=0`` 占位,不写假零值;
* 对 ``track_fans`` 逐个请求 ``relation/stat``;
* 完成后写入当日完成标记并生成日备份。
"""
from __future__ import annotations
import time
from dataclasses import dataclass, field
from typing import Any
from ..bilibili.client import (
VIDEO_UNAVAILABLE_CODES,
BilibiliAPIError,
BilibiliClient,
BilibiliError,
BilibiliHTTPError,
)
from ..config import AppConfig, ConfigStore
from ..repository import Repository
from ..timeutil import is_date, shanghai_date
from .maintenance import backup_daily, prune_daily_backups
@dataclass
class SnapshotSummary:
stat_date: str
started_at: int
total: int = 0
success: int = 0
failed: int = 0
skipped: int = 0
removed: int = 0
fans_ok: int = 0
fans_failed: int = 0
auth_ok: bool | None = None
auth_message: str = ""
backup_path: str | None = None
backup_error: str | None = None
completed: bool = False
already_completed: bool = False
finished_at: int = 0
errors: list[str] = field(default_factory=list)
def as_dict(self) -> dict[str, Any]:
return {
"stat_date": self.stat_date,
"started_at": self.started_at,
"finished_at": self.finished_at,
"elapsed": max(0, self.finished_at - self.started_at),
"total": self.total,
"success": self.success,
"failed": self.failed,
"skipped": self.skipped,
"removed": self.removed,
"fans_ok": self.fans_ok,
"fans_failed": self.fans_failed,
"auth_ok": self.auth_ok,
"auth_message": self.auth_message,
"backup_path": self.backup_path,
"backup_error": self.backup_error,
"completed": self.completed,
"already_completed": self.already_completed,
"errors": self.errors,
}
class SnapshotService:
def __init__(
self,
config_store: ConfigStore,
repository: Repository,
client: BilibiliClient,
*,
clock=time.time,
sleeper=time.sleep,
logger=None,
retries: int = 3,
backoff: float = 5.0,
progress_every: int = 50,
) -> None:
if retries < 1:
raise ValueError("retries 必须 >= 1")
self.config_store = config_store
self.repo = repository
self.client = client
self._clock = clock
self._sleep = sleeper
self.logger = logger
self.retries = retries
self.backoff = backoff
self.progress_every = progress_every
def _now(self) -> int:
return int(self._clock())
def _log(self, level: str, event: str, **fields) -> None:
if self.logger is not None:
getattr(self.logger, level)(event, **fields)
@staticmethod
def completed_key(stat_date: str) -> str:
return f"snapshot:completed:{stat_date}"
def run_if_needed(
self, *, date: str | None = None, force: bool = False
) -> SnapshotSummary:
now = self._now()
stat_date = date or shanghai_date(now)
if not is_date(stat_date):
raise ValueError(f"日期格式必须是 YYYY-MM-DD: {stat_date!r}")
if not force and self.repo.kv_get(self.completed_key(stat_date)) == "1":
self._log("info", "snapshot.skip", stat_date=stat_date, reason="already_completed")
return SnapshotSummary(
stat_date=stat_date,
started_at=now,
finished_at=now,
completed=True,
already_completed=True,
)
return self.run(date=stat_date, force=force)
def run(self, *, date: str | None = None, force: bool = False) -> SnapshotSummary:
cfg = self.config_store.get()
now = self._now()
stat_date = date or shanghai_date(now)
if not is_date(stat_date):
raise ValueError(f"日期格式必须是 YYYY-MM-DD: {stat_date!r}")
summary = SnapshotSummary(stat_date=stat_date, started_at=now)
self._log("info", "snapshot.start", stat_date=stat_date, force=force)
summary.auth_ok, summary.auth_message = self._check_auth(cfg, now)
videos = self.repo.video_aids_for_snapshot()
summary.total = len(videos)
for index, aid in enumerate(videos, 1):
existing = self.repo.get_snapshot(stat_date, aid)
if existing is not None and int(existing.get("ok") or 0) == 1 and not force:
summary.skipped += 1
continue
stat, error, unavailable = self._fetch_view(aid)
if unavailable:
summary.removed += 1
self.repo.set_video_status(aid, "removed", now=now)
self._log(
"warning", "snapshot.removed",
stat_date=stat_date, aid=aid, error=error,
)
elif stat is not None:
self.repo.upsert_snapshot(
stat_date,
aid,
metrics=stat.metrics(),
ok=True,
err=None,
captured_at=self._now(),
)
summary.success += 1
else:
summary.failed += 1
self.repo.upsert_snapshot(
stat_date,
aid,
metrics=None,
ok=False,
err=error,
captured_at=self._now(),
)
summary.errors.append(f"aid {aid}: {error}")
self._log(
"error", "snapshot.failed",
stat_date=stat_date, aid=aid, error=error,
)
if self.progress_every > 0 and (
index % self.progress_every == 0 or index == summary.total
):
self._log(
"info", "snapshot.progress",
stat_date=stat_date, done=index, total=summary.total,
success=summary.success, failed=summary.failed,
skipped=summary.skipped, removed=summary.removed,
)
for mid in cfg.track_fans:
stat, error = self._fetch_fan(mid)
if stat is not None:
self.repo.upsert_up_stat(
stat_date,
mid,
follower=stat.follower,
following=stat.following,
ok=True,
captured_at=self._now(),
)
summary.fans_ok += 1
else:
summary.fans_failed += 1
self.repo.upsert_up_stat(
stat_date,
mid,
follower=None,
following=None,
ok=False,
captured_at=self._now(),
)
summary.errors.append(f"fans mid {mid}: {error}")
self._log(
"error", "snapshot.fan_failed",
stat_date=stat_date, mid=mid, error=error,
)
summary.completed = summary.failed == 0 and summary.fans_failed == 0
self.repo.kv_set(self.completed_key(stat_date), "1" if summary.completed else "0")
self.repo.kv_set("snapshot:last_attempt", stat_date)
if summary.completed:
self.repo.kv_set("snapshot:last_completed", stat_date)
elif self.repo.kv_get("snapshot:last_completed") == stat_date:
self.repo.kv_set("snapshot:last_completed", "")
try:
path = backup_daily(
self.repo.db,
cfg.storage.backup_dir,
stat_date,
logger=self.logger,
)
prune_daily_backups(
cfg.storage.backup_dir,
keep_days=cfg.storage.backup_keep_days,
)
summary.backup_path = str(path)
except Exception as exc: # noqa: BLE001 备份失败不推翻已采集数据
summary.backup_error = str(exc)
self._log("error", "backup.failed", error=str(exc), stat_date=stat_date)
summary.finished_at = self._now()
self._log(
"info" if summary.completed else "warning",
"snapshot.done",
**summary.as_dict(),
)
return summary
# ------------------------------------------------------------ internals
def _check_auth(self, cfg: AppConfig, now: int) -> tuple[bool, str]:
if not cfg.bilibili.cookie:
self.repo.kv_set("auth:last_check", str(now))
self.repo.kv_set("auth:ok", "0")
self.repo.kv_set("auth:message", "未配置 cookie")
return False, "未配置 cookie"
try:
state = self.client.check_login()
except BilibiliError as exc:
message = str(exc)
self.repo.kv_set("auth:last_check", str(now))
self.repo.kv_set("auth:ok", "0")
self.repo.kv_set("auth:message", message)
self._log("warning", "auth.invalid", message=message)
return False, message
self.repo.kv_set("auth:last_check", str(now))
self.repo.kv_set("auth:ok", "1" if state.is_login else "0")
self.repo.kv_set("auth:message", state.message)
if state.mid is not None:
self.repo.kv_set("auth:mid", str(state.mid))
if not state.is_login:
self._log(
"warning", "auth.invalid",
code=state.code, message=state.message,
)
return state.is_login, state.message
def _fetch_view(self, aid: int):
last_error = ""
for attempt in range(1, self.retries + 1):
try:
return self.client.view(aid), None, False
except BilibiliAPIError as exc:
last_error = str(exc)
if exc.code in VIDEO_UNAVAILABLE_CODES:
return None, last_error, True
except BilibiliHTTPError as exc:
last_error = str(exc)
if exc.status == 404:
return None, last_error, True
except BilibiliError as exc:
last_error = str(exc)
if attempt < self.retries:
self._sleep(self.backoff * (2 ** (attempt - 1)))
return None, last_error or "未知错误", False
def _fetch_fan(self, mid: int):
last_error = ""
for attempt in range(1, self.retries + 1):
try:
return self.client.relation_stat(mid), None
except BilibiliError as exc:
last_error = str(exc)
if attempt < self.retries:
self._sleep(self.backoff * (2 ** (attempt - 1)))
return None, last_error or "未知错误"
+59
View File
@@ -0,0 +1,59 @@
# -*- coding: utf-8 -*-
"""上海时区与每日任务时间计算。
B 站数据口径使用 ``Asia/Shanghai``。中国自 1991 年起不再使用夏令时,因此这里
用固定 UTC+8,避免 Windows 上依赖未安装的 ``tzdata`` 包。
"""
from __future__ import annotations
import re
import time
from datetime import datetime, timedelta, timezone
SHANGHAI_TZ = timezone(timedelta(hours=8))
_HHMM_RE = re.compile(r"^([01]\d|2[0-3]):([0-5]\d)$")
_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
def shanghai_datetime(ts: float | None = None) -> datetime:
return datetime.fromtimestamp(time.time() if ts is None else ts, SHANGHAI_TZ)
def shanghai_date(ts: float | None = None) -> str:
return shanghai_datetime(ts).strftime("%Y-%m-%d")
def parse_hhmm(value: str) -> tuple[int, int]:
match = _HHMM_RE.match(value or "")
if not match:
raise ValueError(f"时间必须是 HH:MM(24 小时制): {value!r}")
return int(match.group(1)), int(match.group(2))
def next_daily_run(now: float, hhmm: str) -> float:
"""返回 ``now`` 之后(严格大于)最近的 ``hh:mm`` 上海时间时间戳。"""
hour, minute = parse_hhmm(hhmm)
current = shanghai_datetime(now)
target = current.replace(hour=hour, minute=minute, second=0, microsecond=0)
if target.timestamp() <= now:
target += timedelta(days=1)
return target.timestamp()
def is_date(value: str) -> bool:
if not _DATE_RE.match(value or ""):
return False
try:
datetime.strptime(value, "%Y-%m-%d")
except ValueError:
return False
return True
def day_bounds(date: str) -> tuple[float, float]:
"""返回上海时区某一天的 [start, end) 时间戳。"""
if not is_date(date):
raise ValueError(f"日期格式必须是 YYYY-MM-DD: {date!r}")
start = datetime.strptime(date, "%Y-%m-%d").replace(tzinfo=SHANGHAI_TZ)
return start.timestamp(), (start + timedelta(days=1)).timestamp()
+105
View File
@@ -161,3 +161,108 @@ def info_for(
f"bilibili://music/playlist/playpage/3728825651?page_type=3&oid={aid}&otype=2" f"bilibili://music/playlist/playpage/3728825651?page_type=3&oid={aid}&otype=2"
) )
return item return item
# GET x/web-interface/view?aid=117530286889638(真实七项 stat)
VIEW_PAYLOAD: dict[str, Any] = {
"code": 0,
"message": "0",
"ttl": 1,
"data": {
"aid": 117530286889638,
"bvid": "BV1xvxi6LEZD",
"cid": 43184885058,
"title": "【咻咻满充电专场①】百变音色机能怪,还有什么风格是你驾驭不了的?!",
"pubdate": 1793370433,
"ctime": 1793370433,
"duration": 9504,
"owner": {"mid": 434442072, "name": "_-抱抱-_", "face": ""},
"stat": {
"aid": 117530286889638,
"view": 819,
"danmaku": 7,
"reply": 16,
"favorite": 43,
"coin": 9,
"share": 1,
"now_rank": 0,
"his_rank": 0,
"like": 44,
"dislike": 0,
"evaluation": "",
"vt": 0,
},
},
}
# 当前实测:收藏夹里已不可访问的 aid 返回 62012,而不是旧文档里的 -404
VIEW_UNAVAILABLE: dict[str, Any] = {
"code": 62012,
"message": "62012",
"ttl": 1,
"data": None,
}
# GET x/relation/stat?vmid=37754047
RELATION_PAYLOAD: dict[str, Any] = {
"code": 0,
"message": "0",
"ttl": 1,
"data": {
"mid": 37754047,
"following": 95,
"whisper": 0,
"black": 0,
"follower": 2767268,
"fans_medal_toast": None,
"fans_effect": None,
},
}
# GET x/web-interface/nav(已登录)
NAV_PAYLOAD: dict[str, Any] = {
"code": 0,
"message": "0",
"ttl": 1,
"data": {
"isLogin": True,
"mid": 28802351,
"uname": "冷萃编译器",
"face": "https://i0.hdslb.com/bfs/face/example.jpg",
"level_info": {"current_level": 5},
"vipStatus": 0,
"wbi_img": {
"img_url": "https://i0.hdslb.com/bfs/wbi/example.png",
"sub_url": "https://i0.hdslb.com/bfs/wbi/example-sub.png",
},
},
}
def view_for(
aid: int,
*,
view: int = 819,
like: int = 44,
coin: int = 9,
favorite: int = 43,
share: int = 1,
reply: int = 16,
danmaku: int = 7,
bvid: str | None = None,
) -> dict[str, Any]:
item = copy.deepcopy(VIEW_PAYLOAD)
data = item["data"]
data["aid"] = aid
data["bvid"] = bvid or f"BV{aid}"
data["stat"].update({
"aid": aid,
"view": view,
"like": like,
"coin": coin,
"favorite": favorite,
"share": share,
"reply": reply,
"danmaku": danmaku,
})
return item
+92
View File
@@ -25,8 +25,13 @@ from fixtures import ( # noqa: E402
FOLDERS_PAYLOAD, FOLDERS_PAYLOAD,
IDS_PAYLOAD, IDS_PAYLOAD,
INVALID_INFO, INVALID_INFO,
NAV_PAYLOAD,
RELATION_PAYLOAD,
VALID_INFO, VALID_INFO,
VIEW_PAYLOAD,
VIEW_UNAVAILABLE,
info_for, info_for,
view_for,
) )
@@ -274,6 +279,88 @@ def test_timeout_and_method_forwarded():
assert call["timeout"] == 30 assert call["timeout"] == 30
def test_view_seven_metrics():
transport = QueueTransport([http_json(VIEW_PAYLOAD)])
client, _ = make_client(transport)
stat = client.view(117530286889638)
assert stat.aid == 117530286889638
assert stat.bvid == "BV1xvxi6LEZD"
assert stat.metrics() == {
"view": 819, "like": 44, "favorite": 43, "coin": 9,
"share": 1, "reply": 16, "danmaku": 7,
}
assert "aid=117530286889638" in transport.calls[0]["url"]
def test_view_unavailable_and_malformed():
transport = QueueTransport([
http_json(VIEW_UNAVAILABLE), http_json(VIEW_UNAVAILABLE),
])
client, _ = make_client(transport)
try:
client.view(1)
except BilibiliAPIError as exc:
assert exc.code == 62012
else:
raise AssertionError("不可访问视频应抛业务错误")
payload = view_for(1)
payload["data"]["stat"].pop("coin")
bad = http_json(payload)
transport = QueueTransport([bad, bad])
client, _ = make_client(transport)
try:
client.view(1)
except BilibiliProtocolError as exc:
assert "coin" in str(exc)
else:
raise AssertionError("缺少 stat 字段应报协议错误")
def test_relation_stat():
transport = QueueTransport([http_json(RELATION_PAYLOAD)])
client, _ = make_client(transport)
fan = client.relation_stat(37754047)
assert fan.mid == 37754047
assert fan.follower == 2767268
assert fan.following == 95
assert "vmid=37754047" in transport.calls[0]["url"]
def test_check_login():
transport = QueueTransport([http_json(NAV_PAYLOAD)])
client, _ = make_client(transport)
state = client.check_login()
assert state.is_login is True
assert state.mid == 28802351
assert state.uname == "冷萃编译器"
assert "SESSDATA=secret" in transport.calls[0]["headers"]["cookie"]
transport = QueueTransport([http_json({
"code": 0, "message": "0", "ttl": 1,
"data": {"isLogin": False, "mid": 0, "uname": ""},
})])
client, _ = make_client(transport)
state = client.check_login()
assert state.is_login is False and state.mid == 0
transport = QueueTransport([http_json({"code": -101, "message": "账号未登录", "data": None})])
client, _ = make_client(transport)
state = client.check_login()
assert state.is_login is False
assert state.code == -101
assert "账号未登录" in state.message
def test_check_login_without_cookie_makes_no_request():
transport = QueueTransport([])
client, _ = make_client(transport, cookie="")
state = client.check_login()
assert state.is_login is False
assert state.message == "未配置 cookie"
assert transport.calls == []
def main(): def main():
test_fav_ids_parse_and_anonymous_header() test_fav_ids_parse_and_anonymous_header()
test_anonymous_failure_falls_back_to_full_cookie() test_anonymous_failure_falls_back_to_full_cookie()
@@ -288,6 +375,11 @@ def main():
test_fav_infos_chunk_size_validation() test_fav_infos_chunk_size_validation()
test_pacer_delays_after_first_request() test_pacer_delays_after_first_request()
test_timeout_and_method_forwarded() test_timeout_and_method_forwarded()
test_view_seven_metrics()
test_view_unavailable_and_malformed()
test_relation_stat()
test_check_login()
test_check_login_without_cookie_makes_no_request()
print("OK: B 站只读客户端契约与重试测试全部通过") print("OK: B 站只读客户端契约与重试测试全部通过")
+44 -5
View File
@@ -12,9 +12,11 @@ from pathlib import Path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
sys.stdout.reconfigure(encoding="utf-8") sys.stdout.reconfigure(encoding="utf-8")
from reaction_board.bilibili.models import LoginState # noqa: E402
from reaction_board.cli import main # noqa: E402 from reaction_board.cli import main # noqa: E402
from reaction_board.database import Database # noqa: E402 from reaction_board.database import Database # noqa: E402
from reaction_board.repository import Repository # noqa: E402 from reaction_board.repository import Repository # noqa: E402
import reaction_board.cli as cli # noqa: E402
def write_config(tmp, *, cookie="SESSDATA=x; buvid3=y", csrf="csrf"): def write_config(tmp, *, cookie="SESSDATA=x; buvid3=y", csrf="csrf"):
@@ -86,24 +88,61 @@ def test_sync_requires_auth_without_network():
assert not (Path(tmp) / "app.db").exists() assert not (Path(tmp) / "app.db").exists()
def test_run_once_without_folders(): def test_snapshot_offline_empty_db():
with tempfile.TemporaryDirectory() as tmp:
path = write_config(tmp, cookie="", csrf="")
data = json.loads(path.read_text(encoding="utf-8"))
data["folders"] = []
data["track_fans"] = []
path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
code, output = run_cli(["snapshot", "--config", str(path)])
assert code == 0
assert '"completed": true' in output
assert '"total": 0' in output
assert len(list((Path(tmp) / "backup").glob("app-*.db"))) == 1
def test_auth_check_without_cookie():
with tempfile.TemporaryDirectory() as tmp:
path = write_config(tmp, cookie="", csrf="")
code, output = run_cli(["auth-check", "--config", str(path)])
assert code == 1
payload = json.loads(output)
assert payload["is_login"] is False
assert "未配置 cookie" in payload["message"]
def test_run_once_offline():
class FakeClient:
def check_login(self):
return LoginState(True, 1, "tester", 0, "OK")
with tempfile.TemporaryDirectory() as tmp: with tempfile.TemporaryDirectory() as tmp:
path = write_config(tmp) path = write_config(tmp)
data = json.loads(path.read_text(encoding="utf-8")) data = json.loads(path.read_text(encoding="utf-8"))
data["folders"] = [] data["folders"] = []
data["track_fans"] = []
path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8") path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
code, output = run_cli(["run", "--once", "--config", str(path)])
original = cli._client
cli._client = lambda *args, **kwargs: FakeClient()
try:
code, output = run_cli(["run", "--once", "--config", str(path)])
finally:
cli._client = original
assert code == 0 assert code == 0
assert '"name": "folder_sync"' in output assert '"name": "folder_sync"' in output
assert '"ok": true' in output assert '"name": "snapshot"' in output
def main_test(): def main_test():
test_init_and_status() test_init_and_status()
test_missing_config() test_missing_config()
test_sync_requires_auth_without_network() test_sync_requires_auth_without_network()
test_run_once_without_folders() test_snapshot_offline_empty_db()
print("OK: CLI init/sync/run/status/错误路径测试全部通过") test_auth_check_without_cookie()
test_run_once_offline()
print("OK: CLI init/sync/snapshot/auth-check/run/status/错误路径测试全部通过")
if __name__ == "__main__": if __name__ == "__main__":
+78
View File
@@ -0,0 +1,78 @@
# -*- coding: utf-8 -*-
"""每日备份与保留期测试: uv run python tests/test_maintenance.py"""
import os
import sqlite3
import sys
import tempfile
from pathlib import Path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
sys.stdout.reconfigure(encoding="utf-8")
from reaction_board.database import Database # noqa: E402
from reaction_board.services.maintenance import ( # noqa: E402
backup_daily,
daily_backup_path,
prune_daily_backups,
)
def test_daily_backup_and_restore():
with tempfile.TemporaryDirectory() as tmp:
db = Database(Path(tmp) / "app.db")
db.initialize()
try:
db.write(lambda c: c.execute("INSERT INTO kv(k, v) VALUES ('x', '42')"))
target = backup_daily(db, Path(tmp) / "backup", "2026-10-31")
assert target == daily_backup_path(Path(tmp) / "backup", "2026-10-31")
assert target.name == "app-20261031.db"
assert target.exists()
check = sqlite3.connect(target)
try:
assert check.execute("SELECT v FROM kv WHERE k='x'").fetchone()[0] == "42"
finally:
check.close()
# 覆盖同日备份也安全
db.write(lambda c: c.execute("UPDATE kv SET v='43' WHERE k='x'"))
backup_daily(db, Path(tmp) / "backup", "2026-10-31")
check = sqlite3.connect(target)
try:
assert check.execute("SELECT v FROM kv WHERE k='x'").fetchone()[0] == "43"
finally:
check.close()
finally:
db.close()
def test_prune_daily_backups_only():
with tempfile.TemporaryDirectory() as tmp:
backup_dir = Path(tmp)
old_daily = backup_dir / "app-20260101.db"
fresh_daily = backup_dir / "app-20261031.db"
old_migration = backup_dir / "app-pre-v1-20260101.db"
other = backup_dir / "keep.txt"
for path in (old_daily, fresh_daily, old_migration, other):
path.write_bytes(b"x")
now = 1_800_000_000
old_time = now - 20 * 86400
for path in (old_daily, old_migration):
os.utime(path, (old_time, old_time))
os.utime(fresh_daily, (now, now))
removed = prune_daily_backups(backup_dir, keep_days=14, now=now)
assert removed == [old_daily]
assert fresh_daily.exists()
assert old_migration.exists() # 迁移备份不属于日备份清理范围
assert other.exists()
def main():
test_daily_backup_and_restore()
test_prune_daily_backups_only()
print("OK: 每日备份与保留期测试全部通过")
if __name__ == "__main__":
main()
+19
View File
@@ -110,8 +110,27 @@ def test_max_iterations_and_validation():
raise AssertionError(f"应当失败: {tasks}, {kwargs}") raise AssertionError(f"应当失败: {tasks}, {kwargs}")
def test_next_run_fn_controls_reschedule():
clock = [1_000.0]
calls = []
task = ScheduledTask(
"daily", 86400, lambda: calls.append(clock[0]),
next_run_fn=lambda started: started + 123,
)
scheduler = Scheduler([task], clock=lambda: clock[0])
scheduler.run_due()
assert calls == [1_000.0]
assert task.next_run == 1_123.0
clock[0] = 1_122.0
assert scheduler.run_due() == []
clock[0] = 1_123.0
scheduler.run_due()
assert calls == [1_000.0, 1_123.0]
def main(): def main():
test_run_when_due_and_reschedule() test_run_when_due_and_reschedule()
test_next_run_fn_controls_reschedule()
test_task_error_does_not_block_other_tasks() test_task_error_does_not_block_other_tasks()
test_run_forever_with_stop_event() test_run_forever_with_stop_event()
test_max_iterations_and_validation() test_max_iterations_and_validation()
+331
View File
@@ -0,0 +1,331 @@
# -*- coding: utf-8 -*-
"""每日快照与粉丝采集测试(transport 边界 Mock,不联网):
uv run python tests/test_snapshot.py
"""
import io
import json
import os
import sys
import tempfile
import urllib.parse
from pathlib import Path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
sys.stdout.reconfigure(encoding="utf-8")
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from fixtures import ( # noqa: E402
NAV_PAYLOAD,
RELATION_PAYLOAD,
VIEW_UNAVAILABLE,
view_for,
)
from reaction_board.bilibili.client import ( # noqa: E402
BilibiliClient,
HTTPResponse,
Pacer,
)
from reaction_board.bilibili.models import FavResource # noqa: E402
from reaction_board.config import ConfigStore, FolderConfig # noqa: E402
from reaction_board.database import Database # noqa: E402
from reaction_board.logging_json import JsonLogger # noqa: E402
from reaction_board.repository import Repository # noqa: E402
from reaction_board.services.snapshot import SnapshotService # noqa: E402
DATE = "2026-10-31"
FOLDER = FolderConfig(year=2026, media_id=3700023351, title="2026")
def http_json(payload):
return HTTPResponse(
200, {"content-type": "application/json"},
json.dumps(payload, ensure_ascii=False).encode("utf-8"),
)
class SnapshotTransport:
def __init__(self, views, *, nav=None, fan=None):
self.views = views
self.nav = NAV_PAYLOAD if nav is None else nav
self.fan = RELATION_PAYLOAD if fan is None else fan
self.calls = []
def request(self, method, url, headers, data=None, timeout=30):
self.calls.append(url)
if "/x/web-interface/nav" in url:
return http_json(self.nav)
if "/x/web-interface/view" in url:
aid = int(urllib.parse.parse_qs(
urllib.parse.urlparse(url).query
)["aid"][0])
payload = self.views[aid]
if isinstance(payload, HTTPResponse):
return payload
return http_json(payload)
if "/x/relation/stat" in url:
return http_json(self.fan)
raise AssertionError(f"未预期的 URL: {url}")
def count(self, part):
return sum(1 for url in self.calls if part in url)
def make_service(tmp, transport, *, track_fans=(37754047,), logger=None,
retries=3, cookie="SESSDATA=secret; buvid3=device"):
path = Path(tmp) / "config.json"
path.write_text(json.dumps({
"mid": 28802351,
"csrf": "csrf",
"cookie": cookie,
"storage": {
"database_path": str(Path(tmp) / "app.db"),
"backup_dir": str(Path(tmp) / "backup"),
"backup_keep_days": 14,
},
"folders": [],
"track_fans": list(track_fans),
}, ensure_ascii=False), encoding="utf-8")
store = ConfigStore(path, env={}, require_auth=False)
db = Database(Path(tmp) / "app.db", backup_dir=Path(tmp) / "backup")
db.initialize()
repo = Repository(db, clock=lambda: 1_800_000_000)
repo.ensure_folder(FOLDER)
client = BilibiliClient(
cookie=cookie,
csrf="csrf",
mid=28802351,
transport=transport,
pacer=Pacer(0, 0),
retries=1,
)
service = SnapshotService(
store, repo, client,
clock=lambda: 1_800_000_000,
sleeper=lambda seconds: None,
logger=logger,
retries=retries,
backoff=0,
)
return db, repo, service
def add_video(repo, aid, *, status="active"):
repo.upsert_from_fav(
FavResource(
aid=aid, bvid=f"BV{aid}", title=f"视频{aid}", up_mid=100,
up_name="UP", pubdate=1_700_000_000, duration=600,
cover=None, attr=0, fav_time=None,
),
FOLDER.media_id,
was_added=True,
)
if status != "active":
repo.set_video_status(aid, status)
def test_snapshot_success_resume_and_force():
with tempfile.TemporaryDirectory() as tmp:
transport = SnapshotTransport({
1: view_for(1, view=100, like=10, coin=2, favorite=3,
share=1, reply=4, danmaku=5),
2: view_for(2, view=200, like=20, coin=4, favorite=6,
share=2, reply=8, danmaku=10),
})
db, repo, service = make_service(tmp, transport)
try:
add_video(repo, 1)
add_video(repo, 2)
result = service.run(date=DATE)
assert result.completed is True
assert result.total == 2
assert result.success == 2
assert result.failed == 0
assert result.fans_ok == 1
assert result.fans_failed == 0
assert result.auth_ok is True
assert Path(result.backup_path or "").exists()
row = repo.get_snapshot(DATE, 1)
assert (row["view"], row["like"], row["favorite"], row["coin"],
row["share"], row["reply"], row["danmaku"]) == (100, 10, 3, 2, 1, 4, 5)
assert row["ok"] == 1 and row["src"] == "view"
fan = repo.get_up_stat(DATE, 37754047)
assert fan["follower"] == 2767268 and fan["ok"] == 1
assert repo.kv_get(service.completed_key(DATE)) == "1"
assert repo.kv_get("snapshot:last_completed") == DATE
assert repo.snapshot_counts(DATE) == {"total": 2, "ok": 2, "failed": 0}
# 已完成 -> 不再请求
view_before = transport.count("/web-interface/view")
again = service.run_if_needed(date=DATE)
assert again.already_completed is True
assert transport.count("/web-interface/view") == view_before
# force 重新采集
forced = service.run(date=DATE, force=True)
assert forced.success == 2
assert transport.count("/web-interface/view") == view_before + 2
finally:
db.close()
def test_resume_skips_ok_and_retries_failed():
with tempfile.TemporaryDirectory() as tmp:
transport = SnapshotTransport({
1: view_for(1, view=111),
2: view_for(2, view=222),
})
db, repo, service = make_service(tmp, transport)
try:
add_video(repo, 1)
add_video(repo, 2)
repo.upsert_snapshot(
DATE, 1, metrics={"view": 1, "like": 1, "favorite": 1,
"coin": 1, "share": 1, "reply": 1, "danmaku": 1},
ok=True, err=None, captured_at=1,
)
repo.upsert_snapshot(
DATE, 2, metrics=None, ok=False, err="previous error",
captured_at=1,
)
result = service.run(date=DATE)
assert result.skipped == 1
assert result.success == 1
assert result.completed is True
assert transport.count("/web-interface/view") == 1
assert repo.get_snapshot(DATE, 1)["view"] == 1 # 未被覆盖
assert repo.get_snapshot(DATE, 2)["view"] == 222
finally:
db.close()
def test_unavailable_marks_removed():
with tempfile.TemporaryDirectory() as tmp:
transport = SnapshotTransport({
1: VIEW_UNAVAILABLE,
2: HTTPResponse(404, {}, b"not found"),
})
db, repo, service = make_service(tmp, transport, track_fans=())
try:
add_video(repo, 1)
add_video(repo, 2)
result = service.run(date=DATE)
assert result.removed == 2
assert result.failed == 0
assert result.completed is True
assert repo.get_video(1)["status"] == "removed"
assert repo.get_video(2)["status"] == "removed"
assert repo.get_snapshot(DATE, 1) is None
assert repo.get_snapshot(DATE, 2) is None
finally:
db.close()
def test_other_error_writes_failed_placeholder():
with tempfile.TemporaryDirectory() as tmp:
transport = SnapshotTransport({
1: {"code": -500, "message": "服务异常", "data": None},
})
db, repo, service = make_service(tmp, transport, track_fans=(), retries=3)
try:
add_video(repo, 1)
result = service.run(date=DATE)
assert result.failed == 1
assert result.success == 0
assert result.completed is False
row = repo.get_snapshot(DATE, 1)
assert row["ok"] == 0
assert row["view"] is None and row["like"] is None
assert "code=-500" in row["err"]
assert repo.kv_get(service.completed_key(DATE)) == "0"
# 每次业务失败会走匿名+完整 Cookie 两条请求,服务层重试 3 次
assert transport.count("/web-interface/view") == 6
finally:
db.close()
def test_fan_failure_placeholder():
with tempfile.TemporaryDirectory() as tmp:
transport = SnapshotTransport({}, fan={"code": -500, "message": "x", "data": None})
db, repo, service = make_service(tmp, transport, retries=2)
try:
result = service.run(date=DATE)
assert result.total == 0
assert result.fans_failed == 1
assert result.completed is False
row = repo.get_up_stat(DATE, 37754047)
assert row["ok"] == 0
assert row["follower"] is None and row["following"] is None
finally:
db.close()
def test_auth_invalid_is_warned_but_snapshot_continues():
with tempfile.TemporaryDirectory() as tmp:
stream = io.StringIO()
logger = JsonLogger(stream)
transport = SnapshotTransport(
{}, nav={"code": 0, "message": "0", "ttl": 1,
"data": {"isLogin": False, "mid": 0, "uname": ""}},
fan=RELATION_PAYLOAD,
)
db, repo, service = make_service(tmp, transport, logger=logger)
try:
result = service.run(date=DATE)
assert result.auth_ok is False
assert result.completed is True
assert repo.kv_get("auth:ok") == "0"
events = [json.loads(line) for line in stream.getvalue().splitlines()]
assert any(e["event"] == "auth.invalid" for e in events)
finally:
db.close()
def test_no_cookie_skips_nav_request():
with tempfile.TemporaryDirectory() as tmp:
transport = SnapshotTransport({}, fan=RELATION_PAYLOAD)
db, repo, service = make_service(tmp, transport, cookie="")
try:
result = service.run(date=DATE)
assert result.auth_ok is False
assert result.auth_message == "未配置 cookie"
assert transport.count("/web-interface/nav") == 0
finally:
db.close()
def test_force_failure_clears_last_completed():
with tempfile.TemporaryDirectory() as tmp:
transport = SnapshotTransport({1: view_for(1, view=100)})
db, repo, service = make_service(tmp, transport, track_fans=(), retries=1)
try:
add_video(repo, 1)
first = service.run(date=DATE)
assert first.completed is True
assert repo.kv_get("snapshot:last_completed") == DATE
transport.views[1] = {"code": -500, "message": "x", "data": None}
second = service.run(date=DATE, force=True)
assert second.failed == 1
assert second.completed is False
assert repo.kv_get("snapshot:last_completed") == ""
assert repo.get_snapshot(DATE, 1)["ok"] == 0
finally:
db.close()
def main():
test_snapshot_success_resume_and_force()
test_resume_skips_ok_and_retries_failed()
test_unavailable_marks_removed()
test_other_error_writes_failed_placeholder()
test_fan_failure_placeholder()
test_auth_invalid_is_warned_but_snapshot_continues()
test_no_cookie_skips_nav_request()
test_force_failure_clears_last_completed()
print("OK: 每日快照、断点续跑、失败占位与粉丝采集测试全部通过")
if __name__ == "__main__":
main()
+84
View File
@@ -0,0 +1,84 @@
# -*- coding: utf-8 -*-
"""上海时区与每日时间计算测试: uv run python tests/test_timeutil.py"""
import os
import sys
from datetime import datetime
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
sys.stdout.reconfigure(encoding="utf-8")
from reaction_board.timeutil import ( # noqa: E402
SHANGHAI_TZ,
day_bounds,
is_date,
next_daily_run,
parse_hhmm,
shanghai_date,
shanghai_datetime,
)
def ts(text):
return datetime.strptime(text, "%Y-%m-%d %H:%M:%S").replace(tzinfo=SHANGHAI_TZ).timestamp()
def test_shanghai_date_boundary():
# UTC 2026-10-31 15:59:59 = 上海 2026-10-31 23:59:59
assert shanghai_date(ts("2026-10-31 23:59:59")) == "2026-10-31"
# UTC 2026-10-31 16:00:00 = 上海 2026-11-01 00:00:00
assert shanghai_date(ts("2026-11-01 00:00:00")) == "2026-11-01"
assert shanghai_datetime().tzinfo is not None
def test_parse_hhmm():
assert parse_hhmm("00:01") == (0, 1)
assert parse_hhmm("23:59") == (23, 59)
for bad in ("24:00", "1:01", "00:60", "", "abc"):
try:
parse_hhmm(bad)
except ValueError:
pass
else:
raise AssertionError(f"{bad!r} 应失败")
def test_next_daily_run():
# 当天 00:01 之前 -> 当天 00:01
now = ts("2026-11-01 00:00:10")
nxt = next_daily_run(now, "00:01")
assert nxt == ts("2026-11-01 00:01:00")
# 正好 00:01:00 -> 下一天(严格大于)
assert next_daily_run(ts("2026-11-01 00:01:00"), "00:01") == ts("2026-11-02 00:01:00")
# 当天 00:01 之后 -> 次日
assert next_daily_run(ts("2026-11-01 23:59:59"), "00:01") == ts("2026-11-02 00:01:00")
# 跨月
assert next_daily_run(ts("2026-10-31 12:00:00"), "00:01") == ts("2026-11-01 00:01:00")
def test_day_bounds_and_date_validation():
start, end = day_bounds("2026-11-01")
assert start == ts("2026-11-01 00:00:00")
assert end == ts("2026-11-02 00:00:00")
assert end - start == 86400
assert is_date("2026-11-01")
for bad in ("2026-13-01", "2026-02-30", "2026/11/01", "20261101", ""):
assert not is_date(bad), bad
try:
day_bounds("bad")
except ValueError:
pass
else:
raise AssertionError("非法日期应失败")
def main():
test_shanghai_date_boundary()
test_parse_hhmm()
test_next_daily_run()
test_day_bounds_and_date_validation()
print("OK: 上海时区与每日时间计算测试全部通过")
if __name__ == "__main__":
main()