fix: close data integrity and hot-reload review findings
Backfill same-day snapshots for newly added or restored videos, preserve successful fan snapshots on resume, persist incomplete state before forced re-runs, and split collection completion from backup completion so failed backups can retry without API calls. Reject missing data payloads instead of treating them as empty folders, guard against mass-unfav on suspicious empty remote folders, rebuild Bilibili clients when credentials change, recompute primary folder ownership from active relations with per-folder fav_time, preserve original titles for dropped invalid videos, poll config every 60 seconds, and fix favsort verify interval and mixed resource types.
This commit is contained in:
1 parent
178ab64b18
commit
6a21b5d905
21 files changed
+895
-82
No files matched your search
+23
-4
@@ -281,8 +281,9 @@ PRAGMA synchronous = NORMAL;
|
|||||||
- P0 已实现的命令行:`python app.py init`、`python app.py sync [--full]`、`python app.py status`。
|
- P0 已实现的命令行:`python app.py init`、`python app.py sync [--full]`、`python app.py status`。
|
||||||
|
|
||||||
### 5.2 安全规则- Windows 本地开发时应限制 `config.json` 的文件访问权限;部署到 Linux 时权限必须为 `600`;
|
### 5.2 安全规则- Windows 本地开发时应限制 `config.json` 的文件访问权限;部署到 Linux 时权限必须为 `600`;
|
||||||
- 配置文件每 60 秒检查一次 mtime,收藏夹、分类规则和权重变更后无需重启;
|
- 配置文件每 60 秒检查一次 mtime,凭据、收藏夹、分类规则和权重变更后无需重启;
|
||||||
- Cookie 或 CSRF 改变时重建 B 站请求客户端;
|
- Cookie / CSRF 变化时由 `ClientProvider` 重建 B 站请求客户端,并保留该角色的限速与重试策略;
|
||||||
|
- `schedule.folder_sync_minutes` 与 `schedule.snapshot_at` 在启动时固化,修改后需重启 `run`;
|
||||||
- Telegram 操作必须同时验证 `from.id` 和 `chat.id` 是否为配置的私聊用户;
|
- Telegram 操作必须同时验证 `from.id` 和 `chat.id` 是否为配置的私聊用户;
|
||||||
- 仅允许环境变量临时覆盖敏感配置,不能将真实凭据硬编码进源代码;
|
- 仅允许环境变量临时覆盖敏感配置,不能将真实凭据硬编码进源代码;
|
||||||
- `GET` 路由绝不进行收藏、忽略、拉黑等写操作。
|
- `GET` 路由绝不进行收藏、忽略、拉黑等写操作。
|
||||||
@@ -310,7 +311,8 @@ PRAGMA synchronous = NORMAL;
|
|||||||
- `snapshot` 主键为 `(stat_date, aid)`,保证同日幂等;
|
- `snapshot` 主键为 `(stat_date, aid)`,保证同日幂等;
|
||||||
- `up_stat` 主键为 `(stat_date, mid)`;
|
- `up_stat` 主键为 `(stat_date, mid)`;
|
||||||
- `folder_resource(media_id, aid)` 记录每个收藏夹实际拥有的资源,不能仅依赖 `video.folder_id`;
|
- `folder_resource(media_id, aid)` 记录每个收藏夹实际拥有的资源,不能仅依赖 `video.folder_id`;
|
||||||
- `video.folder_id` 表示视频的主归属夹,用于新视频同 UP 归属和排序;
|
- `folder_resource.fav_time` 记录该视频在该夹的加入时间,用于跨夹主归属选择;
|
||||||
|
- `video.folder_id` 表示视频的主归属夹,按有效收藏关系重算,不依赖收藏夹遍历顺序;
|
||||||
- `up_blacklist` 只在发现任务中生效,收藏夹同步不得用它过滤既有视频;
|
- `up_blacklist` 只在发现任务中生效,收藏夹同步不得用它过滤既有视频;
|
||||||
- `seen_aid` 永久保留搜索去重标记,候选过期删除后也不重复通知;
|
- `seen_aid` 永久保留搜索去重标记,候选过期删除后也不重复通知;
|
||||||
- `fav_action` 保存 Telegram 或签名链接的动作状态,防止重复点击造成重复写入。
|
- `fav_action` 保存 Telegram 或签名链接的动作状态,防止重复点击造成重复写入。
|
||||||
@@ -409,8 +411,11 @@ P1 实现说明:
|
|||||||
|
|
||||||
- 时区使用固定 UTC+8(中国无夏令时),避免 Windows 缺少 `tzdata`;
|
- 时区使用固定 UTC+8(中国无夏令时),避免 Windows 缺少 `tzdata`;
|
||||||
- 当日全部视频与粉丝采集成功后在 `kv` 写入 `snapshot:completed:<date>=1`;启动时未完成会补跑;
|
- 当日全部视频与粉丝采集成功后在 `kv` 写入 `snapshot:completed:<date>=1`;启动时未完成会补跑;
|
||||||
- `ok=0` 的当日记录会在下次补跑时重试;`ok=1` 直接跳过;
|
- 完成标记不是唯一门槛:每次检查 `pending_snapshot_aids(date)`,新入库或从 `dropped` 恢复的视频会立即补采当天快照;
|
||||||
|
- `ok=0` 的当日记录会在下次补跑时重试;`ok=1` 直接跳过;粉丝同样跳过已成功的 `(stat_date, mid)`;
|
||||||
|
- 开始采集前先写 `snapshot:completed:<date>=0`,强制重采中断后不会残留旧的完成标记;
|
||||||
- 每次采集后生成 `data/backup/app-YYYYMMDD.db` 并清理超过 `storage.backup_keep_days` 的日备份;
|
- 每次采集后生成 `data/backup/app-YYYYMMDD.db` 并清理超过 `storage.backup_keep_days` 的日备份;
|
||||||
|
- 采集完成与备份完成分开记录;备份失败时 `snapshot` 返回 1,下次执行只重试备份、不重复请求 B 站;
|
||||||
- Cookie 登录态通过 `x/web-interface/nav` 检查,失效只告警,不阻塞匿名 `view`;
|
- Cookie 登录态通过 `x/web-interface/nav` 检查,失效只告警,不阻塞匿名 `view`;
|
||||||
- Telegram 摘要推迟到 P3,P1 先写结构化 JSON 日志和 `kv` 状态。
|
- Telegram 摘要推迟到 P3,P1 先写结构化 JSON 日志和 `kv` 状态。
|
||||||
|
|
||||||
@@ -589,6 +594,20 @@ uv run python tests/test_moves.py
|
|||||||
- 真实全量联调:795 条视频全部 `ok=1`,0 失败、0 移除,耗时 1124 秒;粉丝 `follower=2767270`;日备份 `data/backup/app-20261031.db` 含 795 条快照;二次执行正确命中完成标记并跳过;
|
- 真实全量联调:795 条视频全部 `ok=1`,0 失败、0 移除,耗时 1124 秒;粉丝 `follower=2767270`;日备份 `data/backup/app-20261031.db` 含 795 条快照;二次执行正确命中完成标记并跳过;
|
||||||
- 自动化测试见 `tests/test_snapshot.py`、`test_maintenance.py`、`test_timeutil.py`,并扩展了 `test_bilibili_client.py` 与 `test_scheduler.py`。
|
- 自动化测试见 `tests/test_snapshot.py`、`test_maintenance.py`、`test_timeutil.py`,并扩展了 `test_bilibili_client.py` 与 `test_scheduler.py`。
|
||||||
|
|
||||||
|
### P1 补丁:数据完整性与热更新修复 ✅
|
||||||
|
|
||||||
|
- 新入库/恢复视频当天补采:`pending_snapshot_aids()` + 同步后立即触发快照;
|
||||||
|
- 凭据热更新:`ClientProvider` 检测 `mid/cookie/csrf` 变化后重建客户端;
|
||||||
|
- 协议防御:`ids` / `infos` 缺失 `data` 或 `data=null` 时中止,不再当作空结果;
|
||||||
|
- 空远端保护:远端 0 条而本地非空时跳过差集,需 `--allow-empty-folder` 才执行清空;
|
||||||
|
- 粉丝续跑:非强制模式跳过已成功的 `(stat_date, mid)`;
|
||||||
|
- 强制重采:开始前先写未完成标记,中断后可续跑;
|
||||||
|
- 备份恢复:采集完成与备份完成分离,备份失败返回 1 并支持无请求重试;
|
||||||
|
- 跨夹主归属:迁移 v2 增加 `folder_resource.fav_time`,按有效关系重算 `video.folder_id`;
|
||||||
|
- `dropped` 视频失效时保留原始标题;
|
||||||
|
- `favsort.py`:`--verify-every` 必须为正数;`sort` 使用真实资源类型,不再硬编码 `2`;
|
||||||
|
- 配置每 60 秒检查 mtime;调度参数变更需重启。
|
||||||
|
|
||||||
### P2:本地看板
|
### P2:本地看板
|
||||||
|
|
||||||
- 总览、榜单、详情、UP、粉丝、候选和系统状态页面;
|
- 总览、榜单、详情、UP、粉丝、候选和系统状态页面;
|
||||||
|
|||||||
@@ -84,7 +84,8 @@ uv run python app.py status
|
|||||||
- `infos` 静默缺失或 `attr=9` 的资源标记为 `removed`,保留历史;
|
- `infos` 静默缺失或 `attr=9` 的资源标记为 `removed`,保留历史;
|
||||||
- 从收藏夹移除的资源标记为 `unfaved`,仍保留数据;
|
- 从收藏夹移除的资源标记为 `unfaved`,仍保留数据;
|
||||||
- 黑名单只作为发现阶段的种子写入,**不会过滤收藏夹同步**;
|
- 黑名单只作为发现阶段的种子写入,**不会过滤收藏夹同步**;
|
||||||
- 收藏夹达到 950 条时预警,系统绝不自动换夹。
|
- 收藏夹达到 950 条时预警,系统绝不自动换夹;
|
||||||
|
- 远端返回 0 条但本地仍有记录时默认按异常跳过差集,确认收藏夹确实被清空后才使用 `--allow-empty-folder`。
|
||||||
|
|
||||||
## P1:每日快照与粉丝追踪
|
## P1:每日快照与粉丝追踪
|
||||||
|
|
||||||
@@ -108,11 +109,13 @@ uv run python app.py auth-check --account all
|
|||||||
|
|
||||||
- `snapshot` 对 `active` / `unfaved` 视频逐条请求 `view`,每条请求间隔 1.0~1.6 秒;
|
- `snapshot` 对 `active` / `unfaved` 视频逐条请求 `view`,每条请求间隔 1.0~1.6 秒;
|
||||||
- 成功写入 `snapshot` 的七项指标;同日 `ok=1` 的记录直接跳过,可断点续跑;
|
- 成功写入 `snapshot` 的七项指标;同日 `ok=1` 的记录直接跳过,可断点续跑;
|
||||||
|
- 当天已完成后新入库或从 `dropped` 恢复的视频会立即补采当天快照,不必等到次日;
|
||||||
- 视频不可访问(`-404` / `62002` / `62012`)标记为 `removed`;
|
- 视频不可访问(`-404` / `62002` / `62012`)标记为 `removed`;
|
||||||
- 其他错误重试 3 次,仍失败写入 `ok=0` 和错误信息,不写伪造的零值;
|
- 其他错误重试 3 次,仍失败写入 `ok=0` 和错误信息,不写伪造的零值;
|
||||||
- 粉丝数写入 `up_stat`,失败同样写 `ok=0`;
|
- 粉丝数写入 `up_stat`,失败同样写 `ok=0`;续跑时已成功的 `(stat_date, mid)` 会跳过,不会被失败结果覆盖;
|
||||||
- 当日全部视频与粉丝采集成功后写入完成标记;服务启动时若当天未完成会自动补跑;
|
- 当日全部视频与粉丝采集成功后写入完成标记;服务启动时若当天未完成会自动补跑;
|
||||||
- 每次采集后生成 `data/backup/app-YYYYMMDD.db`,按 `storage.backup_keep_days` 清理旧日备份;
|
- 每次采集后生成 `data/backup/app-YYYYMMDD.db`,按 `storage.backup_keep_days` 清理旧日备份;
|
||||||
|
- 备份失败时 `snapshot` 返回 1 便于任务告警;再次执行只重试备份,不重复请求 B 站;
|
||||||
- Cookie 失效只告警,不阻塞匿名 `view` 采集;P0/P1 只使用 `bilibili.read`。
|
- Cookie 失效只告警,不阻塞匿名 `view` 采集;P0/P1 只使用 `bilibili.read`。
|
||||||
|
|
||||||
首次真实全量采集结果:795 条视频全部成功、0 失败、0 移除,耗时约 19 分钟;粉丝 `follower=2767270`;日备份为 `data/backup/app-20261031.db`。
|
首次真实全量采集结果:795 条视频全部成功、0 失败、0 移除,耗时约 19 分钟;粉丝 `follower=2767270`;日备份为 `data/backup/app-20261031.db`。
|
||||||
@@ -146,6 +149,8 @@ Copy-Item config.example.json config.json
|
|||||||
- `storage`:数据库与备份目录;
|
- `storage`:数据库与备份目录;
|
||||||
- `schedule` / `notify` / `classify` / `heat_weights`:任务节奏、通知阈值、分类规则和热度权重。
|
- `schedule` / `notify` / `classify` / `heat_weights`:任务节奏、通知阈值、分类规则和热度权重。
|
||||||
|
|
||||||
|
配置每 60 秒检查一次 mtime,凭据、收藏夹、分类规则等变更无需重启即可在下次任务读取;但 `schedule.folder_sync_minutes` 与 `schedule.snapshot_at` 在启动时固化,修改后需重启 `run` 进程。
|
||||||
|
|
||||||
完整配置口径以 [project.md](project.md) 和 [DEVELOPMENT.md](DEVELOPMENT.md) 为准。
|
完整配置口径以 [project.md](project.md) 和 [DEVELOPMENT.md](DEVELOPMENT.md) 为准。
|
||||||
|
|
||||||
## 当前可用:收藏夹排序 CLI
|
## 当前可用:收藏夹排序 CLI
|
||||||
@@ -243,10 +248,12 @@ uv run python tests/test_database.py # 单写队列、事务与并发
|
|||||||
uv run python tests/test_repository.py # 收藏夹/视频状态机与黑名单
|
uv run python tests/test_repository.py # 收藏夹/视频状态机与黑名单
|
||||||
uv run python tests/test_bilibili_client.py # ids/infos/view/fans/nav 契约与重试
|
uv run python tests/test_bilibili_client.py # ids/infos/view/fans/nav 契约与重试
|
||||||
uv run python tests/test_folder_sync.py # 首次导入、差集、失效识别、容量预警
|
uv run python tests/test_folder_sync.py # 首次导入、差集、失效识别、容量预警
|
||||||
uv run python tests/test_snapshot.py # 七项快照、断点续跑、失败占位、粉丝
|
uv run python tests/test_snapshot.py # 七项快照、补采、断点续跑、粉丝、备份重试
|
||||||
uv run python tests/test_maintenance.py # 日备份与保留期
|
uv run python tests/test_maintenance.py # 日备份与保留期
|
||||||
uv run python tests/test_timeutil.py # 上海时区与每日任务时间
|
uv run python tests/test_timeutil.py # 上海时区与每日任务时间
|
||||||
uv run python tests/test_scheduler.py # 内置调度器到期/失败隔离
|
uv run python tests/test_scheduler.py # 内置调度器到期/失败隔离
|
||||||
|
uv run python tests/test_provider.py # 凭据变化重建客户端与配置 revision
|
||||||
|
uv run python tests/test_favsort_run.py # 排序参数校验与混合资源类型
|
||||||
uv run python tests/test_logging.py # JSON 日志脱敏
|
uv run python tests/test_logging.py # JSON 日志脱敏
|
||||||
uv run python tests/test_cli.py # CLI 离线路径
|
uv run python tests/test_cli.py # CLI 离线路径
|
||||||
```
|
```
|
||||||
|
|||||||
+23
-2
@@ -346,6 +346,7 @@ def cmd_plan(cfg, args):
|
|||||||
"current_order": cur,
|
"current_order": cur,
|
||||||
"target_order": tgt,
|
"target_order": tgt,
|
||||||
"moves": moves,
|
"moves": moves,
|
||||||
|
"types": {v["avid"]: v["type"] for v in applied},
|
||||||
"titles": {
|
"titles": {
|
||||||
v["avid"]: {
|
v["avid"]: {
|
||||||
"title": v["title"],
|
"title": v["title"],
|
||||||
@@ -395,8 +396,12 @@ def cmd_run(cfg, args):
|
|||||||
plan = load_plan(media_id)
|
plan = load_plan(media_id)
|
||||||
target = plan["target_order"]
|
target = plan["target_order"]
|
||||||
titles = plan["titles"]
|
titles = plan["titles"]
|
||||||
|
types = plan.get("types") or {}
|
||||||
sc = cfg["sleep"]
|
sc = cfg["sleep"]
|
||||||
verify_every = args.verify_every
|
verify_every = args.verify_every
|
||||||
|
if verify_every < 1:
|
||||||
|
print("[run] --verify-every 必须 >= 1;未执行任何写操作")
|
||||||
|
return 2
|
||||||
|
|
||||||
print("[run] 读取当前顺序 ...")
|
print("[run] 读取当前顺序 ...")
|
||||||
actual = [x["id"] for x in fetch_ids(cfg, media_id)]
|
actual = [x["id"] for x in fetch_ids(cfg, media_id)]
|
||||||
@@ -428,7 +433,13 @@ def cmd_run(cfg, args):
|
|||||||
long_after = random.randint(sc["long_every_min"], sc["long_every_max"])
|
long_after = random.randint(sc["long_every_min"], sc["long_every_max"])
|
||||||
|
|
||||||
for i, m in enumerate(moves, 1):
|
for i, m in enumerate(moves, 1):
|
||||||
body = {"media_id": media_id, "sort": f"{m['pred']}:2:{m['moved']}:2", "csrf": cfg["csrf"]}
|
pred_type = types.get(m["pred"], 2)
|
||||||
|
moved_type = types.get(m["moved"], 2)
|
||||||
|
body = {
|
||||||
|
"media_id": media_id,
|
||||||
|
"sort": f"{m['pred']}:{pred_type}:{m['moved']}:{moved_type}",
|
||||||
|
"csrf": cfg["csrf"],
|
||||||
|
}
|
||||||
print(f"[run] {i}/{total} {titles[str(m['moved'])]['title'][:30]} "
|
print(f"[run] {i}/{total} {titles[str(m['moved'])]['title'][:30]} "
|
||||||
f"-> after {m['pred']}", end=" ", flush=True)
|
f"-> after {m['pred']}", end=" ", flush=True)
|
||||||
resp = post_form(cfg, API["sort"].format(base=BASE), body)
|
resp = post_form(cfg, API["sort"].format(base=BASE), body)
|
||||||
@@ -541,6 +552,16 @@ COMMAND_HELP = {
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _positive_int(value):
|
||||||
|
try:
|
||||||
|
number = int(value)
|
||||||
|
except (TypeError, ValueError) as exc:
|
||||||
|
raise argparse.ArgumentTypeError("必须是整数") from exc
|
||||||
|
if number < 1:
|
||||||
|
raise argparse.ArgumentTypeError("必须 >= 1")
|
||||||
|
return number
|
||||||
|
|
||||||
|
|
||||||
def build_parser():
|
def build_parser():
|
||||||
p = argparse.ArgumentParser(
|
p = argparse.ArgumentParser(
|
||||||
prog="favsort",
|
prog="favsort",
|
||||||
@@ -565,7 +586,7 @@ def build_parser():
|
|||||||
|
|
||||||
pr = sub.add_parser("run", help=COMMAND_HELP["run"])
|
pr = sub.add_parser("run", help=COMMAND_HELP["run"])
|
||||||
pr.add_argument("media_id", type=int, help="收藏夹 media_id")
|
pr.add_argument("media_id", type=int, help="收藏夹 media_id")
|
||||||
pr.add_argument("--verify-every", type=int, default=1, metavar="N",
|
pr.add_argument("--verify-every", type=_positive_int, default=1, metavar="N",
|
||||||
help="每 N 步回读校验一次(默认 1;前 3 步始终逐步校验)")
|
help="每 N 步回读校验一次(默认 1;前 3 步始终逐步校验)")
|
||||||
|
|
||||||
pv = sub.add_parser("verify", help=COMMAND_HELP["verify"])
|
pv = sub.add_parser("verify", help=COMMAND_HELP["verify"])
|
||||||
|
|||||||
+7
-1
@@ -129,7 +129,7 @@
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
- 支持热加载(`SIGHUP` 或每 60 秒检查 mtime),改收藏夹/黑名单不用重启;
|
- 支持热加载(每 60 秒检查 mtime),改收藏夹/黑名单不用重启;`schedule` 中的周期与每日时间需重启生效;
|
||||||
- **黑名单本身放数据库**(`up_blacklist` 表),因为要从通知里动态拉黑;配置文件只做初始种子;
|
- **黑名单本身放数据库**(`up_blacklist` 表),因为要从通知里动态拉黑;配置文件只做初始种子;
|
||||||
- 日志里**永不打印 token / cookie**,启动时只打印"已配置/未配置";
|
- 日志里**永不打印 token / cookie**,启动时只打印"已配置/未配置";
|
||||||
- 容器里把配置目录挂成**可写卷**(`-v ./config:/config`,权限 600),因为 `auth` 扫码续期后要回写 cookie;
|
- 容器里把配置目录挂成**可写卷**(`-v ./config:/config`,权限 600),因为 `auth` 扫码续期后要回写 cookie;
|
||||||
@@ -332,6 +332,7 @@ for folder in 配置的年度夹(2025、2026、…):
|
|||||||
- 被用户手动"移出库"(`dropped`)的视频若又出现在收藏夹(`diff_add`)→ **自动恢复为 `active`**,并补当天快照;这是唯一会"复活"的状态。
|
- 被用户手动"移出库"(`dropped`)的视频若又出现在收藏夹(`diff_add`)→ **自动恢复为 `active`**,并补当天快照;这是唯一会"复活"的状态。
|
||||||
- 监控的是**所有配置的年度夹**(2025、2026…),所以人工把新视频加进旧夹也能被发现;
|
- 监控的是**所有配置的年度夹**(2025、2026…),所以人工把新视频加进旧夹也能被发现;
|
||||||
- `video.folder_id` 以"实际所在夹"为准,若同一 aid 同时出现在多个夹(极少见),取最新加入的那个并告警。
|
- `video.folder_id` 以"实际所在夹"为准,若同一 aid 同时出现在多个夹(极少见),取最新加入的那个并告警。
|
||||||
|
- 上述 `enqueue_snapshot_today` 已实现为 `pending_snapshot_aids()` 检查:完成标记为 1 时仍会为新入库/恢复视频补采当天快照。
|
||||||
|
|
||||||
### 4.3 首次导入
|
### 4.3 首次导入
|
||||||
|
|
||||||
@@ -376,6 +377,8 @@ for folder in 配置的年度夹(2025、2026、…):
|
|||||||
- 唯一键 `(stat_date, aid)`,`INSERT ... ON CONFLICT DO UPDATE`;
|
- 唯一键 `(stat_date, aid)`,`INSERT ... ON CONFLICT DO UPDATE`;
|
||||||
- `stat_date` 用本地日期,不用 UTC(避免 00:01 落到前一天);
|
- `stat_date` 用本地日期,不用 UTC(避免 00:01 落到前一天);
|
||||||
- 失败重试 3 次后写 `ok=0` 占位,**不写假数据**;看板显示"缺失"而不是 0;
|
- 失败重试 3 次后写 `ok=0` 占位,**不写假数据**;看板显示"缺失"而不是 0;
|
||||||
|
- 完成标记之外还要检查当天待采集合:新入库/恢复视频立即补采,`ok=0` 项在续跑时重试;
|
||||||
|
- 强制重采开始前先写未完成标记,中断后仍可续跑;
|
||||||
- 日增量查询时用 `LAG()` 跳过缺失日,不插值。
|
- 日增量查询时用 `LAG()` 跳过缺失日,不插值。
|
||||||
|
|
||||||
### 5.4 粉丝数追踪(新增)
|
### 5.4 粉丝数追踪(新增)
|
||||||
@@ -388,6 +391,7 @@ for folder in 配置的年度夹(2025、2026、…):
|
|||||||
|
|
||||||
- 接口实测:`{"mid":37754047,"following":95,"follower":2767264}`,**匿名`code=0`**,无需登录;
|
- 接口实测:`{"mid":37754047,"following":95,"follower":2767264}`,**匿名`code=0`**,无需登录;
|
||||||
- 唯一键 `(stat_date, mid)`,同样幂等;
|
- 唯一键 `(stat_date, mid)`,同样幂等;
|
||||||
|
- 续跑时已成功的 `(stat_date, mid)` 会跳过,不会因后续失败覆盖为 NULL;
|
||||||
- 只加 1 次请求/天,成本可忽略;
|
- 只加 1 次请求/天,成本可忽略;
|
||||||
- 支持配置多个 mid(`track_fans: [37754047]`),但看板默认只突出咻咻满;
|
- 支持配置多个 mid(`track_fans: [37754047]`),但看板默认只突出咻咻满;
|
||||||
- 粉丝日增 = 今日 - 昨日,缺失日不插值,曲线断开。
|
- 粉丝日增 = 今日 - 昨日,缺失日不插值,曲线断开。
|
||||||
@@ -783,6 +787,8 @@ storage:SQLite WAL,单写线程
|
|||||||
| 26 | `x/relation/stat` 粉丝数 | `code=0`,匿名可用,实测 `follower=2767268` |
|
| 26 | `x/relation/stat` 粉丝数 | `code=0`,匿名可用,实测 `follower=2767268` |
|
||||||
| 27 | `x/web-interface/nav` 登录态 | `code=0`、`isLogin=true`、mid=28802351 |
|
| 27 | `x/web-interface/nav` 登录态 | `code=0`、`isLogin=true`、mid=28802351 |
|
||||||
| 28 | **P1 真实全量快照** | 795 条全部 `ok=1`、0 失败、0 移除;耗时 1124s;粉丝 `follower=2767270`;日备份 `app-20261031.db` 含 795 条快照 |
|
| 28 | **P1 真实全量快照** | 795 条全部 `ok=1`、0 失败、0 移除;耗时 1124s;粉丝 `follower=2767270`;日备份 `app-20261031.db` 含 795 条快照 |
|
||||||
|
| 29 | **空收藏夹 `ids` 契约** | `4413147139` 返回 `{"code":0,"data":[]}`,`data` 键存在且为数组;缺失/`null` 按协议异常处理 |
|
||||||
|
| 30 | **迁移 v2 与增量同步** | 真实库升级到 schema 2,自动生成 `app-pre-v1-*.db`;同步 167+664 条无新增/移除,主归属无矛盾 |
|
||||||
|
|
||||||
**未验证(需授权或外部配合)**:
|
**未验证(需授权或外部配合)**:
|
||||||
|
|
||||||
|
|||||||
@@ -262,6 +262,20 @@ class BilibiliClient:
|
|||||||
raise self._api_error(payload)
|
raise self._api_error(payload)
|
||||||
return payload
|
return payload
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _data_list(payload: Mapping[str, Any], name: str) -> list[Any]:
|
||||||
|
"""区分“真实空数组”与“协议缺字段”。
|
||||||
|
|
||||||
|
实测空收藏夹返回 ``{"code":0,"data":[]}``;缺少 ``data`` 或为 ``null``
|
||||||
|
都不是已知契约,必须中止,避免被误判成空收藏夹后批量改状态。
|
||||||
|
"""
|
||||||
|
if "data" not in payload:
|
||||||
|
raise BilibiliProtocolError(f"{name} 响应缺少 data 字段")
|
||||||
|
data = payload["data"]
|
||||||
|
if not isinstance(data, list):
|
||||||
|
raise BilibiliProtocolError(f"{name} data 不是数组")
|
||||||
|
return data
|
||||||
|
|
||||||
# ------------------------------------------------------------ fav
|
# ------------------------------------------------------------ fav
|
||||||
|
|
||||||
def fav_ids(self, media_id: int) -> list[FavId]:
|
def fav_ids(self, media_id: int) -> list[FavId]:
|
||||||
@@ -269,11 +283,7 @@ class BilibiliClient:
|
|||||||
f"{BASE}/x/v3/fav/resource/ids?media_id={int(media_id)}&platform=web"
|
f"{BASE}/x/v3/fav/resource/ids?media_id={int(media_id)}&platform=web"
|
||||||
)
|
)
|
||||||
payload = self.get_json(url)
|
payload = self.get_json(url)
|
||||||
data = payload.get("data")
|
data = self._data_list(payload, "fav/resource/ids")
|
||||||
if data is None:
|
|
||||||
data = []
|
|
||||||
if not isinstance(data, list):
|
|
||||||
raise BilibiliProtocolError("fav/resource/ids data 不是数组")
|
|
||||||
out: list[FavId] = []
|
out: list[FavId] = []
|
||||||
for index, item in enumerate(data):
|
for index, item in enumerate(data):
|
||||||
if not isinstance(item, Mapping):
|
if not isinstance(item, Mapping):
|
||||||
@@ -320,11 +330,7 @@ class BilibiliClient:
|
|||||||
f"&platform=web&folder_mid={fmid}&folder_id={int(media_id)}"
|
f"&platform=web&folder_mid={fmid}&folder_id={int(media_id)}"
|
||||||
)
|
)
|
||||||
payload = self.get_json(url)
|
payload = self.get_json(url)
|
||||||
data = payload.get("data")
|
data = self._data_list(payload, "fav/resource/infos")
|
||||||
if data is None:
|
|
||||||
data = []
|
|
||||||
if not isinstance(data, list):
|
|
||||||
raise BilibiliProtocolError("fav/resource/infos data 不是数组")
|
|
||||||
chunk_set = set(chunk)
|
chunk_set = set(chunk)
|
||||||
for item in data:
|
for item in data:
|
||||||
if not isinstance(item, Mapping):
|
if not isinstance(item, Mapping):
|
||||||
|
|||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""按账号角色提供 B 站客户端,并在凭据变化时自动重建。"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any, Callable
|
||||||
|
|
||||||
|
from ..config import ConfigStore
|
||||||
|
from .client import BilibiliClient
|
||||||
|
|
||||||
|
|
||||||
|
class ClientProvider:
|
||||||
|
"""缓存客户端;仅当 ``mid`` / ``cookie`` / ``csrf`` 变化时重建。
|
||||||
|
|
||||||
|
``factory`` 负责固化该角色的限速与重试策略,因此重建不会丢失这些设置。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
store: ConfigStore,
|
||||||
|
role: str,
|
||||||
|
factory: Callable[[Any], BilibiliClient],
|
||||||
|
) -> None:
|
||||||
|
if role not in {"read", "write"}:
|
||||||
|
raise ValueError(f"未知账号角色: {role}")
|
||||||
|
self.store = store
|
||||||
|
self.role = role
|
||||||
|
self.factory = factory
|
||||||
|
self._key: tuple[int, str, str] | None = None
|
||||||
|
self._client: BilibiliClient | None = None
|
||||||
|
|
||||||
|
def get(self) -> BilibiliClient:
|
||||||
|
credentials = getattr(self.store.get().bilibili, self.role)
|
||||||
|
key = (credentials.mid, credentials.cookie, credentials.csrf)
|
||||||
|
if self._client is None or key != self._key:
|
||||||
|
self._client = self.factory(credentials)
|
||||||
|
self._key = key
|
||||||
|
return self._client
|
||||||
+58
-12
@@ -21,6 +21,7 @@ from typing import Sequence
|
|||||||
|
|
||||||
from .bilibili.auth_flow import AuthService
|
from .bilibili.auth_flow import AuthService
|
||||||
from .bilibili.client import BilibiliClient, Pacer
|
from .bilibili.client import BilibiliClient, Pacer
|
||||||
|
from .bilibili.provider import ClientProvider
|
||||||
from .config import ConfigError, ConfigStore
|
from .config import ConfigError, ConfigStore
|
||||||
from .database import Database
|
from .database import Database
|
||||||
from .logging_json import JsonLogger
|
from .logging_json import JsonLogger
|
||||||
@@ -53,6 +54,11 @@ def _build_parser() -> argparse.ArgumentParser:
|
|||||||
sync.add_argument(
|
sync.add_argument(
|
||||||
"--full", action="store_true", help="强制全量刷新元数据(忽略 6 小时周期)"
|
"--full", action="store_true", help="强制全量刷新元数据(忽略 6 小时周期)"
|
||||||
)
|
)
|
||||||
|
sync.add_argument(
|
||||||
|
"--allow-empty-folder",
|
||||||
|
action="store_true",
|
||||||
|
help="允许把远端 0 条视为真实清空(默认按异常跳过差集)",
|
||||||
|
)
|
||||||
snapshot = sub.add_parser(
|
snapshot = sub.add_parser(
|
||||||
"snapshot", parents=[common], help="采集当日七项快照与粉丝数"
|
"snapshot", parents=[common], help="采集当日七项快照与粉丝数"
|
||||||
)
|
)
|
||||||
@@ -72,6 +78,11 @@ def _build_parser() -> argparse.ArgumentParser:
|
|||||||
run.add_argument(
|
run.add_argument(
|
||||||
"--once", action="store_true", help="只执行一次到期任务后退出"
|
"--once", action="store_true", help="只执行一次到期任务后退出"
|
||||||
)
|
)
|
||||||
|
run.add_argument(
|
||||||
|
"--allow-empty-folder",
|
||||||
|
action="store_true",
|
||||||
|
help="允许把远端 0 条视为真实清空(默认按异常跳过差集)",
|
||||||
|
)
|
||||||
sub.add_parser("status", parents=[common], help="查看数据库状态(离线)")
|
sub.add_parser("status", parents=[common], help="查看数据库状态(离线)")
|
||||||
return parser
|
return parser
|
||||||
|
|
||||||
@@ -130,7 +141,13 @@ def cmd_sync(args, logger: JsonLogger) -> int:
|
|||||||
try:
|
try:
|
||||||
repo = Repository(db)
|
repo = Repository(db)
|
||||||
client = _client(cfg.bilibili.read, logger)
|
client = _client(cfg.bilibili.read, logger)
|
||||||
service = FolderSyncService(store, repo, client, logger=logger)
|
service = FolderSyncService(
|
||||||
|
store,
|
||||||
|
repo,
|
||||||
|
client,
|
||||||
|
logger=logger,
|
||||||
|
allow_empty_folders=args.allow_empty_folder,
|
||||||
|
)
|
||||||
summary = service.run(full_refresh=args.full)
|
summary = service.run(full_refresh=args.full)
|
||||||
payload = summary.as_dict()
|
payload = summary.as_dict()
|
||||||
payload["database"] = str(cfg.storage.database_path)
|
payload["database"] = str(cfg.storage.database_path)
|
||||||
@@ -157,7 +174,7 @@ def cmd_snapshot(args, logger: JsonLogger) -> int:
|
|||||||
payload["snapshot_counts"] = repo.snapshot_counts(summary.stat_date)
|
payload["snapshot_counts"] = repo.snapshot_counts(summary.stat_date)
|
||||||
payload["counts"] = repo.counts()
|
payload["counts"] = repo.counts()
|
||||||
_print(payload)
|
_print(payload)
|
||||||
return 0 if summary.completed else 1
|
return 0 if summary.completed and not summary.backup_error else 1
|
||||||
finally:
|
finally:
|
||||||
db.close()
|
db.close()
|
||||||
|
|
||||||
@@ -205,16 +222,26 @@ def cmd_run(args, logger: JsonLogger) -> int:
|
|||||||
last: dict[str, object] = {}
|
last: dict[str, object] = {}
|
||||||
try:
|
try:
|
||||||
repo = Repository(db)
|
repo = Repository(db)
|
||||||
folder_client = _client(cfg.bilibili.read, logger)
|
folder_provider = ClientProvider(
|
||||||
folder_service = FolderSyncService(store, repo, folder_client, logger=logger)
|
store, "read", lambda credentials: _client(credentials, logger)
|
||||||
snapshot_client = _client(cfg.bilibili.read, logger, pacer=Pacer(1.0, 1.6), retries=1)
|
)
|
||||||
snapshot_service = SnapshotService(store, repo, snapshot_client, logger=logger)
|
snapshot_provider = ClientProvider(
|
||||||
|
store,
|
||||||
def run_sync():
|
"read",
|
||||||
summary = folder_service.run()
|
lambda credentials: _client(
|
||||||
last["folder"] = summary
|
credentials, logger, pacer=Pacer(1.0, 1.6), retries=1
|
||||||
logger.info("folder_sync.summary", **summary.as_dict())
|
),
|
||||||
return summary
|
)
|
||||||
|
folder_service = FolderSyncService(
|
||||||
|
store,
|
||||||
|
repo,
|
||||||
|
client_provider=folder_provider,
|
||||||
|
logger=logger,
|
||||||
|
allow_empty_folders=args.allow_empty_folder,
|
||||||
|
)
|
||||||
|
snapshot_service = SnapshotService(
|
||||||
|
store, repo, client_provider=snapshot_provider, logger=logger
|
||||||
|
)
|
||||||
|
|
||||||
def run_snapshot():
|
def run_snapshot():
|
||||||
summary = snapshot_service.run_if_needed()
|
summary = snapshot_service.run_if_needed()
|
||||||
@@ -222,6 +249,22 @@ def cmd_run(args, logger: JsonLogger) -> int:
|
|||||||
logger.info("snapshot.summary", **summary.as_dict())
|
logger.info("snapshot.summary", **summary.as_dict())
|
||||||
return summary
|
return summary
|
||||||
|
|
||||||
|
def run_sync():
|
||||||
|
summary = folder_service.run()
|
||||||
|
last["folder"] = summary
|
||||||
|
logger.info("folder_sync.summary", **summary.as_dict())
|
||||||
|
# 新入库/恢复的视频立即补当天快照,而不是等到次日。
|
||||||
|
if summary.added > 0:
|
||||||
|
run_snapshot()
|
||||||
|
return summary
|
||||||
|
|
||||||
|
def run_config_watch():
|
||||||
|
before = store.revision
|
||||||
|
store.get()
|
||||||
|
if store.revision != before:
|
||||||
|
logger.info("config.reloaded", revision=store.revision)
|
||||||
|
return {"revision": store.revision}
|
||||||
|
|
||||||
hh, mm = parse_hhmm(cfg.schedule.snapshot_at)
|
hh, mm = parse_hhmm(cfg.schedule.snapshot_at)
|
||||||
snapshot_at = f"{hh:02d}:{mm:02d}"
|
snapshot_at = f"{hh:02d}:{mm:02d}"
|
||||||
scheduler = Scheduler(
|
scheduler = Scheduler(
|
||||||
@@ -238,6 +281,7 @@ def cmd_run(args, logger: JsonLogger) -> int:
|
|||||||
next_run=0.0,
|
next_run=0.0,
|
||||||
next_run_fn=lambda started: next_daily_run(started, snapshot_at),
|
next_run_fn=lambda started: next_daily_run(started, snapshot_at),
|
||||||
),
|
),
|
||||||
|
ScheduledTask("config_watch", 60.0, run_config_watch),
|
||||||
],
|
],
|
||||||
logger=logger,
|
logger=logger,
|
||||||
)
|
)
|
||||||
@@ -275,6 +319,8 @@ def cmd_status(args, logger: JsonLogger) -> int:
|
|||||||
"last_completed": repo.kv_get("snapshot:last_completed"),
|
"last_completed": repo.kv_get("snapshot:last_completed"),
|
||||||
"auth_ok": repo.kv_get("auth:ok"),
|
"auth_ok": repo.kv_get("auth:ok"),
|
||||||
"auth_message": repo.kv_get("auth:message"),
|
"auth_message": repo.kv_get("auth:message"),
|
||||||
|
"backup_ok": repo.kv_get(f"snapshot:backup:{today}"),
|
||||||
|
"backup_error": repo.kv_get(f"snapshot:backup_error:{today}"),
|
||||||
})
|
})
|
||||||
_print({
|
_print({
|
||||||
"ok": True,
|
"ok": True,
|
||||||
|
|||||||
@@ -478,6 +478,7 @@ class ConfigStore:
|
|||||||
self._config = load_config(path, env=env, require_auth=require_auth)
|
self._config = load_config(path, env=env, require_auth=require_auth)
|
||||||
self._mtime_ns = self._read_mtime_ns()
|
self._mtime_ns = self._read_mtime_ns()
|
||||||
self.last_error: str | None = None
|
self.last_error: str | None = None
|
||||||
|
self.revision = 0
|
||||||
|
|
||||||
def _read_mtime_ns(self) -> int | None:
|
def _read_mtime_ns(self) -> int | None:
|
||||||
try:
|
try:
|
||||||
@@ -499,4 +500,5 @@ class ConfigStore:
|
|||||||
self.last_error = str(exc)
|
self.last_error = str(exc)
|
||||||
return self._config
|
return self._config
|
||||||
self.last_error = None
|
self.last_error = None
|
||||||
|
self.revision += 1
|
||||||
return self._config
|
return self._config
|
||||||
@@ -167,6 +167,13 @@ MIGRATIONS: tuple[Migration, ...] = (
|
|||||||
""",
|
""",
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
Migration(
|
||||||
|
version=2,
|
||||||
|
name="folder_resource fav_time",
|
||||||
|
statements=(
|
||||||
|
"ALTER TABLE folder_resource ADD COLUMN fav_time INTEGER",
|
||||||
|
),
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -166,6 +166,35 @@ class Repository:
|
|||||||
|
|
||||||
# ------------------------------------------------------------ video
|
# ------------------------------------------------------------ video
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _recompute_primary_folder(conn: sqlite3.Connection, aid: int) -> int | None:
|
||||||
|
"""按有效收藏关系重算主归属,避免依赖收藏夹遍历顺序。
|
||||||
|
|
||||||
|
优先使用逐夹 ``fav_time``(后加入者优先),其次 ``first_seen``,
|
||||||
|
最后用年度与 media_id 做确定性决胜。
|
||||||
|
"""
|
||||||
|
row = conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT fr.media_id
|
||||||
|
FROM folder_resource fr
|
||||||
|
JOIN folder f ON f.media_id = fr.media_id
|
||||||
|
WHERE fr.aid = ? AND fr.active = 1 AND f.enabled = 1
|
||||||
|
ORDER BY COALESCE(fr.fav_time, 0) DESC,
|
||||||
|
fr.first_seen DESC,
|
||||||
|
f.year DESC,
|
||||||
|
fr.media_id DESC
|
||||||
|
LIMIT 1
|
||||||
|
""",
|
||||||
|
(int(aid),),
|
||||||
|
).fetchone()
|
||||||
|
if row is None:
|
||||||
|
return None
|
||||||
|
folder_id = int(row["media_id"])
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE video SET folder_id = ? WHERE aid = ?", (folder_id, int(aid))
|
||||||
|
)
|
||||||
|
return folder_id
|
||||||
|
|
||||||
def upsert_from_fav(
|
def upsert_from_fav(
|
||||||
self,
|
self,
|
||||||
resource,
|
resource,
|
||||||
@@ -206,7 +235,8 @@ class Repository:
|
|||||||
ON CONFLICT(aid) DO UPDATE SET
|
ON CONFLICT(aid) DO UPDATE SET
|
||||||
bvid = COALESCE(excluded.bvid, video.bvid),
|
bvid = COALESCE(excluded.bvid, video.bvid),
|
||||||
title = CASE
|
title = CASE
|
||||||
WHEN excluded.status = 'removed' AND video.title != ''
|
WHEN (excluded.attr = 9 OR excluded.status = 'removed')
|
||||||
|
AND video.title != ''
|
||||||
THEN video.title
|
THEN video.title
|
||||||
WHEN excluded.title != '' THEN excluded.title
|
WHEN excluded.title != '' THEN excluded.title
|
||||||
ELSE video.title
|
ELSE video.title
|
||||||
@@ -219,7 +249,10 @@ class Repository:
|
|||||||
END,
|
END,
|
||||||
duration = COALESCE(excluded.duration, video.duration),
|
duration = COALESCE(excluded.duration, video.duration),
|
||||||
cover = COALESCE(excluded.cover, video.cover),
|
cover = COALESCE(excluded.cover, video.cover),
|
||||||
folder_id = excluded.folder_id,
|
folder_id = CASE
|
||||||
|
WHEN video.folder_id IS NULL THEN excluded.folder_id
|
||||||
|
ELSE video.folder_id
|
||||||
|
END,
|
||||||
fav_time = COALESCE(excluded.fav_time, video.fav_time),
|
fav_time = COALESCE(excluded.fav_time, video.fav_time),
|
||||||
status = excluded.status,
|
status = excluded.status,
|
||||||
source = COALESCE(video.source, excluded.source),
|
source = COALESCE(video.source, excluded.source),
|
||||||
@@ -246,19 +279,22 @@ class Repository:
|
|||||||
)
|
)
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"""
|
"""
|
||||||
INSERT INTO folder_resource(media_id, aid, first_seen, active, last_seen)
|
INSERT INTO folder_resource(media_id, aid, first_seen, active,
|
||||||
VALUES (?, ?, ?, 1, ?)
|
last_seen, fav_time)
|
||||||
|
VALUES (?, ?, ?, 1, ?, ?)
|
||||||
ON CONFLICT(media_id, aid) DO UPDATE SET
|
ON CONFLICT(media_id, aid) DO UPDATE SET
|
||||||
active = 1,
|
active = 1,
|
||||||
last_seen = excluded.last_seen
|
last_seen = excluded.last_seen,
|
||||||
|
fav_time = COALESCE(excluded.fav_time, folder_resource.fav_time)
|
||||||
""",
|
""",
|
||||||
(folder_id, resource.aid, now, now),
|
(folder_id, resource.aid, now, now, resource.fav_time),
|
||||||
)
|
)
|
||||||
|
primary = self._recompute_primary_folder(conn, resource.aid)
|
||||||
return UpsertOutcome(
|
return UpsertOutcome(
|
||||||
aid=resource.aid,
|
aid=resource.aid,
|
||||||
status=status,
|
status=status,
|
||||||
previous_folder_id=previous_folder,
|
previous_folder_id=previous_folder,
|
||||||
moved_folder=previous_folder is not None and previous_folder != folder_id,
|
moved_folder=previous_folder is not None and previous_folder != primary,
|
||||||
)
|
)
|
||||||
return self.db.write(run)
|
return self.db.write(run)
|
||||||
|
|
||||||
@@ -304,6 +340,7 @@ class Repository:
|
|||||||
""",
|
""",
|
||||||
(folder_id, aid, now, now),
|
(folder_id, aid, now, now),
|
||||||
)
|
)
|
||||||
|
self._recompute_primary_folder(conn, aid)
|
||||||
return status
|
return status
|
||||||
return self.db.write(run)
|
return self.db.write(run)
|
||||||
|
|
||||||
@@ -320,6 +357,7 @@ class Repository:
|
|||||||
"WHERE media_id = ? AND aid = ?",
|
"WHERE media_id = ? AND aid = ?",
|
||||||
(now, media_id, aid),
|
(now, media_id, aid),
|
||||||
)
|
)
|
||||||
|
self._recompute_primary_folder(conn, aid)
|
||||||
other = conn.execute(
|
other = conn.execute(
|
||||||
"""
|
"""
|
||||||
SELECT 1
|
SELECT 1
|
||||||
@@ -379,6 +417,25 @@ class Repository:
|
|||||||
return [int(row["aid"]) for row in rows]
|
return [int(row["aid"]) for row in rows]
|
||||||
return self.db.read(run)
|
return self.db.read(run)
|
||||||
|
|
||||||
|
def pending_snapshot_aids(self, stat_date: str) -> list[int]:
|
||||||
|
"""当天仍缺成功快照的追踪视频;用于新入库/恢复后的同日补采。"""
|
||||||
|
def run(conn: sqlite3.Connection) -> list[int]:
|
||||||
|
rows = conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT v.aid
|
||||||
|
FROM video v
|
||||||
|
WHERE v.status IN ('active', 'unfaved')
|
||||||
|
AND NOT EXISTS (
|
||||||
|
SELECT 1 FROM snapshot s
|
||||||
|
WHERE s.stat_date = ? AND s.aid = v.aid AND s.ok = 1
|
||||||
|
)
|
||||||
|
ORDER BY v.aid
|
||||||
|
""",
|
||||||
|
(stat_date,),
|
||||||
|
).fetchall()
|
||||||
|
return [int(row["aid"]) for row in rows]
|
||||||
|
return self.db.read(run)
|
||||||
|
|
||||||
def get_snapshot(self, stat_date: str, aid: int) -> dict[str, Any] | None:
|
def get_snapshot(self, stat_date: str, aid: int) -> dict[str, Any] | None:
|
||||||
def run(conn: sqlite3.Connection) -> dict[str, Any] | None:
|
def run(conn: sqlite3.Connection) -> dict[str, Any] | None:
|
||||||
row = conn.execute(
|
row = conn.execute(
|
||||||
|
|||||||
@@ -19,7 +19,6 @@ import time
|
|||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from ..bilibili.client import BilibiliClient
|
|
||||||
from ..config import AppConfig, ConfigStore, FolderConfig
|
from ..config import AppConfig, ConfigStore, FolderConfig
|
||||||
from ..repository import Repository
|
from ..repository import Repository
|
||||||
|
|
||||||
@@ -113,18 +112,29 @@ class FolderSyncService:
|
|||||||
self,
|
self,
|
||||||
config_store: ConfigStore,
|
config_store: ConfigStore,
|
||||||
repository: Repository,
|
repository: Repository,
|
||||||
client: BilibiliClient,
|
client=None,
|
||||||
*,
|
*,
|
||||||
|
client_provider=None,
|
||||||
clock=time.time,
|
clock=time.time,
|
||||||
logger=None,
|
logger=None,
|
||||||
capacity_warn: int = CAPACITY_WARN,
|
capacity_warn: int = CAPACITY_WARN,
|
||||||
|
allow_empty_folders: bool = False,
|
||||||
) -> None:
|
) -> None:
|
||||||
|
if client is None and client_provider is None:
|
||||||
|
raise ValueError("必须提供 client 或 client_provider")
|
||||||
self.config_store = config_store
|
self.config_store = config_store
|
||||||
self.repo = repository
|
self.repo = repository
|
||||||
self.client = client
|
self.client = client
|
||||||
|
self.client_provider = client_provider
|
||||||
self._clock = clock
|
self._clock = clock
|
||||||
self.logger = logger
|
self.logger = logger
|
||||||
self.capacity_warn = capacity_warn
|
self.capacity_warn = capacity_warn
|
||||||
|
self.allow_empty_folders = allow_empty_folders
|
||||||
|
|
||||||
|
def _client(self):
|
||||||
|
if self.client_provider is not None:
|
||||||
|
return self.client_provider.get()
|
||||||
|
return self.client
|
||||||
|
|
||||||
def _now(self) -> int:
|
def _now(self) -> int:
|
||||||
return int(self._clock())
|
return int(self._clock())
|
||||||
@@ -135,6 +145,7 @@ class FolderSyncService:
|
|||||||
|
|
||||||
def run(self, *, full_refresh: bool = False) -> SyncSummary:
|
def run(self, *, full_refresh: bool = False) -> SyncSummary:
|
||||||
cfg = self.config_store.get()
|
cfg = self.config_store.get()
|
||||||
|
client = self._client()
|
||||||
now = self._now()
|
now = self._now()
|
||||||
summary = SyncSummary(started_at=now)
|
summary = SyncSummary(started_at=now)
|
||||||
self._log(
|
self._log(
|
||||||
@@ -151,7 +162,9 @@ class FolderSyncService:
|
|||||||
|
|
||||||
for folder in cfg.folders:
|
for folder in cfg.folders:
|
||||||
try:
|
try:
|
||||||
result = self._sync_folder(cfg, folder, now=now, force_full=full_refresh)
|
result = self._sync_folder(
|
||||||
|
client, cfg, folder, now=now, force_full=full_refresh
|
||||||
|
)
|
||||||
except Exception as exc: # noqa: BLE001 单个夹失败不阻塞其它夹
|
except Exception as exc: # noqa: BLE001 单个夹失败不阻塞其它夹
|
||||||
result = FolderSyncResult(
|
result = FolderSyncResult(
|
||||||
media_id=folder.media_id,
|
media_id=folder.media_id,
|
||||||
@@ -191,6 +204,7 @@ class FolderSyncService:
|
|||||||
|
|
||||||
def _sync_folder(
|
def _sync_folder(
|
||||||
self,
|
self,
|
||||||
|
client,
|
||||||
cfg: AppConfig,
|
cfg: AppConfig,
|
||||||
folder: FolderConfig,
|
folder: FolderConfig,
|
||||||
*,
|
*,
|
||||||
@@ -206,13 +220,20 @@ class FolderSyncService:
|
|||||||
>= cfg.schedule.full_folder_refresh_hours * 3600
|
>= cfg.schedule.full_folder_refresh_hours * 3600
|
||||||
)
|
)
|
||||||
|
|
||||||
resources = self.client.fav_ids(folder.media_id)
|
resources = client.fav_ids(folder.media_id)
|
||||||
remote_map = {item.aid: item for item in resources if item.type == 2}
|
remote_map = {item.aid: item for item in resources if item.type == 2}
|
||||||
remote = set(remote_map)
|
remote = set(remote_map)
|
||||||
local = self.repo.folder_active_aids(folder.media_id)
|
local = self.repo.folder_active_aids(folder.media_id)
|
||||||
added = remote - local
|
added = remote - local
|
||||||
removed = local - remote
|
removed = local - remote
|
||||||
|
|
||||||
|
if not remote and local and not self.allow_empty_folders:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"收藏夹 {folder.media_id} 远端返回 0 条,但本地仍有 {len(local)} 条;"
|
||||||
|
f"疑似接口异常,已跳过差集。确认收藏夹确实被清空后,"
|
||||||
|
f"可使用 --allow-empty-folder 重试"
|
||||||
|
)
|
||||||
|
|
||||||
result = FolderSyncResult(
|
result = FolderSyncResult(
|
||||||
media_id=folder.media_id,
|
media_id=folder.media_id,
|
||||||
year=folder.year,
|
year=folder.year,
|
||||||
@@ -228,7 +249,7 @@ class FolderSyncService:
|
|||||||
targets |= remote
|
targets |= remote
|
||||||
|
|
||||||
if targets:
|
if targets:
|
||||||
infos = self.client.fav_infos(
|
infos = client.fav_infos(
|
||||||
folder.media_id,
|
folder.media_id,
|
||||||
sorted(targets),
|
sorted(targets),
|
||||||
folder_mid=cfg.bilibili.read.mid,
|
folder_mid=cfg.bilibili.read.mid,
|
||||||
|
|||||||
@@ -20,7 +20,6 @@ from typing import Any
|
|||||||
from ..bilibili.client import (
|
from ..bilibili.client import (
|
||||||
VIDEO_UNAVAILABLE_CODES,
|
VIDEO_UNAVAILABLE_CODES,
|
||||||
BilibiliAPIError,
|
BilibiliAPIError,
|
||||||
BilibiliClient,
|
|
||||||
BilibiliError,
|
BilibiliError,
|
||||||
BilibiliHTTPError,
|
BilibiliHTTPError,
|
||||||
)
|
)
|
||||||
@@ -41,6 +40,7 @@ class SnapshotSummary:
|
|||||||
removed: int = 0
|
removed: int = 0
|
||||||
fans_ok: int = 0
|
fans_ok: int = 0
|
||||||
fans_failed: int = 0
|
fans_failed: int = 0
|
||||||
|
fans_skipped: int = 0
|
||||||
auth_ok: bool | None = None
|
auth_ok: bool | None = None
|
||||||
auth_message: str = ""
|
auth_message: str = ""
|
||||||
backup_path: str | None = None
|
backup_path: str | None = None
|
||||||
@@ -63,6 +63,7 @@ class SnapshotSummary:
|
|||||||
"removed": self.removed,
|
"removed": self.removed,
|
||||||
"fans_ok": self.fans_ok,
|
"fans_ok": self.fans_ok,
|
||||||
"fans_failed": self.fans_failed,
|
"fans_failed": self.fans_failed,
|
||||||
|
"fans_skipped": self.fans_skipped,
|
||||||
"auth_ok": self.auth_ok,
|
"auth_ok": self.auth_ok,
|
||||||
"auth_message": self.auth_message,
|
"auth_message": self.auth_message,
|
||||||
"backup_path": self.backup_path,
|
"backup_path": self.backup_path,
|
||||||
@@ -78,8 +79,9 @@ class SnapshotService:
|
|||||||
self,
|
self,
|
||||||
config_store: ConfigStore,
|
config_store: ConfigStore,
|
||||||
repository: Repository,
|
repository: Repository,
|
||||||
client: BilibiliClient,
|
client=None,
|
||||||
*,
|
*,
|
||||||
|
client_provider=None,
|
||||||
clock=time.time,
|
clock=time.time,
|
||||||
sleeper=time.sleep,
|
sleeper=time.sleep,
|
||||||
logger=None,
|
logger=None,
|
||||||
@@ -89,9 +91,12 @@ class SnapshotService:
|
|||||||
) -> None:
|
) -> None:
|
||||||
if retries < 1:
|
if retries < 1:
|
||||||
raise ValueError("retries 必须 >= 1")
|
raise ValueError("retries 必须 >= 1")
|
||||||
|
if client is None and client_provider is None:
|
||||||
|
raise ValueError("必须提供 client 或 client_provider")
|
||||||
self.config_store = config_store
|
self.config_store = config_store
|
||||||
self.repo = repository
|
self.repo = repository
|
||||||
self.client = client
|
self.client = client
|
||||||
|
self.client_provider = client_provider
|
||||||
self._clock = clock
|
self._clock = clock
|
||||||
self._sleep = sleeper
|
self._sleep = sleeper
|
||||||
self.logger = logger
|
self.logger = logger
|
||||||
@@ -102,6 +107,11 @@ class SnapshotService:
|
|||||||
def _now(self) -> int:
|
def _now(self) -> int:
|
||||||
return int(self._clock())
|
return int(self._clock())
|
||||||
|
|
||||||
|
def _client(self):
|
||||||
|
if self.client_provider is not None:
|
||||||
|
return self.client_provider.get()
|
||||||
|
return self.client
|
||||||
|
|
||||||
def _log(self, level: str, event: str, **fields) -> None:
|
def _log(self, level: str, event: str, **fields) -> None:
|
||||||
if self.logger is not None:
|
if self.logger is not None:
|
||||||
getattr(self.logger, level)(event, **fields)
|
getattr(self.logger, level)(event, **fields)
|
||||||
@@ -110,6 +120,14 @@ class SnapshotService:
|
|||||||
def completed_key(stat_date: str) -> str:
|
def completed_key(stat_date: str) -> str:
|
||||||
return f"snapshot:completed:{stat_date}"
|
return f"snapshot:completed:{stat_date}"
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def backup_key(stat_date: str) -> str:
|
||||||
|
return f"snapshot:backup:{stat_date}"
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def backup_error_key(stat_date: str) -> str:
|
||||||
|
return f"snapshot:backup_error:{stat_date}"
|
||||||
|
|
||||||
def run_if_needed(
|
def run_if_needed(
|
||||||
self, *, date: str | None = None, force: bool = False
|
self, *, date: str | None = None, force: bool = False
|
||||||
) -> SnapshotSummary:
|
) -> SnapshotSummary:
|
||||||
@@ -117,7 +135,14 @@ class SnapshotService:
|
|||||||
stat_date = date or shanghai_date(now)
|
stat_date = date or shanghai_date(now)
|
||||||
if not is_date(stat_date):
|
if not is_date(stat_date):
|
||||||
raise ValueError(f"日期格式必须是 YYYY-MM-DD: {stat_date!r}")
|
raise ValueError(f"日期格式必须是 YYYY-MM-DD: {stat_date!r}")
|
||||||
if not force and self.repo.kv_get(self.completed_key(stat_date)) == "1":
|
if force:
|
||||||
|
return self.run(date=stat_date, force=True)
|
||||||
|
|
||||||
|
completed = self.repo.kv_get(self.completed_key(stat_date)) == "1"
|
||||||
|
pending = self.repo.pending_snapshot_aids(stat_date)
|
||||||
|
if completed and not pending:
|
||||||
|
if self.repo.kv_get(self.backup_key(stat_date)) != "1":
|
||||||
|
return self.backup_only(stat_date)
|
||||||
self._log("info", "snapshot.skip", stat_date=stat_date, reason="already_completed")
|
self._log("info", "snapshot.skip", stat_date=stat_date, reason="already_completed")
|
||||||
return SnapshotSummary(
|
return SnapshotSummary(
|
||||||
stat_date=stat_date,
|
stat_date=stat_date,
|
||||||
@@ -126,10 +151,27 @@ class SnapshotService:
|
|||||||
completed=True,
|
completed=True,
|
||||||
already_completed=True,
|
already_completed=True,
|
||||||
)
|
)
|
||||||
return self.run(date=stat_date, force=force)
|
return self.run(date=stat_date, force=False)
|
||||||
|
|
||||||
|
def backup_only(self, stat_date: str) -> SnapshotSummary:
|
||||||
|
"""采集已完成但备份缺失时,只重试备份,不请求任何外部接口。"""
|
||||||
|
now = self._now()
|
||||||
|
cfg = self.config_store.get()
|
||||||
|
summary = SnapshotSummary(
|
||||||
|
stat_date=stat_date,
|
||||||
|
started_at=now,
|
||||||
|
completed=True,
|
||||||
|
already_completed=True,
|
||||||
|
)
|
||||||
|
self._log("info", "snapshot.backup_retry", stat_date=stat_date)
|
||||||
|
self._run_backup(cfg, stat_date, summary)
|
||||||
|
summary.finished_at = self._now()
|
||||||
|
self._log("info", "snapshot.done", **summary.as_dict())
|
||||||
|
return summary
|
||||||
|
|
||||||
def run(self, *, date: str | None = None, force: bool = False) -> SnapshotSummary:
|
def run(self, *, date: str | None = None, force: bool = False) -> SnapshotSummary:
|
||||||
cfg = self.config_store.get()
|
cfg = self.config_store.get()
|
||||||
|
client = self._client()
|
||||||
now = self._now()
|
now = self._now()
|
||||||
stat_date = date or shanghai_date(now)
|
stat_date = date or shanghai_date(now)
|
||||||
if not is_date(stat_date):
|
if not is_date(stat_date):
|
||||||
@@ -137,8 +179,12 @@ class SnapshotService:
|
|||||||
|
|
||||||
summary = SnapshotSummary(stat_date=stat_date, started_at=now)
|
summary = SnapshotSummary(stat_date=stat_date, started_at=now)
|
||||||
self._log("info", "snapshot.start", stat_date=stat_date, force=force)
|
self._log("info", "snapshot.start", stat_date=stat_date, force=force)
|
||||||
|
# 先落“未完成”,保证强制重采或中途中断后仍可续跑。
|
||||||
|
self.repo.kv_set(self.completed_key(stat_date), "0")
|
||||||
|
if self.repo.kv_get("snapshot:last_completed") == stat_date:
|
||||||
|
self.repo.kv_set("snapshot:last_completed", "")
|
||||||
|
|
||||||
summary.auth_ok, summary.auth_message = self._check_auth(cfg, now)
|
summary.auth_ok, summary.auth_message = self._check_auth(client, cfg, now)
|
||||||
|
|
||||||
videos = self.repo.video_aids_for_snapshot()
|
videos = self.repo.video_aids_for_snapshot()
|
||||||
summary.total = len(videos)
|
summary.total = len(videos)
|
||||||
@@ -148,7 +194,7 @@ class SnapshotService:
|
|||||||
summary.skipped += 1
|
summary.skipped += 1
|
||||||
continue
|
continue
|
||||||
|
|
||||||
stat, error, unavailable = self._fetch_view(aid)
|
stat, error, unavailable = self._fetch_view(client, aid)
|
||||||
if unavailable:
|
if unavailable:
|
||||||
summary.removed += 1
|
summary.removed += 1
|
||||||
self.repo.set_video_status(aid, "removed", now=now)
|
self.repo.set_video_status(aid, "removed", now=now)
|
||||||
@@ -193,7 +239,15 @@ class SnapshotService:
|
|||||||
)
|
)
|
||||||
|
|
||||||
for mid in cfg.track_fans:
|
for mid in cfg.track_fans:
|
||||||
stat, error = self._fetch_fan(mid)
|
existing_fan = self.repo.get_up_stat(stat_date, mid)
|
||||||
|
if (
|
||||||
|
not force
|
||||||
|
and existing_fan is not None
|
||||||
|
and int(existing_fan.get("ok") or 0) == 1
|
||||||
|
):
|
||||||
|
summary.fans_skipped += 1
|
||||||
|
continue
|
||||||
|
stat, error = self._fetch_fan(client, mid)
|
||||||
if stat is not None:
|
if stat is not None:
|
||||||
self.repo.upsert_up_stat(
|
self.repo.upsert_up_stat(
|
||||||
stat_date,
|
stat_date,
|
||||||
@@ -228,6 +282,21 @@ class SnapshotService:
|
|||||||
elif self.repo.kv_get("snapshot:last_completed") == stat_date:
|
elif self.repo.kv_get("snapshot:last_completed") == stat_date:
|
||||||
self.repo.kv_set("snapshot:last_completed", "")
|
self.repo.kv_set("snapshot:last_completed", "")
|
||||||
|
|
||||||
|
self._run_backup(cfg, stat_date, summary)
|
||||||
|
|
||||||
|
summary.finished_at = self._now()
|
||||||
|
self._log(
|
||||||
|
"info" if summary.completed else "warning",
|
||||||
|
"snapshot.done",
|
||||||
|
**summary.as_dict(),
|
||||||
|
)
|
||||||
|
return summary
|
||||||
|
|
||||||
|
# ------------------------------------------------------------ internals
|
||||||
|
|
||||||
|
def _run_backup(
|
||||||
|
self, cfg: AppConfig, stat_date: str, summary: SnapshotSummary
|
||||||
|
) -> None:
|
||||||
try:
|
try:
|
||||||
path = backup_daily(
|
path = backup_daily(
|
||||||
self.repo.db,
|
self.repo.db,
|
||||||
@@ -240,28 +309,24 @@ class SnapshotService:
|
|||||||
keep_days=cfg.storage.backup_keep_days,
|
keep_days=cfg.storage.backup_keep_days,
|
||||||
)
|
)
|
||||||
summary.backup_path = str(path)
|
summary.backup_path = str(path)
|
||||||
|
self.repo.kv_set(self.backup_key(stat_date), "1")
|
||||||
|
self.repo.kv_set(self.backup_error_key(stat_date), "")
|
||||||
except Exception as exc: # noqa: BLE001 备份失败不推翻已采集数据
|
except Exception as exc: # noqa: BLE001 备份失败不推翻已采集数据
|
||||||
summary.backup_error = str(exc)
|
summary.backup_error = str(exc)
|
||||||
|
self.repo.kv_set(self.backup_key(stat_date), "0")
|
||||||
|
self.repo.kv_set(self.backup_error_key(stat_date), str(exc))
|
||||||
self._log("error", "backup.failed", error=str(exc), stat_date=stat_date)
|
self._log("error", "backup.failed", error=str(exc), stat_date=stat_date)
|
||||||
|
|
||||||
summary.finished_at = self._now()
|
def _check_auth(
|
||||||
self._log(
|
self, client, cfg: AppConfig, now: int
|
||||||
"info" if summary.completed else "warning",
|
) -> tuple[bool, str]:
|
||||||
"snapshot.done",
|
|
||||||
**summary.as_dict(),
|
|
||||||
)
|
|
||||||
return summary
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ internals
|
|
||||||
|
|
||||||
def _check_auth(self, cfg: AppConfig, now: int) -> tuple[bool, str]:
|
|
||||||
if not cfg.bilibili.read.cookie:
|
if not cfg.bilibili.read.cookie:
|
||||||
self.repo.kv_set("auth:last_check", str(now))
|
self.repo.kv_set("auth:last_check", str(now))
|
||||||
self.repo.kv_set("auth:ok", "0")
|
self.repo.kv_set("auth:ok", "0")
|
||||||
self.repo.kv_set("auth:message", "未配置 cookie")
|
self.repo.kv_set("auth:message", "未配置 cookie")
|
||||||
return False, "未配置 cookie"
|
return False, "未配置 cookie"
|
||||||
try:
|
try:
|
||||||
state = self.client.check_login()
|
state = client.check_login()
|
||||||
except BilibiliError as exc:
|
except BilibiliError as exc:
|
||||||
message = str(exc)
|
message = str(exc)
|
||||||
self.repo.kv_set("auth:last_check", str(now))
|
self.repo.kv_set("auth:last_check", str(now))
|
||||||
@@ -282,11 +347,11 @@ class SnapshotService:
|
|||||||
)
|
)
|
||||||
return state.is_login, state.message
|
return state.is_login, state.message
|
||||||
|
|
||||||
def _fetch_view(self, aid: int):
|
def _fetch_view(self, client, aid: int):
|
||||||
last_error = ""
|
last_error = ""
|
||||||
for attempt in range(1, self.retries + 1):
|
for attempt in range(1, self.retries + 1):
|
||||||
try:
|
try:
|
||||||
return self.client.view(aid), None, False
|
return client.view(aid), None, False
|
||||||
except BilibiliAPIError as exc:
|
except BilibiliAPIError as exc:
|
||||||
last_error = str(exc)
|
last_error = str(exc)
|
||||||
if exc.code in VIDEO_UNAVAILABLE_CODES:
|
if exc.code in VIDEO_UNAVAILABLE_CODES:
|
||||||
@@ -301,11 +366,11 @@ class SnapshotService:
|
|||||||
self._sleep(self.backoff * (2 ** (attempt - 1)))
|
self._sleep(self.backoff * (2 ** (attempt - 1)))
|
||||||
return None, last_error or "未知错误", False
|
return None, last_error or "未知错误", False
|
||||||
|
|
||||||
def _fetch_fan(self, mid: int):
|
def _fetch_fan(self, client, mid: int):
|
||||||
last_error = ""
|
last_error = ""
|
||||||
for attempt in range(1, self.retries + 1):
|
for attempt in range(1, self.retries + 1):
|
||||||
try:
|
try:
|
||||||
return self.client.relation_stat(mid), None
|
return client.relation_stat(mid), None
|
||||||
except BilibiliError as exc:
|
except BilibiliError as exc:
|
||||||
last_error = str(exc)
|
last_error = str(exc)
|
||||||
if attempt < self.retries:
|
if attempt < self.retries:
|
||||||
|
|||||||
@@ -361,6 +361,42 @@ def test_check_login_without_cookie_makes_no_request():
|
|||||||
assert transport.calls == []
|
assert transport.calls == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_data_is_protocol_error():
|
||||||
|
for payload in (
|
||||||
|
{"code": 0, "message": "OK"},
|
||||||
|
{"code": 0, "message": "OK", "data": None},
|
||||||
|
):
|
||||||
|
transport = QueueTransport([http_json(payload)])
|
||||||
|
client, _ = make_client(transport)
|
||||||
|
try:
|
||||||
|
client.fav_ids(1)
|
||||||
|
except BilibiliProtocolError as exc:
|
||||||
|
assert "data" in str(exc)
|
||||||
|
else:
|
||||||
|
raise AssertionError(f"ids 缺少 data 应报协议错误: {payload}")
|
||||||
|
|
||||||
|
transport = QueueTransport([http_json(payload)])
|
||||||
|
client, _ = make_client(transport)
|
||||||
|
try:
|
||||||
|
client.fav_infos(1, [123])
|
||||||
|
except BilibiliProtocolError as exc:
|
||||||
|
assert "data" in str(exc)
|
||||||
|
else:
|
||||||
|
raise AssertionError(f"infos 缺少 data 应报协议错误: {payload}")
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_data_list_is_valid():
|
||||||
|
transport = QueueTransport([http_json({"code": 0, "message": "OK", "data": []})])
|
||||||
|
client, _ = make_client(transport)
|
||||||
|
assert client.fav_ids(4413147139) == []
|
||||||
|
|
||||||
|
transport = QueueTransport([http_json({"code": 0, "message": "OK", "data": []})])
|
||||||
|
client, _ = make_client(transport)
|
||||||
|
result = client.fav_infos(1, [123, 456])
|
||||||
|
assert result.items == {}
|
||||||
|
assert result.missing == (123, 456)
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
test_fav_ids_parse_and_anonymous_header()
|
test_fav_ids_parse_and_anonymous_header()
|
||||||
test_anonymous_failure_falls_back_to_full_cookie()
|
test_anonymous_failure_falls_back_to_full_cookie()
|
||||||
@@ -380,6 +416,8 @@ def main():
|
|||||||
test_relation_stat()
|
test_relation_stat()
|
||||||
test_check_login()
|
test_check_login()
|
||||||
test_check_login_without_cookie_makes_no_request()
|
test_check_login_without_cookie_makes_no_request()
|
||||||
|
test_missing_data_is_protocol_error()
|
||||||
|
test_empty_data_list_is_valid()
|
||||||
print("OK: B 站只读客户端契约与重试测试全部通过")
|
print("OK: B 站只读客户端契约与重试测试全部通过")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+37
-1
@@ -51,7 +51,7 @@ def test_init_and_status():
|
|||||||
path = write_config(tmp)
|
path = write_config(tmp)
|
||||||
code, output = run_cli(["init", "--config", str(path)])
|
code, output = run_cli(["init", "--config", str(path)])
|
||||||
assert code == 0
|
assert code == 0
|
||||||
assert '"schema_version": 1' in output
|
assert '"schema_version": 2' in output
|
||||||
|
|
||||||
db = Database(Path(tmp) / "app.db")
|
db = Database(Path(tmp) / "app.db")
|
||||||
db.initialize()
|
db.initialize()
|
||||||
@@ -174,6 +174,41 @@ def test_run_once_offline():
|
|||||||
assert selected and all(credentials.mid == 1 for credentials in selected)
|
assert selected and all(credentials.mid == 1 for credentials in selected)
|
||||||
|
|
||||||
|
|
||||||
|
def test_snapshot_backup_failure_returns_error_and_retries():
|
||||||
|
import reaction_board.services.snapshot as snapshot_module
|
||||||
|
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
path = write_config(tmp, cookie="", csrf="")
|
||||||
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
data["folders"] = []
|
||||||
|
data["track_fans"] = []
|
||||||
|
path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
|
||||||
|
|
||||||
|
original = snapshot_module.backup_daily
|
||||||
|
|
||||||
|
def fail_backup(*args, **kwargs):
|
||||||
|
raise OSError("disk full")
|
||||||
|
|
||||||
|
snapshot_module.backup_daily = fail_backup
|
||||||
|
try:
|
||||||
|
code, output = run_cli(["snapshot", "--config", str(path)])
|
||||||
|
finally:
|
||||||
|
snapshot_module.backup_daily = original
|
||||||
|
assert code == 1
|
||||||
|
marker = output.find("{\n")
|
||||||
|
assert marker >= 0, output
|
||||||
|
payload = json.loads(output[marker:])
|
||||||
|
assert payload["completed"] is True
|
||||||
|
assert "disk full" in payload["backup_error"]
|
||||||
|
|
||||||
|
code, output = run_cli(["snapshot", "--config", str(path)])
|
||||||
|
assert code == 0
|
||||||
|
marker = output.find("{\n")
|
||||||
|
payload = json.loads(output[marker:])
|
||||||
|
assert payload["backup_error"] is None
|
||||||
|
assert Path(payload["backup_path"]).exists()
|
||||||
|
|
||||||
|
|
||||||
def main_test():
|
def main_test():
|
||||||
test_init_and_status()
|
test_init_and_status()
|
||||||
test_missing_config()
|
test_missing_config()
|
||||||
@@ -182,6 +217,7 @@ def main_test():
|
|||||||
test_auth_check_without_cookie()
|
test_auth_check_without_cookie()
|
||||||
test_auth_check_all_and_auth_dispatch()
|
test_auth_check_all_and_auth_dispatch()
|
||||||
test_run_once_offline()
|
test_run_once_offline()
|
||||||
|
test_snapshot_backup_failure_returns_error_and_retries()
|
||||||
print("OK: CLI init/sync/snapshot/auth-check/run/status/错误路径测试全部通过")
|
print("OK: CLI init/sync/snapshot/auth-check/run/status/错误路径测试全部通过")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ def test_write_read_transaction():
|
|||||||
with tempfile.TemporaryDirectory() as tmp:
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
db = make_db(tmp)
|
db = make_db(tmp)
|
||||||
try:
|
try:
|
||||||
assert db.schema_version == 1
|
assert db.schema_version == 2
|
||||||
db.write(lambda c: c.execute("INSERT INTO kv(k, v) VALUES ('a', '1')"))
|
db.write(lambda c: c.execute("INSERT INTO kv(k, v) VALUES ('a', '1')"))
|
||||||
assert db.read(lambda c: c.execute("SELECT v FROM kv WHERE k='a'").fetchone()[0]) == "1"
|
assert db.read(lambda c: c.execute("SELECT v FROM kv WHERE k='a'").fetchone()[0]) == "1"
|
||||||
|
|
||||||
@@ -94,8 +94,8 @@ def test_close_and_reinitialize():
|
|||||||
|
|
||||||
db2 = make_db(tmp)
|
db2 = make_db(tmp)
|
||||||
try:
|
try:
|
||||||
assert db2.schema_version == 1
|
assert db2.schema_version == 2
|
||||||
assert db2.read(lambda c: c.execute("SELECT MAX(version) FROM schema_version").fetchone()[0]) == 1
|
assert db2.read(lambda c: c.execute("SELECT MAX(version) FROM schema_version").fetchone()[0]) == 2
|
||||||
finally:
|
finally:
|
||||||
db2.close()
|
db2.close()
|
||||||
|
|
||||||
@@ -123,7 +123,7 @@ def test_migration_backup_on_existing_db():
|
|||||||
db = Database(path, backup_dir=Path(tmp) / "backup")
|
db = Database(path, backup_dir=Path(tmp) / "backup")
|
||||||
try:
|
try:
|
||||||
applied = db.initialize()
|
applied = db.initialize()
|
||||||
assert applied == [1]
|
assert applied == [1, 2]
|
||||||
assert len(list((Path(tmp) / "backup").glob("app-pre-v0-*.db"))) == 1
|
assert len(list((Path(tmp) / "backup").glob("app-pre-v0-*.db"))) == 1
|
||||||
finally:
|
finally:
|
||||||
db.close()
|
db.close()
|
||||||
|
|||||||
@@ -0,0 +1,100 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""favsort run 参数校验与资源类型测试(不联网): uv run python tests/test_favsort_run.py"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
|
sys.stdout.reconfigure(encoding="utf-8")
|
||||||
|
|
||||||
|
import favsort # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
class Args:
|
||||||
|
media_id = 1
|
||||||
|
verify_every = 1
|
||||||
|
|
||||||
|
|
||||||
|
def config():
|
||||||
|
return {
|
||||||
|
"mid": 1,
|
||||||
|
"csrf": "csrf",
|
||||||
|
"cookie": "cookie",
|
||||||
|
"overrides": {},
|
||||||
|
"sleep": {
|
||||||
|
"min": 0, "max": 0,
|
||||||
|
"long_every_min": 1000, "long_every_max": 1000,
|
||||||
|
"long_min": 0, "long_max": 0,
|
||||||
|
"retry_wait": 0,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_zero_verify_every_rejected_before_writes():
|
||||||
|
calls = []
|
||||||
|
originals = (favsort.load_plan, favsort.fetch_ids, favsort.post_form)
|
||||||
|
favsort.load_plan = lambda media_id: {"target_order": [], "titles": {}}
|
||||||
|
favsort.fetch_ids = lambda cfg, media_id: calls.append("fetch") or []
|
||||||
|
favsort.post_form = lambda *args, **kwargs: calls.append("post") or {}
|
||||||
|
args = Args()
|
||||||
|
args.verify_every = 0
|
||||||
|
try:
|
||||||
|
code = favsort.cmd_run(config(), args)
|
||||||
|
finally:
|
||||||
|
favsort.load_plan, favsort.fetch_ids, favsort.post_form = originals
|
||||||
|
assert code == 2
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_sort_preserves_resource_types():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
plan = {
|
||||||
|
"media_id": 1,
|
||||||
|
"target_order": [1, 2],
|
||||||
|
"titles": {"1": {"title": "音频"}, "2": {"title": "视频"}},
|
||||||
|
"types": {1: 2, 2: 12},
|
||||||
|
}
|
||||||
|
bodies = []
|
||||||
|
orders = [[2, 1]]
|
||||||
|
originals = (
|
||||||
|
favsort.load_plan,
|
||||||
|
favsort.fetch_ids,
|
||||||
|
favsort.post_form,
|
||||||
|
favsort.plan_path,
|
||||||
|
favsort.journal_path,
|
||||||
|
)
|
||||||
|
favsort.load_plan = lambda media_id: plan
|
||||||
|
favsort.plan_path = lambda media_id: str(Path(tmp) / "plan.json")
|
||||||
|
favsort.journal_path = lambda media_id: str(Path(tmp) / "journal.jsonl")
|
||||||
|
|
||||||
|
def fetch_ids(cfg, media_id):
|
||||||
|
order = orders.pop(0) if orders else [1, 2]
|
||||||
|
return [{"id": aid} for aid in order]
|
||||||
|
|
||||||
|
favsort.fetch_ids = fetch_ids
|
||||||
|
favsort.post_form = lambda cfg, url, body: bodies.append(body) or {"code": 0}
|
||||||
|
args = Args()
|
||||||
|
try:
|
||||||
|
code = favsort.cmd_run(config(), args)
|
||||||
|
finally:
|
||||||
|
(
|
||||||
|
favsort.load_plan,
|
||||||
|
favsort.fetch_ids,
|
||||||
|
favsort.post_form,
|
||||||
|
favsort.plan_path,
|
||||||
|
favsort.journal_path,
|
||||||
|
) = originals
|
||||||
|
assert code == 0
|
||||||
|
assert [body["sort"] for body in bodies] == ["1:2:2:12"]
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
test_zero_verify_every_rejected_before_writes()
|
||||||
|
test_sort_preserves_resource_types()
|
||||||
|
print("OK: favsort run 参数校验与资源类型测试全部通过")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -38,10 +38,11 @@ def http_json(payload):
|
|||||||
|
|
||||||
|
|
||||||
class FakeFavTransport:
|
class FakeFavTransport:
|
||||||
def __init__(self, folders, infos, *, fail_media=None):
|
def __init__(self, folders, infos, *, fail_media=None, omit_ids_data=None):
|
||||||
self.folders = {k: list(v) for k, v in folders.items()}
|
self.folders = {k: list(v) for k, v in folders.items()}
|
||||||
self.infos = dict(infos)
|
self.infos = dict(infos)
|
||||||
self.fail_media = fail_media
|
self.fail_media = fail_media
|
||||||
|
self.omit_ids_data = omit_ids_data
|
||||||
self.ids_calls = []
|
self.ids_calls = []
|
||||||
self.info_calls = []
|
self.info_calls = []
|
||||||
|
|
||||||
@@ -52,6 +53,8 @@ class FakeFavTransport:
|
|||||||
self.ids_calls.append(media_id)
|
self.ids_calls.append(media_id)
|
||||||
if self.fail_media == media_id:
|
if self.fail_media == media_id:
|
||||||
raise TransportError("ids 网络错误")
|
raise TransportError("ids 网络错误")
|
||||||
|
if self.omit_ids_data == media_id:
|
||||||
|
return http_json({"code": 0, "message": "OK", "ttl": 1})
|
||||||
data_rows = [
|
data_rows = [
|
||||||
{"id": aid, "type": 2, "bv_id": f"BV{aid}", "bvid": f"BV{aid}"}
|
{"id": aid, "type": 2, "bv_id": f"BV{aid}", "bvid": f"BV{aid}"}
|
||||||
for aid in self.folders.get(media_id, [])
|
for aid in self.folders.get(media_id, [])
|
||||||
@@ -284,6 +287,51 @@ def test_blacklist_seed_only_once():
|
|||||||
db.close()
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_remote_requires_explicit_opt_in():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
transport = FakeFavTransport(
|
||||||
|
{F1: [1101, 1102], F2: []},
|
||||||
|
{1101: valid(1101), 1102: valid(1102)},
|
||||||
|
)
|
||||||
|
clock = [1_800_000_000]
|
||||||
|
db, repo, service = make_service(tmp, transport, clock)
|
||||||
|
try:
|
||||||
|
service.run()
|
||||||
|
transport.folders[F1] = []
|
||||||
|
clock[0] += 3600
|
||||||
|
summary = service.run()
|
||||||
|
assert summary.ok is False
|
||||||
|
assert "疑似接口异常" in summary.folders[0].error
|
||||||
|
assert repo.get_video(1101)["status"] == "active"
|
||||||
|
assert repo.folder_active_aids(F1) == {1101, 1102}
|
||||||
|
|
||||||
|
service.allow_empty_folders = True
|
||||||
|
summary = service.run()
|
||||||
|
assert summary.ok is True
|
||||||
|
assert repo.get_video(1101)["status"] == "unfaved"
|
||||||
|
assert repo.folder_active_aids(F1) == set()
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_ids_data_aborts_folder():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
transport = FakeFavTransport(
|
||||||
|
{F1: [1101], F2: []},
|
||||||
|
{1101: valid(1101)},
|
||||||
|
omit_ids_data=F1,
|
||||||
|
)
|
||||||
|
clock = [1_800_000_000]
|
||||||
|
db, repo, service = make_service(tmp, transport, clock)
|
||||||
|
try:
|
||||||
|
summary = service.run()
|
||||||
|
assert summary.ok is False
|
||||||
|
assert "data" in summary.folders[0].error
|
||||||
|
assert repo.get_video(1101) is None
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
test_initial_import_and_incremental_skip()
|
test_initial_import_and_incremental_skip()
|
||||||
test_diff_add_and_remove()
|
test_diff_add_and_remove()
|
||||||
@@ -291,6 +339,8 @@ def main():
|
|||||||
test_capacity_warning_does_not_move_folder()
|
test_capacity_warning_does_not_move_folder()
|
||||||
test_folder_error_does_not_block_other_folders()
|
test_folder_error_does_not_block_other_folders()
|
||||||
test_blacklist_seed_only_once()
|
test_blacklist_seed_only_once()
|
||||||
|
test_empty_remote_requires_explicit_opt_in()
|
||||||
|
test_missing_ids_data_aborts_folder()
|
||||||
print("OK: 收藏夹初始导入、差集同步、失效识别与容量预警测试全部通过")
|
print("OK: 收藏夹初始导入、差集同步、失效识别与容量预警测试全部通过")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -37,14 +37,18 @@ def test_fresh_and_idempotent():
|
|||||||
conn = sqlite3.connect(Path(tmp) / "app.db")
|
conn = sqlite3.connect(Path(tmp) / "app.db")
|
||||||
try:
|
try:
|
||||||
applied = run_migrations(conn)
|
applied = run_migrations(conn)
|
||||||
assert applied == [1]
|
assert applied == [1, 2]
|
||||||
assert current_version(conn) == 1
|
assert current_version(conn) == 2
|
||||||
assert EXPECTED_TABLES <= table_names(conn)
|
assert EXPECTED_TABLES <= table_names(conn)
|
||||||
assert "schema_version" in table_names(conn)
|
assert "schema_version" in table_names(conn)
|
||||||
|
columns = {
|
||||||
|
row[1] for row in conn.execute("PRAGMA table_info(folder_resource)")
|
||||||
|
}
|
||||||
|
assert "fav_time" in columns
|
||||||
# 再跑一次不应重复应用
|
# 再跑一次不应重复应用
|
||||||
assert run_migrations(conn) == []
|
assert run_migrations(conn) == []
|
||||||
row = conn.execute("SELECT COUNT(*) FROM schema_version").fetchone()
|
row = conn.execute("SELECT COUNT(*) FROM schema_version").fetchone()
|
||||||
assert row[0] == 1
|
assert row[0] == 2
|
||||||
finally:
|
finally:
|
||||||
conn.close()
|
conn.close()
|
||||||
|
|
||||||
@@ -59,15 +63,15 @@ def test_backup_before_upgrade():
|
|||||||
conn.execute("INSERT INTO kv(k, v) VALUES ('hello', 'world')")
|
conn.execute("INSERT INTO kv(k, v) VALUES ('hello', 'world')")
|
||||||
conn.commit()
|
conn.commit()
|
||||||
|
|
||||||
v2 = Migration(2, "add probe", ("CREATE TABLE probe (id INTEGER PRIMARY KEY)",))
|
v3 = Migration(3, "add probe", ("CREATE TABLE probe (id INTEGER PRIMARY KEY)",))
|
||||||
applied = run_migrations(
|
applied = run_migrations(
|
||||||
conn,
|
conn,
|
||||||
backup_dir=backup_dir,
|
backup_dir=backup_dir,
|
||||||
now=1_800_000_000,
|
now=1_800_000_000,
|
||||||
migrations=MIGRATIONS + (v2,),
|
migrations=MIGRATIONS + (v3,),
|
||||||
)
|
)
|
||||||
assert applied == [2]
|
assert applied == [3]
|
||||||
backups = list(backup_dir.glob("app-pre-v1-*.db"))
|
backups = list(backup_dir.glob("app-pre-v2-*.db"))
|
||||||
assert len(backups) == 1
|
assert len(backups) == 1
|
||||||
check = sqlite3.connect(backups[0])
|
check = sqlite3.connect(backups[0])
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -0,0 +1,106 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""凭据变化时客户端重建测试(不联网): uv run python tests/test_provider.py"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
|
sys.stdout.reconfigure(encoding="utf-8")
|
||||||
|
|
||||||
|
from reaction_board.bilibili.client import BilibiliClient, Pacer # noqa: E402
|
||||||
|
from reaction_board.bilibili.provider import ClientProvider # noqa: E402
|
||||||
|
from reaction_board.config import ConfigStore # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def write_config(path: Path, *, cookie: str, mid: int) -> None:
|
||||||
|
path.write_text(json.dumps({
|
||||||
|
"bilibili": {
|
||||||
|
"read": {"mid": mid, "csrf": "read-csrf", "cookie": cookie},
|
||||||
|
"write": {"mid": 2, "csrf": "write-csrf", "cookie": "write-cookie"},
|
||||||
|
},
|
||||||
|
"folders": [],
|
||||||
|
}, ensure_ascii=False), encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def bump(path: Path) -> None:
|
||||||
|
target = max(path.stat().st_mtime_ns, time.time_ns()) + 1_000_000
|
||||||
|
os.utime(path, ns=(target, target))
|
||||||
|
|
||||||
|
|
||||||
|
def test_rebuild_only_when_credentials_change():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
path = Path(tmp) / "config.json"
|
||||||
|
write_config(path, cookie="cookie-a", mid=1)
|
||||||
|
store = ConfigStore(path, env={})
|
||||||
|
created = []
|
||||||
|
|
||||||
|
def factory(credentials):
|
||||||
|
created.append((credentials.mid, credentials.cookie))
|
||||||
|
return BilibiliClient(
|
||||||
|
cookie=credentials.cookie,
|
||||||
|
csrf=credentials.csrf,
|
||||||
|
mid=credentials.mid,
|
||||||
|
pacer=Pacer(1.0, 1.6),
|
||||||
|
)
|
||||||
|
|
||||||
|
provider = ClientProvider(store, "read", factory)
|
||||||
|
first = provider.get()
|
||||||
|
assert provider.get() is first
|
||||||
|
assert created == [(1, "cookie-a")]
|
||||||
|
|
||||||
|
write_config(path, cookie="cookie-b", mid=1)
|
||||||
|
bump(path)
|
||||||
|
second = provider.get()
|
||||||
|
assert second is not first
|
||||||
|
assert created == [(1, "cookie-a"), (1, "cookie-b")]
|
||||||
|
assert second.pacer.min_delay == 1.0
|
||||||
|
assert second.pacer.max_delay == 1.6
|
||||||
|
|
||||||
|
# 仅 mtime 变化但凭据相同:不重建
|
||||||
|
bump(path)
|
||||||
|
assert provider.get() is second
|
||||||
|
assert len(created) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_mid_change_rebuilds():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
path = Path(tmp) / "config.json"
|
||||||
|
write_config(path, cookie="cookie-a", mid=1)
|
||||||
|
store = ConfigStore(path, env={})
|
||||||
|
provider = ClientProvider(
|
||||||
|
store, "read",
|
||||||
|
lambda credentials: BilibiliClient(mid=credentials.mid),
|
||||||
|
)
|
||||||
|
assert provider.get().mid == 1
|
||||||
|
write_config(path, cookie="cookie-a", mid=9)
|
||||||
|
bump(path)
|
||||||
|
assert provider.get().mid == 9
|
||||||
|
|
||||||
|
|
||||||
|
def test_config_revision_increments():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
path = Path(tmp) / "config.json"
|
||||||
|
write_config(path, cookie="cookie-a", mid=1)
|
||||||
|
store = ConfigStore(path, env={})
|
||||||
|
assert store.revision == 0
|
||||||
|
write_config(path, cookie="cookie-b", mid=1)
|
||||||
|
bump(path)
|
||||||
|
store.get()
|
||||||
|
assert store.revision == 1
|
||||||
|
store.get()
|
||||||
|
assert store.revision == 1
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
test_rebuild_only_when_credentials_change()
|
||||||
|
test_mid_change_rebuilds()
|
||||||
|
test_config_revision_increments()
|
||||||
|
print("OK: 凭据变化重建客户端与配置 revision 测试全部通过")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -18,7 +18,7 @@ F1 = FolderConfig(year=2025, media_id=4451053851, title="2025")
|
|||||||
F2 = FolderConfig(year=2026, media_id=3700023351, title="2026")
|
F2 = FolderConfig(year=2026, media_id=3700023351, title="2026")
|
||||||
|
|
||||||
|
|
||||||
def resource(aid, *, attr=0, up_mid=100, title="视频", bvid=None):
|
def resource(aid, *, attr=0, up_mid=100, title="视频", bvid=None, fav_time=None):
|
||||||
return FavResource(
|
return FavResource(
|
||||||
aid=aid,
|
aid=aid,
|
||||||
bvid=bvid or f"BV{aid}",
|
bvid=bvid or f"BV{aid}",
|
||||||
@@ -29,7 +29,7 @@ def resource(aid, *, attr=0, up_mid=100, title="视频", bvid=None):
|
|||||||
duration=600,
|
duration=600,
|
||||||
cover="http://example/cover.jpg",
|
cover="http://example/cover.jpg",
|
||||||
attr=attr,
|
attr=attr,
|
||||||
fav_time=None,
|
fav_time=fav_time,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -227,6 +227,72 @@ def test_folder_sync_metadata_and_counts():
|
|||||||
db.close()
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_primary_folder_is_order_independent():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
db, repo = setup(tmp)
|
||||||
|
try:
|
||||||
|
repo.upsert_from_fav(resource(1, fav_time=100), F1.media_id, was_added=True)
|
||||||
|
assert repo.get_video(1)["folder_id"] == F1.media_id
|
||||||
|
repo.upsert_from_fav(resource(1, fav_time=200), F2.media_id, was_added=True)
|
||||||
|
assert repo.get_video(1)["folder_id"] == F2.media_id
|
||||||
|
|
||||||
|
# 全量刷新按相反顺序遍历,主归属仍应指向 fav_time 更新的夹
|
||||||
|
repo.upsert_from_fav(resource(1, fav_time=100), F1.media_id, was_added=False)
|
||||||
|
assert repo.get_video(1)["folder_id"] == F2.media_id
|
||||||
|
repo.upsert_from_fav(resource(1, fav_time=200), F2.media_id, was_added=False)
|
||||||
|
assert repo.get_video(1)["folder_id"] == F2.media_id
|
||||||
|
|
||||||
|
# 从主夹移除、另一个夹仍持有时,主归属切换
|
||||||
|
status = repo.mark_resource_removed(F2.media_id, 1)
|
||||||
|
assert status == "active"
|
||||||
|
assert repo.get_video(1)["folder_id"] == F1.media_id
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_dropped_invalid_preserves_original_title():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
db, repo = setup(tmp)
|
||||||
|
try:
|
||||||
|
repo.upsert_from_fav(
|
||||||
|
resource(1, title="原始标题"), F1.media_id, was_added=True
|
||||||
|
)
|
||||||
|
repo.set_video_status(1, "dropped")
|
||||||
|
repo.upsert_from_fav(
|
||||||
|
resource(1, title="已失效视频", attr=9),
|
||||||
|
F1.media_id,
|
||||||
|
was_added=False,
|
||||||
|
)
|
||||||
|
video = repo.get_video(1)
|
||||||
|
assert video["status"] == "dropped"
|
||||||
|
assert video["title"] == "原始标题"
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_pending_snapshot_aids():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
db, repo = setup(tmp)
|
||||||
|
try:
|
||||||
|
repo.upsert_from_fav(resource(1), F1.media_id, was_added=True)
|
||||||
|
repo.upsert_from_fav(resource(2), F1.media_id, was_added=True)
|
||||||
|
repo.upsert_from_fav(resource(3), F1.media_id, was_added=True)
|
||||||
|
repo.set_video_status(3, "unfaved")
|
||||||
|
repo.upsert_snapshot(
|
||||||
|
"2026-10-31", 1,
|
||||||
|
metrics={"view": 1, "like": 1, "favorite": 1, "coin": 1,
|
||||||
|
"share": 1, "reply": 1, "danmaku": 1},
|
||||||
|
ok=True, err=None, captured_at=1,
|
||||||
|
)
|
||||||
|
repo.upsert_snapshot(
|
||||||
|
"2026-10-31", 2,
|
||||||
|
metrics=None, ok=False, err="x", captured_at=1,
|
||||||
|
)
|
||||||
|
assert repo.pending_snapshot_aids("2026-10-31") == [2, 3]
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
test_upsert_and_folder_resource()
|
test_upsert_and_folder_resource()
|
||||||
test_invalid_attr_and_missing()
|
test_invalid_attr_and_missing()
|
||||||
@@ -236,6 +302,9 @@ def main():
|
|||||||
test_multi_folder_moves_primary_and_warns()
|
test_multi_folder_moves_primary_and_warns()
|
||||||
test_blacklist_seed_idempotent()
|
test_blacklist_seed_idempotent()
|
||||||
test_folder_sync_metadata_and_counts()
|
test_folder_sync_metadata_and_counts()
|
||||||
|
test_primary_folder_is_order_independent()
|
||||||
|
test_dropped_invalid_preserves_original_title()
|
||||||
|
test_pending_snapshot_aids()
|
||||||
print("OK: 收藏夹/视频状态机与黑名单测试全部通过")
|
print("OK: 收藏夹/视频状态机与黑名单测试全部通过")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -60,6 +60,8 @@ class SnapshotTransport:
|
|||||||
urllib.parse.urlparse(url).query
|
urllib.parse.urlparse(url).query
|
||||||
)["aid"][0])
|
)["aid"][0])
|
||||||
payload = self.views[aid]
|
payload = self.views[aid]
|
||||||
|
if isinstance(payload, BaseException):
|
||||||
|
raise payload
|
||||||
if isinstance(payload, HTTPResponse):
|
if isinstance(payload, HTTPResponse):
|
||||||
return payload
|
return payload
|
||||||
return http_json(payload)
|
return http_json(payload)
|
||||||
@@ -316,6 +318,115 @@ def test_force_failure_clears_last_completed():
|
|||||||
db.close()
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_new_video_after_completion_backfills_same_day():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
transport = SnapshotTransport({1: view_for(1, view=100)})
|
||||||
|
db, repo, service = make_service(tmp, transport, track_fans=())
|
||||||
|
try:
|
||||||
|
add_video(repo, 1)
|
||||||
|
first = service.run(date=DATE)
|
||||||
|
assert first.completed is True
|
||||||
|
assert transport.count("/web-interface/view") == 1
|
||||||
|
|
||||||
|
# 当天 11:00 新入库:完成标记仍是 1,但必须补采新视频
|
||||||
|
add_video(repo, 2)
|
||||||
|
transport.views[2] = view_for(2, view=222)
|
||||||
|
second = service.run_if_needed(date=DATE)
|
||||||
|
assert second.already_completed is False
|
||||||
|
assert second.success == 1
|
||||||
|
assert second.skipped == 1
|
||||||
|
assert transport.count("/web-interface/view") == 2
|
||||||
|
assert repo.get_snapshot(DATE, 2)["view"] == 222
|
||||||
|
assert repo.get_snapshot(DATE, 1)["view"] == 100
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_fan_resume_preserves_successful_value():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
transport = SnapshotTransport(
|
||||||
|
{1: {"code": -500, "message": "x", "data": None}},
|
||||||
|
)
|
||||||
|
db, repo, service = make_service(tmp, transport, retries=1)
|
||||||
|
try:
|
||||||
|
add_video(repo, 1)
|
||||||
|
first = service.run(date=DATE)
|
||||||
|
assert first.failed == 1
|
||||||
|
assert first.fans_ok == 1
|
||||||
|
assert repo.get_up_stat(DATE, 37754047)["follower"] == 2767268
|
||||||
|
|
||||||
|
transport.views[1] = view_for(1, view=111)
|
||||||
|
transport.fan = {"code": -500, "message": "x", "data": None}
|
||||||
|
second = service.run(date=DATE)
|
||||||
|
assert second.fans_skipped == 1
|
||||||
|
assert second.fans_failed == 0
|
||||||
|
fan = repo.get_up_stat(DATE, 37754047)
|
||||||
|
assert fan["follower"] == 2767268
|
||||||
|
assert fan["ok"] == 1
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_force_interrupt_leaves_incomplete_marker():
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
transport = SnapshotTransport({
|
||||||
|
1: view_for(1),
|
||||||
|
2: RuntimeError("simulated interrupt"),
|
||||||
|
})
|
||||||
|
db, repo, service = make_service(tmp, transport, track_fans=())
|
||||||
|
try:
|
||||||
|
add_video(repo, 1)
|
||||||
|
add_video(repo, 2)
|
||||||
|
repo.kv_set(service.completed_key(DATE), "1")
|
||||||
|
repo.kv_set("snapshot:last_completed", DATE)
|
||||||
|
try:
|
||||||
|
service.run(date=DATE, force=True)
|
||||||
|
except RuntimeError:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
raise AssertionError("应模拟中断")
|
||||||
|
assert repo.kv_get(service.completed_key(DATE)) == "0"
|
||||||
|
assert repo.kv_get("snapshot:last_completed") == ""
|
||||||
|
|
||||||
|
transport.views[2] = view_for(2)
|
||||||
|
resumed = service.run_if_needed(date=DATE)
|
||||||
|
assert resumed.completed is True
|
||||||
|
assert resumed.success == 1
|
||||||
|
assert resumed.skipped == 1
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_backup_failure_retries_without_api():
|
||||||
|
import reaction_board.services.snapshot as snapshot_module
|
||||||
|
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
transport = SnapshotTransport({1: view_for(1)})
|
||||||
|
db, repo, service = make_service(tmp, transport, track_fans=())
|
||||||
|
original = snapshot_module.backup_daily
|
||||||
|
|
||||||
|
def fail_backup(*args, **kwargs):
|
||||||
|
raise OSError("disk full")
|
||||||
|
|
||||||
|
snapshot_module.backup_daily = fail_backup
|
||||||
|
try:
|
||||||
|
first = service.run(date=DATE)
|
||||||
|
finally:
|
||||||
|
snapshot_module.backup_daily = original
|
||||||
|
assert first.completed is True
|
||||||
|
assert first.backup_error
|
||||||
|
assert repo.kv_get(service.backup_key(DATE)) == "0"
|
||||||
|
view_calls = transport.count("/web-interface/view")
|
||||||
|
|
||||||
|
retried = service.run_if_needed(date=DATE)
|
||||||
|
assert retried.already_completed is True
|
||||||
|
assert retried.backup_error is None
|
||||||
|
assert Path(retried.backup_path or "").exists()
|
||||||
|
assert repo.kv_get(service.backup_key(DATE)) == "1"
|
||||||
|
assert transport.count("/web-interface/view") == view_calls
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
test_snapshot_success_resume_and_force()
|
test_snapshot_success_resume_and_force()
|
||||||
test_resume_skips_ok_and_retries_failed()
|
test_resume_skips_ok_and_retries_failed()
|
||||||
@@ -325,6 +436,10 @@ def main():
|
|||||||
test_auth_invalid_is_warned_but_snapshot_continues()
|
test_auth_invalid_is_warned_but_snapshot_continues()
|
||||||
test_no_cookie_skips_nav_request()
|
test_no_cookie_skips_nav_request()
|
||||||
test_force_failure_clears_last_completed()
|
test_force_failure_clears_last_completed()
|
||||||
|
test_new_video_after_completion_backfills_same_day()
|
||||||
|
test_fan_resume_preserves_successful_value()
|
||||||
|
test_force_interrupt_leaves_incomplete_marker()
|
||||||
|
test_backup_failure_retries_without_api()
|
||||||
print("OK: 每日快照、断点续跑、失败占位与粉丝采集测试全部通过")
|
print("OK: 每日快照、断点续跑、失败占位与粉丝采集测试全部通过")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user