fix: close data integrity and hot-reload review findings
Backfill same-day snapshots for newly added or restored videos, preserve successful fan snapshots on resume, persist incomplete state before forced re-runs, and split collection completion from backup completion so failed backups can retry without API calls. Reject missing data payloads instead of treating them as empty folders, guard against mass-unfav on suspicious empty remote folders, rebuild Bilibili clients when credentials change, recompute primary folder ownership from active relations with per-folder fav_time, preserve original titles for dropped invalid videos, poll config every 60 seconds, and fix favsort verify interval and mixed resource types.
This commit is contained in:
1 parent
178ab64b18
commit
6a21b5d905
21 files changed
+895
-82
No files matched your search
@@ -84,7 +84,8 @@ uv run python app.py status
|
||||
- `infos` 静默缺失或 `attr=9` 的资源标记为 `removed`,保留历史;
|
||||
- 从收藏夹移除的资源标记为 `unfaved`,仍保留数据;
|
||||
- 黑名单只作为发现阶段的种子写入,**不会过滤收藏夹同步**;
|
||||
- 收藏夹达到 950 条时预警,系统绝不自动换夹。
|
||||
- 收藏夹达到 950 条时预警,系统绝不自动换夹;
|
||||
- 远端返回 0 条但本地仍有记录时默认按异常跳过差集,确认收藏夹确实被清空后才使用 `--allow-empty-folder`。
|
||||
|
||||
## P1:每日快照与粉丝追踪
|
||||
|
||||
@@ -108,11 +109,13 @@ uv run python app.py auth-check --account all
|
||||
|
||||
- `snapshot` 对 `active` / `unfaved` 视频逐条请求 `view`,每条请求间隔 1.0~1.6 秒;
|
||||
- 成功写入 `snapshot` 的七项指标;同日 `ok=1` 的记录直接跳过,可断点续跑;
|
||||
- 当天已完成后新入库或从 `dropped` 恢复的视频会立即补采当天快照,不必等到次日;
|
||||
- 视频不可访问(`-404` / `62002` / `62012`)标记为 `removed`;
|
||||
- 其他错误重试 3 次,仍失败写入 `ok=0` 和错误信息,不写伪造的零值;
|
||||
- 粉丝数写入 `up_stat`,失败同样写 `ok=0`;
|
||||
- 粉丝数写入 `up_stat`,失败同样写 `ok=0`;续跑时已成功的 `(stat_date, mid)` 会跳过,不会被失败结果覆盖;
|
||||
- 当日全部视频与粉丝采集成功后写入完成标记;服务启动时若当天未完成会自动补跑;
|
||||
- 每次采集后生成 `data/backup/app-YYYYMMDD.db`,按 `storage.backup_keep_days` 清理旧日备份;
|
||||
- 备份失败时 `snapshot` 返回 1 便于任务告警;再次执行只重试备份,不重复请求 B 站;
|
||||
- Cookie 失效只告警,不阻塞匿名 `view` 采集;P0/P1 只使用 `bilibili.read`。
|
||||
|
||||
首次真实全量采集结果:795 条视频全部成功、0 失败、0 移除,耗时约 19 分钟;粉丝 `follower=2767270`;日备份为 `data/backup/app-20261031.db`。
|
||||
@@ -146,6 +149,8 @@ Copy-Item config.example.json config.json
|
||||
- `storage`:数据库与备份目录;
|
||||
- `schedule` / `notify` / `classify` / `heat_weights`:任务节奏、通知阈值、分类规则和热度权重。
|
||||
|
||||
配置每 60 秒检查一次 mtime,凭据、收藏夹、分类规则等变更无需重启即可在下次任务读取;但 `schedule.folder_sync_minutes` 与 `schedule.snapshot_at` 在启动时固化,修改后需重启 `run` 进程。
|
||||
|
||||
完整配置口径以 [project.md](project.md) 和 [DEVELOPMENT.md](DEVELOPMENT.md) 为准。
|
||||
|
||||
## 当前可用:收藏夹排序 CLI
|
||||
@@ -243,10 +248,12 @@ uv run python tests/test_database.py # 单写队列、事务与并发
|
||||
uv run python tests/test_repository.py # 收藏夹/视频状态机与黑名单
|
||||
uv run python tests/test_bilibili_client.py # ids/infos/view/fans/nav 契约与重试
|
||||
uv run python tests/test_folder_sync.py # 首次导入、差集、失效识别、容量预警
|
||||
uv run python tests/test_snapshot.py # 七项快照、断点续跑、失败占位、粉丝
|
||||
uv run python tests/test_snapshot.py # 七项快照、补采、断点续跑、粉丝、备份重试
|
||||
uv run python tests/test_maintenance.py # 日备份与保留期
|
||||
uv run python tests/test_timeutil.py # 上海时区与每日任务时间
|
||||
uv run python tests/test_scheduler.py # 内置调度器到期/失败隔离
|
||||
uv run python tests/test_provider.py # 凭据变化重建客户端与配置 revision
|
||||
uv run python tests/test_favsort_run.py # 排序参数校验与混合资源类型
|
||||
uv run python tests/test_logging.py # JSON 日志脱敏
|
||||
uv run python tests/test_cli.py # CLI 离线路径
|
||||
```
|
||||
|
||||
Reference in new issue
Block a user