feat: EOD 23:15 seal + midnight seed shell for video/fans
Move daily snapshot/fans to 23:15 and add 00:01 seed that clones last_completed ok=1 rows into today without advancing completed. Evening runs force through seed markers; growth previous is per-aid adjacent. Home shows short-update clock + 全站数据 date, and 收录视频 counts.
This commit is contained in:
1 parent
374b138041
commit
8a9cb21408
17 files changed
+520
-39
No files matched your search
+7
-4
@@ -294,7 +294,8 @@ PRAGMA synchronous = NORMAL;
|
||||
"up_blacklist_seed": [261052419, 1908537643, 1701121088, 542262],
|
||||
"schedule": {
|
||||
"discover_minutes": 60,
|
||||
"snapshot_at": "00:01",
|
||||
"snapshot_at": "23:15",
|
||||
"seed_at": "00:01",
|
||||
"candidate_ttl_days": 7
|
||||
},
|
||||
"notify": { "min_score": 2, "max_items_per_message": 10 }
|
||||
@@ -308,7 +309,7 @@ PRAGMA synchronous = NORMAL;
|
||||
### 5.2 安全规则- Windows 本地开发时应限制 `config.json` 的文件访问权限;部署到 Linux 时权限必须为 `600`;
|
||||
- 配置文件每 60 秒检查一次 mtime,凭据、分类规则和权重变更后无需重启;
|
||||
- Cookie / CSRF 变化时由 `ClientProvider` 重建 B 站请求客户端,并保留该角色的限速与重试策略;
|
||||
- `schedule.discover_minutes` 与 `schedule.snapshot_at` 在启动时固化,修改后需重启 `run`;
|
||||
- `schedule.discover_minutes`、`schedule.snapshot_at` 与 `schedule.seed_at` 在启动时固化,修改后需重启 `run`;
|
||||
- Telegram 操作必须同时验证 `from.id` 和 `chat.id` 是否为配置的私聊用户;
|
||||
- 仅允许环境变量临时覆盖敏感配置,不能将真实凭据硬编码进源代码;
|
||||
- `GET` 路由绝不进行入库、忽略、拉黑等写操作。
|
||||
@@ -434,7 +435,7 @@ P3 实现说明:
|
||||
|
||||
### 7.4 每日快照与粉丝采集
|
||||
|
||||
每天 `00:01`(`Asia/Shanghai`)执行:
|
||||
每天 `23:15`(`Asia/Shanghai`)执行全量落盘;`00:01` 用上一完成日克隆当日壳(seed):
|
||||
|
||||
- 对 `status IN ('active', 'unfaved')` 的视频逐条请求 `view?aid=`;
|
||||
- 成功写入播放、点赞、收藏、投币、转发、评论、弹幕七项数据,并把 `view` 的 `owner.face` 写入 `video.up_face`(UP 头像,供 UP 列表/详情展示);
|
||||
@@ -450,7 +451,9 @@ P3 实现说明:
|
||||
P1 实现说明:
|
||||
|
||||
- 时区使用固定 UTC+8(中国无夏令时),避免 Windows 缺少 `tzdata`;
|
||||
- 当日全部视频与粉丝采集成功后在 `kv` 写入 `snapshot:completed:<date>=1`;启动时未完成会补跑;
|
||||
- 午夜 `seed_at`(默认 `00:01`)把 `snapshot:last_completed` 的 ok=1 行克隆为当日壳,写入 `snapshot:seeded:<date>=1` / `fans:seeded:<date>=1`,**不**推进完成日;
|
||||
- 晚间 `snapshot_at`(默认 `23:15`)对 seeded 日 force 重采;成功后写 `snapshot:completed:<date>=1`、推进 `last_completed` 并清除 seeded;
|
||||
- 当日全部视频与粉丝采集成功后在 `kv` 写入 `snapshot:completed:<date>=1`;启动时未完成会补跑(纯 seed 日不误标完成);
|
||||
- 完成标记不是唯一门槛:每次检查 `pending_snapshot_aids(date)`,新入库或从 `dropped` 恢复的视频会立即补采当天快照;
|
||||
- `ok=0` 的当日记录会在下次补跑时重试;`ok=1` 直接跳过;粉丝同样跳过已成功的 `(stat_date, mid)`;
|
||||
- 开始采集前先写 `snapshot:completed:<date>=0`,强制重采中断后不会残留旧的完成标记;
|
||||
|
||||
Reference in new issue
Block a user