feat: EOD 23:15 seal + midnight seed shell for video/fans
Move daily snapshot/fans to 23:15 and add 00:01 seed that clones last_completed ok=1 rows into today without advancing completed. Evening runs force through seed markers; growth previous is per-aid adjacent. Home shows short-update clock + 全站数据 date, and 收录视频 counts.
This commit is contained in:
1 parent
374b138041
commit
8a9cb21408
17 files changed
+520
-39
No files matched your search
+6
-6
@@ -117,7 +117,7 @@ Telegram 私聊发链接/BV 号 ───┘
|
||||
145178393, 66389997, 293976264, 3026109
|
||||
],
|
||||
"schedule": { "discover_minutes": 60,
|
||||
"snapshot_at": "00:01", "candidate_ttl_days": 7 },
|
||||
"snapshot_at": "23:15", "seed_at": "00:01", "candidate_ttl_days": 7 },
|
||||
"discover": { "keywords": ["咻咻满"], "pages": 3, "window_days": 7,
|
||||
"candidate_min_score": 4 },
|
||||
"notify": { "min_score": 4, "max_items_per_message": 10 },
|
||||
@@ -377,7 +377,7 @@ for folder in 配置的年度夹(2025、2026、…):
|
||||
### 5.1 任务
|
||||
|
||||
```
|
||||
每天 00:01:00(Asia/Shanghai):
|
||||
每天 23:15(Asia/Shanghai)全量落盘;00:01 克隆当日壳:
|
||||
videos = db.where(status='active') # 跨所有年度夹
|
||||
for v in videos:
|
||||
if exists(snapshot[today, v.aid]) and ok: continue # 断点续跑
|
||||
@@ -397,13 +397,13 @@ for folder in 配置的年度夹(2025、2026、…):
|
||||
| 1600(2027 年末) | 1600 | 约 33 分钟 |
|
||||
| 3000(2029 年) | 3000 | 约 60 分钟 |
|
||||
|
||||
- 00:01 启动,最晚 01:00 前完成,不影响白天使用;
|
||||
- 23:15 启动全量落盘;00:01 仅本地克隆壳,不影响白天使用;
|
||||
- 若单条 0.1s + sleep 1.2s ≈ 1.3s,实际按 1.2s 估算偏保守。
|
||||
|
||||
### 5.3 幂等与口径
|
||||
|
||||
- 唯一键 `(stat_date, aid)`,`INSERT ... ON CONFLICT DO UPDATE`;
|
||||
- `stat_date` 用本地日期,不用 UTC(避免 00:01 落到前一天);
|
||||
- `stat_date` 用本地日期,不用 UTC(避免日界落到前一天);
|
||||
- 失败重试 3 次后写 `ok=0` 占位,**不写假数据**;看板显示"缺失"而不是 0;
|
||||
- 完成标记之外还要检查当天待采集合:新入库/恢复视频立即补采,`ok=0` 项在续跑时重试;
|
||||
- 强制重采开始前先写未完成标记,中断后仍可续跑;
|
||||
@@ -412,7 +412,7 @@ for folder in 配置的年度夹(2025、2026、…):
|
||||
### 5.4 粉丝数追踪(新增)
|
||||
|
||||
```
|
||||
每天 00:01 与视频快照同一个任务里,追加 1 次请求:
|
||||
每天 23:15 与视频快照同一个调度窗口里,追加粉丝请求:
|
||||
r = GET x/relation/stat?vmid=37754047 # 匿名可用,实测 0.1s
|
||||
upsert(up_stat, today, mid=37754047, follower=r.follower, following=r.following)
|
||||
```
|
||||
@@ -782,7 +782,7 @@ config.docker.example.json 服务器配置样例(0.0.0.0 + /data 绝对路径
|
||||
**进程结构**
|
||||
|
||||
```
|
||||
scheduler 线程:discovery(每 60min) / snapshot(每日 00:01) / candidate_cleanup(每日)
|
||||
scheduler 线程:discovery(每 60min) / snapshot_seed(每日 00:01) / snapshot+fans(每日 23:15) / snapshot_recent(小时) / candidate_cleanup(每日)
|
||||
web 线程:ThreadingHTTPServer :8080
|
||||
telegram:getUpdates long polling(回调 + 私聊文本入库)
|
||||
storage:SQLite WAL,单写线程
|
||||
|
||||
Reference in new issue
Block a user