Production evening seal recorded full-site snapshot as success with elapsed=0 / already_completed=true while hourly sync was overlapping. Mutual skip between snapshot and snapshot_recent was recorded as ok=true, then next_run advanced to the next day — so the real seal was missed until a manual --force.
Later: post-seed restart forced a daytime full scrape (clone punched early) and left the evening 23:15 seal skipped. Root cause: intent lived only in in-memory next_run.
## Background
Production evening seal recorded full-site `snapshot` as success with `elapsed=0` / `already_completed=true` while hourly sync was overlapping. Mutual skip between `snapshot` and `snapshot_recent` was recorded as `ok=true`, then `next_run` advanced to the next day — so the real seal was missed until a manual `--force`.
Later: post-seed restart forced a daytime full scrape (clone punched early) and left the evening 23:15 seal skipped. Root cause: intent lived only in in-memory `next_run`.
## Changes (!7 original)
1. **Full-site snapshot never skips** because another task is running (removed mutex).
2. **Quiet window 23:00–01:00 Asia/Shanghai** for discovery / snapshot_recent / candidate_cleanup.
3. **Fans** short-path (no quiet deferral / no mutex with snapshot).
4. **Skip ≠ success** (`ok=false` + `skipped=true`).
5. **Retry / makeup** via `TaskControl.next_run` (~5 min; seed ~10 min).
6. System UI + `/api/system` next_runs aligned.
7. **`snapshot-seed` CLI**.
8. Seed deferral when snapshot running / `no_source`.
## Review follow-ups (kept)
1. snapshot-seed: check `completed_key` before `no_source`.
2. `ok=False` → `scheduler.task_fail` (no false `task_ok`).
3. candidate_cleanup failure → ~5 min retry.
4. (superseded by #8) no daytime force-seal just because seeded.
## #8 on this branch — task state persistence
Implements https://git.ofcss.com/kcloud/re-xxm/issues/8 on tip `f408da93c20237bb7bcfc36e74346452fd02ac0f`.
- New `reaction_board/task_state.py`: kv `task:state:{name}` with `status` / `due_at` / `stat_date` / `progress` / `updated_at`.
- Scheduler startup uses `startup_next_run` (recover stale `running`, honor future `due_at`); **no** daily `next_run=0`.
- Seed success enqueues evening seal (`ensure_pending` with `due_at=snapshot_at`); daytime restart does not force-scrape.
- Seal path: `run_videos(..., force=True, keep_scraped=True)` punches `src=seed`, keeps non-seed ok=1 as progress; CLI `--force` still full refetch.
- Fail/skip/quiet → `failed`/`deferred` + future `due_at`; success → `done` + next wall/interval due.
- Web system next_runs prefer `task:state.due_at`.
## Test plan
- `uv run python tests/test_task_state.py`
- `uv run python tests/test_cli.py`
- `uv run python tests/test_scheduler.py`
- `uv run python tests/test_snapshot.py`
- `uv run python tests/test_timeutil.py`
- `uv run python tests/test_web_queries.py`
Do not merge until reviewed.
Incident: evening full snapshot could be recorded as ok with
already_completed/elapsed=0 after mutual skip with hourly sync, then
next_run jumped to the next day.
- Full-site snapshot never skips because another task is running
- 23:00-01:00 Asia/Shanghai: defer discovery / snapshot_recent /
candidate_cleanup to after 01:00; fans never defers
- Skip records ok=false + skipped=true (not success / not completed)
- Skip/fail reschedule soon (~5m); daily success still next snapshot_at
- TaskControl return value drives next_run; system UI shows skip fields
Add `python app.py snapshot-seed [--date]` (offline clone, Debian /
docker compose exec friendly). When seed fires while full-site snapshot
is running, or last_completed is missing (no_source), record skip as
non-success and retry in 10 minutes until a real seed succeeds; fans
and quiet-window rules unchanged.
Schedule snapshot_seed/snapshot/fans to the next wall-clock seed_at /
snapshot_at on process start (optionally from last task record), instead
of next_run=0. Force-seal of seed shells only after today's snapshot_at
(or CLI --force), so a daytime restart cannot scrape the whole site.
Replace process-start next_run=0 guesses with kv task:state records
(status, due_at, stat_date, progress). Startup recovers stale running,
honors future due_at for delayed retries, and resumes the same
stat_date. Seed only enqueues evening seal; seal force keeps non-seed
ok=1 as progress (keep_scraped). Keeps !7 fail-logging and cleanup retry.
- Cross-midnight: unfinished yesterday seals ASAP with yesterday's stat_date;
force seal ignores today's snapshot_at gate when stat_date < today.
- Off-plan already_completed: compare completion started_at (not finished_at)
to today's planned slot; earlier → still run at slot for all daily tasks.
- Cold start: bootstrap task:state from task:last / seed markers.
- ensure_pending: do not glue new due onto a different day's unfinished state.
- run_if_needed(force): keep_scraped True for seal, False for CLI --force.
- Scheduler due_reader: pick up disk due_at each loop (seed→evening seal).
tip f6bef59:按 #8 产品规则补齐跨天续跑、日更 off-plan(started_at vs 计划点)、冷启动回填、keep_scraped 分流、同进程 due_at 刷新。未合入。
tip [`f6bef59`](https://git.ofcss.com/kcloud/re-xxm/commit/f6bef59e073f096d7d112aac6537c6b56391bf51):按 #8 产品规则补齐跨天续跑、日更 off-plan(`started_at` vs 计划点)、冷启动回填、`keep_scraped` 分流、同进程 `due_at` 刷新。未合入。
kairee
merged commit 764bde65d9 into main2026-11-03 02:40:10 +08:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Background
Production evening seal recorded full-site
snapshotas success withelapsed=0/already_completed=truewhile hourly sync was overlapping. Mutual skip betweensnapshotandsnapshot_recentwas recorded asok=true, thennext_runadvanced to the next day — so the real seal was missed until a manual--force.Later: post-seed restart forced a daytime full scrape (clone punched early) and left the evening 23:15 seal skipped. Root cause: intent lived only in in-memory
next_run.Changes (!7 original)
ok=false+skipped=true).TaskControl.next_run(~5 min; seed ~10 min)./api/systemnext_runs aligned.snapshot-seedCLI.no_source.Review follow-ups (kept)
completed_keybeforeno_source.ok=False→scheduler.task_fail(no falsetask_ok).#8 on this branch — task state persistence
Implements #8 on tip
f408da93c20237bb7bcfc36e74346452fd02ac0f.reaction_board/task_state.py: kvtask:state:{name}withstatus/due_at/stat_date/progress/updated_at.startup_next_run(recover stalerunning, honor futuredue_at); no dailynext_run=0.ensure_pendingwithdue_at=snapshot_at); daytime restart does not force-scrape.run_videos(..., force=True, keep_scraped=True)punchessrc=seed, keeps non-seed ok=1 as progress; CLI--forcestill full refetch.failed/deferred+ futuredue_at; success →done+ next wall/interval due.task:state.due_at.Test plan
uv run python tests/test_task_state.pyuv run python tests/test_cli.pyuv run python tests/test_scheduler.pyuv run python tests/test_snapshot.pyuv run python tests/test_timeutil.pyuv run python tests/test_web_queries.pyDo not merge until reviewed.
@michal 测例 NEED-CHANGES 已补,请对 tip
dcf37c4a1d1afc05394de43c0f6a2c6e3fad84b3(dcf37c4)复审。H7
test_fail_or_defer_preserves_progress:先写入非空 progress,再mark_failed/mark_deferred(progress=None),load_state断言 progress +stat_date保留,只改 status +due_at。H8
test_defer_snapshot_seed_schedules_ten_minutes:task:statestatus=deferred,due_at ≈ finished+600test_quiet_window_defer_persists_deferred_state:discovery 静默窗 defer →deferred+due_at == next_after_snapshot_quiet_window,且不写成功完成类 kvSoft
test_seeded_before_snapshot_at_does_not_force_run:due_at 刻意用 22:00 ≠ 墙钟 23:15test_candidate_cleanup_failure_retries_soon:兼读task:statefailed + due_at本地:
uv run python tests/test_task_state.py/uv run python tests/test_cli.py均 OK。未 merge。位置:reaction_board/cli.py:370-388、reaction_board/task_state.py:272-275
_initial_daily_next_run() 现在只认 task:state。旧版本数据库只有 task:last:*,本分支没有迁移或回填,因此升级后的第一次启动:
实际复现:写入一条 task:last:snapshot 的失败记录,重启时间为 2026-11-03 23:25,下一次全站快照被排到:
2026-11-04 23:15
这正是原本要修的事故:23:15 的失败/未完成 seal 在升级重启后被推迟到次日。首次安装如果在 23:15 之后启动,也同样会跳过当天快照。
建议:启动时从 task:last、snapshot:last_completed、snapshot:completed: 引导 task:state,或在 _initial_daily_next_run() 保留旧的 task:last 回退;当天未完成且已过 snapshot_at 时应立即补跑。补充“只有 task:last、没有 task:state”的升级测试。
位置:reaction_board/task_state.py:227-238、reaction_board/cli.py:1009-1064
ensure_pending() 在已有未完成状态时执行:
due = min(prev.due_at, due_at)
sd = prev.stat_date or stat_date
这会把昨天失败/延后的 due_at 和 stat_date 原样保留。例如:
随后 resume_stat_date() 返回 2026-11-02,今晚 23:15 的 run_snapshot() 会 force 昨天,而不是今天;今天的 seed 壳可能永远不落 seal。
更直接的跨午夜场景:昨天 23:15 的 seal 在 00:10 重启:
新增的 test_restart_mid_run_resumes_same_stat_date 只验证了 state 字段,没有走真实的 run_snapshot() 路径,因此没有覆盖这个缺陷。
建议:ensure_pending() 在 prev.stat_date != stat_date 时应重置为新的 due_at/stat_date;跨午夜恢复的 seal 不能再用当前墙钟 snapshot_at 判定为“白天 seed 壳”,应允许它继续完成昨天的 seal;同时补充“23:15 运行中崩溃、00:10 重启”的回归测试。
位置:reaction_board/services/snapshot.py:157-161、reaction_board/cli.py:538
run_if_needed(force=True) 现在固定调用:
run_videos(date=stat_date, force=True, keep_scraped=True)
keep_scraped=True 会跳过所有非 seed 的 ok=1 行,而 CLI 的 snapshot --force 正是通过 run_if_needed(force=args.force) 执行。实测存在已采集行时:
run_if_needed(force=True): skipped=1 success=0 view_calls=0
run_videos(force=True): skipped=0 success=1 view_calls=1
这与 run_videos() 文档中“CLI --force 不传此标志,会重采全部成功行”的说明直接矛盾,也会让手动 --force 恢复失效。
建议:给 run_if_needed() 增加 keep_scraped 参数,默认 False;调度器 seal 传 True,CLI --force 传 False。补充真实 run_if_needed(force=True) 的重采用例。
位置:reaction_board/cli.py:423-428、reaction_board/cli.py:1272-1289
_enqueue_evening_seal() 只写 task:state,但 Scheduler 的 next_run 只在 cmd_run 启动时读取一次。若 seed 在 snapshot_at 之后成功(例如手动 snapshot-seed 或延迟重试),state 会写入 due_at=now 的 pending seal,但当前进程的 snapshot.next_run 仍是次日 23:15,今天不会触发。
建议:调度循环每轮重新核对 task:state.due_at,或在 _enqueue_evening_seal() 后同步更新对应 ScheduledTask.next_run。
tip
f6bef59:按 #8 产品规则补齐跨天续跑、日更 off-plan(started_atvs 计划点)、冷启动回填、keep_scraped分流、同进程due_at刷新。未合入。