fix(scheduler): no false skip for full snapshot; quiet window + retry #7

Merged
kairee merged 7 commits from auto/sched-no-false-skip-4821 into main 2026-11-03 02:40:10 +08:00

Background

Production evening seal recorded full-site snapshot as success with elapsed=0 / already_completed=true while hourly sync was overlapping. Mutual skip between snapshot and snapshot_recent was recorded as ok=true, then next_run advanced to the next day — so the real seal was missed until a manual --force.

Later: post-seed restart forced a daytime full scrape (clone punched early) and left the evening 23:15 seal skipped. Root cause: intent lived only in in-memory next_run.

Changes (!7 original)

  1. Full-site snapshot never skips because another task is running (removed mutex).
  2. Quiet window 23:00–01:00 Asia/Shanghai for discovery / snapshot_recent / candidate_cleanup.
  3. Fans short-path (no quiet deferral / no mutex with snapshot).
  4. Skip ≠ success (ok=false + skipped=true).
  5. Retry / makeup via TaskControl.next_run (~5 min; seed ~10 min).
  6. System UI + /api/system next_runs aligned.
  7. snapshot-seed CLI.
  8. Seed deferral when snapshot running / no_source.

Review follow-ups (kept)

  1. snapshot-seed: check completed_key before no_source.
  2. ok=False → scheduler.task_fail (no false task_ok).
  3. candidate_cleanup failure → ~5 min retry.
  4. (superseded by #8) no daytime force-seal just because seeded.

#8 on this branch — task state persistence

Implements #8 on tip f408da93c20237bb7bcfc36e74346452fd02ac0f.

  • New reaction_board/task_state.py: kv task:state:{name} with status / due_at / stat_date / progress / updated_at.
  • Scheduler startup uses startup_next_run (recover stale running, honor future due_at); no daily next_run=0.
  • Seed success enqueues evening seal (ensure_pending with due_at=snapshot_at); daytime restart does not force-scrape.
  • Seal path: run_videos(..., force=True, keep_scraped=True) punches src=seed, keeps non-seed ok=1 as progress; CLI --force still full refetch.
  • Fail/skip/quiet → failed/deferred + future due_at; success → done + next wall/interval due.
  • Web system next_runs prefer task:state.due_at.

Test plan

  • uv run python tests/test_task_state.py
  • uv run python tests/test_cli.py
  • uv run python tests/test_scheduler.py
  • uv run python tests/test_snapshot.py
  • uv run python tests/test_timeutil.py
  • uv run python tests/test_web_queries.py

Do not merge until reviewed.

## Background Production evening seal recorded full-site `snapshot` as success with `elapsed=0` / `already_completed=true` while hourly sync was overlapping. Mutual skip between `snapshot` and `snapshot_recent` was recorded as `ok=true`, then `next_run` advanced to the next day — so the real seal was missed until a manual `--force`. Later: post-seed restart forced a daytime full scrape (clone punched early) and left the evening 23:15 seal skipped. Root cause: intent lived only in in-memory `next_run`. ## Changes (!7 original) 1. **Full-site snapshot never skips** because another task is running (removed mutex). 2. **Quiet window 23:00–01:00 Asia/Shanghai** for discovery / snapshot_recent / candidate_cleanup. 3. **Fans** short-path (no quiet deferral / no mutex with snapshot). 4. **Skip ≠ success** (`ok=false` + `skipped=true`). 5. **Retry / makeup** via `TaskControl.next_run` (~5 min; seed ~10 min). 6. System UI + `/api/system` next_runs aligned. 7. **`snapshot-seed` CLI**. 8. Seed deferral when snapshot running / `no_source`. ## Review follow-ups (kept) 1. snapshot-seed: check `completed_key` before `no_source`. 2. `ok=False` → `scheduler.task_fail` (no false `task_ok`). 3. candidate_cleanup failure → ~5 min retry. 4. (superseded by #8) no daytime force-seal just because seeded. ## #8 on this branch — task state persistence Implements https://git.ofcss.com/kcloud/re-xxm/issues/8 on tip `f408da93c20237bb7bcfc36e74346452fd02ac0f`. - New `reaction_board/task_state.py`: kv `task:state:{name}` with `status` / `due_at` / `stat_date` / `progress` / `updated_at`. - Scheduler startup uses `startup_next_run` (recover stale `running`, honor future `due_at`); **no** daily `next_run=0`. - Seed success enqueues evening seal (`ensure_pending` with `due_at=snapshot_at`); daytime restart does not force-scrape. - Seal path: `run_videos(..., force=True, keep_scraped=True)` punches `src=seed`, keeps non-seed ok=1 as progress; CLI `--force` still full refetch. - Fail/skip/quiet → `failed`/`deferred` + future `due_at`; success → `done` + next wall/interval due. - Web system next_runs prefer `task:state.due_at`. ## Test plan - `uv run python tests/test_task_state.py` - `uv run python tests/test_cli.py` - `uv run python tests/test_scheduler.py` - `uv run python tests/test_snapshot.py` - `uv run python tests/test_timeutil.py` - `uv run python tests/test_web_queries.py` Do not merge until reviewed.
bot-ci Bot added 1 commit 2026-11-03 00:07:48 +08:00
Incident: evening full snapshot could be recorded as ok with
already_completed/elapsed=0 after mutual skip with hourly sync, then
next_run jumped to the next day.

- Full-site snapshot never skips because another task is running
- 23:00-01:00 Asia/Shanghai: defer discovery / snapshot_recent /
  candidate_cleanup to after 01:00; fans never defers
- Skip records ok=false + skipped=true (not success / not completed)
- Skip/fail reschedule soon (~5m); daily success still next snapshot_at
- TaskControl return value drives next_run; system UI shows skip fields
bot-ci Bot added 1 commit 2026-11-03 00:22:20 +08:00
Add `python app.py snapshot-seed [--date]` (offline clone, Debian /
docker compose exec friendly). When seed fires while full-site snapshot
is running, or last_completed is missing (no_source), record skip as
non-success and retry in 10 minutes until a real seed succeeds; fans
and quiet-window rules unchanged.
kairee requested changes 2026-11-03 01:04:37 +08:00
Dismissed
kairee left a comment
Owner
  1. snapshot-seed 会把“当天已完成”的正常情形误判为 no_source,并持续 10 分钟重试。
  • 位置:reaction_board/services/snapshot.py:541-550、reaction_board/cli.py:895-910
  • run_seed_from_last_completed() 先判断 source >= stat_date,而全站快照当天完成后的正常状态就是:
  • snapshot:last_completed == stat_date
  • snapshot:completed:<stat_date> == "1"
  • 因为检查顺序,snapshot:completed 的 already_completed 分支永远到不了,手动执行 snapshot-seed 会返回非 0;服务在重启后也会把这个正常的无需 seed 状态记作跳过,并每 10 分钟再试一次,直到跨日。
  • 建议:先检查目标日的 completed_key(stat_date),已完成则返回 reason="already_completed" 的成功结果;只有目标日未完成时,才把缺失或不早于目标日的 last_completed 判为 no_source。补充回归测试覆盖 last_completed == target_date 且目标日已完成。
  1. 返回 TaskControl(ok=False) 的任务仍被调度器记录为 scheduler.task_ok。
  • 位置:reaction_board/scheduler.py:91-106
  • 当前实现在线 94 无条件写入 scheduler.task_ok,之后才依据 TaskControl.ok 将 TaskResult.ok 改为 False。因此 quiet-window skip、seed 的 no_source、快照业务失败等路径,会同时产生“任务成功”的调度日志。
  • 这与本 PR 的“Skip ≠ success”目标冲突,也会误导基于结构化日志的排障或监控。
  • 建议:在解析 TaskControl、确定最终 ok/error 后再记录日志;ok=False 时写失败/跳过对应事件,而非 scheduler.task_ok。补充带 logger 的调度器测试。
  1. candidate_cleanup 的失败不会按 PR 所述约 5 分钟补跑。
  • 位置:reaction_board/cli.py:727-755、reaction_board/scheduler.py:96-101
  • run_cleanup() 成功时返回 TaskControl,但未捕获异常并返回失败控制结果。数据库操作异常会由 Scheduler 捕获,再按该任务的默认 86400s interval 重排,而不是短间隔重试;同时不会写入该任务的失败记录。
  • 这与 PR 描述及 DEVELOPMENT.md 中“skip / fail 后短间隔补跑”的口径不一致。
  • 建议:参照 run_discover(),捕获异常、调用 _record_task() 并返回 TaskControl(ok=False, next_run=_retry_soon(...));新增失败补跑测试。
1. snapshot-seed 会把“当天已完成”的正常情形误判为 no_source,并持续 10 分钟重试。 - 位置:reaction_board/services/snapshot.py:541-550、reaction_board/cli.py:895-910 - run_seed_from_last_completed() 先判断 source >= stat_date,而全站快照当天完成后的正常状态就是: - snapshot:last_completed == stat_date - snapshot:completed:<stat_date> == "1" - 因为检查顺序,snapshot:completed 的 already_completed 分支永远到不了,手动执行 snapshot-seed 会返回非 0;服务在重启后也会把这个正常的无需 seed 状态记作跳过,并每 10 分钟再试一次,直到跨日。 - 建议:先检查目标日的 completed_key(stat_date),已完成则返回 reason="already_completed" 的成功结果;只有目标日未完成时,才把缺失或不早于目标日的 last_completed 判为 no_source。补充回归测试覆盖 last_completed == target_date 且目标日已完成。 2. 返回 TaskControl(ok=False) 的任务仍被调度器记录为 scheduler.task_ok。 - 位置:reaction_board/scheduler.py:91-106 - 当前实现在线 94 无条件写入 scheduler.task_ok,之后才依据 TaskControl.ok 将 TaskResult.ok 改为 False。因此 quiet-window skip、seed 的 no_source、快照业务失败等路径,会同时产生“任务成功”的调度日志。 - 这与本 PR 的“Skip ≠ success”目标冲突,也会误导基于结构化日志的排障或监控。 - 建议:在解析 TaskControl、确定最终 ok/error 后再记录日志;ok=False 时写失败/跳过对应事件,而非 scheduler.task_ok。补充带 logger 的调度器测试。 3. candidate_cleanup 的失败不会按 PR 所述约 5 分钟补跑。 - 位置:reaction_board/cli.py:727-755、reaction_board/scheduler.py:96-101 - run_cleanup() 成功时返回 TaskControl,但未捕获异常并返回失败控制结果。数据库操作异常会由 Scheduler 捕获,再按该任务的默认 86400s interval 重排,而不是短间隔重试;同时不会写入该任务的失败记录。 - 这与 PR 描述及 DEVELOPMENT.md 中“skip / fail 后短间隔补跑”的口径不一致。 - 建议:参照 run_discover(),捕获异常、调用 _record_task() 并返回 TaskControl(ok=False, next_run=_retry_soon(...));新增失败补跑测试。
bot-ci Bot added 2 commits 2026-11-03 01:09:55 +08:00
- seed: check completed_key before no_source so same-day last_completed
  is already_completed (not 10m retry)
- scheduler: resolve TaskControl ok/error before logging; ok=False logs
  task_fail, not task_ok
- candidate_cleanup: catch failures, record task, retry ~5m like discover
Schedule snapshot_seed/snapshot/fans to the next wall-clock seed_at /
snapshot_at on process start (optionally from last task record), instead
of next_run=0. Force-seal of seed shells only after today's snapshot_at
(or CLI --force), so a daytime restart cannot scrape the whole site.
kairee approved these changes 2026-11-03 01:23:06 +08:00
bot-ci Bot added 1 commit 2026-11-03 01:50:12 +08:00
Replace process-start next_run=0 guesses with kv task:state records
(status, due_at, stat_date, progress). Startup recovers stale running,
honors future due_at for delayed retries, and resumes the same
stat_date. Seed only enqueues evening seal; seal force keeps non-seed
ok=1 as progress (keep_scraped). Keeps !7 fail-logging and cleanup retry.
bot-ci Bot added 1 commit 2026-11-03 01:59:26 +08:00
Address Michał NEED-CHANGES on !7 tip f408da93: fail/defer omit-progress
preserves progress/stat_date; seed + quiet-window defer assert
status=deferred and future due_at on task:state. Soft: seeded due≠wall;
cleanup failure reads task:state.

@michal 测例 NEED-CHANGES 已补,请对 tip dcf37c4a1d1afc05394de43c0f6a2c6e3fad84b3(dcf37c4)复审。

H7 test_fail_or_defer_preserves_progress:先写入非空 progress,再 mark_failed / mark_deferred(progress=None),load_state 断言 progress + stat_date 保留,只改 status + due_at。

H8

  • 扩展 test_defer_snapshot_seed_schedules_ten_minutes:task:state status=deferred,due_at ≈ finished+600
  • 新增 test_quiet_window_defer_persists_deferred_state:discovery 静默窗 defer → deferred + due_at == next_after_snapshot_quiet_window,且不写成功完成类 kv

Soft

  • test_seeded_before_snapshot_at_does_not_force_run:due_at 刻意用 22:00 ≠ 墙钟 23:15
  • test_candidate_cleanup_failure_retries_soon:兼读 task:state failed + due_at

本地:uv run python tests/test_task_state.py / uv run python tests/test_cli.py 均 OK。未 merge。

@michal 测例 NEED-CHANGES 已补,请对 tip `dcf37c4a1d1afc05394de43c0f6a2c6e3fad84b3`(`dcf37c4`)复审。 **H7** `test_fail_or_defer_preserves_progress`:先写入非空 progress,再 `mark_failed` / `mark_deferred`(`progress=None`),`load_state` 断言 progress + `stat_date` 保留,只改 status + `due_at`。 **H8** - 扩展 `test_defer_snapshot_seed_schedules_ten_minutes`:`task:state` `status=deferred`,`due_at ≈ finished+600` - 新增 `test_quiet_window_defer_persists_deferred_state`:discovery 静默窗 defer → `deferred` + `due_at == next_after_snapshot_quiet_window`,且不写成功完成类 kv **Soft** - `test_seeded_before_snapshot_at_does_not_force_run`:due_at 刻意用 22:00 ≠ 墙钟 23:15 - `test_candidate_cleanup_failure_retries_soon`:兼读 `task:state` failed + due_at 本地:`uv run python tests/test_task_state.py` / `uv run python tests/test_cli.py` 均 OK。未 merge。
Owner
  1. 升级/首次启动没有把旧 task:last 迁移到 task:state,原始事故仍会重现
    位置:reaction_board/cli.py:370-388、reaction_board/task_state.py:272-275
    _initial_daily_next_run() 现在只认 task:state。旧版本数据库只有 task:last:*,本分支没有迁移或回填,因此升级后的第一次启动:
  • task:state 为空;
  • startup_next_run() 直接返回 wall_due_at,即下一个 snapshot_at。
    实际复现:写入一条 task:last:snapshot 的失败记录,重启时间为 2026-11-03 23:25,下一次全站快照被排到:
    2026-11-04 23:15
    这正是原本要修的事故:23:15 的失败/未完成 seal 在升级重启后被推迟到次日。首次安装如果在 23:15 之后启动,也同样会跳过当天快照。
    建议:启动时从 task:last、snapshot:last_completed、snapshot:completed: 引导 task:state,或在 _initial_daily_next_run() 保留旧的 task:last 回退;当天未完成且已过 snapshot_at 时应立即补跑。补充“只有 task:last、没有 task:state”的升级测试。
  1. 跨午夜恢复/跨日 active 状态会把昨天带进今天
    位置:reaction_board/task_state.py:227-238、reaction_board/cli.py:1009-1064
    ensure_pending() 在已有未完成状态时执行:
    due = min(prev.due_at, due_at)
    sd = prev.stat_date or stat_date
    这会把昨天失败/延后的 due_at 和 stat_date 原样保留。例如:
  • 昨天 snapshot 失败,state 为 stat_date=2026-11-02;
  • 今天 00:01 seed 成功,ensure_pending() 想挂今天的 seal;
  • 结果 state 仍是 due_at=2026-11-02 23:20、stat_date=2026-11-02。
    随后 resume_stat_date() 返回 2026-11-02,今晚 23:15 的 run_snapshot() 会 force 昨天,而不是今天;今天的 seed 壳可能永远不落 seal。
    更直接的跨午夜场景:昨天 23:15 的 seal 在 00:10 重启:
  • state 已恢复为 pending,stat_date=2026-11-02;
  • 当前时间 00:10 未到 23:15,run_snapshot() 的 force_seed 为 false;
  • run_if_needed(date=2026-11-02) 返回 reason="seeded_pending_seal";
  • 代码把下一次运行推到 2026-11-03 23:15,并保留 stat_date=2026-11-02;
  • 今晚 23:15 会 force 昨天,今天的数据反而没有 seal。
    新增的 test_restart_mid_run_resumes_same_stat_date 只验证了 state 字段,没有走真实的 run_snapshot() 路径,因此没有覆盖这个缺陷。
    建议:ensure_pending() 在 prev.stat_date != stat_date 时应重置为新的 due_at/stat_date;跨午夜恢复的 seal 不能再用当前墙钟 snapshot_at 判定为“白天 seed 壳”,应允许它继续完成昨天的 seal;同时补充“23:15 运行中崩溃、00:10 重启”的回归测试。
  1. CLI snapshot --force 不再强制重采
    位置:reaction_board/services/snapshot.py:157-161、reaction_board/cli.py:538
    run_if_needed(force=True) 现在固定调用:
    run_videos(date=stat_date, force=True, keep_scraped=True)
    keep_scraped=True 会跳过所有非 seed 的 ok=1 行,而 CLI 的 snapshot --force 正是通过 run_if_needed(force=args.force) 执行。实测存在已采集行时:
    run_if_needed(force=True): skipped=1 success=0 view_calls=0
    run_videos(force=True): skipped=0 success=1 view_calls=1
    这与 run_videos() 文档中“CLI --force 不传此标志,会重采全部成功行”的说明直接矛盾,也会让手动 --force 恢复失效。
    建议:给 run_if_needed() 增加 keep_scraped 参数,默认 False;调度器 seal 传 True,CLI --force 传 False。补充真实 run_if_needed(force=True) 的重采用例。
  2. 运行期 task:state.due_at 不会同步到当前 Scheduler
    位置:reaction_board/cli.py:423-428、reaction_board/cli.py:1272-1289
    _enqueue_evening_seal() 只写 task:state,但 Scheduler 的 next_run 只在 cmd_run 启动时读取一次。若 seed 在 snapshot_at 之后成功(例如手动 snapshot-seed 或延迟重试),state 会写入 due_at=now 的 pending seal,但当前进程的 snapshot.next_run 仍是次日 23:15,今天不会触发。
    建议:调度循环每轮重新核对 task:state.due_at,或在 _enqueue_evening_seal() 后同步更新对应 ScheduledTask.next_run。
1. 升级/首次启动没有把旧 task:last 迁移到 task:state,原始事故仍会重现 位置:reaction_board/cli.py:370-388、reaction_board/task_state.py:272-275 _initial_daily_next_run() 现在只认 task:state。旧版本数据库只有 task:last:*,本分支没有迁移或回填,因此升级后的第一次启动: - task:state 为空; - startup_next_run() 直接返回 wall_due_at,即下一个 snapshot_at。 实际复现:写入一条 task:last:snapshot 的失败记录,重启时间为 2026-11-03 23:25,下一次全站快照被排到: 2026-11-04 23:15 这正是原本要修的事故:23:15 的失败/未完成 seal 在升级重启后被推迟到次日。首次安装如果在 23:15 之后启动,也同样会跳过当天快照。 建议:启动时从 task:last、snapshot:last_completed、snapshot:completed:<today> 引导 task:state,或在 _initial_daily_next_run() 保留旧的 task:last 回退;当天未完成且已过 snapshot_at 时应立即补跑。补充“只有 task:last、没有 task:state”的升级测试。 2. 跨午夜恢复/跨日 active 状态会把昨天带进今天 位置:reaction_board/task_state.py:227-238、reaction_board/cli.py:1009-1064 ensure_pending() 在已有未完成状态时执行: due = min(prev.due_at, due_at) sd = prev.stat_date or stat_date 这会把昨天失败/延后的 due_at 和 stat_date 原样保留。例如: - 昨天 snapshot 失败,state 为 stat_date=2026-11-02; - 今天 00:01 seed 成功,ensure_pending() 想挂今天的 seal; - 结果 state 仍是 due_at=2026-11-02 23:20、stat_date=2026-11-02。 随后 resume_stat_date() 返回 2026-11-02,今晚 23:15 的 run_snapshot() 会 force 昨天,而不是今天;今天的 seed 壳可能永远不落 seal。 更直接的跨午夜场景:昨天 23:15 的 seal 在 00:10 重启: - state 已恢复为 pending,stat_date=2026-11-02; - 当前时间 00:10 未到 23:15,run_snapshot() 的 force_seed 为 false; - run_if_needed(date=2026-11-02) 返回 reason="seeded_pending_seal"; - 代码把下一次运行推到 2026-11-03 23:15,并保留 stat_date=2026-11-02; - 今晚 23:15 会 force 昨天,今天的数据反而没有 seal。 新增的 test_restart_mid_run_resumes_same_stat_date 只验证了 state 字段,没有走真实的 run_snapshot() 路径,因此没有覆盖这个缺陷。 建议:ensure_pending() 在 prev.stat_date != stat_date 时应重置为新的 due_at/stat_date;跨午夜恢复的 seal 不能再用当前墙钟 snapshot_at 判定为“白天 seed 壳”,应允许它继续完成昨天的 seal;同时补充“23:15 运行中崩溃、00:10 重启”的回归测试。 3. CLI snapshot --force 不再强制重采 位置:reaction_board/services/snapshot.py:157-161、reaction_board/cli.py:538 run_if_needed(force=True) 现在固定调用: run_videos(date=stat_date, force=True, keep_scraped=True) keep_scraped=True 会跳过所有非 seed 的 ok=1 行,而 CLI 的 snapshot --force 正是通过 run_if_needed(force=args.force) 执行。实测存在已采集行时: run_if_needed(force=True): skipped=1 success=0 view_calls=0 run_videos(force=True): skipped=0 success=1 view_calls=1 这与 run_videos() 文档中“CLI --force 不传此标志,会重采全部成功行”的说明直接矛盾,也会让手动 --force 恢复失效。 建议:给 run_if_needed() 增加 keep_scraped 参数,默认 False;调度器 seal 传 True,CLI --force 传 False。补充真实 run_if_needed(force=True) 的重采用例。 4. 运行期 task:state.due_at 不会同步到当前 Scheduler 位置:reaction_board/cli.py:423-428、reaction_board/cli.py:1272-1289 _enqueue_evening_seal() 只写 task:state,但 Scheduler 的 next_run 只在 cmd_run 启动时读取一次。若 seed 在 snapshot_at 之后成功(例如手动 snapshot-seed 或延迟重试),state 会写入 due_at=now 的 pending seal,但当前进程的 snapshot.next_run 仍是次日 23:15,今天不会触发。 建议:调度循环每轮重新核对 task:state.due_at,或在 _enqueue_evening_seal() 后同步更新对应 ScheduledTask.next_run。
bot-ci Bot added 1 commit 2026-11-03 02:35:27 +08:00
- Cross-midnight: unfinished yesterday seals ASAP with yesterday's stat_date;
  force seal ignores today's snapshot_at gate when stat_date < today.
- Off-plan already_completed: compare completion started_at (not finished_at)
  to today's planned slot; earlier → still run at slot for all daily tasks.
- Cold start: bootstrap task:state from task:last / seed markers.
- ensure_pending: do not glue new due onto a different day's unfinished state.
- run_if_needed(force): keep_scraped True for seal, False for CLI --force.
- Scheduler due_reader: pick up disk due_at each loop (seed→evening seal).

tip f6bef59:按 #8 产品规则补齐跨天续跑、日更 off-plan(started_at vs 计划点)、冷启动回填、keep_scraped 分流、同进程 due_at 刷新。未合入。

tip [`f6bef59`](https://git.ofcss.com/kcloud/re-xxm/commit/f6bef59e073f096d7d112aac6537c6b56391bf51):按 #8 产品规则补齐跨天续跑、日更 off-plan(`started_at` vs 计划点)、冷启动回填、`keep_scraped` 分流、同进程 `due_at` 刷新。未合入。
kairee merged commit 764bde65d9 into main 2026-11-03 02:40:10 +08:00
kairee deleted branch auto/sched-no-false-skip-4821 2026-11-03 02:40:19 +08:00
Sign in to join this conversation.
No Reviewers
No labels
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kcloud/re-xxm#7