fix(scheduler): no false skip for full snapshot; quiet window + retry #7

Open

bot-ci Bot wants to merge 6 commits from auto/sched-no-false-skip-4821 into main

pull from: auto/sched-no-false-skip-4821

merge into: kcloud:main

kcloud:main

Conversation 3 Commits 6 Files Changed 15

+2684 -503

bot-ci Bot commented 2026-11-03 00:07:48 +08:00

Copy link

Copy source

Background

Production evening seal recorded full-site snapshot as success with elapsed=0 / already_completed=true while hourly sync was overlapping. Mutual skip between snapshot and snapshot_recent was recorded as ok=true, then next_run advanced to the next day — so the real seal was missed until a manual --force.

Later: post-seed restart forced a daytime full scrape (clone punched early) and left the evening 23:15 seal skipped. Root cause: intent lived only in in-memory next_run.

Changes (!7 original)

1. Full-site snapshot never skips because another task is running (removed mutex).
2. Quiet window 23:00–01:00 Asia/Shanghai for discovery / snapshot_recent / candidate_cleanup.
3. Fans short-path (no quiet deferral / no mutex with snapshot).
4. Skip ≠ success (ok=false + skipped=true).
5. Retry / makeup via TaskControl.next_run (~5 min; seed ~10 min).
6. System UI + /api/system next_runs aligned.
7. snapshot-seed CLI.
8. Seed deferral when snapshot running / no_source.

Review follow-ups (kept)

1. snapshot-seed: check completed_key before no_source.
2. ok=False → scheduler.task_fail (no false task_ok).
3. candidate_cleanup failure → ~5 min retry.
4. (superseded by #8) no daytime force-seal just because seeded.

#8 on this branch — task state persistence

Implements #8 on tip f408da93c20237bb7bcfc36e74346452fd02ac0f.

• New reaction_board/task_state.py: kv task:state:{name} with status / due_at / stat_date / progress / updated_at.
• Scheduler startup uses startup_next_run (recover stale running, honor future due_at); no daily next_run=0.
• Seed success enqueues evening seal (ensure_pending with due_at=snapshot_at); daytime restart does not force-scrape.
• Seal path: run_videos(..., force=True, keep_scraped=True) punches src=seed, keeps non-seed ok=1 as progress; CLI --force still full refetch.
• Fail/skip/quiet → failed/deferred + future due_at; success → done + next wall/interval due.
• Web system next_runs prefer task:state.due_at.

Test plan

• uv run python tests/test_task_state.py
• uv run python tests/test_cli.py
• uv run python tests/test_scheduler.py
• uv run python tests/test_snapshot.py
• uv run python tests/test_timeutil.py
• uv run python tests/test_web_queries.py

Do not merge until reviewed.

## Background Production evening seal recorded full-site `snapshot` as success with `elapsed=0` / `already_completed=true` while hourly sync was overlapping. Mutual skip between `snapshot` and `snapshot_recent` was recorded as `ok=true`, then `next_run` advanced to the next day — so the real seal was missed until a manual `--force`. Later: post-seed restart forced a daytime full scrape (clone punched early) and left the evening 23:15 seal skipped. Root cause: intent lived only in in-memory `next_run`. ## Changes (!7 original) 1. **Full-site snapshot never skips** because another task is running (removed mutex). 2. **Quiet window 23:00–01:00 Asia/Shanghai** for discovery / snapshot_recent / candidate_cleanup. 3. **Fans** short-path (no quiet deferral / no mutex with snapshot). 4. **Skip ≠ success** (`ok=false` + `skipped=true`). 5. **Retry / makeup** via `TaskControl.next_run` (~5 min; seed ~10 min). 6. System UI + `/api/system` next_runs aligned. 7. **`snapshot-seed` CLI**. 8. Seed deferral when snapshot running / `no_source`. ## Review follow-ups (kept) 1. snapshot-seed: check `completed_key` before `no_source`. 2. `ok=False` → `scheduler.task_fail` (no false `task_ok`). 3. candidate_cleanup failure → ~5 min retry. 4. (superseded by #8) no daytime force-seal just because seeded. ## #8 on this branch — task state persistence Implements https://git.ofcss.com/kcloud/re-xxm/issues/8 on tip `f408da93c20237bb7bcfc36e74346452fd02ac0f`. - New `reaction_board/task_state.py`: kv `task:state:{name}` with `status` / `due_at` / `stat_date` / `progress` / `updated_at`. - Scheduler startup uses `startup_next_run` (recover stale `running`, honor future `due_at`); **no** daily `next_run=0`. - Seed success enqueues evening seal (`ensure_pending` with `due_at=snapshot_at`); daytime restart does not force-scrape. - Seal path: `run_videos(..., force=True, keep_scraped=True)` punches `src=seed`, keeps non-seed ok=1 as progress; CLI `--force` still full refetch. - Fail/skip/quiet → `failed`/`deferred` + future `due_at`; success → `done` + next wall/interval due. - Web system next_runs prefer `task:state.due_at`. ## Test plan - `uv run python tests/test_task_state.py` - `uv run python tests/test_cli.py` - `uv run python tests/test_scheduler.py` - `uv run python tests/test_snapshot.py` - `uv run python tests/test_timeutil.py` - `uv run python tests/test_web_queries.py` Do not merge until reviewed.

bot-ci Bot added 1 commit 2026-11-03 00:07:48 +08:00

fix(scheduler): drop cross-task mutex, quiet window + retry ... 9969e65202

Incident: evening full snapshot could be recorded as ok with
already_completed/elapsed=0 after mutual skip with hourly sync, then
next_run jumped to the next day.

- Full-site snapshot never skips because another task is running
- 23:00-01:00 Asia/Shanghai: defer discovery / snapshot_recent /
  candidate_cleanup to after 01:00; fans never defers
- Skip records ok=false + skipped=true (not success / not completed)
- Skip/fail reschedule soon (~5m); daily success still next snapshot_at
- TaskControl return value drives next_run; system UI shows skip fields

bot-ci Bot added 1 commit 2026-11-03 00:22:20 +08:00

feat(scheduler): add snapshot-seed CLI and 10m seed deferral ... bb18d232f6

Add `python app.py snapshot-seed [--date]` (offline clone, Debian /
docker compose exec friendly). When seed fires while full-site snapshot
is running, or last_completed is missing (no_source), record skip as
non-success and retry in 10 minutes until a real seed succeeds; fans
and quiet-window rules unchanged.

kairee requested changes 2026-11-03 01:04:37 +08:00

Dismissed

kairee left a comment

Owner

Copy link

Copy source

1. snapshot-seed 会把“当天已完成”的正常情形误判为 no_source，并持续 10 分钟重试。

• 位置：reaction_board/services/snapshot.py:541-550、reaction_board/cli.py:895-910
• run_seed_from_last_completed() 先判断 source >= stat_date，而全站快照当天完成后的正常状态就是：
• snapshot:last_completed == stat_date
• snapshot:completed:<stat_date> == "1"
• 因为检查顺序，snapshot:completed 的 already_completed 分支永远到不了，手动执行 snapshot-seed 会返回非 0；服务在重启后也会把这个正常的无需 seed 状态记作跳过，并每 10 分钟再试一次，直到跨日。
• 建议：先检查目标日的 completed_key(stat_date)，已完成则返回 reason="already_completed" 的成功结果；只有目标日未完成时，才把缺失或不早于目标日的 last_completed 判为 no_source。补充回归测试覆盖 last_completed == target_date 且目标日已完成。

2. 返回 TaskControl(ok=False) 的任务仍被调度器记录为 scheduler.task_ok。

• 位置：reaction_board/scheduler.py:91-106
• 当前实现在线 94 无条件写入 scheduler.task_ok，之后才依据 TaskControl.ok 将 TaskResult.ok 改为 False。因此 quiet-window skip、seed 的 no_source、快照业务失败等路径，会同时产生“任务成功”的调度日志。
• 这与本 PR 的“Skip ≠ success”目标冲突，也会误导基于结构化日志的排障或监控。
• 建议：在解析 TaskControl、确定最终 ok/error 后再记录日志；ok=False 时写失败/跳过对应事件，而非 scheduler.task_ok。补充带 logger 的调度器测试。

3. candidate_cleanup 的失败不会按 PR 所述约 5 分钟补跑。

• 位置：reaction_board/cli.py:727-755、reaction_board/scheduler.py:96-101
• run_cleanup() 成功时返回 TaskControl，但未捕获异常并返回失败控制结果。数据库操作异常会由 Scheduler 捕获，再按该任务的默认 86400s interval 重排，而不是短间隔重试；同时不会写入该任务的失败记录。
• 这与 PR 描述及 DEVELOPMENT.md 中“skip / fail 后短间隔补跑”的口径不一致。
• 建议：参照 run_discover()，捕获异常、调用 _record_task() 并返回 TaskControl(ok=False, next_run=_retry_soon(...))；新增失败补跑测试。

1. snapshot-seed 会把“当天已完成”的正常情形误判为 no_source，并持续 10 分钟重试。 - 位置：reaction_board/services/snapshot.py:541-550、reaction_board/cli.py:895-910 - run_seed_from_last_completed() 先判断 source >= stat_date，而全站快照当天完成后的正常状态就是： - snapshot:last_completed == stat_date - snapshot:completed:<stat_date> == "1" - 因为检查顺序，snapshot:completed 的 already_completed 分支永远到不了，手动执行 snapshot-seed 会返回非 0；服务在重启后也会把这个正常的无需 seed 状态记作跳过，并每 10 分钟再试一次，直到跨日。 - 建议：先检查目标日的 completed_key(stat_date)，已完成则返回 reason="already_completed" 的成功结果；只有目标日未完成时，才把缺失或不早于目标日的 last_completed 判为 no_source。补充回归测试覆盖 last_completed == target_date 且目标日已完成。 2. 返回 TaskControl(ok=False) 的任务仍被调度器记录为 scheduler.task_ok。 - 位置：reaction_board/scheduler.py:91-106 - 当前实现在线 94 无条件写入 scheduler.task_ok，之后才依据 TaskControl.ok 将 TaskResult.ok 改为 False。因此 quiet-window skip、seed 的 no_source、快照业务失败等路径，会同时产生“任务成功”的调度日志。 - 这与本 PR 的“Skip ≠ success”目标冲突，也会误导基于结构化日志的排障或监控。 - 建议：在解析 TaskControl、确定最终 ok/error 后再记录日志；ok=False 时写失败/跳过对应事件，而非 scheduler.task_ok。补充带 logger 的调度器测试。 3. candidate_cleanup 的失败不会按 PR 所述约 5 分钟补跑。 - 位置：reaction_board/cli.py:727-755、reaction_board/scheduler.py:96-101 - run_cleanup() 成功时返回 TaskControl，但未捕获异常并返回失败控制结果。数据库操作异常会由 Scheduler 捕获，再按该任务的默认 86400s interval 重排，而不是短间隔重试；同时不会写入该任务的失败记录。 - 这与 PR 描述及 DEVELOPMENT.md 中“skip / fail 后短间隔补跑”的口径不一致。 - 建议：参照 run_discover()，捕获异常、调用 _record_task() 并返回 TaskControl(ok=False, next_run=_retry_soon(...))；新增失败补跑测试。

bot-ci Bot referenced this issue from a commit 2026-11-03 01:09:55 +08:00

fix(scheduler): address !7 REQUEST_CHANGES

bot-ci Bot added 2 commits 2026-11-03 01:09:55 +08:00

fix(scheduler): address !7 REQUEST_CHANGES ... 1697573d95

- seed: check completed_key before no_source so same-day last_completed
  is already_completed (not 10m retry)
- scheduler: resolve TaskControl ok/error before logging; ok=False logs
  task_fail, not task_ok
- candidate_cleanup: catch failures, record task, retry ~5m like discover

fix(scheduler): do not force full snapshot on post-seed restart ... 51ffa44577

Schedule snapshot_seed/snapshot/fans to the next wall-clock seed_at /
snapshot_at on process start (optionally from last task record), instead
of next_run=0. Force-seal of seed shells only after today's snapshot_at
(or CLI --force), so a daytime restart cannot scrape the whole site.

kairee approved these changes 2026-11-03 01:23:06 +08:00

bot-ci Bot referenced this pull request 2026-11-03 01:42:05 +08:00

调度：任务状态与进度持久化（跨重启/重建） #8

bot-ci Bot referenced this issue from a commit 2026-11-03 01:50:12 +08:00

feat(scheduler): persist task state/progress/due_at across restart (#8)

bot-ci Bot added 1 commit 2026-11-03 01:50:12 +08:00

feat(scheduler): persist task state/progress/due_at across restart (#8) ... f408da93c2

Replace process-start next_run=0 guesses with kv task:state records
(status, due_at, stat_date, progress). Startup recovers stale running,
honors future due_at for delayed retries, and resumes the same
stat_date. Seed only enqueues evening seal; seal force keeps non-seed
ok=1 as progress (keep_scraped). Keeps !7 fail-logging and cleanup retry.

bot-ci Bot referenced this issue from a commit 2026-11-03 01:59:26 +08:00

test(scheduler): nail H7 progress keep + H8 deferred task:state

bot-ci Bot added 1 commit 2026-11-03 01:59:26 +08:00

test(scheduler): nail H7 progress keep + H8 deferred task:state ... dcf37c4a1d

Address Michał NEED-CHANGES on !7 tip f408da93: fail/defer omit-progress
preserves progress/stat_date; seed + quiet-window defer assert
status=deferred and future due_at on task:state. Soft: seeded due≠wall;
cleanup failure reads task:state.

bot-ci Bot commented 2026-11-03 01:59:35 +08:00

Copy link

Copy source

@michal 测例 NEED-CHANGES 已补，请对 tip dcf37c4a1d1afc05394de43c0f6a2c6e3fad84b3（dcf37c4）复审。

H7 test_fail_or_defer_preserves_progress：先写入非空 progress，再 mark_failed / mark_deferred（progress=None），load_state 断言 progress + stat_date 保留，只改 status + due_at。

H8

• 扩展 test_defer_snapshot_seed_schedules_ten_minutes：task:state status=deferred，due_at ≈ finished+600
• 新增 test_quiet_window_defer_persists_deferred_state：discovery 静默窗 defer → deferred + due_at == next_after_snapshot_quiet_window，且不写成功完成类 kv

Soft

• test_seeded_before_snapshot_at_does_not_force_run：due_at 刻意用 22:00 ≠ 墙钟 23:15
• test_candidate_cleanup_failure_retries_soon：兼读 task:state failed + due_at

本地：uv run python tests/test_task_state.py / uv run python tests/test_cli.py 均 OK。未 merge。

@michal 测例 NEED-CHANGES 已补，请对 tip `dcf37c4a1d1afc05394de43c0f6a2c6e3fad84b3`（`dcf37c4`）复审。 **H7** `test_fail_or_defer_preserves_progress`：先写入非空 progress，再 `mark_failed` / `mark_deferred`（`progress=None`），`load_state` 断言 progress + `stat_date` 保留，只改 status + `due_at`。 **H8** - 扩展 `test_defer_snapshot_seed_schedules_ten_minutes`：`task:state` `status=deferred`，`due_at ≈ finished+600` - 新增 `test_quiet_window_defer_persists_deferred_state`：discovery 静默窗 defer → `deferred` + `due_at == next_after_snapshot_quiet_window`，且不写成功完成类 kv **Soft** - `test_seeded_before_snapshot_at_does_not_force_run`：due_at 刻意用 22:00 ≠ 墙钟 23:15 - `test_candidate_cleanup_failure_retries_soon`：兼读 `task:state` failed + due_at 本地：`uv run python tests/test_task_state.py` / `uv run python tests/test_cli.py` 均 OK。未 merge。

You are not authorized to merge this pull request.

This pull request can be merged automatically.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin auto/sched-no-false-skip-4821:auto/sched-no-false-skip-4821

git checkout auto/sched-no-false-skip-4821

Sign in to join this conversation.

Reviewers

No Reviewers

kairee

Labels

No items

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

bot-ci Bot kairee (kairee)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kcloud/re-xxm#7