Troubleshoot the opt job that failed, not the highest-numbered one

[calvinp0]

Top of stack #942 — sits on #941. This PR's diff shows only its own layer.

#943  Troubleshoot the opt job that failed, not the highest-numbered one   ← you are here
 └── #941  Only let a geometry-determining job reject a TS guess
      └── #940  Stop conflating a TS guess's identity with its position
           └── #817  Match TS guesses by identity, not list position       (base: main)

What went wrong

A production run died with:

SchedulerError: opt job for TS0 seems right, yet "run_opt_job" was called.

parse_opt_geo() reaches troubleshooting only on the failure branch, and it already holds the job that failed — but it passed only the species label:

elif self.trsh_ess_jobs:
    self.troubleshoot_opt_jobs(label=label)      # the failed `job` is in scope, and dropped

troubleshoot_opt_jobs() then re-derived a job by scanning for the largest job number:

for job_name in self.job_dict[label]['opt'].keys():
    job_name_int = int(job_name[5:])
    if job_name_int > latest_job_num:            # "latest" == biggest number
        previous_job_num = latest_job_num
        latest_job_num = job_name_int
        job = self.job_dict[label]['opt'][job_name]

Job numbers are only monotonic within a single execution. A project that has been restarted holds jobs from several numbering eras at once, so the highest number can belong to an old job.

In the run that crashed, job_dict['TS0']['opt'] held 102 opt jobs numbered 1332–9779 (22 of them 9xxx), while the chronologically newest was opt_a2098:

newest by mtime	opt_a2098 — 08-01 00:51
highest by number	opt_a9779 — an earlier era

So when opt_a2098 failed, troubleshooting selected a stale 9xxx job, found it converged and run with a fine grid, and hit the sanity guard — which raises and aborts the entire run, discarding 3.5 hours of completed optimization plus everything queued behind it.

The fix

• troubleshoot_opt_jobs(label, job=None) accepts the job to troubleshoot; parse_opt_geo() passes the one that actually failed.
• The fine/coarse pairing no longer decrements a job number. New get_preceding_opt_job() locates the predecessor by the job's identity (candidate is job).
• New get_latest_opt_job() resolves recency from the insertion order of job_dict[label]['opt'] rather than from the job number, for callers that supply no job.
• An absent opt job is logged and returns, instead of raising AttributeError on None.

The SchedulerError guard is deliberately kept. It is a real invariant check, and with the correct job passed it is unreachable from this path — since parse_opt_geo() only troubleshoots when the status is not done. Removing it would hide a future bug rather than fix one.

Evidence

6 new tests in TestTroubleshootOptJobIdentity. The central one calls troubleshoot_opt_jobs(label=...) with the original signature, against a job_dict seeded exactly like the failing project — a stale opt_a9755 (done, fine) plus the newer opt_a2098 that failed. Without this change it fails with the reported error verbatim:

arc.exceptions.SchedulerError: opt job for spc_test seems right, yet "run_opt_job" was called.

The other five cover insertion-order recency, identity-based predecessor lookup, an unregistered job, and the empty-job_dict path.

245 tests pass across all four stack layers applied together (scheduler_test, output_test, species_test, checks/ts_test, job/trsh_test).

Why this belongs on this stack

Same root pattern as the three layers below: an integer that merely looks like an ordering, used as identity. #940 was list position vs TSGuess.index; #941 was a tsg<i> adapter number used as a list subscript; this is a job number used as a timestamp. Each is a valid ordering within one context and silently wrong across contexts — and each surfaces as a function that receives a label and re-derives an object the caller already held.

[Copilot]

Pull request overview

This PR fixes opt-job troubleshooting in Scheduler so it targets the actual failed optimization job (or, if none is supplied, the most recently spawned job) instead of re-deriving a “latest” job by taking the highest job number—preventing incorrect troubleshooting behavior in restarted projects where job numbers are not globally monotonic.

Changes:

• Pass the failed job from parse_opt_geo() into troubleshoot_opt_jobs() so troubleshooting operates on the correct job instance.
• Add get_latest_opt_job() (insertion-order “latest”) and get_preceding_opt_job() (identity-based predecessor lookup) to replace job-number-based selection/decrement logic.
• Add a focused regression test suite (TestTroubleshootOptJobIdentity) covering identity selection, insertion-order fallback, predecessor lookup, unregistered jobs, and empty job lists.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
arc/scheduler.py	Updates opt-job troubleshooting to accept the failed job explicitly, and adds helpers to find latest/preceding jobs by insertion order and object identity.
arc/scheduler_test.py	Adds regression tests ensuring troubleshooting targets the correct failed opt job in restarted-project scenarios.

Suppressed comments (1)

arc/scheduler.py:3620

• This signature uses quoted type annotations for JobAdapter. Since the file has from __future__ import annotations and already uses unquoted annotations elsewhere, dropping the quotes here keeps typing style consistent.

    def get_preceding_opt_job(self, label: str, job: 'JobAdapter') -> 'JobAdapter | None':

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.80%. Comparing base (0ff9e91) to head (5a202e9).

Additional details and impacted files

@@                       Coverage Diff                        @@
##           fix_ts_switch_job_type_guard     #943      +/-   ##
================================================================
+ Coverage                         63.76%   63.80%   +0.03%     
================================================================
  Files                               114      114              
  Lines                             38407    38412       +5     
  Branches                          10037    10038       +1     
================================================================
+ Hits                              24491    24508      +17     
+ Misses                            11013    10997      -16     
- Partials                           2903     2907       +4     

Flag	Coverage Δ
functionaltests	63.80% <ø> (+0.03%)	⬆️
unittests	63.80% <ø> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.