Repository navigation
fix(ct_monitor): keep scanning after a log fails its check - #1349
Merged
Merged
Conversation
The scan loop propagated the first failure with `?`, which returned before `last_checked` was ever assigned. Sixty seconds later the same page of up to 10 000 logs was re-fetched and the pass stopped at the same entry -- forever. `run` only logs the error, so the monitor went on looking healthy while checking nothing. The trigger does not have to be a real mis-issuance. `check_one_log` fetches each certificate from crt.sh and parses it as PEM, so a single 429 whose HTML body fails to parse wedges the monitor just as effectively. On a first run `last_checked` is `None`, so a domain with any certificate history issues up to 10 000 sequential crt.sh fetches, which will itself provoke the rate limiting that causes the wedge. It is also a detection-evasion primitive: one benign unknown key -- a rotated gateway key, say -- masks every log recorded after it. So a failing log is now reported and the pass continues, the watermark advances, and the failures are summarised into one error at the end. The trade is that a certificate that genuinely should not exist is alerted on once instead of every minute; the alternative was alerting forever about one certificate and never looking at another. Extract the loop into `scan`, away from the HTTP calls, so it can be tested without a network -- the crate had no tests at all. Stopping at the previous watermark and reporting one that fell off the page are unchanged, and both are now covered.
This was referenced Sep 24, 2026
kvinwang
added a commit
that referenced
this pull request
Sep 25, 2026
…d ct_monitor tc-kms-build-001 now requires --locked on every dstack cargo build in the builder images (#1403), a versioned, integrity-checked Vue script on the onboarding page (#1396), and the ct_monitor scan tests (#1349), whose binary talks to crt.sh directly and cannot run hermetically. Signed-off-by: Kevin Wang <wy721@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
check_new_logsreturned on the first failing log with?, beforelast_checkedwas advanced. Every following pass re-fetched the same page and stopped at the same entry, whilerunonly logged the error — the monitor looked healthy but checked nothing. A single crt.sh 429 whose body fails to parse as PEM is enough to trigger it.Fix
A failing log is reported and the pass continues; the watermark advances and failures are summarised into one error. The loop is extracted into
scanso it can be tested without a network.Trade-off: a failing certificate is now alerted on once per appearance instead of every minute forever.
Verification
cargo test -p ct_monitor: 6 passed (the crate had no tests before).Split out of #1281.