Repository navigation
Incus 'Lost connection to the event listener' during baseline COW copy fails the item terminally instead of retrying as transient infra - #769
Merged
Conversation
…after readiness When incusd crashes or restarts mid-provisioning, in-flight incus client calls fail with "Lost connection to the event listener ... websocket: close 1006" or unix-socket connect errors, and work items went straight to Failed instead of WaitingForTransientRetry. Extend the existing transient-infrastructure classifier with the daemon-connection-loss signatures, retry daemon-connectivity CLI failures in-band after a bounded incus query /1.0 readiness wait, mark ambiguous allow-failure exec results as ExecutionUnavailable, and let allow-failure terminal throws carry the matched allowlisted signature so the classifier still sees the loss. Half-created instances are still cleaned up by the existing creation catch path, and exhausted retries defer through SandboxProvisioningDeferredException. Adds hot-reloadable DaemonUnavailableRetryAttempts, DaemonReadyTimeout, DaemonReadyPollInterval, and DaemonReadyProbeTimeout options. CodeyBox-Prompt-Revision: 1 Co-Authored-By: CodeyBox <noreply@codeybox.invalid>
…argets Audit rework on the daemon-restart handling: - RunCheckedAsync in-place retry is now opt-in per call site (allowDaemonConnectivityRetry) and never applies to `incus exec` argv, whose guest command may have partially run and whose stderr is guest-controlled. - RunAllowFailureAsync no longer rewrites exec-argv failures to ExecutionUnavailable: guest stderr can forge daemon-loss signatures. The guest completion sentinel remains the exec ambiguity oracle. - Instance-creating steps (init/copy/baseline init) pass a prepareForDaemonRetryAsync callback that proves the target absent — deleting a half-created instance through the verified-ownership delete path — before the identical argv is retried; otherwise the original transient failure surfaces for deferred recheck. - Readiness wait recomputes the remaining budget after each probe so the bound is DaemonReadyTimeout, not timeout plus one poll interval. - Transient classification at throw sites now rides a typed IncusTransientInfrastructureException instead of smuggling the matched signature through exception-message prose. - Docs/comments scoped to checked, explicitly safe operations; unsafe steps classify as transient and defer through ProvisioningRetryRecheckIn. CodeyBox-Prompt-Revision: 1 Co-Authored-By: CodeyBox <noreply@codeybox.invalid>
PrepareInstanceAbsentForRetryAsync runs inside the caller's operation-gate lease, but its delete re-entered the gate (DeleteInstanceAsync defaulted to heavyOperation:true), deadlocking capacity-1 provisioning until cancellation. Thread callerHoldsOperationGate through DeleteVerifiedOwnedInstanceAsync / DeleteInstanceAsync so the documented heavyOperation:false guarantee holds; other callers keep existing behavior. Add a capacity-1 COW-copy regression test proving the half-created instance is deleted and the copy retried instead of hanging. CodeyBox-Prompt-Revision: 1 Co-Authored-By: CodeyBox <noreply@codeybox.invalid>
Guest stderr merged by incus exec must not mint daemon-connectivity transient faults; only non-exec steps classify those as retryable. CodeyBox-Prompt-Revision: 1 Co-Authored-By: CodeyBox <noreply@codeybox.invalid>
Checked exec argv merges guest stderr into the captured stream, so embedding it in the failure message let a guest printing a daemon-loss signature forge a transient provisioning deferral via TryClassify. Exec failures now throw IncusExecCommandException without guest output, and TryClassify suppresses daemon-connectivity faults on exec-marked chains while still classifying non-daemon faults. Also splits the daemon probe catch so no unused variable remains. CodeyBox-Prompt-Revision: 1 Co-Authored-By: CodeyBox <noreply@codeybox.invalid>
Exec stderr merges guest output, so any allowlisted signature there - daemon or deadline - must stay terminal. Genuine exec transport loss uses ExecutionUnavailable and the sentinel check. CodeyBox-Prompt-Revision: 1 Co-Authored-By: CodeyBox <noreply@codeybox.invalid>
IncusSandboxOptions and Program summaries claimed exec daemon-loss matches defer as transient; code keeps exec-stderr matches terminal with genuine loss reported out-of-band via ExecutionUnavailable. CodeyBox-Prompt-Revision: 1 Co-Authored-By: CodeyBox <noreply@codeybox.invalid> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated via CodeyBox — work item a739483aefc446339da4fd3fde58104c
Initiated by CodeyBox operator
Co-Authored-By: CodeyBox noreply@codeybox.invalid
🤖 Generated with CodeyBox