[WIP]OCPEDGE-2827: tolerate one NotReady CP node in EnsureNodesReady for degraded TNF - #31442
[WIP]OCPEDGE-2827: tolerate one NotReady CP node in EnsureNodesReady for degraded TNF#31442Neilhamza wants to merge 1 commit into
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: automatic mode |
openshift-ci-robot
added
the
jira/valid-reference
|
@Neilhamza: This pull request references OCPEDGE-2827 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
Walkthrough
ChangesNode readiness handling
Estimated code review effort: 1 (Trivial) | ~5 minutes Suggested reviewers: Caution Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional.
❌ Failed checks (1 error, 1 warning)
✅ Passed checks (13 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Neilhamza The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/extended/node/node_utils.go`:
- Around line 1006-1009: Update isControlPlaneNode to return (bool, error),
preserving the classification result while propagating failures from the
Nodes().Get lookup instead of converting them to false. Update EnsureNodesReady
and all callers to handle and assert the returned error so API lookup failures
are reported distinctly from a NotReady node.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 573730f8-9b15-4f19-ae06-966c3ffb9ad1
📒 Files selected for processing (1)
test/extended/node/node_utils.go
| func isControlPlaneNode(ctx context.Context, oc *exutil.CLI, nodeName string) bool { | ||
| node, err := oc.AdminKubeClient().CoreV1().Nodes().Get(ctx, nodeName, metav1.GetOptions{}) | ||
| if err != nil { | ||
| return false |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Propagate the node lookup error.
Line [1007] can fail, but Line [1008] silently converts that failure into false. A transient API error will then fall through to the readiness assertion and be reported as a NotReady node instead of an inability to classify it. Return (bool, error) and propagate/assert the error from EnsureNodesReady, or reuse the node returned by the initial list.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@test/extended/node/node_utils.go` around lines 1006 - 1009, Update
isControlPlaneNode to return (bool, error), preserving the classification result
while propagating failures from the Nodes().Get lookup instead of converting
them to false. Update EnsureNodesReady and all callers to handle and assert the
returned error so API lookup failures are reported distinctly from a NotReady
node.
Source: Path instructions
5df8d5b to
37eeb67
Compare
|
/payload-aggregate periodic-ci-openshift-release-main-nightly-4.23-e2e-metal-ovn-two-node-fencing-degraded 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/8e038bb0-8aa5-11f1-9537-77918f9bbb03-0 |
|
/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/04a6a3b0-8aa6-11f1-9fe7-da6c8b8c8de2-0 |
|
Scheduling required tests: |
|
/retest |
|
/payload-aggregate periodic-ci-openshift-release-main-nightly-4.23-e2e-metal-ovn-two-node-fencing-degraded 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/f4990ee0-8b1d-11f1-9b3a-26d65f39e96e-0 |
|
/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn 5 |
|
/retest |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/fa2112e0-8b1d-11f1-8f1b-d09c5195f877-0 |
|
/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9d156510-8b22-11f1-9ddd-15883f653acd-0 |
37eeb67 to
9713c89
Compare
|
@coderabbitai perform full review |
|
@Neilhamza: This pull request references OCPEDGE-2827 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target either version "5.0." or "openshift-5.0.", but it targets "openshift-4.23" instead. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
✅ Action performedFull review finished. |
9713c89 to
2a6e844
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
test/extended/node/node_utils.go (1)
1002-1006: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd boundary coverage for the readiness exception.
Cover exactly one degraded TNF control-plane/master node, plus rejection of worker nodes, multiple NotReady nodes, non-degraded clusters, and non-TNF topologies.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/extended/node/node_utils.go` around lines 1002 - 1006, Add focused tests for the readiness exception around hasControlPlaneLabel and the IsTwoNodeFencing path: verify tolerance only for exactly one NotReady control-plane/master node on a degraded TNF cluster, and verify rejection for a worker node, multiple NotReady nodes, non-degraded clusters, and non-TNF topologies.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@test/extended/node/node_utils.go`:
- Around line 1002-1006: Add focused tests for the readiness exception around
hasControlPlaneLabel and the IsTwoNodeFencing path: verify tolerance only for
exactly one NotReady control-plane/master node on a degraded TNF cluster, and
verify rejection for a worker node, multiple NotReady nodes, non-degraded
clusters, and non-TNF topologies.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 047dbe9c-c20a-4076-a9d0-2fa8ad876bb3
📒 Files selected for processing (1)
test/extended/node/node_utils.go
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/extended/node/node_utils.go`:
- Around line 995-998: Update the degraded TNF exception in the node-readiness
validation flow to inspect the sole entry in notReadyNodes and return only when
that node is a control-plane node. Preserve the existing BeEmpty() failure path
for a single NotReady worker, while retaining the current cluster-state and
two-node-fencing checks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: f4031444-91b4-4fac-ab1e-4f71a31410b3
📒 Files selected for processing (1)
test/extended/node/node_utils.go
|
/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/52de57b0-8b25-11f1-8066-11b38563b107-0 |
|
/payload-aggregate periodic-ci-openshift-release-main-nightly-4.23-e2e-metal-ovn-two-node-fencing-degraded 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/5b81cc80-8b25-11f1-8733-40b42c1be0a9-0 |
2a6e844 to
eb0d22c
Compare
|
/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/33c606b0-8b26-11f1-909d-c5872d2834f4-0 |
|
/payload-aggregate periodic-ci-openshift-release-main-nightly-4.23-e2e-metal-ovn-two-node-fencing-degraded 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/b4621160-8b26-11f1-8797-203235165492-0 |
|
Scheduling required tests: |
|
/retest |
|
/payload-job periodic-ci-openshift-release-main-nightly-4.23-e2e-metal-ovn-two-node-fencing-degraded |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/fa65f100-8b52-11f1-8b16-e29987bb222c-0 |
…egraded TNF EnsureNodesReady checks that all nodes are Ready before running tests. On intentionally degraded Two Node Fencing clusters (one master shut down), this blocks every test suite that uses the check (13 files). When DEGRADED_NODE=true (sets exutil.ClusterDegraded) and the cluster topology is DualReplica (TNF), tolerate exactly one NotReady node provided it is a control-plane node (verified via getControlPlaneNodes). Condition order is ClusterDegraded (bool, free) → len==1 (free) → IsTwoNodeFencing (API call) → control-plane label check, so healthy/non-degraded runs short-circuit before touching the API. The check remains strict for: - Non-degraded clusters (no env var) - Non-TNF topologies - More than one NotReady node (both TNF nodes down) - NotReady worker nodes on a TNF cluster with added compute Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
eb0d22c to
68527bd
Compare
|
/jira refresh |
|
@Neilhamza: This pull request references OCPEDGE-2827 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target either version "5.0." or "openshift-5.0.", but it targets "openshift-4.23" instead. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/payload-aggregate periodic-ci-openshift-release-main-nightly-4.23-e2e-metal-ovn-two-node-fencing-degraded 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9f113a60-8bf9-11f1-8d54-dda4ca122171-0 |
|
/payload-aggregate periodic-ci-openshift-release-main-nightly-4.23-e2e-metal-ovn-two-node-fencing-degraded 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/3b203f50-8bfa-11f1-842b-e23cb2cc7c52-0 |
Neilhamza
changed the title
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
Scheduling required tests: |
2 participants
Summary
On intentionally degraded Two Node Fencing (TNF) clusters where one master is shut down,
EnsureNodesReadyinBeforeEachhooks blocks all 13 test files undertest/extended/node/because it fails on any NotReady node with no bypass.This adds a targeted carve-out: when
DEGRADED_NODE=true(already wired via PR #30649) AND the cluster topology isDualReplica(TNF) AND exactly one node is NotReady AND that node is a control-plane node (verified viagetControlPlaneNodes), the precondition gate passes and the test runs normally.Condition order:
ClusterDegraded(bool, free) ->len==1(free) ->IsTwoNodeFencing(API call) -> CP label check, so healthy/non-degraded runs short-circuit before any API call.The check remains strict for:
Validation
Aggregated degraded run against this PR shows zero EnsureNodesReady/BeforeEach failures in
test/extended/node/— the 9 previously blocked sig-node tests now run and pass. Remaining lane failures are unrelated (TLS port-forward, UDN network segmentation, PLR resize — tracked separately in the release repo skip list).Failing CI lanes
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-degradedperiodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ovn-two-node-fencing-dualstack-degradedJira: https://redhat.atlassian.net/browse/OCPEDGE-2827