OCPBUGS-100065: on-prem: tune API VIP haproxy health checks

[mkowalski]

What

Tunes the health-check parameters of the on-prem API VIP haproxy masters backend (single server-line change in the MCO template):

knob	before	after	why
rise	1	3	require 3 consecutive passing /readyz probes (+10s buffer) before re-adding a rebooted master
weight / slowstart	1 / none	256 / 60s	ramp a newly risen server's share of new connections from ~4% to 100% over 60s instead of instantly taking a full round-robin share; large base weight is needed for ramp granularity (relative weights stay equal)
fall / fastinter	3 / none	2 / 2s	mark a dying master down in ~4s instead of ~15s
on-marked-down shutdown-sessions	—	added	terminate established client connections to a marked-down server so clients reconnect to healthy masters instead of timing out

Why

On metal-ipi upgrades, the kube-apiserver on a freshly rebooted master passes /readyz (core API works over host network) ~20-30s before the node's OVN pod network converges. With rise 1, haproxy re-adds it to the VIP rotation on a single passing probe and immediately routes ~1/3 of new connections to it; requests proxied to aggregated APIs (oauth-apiserver, openshift-apiserver) hang, and broken http2 backend connections stay pinned for up to 45s, producing 10-15s of oauth-api-new-connections disruption in ~30-50% of master-updating upgrade runs. Full analysis in OCPBUGS-100065.

This is a downstream mitigation for the routing half of the bug; the aggregator-side fixes are openshift/kubernetes#2730 (readyz reachability semantics) and openshift/kubernetes#2732 (fast http2 dead-connection detection, being upstreamed via kubernetes/kubernetes#141318).

Expected impact / validation

slowstart reduces the probability of a new connection landing on the not-yet-converged kas during the vulnerable ~40-70s window from ~33% to a few percent early in the window. We will benchmark by running dozens of pj-rehearse runs of e2e-metal-ipi-upgrade-ovn-ipv6 / e2e-metal-ipi-ovn-upgrade-runc and comparing oauth-api-new-connections disruption totals against baseline (baseline data in the Jira).

Risk

• Longer re-add delay (+10s) and 60s traffic ramp slightly extend the reduced-capacity period after a master reboot (2 masters carry most new connections for ~1 min).
• shutdown-sessions kills established client connections when a master is marked down; clients reconnect via the VIP to healthy masters. This is intentionally aggressive to shrink VIP-failover disruption episodes and is called out for reviewer attention.
• pkg/controller/template render tests pass.

---

This PR description was generated using AI. Please verify before acting on it.

Summary by CodeRabbit

• Improvements
  • Improved HAProxy backend server traffic distribution and health monitoring.
  • Added faster failure detection and recovery behavior.
  • Enabled gradual traffic ramp-up when servers return to service.
  • Ensured active sessions close when a server is marked unavailable.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@mkowalski: This pull request references Jira Issue OCPBUGS-100065, which is invalid:

• expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What

Tunes the health-check parameters of the on-prem API VIP haproxy masters backend (single server-line change in the MCO template):

knob	before	after	why
rise	1	3	require 3 consecutive passing /readyz probes (+10s buffer) before re-adding a rebooted master
weight / slowstart	1 / none	256 / 60s	ramp a newly risen server's share of new connections from ~4% to 100% over 60s instead of instantly taking a full round-robin share; large base weight is needed for ramp granularity (relative weights stay equal)
fall / fastinter	3 / none	2 / 2s	mark a dying master down in ~4s instead of ~15s
on-marked-down shutdown-sessions	—	added	terminate established client connections to a marked-down server so clients reconnect to healthy masters instead of timing out

Why

On metal-ipi upgrades, the kube-apiserver on a freshly rebooted master passes /readyz (core API works over host network) ~20-30s before the node's OVN pod network converges. With rise 1, haproxy re-adds it to the VIP rotation on a single passing probe and immediately routes ~1/3 of new connections to it; requests proxied to aggregated APIs (oauth-apiserver, openshift-apiserver) hang, and broken http2 backend connections stay pinned for up to 45s, producing 10-15s of oauth-api-new-connections disruption in ~30-50% of master-updating upgrade runs. Full analysis in OCPBUGS-100065.

This is a downstream mitigation for the routing half of the bug; the aggregator-side fixes are openshift/kubernetes#2730 (readyz reachability semantics) and openshift/kubernetes#2732 (fast http2 dead-connection detection, being upstreamed via kubernetes/kubernetes#141318).

Expected impact / validation

slowstart reduces the probability of a new connection landing on the not-yet-converged kas during the vulnerable ~40-70s window from ~33% to a few percent early in the window. We will benchmark by running dozens of pj-rehearse runs of e2e-metal-ipi-upgrade-ovn-ipv6 / e2e-metal-ipi-ovn-upgrade-runc and comparing oauth-api-new-connections disruption totals against baseline (baseline data in the Jira).

Risk

• Longer re-add delay (+10s) and 60s traffic ramp slightly extend the reduced-capacity period after a master reboot (2 masters carry most new connections for ~1 min).
• shutdown-sessions kills established client connections when a master is marked down; clients reconnect via the VIP to healthy masters. This is intentionally aggressive to shrink VIP-failover disruption episodes and is called out for reviewer attention.
• pkg/controller/template render tests pass.

---

This PR description was generated using AI. Please verify before acting on it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 79158c44-6c4a-4bff-9b13-bcb44776d598

📥 Commits

Reviewing files that changed from the base of the PR and between 0df05f6 and cc9e0fb.

📒 Files selected for processing (1)

• templates/master/00-master/on-prem/files/haproxy-haproxy.yaml

---

Walkthrough

The HAProxy server template now uses a higher server weight, faster health checks, updated failure and recovery thresholds, a 60-second slow start, and session shutdown when the server is marked down.

Changes

HAProxy server tuning

Layer / File(s)	Summary
Update HAProxy server settings templates/master/00-master/on-prem/files/haproxy-haproxy.yaml	The server weight changes from 1 to 256. Health checks use fastinter 2s, fall 2, and rise 3. The entry adds slowstart 60s and on-marked-down shutdown-sessions.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Mergeability Score: ⚪ Minimal · up to cc9e0

This localized HAProxy health-check tuning change is merge-ready after normal checks and review; no actionable merge-blocking risk remains.

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the on-prem API VIP HAProxy health-check tuning and includes the related issue identifier.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The only changed file is an HAProxy YAML template server line; the diff introduces no Ginkgo test titles or dynamic test-name values.
Test Structure And Quality	✅ Passed	The diff changes only one HAProxy YAML server line; it adds no Ginkgo test code, so the listed test-structure checks are not applicable.
Microshift Test Compatibility	✅ Passed	The exact diff changes only the HAProxy template and adds no Ginkgo test, API reference, or MicroShift-related test code; this check is inapplicable.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The diff changes only one HAProxy template server line; it adds no Ginkgo e2e tests, so SNO test compatibility rules do not apply.
Topology-Aware Scheduling Compatibility	✅ Passed	The diff changes only HAProxy backend health-check and session settings; the target file contains no affinity, topology spread, node selector, toleration, replica, or PDB scheduling constraints.
Ote Binary Stdout Contract	✅ Passed	The PR changes only the HAProxy YAML template; its diff contains no OTE binary, main, suite-setup, or stdout-writing code.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The PR changes one HAProxy template line only; it adds no Ginkgo e2e tests, IPv4 assumptions, or external connectivity requirements.
No-Weak-Crypto	✅ Passed	The parent-to-HEAD diff changes only one HAProxy server line; its added directives contain no MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, crypto implementation, or secret comparison.
Container-Privileges	✅ Passed	The only changed line adds HAProxy health-check directives; it introduces no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, or escalation setting.
No-Sensitive-Data-In-Logs	✅ Passed	The only diff changes HAProxy server health-check options; log, log-format, stats auth, and templated server names are unchanged, so no new sensitive-data logging is introduced.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mkowalski
Once this PR has been reviewed and has the lgtm label, please assign proietfb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mkowalski]

/payload-aggregate periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-upgrade-ovn-ipv6 10

[mkowalski]

/hold

We are benchmarking this change now, similar to https://redhat.atlassian.net/browse/OCPBUGS-100065?focusedCommentId=17805935

[openshift-ci]

@mkowalski: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-upgrade-ovn-ipv6

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/25a0f2f0-970d-11f1-995d-630277880b32-0

[mkowalski]

Payload validation results

10x /payload-aggregate periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-upgrade-ovn-ipv6 completed (run page, aggregator — aggregation passed). 8 usable runs; 2 runs failed at ~19m before upgrading (no disruption data, unrelated infra failures).

oauth-api-new-connections disruption totals: 0, 0, 0, 0, 1, 2, 7, 11 seconds (one 0 is partial data; openshift-api-new-connections max 8s; reused connections ≤1s everywhere).

Comparison against the data in OCPBUGS-100065 (this PR tested alone, without openshift/kubernetes#2730 / openshift/kubernetes#2732):

metric	baseline	this PR	#2730 alone	#2730 + #2732
runs ≥10s	~31-50%	1/8 (11s)	1/11 (12s)	0/10
max	16s+	11s	12s	3s
median	~5-10s	~0.5s	~1s	~0.5s

Takeaways:

• The health-check tuning alone performs roughly on par with the kas readyz fix (Bug 1991067: [on-prem] Set coredns bufsize to 512 #2730 alone): median disruption collapses from ~5-10s to sub-second.
• The 7s/11s tail is the expected residual: slowstart is probabilistic and haproxy cannot address the aggregator's http2 dead-connection pinning — the Bug 1991067: [on-prem] Set coredns bufsize to 512 #2730 + test CI for no code #2732 combination remains the only configuration bounding disruption at ~3s.
• This change is complementary to and safe to keep alongside the kube-apiserver fixes.

---

This comment was generated using AI. Please verify before acting on it.

[mkowalski]

/payload-aggregate-with-prs periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-upgrade-ovn-ipv6 10 openshift/kubernetes#2730

[openshift-ci]

@mkowalski: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-upgrade-ovn-ipv6

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/54336a50-9a21-11f1-86eb-41ee0bc7cc69-0

[mkowalski]

Payload validation results: this PR + openshift/kubernetes#2730 combined

10x /payload-aggregate-with-prs periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-upgrade-ovn-ipv6 10 openshift/kubernetes#2730 completed (run page, aggregator). 9 usable runs (1 run lost disruption data early).

oauth-api-new-connections disruption totals, sorted: 0, 0, 0, 0, 1, 2, 4, 6, 14 seconds.

metric	baseline	#6400 alone	#2730 alone	#6400 + #2730	#2730 + #2732
runs ≥10s	~31-50%	1/8 (11s)	1/11 (12s)	1/9 (14s)	0/10
max	16s+	11s	12s	14s	3s
median	~5-10s	~0.5s	~1s	~1s	~0.5s

Conclusions

• Combining OCPBUGS-100065: on-prem: tune API VIP haproxy health checks #6400 and Bug 1991067: [on-prem] Set coredns bufsize to 512 #2730 does not improve the results. They are redundant defenses against the same half of the bug (routing new connections to a kube-apiserver whose pod network has not converged): OCPBUGS-100065: on-prem: tune API VIP haproxy health checks #6400 gates VIP membership at the load balancer, Bug 1991067: [on-prem] Set coredns bufsize to 512 #2730 gates it in /readyz. Either one alone is enough — OCPBUGS-100065: on-prem: tune API VIP haproxy health checks #6400 alone (max 11s), Bug 1991067: [on-prem] Set coredns bufsize to 512 #2730 alone (max 12s), and both together (max 14s) all land on the same residual ceiling.
• OCPBUGS-100065: UPSTREAM: <carry>: kube-aggregator: fast http2 health checking for backend connections kubernetes#2732 is imperative to fix this properly. Three independent experiments now converge on the same ~11-14s tail, and the outlier run in this aggregate shows the exact signature test CI for no code #2732 addresses: timeout awaiting response headers → 503 ... http2: client connection lost for ~45s — backend connections that broke after readiness (which no readiness gate or LB health check can protect) staying pinned until the default http2 ReadIdleTimeout+PingTimeout expires. Only the Bug 1991067: [on-prem] Set coredns bufsize to 512 #2730 + test CI for no code #2732 combination bounds disruption at ~3s.

Note: the aggregator reported FAILURE, but for reasons unrelated to this change — all 9 runs uniformly failed the [sig-auth][Feature:SecurityPenetration] suite (0/9 passes) plus a few other tests that passed 10/10 in the Aug 13 aggregate on the same job, pointing to a payload-wide regression in the 2026-08-17 CI build. The disruption invariants themselves passed except the single 14s outlier (threshold 14.21s).

---

This comment was generated using AI. Please verify before acting on it.

[mkowalski]

/retest-required

[mkowalski]

/hold cancel

[openshift-ci]

@mkowalski: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/bootstrap-unit	cc9e0fb	link	true	/test bootstrap-unit

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.