BGP VIP management: render an IPv6 ToR peer on dual-stack deployments - #1945
Open
mkowalski wants to merge 2 commits into
Open
BGP VIP management: render an IPv6 ToR peer on dual-stack deployments#1945mkowalski wants to merge 2 commits into
mkowalski wants to merge 2 commits into
Conversation
With IP_STACK=v4v6 the install-config carries one API/ingress VIP per address family, but bgpVIPConfig only rendered the IPv4 ToR peer, so the secondary-family VIPs had no same-family BGP session to be advertised over. Render a second peer for the ToR's address on the external IPv6 subnet when one exists (override: BGP_VIP_PEER_ADDRESS_V6). The ToR speaker itself already listens on both families (ENABLE_BGP_TOR renders per-family listen ranges). Validated on a dual-stack dev-scripts cluster: both sessions establish per node and both VIP families are advertised to the ToR. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
mkowalski
added a commit
to mkowalski/openshift-release
that referenced
this pull request
Aug 6, 2026
The three coexistence jobs now install IP_STACK=v4v6 clusters (with the DHCP_LEASE_EXPIRY=0 workaround the existing dual-stack lanes carry) and every verify step asserts both address families: - bgp-vip-verify (shared with the single-stack base lane, family-driven by the cluster's VIP lists): per-family ToR path counts for every API and ingress VIP (/32 and /128) and the console exercised over the ingress VIP of each family - ovn-bgp verify: unique Established route-reflector sessions equal to one per node InternalIP (per family), per-node pod subnets asserted in the matching address-family table, and the pod datapath exercised over both agnhost families - metallb pre/verify: per-family IPAddressPool ranges (.70-.90 and ::70-::90), a second BGPPeer for the ToR's IPv6 address, the lb-echo Service becomes ipFamilyPolicy PreferDualStack, and the verify asserts one LoadBalancer IP per family, each advertised from every node and reachable from the hypervisor Two fallback-path fixes found while validating on a live dual-stack cluster: wait for the MetalLB operator to create controller/speaker before rollout-waiting them (the CR-accept-to-workload window raced), and set METALLB_BGP_TYPE=frr-k8s on the dev-manifest webhook server (hardcoded 'native' rejects IPv6 pools that frr-k8s mode supports). Validated end to end on a live dual-stack (v4v6) BGP-VIP cluster with all three producers active: all three verify steps exit 0, both LB families answer, both agnhost families reachable from pods, console 200 over both ingress VIP families. Requires openshift-metal3/dev-scripts#1945 (IPv6 ToR peer in the knob) for CI installs, and the feature-side dual-stack fixes (MCO#6326 per-family kube-vip instances, kube-vip#1671 + the RT-mode NODAD fix) - red by design until those land, like everything else in this PR. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com>
The FRR timer-format and dead port-field bugs on installer#10718 went undetected because e2e only ever deployed the defaults-only happy path: optional fields that are never set produce no config lines, so broken or unrendered template branches are invisible. Set port, holdTime, keepaliveTime, password, bfdEnabled and ebgpMultiHop on every rendered peer (overridable via BGP_VIP_* knobs), and configure the ToR to match: session password and BFD on the CLUSTER peer-group, bfdd enabled, and the BFD control/echo UDP ports opened. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com>
Member
Author
|
Pushed 67c4c6e: BGP_VIP_MANAGEMENT deployments now set every optional peer field (port, holdTime/keepaliveTime, password, bfdEnabled, ebgpMultiHop; overridable via This comment was generated using AI. Please verify before acting on it. |
|
@mkowalski: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
openshift-merge-bot Bot
pushed a commit
to openshift/release
that referenced
this pull request
Aug 13, 2026
…rtisements, day-2 MetalLB) (#82912) * OPNET-803: installer: add e2e-metal-ipi-bgp-vip-ovn-bgp coexistence lane Combined lane for BGP-based VIP management (enhancement 1982) and OVN-K BGP route advertisements. New baremetalds-e2e-bgp-vip-ovn-bgp workflow: dev-scripts install with BGP_VIP_MANAGEMENT=true + ENABLE_BGP_TOR=true, then the existing baremetalds-e2e-ovn-bgp-pre step (external FRR route reflector, routeAdvertisements enabled, FRRConfiguration and RouteAdvertisements CRs). Tests run the existing VIP acceptance step, a new coexistence verify step (both consumers' CRs present, every node - including control plane nodes served by frr-k8s static pods, not the DaemonSet - Established at the route reflector, per-node pod subnets advertised, pod-to-external datapath over a BGP-imported route) and the conformance suite. baremetalds-e2e-ovn-bgp-pre namespace creation made idempotent (oc create namespace fails under errexit when openshift-frr-k8s already exists, as it does on BGP-VIP-managed clusters). Wired as an optional, on-demand presubmit on openshift/installer, red by design until the feature PRs merge (installer#10718, MCO#6326, CNO#3047), like the base e2e-metal-ipi-bgp-vip lane. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com> * ovn/bgp: wait for RouteAdvertisements CRD before applying CRs Found by a dry run of the coexistence lane on a live dev-scripts cluster with BGP VIP management: when frr-k8s is already deployed, the daemonset rollout waits return immediately and the RouteAdvertisements apply races CNO's CRD creation ('no matches for kind RouteAdvertisements'). On the existing ovn-bgp lanes the race is masked by the daemonset creation wait. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com> * bgp-vip/ovn-bgp: address review findings Verify step: bounded polling before asserting CR presence, session count and route state (BGP/CR reconciliation may still be converging when the step starts); container runtime detection matching the pre-step's docker fallback instead of hard-coded podman; assert each node's exact OVN pod subnet (from the k8s.ovn.org/node-subnets annotation) at the route reflector instead of a route count that unrelated prefixes (agnhost network) could satisfy; curl --fail --show-error so HTTP errors fail the datapath check. Pre step: make the RouteAdvertisements CRD wait finite (10 min deadline, dump CRD state on timeout) and move it before the FRR_IMAGE managementState=Unmanaged transition, which would stop CNO from ever creating the CRD. Re-validated against a live BGP-VIP dev-scripts cluster with route advertisements active: all four verify checks pass. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com> * bgp-vip/ovn-bgp: document security justification in workflow docs Container-Privileges pre-merge check: EXTRA_MG_ARGS=--host-network scopes to the post-run must-gather pod only (diagnostics must survive a broken BGP-advertised pod-network path); the privileged FRR/agnhost containers are in the pre-existing shared ovn-bgp-pre step and run on the ephemeral CI hypervisor, not the cluster. Both mirror the existing baremetalds-e2e-ovn-bgp-* lanes unchanged. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com> * bgp-vip/metallb: add day-2 MetalLB install pre step Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com> * bgp-vip/metallb: add coexistence verify step Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com> * installer: add e2e-metal-ipi-bgp-vip-metallb day-2 coexistence lane Third bgp-vip lane: day-2 MetalLB operator install (OLM with upstream manifest fallback) in frr-k8s-external mode on a BGP-VIP-managed cluster, peering with the existing ToR. Verifies the same-neighbor FRRConfiguration merge (no session duplication), LoadBalancer IP advertisement from every node, datapath over the BGP route, and that the VIP acceptance criteria still hold. Validated end to end against a live dev-scripts cluster before submission. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com> * installer: add e2e-metal-ipi-bgp-vip-ovn-bgp-metallb three-producer lane Full coexistence lane composing the two previous ones: BGP-based VIP management + OVN-K route advertisements + day-2 MetalLB in frr-k8s-external mode on a single cluster - three FRRConfiguration producers sharing the same frr-k8s instances (static pods on masters, DaemonSet on workers) and the same external speakers. Runs all three verify steps in order, then the conformance suite. This exact producer combination was validated end to end on a live dev-scripts cluster (all three verifies green) before the lane was added. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com> * bgp-vip/metallb: source the fallback manifests from openshift/metallb-operator Switch the no-catalog fallback from a pinned upstream main SHA to the openshift/metallb-operator release branch (default release-5.0): the fork's branches track OCP releases and carry the downstream API shape (frr-k8s-external backend), so the lane does not rot the way an upstream main pin would. The SCC grant stays: the fork's bin manifest ships the same uid-65534 webhook-server deployment (the productized operator only avoids it via the CSV's --external-metallb-webhook-server mode, which is OLM-only). Validated live: fork manifest applied over the dry-run cluster, operator + MetalLB rolled out, all three coexistence verifies still pass. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com> * bgp-vip: address review findings across the coexistence lanes - both datapath curls discard the response body (-o /dev/null): the pod hostname must not land in CI logs (No-Sensitive-Data-In-Logs) - vtysh summary parsing accepts both JSON schemas (.ipv4Unicast.peers on FRR 8.x, flat .peers on 10.x) - ovn-bgp verify: require exactly one IPv4 pod subnet per node from the node-subnets annotations before trusting the route-reflector result (an absent annotation previously shrank the loop instead of failing) - metallb pool moved to 192.168.111.70-90, off the dev-scripts DHCP range (.20-.60) that the previous .30-.50 overlapped - datapath gated on backend readiness: lb-echo rollout wait in the pre step, poll-wrapped curl in the verify step Re-validated live: pool reapplied, service recycled (got .70), all three coexistence verifies exit 0. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com> * bgp-vip: convert the coexistence lanes to dual-stack The three coexistence jobs now install IP_STACK=v4v6 clusters (with the DHCP_LEASE_EXPIRY=0 workaround the existing dual-stack lanes carry) and every verify step asserts both address families: - bgp-vip-verify (shared with the single-stack base lane, family-driven by the cluster's VIP lists): per-family ToR path counts for every API and ingress VIP (/32 and /128) and the console exercised over the ingress VIP of each family - ovn-bgp verify: unique Established route-reflector sessions equal to one per node InternalIP (per family), per-node pod subnets asserted in the matching address-family table, and the pod datapath exercised over both agnhost families - metallb pre/verify: per-family IPAddressPool ranges (.70-.90 and ::70-::90), a second BGPPeer for the ToR's IPv6 address, the lb-echo Service becomes ipFamilyPolicy PreferDualStack, and the verify asserts one LoadBalancer IP per family, each advertised from every node and reachable from the hypervisor Two fallback-path fixes found while validating on a live dual-stack cluster: wait for the MetalLB operator to create controller/speaker before rollout-waiting them (the CR-accept-to-workload window raced), and set METALLB_BGP_TYPE=frr-k8s on the dev-manifest webhook server (hardcoded 'native' rejects IPv6 pools that frr-k8s mode supports). Validated end to end on a live dual-stack (v4v6) BGP-VIP cluster with all three producers active: all three verify steps exit 0, both LB families answer, both agnhost families reachable from pods, console 200 over both ingress VIP families. Requires openshift-metal3/dev-scripts#1945 (IPv6 ToR peer in the knob) for CI installs, and the feature-side dual-stack fixes (MCO#6326 per-family kube-vip instances, kube-vip#1671 + the RT-mode NODAD fix) - red by design until those land, like everything else in this PR. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com> * bgp-vip/metallb: reference OCPBUGS-105394/-105395 on the env workarounds Both are dev-manifest env drift from the CSV (which sets DEPLOY_SERVICEMONITORS=true and METALLB_BGP_TYPE=frr); annotate so the workarounds are removed when the bugs are fixed. Comment-only change. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com> * bgp-vip/verify: assert FRR runtime state at the ToR The FRR timer-format bug on installer#10718 was invisible to this lane: FRR fails soft on an invalid config line (logs and skips the stanza), the pods stay Ready and the sessions still establish on default timers, so nothing went red. Add a step that asserts every BGP session at the ToR is Established with the negotiated timers matching the configured holdTime/keepaliveTime (90/30, the dev-scripts BGP_VIP_* defaults) and BFD Up - a silently rejected timers line leaves the FRR defaults in place and now fails the lane. Pairs with the dev-scripts change that sets every optional peer field. Assisted-By: Claude Fable 5 Signed-off-by: Mat Kowalski <mko@redhat.com> --------- Signed-off-by: Mat Kowalski <mko@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1939: with
IP_STACK=v4v6the install-config carries one API/ingress VIP per address family, butbgpVIPConfigonly rendered the IPv4 ToR peer — the secondary-family VIPs had no same-family BGP session to be advertised over. Render a second peer for the ToR's address on the external IPv6 subnet when one exists (override:BGP_VIP_PEER_ADDRESS_V6). The ToR speaker from #1929 already listens on both families.Validated on a live dual-stack dev-scripts cluster: both sessions establish per node and both VIP families are advertised to the ToR (v4 /32s and v6 /128s).
Needed for the dual-stack conversion of the
e2e-metal-ipi-bgp-vip*CI lanes (openshift/release#82912).This PR description was generated using AI. Please verify before acting on it.