From 4edc902bccc83cbff98ede12dc934657018329bd Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Thu, 16 Jul 2026 21:46:01 +0200 Subject: [PATCH 01/19] Release notes for 26.7.0 New platform features. Discovered via Claude, using a generated script. This will later inform a skill. Note: It misses release note comments inside PR descriptions. Working on a fix. --- modules/ROOT/pages/release-notes.adoc | 2 + .../partials/release-notes/release-26.7.adoc | 152 ++++++++++++++ scripts/gather-release-notes.sh | 186 ++++++++++++++++++ 3 files changed, 340 insertions(+) create mode 100644 modules/ROOT/partials/release-notes/release-26.7.adoc create mode 100755 scripts/gather-release-notes.sh diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index e320758f5..f7d3d2b47 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -7,6 +7,8 @@ The Stackable Data Platform consists of multiple operators that work together. Periodically a platform release is made, including all components of the platform at a specific version. // WARNING: Please keep the empty newlines, otherwise headings are broken. +include::partial$release-notes/release-26.7.adoc[] + include::partial$release-notes/release-26.3.adoc[] include::partial$release-notes/release-25.11.adoc[] diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc new file mode 100644 index 000000000..388f4a8ff --- /dev/null +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -0,0 +1,152 @@ +// DRAFT release notes for SDP 26.7.0. +// Sourced from stackabletech issues/PRs labelled (release/26.7.0 OR scheduled-for/26.7.0) +// AND (release-note OR release-note/action-required). 13 issues + 29 PRs. +// All `// TODO(verify)` markers must be resolved before this is merged/published. + +== Release 26.7 + +=== 26.7.0 + +// TODO(verify): exact release date. Tracking issue #847 target was 2026-07-17, shifted ~10 days -> ~2026-07-27. +Released on 2026-07-27. + +// TODO(highlights): written last, once all sections below are agreed. +// [TIP,caption=Release highlights] +// ==== +// * Highlight 1 +// ==== + +// TODO(breaking): overview of breaking changes, written last. Candidates (action-required / chore!): +// * Generic database connection (Druid field rename) - link:#druid-features-26_7_0 +// * Apache Airflow EXPERIMENTAL_FILE_* rename - link:#airflow-removals-26_7_0 +// * Apache Superset EXPERIMENTAL_FILE_* rename - link:#superset-removals-26_7_0 +// * Apache NiFi 1.x CRD fields removed - link:#nifi-removals-26_7_0 +// * Apache Spark application templates now namespaced - link:#spark-fixes-26_7_0 +// [WARNING,caption=Overview of breaking changes] +// ==== +// ==== + +==== New platform features + +===== General + +====== Miscellaneous + +* The SDP now provides a generic database connection mechanism, giving a consistent way to configure connections to external SQL databases across operators. + It has been rolled out to the Apache Airflow (https://github.com/stackabletech/airflow-operator/pull/754[airflow-operator#754]), Apache Hive (https://github.com/stackabletech/hive-operator/pull/674[hive-operator#674]) and Apache Superset (https://github.com/stackabletech/superset-operator/pull/722[superset-operator#722]) operators, and underpins new database connector support in Apache Druid and Trino (see the respective sections below). ++ +-- +[WARNING] +==== +*Breaking:* In the Stackable operator for Apache Druid the field `metadata_storage_database` has been renamed to `metadata_database`. +==== + +// TODO(verify): xref to the generic database connection concept/usage docs once the page exists. +Tracked in https://github.com/stackabletech/issues/issues/238[issues#238]. +-- +* Operators that support OIDC authentication can now configure the `clientAuthenticationMethod`. + This resolves the Apache Druid OIDC issue where a provider advertising `private_key_jwt` could cause authentication to fail (a known issue in the 26.3 release). + // TODO(verify): confirm which operators expose this field and add the relevant xref. + Tracked in https://github.com/stackabletech/issues/issues/838[issues#838]. +* Operators can now resolve product images from registries other than `oci.stackable.tech`, making it possible to install the SDP from a mirror or private registry. + // TODO(verify): confirm this landed for 26.7.0 (issue #716 is a tracking epic) and add the xref to the image-registry docs. + Tracked in https://github.com/stackabletech/issues/issues/716[issues#716]. + +===== Apache Airflow + +Apache Airflow can now authenticate git-sync against a repository using a custom CA, adding first-class support for CAs alongside the existing SSH and basic-auth options. +// TODO(verify): xref to the DAG mounting / git-sync usage guide. +Implemented in https://github.com/stackabletech/airflow-operator/pull/750[airflow-operator#750]. + +===== Apache NiFi + +Apache NiFi can now authenticate git-sync against a repository using a custom CA, adding first-class support for CAs. +// TODO(verify): xref to the NiFi git-sync usage guide. +Implemented in https://github.com/stackabletech/nifi-operator/pull/903[nifi-operator#903]. + +===== Apache Superset + +Two new roles, `worker` and `beat` (limited to a single replica), have been added so that Celery workers and the Celery beat scheduler can be enabled. +// TODO(verify): the results backend currently supports Redis only (no S3); confirm whether to call this out. xref to Superset usage guide. +Implemented in https://github.com/stackabletech/superset-operator/pull/724[superset-operator#724]. + +===== Open Policy Agent + +The OPA operator now supports typed `configOverrides` for OPA's `config.json`, via JSON merge patch, JSON patches or a full file replacement. +// TODO(verify): xref to the OPA configuration overrides docs. +Implemented in https://github.com/stackabletech/opa-operator/pull/818[opa-operator#818]. + +===== OpenSearch + +* Security configuration files are now hot-reloaded when changed, so that settings managed by the operator are applied without Pod restarts. + // TODO(verify): xref to the OpenSearch security docs. + Implemented in https://github.com/stackabletech/opensearch-operator/pull/130[opensearch-operator#130]. +* The operator now writes the actual node FQDNs into `plugins.security.nodes_dn` instead of a generic subject DN. + // TODO(verify): confirm feature vs fix categorisation for opensearch-operator#144. + Implemented in https://github.com/stackabletech/opensearch-operator/pull/144[opensearch-operator#144]. + +===== Trino + +* Trino now supports the PostgreSQL connector, allowing PostgreSQL databases to be queried as a Trino catalog. + // TODO(verify): xref to the Trino PostgreSQL connector docs. + Implemented in https://github.com/stackabletech/trino-operator/pull/883[trino-operator#883]. +* A new `.spec.name.inferred.replaceHyphensWithUnderscores` field on `TrinoCatalog` allows tweaking the catalog name in Trino by replacing `-` with `_`. + This lets you use valid Kubernetes resource names while keeping the convenience of `_` in catalog names. + Implemented in https://github.com/stackabletech/trino-operator/pull/903[trino-operator#903]. + +===== Stackable commons-operator + +The commons-operator restart-controller can now be told to ignore specific ConfigMaps and Secrets. +Set the `restarter.stackable.tech/ignore` label on a ConfigMap or Secret, or the `restarter.stackable.tech/ignore-configmap.x` / `restarter.stackable.tech/ignore-secret.x` annotations on a StatefulSet, to exclude them from triggering restarts. +// TODO(verify): xref to the restarter docs. +Implemented in https://github.com/stackabletech/commons-operator/pull/410[commons-operator#410]. + +===== Stackable secret-operator + +* The name of the CA certificate entry written by a `TrustStore` is now configurable, so it can be aligned with what consumers such as OpenShift expect (`tls.crt` vs `ca.crt`). + // TODO(verify): xref to the TrustStore docs. + Implemented in https://github.com/stackabletech/secret-operator/pull/679[secret-operator#679]. +* Domain components (DCs) can now be added to the subject DN of TLS certificates via the `secrets.stackable.tech/backend.autotls.cert.domain-components-in-subject-dn` volume annotation. + // TODO(verify): xref to the secret-operator volume docs. + Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret-operator#708]. + +// TODO(verify): stackablectl#187 (uninstall Stacks and Demos) - stackablectl is versioned separately; confirm whether it belongs in these platform release notes. + +// ============================================================================ +// TODO: remaining sections (next review rounds) - stubbed to keep the include valid. +// ============================================================================ + +==== Platform improvements + +// TODO: NiFi startup times (nifi#953), OPA DaemonSet maxSurge (opa#819), +// typed OpenSearch configOverrides (opensearch#137), General > Vulnerabilities (CVE counts TODO). + +==== Platform fixes + +// TODO: internal-secret mutability (issues#843; airflow#770, trino#876), +// SecretClasses requiring client cert regression (issues#547), +// Spark connect user args (spark#696), Spark graceful SIGTERM shutdown (docker-images#1564), +// Spark application templates namespaced (spark#694/#720 - BREAKING). + +==== Platform deprecations + +// TODO: product version deprecations (from docker-images version issues). + +==== Platform removals + +// TODO: NiFi 1.x CRD fields (nifi#960 - BREAKING), Airflow/Superset EXPERIMENTAL_FILE_* rename +// (airflow#777, superset#721 - BREAKING), removed product versions. + +==== Supported versions + +// TODO(UNVERIFIED - reconcile against portal.stackable.build screenshot before publishing): +// product versions (new/LTS/deprecated/removed), Kubernetes versions, OpenShift versions. + +==== Upgrade from 26.3 + +// TODO: stackablectl + Helm uninstall/install for 26.3 -> 26.7, plus product-specific upgrade notes. + +[#known-issues-26_7_0] +==== Known issues + +// TODO(verify): none known yet. diff --git a/scripts/gather-release-notes.sh b/scripts/gather-release-notes.sh new file mode 100755 index 000000000..8368d49fe --- /dev/null +++ b/scripts/gather-release-notes.sh @@ -0,0 +1,186 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Gather the GitHub issues and pull requests that feed the release notes for a +# given SDP release, straight from the stackabletech org. +# +# It searches for items carrying BOTH: +# - a release label: release/ OR scheduled-for/ +# - a release-note label: release-note OR release-note/action-required +# +# and prints a report grouped by repository, including the "Release notes" +# snippet from each item body (the section the authors write in the PR template). +# +# It also flags two things that need human attention before a release: +# - items still labelled `scheduled-for/` (merged PRs should be +# re-labelled `release/`; open ones need to land or be dropped). +# - open / unmerged PRs in the set. +# +# Counts are the ground truth from pagination, NOT the GitHub search +# `total_count` (the web UI and that field can be misleading). +# +# Requires: gh (authenticated), jq. +# +# Usage: +# scripts/gather-release-notes.sh -v [--json] +# +# -v, --version Required. e.g. 26.7.0 +# --json Dump the raw combined JSON (all items) instead of the report. +# +# Examples: +# scripts/gather-release-notes.sh -v 26.7.0 +# scripts/gather-release-notes.sh -v 26.7.0 --json > items.json + +# ------------------------------ +# Args parsing +# ------------------------------ + +VERSION="" +OUTPUT="report" + +while [[ "$#" -gt 0 ]]; do + case $1 in + -v|--version) VERSION="$2"; shift ;; + --json) OUTPUT="json" ;; + -h|--help) sed -n '3,30p' "$0"; exit 0 ;; + *) echo "Unknown parameter passed: $1" >&2; exit 1 ;; + esac + shift +done + +if [ -z "$VERSION" ]; then + echo "Usage: gather-release-notes.sh -v [--json]" >&2 + exit 1 +fi + +if [[ ! "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + echo "Invalid version format. Please use the major.minor.patch format (e.g. 26.7.0)." >&2 + exit 1 +fi + +for tool in gh jq; do + if ! command -v "$tool" >/dev/null 2>&1; then + echo "Required tool '$tool' not found on PATH." >&2 + exit 1 + fi +done + +# Minor version, e.g. 26.7 from 26.7.0 +MINOR=$(echo "$VERSION" | cut -d. -f1,2) + +RELEASE_LABEL="release/$VERSION" +SCHEDULED_LABEL="scheduled-for/$VERSION" +QUERY="org:stackabletech label:$SCHEDULED_LABEL,$RELEASE_LABEL label:release-note,release-note/action-required" + +# ------------------------------ +# Fetch (single paginated search; the search API already returns bodies) +# ------------------------------ + +ITEMS_FILE=$(mktemp) +trap 'rm -f "$ITEMS_FILE"' EXIT + +# --paginate over search returns one JSON object per page; slurp the .items +# arrays together and de-duplicate by html_url (defensive against overlap). +gh api -X GET search/issues --paginate -f q="$QUERY" --jq '.items[]' \ + | jq -s 'unique_by(.html_url)' > "$ITEMS_FILE" + +if [ "$OUTPUT" = "json" ]; then + cat "$ITEMS_FILE" + exit 0 +fi + +# ------------------------------ +# Helper: is this item an action-required one? +# ------------------------------ +is_action_required() { echo "$1" | jq -e 'any(.labels[].name; . == "release-note/action-required")' >/dev/null; } + +# ------------------------------ +# Counts (ground truth = number of items actually returned) +# ------------------------------ +TOTAL=$(jq 'length' "$ITEMS_FILE") +N_ISSUES=$(jq '[.[] | select(.pull_request == null)] | length' "$ITEMS_FILE") +N_PRS=$(jq '[.[] | select(.pull_request != null)] | length' "$ITEMS_FILE") + +echo "# Release-notes source data for $VERSION (minor: $MINOR)" +echo "#" +echo "# Query: (label:$SCHEDULED_LABEL OR label:$RELEASE_LABEL)" +echo "# AND (label:release-note OR label:release-note/action-required)" +echo +echo "## Counts (ground truth via pagination, not search total_count)" +echo " Issues: $N_ISSUES" +echo " PRs: $N_PRS" +echo " Total: $TOTAL" +echo + +# ------------------------------ +# Hygiene flags +# ------------------------------ +echo "## Hygiene flags" +echo +echo "### Items still labelled '$SCHEDULED_LABEL'" +echo "# (merged PRs should be re-labelled '$RELEASE_LABEL'; open ones must land or be dropped)" +SCHEDULED=$(jq -r --arg L "$SCHEDULED_LABEL" ' + [.[] | select(any(.labels[].name; . == $L))] + | if length == 0 then " (none)" else + .[] | " \(.repository_url | sub(".*/repos/stackabletech/";"")) #\(.number) [" + + (if .pull_request == null then "issue" + elif .pull_request.merged_at != null then "PR-MERGED" + else "PR-OPEN" end) + "] \(.title)" + end' "$ITEMS_FILE") +echo "$SCHEDULED" +echo +echo "### Open / unmerged PRs in the set (resolve before release)" +OPEN_PRS=$(jq -r ' + [.[] | select(.pull_request != null and .state == "open")] + | if length == 0 then " (none)" else + .[] | " \(.repository_url | sub(".*/repos/stackabletech/";"")) #\(.number) \(.title)" + end' "$ITEMS_FILE") +echo "$OPEN_PRS" +echo + +# ------------------------------ +# Items grouped by repository, with release-note snippet +# ------------------------------ +echo "## Items grouped by repository" + +REPOS=$(jq -r '[.[] | .repository_url | sub(".*/repos/stackabletech/";"")] | unique | .[]' "$ITEMS_FILE") + +while IFS= read -r repo; do + [ -z "$repo" ] && continue + echo + echo "### $repo" + # Iterate items for this repo (compact JSON per line) + jq -c --arg R "$repo" '.[] | select((.repository_url | sub(".*/repos/stackabletech/";"")) == $R)' "$ITEMS_FILE" \ + | while IFS= read -r item; do + number=$(echo "$item" | jq -r '.number') + title=$(echo "$item" | jq -r '.title') + url=$(echo "$item" | jq -r '.html_url') + labels=$(echo "$item" | jq -r '[.labels[].name] | join(", ")') + if echo "$item" | jq -e '.pull_request == null' >/dev/null; then + kind="issue" + elif echo "$item" | jq -e '.pull_request.merged_at != null' >/dev/null; then + kind="PR-merged" + else + kind="PR-$(echo "$item" | jq -r '.state')" + fi + marker="" + if is_action_required "$item"; then marker=" **ACTION-REQUIRED**"; fi + + echo "- #$number [$kind]$marker $title" + echo " $url" + echo " labels: $labels" + # Extract the "Release note(s)" section from the body, if present. + snippet=$(echo "$item" | jq -r '.body // ""' | awk ' + BEGIN { grab=0 } + /^#+[[:space:]]*[Rr]elease[[:space:]]*[Nn]ote/ { grab=1; next } + grab && /^#+[[:space:]]/ { exit } + grab { print } + ' | sed 's/^/ /' | sed '/^[[:space:]]*$/d') + if [ -n "$snippet" ]; then + echo " release-note snippet:" + echo "$snippet" + else + echo " release-note snippet: (none found in body)" + fi + done +done <<< "$REPOS" From c48ed9259a01de66a8410198d61f3c46074bb708 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 10:45:34 +0200 Subject: [PATCH 02/19] Improve generated script It no longer tries to find the "release notes" since they are quite free form. Instead, an LLM can do that but after the script pulls the content. --- scripts/gather-release-notes.sh | 83 ++++++++++++++++----------------- 1 file changed, 41 insertions(+), 42 deletions(-) diff --git a/scripts/gather-release-notes.sh b/scripts/gather-release-notes.sh index 8368d49fe..e93408cb2 100755 --- a/scripts/gather-release-notes.sh +++ b/scripts/gather-release-notes.sh @@ -85,15 +85,28 @@ gh api -X GET search/issues --paginate -f q="$QUERY" --jq '.items[]' \ | jq -s 'unique_by(.html_url)' > "$ITEMS_FILE" if [ "$OUTPUT" = "json" ]; then - cat "$ITEMS_FILE" + # Enrich each item with its comments and a trimmed set of fields, so the + # release-note text (which authors write free-form, in the body OR a + # comment) is available in one place for a human/agent to read and + # interpret. Deliberately NO regex extraction here - see the note below. + jq -c '.[]' "$ITEMS_FILE" | while IFS= read -r item; do + full=$(echo "$item" | jq -r '.repository_url | sub(".*/repos/";"")') + number=$(echo "$item" | jq -r '.number') + comments=$(gh api "repos/$full/issues/$number/comments" --jq '[.[] | {author: .user.login, body}]' 2>/dev/null || echo '[]') + echo "$item" | jq -c --argjson comments "$comments" '{ + repo: (.repository_url | sub(".*/repos/";"")), + number, title, html_url, state, + is_pr: (.pull_request != null), + merged: (.pull_request.merged_at != null), + labels: [.labels[].name], + action_required: (any(.labels[].name; . == "release-note/action-required")), + body: (.body // ""), + comments: $comments + }' + done | jq -s '.' exit 0 fi -# ------------------------------ -# Helper: is this item an action-required one? -# ------------------------------ -is_action_required() { echo "$1" | jq -e 'any(.labels[].name; . == "release-note/action-required")' >/dev/null; } - # ------------------------------ # Counts (ground truth = number of items actually returned) # ------------------------------ @@ -139,9 +152,19 @@ echo "$OPEN_PRS" echo # ------------------------------ -# Items grouped by repository, with release-note snippet +# Item inventory grouped by repository (metadata only). +# +# NOTE: we deliberately do NOT try to extract the release-note text here. +# Authors write it free-form - under headings like `### Release notes`, an +# inline `Release-Note:` label, a bare `# Release notes` in a *linked* comment, +# or interleaved with reviewer instructions. A regex can't reliably tell a real +# note from a template checklist line ("- [ ] Release note snippet added") or +# reviewer chatter. Reading and interpreting that text is the skill's job: +# run this script with `--json` to get each item's full body + comments, then +# read them. This inventory is just the deterministic "what's in the release". # ------------------------------ -echo "## Items grouped by repository" +echo "## Item inventory grouped by repository" +echo "# For the release-note text of each item, run this script with --json and read the body + comments." REPOS=$(jq -r '[.[] | .repository_url | sub(".*/repos/stackabletech/";"")] | unique | .[]' "$ITEMS_FILE") @@ -149,38 +172,14 @@ while IFS= read -r repo; do [ -z "$repo" ] && continue echo echo "### $repo" - # Iterate items for this repo (compact JSON per line) - jq -c --arg R "$repo" '.[] | select((.repository_url | sub(".*/repos/stackabletech/";"")) == $R)' "$ITEMS_FILE" \ - | while IFS= read -r item; do - number=$(echo "$item" | jq -r '.number') - title=$(echo "$item" | jq -r '.title') - url=$(echo "$item" | jq -r '.html_url') - labels=$(echo "$item" | jq -r '[.labels[].name] | join(", ")') - if echo "$item" | jq -e '.pull_request == null' >/dev/null; then - kind="issue" - elif echo "$item" | jq -e '.pull_request.merged_at != null' >/dev/null; then - kind="PR-merged" - else - kind="PR-$(echo "$item" | jq -r '.state')" - fi - marker="" - if is_action_required "$item"; then marker=" **ACTION-REQUIRED**"; fi - - echo "- #$number [$kind]$marker $title" - echo " $url" - echo " labels: $labels" - # Extract the "Release note(s)" section from the body, if present. - snippet=$(echo "$item" | jq -r '.body // ""' | awk ' - BEGIN { grab=0 } - /^#+[[:space:]]*[Rr]elease[[:space:]]*[Nn]ote/ { grab=1; next } - grab && /^#+[[:space:]]/ { exit } - grab { print } - ' | sed 's/^/ /' | sed '/^[[:space:]]*$/d') - if [ -n "$snippet" ]; then - echo " release-note snippet:" - echo "$snippet" - else - echo " release-note snippet: (none found in body)" - fi - done + jq -r --arg R "$repo" ' + def kind: if .pull_request == null then "issue" + elif .pull_request.merged_at != null then "PR-merged" + else "PR-" + .state end; + def ar: if any(.labels[].name; . == "release-note/action-required") + then " **ACTION-REQUIRED**" else "" end; + [.[] | select((.repository_url | sub(".*/repos/stackabletech/";"")) == $R)] + | sort_by(.number)[] + | "- #\(.number) [\(kind)]\(ar) \(.title)\n \(.html_url)\n labels: \([.labels[].name] | join(", "))" + ' "$ITEMS_FILE" done <<< "$REPOS" From c6b4621c256994b4243c861a2193a4bc39a1d497 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 13:27:55 +0200 Subject: [PATCH 03/19] Rearranged release notes after catching the missing release notes from issues. --- .../partials/release-notes/release-26.7.adoc | 355 ++++++++++++++---- 1 file changed, 290 insertions(+), 65 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 388f4a8ff..e6cd98d4d 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -1,30 +1,37 @@ // DRAFT release notes for SDP 26.7.0. // Sourced from stackabletech issues/PRs labelled (release/26.7.0 OR scheduled-for/26.7.0) -// AND (release-note OR release-note/action-required). 13 issues + 29 PRs. -// All `// TODO(verify)` markers must be resolved before this is merged/published. +// AND (release-note OR release-note/action-required): 13 issues + 29 PRs. +// Wording is taken from the authors' own release-note snippets (body or comment) where they wrote one. +// Items with NO author snippet are drafted from the PR body and marked `// TODO(author snippet)`. +// All `// TODO(...)` markers must be resolved before this is merged/published. == Release 26.7 === 26.7.0 -// TODO(verify): exact release date. Tracking issue #847 target was 2026-07-17, shifted ~10 days -> ~2026-07-27. +// TODO(verify): exact release date. Tracking issue #847 target was 2026-07-17, shifted ~10 days. Released on 2026-07-27. -// TODO(highlights): written last, once all sections below are agreed. +// TODO(highlights): write last, once all sections below are agreed. Likely candidates: +// * Generic database connection mechanism rolled out across operators. +// * Trino gains a first-class PostgreSQL connector. +// * Custom OIDC clientAuthenticationMethod (fixes the 26.3 Druid OIDC issue). // [TIP,caption=Release highlights] // ==== -// * Highlight 1 // ==== -// TODO(breaking): overview of breaking changes, written last. Candidates (action-required / chore!): -// * Generic database connection (Druid field rename) - link:#druid-features-26_7_0 -// * Apache Airflow EXPERIMENTAL_FILE_* rename - link:#airflow-removals-26_7_0 -// * Apache Superset EXPERIMENTAL_FILE_* rename - link:#superset-removals-26_7_0 -// * Apache NiFi 1.x CRD fields removed - link:#nifi-removals-26_7_0 -// * Apache Spark application templates now namespaced - link:#spark-fixes-26_7_0 -// [WARNING,caption=Overview of breaking changes] -// ==== -// ==== +[WARNING,caption=Overview of breaking changes] +==== +The following components of the SDP contain breaking changes for this release: + +* link:#generic-db-26_7_0[Generic database connection] (Apache Airflow, Apache Druid, Apache Hive, Apache Superset) +* link:#airflow-removals-26_7_0[Apache Airflow] (renamed `EXPERIMENTAL_FILE_*` settings) +* link:#nifi-removals-26_7_0[Apache NiFi] (removed NiFi 1.x CRD fields) +* link:#spark-fixes-26_7_0[Apache Spark] (application templates are now namespaced) +* link:#superset-removals-26_7_0[Apache Superset] (renamed `EXPERIMENTAL_FILE_*` settings) +// TODO(verify): confirm the image-registry change (issues#716) shipped in 26.7.0 and is breaking for non-Helm installs; if so keep the line below, otherwise remove it. +// * link:#registry-26_7_0[Image registry / repository resolution] +==== ==== New platform features @@ -32,121 +39,339 @@ Released on 2026-07-27. ====== Miscellaneous -* The SDP now provides a generic database connection mechanism, giving a consistent way to configure connections to external SQL databases across operators. - It has been rolled out to the Apache Airflow (https://github.com/stackabletech/airflow-operator/pull/754[airflow-operator#754]), Apache Hive (https://github.com/stackabletech/hive-operator/pull/674[hive-operator#674]) and Apache Superset (https://github.com/stackabletech/superset-operator/pull/722[superset-operator#722]) operators, and underpins new database connector support in Apache Druid and Trino (see the respective sections below). +* [[generic-db-26_7_0]]The SDP now provides a generic database connection mechanism, giving a consistent, typed way to configure connections to external SQL databases across operators. + The same typed structure is used in every CRD, so it can be copied between stacklets. + -- [WARNING] ==== -*Breaking:* In the Stackable operator for Apache Druid the field `metadata_storage_database` has been renamed to `metadata_database`. +*Breaking:* This replaces the previous free-form connection strings with a typed struct, and requires changes to your custom resources (and, in some cases, your Secrets): + +* Apache Airflow: the `.clusterConfig.credentialsSecret` field is renamed to `.clusterConfig.credentialsSecretName`, and the database connection becomes a typed struct. See https://github.com/stackabletech/airflow-operator/pull/754[airflow-operator#754]. +* Apache Druid: the `metadataStorageDatabase` field is renamed to `metadataDatabase` and restructured. See https://github.com/stackabletech/druid-operator/pull/814[druid-operator#814]. +* Apache Hive: the connection becomes a typed struct; because the `database` field is removed, you must record the `username` and add it to the Secret named in `credentialsSecretName` under the `username` key before upgrading. See https://github.com/stackabletech/hive-operator/pull/674[hive-operator#674]. +* Apache Superset: the connection becomes a typed struct, and the operator now uses an internal Secret for the Superset `SECRET_KEY` (created automatically if missing). See https://github.com/stackabletech/superset-operator/pull/722[superset-operator#722]. ==== -// TODO(verify): xref to the generic database connection concept/usage docs once the page exists. +// TODO(verify): xref to the per-operator database-connection usage guides (docs checking). Tracked in https://github.com/stackabletech/issues/issues/238[issues#238]. -- -* Operators that support OIDC authentication can now configure the `clientAuthenticationMethod`. - This resolves the Apache Druid OIDC issue where a provider advertising `private_key_jwt` could cause authentication to fail (a known issue in the 26.3 release). - // TODO(verify): confirm which operators expose this field and add the relevant xref. +* Setting a custom client authentication method is now supported for Apache Airflow, Apache Superset and Apache Druid. + Since Apache Druid 35 and 36, OIDC authentication can fail if the provider (for example Keycloak) advertises `private_key_jwt`, because Druid may attempt to use it. + This feature lets you override the method so that OIDC authentication works again (this was a known issue in the 26.3 release). Tracked in https://github.com/stackabletech/issues/issues/838[issues#838]. -* Operators can now resolve product images from registries other than `oci.stackable.tech`, making it possible to install the SDP from a mirror or private registry. - // TODO(verify): confirm this landed for 26.7.0 (issue #716 is a tracking epic) and add the xref to the image-registry docs. - Tracked in https://github.com/stackabletech/issues/issues/716[issues#716]. ++ +// TODO(hygiene): issues#838 is still labelled scheduled-for/26.7.0 - should be release/26.7.0. + +// TODO(verify - image registry, issues#716): the author's draft note says the SDP now also publishes Helm Charts to quay.io directly from CI, and that operators installed from there pull product images from quay.io. However the quay.io part may not have shipped for 26.7.0, lfrancke's review questions were unresolved, and it is only breaking for non-Helm installs (a new env var / CLI arg is required). Confirm scope before writing. See https://github.com/stackabletech/issues/issues/716[issues#716]. ===== Apache Airflow -Apache Airflow can now authenticate git-sync against a repository using a custom CA, adding first-class support for CAs alongside the existing SSH and basic-auth options. +The Apache Airflow operator now allows specifying CA certificates when using git-sync. // TODO(verify): xref to the DAG mounting / git-sync usage guide. Implemented in https://github.com/stackabletech/airflow-operator/pull/750[airflow-operator#750]. ===== Apache NiFi -Apache NiFi can now authenticate git-sync against a repository using a custom CA, adding first-class support for CAs. +The Apache NiFi operator now allows specifying CA certificates when using git-sync. // TODO(verify): xref to the NiFi git-sync usage guide. Implemented in https://github.com/stackabletech/nifi-operator/pull/903[nifi-operator#903]. ===== Apache Superset -Two new roles, `worker` and `beat` (limited to a single replica), have been added so that Celery workers and the Celery beat scheduler can be enabled. -// TODO(verify): the results backend currently supports Redis only (no S3); confirm whether to call this out. xref to Superset usage guide. +Apache Superset now supports running asynchronous queries via Celery. +Two new roles, `worker` and `beat` (limited to a single replica), have been added. +Redis is supported as the broker queue and results backend. +// TODO(verify): xref to the Superset usage guide. Implemented in https://github.com/stackabletech/superset-operator/pull/724[superset-operator#724]. ===== Open Policy Agent -The OPA operator now supports typed `configOverrides` for OPA's `config.json`, via JSON merge patch, JSON patches or a full file replacement. -// TODO(verify): xref to the OPA configuration overrides docs. +The Open Policy Agent operator now supports `configOverrides` for its `config.json`. +JSON Patches and JSON Merge Patches let you override the nested structure of the file. +// TODO(verify): xref to the OPA configuration/environment overrides docs. Implemented in https://github.com/stackabletech/opa-operator/pull/818[opa-operator#818]. ===== OpenSearch -* Security configuration files are now hot-reloaded when changed, so that settings managed by the operator are applied without Pod restarts. +* Settings managed by the operator are now hot-reloaded when changed, without Pod restarts. // TODO(verify): xref to the OpenSearch security docs. Implemented in https://github.com/stackabletech/opensearch-operator/pull/130[opensearch-operator#130]. -* The operator now writes the actual node FQDNs into `plugins.security.nodes_dn` instead of a generic subject DN. - // TODO(verify): confirm feature vs fix categorisation for opensearch-operator#144. - Implemented in https://github.com/stackabletech/opensearch-operator/pull/144[opensearch-operator#144]. +* The OpenSearch operator now supports JSON Merge Patches, JSON Patches and user-provided JSON files in `configOverrides`, making it easier to define nested types and non-string values. + // TODO(verify): xref to the OpenSearch configuration/environment overrides docs. Confirm feature vs improvement placement (author wrote "OpenSearch" without a section). + Implemented in https://github.com/stackabletech/opensearch-operator/pull/137[opensearch-operator#137]. ===== Trino -* Trino now supports the PostgreSQL connector, allowing PostgreSQL databases to be queried as a Trino catalog. +* Trino now has first-class support for the PostgreSQL connector, built on the new generic database connection mechanism. + Previously, users had to fall back to the `generic` connector. // TODO(verify): xref to the Trino PostgreSQL connector docs. Implemented in https://github.com/stackabletech/trino-operator/pull/883[trino-operator#883]. * A new `.spec.name.inferred.replaceHyphensWithUnderscores` field on `TrinoCatalog` allows tweaking the catalog name in Trino by replacing `-` with `_`. This lets you use valid Kubernetes resource names while keeping the convenience of `_` in catalog names. Implemented in https://github.com/stackabletech/trino-operator/pull/903[trino-operator#903]. -===== Stackable commons-operator +===== Stackable secret-operator -The commons-operator restart-controller can now be told to ignore specific ConfigMaps and Secrets. -Set the `restarter.stackable.tech/ignore` label on a ConfigMap or Secret, or the `restarter.stackable.tech/ignore-configmap.x` / `restarter.stackable.tech/ignore-secret.x` annotations on a StatefulSet, to exclude them from triggering restarts. -// TODO(verify): xref to the restarter docs. -Implemented in https://github.com/stackabletech/commons-operator/pull/410[commons-operator#410]. +The Stackable secret-operator now supports adding domain components (DCs) to the subject DN of TLS certificates, via the `secrets.stackable.tech/backend.autotls.cert.domain-components-in-subject-dn` volume annotation. +// TODO(verify): xref to the secret-operator volume docs. +Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret-operator#708]. -===== Stackable secret-operator +// TODO(decide): stackablectl#187 (uninstall Stacks and Demos) - stackablectl/stackable-cockpit is versioned separately from the SDP. If it belongs here, the author's note is: +// - All demos and stacks can now be installed in non-default namespaces. +// - `stackablectl demo uninstall` / `stackablectl stack uninstall` remove the demo/stack namespace, the Stackable operators and CRDs, and the operator namespace (use `--skip-operators-and-crds` to keep operators/CRDs). +// See https://github.com/stackabletech/stackablectl/issues/187[stackablectl#187]. + +==== Platform improvements -* The name of the CA certificate entry written by a `TrustStore` is now configurable, so it can be aligned with what consumers such as OpenShift expect (`tls.crt` vs `ca.crt`). - // TODO(verify): xref to the TrustStore docs. - Implemented in https://github.com/stackabletech/secret-operator/pull/679[secret-operator#679]. -* Domain components (DCs) can now be added to the subject DN of TLS certificates via the `secrets.stackable.tech/backend.autotls.cert.domain-components-in-subject-dn` volume annotation. - // TODO(verify): xref to the secret-operator volume docs. - Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret-operator#708]. +===== General -// TODO(verify): stackablectl#187 (uninstall Stacks and Demos) - stackablectl is versioned separately; confirm whether it belongs in these platform release notes. +====== Vulnerabilities -// ============================================================================ -// TODO: remaining sections (next review rounds) - stubbed to keep the include valid. -// ============================================================================ +// TODO(verify): CVE counts. e.g. "NNN CVEs were fixed in the Stackable product images. This includes N critical and NN high-severity CVEs." -==== Platform improvements +====== Miscellaneous + +* ConfigMaps and Secrets can now be excluded from the restart-controller. + Set the `restarter.stackable.tech/ignore` label on a ConfigMap or Secret, or the `restarter.stackable.tech/ignore-configmap.x` / `restarter.stackable.tech/ignore-secret.x` annotations on a StatefulSet. + // TODO(verify): xref to the restarter docs. + Implemented in https://github.com/stackabletech/commons-operator/pull/410[commons-operator#410]. + +===== Apache NiFi -// TODO: NiFi startup times (nifi#953), OPA DaemonSet maxSurge (opa#819), -// typed OpenSearch configOverrides (opensearch#137), General > Vulnerabilities (CVE counts TODO). +// TODO(author snippet): nifi-operator#953 has no author-written release note (only CI links); wording below is drafted from the PR body - please confirm. +When the number of NiFi nodes is fixed (all role-group replicas are set, i.e. no autoscaling), the operator now sets `nifi.cluster.flow.election.max.candidates` to that node count. +The cluster then finishes flow election as soon as all expected nodes have connected, instead of waiting out `nifi.cluster.flow.election.max.wait.time` (5 minutes), which greatly reduces cold-start times. +Implemented in https://github.com/stackabletech/nifi-operator/pull/953[nifi-operator#953]. + +===== Open Policy Agent + +The OPA DaemonSet now uses `maxSurge=1` and `maxUnavailable=0` for its rolling update strategy. +During rolling updates (for example when an OPA config change triggers a restart), a new OPA Pod is created and must become ready before the old one is terminated. +This removes the brief availability gap that previously caused errors in products such as Trino. +Implemented in https://github.com/stackabletech/opa-operator/pull/819[opa-operator#819]. + +===== OpenSearch + +The verification of the OpenSearch nodes' certificates has been extended. +The nodes' certificates now also include the FQDNs of the nodes, which are verified by the other nodes using the `plugins.security.nodes_dn` configuration setting. +Previously, it was only verified that the certificates were issued by the secret-operator. +Implemented in https://github.com/stackabletech/opensearch-operator/pull/144[opensearch-operator#144]. + +===== Stackable secret-operator + +The name of the CA certificate entry written by a `TrustStore` is now configurable. +This makes it possible to align with what consumers such as OpenShift expect, which reads the CA from a Secret under the `tls.crt` key rather than `ca.crt`. +// TODO(author snippet): secret-operator#679 has no author-written release note (only a CRD-naming discussion); wording drafted from the PR body - please confirm. +// TODO(verify): xref to the TrustStore docs. +Implemented in https://github.com/stackabletech/secret-operator/pull/679[secret-operator#679]. ==== Platform fixes -// TODO: internal-secret mutability (issues#843; airflow#770, trino#876), -// SecretClasses requiring client cert regression (issues#547), -// Spark connect user args (spark#696), Spark graceful SIGTERM shutdown (docker-images#1564), -// Spark application templates namespaced (spark#694/#720 - BREAKING). +===== General + +* All operators now request only the parts of the secret data they actually need for operator-managed secret volumes. + This uses the xref:secret-operator:volume.adoc#_secrets_stackable_techprovision_parts[`secrets.stackable.tech/provision-parts`] annotation introduced in SDP 26.3.0. + This resolves the regression where SecretClasses used only to validate server certificates also required a client certificate (`tls.crt` / `tls.key`) to be provided. + Fixed in https://github.com/stackabletech/issues/issues/547[issues#547]. +* Internal secrets are now stored in mutable Kubernetes Secret objects. + Previously these objects were immutable and, if deleted, were recreated with new random values; because immutable Secrets are cached, this could disrupt clusters (for example workers failing to join). + This release migrates existing internal Secrets to mutable versions for Trino (https://github.com/stackabletech/trino-operator/pull/876[trino-operator#876]) and Apache Airflow (https://github.com/stackabletech/airflow-operator/pull/770[airflow-operator#770]). + Tracked in https://github.com/stackabletech/issues/issues/843[issues#843]. + +===== Apache Airflow + +OPA authorization now works for Apache Airflow 3 with any user role. +Previously, OPA rulesets were only applied for users with the `Admin` role. +// TODO(hygiene): airflow-operator#800 is still labelled scheduled-for/26.7.0 - should be release/26.7.0. Underlying fix shipped in docker-images#1512. +Fixed in https://github.com/stackabletech/airflow-operator/pull/800[airflow-operator#800]. + +===== Apache NiFi + +// TODO(author snippet): no release note was written on nifi-opa-plugin#30 (issue) or #31 (fix PR); wording below is a suggestion drafted from the issue/PR - please confirm. +// TODO(hygiene): the issue nifi-opa-plugin#30 is still OPEN and labelled scheduled-for/26.7.0. The fix (nifi-opa-plugin#31) is merged and labelled release/26.7.0. Close the issue and/or move its label to release/26.7.0. +The NiFi OPA authorizer plugin no longer exhausts system threads under load. +Previously, the plugin created a new HTTP client for every authorization request, which could spawn thousands of native threads under NiFi load and lead to errors such as `EAGAIN` or out-of-memory conditions. +With this release the plugin uses a single shared, pooled HTTP client for OPA authorization requests. +No configuration changes are required, as defaults are provided; the pool can be tuned via the `OPA_HTTP_POOL_SIZE`, `OPA_HTTP_CONNECT_TIMEOUT_MS` and `OPA_HTTP_REQUEST_TIMEOUT_MS` environment variables. +Fixed in https://github.com/stackabletech/nifi-opa-plugin/issues/30[nifi-opa-plugin#30]. + +[#spark-fixes-26_7_0] +===== Apache Spark + +* *Breaking:* Spark application templates are now namespaced objects instead of cluster-wide objects. + They were unintentionally released as cluster-wide objects in SDP 26.3. + You must recreate any templates in the namespaces where the applications run. + Fixed in https://github.com/stackabletech/spark-k8s-operator/pull/694[spark-k8s-operator#694] (reapplied in https://github.com/stackabletech/spark-k8s-operator/pull/720[spark-k8s-operator#720]). + // TODO(hygiene): spark-k8s-operator#720 is still labelled scheduled-for/26.7.0 - should be release/26.7.0. +* User arguments are now correctly propagated to the Spark Connect server. + Previously, the newline concatenation in the Spark Connect server entrypoint was wrong, so user arguments were dropped. + Fixed in https://github.com/stackabletech/spark-k8s-operator/pull/696[spark-k8s-operator#696]. +* Spark executors now shut down gracefully on `SIGTERM`. + Previously the executor did not receive `SIGTERM` when the driver did, so it was `SIGKILL`ed after the grace period. + Fixed in https://github.com/stackabletech/docker-images/pull/1564[docker-images#1564]. + +===== Open Policy Agent + +// TODO(BLOCKER - unmerged): the fix PR opa-operator#859 is still OPEN (not merged) and the issue opa-operator#858 is still OPEN and labelled scheduled-for/26.7.0. This note must not ship until #859 is merged; otherwise remove this entry. +The Microsoft Entra backend of the OPA user-info-fetcher now retrieves complete group memberships for policy checks. +Previously, it read only the first page of a user's group memberships, so users belonging to many groups received an incomplete group list. +With this release, the Entra backend follows the result pages until every page has been retrieved (bounded to 100 pages as a safeguard). +Fixed in https://github.com/stackabletech/opa-operator/issues/858[opa-operator#858]. ++ +// TODO(orphan note - needs its own labelled source): lfrancke's release note on issue #858 also describes a SECOND change not traceable to #858/#859 - "the user-info-fetcher's outbound HTTP requests (to Microsoft Entra, Keycloak, and the XFSC AAS) now have a 60-second overall timeout and a 15-second connection timeout (previously none, so a slow backend could hang lookups indefinitely)". Find the PR that implemented this, confirm it's in 26.7.0, and label it - then add it here. + +===== Trino + +Hot-reloading of password file Secrets works again. +In the previous release a change to the Secret triggered a Pod restart; the password file Secrets are now excluded from the restart-controller, so hot-reloading is supported again. +Fixed in https://github.com/stackabletech/trino-operator/pull/868[trino-operator#868]. ==== Platform deprecations -// TODO: product version deprecations (from docker-images version issues). +// TODO(verify): no feature/API deprecations found in the labelled items (only product-version deprecations, which are listed under Supported versions below). Confirm there are none. ==== Platform removals -// TODO: NiFi 1.x CRD fields (nifi#960 - BREAKING), Airflow/Superset EXPERIMENTAL_FILE_* rename -// (airflow#777, superset#721 - BREAKING), removed product versions. +[#airflow-removals-26_7_0] +===== Apache Airflow + +*Breaking:* `EXPERIMENTAL_FILE_HEADER` and `EXPERIMENTAL_FILE_FOOTER` in `webserver_config.py` (for arbitrary Python code) have been renamed to `FILE_HEADER` and `FILE_FOOTER`. +Removed in https://github.com/stackabletech/airflow-operator/pull/777[airflow-operator#777]. + +[#nifi-removals-26_7_0] +===== Apache NiFi + +// TODO(author snippet): nifi-operator#960 has no author-written release note (only a CI link); wording drafted from the PR body - please confirm. +*Breaking:* The NiFi 1.x configuration CRD fields and code have been removed, following the removal of NiFi 1.28.1. +This removes the Prometheus reporting-task Job and its `spec.clusterConfig.createReportingTaskJob` field, the pre-2.x non-rolling upgrade handling, the dedicated metrics port, and the sensitive-properties algorithms that were only supported on NiFi 1.x. +Removed in https://github.com/stackabletech/nifi-operator/pull/960[nifi-operator#960]. + +[#superset-removals-26_7_0] +===== Apache Superset + +*Breaking:* `EXPERIMENTAL_FILE_HEADER` and `EXPERIMENTAL_FILE_FOOTER` in `superset_config.py` (for arbitrary Python code) have been renamed to `FILE_HEADER` and `FILE_FOOTER`. +Removed in https://github.com/stackabletech/superset-operator/pull/721[superset-operator#721]. ==== Supported versions -// TODO(UNVERIFIED - reconcile against portal.stackable.build screenshot before publishing): -// product versions (new/LTS/deprecated/removed), Kubernetes versions, OpenShift versions. +===== Product versions + +// TODO(UNVERIFIED - stop-gap from GitHub): the list below is derived only from the docker-images "Update major/minor versions" issues that carry a release-note label (Airflow, Druid, NiFi, OpenSearch, Superset, ZooKeeper). +// The other products (HBase, Hadoop, Hive, Kafka, Spark, OPA, Trino, Vector) are NOT yet captured here because their version issues are not labelled. +// RECONCILE the complete set against a portal.stackable.build screenshot (source of truth; hub.stackable.tech later) before publishing. + +As with previous SDP releases, many product images have been updated to newer versions. +Refer to the xref:operators:supported_versions.adoc[supported versions] documentation for a complete overview including LTS versions or deprecations. + +====== New versions + +The following new product versions are now supported: + +* Apache Airflow: https://github.com/stackabletech/docker-images/issues/1489[3.2.1] +** Note: the Airflow 3.2.1 webserver no longer serves metrics, and this version uses Python 3.14. // TODO(verify) whether the metrics change is intended. +* Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[37.0.0 (LTS)] +** Note: Druid 37.0.0 switched to AWS SDK v2. TLS is now enforced for S3 connections, and the S3 `region` is now required (the operator passes `aws.region` from `S3Connection.region`). +* Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[2.9.0] +* OpenSearch: https://github.com/stackabletech/docker-images/issues/1500[3.6.0] +* Apache Superset: https://github.com/stackabletech/docker-images/issues/1504[6.1.0] +* Apache ZooKeeper: https://github.com/stackabletech/docker-images/issues/1506[3.9.5 (LTS)] + +====== Deprecated versions + +The following product versions are deprecated and will be removed in a later release: + +* Apache Airflow: https://github.com/stackabletech/docker-images/issues/1489[3.1.6] +* Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[35.0.1], https://github.com/stackabletech/docker-images/issues/1490[30.0.1] +* Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[2.7.2] +* OpenSearch: https://github.com/stackabletech/docker-images/issues/1500[3.4.0] +* Apache Superset: https://github.com/stackabletech/docker-images/issues/1504[6.0.0] +* Apache ZooKeeper: https://github.com/stackabletech/docker-images/issues/1506[3.9.4] + +====== Removed versions + +The following product versions are no longer supported. +These images for released product versions remain available in the https://oci.stackable.tech/[Stackable OCI Registry,window=_blank]. +Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[in the project overview documentation,window=_blank]. + +* Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[34.0.0] +* Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[1.28.1] +// TODO(verify): complete this list once the full product-version set is reconciled against the portal. + +===== Kubernetes versions + +// TODO(verify): supported Kubernetes versions for 26.7.0. In 26.3 these were 1.35-1.31. Confirm against the release definition. + +===== OpenShift versions + +// TODO(verify): supported OpenShift versions for 26.7.0. In 26.3 these were 4.21-4.18. Confirm against the release definition. ==== Upgrade from 26.3 -// TODO: stackablectl + Helm uninstall/install for 26.3 -> 26.7, plus product-specific upgrade notes. +// TODO(verify): consult/collect product-specific upgrade notes (e.g. the generic database connection migrations for Airflow/Druid/Hive/Superset, and the Druid 37 S3 TLS/region change). + +===== Using stackablectl + +====== Upgrade with a single command + +[source,console] +---- +$ stackablectl release upgrade 26.7 +---- + +====== Upgrade with multiple consecutive commands + +Uninstall the `26.3` release + +[source,console] +---- +$ stackablectl release uninstall 26.3 + +Uninstalled release '26.3' + +Use "stackablectl release list" to list available releases. +# ... +---- + +Install the `26.7` release + +[source,console] +---- +$ stackablectl release install 26.7 + +Installed release '26.7' + +Use "stackablectl operator installed" to list installed operators. +---- + +===== Using Helm + +Use `helm list` to list the currently installed operators. + +You can use the following command to uninstall all operators that are part of the `26.3` release: + +[source,console] +---- +$ helm uninstall airflow-operator -operator ... +release "airflow-operator" uninstalled +release "-operator" uninstalled +... +---- + +Install the `26.7` release + +[NOTE] +==== +`helm repo` subcommands are not supported for OCI registries. +The operators are installed directly, without adding the Helm Chart repository first. +==== + +[source,console] +---- +helm install --wait -operator oci://oci.stackable.tech/sdp-charts/-operator --version 26.7.0 +---- [#known-issues-26_7_0] ==== Known issues -// TODO(verify): none known yet. +// TODO(verify): none identified yet from the labelled items. From de1a970595c61032d203110794e01912b6f4b0a8 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 15:49:18 +0200 Subject: [PATCH 04/19] remove a todo --- modules/ROOT/partials/release-notes/release-26.7.adoc | 1 - 1 file changed, 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index e6cd98d4d..7ed0b3955 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -213,7 +213,6 @@ Fixed in https://github.com/stackabletech/nifi-opa-plugin/issues/30[nifi-opa-plu ===== Open Policy Agent -// TODO(BLOCKER - unmerged): the fix PR opa-operator#859 is still OPEN (not merged) and the issue opa-operator#858 is still OPEN and labelled scheduled-for/26.7.0. This note must not ship until #859 is merged; otherwise remove this entry. The Microsoft Entra backend of the OPA user-info-fetcher now retrieves complete group memberships for policy checks. Previously, it read only the first page of a user's group memberships, so users belonging to many groups received an incomplete group list. With this release, the Entra backend follows the result pages until every page has been retrieved (bounded to 100 pages as a safeguard). From c79a9611f26036e7649676101dc9b97622b3065d Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 15:56:12 +0200 Subject: [PATCH 05/19] nifi cold start --- modules/ROOT/partials/release-notes/release-26.7.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 7ed0b3955..94c98a652 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -139,9 +139,9 @@ Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret- ===== Apache NiFi -// TODO(author snippet): nifi-operator#953 has no author-written release note (only CI links); wording below is drafted from the PR body - please confirm. -When the number of NiFi nodes is fixed (all role-group replicas are set, i.e. no autoscaling), the operator now sets `nifi.cluster.flow.election.max.candidates` to that node count. -The cluster then finishes flow election as soon as all expected nodes have connected, instead of waiting out `nifi.cluster.flow.election.max.wait.time` (5 minutes), which greatly reduces cold-start times. +For clusters with a fixed number of nodes, the operator now sets `nifi.cluster.flow.election.max.candidates` to the total node count. +Cold-start flow election then completes as soon as all expected nodes report in, typically seconds instead of the up-to-5-minute `nifi.cluster.flow.election.max.wait.time` timeout. +The timeout stays at NiFi's upstream default of 5 minutes as a fallback, so there is no correctness change if fewer nodes than expected show up. Implemented in https://github.com/stackabletech/nifi-operator/pull/953[nifi-operator#953]. ===== Open Policy Agent From e286a0322b7e6f49187dd2d1ce581265591c4499 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 15:57:36 +0200 Subject: [PATCH 06/19] kafka pod FQDNs in cert DNs --- modules/ROOT/partials/release-notes/release-26.7.adoc | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 94c98a652..8eb3e9a37 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -137,6 +137,14 @@ Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret- // TODO(verify): xref to the restarter docs. Implemented in https://github.com/stackabletech/commons-operator/pull/410[commons-operator#410]. +===== Apache Kafka + +The FQDNs of the Kafka Pods are now included in the subject DNs of their TLS certificates. +OPA Rego rules can now check these FQDNs. +Existing OPA rules may need to be adjusted. +// TODO(verify): xref to the Kafka security usage guide (author linked .../kafka/usage-guide/security/#_authorization). +Implemented in https://github.com/stackabletech/kafka-operator/pull/972[kafka-operator#972]. + ===== Apache NiFi For clusters with a fixed number of nodes, the operator now sets `nifi.cluster.flow.election.max.candidates` to the total node count. From 6993019d1b7ea6d9980d65470ae8819eba5e1263 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 15:58:55 +0200 Subject: [PATCH 07/19] TrustStore .spec.tlsPemCaName no configurable --- modules/ROOT/partials/release-notes/release-26.7.adoc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 8eb3e9a37..304d89d0b 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -168,9 +168,8 @@ Implemented in https://github.com/stackabletech/opensearch-operator/pull/144[ope ===== Stackable secret-operator -The name of the CA certificate entry written by a `TrustStore` is now configurable. -This makes it possible to align with what consumers such as OpenShift expect, which reads the CA from a Secret under the `tls.crt` key rather than `ca.crt`. -// TODO(author snippet): secret-operator#679 has no author-written release note (only a CRD-naming discussion); wording drafted from the PR body - please confirm. +The name of the key in the ConfigMap/Secret that holds the PEM-encoded CA certificate of a `TrustStore` is now configurable, via `.spec.tlsPemCaName`. +This is needed, for example, to use the generated Secret within an OpenShift Ingress. // TODO(verify): xref to the TrustStore docs. Implemented in https://github.com/stackabletech/secret-operator/pull/679[secret-operator#679]. From bc2bfc926aa1d581a71a4f8eef2b150d171c382a Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 16:00:09 +0200 Subject: [PATCH 08/19] NiFi 1 CRD fields gone, and sensitiveProperties removedl --- modules/ROOT/partials/release-notes/release-26.7.adoc | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 304d89d0b..30ba20633 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -248,9 +248,13 @@ Removed in https://github.com/stackabletech/airflow-operator/pull/777[airflow-op [#nifi-removals-26_7_0] ===== Apache NiFi -// TODO(author snippet): nifi-operator#960 has no author-written release note (only a CI link); wording drafted from the PR body - please confirm. -*Breaking:* The NiFi 1.x configuration CRD fields and code have been removed, following the removal of NiFi 1.28.1. -This removes the Prometheus reporting-task Job and its `spec.clusterConfig.createReportingTaskJob` field, the pre-2.x non-rolling upgrade handling, the dedicated metrics port, and the sensitive-properties algorithms that were only supported on NiFi 1.x. +*Breaking:* Support for NiFi 1.x has been dropped, along with a few CRD fields that only applied to 1.x and had no effect on NiFi 2.x: + +* `spec.clusterConfig.createReportingTaskJob` (including `enabled` and `podOverrides`) is removed. + NiFi 2.x exposes Prometheus metrics directly at `/nifi-api/flow/metrics/prometheus`, so the reporting-task Job is obsolete. +* The deprecated `spec.clusterConfig.sensitiveProperties.algorithm` values are removed. + Use a supported algorithm instead: `nifiArgon2AesGcm256` (default) or `nifiPbkdf2AesGcm256`. + Removed in https://github.com/stackabletech/nifi-operator/pull/960[nifi-operator#960]. [#superset-removals-26_7_0] From b991f64147122ac2f15da76a32975d610156dcf3 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 19:00:23 +0200 Subject: [PATCH 09/19] Nifi OPA Authorizer thread exhaustion --- modules/ROOT/partials/release-notes/release-26.7.adoc | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 30ba20633..1b33bc00f 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -195,12 +195,10 @@ Fixed in https://github.com/stackabletech/airflow-operator/pull/800[airflow-oper ===== Apache NiFi -// TODO(author snippet): no release note was written on nifi-opa-plugin#30 (issue) or #31 (fix PR); wording below is a suggestion drafted from the issue/PR - please confirm. -// TODO(hygiene): the issue nifi-opa-plugin#30 is still OPEN and labelled scheduled-for/26.7.0. The fix (nifi-opa-plugin#31) is merged and labelled release/26.7.0. Close the issue and/or move its label to release/26.7.0. -The NiFi OPA authorizer plugin no longer exhausts system threads under load. -Previously, the plugin created a new HTTP client for every authorization request, which could spawn thousands of native threads under NiFi load and lead to errors such as `EAGAIN` or out-of-memory conditions. -With this release the plugin uses a single shared, pooled HTTP client for OPA authorization requests. -No configuration changes are required, as defaults are provided; the pool can be tuned via the `OPA_HTTP_POOL_SIZE`, `OPA_HTTP_CONNECT_TIMEOUT_MS` and `OPA_HTTP_REQUEST_TIMEOUT_MS` environment variables. +The NiFi OPA authorizer plugin no longer exhausts native threads under load. +Previously, the plugin could create a new HTTP client for each authorization request, spawning large numbers of selector and executor threads under NiFi load and eventually exhausting native threads. +With this release, all OPA authorization requests share a single pooled HTTP client with a fixed-size daemon-thread executor. +Existing deployments need no configuration changes, as defaults are provided; the client is configurable via `OPA_HTTP_POOL_SIZE` (default 32), `OPA_HTTP_CONNECT_TIMEOUT_MS` (default 2000) and `OPA_HTTP_REQUEST_TIMEOUT_MS` (default 5000). Fixed in https://github.com/stackabletech/nifi-opa-plugin/issues/30[nifi-opa-plugin#30]. [#spark-fixes-26_7_0] From 109a11bab5709a9e69f434a231ff32c7ab4ce982 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 22:17:02 +0200 Subject: [PATCH 10/19] Product changes Based on data in portal, but with a correction for Spark --- .../partials/release-notes/release-26.7.adoc | 68 ++++++++++++++++--- 1 file changed, 58 insertions(+), 10 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 1b33bc00f..d7f3352a3 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -265,25 +265,42 @@ Removed in https://github.com/stackabletech/superset-operator/pull/721[superset- ===== Product versions -// TODO(UNVERIFIED - stop-gap from GitHub): the list below is derived only from the docker-images "Update major/minor versions" issues that carry a release-note label (Airflow, Druid, NiFi, OpenSearch, Superset, ZooKeeper). -// The other products (HBase, Hadoop, Hive, Kafka, Spark, OPA, Trino, Vector) are NOT yet captured here because their version issues are not labelled. -// RECONCILE the complete set against a portal.stackable.build screenshot (source of truth; hub.stackable.tech later) before publishing. +// Reconciled against portal.stackable.build (source of truth) by diffing the 26.3 and 26.7 exports. +// Added = in 26.7 not in 26.3; Removed = in 26.3 not in 26.7; statuses (LTS/experimental/deprecated) from the 26.7 export. +// Overrides where the portal was wrong: +// - Spark 4.1.2 appears in the 26.3 portal export by mistake (it was not in 26.3.0; docker-images#1419 added only 3.5.8). It is treated as ADDED in 26.7. Portal maintainer to be notified. +// - Vector is absent from the portal export; taken from docker-images#1486 (add 0.55.0, remove 0.52.0). Portal maintainer notified. +// Deprecated list = versions newly deprecated this release (status changed to deprecated vs 26.3), not the full standing deprecated set. As with previous SDP releases, many product images have been updated to newer versions. Refer to the xref:operators:supported_versions.adoc[supported versions] documentation for a complete overview including LTS versions or deprecations. +====== New LTS versions + +The following product versions were already available before but are now marked as the xref:compliance:policies.adoc#_product_lifecycle_policy[LTS] version: + +* Apache Hive: https://github.com/stackabletech/docker-images/issues/1495[4.2.0 (LTS)] + ====== New versions The following new product versions are now supported: -* Apache Airflow: https://github.com/stackabletech/docker-images/issues/1489[3.2.1] -** Note: the Airflow 3.2.1 webserver no longer serves metrics, and this version uses Python 3.14. // TODO(verify) whether the metrics change is intended. +* Apache Airflow: https://github.com/stackabletech/docker-images/issues/1489[3.2.2] +** Note: this Airflow version uses Python 3.14. // TODO(verify): the docker-images#1489 note about the webserver no longer serving metrics referred to 3.2.1; confirm it still applies to 3.2.2. * Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[37.0.0 (LTS)] ** Note: Druid 37.0.0 switched to AWS SDK v2. TLS is now enforced for S3 connections, and the S3 `region` is now required (the operator passes `aws.region` from `S3Connection.region`). +* Apache HBase: https://github.com/stackabletech/docker-images/issues/1491[2.6.6 (LTS)] +* Apache Hadoop HDFS: https://github.com/stackabletech/docker-images/issues/1494[3.4.3 (LTS)], https://github.com/stackabletech/docker-images/issues/1494[3.5.0] +* Apache Kafka: https://github.com/stackabletech/docker-images/issues/1497[3.9.2 (LTS)], https://github.com/stackabletech/docker-images/issues/1497[4.2.1 (experimental)] * Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[2.9.0] -* OpenSearch: https://github.com/stackabletech/docker-images/issues/1500[3.6.0] +* Apache Phoenix: https://github.com/stackabletech/docker-images/issues/1491[5.3.1 (LTS)] +* Apache Spark: https://github.com/stackabletech/docker-images/issues/1502[4.1.2 (LTS)] * Apache Superset: https://github.com/stackabletech/docker-images/issues/1504[6.1.0] * Apache ZooKeeper: https://github.com/stackabletech/docker-images/issues/1506[3.9.5 (LTS)] +* Open Policy Agent: https://github.com/stackabletech/docker-images/issues/1499[1.16.2] +* OpenSearch: https://github.com/stackabletech/docker-images/issues/1500[3.6.0] +* Trino: https://github.com/stackabletech/docker-images/issues/1505[481] +* Vector: https://github.com/stackabletech/docker-images/issues/1486[0.55.0] ====== Deprecated versions @@ -291,10 +308,17 @@ The following product versions are deprecated and will be removed in a later rel * Apache Airflow: https://github.com/stackabletech/docker-images/issues/1489[3.1.6] * Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[35.0.1], https://github.com/stackabletech/docker-images/issues/1490[30.0.1] +* Apache Hadoop HDFS: https://github.com/stackabletech/docker-images/issues/1494[3.4.2] +* Apache Hive: https://github.com/stackabletech/docker-images/issues/1495[4.0.1] +* Apache Kafka: https://github.com/stackabletech/docker-images/issues/1497[3.9.1] * Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[2.7.2] -* OpenSearch: https://github.com/stackabletech/docker-images/issues/1500[3.4.0] +* Apache Phoenix: https://github.com/stackabletech/docker-images/issues/1491[5.3.0] +* Apache Spark: https://github.com/stackabletech/docker-images/issues/1502[3.5.8], https://github.com/stackabletech/docker-images/issues/1502[4.1.1] * Apache Superset: https://github.com/stackabletech/docker-images/issues/1504[6.0.0] * Apache ZooKeeper: https://github.com/stackabletech/docker-images/issues/1506[3.9.4] +* Open Policy Agent: https://github.com/stackabletech/docker-images/issues/1499[1.12.3] +* OpenSearch: https://github.com/stackabletech/docker-images/issues/1500[3.4.0] +* Trino: https://github.com/stackabletech/docker-images/issues/1505[479] ====== Removed versions @@ -303,16 +327,40 @@ These images for released product versions remain available in the https://oci.s Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[in the project overview documentation,window=_blank]. * Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[34.0.0] +* Apache HBase: https://github.com/stackabletech/docker-images/issues/1491[2.6.3] +* Apache Hive: https://github.com/stackabletech/docker-images/issues/1495[4.1.0] * Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[1.28.1] -// TODO(verify): complete this list once the full product-version set is reconciled against the portal. +* Apache Phoenix: https://github.com/stackabletech/docker-images/issues/1491[5.2.1] +* Apache Spark: https://github.com/stackabletech/docker-images/issues/1502[3.5.7], https://github.com/stackabletech/docker-images/issues/1502[4.0.1] +* Open Policy Agent: https://github.com/stackabletech/docker-images/issues/1499[1.8.0] +* Vector: https://github.com/stackabletech/docker-images/issues/1486[0.52.0] ===== Kubernetes versions -// TODO(verify): supported Kubernetes versions for 26.7.0. In 26.3 these were 1.35-1.31. Confirm against the release definition. +This release supports the following Kubernetes versions: + +* `1.36` +* `1.35` +* `1.34` +* `1.33` +* `1.32` +* `1.31` + +// Portal diff shows no Kubernetes versions were dropped between 26.3 and 26.7 (1.36 added). Confirm before publishing. +No Kubernetes versions were dropped in this release. ===== OpenShift versions -// TODO(verify): supported OpenShift versions for 26.7.0. In 26.3 these were 4.21-4.18. Confirm against the release definition. +This release is available in the RedHat Certified Operator Catalog for the following OpenShift versions: + +* `4.22` +* `4.21` +* `4.20` +* `4.19` +* `4.18` + +// Portal diff shows no OpenShift versions were dropped between 26.3 and 26.7 (4.22 added). Confirm before publishing. +No OpenShift versions were dropped in this release. ==== Upgrade from 26.3 From 10a619430bc00f5f6237eb08623d62210f53f907 Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 22 Jul 2026 17:29:35 +0200 Subject: [PATCH 11/19] docs: Tweak 26.7.0 release notes --- .../partials/release-notes/release-26.7.adoc | 225 ++++++++++-------- 1 file changed, 132 insertions(+), 93 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index d7f3352a3..cc2377ebe 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -1,16 +1,9 @@ -// DRAFT release notes for SDP 26.7.0. -// Sourced from stackabletech issues/PRs labelled (release/26.7.0 OR scheduled-for/26.7.0) -// AND (release-note OR release-note/action-required): 13 issues + 29 PRs. -// Wording is taken from the authors' own release-note snippets (body or comment) where they wrote one. -// Items with NO author snippet are drafted from the PR body and marked `// TODO(author snippet)`. -// All `// TODO(...)` markers must be resolved before this is merged/published. - == Release 26.7 +:secret-volume-annotation: secret-operator:volume.adoc##_secrets_stackable_techbackend_autotls_cert_domain_components_in_subject_dn === 26.7.0 -// TODO(verify): exact release date. Tracking issue #847 target was 2026-07-17, shifted ~10 days. -Released on 2026-07-27. +Released on 2026-07-21. // TODO(highlights): write last, once all sections below are agreed. Likely candidates: // * Generic database connection mechanism rolled out across operators. @@ -25,10 +18,10 @@ Released on 2026-07-27. The following components of the SDP contain breaking changes for this release: * link:#generic-db-26_7_0[Generic database connection] (Apache Airflow, Apache Druid, Apache Hive, Apache Superset) -* link:#airflow-removals-26_7_0[Apache Airflow] (renamed `EXPERIMENTAL_FILE_*` settings) +* link:#airflow-improvements-26_7_0[Apache Airflow] (renamed `EXPERIMENTAL_FILE_*` settings) * link:#nifi-removals-26_7_0[Apache NiFi] (removed NiFi 1.x CRD fields) * link:#spark-fixes-26_7_0[Apache Spark] (application templates are now namespaced) -* link:#superset-removals-26_7_0[Apache Superset] (renamed `EXPERIMENTAL_FILE_*` settings) +* link:#superset-improvements-26_7_0[Apache Superset] (renamed `EXPERIMENTAL_FILE_*` settings) // TODO(verify): confirm the image-registry change (issues#716) shipped in 26.7.0 and is breaking for non-Helm installs; if so keep the line below, otherwise remove it. // * link:#registry-26_7_0[Image registry / repository resolution] ==== @@ -37,6 +30,22 @@ The following components of the SDP contain breaking changes for this release: ===== General +====== Artifacts on quay.io + +SDP artifacts are now published to quay.io directly from our CI pipelines. +Previously, only the operator and product container images were available on quay.io, manually mirrored from oci.stackable.tech, and operators defaulted to pulling product container images from oci.stackable.tech. +Additionally, operator Helm Charts are now published to quay.io as well. +When installing these Helm Charts, the operators are automatically instructed to use product container images from quay.io. + +[CAUTION] +==== +To publish operator Helm Charts to quay.io, the repository layout was adjusted to accommodate the new artifacts. +It closely mirrors the layout of oci.stackable.tech. +See xref:concepts:artifact-registries/index.adoc[the artifact registries page] for a detailed explanation of the new layout. +==== + +Tracked in https://github.com/stackabletech/issues/issues/716[issues#716]. + ====== Miscellaneous * [[generic-db-26_7_0]]The SDP now provides a generic database connection mechanism, giving a consistent, typed way to configure connections to external SQL databases across operators. @@ -47,10 +56,16 @@ The following components of the SDP contain breaking changes for this release: ==== *Breaking:* This replaces the previous free-form connection strings with a typed struct, and requires changes to your custom resources (and, in some cases, your Secrets): -* Apache Airflow: the `.clusterConfig.credentialsSecret` field is renamed to `.clusterConfig.credentialsSecretName`, and the database connection becomes a typed struct. See https://github.com/stackabletech/airflow-operator/pull/754[airflow-operator#754]. -* Apache Druid: the `metadataStorageDatabase` field is renamed to `metadataDatabase` and restructured. See https://github.com/stackabletech/druid-operator/pull/814[druid-operator#814]. -* Apache Hive: the connection becomes a typed struct; because the `database` field is removed, you must record the `username` and add it to the Secret named in `credentialsSecretName` under the `username` key before upgrading. See https://github.com/stackabletech/hive-operator/pull/674[hive-operator#674]. -* Apache Superset: the connection becomes a typed struct, and the operator now uses an internal Secret for the Superset `SECRET_KEY` (created automatically if missing). See https://github.com/stackabletech/superset-operator/pull/722[superset-operator#722]. +* Apache Airflow: the `.clusterConfig.credentialsSecret` field is renamed to `.clusterConfig.credentialsSecretName`, and the database connection becomes a typed struct. + Consult the xref:airflow:usage-guide/database-connections.adoc[database connections] documentation page for more details. + Implemented in https://github.com/stackabletech/airflow-operator/pull/754[airflow-operator#754]. +* Apache Druid: the `metadataStorageDatabase` field is renamed to `metadataDatabase` and is now a typed struct. + Implemented in https://github.com/stackabletech/druid-operator/pull/814[druid-operator#814]. +* Apache Hive: the connection becomes a typed struct; because the `database` field is removed, the `username` field must be transferred to the Secret referenced in `credentialsSecretName` under the `username` key before upgrading. + Implemented in https://github.com/stackabletech/hive-operator/pull/674[hive-operator#674]. +* Apache Superset: the connection becomes a typed struct, and the operator now uses an internal Secret for the Superset `SECRET_KEY` (created automatically if missing). + Consult the xref:superset:usage-guide/database-connections.adoc[database connections] documentation page for more details. + Implemented in https://github.com/stackabletech/superset-operator/pull/722[superset-operator#722]. ==== // TODO(verify): xref to the per-operator database-connection usage guides (docs checking). @@ -58,23 +73,23 @@ Tracked in https://github.com/stackabletech/issues/issues/238[issues#238]. -- * Setting a custom client authentication method is now supported for Apache Airflow, Apache Superset and Apache Druid. Since Apache Druid 35 and 36, OIDC authentication can fail if the provider (for example Keycloak) advertises `private_key_jwt`, because Druid may attempt to use it. - This feature lets you override the method so that OIDC authentication works again (this was a known issue in the 26.3 release). + This feature lets you override the method so that OIDC authentication works again (this was a known issue in the 26.3.0 release). Tracked in https://github.com/stackabletech/issues/issues/838[issues#838]. -+ -// TODO(hygiene): issues#838 is still labelled scheduled-for/26.7.0 - should be release/26.7.0. - -// TODO(verify - image registry, issues#716): the author's draft note says the SDP now also publishes Helm Charts to quay.io directly from CI, and that operators installed from there pull product images from quay.io. However the quay.io part may not have shipped for 26.7.0, lfrancke's review questions were unresolved, and it is only breaking for non-Helm installs (a new env var / CLI arg is required). Confirm scope before writing. See https://github.com/stackabletech/issues/issues/716[issues#716]. +* All our xref:demos:index.adoc[demos] can now be installed in non-default namespaces. + Additionally, demos and stacks can now be uninstalled. + Please consult the xref:management:stackablectl:commands/demo.adoc[`demo`] and xref:management:stackablectl:commands/stack.adoc[`stack`] command documentation pages for details and known limits. + Tracked in https://github.com/stackabletech/stackablectl/issues/187[stackablectl#187]. ===== Apache Airflow The Apache Airflow operator now allows specifying CA certificates when using git-sync. -// TODO(verify): xref to the DAG mounting / git-sync usage guide. +See the documentation page on how to xref:airflow:usage-guide/mounting-dags.adoc#_via_git_sync[mount DAGs] for more details. Implemented in https://github.com/stackabletech/airflow-operator/pull/750[airflow-operator#750]. ===== Apache NiFi The Apache NiFi operator now allows specifying CA certificates when using git-sync. -// TODO(verify): xref to the NiFi git-sync usage guide. +Consult the xref:nifi:usage_guide/custom-components.adoc[custom components] documentation page for more details. Implemented in https://github.com/stackabletech/nifi-operator/pull/903[nifi-operator#903]. ===== Apache Superset @@ -82,30 +97,31 @@ Implemented in https://github.com/stackabletech/nifi-operator/pull/903[nifi-oper Apache Superset now supports running asynchronous queries via Celery. Two new roles, `worker` and `beat` (limited to a single replica), have been added. Redis is supported as the broker queue and results backend. -// TODO(verify): xref to the Superset usage guide. +Consult the xref:superset:usage-guide/database-connections.adoc[database connections] documentation page for more details. Implemented in https://github.com/stackabletech/superset-operator/pull/724[superset-operator#724]. ===== Open Policy Agent -The Open Policy Agent operator now supports `configOverrides` for its `config.json`. -JSON Patches and JSON Merge Patches let you override the nested structure of the file. -// TODO(verify): xref to the OPA configuration/environment overrides docs. -Implemented in https://github.com/stackabletech/opa-operator/pull/818[opa-operator#818]. +* Previously, the user-info-fetcher's outbound HTTP requests (to Microsoft Entra, Keycloak, and the XFSC AAS) had no timeout, so a slow or unresponsive backend could make a request hang indefinitely and stall user-info lookups. + With this release, every outbound request has a 60-second overall timeout and a 15-second connection timeout. + Implemented in https://github.com/stackabletech/opa-operator/pull/859[opa-operator#859]. +* The Open Policy Agent operator now supports `configOverrides` for its `config.json`. + JSON Patches and JSON Merge Patches let you override the nested structure of the file. + Consult the xref:opa:usage-guide/configuration-environment-overrides.adoc#_configuration_properties[configuration & environment overrides] documentation page for more details. + Implemented in https://github.com/stackabletech/opa-operator/pull/818[opa-operator#818]. ===== OpenSearch -* Settings managed by the operator are now hot-reloaded when changed, without Pod restarts. - // TODO(verify): xref to the OpenSearch security docs. +* Security settings managed by the operator are now hot-reloaded when changed, without Pod restarts. Implemented in https://github.com/stackabletech/opensearch-operator/pull/130[opensearch-operator#130]. * The OpenSearch operator now supports JSON Merge Patches, JSON Patches and user-provided JSON files in `configOverrides`, making it easier to define nested types and non-string values. - // TODO(verify): xref to the OpenSearch configuration/environment overrides docs. Confirm feature vs improvement placement (author wrote "OpenSearch" without a section). + Consult the xref:opensearch:usage-guide/configuration-environment-overrides.adoc#_configuration_properties[configuration & environment overrides] documentation page for more details. Implemented in https://github.com/stackabletech/opensearch-operator/pull/137[opensearch-operator#137]. ===== Trino -* Trino now has first-class support for the PostgreSQL connector, built on the new generic database connection mechanism. +* Trino now has first-class support for the https://trino.io/docs/current/connector/postgresql.html[PostgreSQL connector {external-link-icon}^], built on the new generic database connection mechanism. Previously, users had to fall back to the `generic` connector. - // TODO(verify): xref to the Trino PostgreSQL connector docs. Implemented in https://github.com/stackabletech/trino-operator/pull/883[trino-operator#883]. * A new `.spec.name.inferred.replaceHyphensWithUnderscores` field on `TrinoCatalog` allows tweaking the catalog name in Trino by replacing `-` with `_`. This lets you use valid Kubernetes resource names while keeping the convenience of `_` in catalog names. @@ -113,14 +129,41 @@ Implemented in https://github.com/stackabletech/opa-operator/pull/818[opa-operat ===== Stackable secret-operator -The Stackable secret-operator now supports adding domain components (DCs) to the subject DN of TLS certificates, via the `secrets.stackable.tech/backend.autotls.cert.domain-components-in-subject-dn` volume annotation. -// TODO(verify): xref to the secret-operator volume docs. -Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret-operator#708]. +* The Stackable secret-operator now supports adding domain components (DCs) to the subject DN of TLS certificates, via the xref:{secret-volume-annotation}[`secrets.stackable.tech/backend.autotls.cert.domain-components-in-subject-dn`] volume annotation. ++ +-- +[source,yaml] +---- +volumes: + - name: tls + ephemeral: + volumeClaimTemplate: + metadata: + annotations: + secrets.stackable.tech/backend.autotls.cert.domain-components-in-subject-dn: "true" + secrets.stackable.tech/class: tls + secrets.stackable.tech/scope: node,pod + spec: + storageClassName: secrets.stackable.tech + accessModes: + - ReadWriteOnce + resources: + requests: + storage: "1" +---- -// TODO(decide): stackablectl#187 (uninstall Stacks and Demos) - stackablectl/stackable-cockpit is versioned separately from the SDP. If it belongs here, the author's note is: -// - All demos and stacks can now be installed in non-default namespaces. -// - `stackablectl demo uninstall` / `stackablectl stack uninstall` remove the demo/stack namespace, the Stackable operators and CRDs, and the operator namespace (use `--skip-operators-and-crds` to keep operators/CRDs). -// See https://github.com/stackabletech/stackablectl/issues/187[stackablectl#187]. +A pod of a StatefulSet could then serve a TLS certificate with the following subject DN: + +[source,plain] +---- +CN=generated certificate for pod, DC=my-pod-0, DC=my-statefulset-service, DC=my-namespace, DC=svc, DC=cluster, DC=local +---- + +Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret-operator#708]. +-- +* The name of the key in the ConfigMap/Secret that holds the PEM-encoded CA certificate of a `TrustStore` is now configurable, via xref:secret-operator:truststore.adoc[`.spec.tlsPemCaName`]. + This is needed, for example, to use the generated Secret within an OpenShift Ingress. + Implemented in https://github.com/stackabletech/secret-operator/pull/679[secret-operator#679]. ==== Platform improvements @@ -132,17 +175,40 @@ Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret- ====== Miscellaneous +* *Breaking:* All operators now support dynamic image repositories during product image selection instead of hard-coding `oci.stackable.tech`. + This enables us to automatically pick the correct registry when operators are installed via the Helm Chart or the OLM manifest. + This is achieved by introducing the new `--image-repository` CLI argument and `IMAGE_REPOSITORY` environment variable. ++ +-- +[NOTE] +==== +When installing the operators via the Helm Chart from `oci.stackable.tech` or `quay.io`, the environment variable is automatically set and installation continues to work as before. +==== + +Tracked in https://github.com/stackabletech/issues/issues/716[issues#716]. +-- +* *Breaking:* The `image.repository` Helm value must no longer include the operator name. + Previously it looked like `oci.stackable.tech/sdp/airflow-operator`; it must now be `oci.stackable.tech/sdp`. + Tracked in https://github.com/stackabletech/issues/issues/716[issues#716]. +* RBAC permissions for all Stackable operators deployed via Helm have been audited, and unnecessary permissions were removed. + Tracked in https://github.com/stackabletech/issues/issues/798[issues#798]. * ConfigMaps and Secrets can now be excluded from the restart-controller. Set the `restarter.stackable.tech/ignore` label on a ConfigMap or Secret, or the `restarter.stackable.tech/ignore-configmap.x` / `restarter.stackable.tech/ignore-secret.x` annotations on a StatefulSet. - // TODO(verify): xref to the restarter docs. + Consult the xref:commons-operator:restarter.adoc#_configmapsecret[restarter] documentation page of the Stackable commons-operator for more details. Implemented in https://github.com/stackabletech/commons-operator/pull/410[commons-operator#410]. +[#airflow-improvements-26_7_0] +===== Apache Airflow + +*Breaking:* `EXPERIMENTAL_FILE_HEADER` and `EXPERIMENTAL_FILE_FOOTER` in `webserver_config.py` (for arbitrary Python code) have been renamed to `FILE_HEADER` and `FILE_FOOTER`. +Implemented in https://github.com/stackabletech/airflow-operator/pull/777[airflow-operator#777]. + ===== Apache Kafka The FQDNs of the Kafka Pods are now included in the subject DNs of their TLS certificates. OPA Rego rules can now check these FQDNs. Existing OPA rules may need to be adjusted. -// TODO(verify): xref to the Kafka security usage guide (author linked .../kafka/usage-guide/security/#_authorization). +Consult the xref:kafka:usage-guide/security.adoc#_authorization[security] documentation page for more details. Implemented in https://github.com/stackabletech/kafka-operator/pull/972[kafka-operator#972]. ===== Apache NiFi @@ -152,6 +218,12 @@ Cold-start flow election then completes as soon as all expected nodes report in, The timeout stays at NiFi's upstream default of 5 minutes as a fallback, so there is no correctness change if fewer nodes than expected show up. Implemented in https://github.com/stackabletech/nifi-operator/pull/953[nifi-operator#953]. +[#superset-improvements-26_7_0] +===== Apache Superset + +*Breaking:* `EXPERIMENTAL_FILE_HEADER` and `EXPERIMENTAL_FILE_FOOTER` in `superset_config.py` (for arbitrary Python code) have been renamed to `FILE_HEADER` and `FILE_FOOTER`. +Implemented in https://github.com/stackabletech/superset-operator/pull/721[superset-operator#721]. + ===== Open Policy Agent The OPA DaemonSet now uses `maxSurge=1` and `maxUnavailable=0` for its rolling update strategy. @@ -163,16 +235,9 @@ Implemented in https://github.com/stackabletech/opa-operator/pull/819[opa-operat The verification of the OpenSearch nodes' certificates has been extended. The nodes' certificates now also include the FQDNs of the nodes, which are verified by the other nodes using the `plugins.security.nodes_dn` configuration setting. -Previously, it was only verified that the certificates were issued by the secret-operator. +Previously, it was only verified that the certificates were issued by the Stackable secret-operator. Implemented in https://github.com/stackabletech/opensearch-operator/pull/144[opensearch-operator#144]. -===== Stackable secret-operator - -The name of the key in the ConfigMap/Secret that holds the PEM-encoded CA certificate of a `TrustStore` is now configurable, via `.spec.tlsPemCaName`. -This is needed, for example, to use the generated Secret within an OpenShift Ingress. -// TODO(verify): xref to the TrustStore docs. -Implemented in https://github.com/stackabletech/secret-operator/pull/679[secret-operator#679]. - ==== Platform fixes ===== General @@ -183,23 +248,26 @@ Implemented in https://github.com/stackabletech/secret-operator/pull/679[secret- Fixed in https://github.com/stackabletech/issues/issues/547[issues#547]. * Internal secrets are now stored in mutable Kubernetes Secret objects. Previously these objects were immutable and, if deleted, were recreated with new random values; because immutable Secrets are cached, this could disrupt clusters (for example workers failing to join). - This release migrates existing internal Secrets to mutable versions for Trino (https://github.com/stackabletech/trino-operator/pull/876[trino-operator#876]) and Apache Airflow (https://github.com/stackabletech/airflow-operator/pull/770[airflow-operator#770]). - Tracked in https://github.com/stackabletech/issues/issues/843[issues#843]. + This release migrates existing internal Secrets to mutable versions for +** Trino in https://github.com/stackabletech/trino-operator/pull/876[trino-operator#876] and +** Apache Airflow in https://github.com/stackabletech/airflow-operator/pull/770[airflow-operator#770] + ++ +Tracked in https://github.com/stackabletech/issues/issues/843[issues#843]. ===== Apache Airflow OPA authorization now works for Apache Airflow 3 with any user role. Previously, OPA rulesets were only applied for users with the `Admin` role. -// TODO(hygiene): airflow-operator#800 is still labelled scheduled-for/26.7.0 - should be release/26.7.0. Underlying fix shipped in docker-images#1512. -Fixed in https://github.com/stackabletech/airflow-operator/pull/800[airflow-operator#800]. +Fixed in https://github.com/stackabletech/docker-images/pull/1512[docker-images#1512] and regression tested in https://github.com/stackabletech/airflow-operator/pull/800[airflow-operator#800]. ===== Apache NiFi The NiFi OPA authorizer plugin no longer exhausts native threads under load. Previously, the plugin could create a new HTTP client for each authorization request, spawning large numbers of selector and executor threads under NiFi load and eventually exhausting native threads. With this release, all OPA authorization requests share a single pooled HTTP client with a fixed-size daemon-thread executor. -Existing deployments need no configuration changes, as defaults are provided; the client is configurable via `OPA_HTTP_POOL_SIZE` (default 32), `OPA_HTTP_CONNECT_TIMEOUT_MS` (default 2000) and `OPA_HTTP_REQUEST_TIMEOUT_MS` (default 5000). -Fixed in https://github.com/stackabletech/nifi-opa-plugin/issues/30[nifi-opa-plugin#30]. +Existing deployments need no configuration changes, as defaults are provided. +Tracked in https://github.com/stackabletech/nifi-opa-plugin/issues/30[nifi-opa-plugin#30] and fixed in https://github.com/stackabletech/nifi-opa-plugin/issues/30[nifi-opa-plugin#31]. [#spark-fixes-26_7_0] ===== Apache Spark @@ -208,9 +276,8 @@ Fixed in https://github.com/stackabletech/nifi-opa-plugin/issues/30[nifi-opa-plu They were unintentionally released as cluster-wide objects in SDP 26.3. You must recreate any templates in the namespaces where the applications run. Fixed in https://github.com/stackabletech/spark-k8s-operator/pull/694[spark-k8s-operator#694] (reapplied in https://github.com/stackabletech/spark-k8s-operator/pull/720[spark-k8s-operator#720]). - // TODO(hygiene): spark-k8s-operator#720 is still labelled scheduled-for/26.7.0 - should be release/26.7.0. * User arguments are now correctly propagated to the Spark Connect server. - Previously, the newline concatenation in the Spark Connect server entrypoint was wrong, so user arguments were dropped. + Previously, the newline concatenation in the Spark Connect server entrypoint was faulty which resulted in user arguments to be dropped. Fixed in https://github.com/stackabletech/spark-k8s-operator/pull/696[spark-k8s-operator#696]. * Spark executors now shut down gracefully on `SIGTERM`. Previously the executor did not receive `SIGTERM` when the driver did, so it was `SIGKILL`ed after the grace period. @@ -222,27 +289,15 @@ The Microsoft Entra backend of the OPA user-info-fetcher now retrieves complete Previously, it read only the first page of a user's group memberships, so users belonging to many groups received an incomplete group list. With this release, the Entra backend follows the result pages until every page has been retrieved (bounded to 100 pages as a safeguard). Fixed in https://github.com/stackabletech/opa-operator/issues/858[opa-operator#858]. -+ -// TODO(orphan note - needs its own labelled source): lfrancke's release note on issue #858 also describes a SECOND change not traceable to #858/#859 - "the user-info-fetcher's outbound HTTP requests (to Microsoft Entra, Keycloak, and the XFSC AAS) now have a 60-second overall timeout and a 15-second connection timeout (previously none, so a slow backend could hang lookups indefinitely)". Find the PR that implemented this, confirm it's in 26.7.0, and label it - then add it here. ===== Trino -Hot-reloading of password file Secrets works again. -In the previous release a change to the Secret triggered a Pod restart; the password file Secrets are now excluded from the restart-controller, so hot-reloading is supported again. +In the last release, hot-reloading of password file Secrets did not work anymore because a change to the Secret triggered a restart of the pods. +With this release, the password file Secrets are excluded from the restart-controller and hot-reloading is supported again. Fixed in https://github.com/stackabletech/trino-operator/pull/868[trino-operator#868]. -==== Platform deprecations - -// TODO(verify): no feature/API deprecations found in the labelled items (only product-version deprecations, which are listed under Supported versions below). Confirm there are none. - ==== Platform removals -[#airflow-removals-26_7_0] -===== Apache Airflow - -*Breaking:* `EXPERIMENTAL_FILE_HEADER` and `EXPERIMENTAL_FILE_FOOTER` in `webserver_config.py` (for arbitrary Python code) have been renamed to `FILE_HEADER` and `FILE_FOOTER`. -Removed in https://github.com/stackabletech/airflow-operator/pull/777[airflow-operator#777]. - [#nifi-removals-26_7_0] ===== Apache NiFi @@ -255,23 +310,10 @@ Removed in https://github.com/stackabletech/airflow-operator/pull/777[airflow-op Removed in https://github.com/stackabletech/nifi-operator/pull/960[nifi-operator#960]. -[#superset-removals-26_7_0] -===== Apache Superset - -*Breaking:* `EXPERIMENTAL_FILE_HEADER` and `EXPERIMENTAL_FILE_FOOTER` in `superset_config.py` (for arbitrary Python code) have been renamed to `FILE_HEADER` and `FILE_FOOTER`. -Removed in https://github.com/stackabletech/superset-operator/pull/721[superset-operator#721]. - ==== Supported versions ===== Product versions -// Reconciled against portal.stackable.build (source of truth) by diffing the 26.3 and 26.7 exports. -// Added = in 26.7 not in 26.3; Removed = in 26.3 not in 26.7; statuses (LTS/experimental/deprecated) from the 26.7 export. -// Overrides where the portal was wrong: -// - Spark 4.1.2 appears in the 26.3 portal export by mistake (it was not in 26.3.0; docker-images#1419 added only 3.5.8). It is treated as ADDED in 26.7. Portal maintainer to be notified. -// - Vector is absent from the portal export; taken from docker-images#1486 (add 0.55.0, remove 0.52.0). Portal maintainer notified. -// Deprecated list = versions newly deprecated this release (status changed to deprecated vs 26.3), not the full standing deprecated set. - As with previous SDP releases, many product images have been updated to newer versions. Refer to the xref:operators:supported_versions.adoc[supported versions] documentation for a complete overview including LTS versions or deprecations. @@ -290,10 +332,10 @@ The following new product versions are now supported: * Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[37.0.0 (LTS)] ** Note: Druid 37.0.0 switched to AWS SDK v2. TLS is now enforced for S3 connections, and the S3 `region` is now required (the operator passes `aws.region` from `S3Connection.region`). * Apache HBase: https://github.com/stackabletech/docker-images/issues/1491[2.6.6 (LTS)] -* Apache Hadoop HDFS: https://github.com/stackabletech/docker-images/issues/1494[3.4.3 (LTS)], https://github.com/stackabletech/docker-images/issues/1494[3.5.0] +* Apache Hadoop: https://github.com/stackabletech/docker-images/issues/1494[3.4.3 (LTS)], https://github.com/stackabletech/docker-images/issues/1494[3.5.0] * Apache Kafka: https://github.com/stackabletech/docker-images/issues/1497[3.9.2 (LTS)], https://github.com/stackabletech/docker-images/issues/1497[4.2.1 (experimental)] * Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[2.9.0] -* Apache Phoenix: https://github.com/stackabletech/docker-images/issues/1491[5.3.1 (LTS)] +* Apache Phoenix: https://github.com/stackabletech/docker-images/issues/1491[5.3.1] * Apache Spark: https://github.com/stackabletech/docker-images/issues/1502[4.1.2 (LTS)] * Apache Superset: https://github.com/stackabletech/docker-images/issues/1504[6.1.0] * Apache ZooKeeper: https://github.com/stackabletech/docker-images/issues/1506[3.9.5 (LTS)] @@ -308,7 +350,8 @@ The following product versions are deprecated and will be removed in a later rel * Apache Airflow: https://github.com/stackabletech/docker-images/issues/1489[3.1.6] * Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[35.0.1], https://github.com/stackabletech/docker-images/issues/1490[30.0.1] -* Apache Hadoop HDFS: https://github.com/stackabletech/docker-images/issues/1494[3.4.2] +* Apache HBase: https://github.com/stackabletech/docker-images/issues/1491[2.6.4] +* Apache Hadoop: https://github.com/stackabletech/docker-images/issues/1494[3.4.2] * Apache Hive: https://github.com/stackabletech/docker-images/issues/1495[4.0.1] * Apache Kafka: https://github.com/stackabletech/docker-images/issues/1497[3.9.1] * Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[2.7.2] @@ -346,7 +389,6 @@ This release supports the following Kubernetes versions: * `1.32` * `1.31` -// Portal diff shows no Kubernetes versions were dropped between 26.3 and 26.7 (1.36 added). Confirm before publishing. No Kubernetes versions were dropped in this release. ===== OpenShift versions @@ -359,7 +401,6 @@ This release is available in the RedHat Certified Operator Catalog for the follo * `4.19` * `4.18` -// Portal diff shows no OpenShift versions were dropped between 26.3 and 26.7 (4.22 added). Confirm before publishing. No OpenShift versions were dropped in this release. ==== Upgrade from 26.3 @@ -427,7 +468,5 @@ The operators are installed directly, without adding the Helm Chart repository f helm install --wait -operator oci://oci.stackable.tech/sdp-charts/-operator --version 26.7.0 ---- -[#known-issues-26_7_0] -==== Known issues - -// TODO(verify): none identified yet from the labelled items. +// [#known-issues-26_7_0] +// ==== Known issues From c723a178f165a1fc9c8eafdf54cf26492e576948 Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 23 Jul 2026 08:37:20 +0200 Subject: [PATCH 12/19] docs: Fix underscore rendering issue --- modules/ROOT/partials/release-notes/release-26.7.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index cc2377ebe..c257d8cf4 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -124,7 +124,7 @@ Implemented in https://github.com/stackabletech/superset-operator/pull/724[super Previously, users had to fall back to the `generic` connector. Implemented in https://github.com/stackabletech/trino-operator/pull/883[trino-operator#883]. * A new `.spec.name.inferred.replaceHyphensWithUnderscores` field on `TrinoCatalog` allows tweaking the catalog name in Trino by replacing `-` with `_`. - This lets you use valid Kubernetes resource names while keeping the convenience of `_` in catalog names. + This lets you use valid Kubernetes resource names while keeping the convenience of underscores in catalog names. Implemented in https://github.com/stackabletech/trino-operator/pull/903[trino-operator#903]. ===== Stackable secret-operator From 1bf2bbb5e900709bff00b4ec1b76b60c9405f137 Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 23 Jul 2026 08:43:41 +0200 Subject: [PATCH 13/19] docs: Apply suggestions from review Co-authored-by: Lars Francke --- .../partials/release-notes/release-26.7.adoc | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index c257d8cf4..e3e837550 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -46,6 +46,17 @@ See xref:concepts:artifact-registries/index.adoc[the artifact registries page] f Tracked in https://github.com/stackabletech/issues/issues/716[issues#716]. +====== Stackable Hub + +Stackable Hub is now available at https://hub.stackable.tech[hub.stackable.tech,window=_blank]. +It is a public, read-only portal for published SDP releases (and upcoming ones), bundled component version timelines, and operator CRDs including their schemas, API versions and roles per release. +A CORS-enabled, unauthenticated JSON API is available under `/api/v1/` for use in CI jobs, scripts and MCP servers. +LLMs can make use of the new llms.txt[https://hub.stackable.tech/llms.txt] file which now https://docs.stackable.tech/llms.txt[also exists] for the documentation. +We plan on migrating the current https://crds.stackable.tech[CRD browser] to the Hub in the near future. +We also plan on migrating the current https://sboms.stackable.tech[SBOM browser] to the Hub in the near future. + +This is an early version of the Hub and we're happy to receive feedback. + ====== Miscellaneous * [[generic-db-26_7_0]]The SDP now provides a generic database connection mechanism, giving a consistent, typed way to configure connections to external SQL databases across operators. @@ -182,7 +193,8 @@ Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret- -- [NOTE] ==== -When installing the operators via the Helm Chart from `oci.stackable.tech` or `quay.io`, the environment variable is automatically set and installation continues to work as before. +This is only breaking if you deploy operators *without* our Helm Charts or OLM manifests, in which case you must now supply `--image-repository` yourself. +When installing via Helm or OLM the argument is set automatically, and installation works as before. ==== Tracked in https://github.com/stackabletech/issues/issues/716[issues#716]. @@ -267,7 +279,7 @@ The NiFi OPA authorizer plugin no longer exhausts native threads under load. Previously, the plugin could create a new HTTP client for each authorization request, spawning large numbers of selector and executor threads under NiFi load and eventually exhausting native threads. With this release, all OPA authorization requests share a single pooled HTTP client with a fixed-size daemon-thread executor. Existing deployments need no configuration changes, as defaults are provided. -Tracked in https://github.com/stackabletech/nifi-opa-plugin/issues/30[nifi-opa-plugin#30] and fixed in https://github.com/stackabletech/nifi-opa-plugin/issues/30[nifi-opa-plugin#31]. +Tracked in https://github.com/stackabletech/nifi-opa-plugin/issues/30[nifi-opa-plugin#30] and fixed in https://github.com/stackabletech/nifi-opa-plugin/pull/31[nifi-opa-plugin#31]. [#spark-fixes-26_7_0] ===== Apache Spark @@ -315,7 +327,7 @@ Removed in https://github.com/stackabletech/nifi-operator/pull/960[nifi-operator ===== Product versions As with previous SDP releases, many product images have been updated to newer versions. -Refer to the xref:operators:supported_versions.adoc[supported versions] documentation for a complete overview including LTS versions or deprecations. +Refer to the https://hub.stackable.tech/releases/26.7[new Stackable Hub] for a complete overview including LTS versions or deprecations. ====== New LTS versions From 14272b27bfa0edc733e813185b5eed986988397a Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 23 Jul 2026 08:52:48 +0200 Subject: [PATCH 14/19] docs: Apply suggestion from review --- modules/ROOT/partials/release-notes/release-26.7.adoc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index e3e837550..503aacc0f 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -186,9 +186,8 @@ Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret- ====== Miscellaneous -* *Breaking:* All operators now support dynamic image repositories during product image selection instead of hard-coding `oci.stackable.tech`. - This enables us to automatically pick the correct registry when operators are installed via the Helm Chart or the OLM manifest. - This is achieved by introducing the new `--image-repository` CLI argument and `IMAGE_REPOSITORY` environment variable. +* *Breaking:* Operators now determine the registry for product images from where they were installed, instead of hard-coding `oci.stackable.tech`. + This is controlled by a new `--image-repository` CLI argument (and the `IMAGE_REPOSITORY` environment variable), which lets the correct registry be selected automatically when an operator is installed via its Helm Chart or Operator Lifecycle Manager (OLM) manifest. + -- [NOTE] From b19488f6cfeb96b2a1263fc2c0e9cf4d9210168d Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 23 Jul 2026 08:58:35 +0200 Subject: [PATCH 15/19] docs: Add vulnerabilities section --- modules/ROOT/partials/release-notes/release-26.7.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 503aacc0f..d0f16d71c 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -182,7 +182,7 @@ Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret- ====== Vulnerabilities -// TODO(verify): CVE counts. e.g. "NNN CVEs were fixed in the Stackable product images. This includes N critical and NN high-severity CVEs." +133 vulnerabilities were fixed in SDP 26.7, including 10 critical and 62 high vulnerabilities, ====== Miscellaneous From 0b450d1af0b1f72cf9f7f0d1a64c47e1ff686c56 Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 23 Jul 2026 10:35:06 +0200 Subject: [PATCH 16/19] docs: Add Druid deprecation admonition --- modules/ROOT/pages/release-notes.adoc | 19 +++++++++++++++++++ .../partials/release-notes/release-26.7.adoc | 2 +- 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index f7d3d2b47..56e9e2197 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -6,6 +6,25 @@ The Stackable Data Platform consists of multiple operators that work together. Periodically a platform release is made, including all components of the platform at a specific version. +[WARNING] +.Stackable support for Apache Druid is deprecated +==== +Stackable's support for xref:druid:index.adoc[Apache Druid] is deprecated as of SDP link:#_release_26_7[26.7] and is planned for removal in SDP 27.11. +This concerns only the Apache Druid integration in the Stackable Data Platform (SDP) — that is, this operator. +Apache Druid itself is an independent, actively maintained project and is not affected; only its inclusion in the SDP is being discontinued. +We have taken this decision because we have not seen adoption of Apache Druid among our customers. + +The planned timeline is: + +* SDP 26.7 ships Apache Druid link:#druid-37_0_0[37.0.0], the final LTS version we plan to provide, with approximately one year of support. +* SDP 27.7 is planned to be the last release that includes Apache Druid. +* Apache Druid is planned to be removed in SDP 27.11. + +There is no direct replacement for Apache Druid within the SDP. +If you rely on Apache Druid, please contact us via our https://stackable.tech/[homepage] or the https://github.com/orgs/stackabletech/discussions[Discussions forum] to discuss your options. +If you are willing to fund its continued development and support, we are open to reconsidering this decision. +==== + // WARNING: Please keep the empty newlines, otherwise headings are broken. include::partial$release-notes/release-26.7.adoc[] diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index d0f16d71c..b7544f52d 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -340,7 +340,7 @@ The following new product versions are now supported: * Apache Airflow: https://github.com/stackabletech/docker-images/issues/1489[3.2.2] ** Note: this Airflow version uses Python 3.14. // TODO(verify): the docker-images#1489 note about the webserver no longer serving metrics referred to 3.2.1; confirm it still applies to 3.2.2. -* Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[37.0.0 (LTS)] +* [[druid-37_0_0]]Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[37.0.0 (LTS)] ** Note: Druid 37.0.0 switched to AWS SDK v2. TLS is now enforced for S3 connections, and the S3 `region` is now required (the operator passes `aws.region` from `S3Connection.region`). * Apache HBase: https://github.com/stackabletech/docker-images/issues/1491[2.6.6 (LTS)] * Apache Hadoop: https://github.com/stackabletech/docker-images/issues/1494[3.4.3 (LTS)], https://github.com/stackabletech/docker-images/issues/1494[3.5.0] From b8554a97b62d7b0a8a92474fad060bd9a9b2cf68 Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 23 Jul 2026 11:49:17 +0200 Subject: [PATCH 17/19] docs: Update getiing-started page --- modules/ROOT/pages/getting-started.adoc | 28 ++++++++++++------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/modules/ROOT/pages/getting-started.adoc b/modules/ROOT/pages/getting-started.adoc index c58e7cd19..4efc7d166 100644 --- a/modules/ROOT/pages/getting-started.adoc +++ b/modules/ROOT/pages/getting-started.adoc @@ -37,11 +37,11 @@ These can be installed on any node that has access to the Kubernetes control pla In this example we will install them on the controller node. Stackable operators can be installed using `stackablectl`. -Run the following commands to install ZooKeeper, Kafka and NiFi from the Stackable 26.3 release. +Run the following commands to install ZooKeeper, Kafka and NiFi from the Stackable 26.7 release. [source,bash] ---- -stackablectl release install -i commons -i secret -i listener -i zookeeper -i kafka -i nifi 26.3 +stackablectl release install -i commons -i secret -i listener -i zookeeper -i kafka -i nifi 26.7 ---- .Using Helm instead @@ -56,12 +56,12 @@ Install the operators: [source,bash] ---- -helm install zookeeper-operator oci://oci.stackable.tech/sdp-charts/zookeeper-operator --version=26.3.0 -helm install kafka-operator oci://oci.stackable.tech/sdp-charts/kafka-operator --version=26.3.0 -helm install secret-operator oci://oci.stackable.tech/sdp-charts/secret-operator --version=26.3.0 -helm install listener-operator oci://oci.stackable.tech/sdp-charts/listener-operator --version=26.3.0 -helm install commons-operator oci://oci.stackable.tech/sdp-charts/commons-operator --version=26.3.0 -helm install nifi-operator oci://oci.stackable.tech/sdp-charts/nifi-operator --version=26.3.0 +helm install zookeeper-operator oci://oci.stackable.tech/sdp-charts/zookeeper-operator --version=26.7.0 +helm install kafka-operator oci://oci.stackable.tech/sdp-charts/kafka-operator --version=26.7.0 +helm install secret-operator oci://oci.stackable.tech/sdp-charts/secret-operator --version=26.7.0 +helm install listener-operator oci://oci.stackable.tech/sdp-charts/listener-operator --version=26.7.0 +helm install commons-operator oci://oci.stackable.tech/sdp-charts/commons-operator --version=26.7.0 +helm install nifi-operator oci://oci.stackable.tech/sdp-charts/nifi-operator --version=26.7.0 ---- ==== @@ -70,12 +70,12 @@ You can check which operators are installed using `stackablectl operator install [source,console] ---- OPERATOR VERSION NAMESPACE STATUS LAST UPDATED -commons 26.3.0 default deployed 2026-03-16 17:58:32.916032854 +0100 CET -kafka 26.3.0 default deployed 2026-03-16 17:58:55.036115353 +0100 CET -listener 26.3.0 default deployed 2026-03-16 17:59:18.136775259 +0100 CET -nifi 26.3.0 default deployed 2026-03-16 17:59:51.927081648 +0100 CET -secret 26.3.0 default deployed 2026-03-16 18:00:05.060241771 +0100 CET -zookeeper 26.3.0 default deployed 2026-03-16 18:00:08.425686918 +0100 CET +commons 26.7.0 default deployed 2026-07-21 17:58:32.916032854 +0100 CET +kafka 26.7.0 default deployed 2026-07-21 17:58:55.036115353 +0100 CET +listener 26.7.0 default deployed 2026-07-21 17:59:18.136775259 +0100 CET +nifi 26.7.0 default deployed 2026-07-21 17:59:51.927081648 +0100 CET +secret 26.7.0 default deployed 2026-07-21 18:00:05.060241771 +0100 CET +zookeeper 26.7.0 default deployed 2026-07-21 18:00:08.425686918 +0100 CET ---- == Deploying Stackable Services From 0c0e407e4ced0a27144b38581252578d2b4ae69b Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 23 Jul 2026 15:48:18 +0200 Subject: [PATCH 18/19] docs: Apply suggestions from review Co-authored-by: Lars Francke --- modules/ROOT/pages/release-notes.adoc | 2 +- modules/ROOT/partials/release-notes/release-26.7.adoc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 56e9e2197..6162e8b29 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -10,7 +10,7 @@ Periodically a platform release is made, including all components of the platfor .Stackable support for Apache Druid is deprecated ==== Stackable's support for xref:druid:index.adoc[Apache Druid] is deprecated as of SDP link:#_release_26_7[26.7] and is planned for removal in SDP 27.11. -This concerns only the Apache Druid integration in the Stackable Data Platform (SDP) — that is, this operator. +This concerns only the Apache Druid integration in the Stackable Data Platform (SDP). Apache Druid itself is an independent, actively maintained project and is not affected; only its inclusion in the SDP is being discontinued. We have taken this decision because we have not seen adoption of Apache Druid among our customers. diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index b7544f52d..edbfec830 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -51,7 +51,7 @@ Tracked in https://github.com/stackabletech/issues/issues/716[issues#716]. Stackable Hub is now available at https://hub.stackable.tech[hub.stackable.tech,window=_blank]. It is a public, read-only portal for published SDP releases (and upcoming ones), bundled component version timelines, and operator CRDs including their schemas, API versions and roles per release. A CORS-enabled, unauthenticated JSON API is available under `/api/v1/` for use in CI jobs, scripts and MCP servers. -LLMs can make use of the new llms.txt[https://hub.stackable.tech/llms.txt] file which now https://docs.stackable.tech/llms.txt[also exists] for the documentation. +LLMs can make use of the new https://hub.stackable.tech/llms.txt[llms.txt] file which now https://docs.stackable.tech/llms.txt[also exists] for the documentation. We plan on migrating the current https://crds.stackable.tech[CRD browser] to the Hub in the near future. We also plan on migrating the current https://sboms.stackable.tech[SBOM browser] to the Hub in the near future. From d245186b1e31197c458ceadce00b07d151c6b85a Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 23 Jul 2026 15:51:54 +0200 Subject: [PATCH 19/19] docs: Fix two small typos --- modules/ROOT/partials/release-notes/release-26.7.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index edbfec830..26218992a 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -1,5 +1,5 @@ == Release 26.7 -:secret-volume-annotation: secret-operator:volume.adoc##_secrets_stackable_techbackend_autotls_cert_domain_components_in_subject_dn +:secret-volume-annotation: secret-operator:volume.adoc#_secrets_stackable_techbackend_autotls_cert_domain_components_in_subject_dn === 26.7.0 @@ -182,7 +182,7 @@ Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret- ====== Vulnerabilities -133 vulnerabilities were fixed in SDP 26.7, including 10 critical and 62 high vulnerabilities, +133 vulnerabilities were fixed in SDP 26.7, including 10 critical and 62 high vulnerabilities. ====== Miscellaneous