Preserve customized Python test fixtures

.chronus/changes/python-regeneration-fixtures-2026-7-28.md

@@ -0,0 +1,7 @@
+---
+changeKind: internal
+packages:
+  - "@typespec/http-client-python"
+---
+
+Track only hand-authored test fixture files instead of whole generated packages, and clean generated test code with `git clean` during regeneration.

packages/http-client-python/.gitignore

@@ -14,3 +14,56 @@ alpha
 # pip component detection reports (generated during build)
 component-detection-pip-report.json
 generator/component-detection-pip-report.json
+
+# Generated SDK fixtures with hand-authored/customized code required by tests. (begin)
+#
+# Generated SDK packages under `tests/generated/` are recreated from the specs by
+# `npm run regenerate`, so they are not committed. Only the hand-authored files
+# that regeneration deliberately preserves are tracked; every exception below
+# documents why the file cannot be regenerated.
+#
+# The repo-root .gitignore excludes `tests/**/generated/` as a whole directory,
+# which would stop git from ever descending into it, so re-include the directory
+# itself first and then ignore its contents.
+!tests/**/generated/
+tests/generated/**
+# Re-include directories so the exceptions below are reachable (git does not
+# descend into an ignored directory).
+!tests/generated/**/
+
+# `_operations/_patch.py` adds `patch_added_operation`, asserted by
+# tests/mock_api/shared/test_patch.py.
+!tests/generated/azure/authentication-api-key/authentication/apikey/_operations/_patch.py
+!tests/generated/azure/authentication-api-key/authentication/apikey/aio/_operations/_patch.py
+!tests/generated/unbranded/authentication-api-key/authentication/apikey/_operations/_patch.py
+!tests/generated/unbranded/authentication-api-key/authentication/apikey/aio/_operations/_patch.py
+
+# `models/_patch.py` registers the custom GeoJSON serializer/deserializer used by
+# tests/mock_api/azure/test_client_generator_core_alternate_type.py.
+!tests/generated/azure/azure-client-generator-core-alternate-type/specs/azure/clientgenerator/core/alternatetype/models/_patch.py
+
+# `pyproject.toml` carries a hand-added `[tool.azure-sdk-build]` section. It is
+# the fixture for the emitter behavior that keeps existing `[tool.*]` fields when
+# it rewrites `pyproject.toml`, so the file has to exist before regeneration.
+!tests/generated/azure/authentication-union/pyproject.toml
+
+# `generation-subdir`/`generation-subdir2` emit into a `_generated/` subfolder;
+# the surrounding package is a hand-written wrapper (`CustomizedClient` /
+# `AddedClient`) that regeneration must leave untouched.
+!tests/generated/azure/generation-subdir/generation/subdir/__init__.py
+!tests/generated/azure/generation-subdir/generation/subdir/_client.py
+!tests/generated/azure/generation-subdir/generation/subdir/aio/__init__.py
+!tests/generated/azure/generation-subdir/generation/subdir/aio/_client.py
+!tests/generated/unbranded/generation-subdir/generation/subdir/__init__.py
+!tests/generated/unbranded/generation-subdir/generation/subdir/_client.py
+!tests/generated/unbranded/generation-subdir/generation/subdir/aio/__init__.py
+!tests/generated/unbranded/generation-subdir/generation/subdir/aio/_client.py
+!tests/generated/azure/generation-subdir2/generation/subdir2/__init__.py
+!tests/generated/azure/generation-subdir2/generation/subdir2/_client.py
+!tests/generated/azure/generation-subdir2/generation/subdir2/aio/__init__.py
+!tests/generated/azure/generation-subdir2/generation/subdir2/aio/_client.py
+!tests/generated/unbranded/generation-subdir2/generation/subdir2/__init__.py
+!tests/generated/unbranded/generation-subdir2/generation/subdir2/_client.py
+!tests/generated/unbranded/generation-subdir2/generation/subdir2/aio/__init__.py
+!tests/generated/unbranded/generation-subdir2/generation/subdir2/aio/_client.py
+# Generated SDK fixtures with hand-authored/customized code required by tests. (end)

packages/http-client-python/CONTRIBUTING.md

@@ -135,22 +135,6 @@ When you open a PR against this package:
 
 Both must pass before your PR can be merged.
 
-### Manual Regeneration Testing
-
-You can manually trigger the [TypeSpec Python Regenerate Tests](https://github.com/Azure/azure-sdk-for-python/actions/workflows/typespec-python-regenerate.yml) workflow in `azure-sdk-for-python` to regenerate tests. The workflow checks out `microsoft/typespec` (at the ref you specify, defaulting to `main`), builds the regeneration infrastructure, installs the target emitter from npm, and runs the full regeneration.
-
-### Post-Release: Updating azure-sdk-for-python
-
-Once a new version of this emitter is released, follow these steps to update `azure-sdk-for-python`:
-
-1. Follow skill [emitter-package-update](https://github.com/Azure/azure-sdk-for-python/tree/main/.github/skills/emitter-package-update) to **create a PR** to update `eng/emitter-package.json` and `eng/emitter-package-lock.json` and submit them to `azure-sdk-for-python`.
-
-2. **Automatic regeneration**: Once the PR merges to `main`, the [TypeSpec Python Regenerate Tests](https://github.com/Azure/azure-sdk-for-python/actions/workflows/typespec-python-regenerate.yml) workflow triggers automatically (it watches for changes to `eng/emitter-package.json`). It regenerates all test code and creates a follow-up issue with the updated generated files.
-
-3. **Generated code location**: The regenerated tests are checked in at [`eng/tools/azure-sdk-tools/emitter/generated`](https://github.com/Azure/azure-sdk-for-python/tree/typespec-python-generated-tests/eng/tools/azure-sdk-tools/emitter/generated) in `azure-sdk-for-python`, split into:
-   - `azure/` — Tests generated with the branded emitter (Azure HTTP specs)
-   - `unbranded/` — Tests generated with the unbranded emitter (HTTP specs)
-
 ## Release Process
 
 The release process for `@typespec/http-client-python` follows these steps:

packages/http-client-python/eng/scripts/ci/regenerate-common.ts

@@ -16,8 +16,7 @@
 import { compile, NodeHost } from "@typespec/compiler";
 import { execSync } from "child_process";
 import { existsSync, rmSync } from "fs";
-import { access, cp, mkdir, mkdtemp, readdir, writeFile } from "fs/promises";
-import { tmpdir } from "os";
+import { access, mkdir, readdir, writeFile } from "fs/promises";
 import { dirname, join, relative, resolve } from "path";
 import pc from "picocolors";
 
@@ -655,6 +654,28 @@ export async function runParallel(
   return results;
 }
 
+/**
+ * Removes every generated file under `tests/generated`, keeping only the
+ * hand-authored files that are tracked in git. `git clean -xdf` deletes ignored
+ * and untracked files but never tracked ones, so `.gitignore` stays the single
+ * source of truth for which fixtures survive regeneration (each exception there
+ * documents why it cannot be regenerated).
+ */
+export async function cleanGeneratedCode(generatedFolder: string): Promise<void> {
+  const repoDir = resolve(generatedFolder, "..");
+  const testsGeneratedDir = resolve(repoDir, "tests/generated");
+  if (!existsSync(testsGeneratedDir)) return;
+
+  console.log(pc.cyan(`\n${"=".repeat(60)}`));
+  console.log(pc.cyan(`Cleaning generated test packages`));
+  console.log(pc.cyan(`${"=".repeat(60)}\n`));
+
+  execSync(`git clean -x -d -f -q -- "${testsGeneratedDir}"`, {
+    cwd: repoDir,
+    stdio: ["ignore", "inherit", "inherit"],
+  });
+}
+
 /**
  * Pre-create the marker files that the test harness expects to find before
  * regeneration so it can verify they're cleared/preserved correctly.
@@ -698,99 +719,3 @@ export async function preprocess(flavor: string, generatedFolder: string): Promi
     }),
   );
 }
-
-/**
- * Resets the `tests/generated/{azure,unbranded}` baseline by sparse-checking-out
- * `eng/tools/azure-sdk-tools/emitter/generated` from the Azure/azure-sdk-for-python repo, then
- * deleting a couple of fully-generated package folders so regeneration has to
- * recreate them from scratch (smoke test of full-emit path).
- *
- * `generatedFolder` is the per-repo `generator/` directory; baseline lands at
- * `<generatedFolder>/../tests/generated`.
- */
-export async function prepareBaselineOfGeneratedCode(generatedFolder: string): Promise<void> {
-  const repoUrl = "https://github.com/Azure/azure-sdk-for-python.git";
-  const branch = "typespec-python-generated-tests";
-  const sourceSubdir = "eng/tools/azure-sdk-tools/emitter/generated";
-  const testsGeneratedDir = resolve(generatedFolder, "../tests/generated");
-
-  console.log(pc.cyan(`\n${"=".repeat(60)}`));
-  console.log(pc.cyan(`Resetting baseline from ${repoUrl} (${branch}/${sourceSubdir})`));
-  console.log(pc.cyan(`${"=".repeat(60)}\n`));
-
-  // Wipe tests/generated
-  if (existsSync(testsGeneratedDir)) {
-    console.log(pc.dim(`Removing ${testsGeneratedDir}`));
-    rmSync(testsGeneratedDir, { recursive: true, force: true });
-  }
-
-  // Sparse-checkout the baseline folder into a temp directory
-  const tempDir = await mkdtemp(join(tmpdir(), "azsdk-baseline-"));
-  try {
-    console.log(pc.dim(`Cloning into ${tempDir}`));
-    const run = (cmd: string) =>
-      execSync(cmd, { cwd: tempDir, stdio: ["ignore", "ignore", "inherit"] });
-
-    run(`git init`);
-    run(`git config core.longpaths true`);
-    run(`git remote add origin ${repoUrl}`);
-    run(`git config core.sparseCheckout true`);
-    run(`git sparse-checkout init --cone`);
-    run(`git sparse-checkout set ${sourceSubdir}`);
-    run(`git fetch --depth 1 origin ${branch}`);
-    run(`git checkout FETCH_HEAD`);
-
-    // we don't copy whole generated folder, just the specific subfolders needed for tests
-    // to verify correct preservation/deletion of files and folders during regeneration,
-    // to avoid accidentally including any manually edited code that might be in the repo
-    // and cause confusion when it doesn't get updated during regeneration
-    const legacyCodePathNeededForTests = [
-      "azure/authentication-api-key",
-      "unbranded/authentication-api-key",
-      "azure/authentication-union",
-      "azure/generation-subdir",
-      "azure/generation-subdir2",
-      "unbranded/generation-subdir",
-      "unbranded/generation-subdir2",
-      "azure/azure-client-generator-core-alternate-type",
-    ];
-
-    const sourceRoot = join(tempDir, ...sourceSubdir.split("/"));
-    for (const subPath of legacyCodePathNeededForTests) {
-      const segments = subPath.split("/");
-      const src = join(sourceRoot, ...segments);
-      const dest = join(testsGeneratedDir, ...segments);
-      if (!existsSync(src)) {
-        console.warn(pc.yellow(`Baseline folder not found: ${src}`));
-        continue;
-      }
-      console.log(pc.dim(`Copying ${subPath} -> ${dest}`));
-      await mkdir(dirname(dest), { recursive: true });
-      await cp(src, dest, { recursive: true });
-    }
-
-    console.log(pc.green(`Baseline reset complete.\n`));
-  } finally {
-    rmSync(tempDir, { recursive: true, force: true });
-  }
-
-  // Smoke test the full-emit path: delete a couple of fully-generated package
-  // folders and every README.md so regeneration has to recreate them.
-  const deleteIfExists = (path: string) => {
-    if (!existsSync(path)) return;
-    console.log(pc.dim(`Deleting ${path}`));
-    rmSync(path, { recursive: true, force: true });
-  };
-
-  deleteIfExists(join(testsGeneratedDir, "azure", "authentication-http-custom"));
-  deleteIfExists(join(testsGeneratedDir, "unbranded", "encode-array"));
-
-  if (existsSync(testsGeneratedDir)) {
-    const entries = await readdir(testsGeneratedDir, { recursive: true, withFileTypes: true });
-    for (const entry of entries) {
-      if (entry.isFile() && entry.name === "README.md") {
-        deleteIfExists(join(entry.parentPath, entry.name));
-      }
-    }
-  }
-}

packages/http-client-python/eng/scripts/ci/regenerate.ts

@@ -22,8 +22,8 @@ import { parseArgs } from "util";
 
 import {
   buildTaskGroups,
+  cleanGeneratedCode,
   getSubdirectories,
-  prepareBaselineOfGeneratedCode,
   preprocess,
   RegenerateContext,
   RegenerateFlags,
@@ -256,7 +256,7 @@ async function main() {
   const startTime = performance.now();
   let success: boolean;
 
-  await prepareBaselineOfGeneratedCode(GENERATED_FOLDER);
+  await cleanGeneratedCode(GENERATED_FOLDER);
 
   if (flavor) {
     success = await regenerateFlavor(flavor, name, debug, jobs);

packages/http-client-python/generator/pygen/codegen/serializers/types_serializer.py

@@ -189,9 +189,7 @@ def typeddict_models(self) -> list[ModelType]:
         property — are kept, ensuring no forward reference is left undefined.
         """
         needed = self._types_file_model_names
-        candidates = [
-            m for m in self._models if m.base != "json" and not m.discriminated_subtypes and m.name in needed
-        ]
+        candidates = [m for m in self._models if m.base != "json" and not m.discriminated_subtypes and m.name in needed]
         seen_names: dict[str, "ModelType"] = {}
         result: list["ModelType"] = []
         for m in candidates:

packages/http-client-python/package.json

@@ -50,7 +50,8 @@
     "typecheck": "tsx ./eng/scripts/ci/typecheck.ts",
     "typecheck:generated": "tsx ./eng/scripts/ci/typecheck.ts --generated",
     "regenerate": "tsx ./eng/scripts/ci/regenerate.ts",
-    "ci": "npm run test:emitter && npm run ci:generated",
+    "check-unstaged": "git -c core.safecrlf=false diff --ignore-space-at-eol --exit-code HEAD -- .",
+    "ci": "npm run check-unstaged && npm run test:emitter && npm run ci:generated",
     "ci:generated": "tsx ./eng/scripts/ci/run-tests.ts --generator --env=ci",
     "change:version": "pnpm chronus version --ignore-policies --only @typespec/http-client-python",
     "change:add": "pnpm chronus add",
@@ -129,6 +130,7 @@
     "rimraf": "~6.1.2",
     "typescript": "~5.9.2",
     "typescript-eslint": "^8.49.0",
-    "vitest": "^4.0.15"
+    "vitest": "^4.0.15",
+    "prettier": "^3.9.5"
   }
 }

packages/http-client-python/tests/generated/azure/authentication-api-key/authentication/apikey/_operations/_patch.py

@@ -0,0 +1,37 @@
+# coding=utf-8
+
+"""Customize generated code here.
+
+Follow our quickstart for examples: https://aka.ms/azsdk/python/dpcodegen/python/customize
+"""
+
+from typing import List
+
+from ._operations import _ApiKeyClientOperationsMixin as Generated
+
+
+class _ApiKeyClientOperationsMixin(Generated):
+
+    def patch_added_operation(self) -> bool:
+        """This is a placeholder for the operation that was added in the patch.
+
+        This method is a no-op and does not perform any action.
+
+        :return: Always returns True.
+        :rtype: bool
+        """
+        return True
+
+
+__all__: List[str] = [
+    "_ApiKeyClientOperationsMixin"
+]  # Add all objects you want publicly available to users at this package level
+
+
+def patch_sdk():
+    """Do not remove from this file.
+
+    `patch_sdk` is a last resort escape hatch that allows you to do customizations
+    you can't accomplish using the techniques described in
+    https://aka.ms/azsdk/python/dpcodegen/python/customize
+    """

packages/http-client-python/tests/generated/azure/authentication-api-key/authentication/apikey/aio/_operations/_patch.py

@@ -0,0 +1,37 @@
+# coding=utf-8
+
+"""Customize generated code here.
+
+Follow our quickstart for examples: https://aka.ms/azsdk/python/dpcodegen/python/customize
+"""
+
+from typing import List
+
+from ._operations import _ApiKeyClientOperationsMixin as Generated
+
+
+class _ApiKeyClientOperationsMixin(Generated):
+
+    def patch_added_operation(self) -> bool:
+        """This is a placeholder for the operation that was added in the patch.
+
+        This method is a no-op and does not perform any action.
+
+        :return: Always returns True.
+        :rtype: bool
+        """
+        return True
+
+
+__all__: List[str] = [
+    "_ApiKeyClientOperationsMixin"
+]  # Add all objects you want publicly available to users at this package level
+
+
+def patch_sdk():
+    """Do not remove from this file.
+
+    `patch_sdk` is a last resort escape hatch that allows you to do customizations
+    you can't accomplish using the techniques described in
+    https://aka.ms/azsdk/python/dpcodegen/python/customize
+    """