feat(ci): automated proto drift detection and SDK sync notifications

[Ygnas]

User Story

As a contributor changing proto definitions, I want to know immediately if my changes affect any SDK, so that I can regenerate the stubs myself or trust that SDK maintainers will be notified automatically. Contributors shouldn't need to understand every SDK's internals to make a proto change.

As an SDK maintainer, I want to be notified when proto changes introduce fields or RPCs that my SDK's converter layer doesn't yet expose, enabling planned and prioritized SDK update work without manually tracking every proto change.

Problem Statement

OpenShell maintains four SDKs (Go, TypeScript, Python, Rust) across different teams. Proto file changes have varying impacts per SDK:

Go is unique in committing generated proto stubs to the repository. Go's source-level dependency model (go get clones and compiles sources directly) necessitates this. When protos change, committed stubs must be regenerated and re-committed. The existing go:proto:check CI task verifies stub consistency but is a simple pass/fail gate without notifications.

TypeScript, Python, and Rust all generate stubs at build time and gitignore them. TypeScript uses buf generate (output to src/gen/, gitignored). Python runs grpc_tools.protoc via a python:proto task before builds. Rust uses tonic-prost-build in a build.rs script at compile time. Stub drift is structurally impossible for these three SDKs.

All four SDKs share a deeper problem: each has a hand-written domain type layer (converters, typed wrappers, From impls) mapping proto types into idiomatic language constructs. A new proto field compiles and tests pass, but the field is silently absent from the SDK's public API. SDK consumers discover missing fields only through trial and error. This "converter coverage gap" grows silently as proto definitions evolve.

SDK	Stubs Committed?	Generation Tool	Domain Type Layer	Stub Drift Risk
Go	Yes (source-level deps require it)	buf generate	types/ + converter/	Yes
TypeScript	No (src/gen/ gitignored)	buf generate	Curated API + raw escape hatch	No
Python	No (_proto/ gitignored)	grpc_tools.protoc	sandbox.py dataclass wrappers	No
Rust	No (compile-time build.rs)	tonic-prost-build	types.rs with From impls	No

Impact / Why This Matters

Current behavior: Proto changes land without any notification to SDK maintainers. For Go, contributors must know to regenerate and commit stubs. For all four SDKs, new proto fields are silently ignored by the domain type layer.

Current workaround: SDK maintainers manually monitor proto changes by watching the commit log. This is error-prone and doesn't scale.

Why the workaround is insufficient: As the contributor base grows, proto changes come from many people who may not be aware of SDK implications. SDK consumers depend on proto fields being exposed through the SDK. Silent gaps create downstream surprises.

Proposed Design

Stub drift detection + issue lifecycle

Scope: Go SDK (the only SDK committing generated stubs). Infrastructure is designed to be extensible if other SDKs' build models change.

• Per-SDK mise tasks (go:proto:drift) that regenerate stubs in a temp directory and compare against committed files, outputting structured JSON
• A PR-triggered GitHub workflow annotating PRs with drift warnings (non-blocking, informational). Contributors see messages like: "Your proto change affects SDK stubs. The Go SDK stubs are 3 files behind."
• A daily cron workflow that:
  • Runs drift detection for each SDK
  • If drift is found, runs a build verification pipeline (regenerate, build, test)
  • Creates or updates a GitHub issue per SDK (deduplicated by sdk:{sdk}:sync label, matching the repo's existing area:sdk:{language} convention)
  • Auto-closes issues when drift resolves
• A Python script for issue body generation and lifecycle management via gh

This makes contributions transparent without blocking anyone. Contributors see warnings; SDK maintainers get actionable issues with fix commands.

Extensibility

The system should use a config-driven approach where adding a new SDK requires only:

1. A drift detection mise task for that language
2. A config entry with display name, source directories, and build/test tasks

This pattern scales to all current and future SDKs.

Future extensions

Converter coverage analysis — compare proto message fields against what each SDK's converter/domain-type layer actually maps, reporting fields present in proto but absent from SDK type mappings. This would detect the deeper "business logic gap" that stub comparison cannot catch. Scope would cover all four SDKs.

Acceptance Criteria

• PR-triggered workflow annotates proto-touching PRs with SDK drift warnings (Go; extensible)
• Daily cron creates issues when drift is detected, with fix commands and affected file lists
• Issues auto-close when drift resolves
• Currently covers Go SDK stub drift (only SDK that commits stubs); matrix design supports future SDKs
• Adding a new SDK requires only a mise task + config entry
• Unit tests cover issue body generation and lifecycle management

Alternatives Considered

Fail CI on proto drift: Rejected. This would block proto contributors until every affected SDK is updated. Contributors should be able to land proto changes without understanding SDK internals. SDK maintenance is the SDK maintainer's responsibility, triggered by notifications.

Wiki dashboard: Considered and deferred. Wiki pages have low discoverability. PR annotations and auto-filed issues are more visible to those who need to act.

Integrate into existing branch-checks.yml: Worth discussing. A "Proto Drift" row alongside existing checks increases visibility. A separate workflow is easier to iterate on initially and can be promoted later.

Extend stub drift to TypeScript/Python/Rust: Not needed. All three generate stubs at build time and gitignore them, making stub drift structurally impossible. Go is the only SDK where stubs are committed. All four SDKs would benefit from the converter coverage analysis extension.

cc @maxdubrinsky @drew @mrunalp @Gkrumbach07 @rhuss

[elezar]

📋 triage-agent

Triage Assessment

Classification: needs-information

Summary

The notification and compatibility goal is clear, and the existing blocking
Go proto check does not provide the proposed SDK-owner notification and
finding lifecycle. Before the proposal can be technically validated, the
distribution and ownership model needs to cover SDKs consistently regardless
of where their source repository is hosted.

Why out-of-tree SDKs matter

Supporting out-of-tree SDKs allows language communities and partner teams to
own their implementation, release cadence, and support policy without
coupling every SDK change to the OpenShell repository. It also lets new or
experimental SDKs emerge without increasing the core repository's
maintenance burden. A uniform pull-based compatibility process keeps these
SDKs aligned with the OpenShell contract without requiring OpenShell to know
about or coordinate every consumer.

Information requested

Please clarify the proposal around a consumer-driven pull model that treats
in-tree and out-of-tree SDKs identically:

1. What versioned contract does an SDK pull: every main revision,
   development releases, stable releases, or another channel?
2. How does an SDK record the contract revision it currently supports?
3. How does the SDK detect stub and domain/converter compatibility gaps?
4. How does the SDK create, update, and resolve findings in its own repository
   without registration or cross-repository credentials in OpenShell?
5. Should the existing blocking go:proto:check remain as a producer-side
   integrity check, independently of the consumer notification process?

The same compatibility contract and finding lifecycle should apply whether
the SDK lives inside or outside the OpenShell repository.

[Ygnas]

Scope

This proposal only covers in-tree SDKs. Right now that means Go and TypeScript. Python and Rust regenerate stubs at build time, so they don't need stub drift detection. Adding a new SDK takes a mise task and a config entry.

Triage questions

1. What versioned contract does an SDK pull?

The contract is main HEAD. All the in-tree SDKs live in the same repo as the proto definitions, so the daily cron and PR checks both compare against whatever proto/ looks like on the branch or main.

2. How does an SDK record which contract revision it supports?

For Go, the committed stubs (sdk/go/proto/*.pb.go) are the record. They're a snapshot of the last-synced proto revision. TypeScript generates stubs at build time and gitignores them, so the record is implicit: whatever main HEAD compiles against.

3. How does the SDK detect stub and converter gaps?

Two layers, both on this branch.

Go is the only SDK that commits stubs, so it's the only one with actual stub drift. The go:proto:drift mise task regenerates to a temp directory and diffs against committed files. If something changed, that's drift.

For both Go and TypeScript, there's also build verification: regenerate bindings, then run build and test. If the build fails, the converter layer has a gap. The go:proto:build-check and sdk:ts:proto:build-check tasks handle this in the cron workflow. TypeScript is a bit different here because stubs are gitignored, so its drift detection is really just build verification. The sdk:ts:proto:drift task regenerates protos and typechecks the handwritten SDK code against them.

Converter coverage analysis (comparing proto message fields against what the SDK actually maps) is a future extension. That would catch silent gaps where the build passes but a new field isn't in the SDK's public API.

4. How does the SDK create, update, and resolve findings?

The cron workflow manages issues in the OpenShell repo using gh CLI with GITHUB_TOKEN. Issues are deduplicated by sdk:{name}:sync labels and auto-closed when drift resolves. Everything stays in one repo, no cross-repo credentials needed.

5. Should go:proto:check stay?

Yes. go:proto:check is a CI gate that prevents merging stale committed stubs. It answers "are the .pb.go files current?" The drift notification workflow answers a different question: "has proto evolved past what the SDK exposes?" They serve different purposes and both checks are needed independently.

[Ygnas]

@elezar I answered all five triage questions in my comment above. Is something else needed to move this from needs-info? If anything is not clear I am happy to explain more.