Add extensible chat client routing

[joshuajyue]

Resolves #7647

Summary

Adds experimental chat-routing primitives for selecting configured IChatClient instances without imposing a particular routing policy.

• RoutingChatClient supports one-shot client selection.
• FailoverChatClient retries uncanceled failures that occur before output is exposed.
• OrderedFailoverChatClient provides ordered fallback.
• SemanticRoutingChatClient selects clients using cached profile embeddings.
• RoutingContext exposes mutable request messages and options.
• FailoverChatClientAttempt reports invocation outcome, duration, time to first update, completion, and output commitment.

API shape

Routing policies have two seams:

protected override ValueTask<IChatClient> SelectClientAsync(
    RoutingContext context,
    CancellationToken cancellationToken);

protected override ValueTask OnRoutingUpdateAsync(
    RoutingContext context,
    FailoverChatClientAttempt? attempt,
    bool isTerminal,
    CancellationToken cancellationToken);

SelectClientAsync runs before every invocation. OnRoutingUpdateAsync reports each attempt so policy state can influence the next selection.

isTerminal means the base will not select another client after the callback completes. A null attempt represents selection terminating before a client was invoked.

Failover behavior

• Cancellation never triggers reselection.
• Streaming fallback occurs only before the first update is exposed.
• Mid-stream failures and caller abandonment are terminal.
• Enumerator disposal failures are included in the attempt.
• Attempt limits are request-local.
• Null selector results are treated as terminal selection failures.
• Ordered failover retains request state only between a nonterminal update and the immediately following selection; it holds no state while a provider invocation or stream is active.
• Streaming callers must dispose active enumerators for terminal observation and inner-resource cleanup.

Semantic routing

SemanticRoutingChatClient:

• Lazily embeds and caches profile utterances.
• Embeds the final user message for selection.
• Supports global top-K profile matching.
• Aggregates matches per client using mean or sum.
• Preserves the existing single-best-match behavior with topK: 1.
• Uses a required default client when no profile clears the threshold.
• Supports caller-controlled ownership through leaveOpen.

Tests

Coverage includes selection failures, retries, cancellation, attempt limits, streaming commitment, early disposal, enumerator failures, concurrent requests, state cleanup, semantic thresholds, top-K aggregation, caching, and ownership.

• 753 tests passed on net10.0
• net472 test project builds successfully

Microsoft Reviewers: Open in CodeFlow

[joshuajyue]

Addressed in commit 0abbfd1, along with some cancellation races

[jozkee]

Looking good.

[joshuajyue]

Thank you David for the feedback, addressed in 8b79ae4

[joshuajyue]

Sorry y'all, I had a change of thought -- just one last thing I felt was a bit unresolved:

I actually think the small per-attempt selection result mentioned earlier is more ergonomic than requiring every option variation to be represented by a configured client:

RoutingSelection(IChatClient client, ChatOptions? options)

The client reference still remains the routing identity. The returned options are simply complete, ephemeral options for that attempt, while RoutingContext.ChatOptions remains the clean request baseline. No merging of options necessary -- the user shapes on top of a clone of caller options. This makes dynamic model/reasoning/temperature selection accessible through the selector without leaking options across failover attempts or treating context mutation as a hidden output.

ConfigureOptionsChatClient remains a way to modify options — it’s just not the only way anymore.

@jozkee @PranavSenthilnathan let me know what you guys think when you have time.

I kicked off a Copilot session to write it out and push it here — if y’all prefer the old way then I’ll just revert it.