feat: make MaxRetries configurable in OpenAI provider#271
Conversation
Add MaxRetries *int to config.OpenAI. When set, option.WithMaxRetries is passed to the SDK client in both responses and chat completions interceptors. Nil preserves the SDK default (2 retries); 0 disables retries entirely. Update TestClientAndConnectionError and TestUpstreamError to set MaxRetries=0, eliminating retry delays and speeding up these tests.
mtojek
changed the title
|
It would be nice to have some setting for it (env var / flag) but it can be a follow up. |
When MaxRetries is not set in config, fall back to the OPENAI_MAX_RETRIES environment variable. Invalid values are silently ignored, preserving the SDK default of 2 retries.
|
Let me know your thoughts @pawbana 👍 |
|
LGTM although I missed before that messages interceptor is missing. |
Add MaxRetries *int to config.Anthropic and config.Copilot, readable via ANTHROPIC_MAX_RETRIES and COPILOT_MAX_RETRIES env vars respectively. Pass option.WithMaxRetries in messages interceptor when set. Copilot propagates MaxRetries to the config.OpenAI it builds for interceptors.
Thanks for suggestion! I extended support to Anthropic and Copilot 👍 |
| cfg.APIDumpDir = os.Getenv("BRIDGE_DUMP_DIR") | ||
| } | ||
| if cfg.MaxRetries == nil { | ||
| if v := os.Getenv("COPILOT_MAX_RETRIES"); v != "" { |
There was a problem hiding this comment.
This is a bit weird in Copilot case as we've recently added support for multiple providers and multiple Copilot providers (normal / business / enterprise) in coder/coder but looking at what configuration options are possible in aibridge I think it is ok.
2 participants
Fixes: #115
Add
MaxRetries *inttoconfig.OpenAI. When set,option.WithMaxRetriesis passed to the SDK client in both responses and chat completions interceptors. Nil preserves the SDK default (2 retries); 0 disables retries entirely.Update
TestClientAndConnectionErrorandTestUpstreamErrorto setMaxRetries=0, eliminating retry delays and speeding up these tests.