fix: skip prompt_cache_retention parameter for Azure endpoints #10821
+79
−0
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related GitHub Issue
Closes: #10782
Description
This PR attempts to address Issue #10782 by fixing Azure Foundry compatibility with the OpenAI Native handler.
Azure Foundry/AI does not support the
prompt_cache_retentionparameter that OpenAI's API accepts. When users configure the OpenAI Native handler with an Azure Foundry endpoint (e.g.,*.openai.azure.comor*.services.ai.azure.com), API calls would fail because Azure rejects this unsupported parameter.Implementation Details:
isAzureEndpoint()helper method to detect Azure base URLsgetPromptCacheRetention()to returnundefinedfor Azure endpoints, effectively skipping the parameter in API requests*.azure.comand*.azure-api.netTest Procedure
.azure.comendpoints skipprompt_cache_retention*.services.ai.azure.comendpoints skipprompt_cache_retention*.azure-api.netendpoints skipprompt_cache_retentionprompt_cache_retentionfor eligible modelsprompt_cache_retentionTo run tests:
cd src && npx vitest run api/providers/__tests__/openai-native-usage.spec.tsPre-Submission Checklist
Documentation Updates
Additional Notes
Feedback and guidance are welcome! This is an attempt to help users who are locked to Azure Foundry endpoints in their work tenants.
Important
Fixes Azure endpoint compatibility by skipping unsupported
prompt_cache_retentionparameter inOpenAiNativeHandler.getPromptCacheRetention()inopenai-native.tsnow returnsundefinedfor Azure endpoints, skippingprompt_cache_retentionin requests.isAzureEndpoint()added to detect Azure URLs (*.azure.com,*.azure-api.net).openai-native-usage.spec.tsto verify Azure endpoint detection and parameter omission..azure.com,*.services.ai.azure.com, and*.azure-api.netendpoints, as well as non-Azure and default OpenAI endpoints.This description was created by
for 287e843. You can customize this summary. It will automatically update as commits are pushed.