OpenAI + Anthropic Taste Test

Use this skill from the
`Azure-Samples/openai-anthropic-taste-test`
workspace. Read the root README.md and azure.yaml before changing or deploying anything.

Goal

Get the customer from a clean machine to a verified blind A/B page with the least interaction
possible:

1. Reuse existing authentication and azd environment state.
2. Select full Azure hosting when RBAC permits it; otherwise select local hosting automatically.
3. Preview infrastructure before deployment.
4. Provision with no API keys.
5. Verify both model endpoints and the browser flow.
6. Report the resource group, application URL or local URL, model IDs, and any remaining manual
   prerequisite.

Hard rules

• Use Microsoft Entra authentication only. Never request, create, print, or store API keys.
• Run azd provision --preview before azd provision or azd up.
• Never guess CLAUDE_ORGANIZATION_NAME; it is customer attestation data. Ask only when it cannot
  be resolved from the active azd environment.
• Do not persist tenant IDs, subscription IDs, tokens, customer names, or deployment logs in tracked
  files. azd environment state belongs under the gitignored .azure/ folder.
• Reuse cached browser authentication. Run azd auth login --check-status first, and invoke browser
  login only when the cache is missing or for the wrong tenant. Do not use device-code login unless
  the customer explicitly requests it.
• Do not weaken the infrastructure to key-based authentication to work around RBAC.
• On failure, clean up partially provisioned resources unless the customer asks to retain them.

Choose a path

Condition	Path
Customer only wants to see the UI	Sample mode
Customer has Microsoft.Authorization/roleAssignments/write	Azure-hosted mode
Customer lacks role-assignment permission but has Foundry data access	Local-hosted mode
Customer explicitly requests one mode	Honor that mode

Sample mode — no Azure required

dotnet run --project src/TasteTest --launch-profile sample

Verify http://localhost:5050/health, open http://localhost:5050, submit a seed prompt, pick a
winner, and continue the winning conversation.

PLAN — resolve context once

Resolve these values in order from explicit user input, azd env get-values, and the active Azure
context. Ask only for values still missing:

• Environment name
• Subscription ID
• Tenant ID when the subscription is outside the current tenant
• Azure location; default to eastus2 if the customer has no preference
• Legal organization name for the Anthropic attestation
• Country code; default US
• Industry; default technology

Use the model defaults in infra/main.parameters.json unless the customer asks to change them:
gpt-6.1-sol version 2026-09-29 and claude-sonnet-5-5 version 2 (Hosted on Azure).
Both use GlobalStandard; choose eastus2 or swedencentral for generally available defaults.
Both default to capacity 10 (10,000 TPM). Sonnet also passed a smoke test at capacity 1,
but that allocation permits only one request per minute and is not the interactive-demo default.
Use .NET SDK 10.0.402+ and azd 1.37.1+. The application requests low reasoning effort through the
same MEAI options for both vendors; Anthropic 13 uses adaptive thinking for that setting.

Check authentication:

azd auth login --check-status

When login is required, use the enterprise browser flow:

azd auth login --tenant-id <tenant-id>

Create or select an environment:

azd env new <environment> `
  --subscription <subscription-id> `
  --location eastus2 `
  --no-prompt

azd env set AZURE_TENANT_ID <tenant-id>
azd env set CLAUDE_ORGANIZATION_NAME "<legal-entity>"
azd env set CLAUDE_COUNTRY_CODE US
azd env set CLAUDE_INDUSTRY technology

DEPLOY — full Azure hosting

Use this path when the principal can create role assignments. It creates the Foundry account and
project, both model deployments, Container Apps, ACR, managed identity, and Log Analytics.

azd env set HOSTING_MODE containerapp
azd env set ASSIGN_INFERENCE_ROLE_TO_DEPLOYER true
azd provision --preview --no-prompt
azd up --no-prompt

After deployment:

1. Read SERVICE_WEB_URI from azd env get-values.
2. GET <SERVICE_WEB_URI>/health and require HTTP 200.
3. Open SERVICE_WEB_URI.
4. Complete the browser verification under VERIFY.

DEPLOY — no-RBAC local hosting

Use this path when the principal cannot perform
Microsoft.Authorization/roleAssignments/write. It creates no role assignments and no application
hosting. The signed-in developer must already have data-plane access through Azure AI Developer,
Foundry Owner, or Cognitive Services User.

azd env set HOSTING_MODE local
azd env set ASSIGN_INFERENCE_ROLE_TO_DEPLOYER false
azd provision --preview --no-prompt
azd provision --no-prompt

The application uses DefaultAzureCredential, whose developer chain includes Azure Developer CLI,
Azure CLI, and supported IDE credentials. Reuse a cached credential for the target tenant rather than
prompting for another login. Only if no usable credential exists, use browser-based Azure CLI login:

az login --tenant <tenant-id>
powershell.exe -NoProfile -ExecutionPolicy RemoteSigned -File .\scripts\run-local.ps1

On macOS or Linux:

az login --tenant <tenant-id>
./scripts/run-local.sh

Verify http://localhost:5050/health or the URL printed by dotnet run, then complete the browser
verification below.

VERIFY — prove the experience

1. Before sending a prompt, inspect the DOM for provider/model strings. It must contain none of:
   OpenAI, Anthropic, gpt-, claude-, Responses API, or Messages API.
2. Select one of the seed prompts.
3. Submit it and confirm both lane A and lane B stream concurrently.
4. Confirm the vote buttons enable only after both lanes finish successfully. While waiting or after failure, they stay disabled with an explanation.
5. Pick either lane.
6. Confirm both identities appear, the selected lane is marked WINNER, and only that response
   pane remains.
7. Submit a follow-up and confirm only the winning model responds.
8. Select New taste test and confirm both identities disappear from the DOM and lane ordering is
   randomized again.
9. Cancel a streamed comparison and retry. The partial answers remain labeled as incomplete, but
   neither lane includes that attempt in the next request's history.
10. Confirm the submit button disables promptly, Cancel remains responsive before the first text,
    and a double click does not queue a second comparison or leave a stale error.
11. For live verification, require no preview banner, completed answers from both native SDKs,
    and an enabled vote. Canned preview responses are not evidence of working inference.

For a direct Claude smoke test, use the provisioned endpoint and deployment from
azd env get-values. Acquire an Entra token for https://ai.azure.com/.default, POST a minimal
Messages API request to <AZURE_FOUNDRY_ENDPOINT>/anthropic/v1/messages, and require a successful
text response. Never print the token.

DIAGNOSE

Error	Meaning	Action
roleAssignments/write authorization failure	Contributor can provision resources but cannot grant managed-identity access	Switch to local hosting, or request Role Based Access Control Administrator
no valid payment method / plan cannot be purchased with a free subscription	The Marketplace plan is not eligible on that subscription	Use an eligible subscription. Some subscriptions are provisioned with a different pre-accepted plan, so confirm the available plans with the subscription administrator.
AnthropicOrganizationCreationException	Attestation fields are missing or invalid	Set the legal entity, two-letter country, and lowercase industry
InsufficientQuota	Model capacity is unavailable in the selected region	Lower capacity, select another supported region, or request quota
Local app returns 401/403	DefaultAzureCredential resolved a different tenant/account or the user lacks data access	Sign Azure CLI into the target tenant and verify Azure AI Developer, Foundry Owner, or Cognitive Services User
Azure-hosted app returns 401/403	Managed-identity role assignment has not propagated	Verify Cognitive Services User on the Foundry account and retry after propagation
One lane fails while the other succeeds	Provider-specific quota, deployment, auth, or endpoint problem	Check server logs; the browser intentionally hides provider details before reveal
Empty answer or token-limit message	Reasoning can consume the output budget before visible text	Use the 4096-token default, restart the local app after settings change, then Retry comparison. Check terminal finish reason and usage in server logs.
PowerShell reports scripts are disabled	The launcher was blocked before the app started	Use the process-scoped Windows launcher above. Do not override organization-enforced policies.

MODIFY

Change models through azd environment values, then re-run preview and provision:

azd env set CLAUDE_MODEL_NAME claude-sonnet-5-5
azd env set CLAUDE_MODEL_VERSION 2
azd env set CLAUDE_MODEL_CAPACITY 10
azd env set AOAI_MODEL_NAME gpt-6.1-sol
azd env set AOAI_MODEL_VERSION 2026-09-29
azd provision --preview --no-prompt
azd provision --no-prompt

Always verify current model versions, region support, and quota before changing defaults.

TEARDOWN

Confirm the selected environment and resource group before deleting:

azd env get-values
azd down --purge --force --no-prompt

After a failed deployment, verify that the resource group and any soft-deleted Cognitive Services
account are gone. Do not delete resources outside the selected azd environment.