From 49982a1db888f11dad4a6b903864f914853882f5 Mon Sep 17 00:00:00 2001 From: 777genius Date: Sun, 26 Apr 2026 10:21:18 +0300 Subject: [PATCH] feat: improve opencode model verification ux --- README.md | 2 +- .../opencode-model-gauntlet-results.md | 1059 +++++++++ .../ui/RuntimeProviderManagementPanelView.tsx | 13 +- .../team/dialogs/TeamModelSelector.tsx | 74 +- .../utils/openCodeModelRecommendations.ts | 1154 ++++++++- .../OpenCodeSemanticMessaging.live.test.ts | 2 + ...OpenCodeSemanticModelGauntlet.live.test.ts | 2113 +++++++++++++++++ .../OpenCodeSemanticModelMatrix.live.test.ts | 1 + .../services/team/openCodeLiveTestHarness.ts | 12 + .../TeamModelSelectorDisabledState.test.ts | 60 +- ...RuntimeProviderManagementPanelView.test.ts | 76 +- .../openCodeModelRecommendations.test.ts | 850 ++++++- 12 files changed, 5182 insertions(+), 234 deletions(-) create mode 100644 docs/research/opencode-model-gauntlet-results.md create mode 100644 test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts diff --git a/README.md b/README.md index 669e9c28..daed42be 100644 --- a/README.md +++ b/README.md @@ -188,7 +188,7 @@ For feature architecture and implementation guidance: | **Per-task code review** | ✅ Accept / reject / comment | ⚠️ PR-level only | ⚠️ File-level only | ✅ BugBot on PRs | ❌ | | **Flexible autonomy** | ✅ Granular settings, per-action approval, notifications | ❌ | ⚠️ Plan approval only | ✅ | ✅ | | **Git worktree isolation** | ✅ Optional | ⚠️ Mandatory | ⚠️ Mandatory | ✅ | ✅ | -| **Multi-agent backend** | ✅ Claude, Codex, more coming soon | ✅ 6+ agents | ✅ 11 providers | ✅ Multi-model | — | +| **Multi-agent backend** | ✅ Codex, Claude, and 75+ providers | ✅ 6+ agents | ✅ 11 providers | ✅ Multi-model | — | | **Price** | **Free** | Free / $30 user/mo | Free | $0–$200/mo | Provider subscription | --- diff --git a/docs/research/opencode-model-gauntlet-results.md b/docs/research/opencode-model-gauntlet-results.md new file mode 100644 index 00000000..c80fc79c --- /dev/null +++ b/docs/research/opencode-model-gauntlet-results.md @@ -0,0 +1,1059 @@ +# OpenCode Model Gauntlet Results + +Generated: 2026-04-26T01:18:27Z + +## Methodology + +The gauntlet uses production-path OpenCode Agent Teams launch and delivery code. It checks launch/bootstrap, direct user-to-member delivery, teammate-to-teammate relay, chained relay to a third teammate, near-concurrent delivery, taskRefs preservation, transcript hygiene, duplicate visible reply prevention, and latency. + +Verdicts: + +- `Recommended` requires at least 3 successful runs, average score >= 90, consistency score >= 85, no hard failures, no provider-infra failures, and no harness failures. +- `Strong candidate` means the model performed well in counted behavioral runs and has at least one full successful run, but does not yet satisfy the stricter repeated-run recommendation threshold. +- `Tested only` means the model has real gauntlet evidence but showed behavioral/runtime instability. +- `Infra blocked` means provider/API/catalog/credit errors prevented useful model judgment. + +Provider-infra runs are reported separately and are not counted as model behavior. They still block a `Recommended` verdict until rerun succeeds. + +Credit/key/max-token allowance failures are treated as inconclusive provider-infra samples, not as model-quality verdicts. They are intentionally not surfaced as `Not recommended` in the product UI and should be rerun after balance or key limits are restored. + +Current generated reports also include scoring weights, readiness score, consistency score, score spread, recommendation blockers, weighted stage impact, weakest-stage pass rates, weakest `taskRefs` pass rates, protocol-violation totals, stage durations, and primary failure categories. This makes the scorecard more actionable than a single average: a model can now be separated into provider-infra blocked, runtime-transport unstable, or model-behavior weak. + +The readiness score is not a replacement for the `Recommended` gate. It is a practical ranking signal that combines behavioral average, counted pass rate, `taskRefs` preservation, protocol cleanliness, consistency, and provider-infra cleanliness. The `Recommended` verdict remains stricter and still requires repeated successful runs with no hard/provider/harness failures. + +The consistency score protects against misleading averages. A model with one excellent run and one weak run can keep a high average, but it will now show score spread and can be blocked from `Recommended` when consistency is below the configured threshold. + +Confidence: + +- `high`: at least 3 counted behavioral runs and no provider-infra contamination. +- `medium`: at least 2 counted behavioral runs. +- `low`: 1 counted behavioral run. +- `blocked`: no useful behavioral sample because infra or harness failures dominated. + +Current UI policy: all previously passing OpenCode routes are shown as `Tested`, not `Recommended`. A model should only be promoted to `Recommended` after a fresh 3-run gauntlet passes the current average, consistency, hard-failure, provider-infra, and harness-failure gates. + +## OpenRouter Rank 11-20 Fresh Batch + +Source: OpenRouter activity screenshot ranks 11-20, excluding `openrouter/google/gemini-2.5-flash` and `openrouter/z-ai/glm-5.1` because both already had clean 100/100 gauntlet evidence. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-rank11-20-gauntlet-1777183582" OPENCODE_E2E_MODELS="openrouter/google/gemini-2.5-flash-lite,openrouter/nvidia/nemotron-3-super-120b-a12b:free,openrouter/xiaomi/mimo-v2-pro,openrouter/openai/gpt-5.4,openrouter/openai/gpt-oss-120b,openrouter/google/gemini-3.1-pro-preview,openrouter/moonshotai/kimi-k2.5,openrouter/qwen/qwen3.6-plus" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +Host note: the run produced a usable report, then the harness failed while the machine was out of disk space (`ENOSPC`). Treat `openrouter/moonshotai/kimi-k2.5` as inconclusive because its failure was directly caused by `ENOSPC`. The other completed rows are useful model signals. + +| Model | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| --- | --- | ---: | ---: | ---: | --- | ---: | --- | +| `openrouter/google/gemini-2.5-flash-lite` | Tested only | 54 | 35 | 0/1 | model-behavior | 246674ms | Passed launch/direct reply, then timed out or failed peer relay and concurrent delivery. | +| `openrouter/nvidia/nemotron-3-super-120b-a12b:free` | Tested only | 54 | 35 | 0/1 | model-behavior | 202862ms | Passed launch/direct reply, then failed peer relay through OpenCode delivery. | +| `openrouter/xiaomi/mimo-v2-pro` | Strong candidate | 100 | 100 | 1/1 | none | 163958ms | Passed launch, direct reply, peer relays, concurrent replies, taskRefs, transcript hygiene, and duplicate guard. | +| `openrouter/openai/gpt-5.4` | Tested only | 55 | 70 | 0/1 | runtime-transport | 290561ms | Passed direct and peer relays, then failed concurrent Tom reply, taskRefs, and duplicate-token correctness. | +| `openrouter/openai/gpt-oss-120b` | Infra blocked | 0 | 65 | 0/0 | provider-infra | 464553ms | Passed early stages, then timed out during concurrent delivery and taskRefs checks. | +| `openrouter/google/gemini-3.1-pro-preview` | Tested only | 40 | 0 | 0/1 | model-behavior | 19977ms | Failed launch readiness before usable Agent Teams behavior. | +| `openrouter/moonshotai/kimi-k2.5` | Inconclusive | 40 | 0 | 0/1 | host ENOSPC | 1720ms | Failure happened while materializing task fixtures because the host disk was full. Rerun after disk cleanup. | +| `openrouter/qwen/qwen3.6-plus` | Tested only | 68 | 95 | 0/1 | model-behavior | 224986ms | Completed all functional stages but failed duplicate-token/protocol cleanliness, so it is not safe as `Tested`. | + +Additional context from the same screenshot: `openrouter/google/gemini-2.5-flash` and `openrouter/z-ai/glm-5.1` were already tracked as `Tested` from earlier 1/1 100/100 gauntlets. After this batch, `openrouter/xiaomi/mimo-v2-pro` is also tracked as `Tested`. No model from this batch should be marked `Recommended` yet because none has a fresh 3-run clean pass under the current gate. + +## MiniMax M2.7 Fresh Rerun + +Source: targeted rerun after a previous near-concurrent runtime transport failure. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-minimax-m27-rerun-1777183082" OPENCODE_E2E_MODELS="openrouter/minimax/minimax-m2.7" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| --- | ---: | ---: | --- | ---: | ---: | ---: | --- | ---: | --- | +| `openrouter/minimax/minimax-m2.7` | 0.30 | 1.20 | Strong candidate | 100 | 100 | 1/1 | none | 141289ms | Fresh rerun passed launch, direct reply, peer relays, concurrent replies, taskRefs, transcript hygiene, and duplicate guard. | + +Interpretation: the latest rerun clears the previous near-concurrent delivery failure for one sample. Keep it as `Tested`, not `Recommended`, because the combined history still includes one prior runtime transport failure and the current recommendation gate requires repeated clean runs. + +## Cheap Top Models Single-Run + +Source: top models from the OpenRouter activity screenshot, filtered by lower OpenRouter API pricing first. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-cheap-top-gauntlet-1777157371" OPENCODE_E2E_MODELS="openrouter/minimax/minimax-m2.5,openrouter/x-ai/grok-4.1-fast,openrouter/deepseek/deepseek-v3.2,openrouter/minimax/minimax-m2.7,openrouter/google/gemini-3-flash-preview" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ------------------------------------------ | ---------: | ----------: | ---------------- | --------: | --: | --------: | ----------------- | -------: | ---------------------------------------------------------------------------- | +| `openrouter/minimax/minimax-m2.5` | 0.15 | 1.15 | Strong candidate | 100 | 100 | 1/1 | none | 163891ms | Passed launch, direct, peer relay, multi-hop, concurrent, taskRefs, hygiene. | +| `openrouter/x-ai/grok-4.1-fast` | 0.20 | 0.50 | Strong candidate | 100 | 100 | 1/1 | none | 123333ms | Passed all gauntlet stages and was cheaper/faster than MiniMax M2.5 here. | +| `openrouter/deepseek/deepseek-v3.2` | 0.252 | 0.378 | Tested only | 55 | 70 | 0/1 | runtime-transport | 316760ms | Failed concurrent Tom reply, lost taskRefs for concurrentTom, protocol miss. | +| `openrouter/minimax/minimax-m2.7` | 0.30 | 1.20 | Tested only | 55 | 70 | 0/1 | runtime-transport | 305447ms | Same concurrent Tom/runtime-transport failure pattern as DeepSeek V3.2. | +| `openrouter/google/gemini-3-flash-preview` | 0.50 | 3.00 | Strong candidate | 100 | 100 | 1/1 | none | 85298ms | Passed all gauntlet stages and was the fastest in this cheap-top batch. | + +Interpretation: for the cheap top-model lane, the first models worth repeating with 3-run gauntlet are `openrouter/google/gemini-3-flash-preview`, `openrouter/x-ai/grok-4.1-fast`, and `openrouter/minimax/minimax-m2.5`. `openrouter/deepseek/deepseek-v3.2` and `openrouter/minimax/minimax-m2.7` should stay below recommendation until their concurrent delivery/runtime-transport failures are resolved or disproven by repeat runs. + +## Diverse Models Single-Run + +Source: broader OpenRouter model mix, biased toward moderate pricing and different providers. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-diverse-gauntlet-1777158534" OPENCODE_E2E_MODELS="openrouter/qwen/qwen3-coder-flash,openrouter/qwen/qwen3-coder,openrouter/google/gemini-3.1-flash-lite-preview,openrouter/mistralai/devstral-2512,openrouter/moonshotai/kimi-k2.6,openrouter/openai/gpt-5.4-mini,openrouter/xiaomi/mimo-v2-pro" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ------------------------------------------------- | ---------: | ----------: | ---------------- | --------: | --: | --------: | ----------------- | -------: | ---------------------------------------------------------------------------- | +| `openrouter/qwen/qwen3-coder-flash` | 0.195 | 0.975 | Tested only | 60 | 50 | 0/1 | model-behavior | 281611ms | Passed direct and first relay, then failed chained/concurrent reply flow. | +| `openrouter/qwen/qwen3-coder` | 0.22 | 1.00 | Tested only | 55 | 70 | 0/1 | runtime-transport | 271746ms | Failed concurrent Tom reply and taskRefs, with one protocol token issue. | +| `openrouter/google/gemini-3.1-flash-lite-preview` | 0.25 | 1.50 | Strong candidate | 100 | 100 | 1/1 | none | 88292ms | Passed launch, direct, peer relay, multi-hop, concurrent, taskRefs, hygiene. | +| `openrouter/mistralai/devstral-2512` | 0.40 | 2.00 | Strong candidate | 100 | 100 | 1/1 | none | 86424ms | Passed all stages and was the fastest in this diverse batch. | +| `openrouter/moonshotai/kimi-k2.6` | 0.7448 | 4.655 | Tested only | 60 | 50 | 0/1 | model-behavior | 282340ms | Same chained/concurrent weakness pattern as Qwen3 Coder Flash. | +| `openrouter/openai/gpt-5.4-mini` | 0.75 | 4.50 | Tested only | 55 | 70 | 0/1 | runtime-transport | 267937ms | Failed concurrent Tom reply and taskRefs despite passing earlier stages. | +| `openrouter/xiaomi/mimo-v2-pro` | 1.00 | 3.00 | Strong candidate | 100 | 100 | 1/1 | none | 115528ms | Passed all stages; slower than Devstral and Gemini Flash Lite in this run. | + +Interpretation: the next 3-run promotion candidates from this diverse batch are `openrouter/mistralai/devstral-2512`, `openrouter/google/gemini-3.1-flash-lite-preview`, and `openrouter/xiaomi/mimo-v2-pro`. Keep `openrouter/qwen/qwen3-coder-flash`, `openrouter/qwen/qwen3-coder`, `openrouter/moonshotai/kimi-k2.6`, and `openrouter/openai/gpt-5.4-mini` as `Tested only` until repeat runs prove they can handle chained relay plus concurrent delivery reliably. + +## Additional Catalog Models Single-Run + +Source: newer OpenRouter catalog entries across DeepSeek, Xiaomi, Z.ai, Qwen, MiniMax, and Mistral. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-diverse-gauntlet-1777160285" OPENCODE_E2E_MODELS="openrouter/deepseek/deepseek-v4-flash,openrouter/xiaomi/mimo-v2.5,openrouter/xiaomi/mimo-v2.5-pro,openrouter/z-ai/glm-5.1,openrouter/qwen/qwen3.6-plus,openrouter/minimax/minimax-m2-her,openrouter/mistralai/mistral-small-2603" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ----------------------------------------- | ---------: | ----------: | ---------------- | --------: | --: | --------: | ---------------- | -------: | ---------------------------------------------------------------------------- | +| `openrouter/deepseek/deepseek-v4-flash` | 0.14 | 0.28 | Infra blocked | 0 | 35 | 0/1 | provider-infra | 397985ms | Launched and replied directly, then hit OpenCode tool/runtime connectivity. | +| `openrouter/xiaomi/mimo-v2.5` | 0.40 | 2.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5365ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/xiaomi/mimo-v2.5-pro` | 1.00 | 3.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5240ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/z-ai/glm-5.1` | 1.05 | 3.50 | Strong candidate | 100 | 100 | 1/1 | none | 132735ms | Passed launch, direct, peer relay, multi-hop, concurrent, taskRefs, hygiene. | +| `openrouter/qwen/qwen3.6-plus` | 0.325 | 1.95 | Tested only | 61 | 85 | 0/1 | model-behavior | 128215ms | Passed stages but failed taskRefs/noDuplicateTokens, so not a candidate. | +| `openrouter/minimax/minimax-m2-her` | 0.30 | 1.20 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5347ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/mistralai/mistral-small-2603` | 0.15 | 0.60 | Tested only | 60 | 50 | 0/1 | model-behavior | 238670ms | Passed direct and first relay, then failed chained/concurrent reply flow. | + +Interpretation: `openrouter/z-ai/glm-5.1` is the only new repeat-run candidate from this batch. `openrouter/qwen/qwen3.6-plus` is interesting but should remain `Tested only`: it completed most stages but failed metadata/dedup correctness, which is exactly the class of issue that breaks the Messages UI. Xiaomi v2.5/v2.5 Pro and MiniMax M2 HER are currently unusable through this OpenCode route because OpenCode does not expose those provider-scoped model ids in its live catalog. + +## Flash And Compact Models Single-Run + +Source: lower-cost flash/compact routes across Z.ai, Qwen, StepFun, MiniMax, Xiaomi, and Mistral. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-diverse-gauntlet-1777161454" OPENCODE_E2E_MODELS="openrouter/z-ai/glm-5-turbo,openrouter/z-ai/glm-4.7-flash,openrouter/qwen/qwen3.5-flash-02-23,openrouter/stepfun/step-3.5-flash,openrouter/minimax/minimax-m2.1,openrouter/xiaomi/mimo-v2-flash,openrouter/mistralai/ministral-14b-2512" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ----------------------------------------- | ---------: | ----------: | ---------------- | --------: | --: | --------: | ----------------- | -------: | ---------------------------------------------------------------------------- | +| `openrouter/z-ai/glm-5-turbo` | 1.20 | 4.00 | Tested only | 54 | 35 | 0/1 | model-behavior | 130954ms | Passed direct reply, then failed peer relay and concurrent delivery. | +| `openrouter/z-ai/glm-4.7-flash` | 0.06 | 0.40 | Strong candidate | 100 | 100 | 1/1 | none | 243535ms | Passed all stages, but was slow enough to need repeat latency validation. | +| `openrouter/qwen/qwen3.5-flash-02-23` | 0.065 | 0.26 | Tested only | 68 | 95 | 0/1 | model-behavior | 93897ms | Completed the flow but failed duplicate-token correctness. | +| `openrouter/stepfun/step-3.5-flash` | 0.10 | 0.30 | Strong candidate | 100 | 100 | 1/1 | none | 97550ms | Passed launch, direct, peer relay, multi-hop, concurrent, taskRefs, hygiene. | +| `openrouter/minimax/minimax-m2.1` | 0.29 | 0.95 | Strong candidate | 100 | 100 | 1/1 | none | 115326ms | Passed all stages; cheaper than M2.5/M2.7 and worth a 3-run check. | +| `openrouter/xiaomi/mimo-v2-flash` | 0.09 | 0.29 | Tested only | 55 | 70 | 0/1 | runtime-transport | 263893ms | Failed concurrent Tom reply and taskRefs despite passing earlier stages. | +| `openrouter/mistralai/ministral-14b-2512` | 0.20 | 0.20 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5364ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: `openrouter/stepfun/step-3.5-flash` and `openrouter/minimax/minimax-m2.1` are the most practical new repeat-run candidates from this batch: both passed 100/100 and stayed near 1.5-2 minutes. `openrouter/z-ai/glm-4.7-flash` also passed, but its p50 was 243s, so it needs latency repeat validation before promotion. `openrouter/qwen/qwen3.5-flash-02-23` is not safe despite 95/100 because duplicate visible reply tokens are a user-facing Messages UI bug. + +## Pro And Small Alternative Models Single-Run + +Source: another non-overlapping batch across DeepSeek, Z.ai, Qwen, Mistral, Baidu, and Nous. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-diverse-gauntlet-1777162728" OPENCODE_E2E_MODELS="openrouter/deepseek/deepseek-v4-pro,openrouter/z-ai/glm-4.7,openrouter/qwen/qwen3.5-35b-a3b,openrouter/mistralai/ministral-8b-2512,openrouter/mistralai/ministral-3b-2512,openrouter/baidu/ernie-4.5-21b-a3b,openrouter/nousresearch/hermes-4-70b" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ---------------------------------------- | ---------: | ----------: | ---------------- | --------: | --: | --------: | ---------------- | -------: | ---------------------------------------------------------------------------- | +| `openrouter/deepseek/deepseek-v4-pro` | 0.435 | 0.87 | Tested only | 54 | 35 | 0/1 | model-behavior | 297105ms | Passed direct reply, then timed out on peer relay and concurrent delivery. | +| `openrouter/z-ai/glm-4.7` | 0.38 | 1.74 | Strong candidate | 100 | 100 | 1/1 | none | 170582ms | Passed launch, direct, peer relay, multi-hop, concurrent, taskRefs, hygiene. | +| `openrouter/qwen/qwen3.5-35b-a3b` | 0.1625 | 1.30 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5414ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/mistralai/ministral-8b-2512` | 0.15 | 0.15 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5354ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/mistralai/ministral-3b-2512` | 0.10 | 0.10 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5312ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/baidu/ernie-4.5-21b-a3b` | 0.07 | 0.28 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5335ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/nousresearch/hermes-4-70b` | 0.13 | 0.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 7575ms | Not found in the live OpenCode provider-scoped catalog or launch handshake. | + +Interpretation: `openrouter/z-ai/glm-4.7` is the only new repeat-run candidate from this batch. `openrouter/deepseek/deepseek-v4-pro` was a useful negative result: it is inexpensive for a large pro model, but failed the Agent Teams relay scenario badly. The other five routes are currently catalog/launch blocked in OpenCode, so they should not appear as viable Agent Teams choices until OpenCode exposes them. + +## Legacy And Broad Provider Models Single-Run + +Source: another broad batch across Anthropic, xAI, Google, Mistral, DeepSeek, Qwen, and Z.ai. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-diverse-gauntlet-1777163463" OPENCODE_E2E_MODELS="openrouter/anthropic/claude-haiku-4.5,openrouter/x-ai/grok-4,openrouter/google/gemini-2.0-flash-001,openrouter/mistralai/devstral-small,openrouter/deepseek/deepseek-chat-v3.1,openrouter/qwen/qwen3-30b-a3b,openrouter/z-ai/glm-4.5-air" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ---------------------------------------- | ---------: | ----------: | ---------------- | --------: | --: | --------: | ---------------- | -------: | ---------------------------------------------------------------------------- | +| `openrouter/anthropic/claude-haiku-4.5` | 1.00 | 5.00 | Strong candidate | 100 | 100 | 1/1 | none | 91040ms | Passed launch, direct, peer relay, multi-hop, concurrent, taskRefs, hygiene. | +| `openrouter/x-ai/grok-4` | 3.00 | 15.00 | Tested only | 54 | 35 | 0/1 | model-behavior | 165700ms | Passed direct reply, then failed peer relay with empty assistant turn. | +| `openrouter/google/gemini-2.0-flash-001` | 0.10 | 0.40 | Tested only | 60 | 50 | 0/1 | model-behavior | 113041ms | Passed direct and first relay, then failed chained/concurrent relay. | +| `openrouter/mistralai/devstral-small` | 0.10 | 0.30 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5603ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/deepseek/deepseek-chat-v3.1` | 0.15 | 0.75 | Tested only | 52 | 70 | 0/1 | model-behavior | 271581ms | Failed concurrent replies, taskRefs, and duplicate-token correctness. | +| `openrouter/qwen/qwen3-30b-a3b` | 0.08 | 0.28 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5477ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/z-ai/glm-4.5-air` | 0.13 | 0.85 | Infra blocked | 0 | 35 | 0/1 | provider-infra | 140700ms | Passed direct reply, then failed peer relay through OpenCode tool handling. | + +Interpretation: `openrouter/anthropic/claude-haiku-4.5` is the only new strong result in this batch and is now a serious cheap-Anthropic repeat-run candidate. `openrouter/x-ai/grok-4` is a useful negative result because `grok-4.1-fast` passed earlier while full `grok-4` failed this Agent Teams relay scenario. Older `gemini-2.0-flash-001` also looks unsafe compared with newer Gemini 3 Flash routes. + +## Older Strong Families Regression Batch + +Source: another non-overlapping batch covering older/alternate routes from families that looked promising elsewhere. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-diverse-gauntlet-1777164494" OPENCODE_E2E_MODELS="openrouter/anthropic/claude-sonnet-4.5,openrouter/anthropic/claude-opus-4.5,openrouter/google/gemini-2.5-flash-lite,openrouter/mistralai/mistral-medium-3,openrouter/x-ai/grok-3-mini,openrouter/qwen/qwen3-next-80b-a3b-instruct,openrouter/z-ai/glm-4.6" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| --------------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ----------------- | -------: | ---------------------------------------------------------------------------- | +| `openrouter/anthropic/claude-sonnet-4.5` | 3.00 | 15.00 | Infra blocked | 0 | 70 | 0/1 | provider-infra | 281971ms | Passed most flow, then failed concurrent Tom/taskRefs/protocol handling. | +| `openrouter/anthropic/claude-opus-4.5` | 5.00 | 25.00 | Infra blocked | 0 | 70 | 0/1 | provider-infra | 274162ms | Same concurrent Tom/taskRefs/protocol failure pattern as Sonnet 4.5. | +| `openrouter/google/gemini-2.5-flash-lite` | 0.10 | 0.40 | Tested only | 52 | 70 | 0/1 | runtime-transport | 264105ms | Failed concurrent Tom reply and taskRefs, confirming older negative signal. | +| `openrouter/mistralai/mistral-medium-3` | 0.40 | 2.00 | Tested only | 46 | 15 | 0/1 | runtime-transport | 217744ms | Timed out already on direct reply, not viable for Agent Teams. | +| `openrouter/x-ai/grok-3-mini` | 0.30 | 0.50 | Tested only | 54 | 35 | 0/1 | model-behavior | 142579ms | Passed direct reply, then failed peer relay/concurrent delivery. | +| `openrouter/qwen/qwen3-next-80b-a3b-instruct` | 0.09 | 1.10 | Infra blocked | 0 | 35 | 0/1 | provider-infra | 237510ms | Passed direct reply, then timed out on peer relay through OpenCode. | +| `openrouter/z-ai/glm-4.6` | 0.39 | 1.90 | Tested only | 54 | 35 | 0/1 | runtime-transport | 293919ms | Fresh gauntlet regressed vs older smoke: direct reply only, peer relay fail. | + +Interpretation: this batch produced no new repeat-run candidate. It also downgrades confidence in several older `Tested` routes: `openrouter/z-ai/glm-4.6` and `openrouter/mistralai/mistral-medium-3` should not be promoted without a future clean gauntlet. The newer winners from the same families remain more interesting: `glm-4.7`, `glm-5.1`, `grok-4.1-fast`, and `gemini-3-flash-preview`. + +## Broad Cheap Alternatives Single-Run + +Source: another broad batch across OpenAI, Qwen, xAI, MiniMax, and Google cheap/compact routes. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-broad-cheap-gauntlet-1777166547" OPENCODE_E2E_MODELS="openrouter/openai/gpt-oss-120b,openrouter/openai/gpt-5-nano,openrouter/qwen/qwen3-coder-30b-a3b-instruct,openrouter/qwen/qwen3-coder-next,openrouter/x-ai/grok-4-fast,openrouter/minimax/minimax-m2,openrouter/google/gemini-2.0-flash-lite-001" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ---------------------------------------------- | ---------: | ----------: | ---------------- | --------: | --: | --------: | ----------------- | -------: | ------------------------------------------------------------------------------------------------ | +| `openrouter/openai/gpt-oss-120b` | 0.039 | 0.19 | Tested only | 60 | 50 | 0/1 | model-behavior | 237273ms | Passed launch, direct, and first relay, then failed chained relay, concurrent, and hygiene. | +| `openrouter/openai/gpt-5-nano` | 0.05 | 0.40 | Strong candidate | 100 | 100 | 1/1 | none | 235304ms | Passed launch, direct, peer relays, concurrent replies, taskRefs, transcript hygiene. | +| `openrouter/qwen/qwen3-coder-30b-a3b-instruct` | 0.07 | 0.27 | Tested only | 55 | 70 | 0/1 | runtime-transport | 285835ms | Passed direct and peer relay, then failed concurrent reply, taskRefs, and duplicate-token check. | +| `openrouter/qwen/qwen3-coder-next` | 0.14 | 0.80 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5448ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/x-ai/grok-4-fast` | 0.20 | 0.50 | Infra blocked | 0 | 60 | 0/1 | provider-infra | 272477ms | Passed most flow, then failed concurrent Tom reply, taskRefs, hygiene, and duplicate check. | +| `openrouter/minimax/minimax-m2` | 0.255 | 1.00 | Strong candidate | 100 | 100 | 1/1 | none | 116814ms | Passed launch, direct, peer relays, concurrent replies, taskRefs, transcript hygiene. | +| `openrouter/google/gemini-2.0-flash-lite-001` | 0.075 | 0.30 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5321ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: `openrouter/minimax/minimax-m2` is the best new practical candidate from this batch because it passed 100/100 with much lower latency than `gpt-5-nano`. `openrouter/openai/gpt-5-nano` also passed cleanly, but its single run was slow enough to require repeat validation. `openrouter/openai/gpt-oss-120b`, `openrouter/qwen/qwen3-coder-30b-a3b-instruct`, and `openrouter/x-ai/grok-4-fast` are not safe for Agent Teams despite partial success, because they failed exactly the multi-agent relay/concurrent behaviors we care about. + +## Broad General Alternatives Single-Run + +Source: another broad batch across OpenAI, Qwen, DeepSeek, Mistral, and Nvidia routes. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-broad-general-gauntlet-1777167901" OPENCODE_E2E_MODELS="openrouter/openai/gpt-5.4-nano,openrouter/openai/gpt-4.1-nano,openrouter/qwen/qwen-plus,openrouter/qwen/qwen3-235b-a22b-2507,openrouter/deepseek/deepseek-v3.2-exp,openrouter/mistralai/codestral-2508,openrouter/nvidia/nemotron-3-super-120b-a12b" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ---------------------------------------------- | ---------: | ----------: | ---------------- | --------: | --: | --------: | ----------------- | -------: | ----------------------------------------------------------------------------------------------- | +| `openrouter/openai/gpt-5.4-nano` | 0.20 | 1.25 | Tested only | 55 | 70 | 0/1 | runtime-transport | 262911ms | Passed direct and peer relays, then failed concurrent Tom reply, taskRefs, and duplicate check. | +| `openrouter/openai/gpt-4.1-nano` | 0.10 | 0.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5317ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen-plus` | 0.26 | 0.78 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5230ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen3-235b-a22b-2507` | 0.071 | 0.10 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5356ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/deepseek/deepseek-v3.2-exp` | 0.27 | 0.41 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5195ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/mistralai/codestral-2508` | 0.30 | 0.90 | Strong candidate | 100 | 100 | 1/1 | none | 79312ms | Passed launch, direct, peer relays, concurrent replies, taskRefs, transcript hygiene. | +| `openrouter/nvidia/nemotron-3-super-120b-a12b` | 0.09 | 0.45 | Tested only | 54 | 35 | 0/1 | model-behavior | 150382ms | Passed direct reply, then failed peer relay, concurrent delivery, hygiene, and latency. | + +Interpretation: `openrouter/mistralai/codestral-2508` is the standout here: it had the fastest clean 100/100 pass among the latest two batches and was already on the tested list. `openrouter/openai/gpt-5.4-nano` looks unsafe despite decent partial progress because it failed the concurrent/taskRefs path. `openrouter/nvidia/nemotron-3-super-120b-a12b` is not a viable Agent Teams model right now: it got only the first direct reply right. + +## Broad Affordable Alternatives Single-Run + +Source: another broad batch across Mistral, Cohere, Reka, Google Gemma, and Qwen low-cost routes. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-broad-affordable-gauntlet-1777168585" OPENCODE_E2E_MODELS="openrouter/mistralai/mistral-small-3.2-24b-instruct,openrouter/mistralai/mistral-nemo,openrouter/cohere/command-r7b-12-2024,openrouter/cohere/command-r-08-2024,openrouter/rekaai/reka-flash-3,openrouter/google/gemma-4-31b-it,openrouter/qwen/qwen3-32b" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ----------------------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ----------------- | -------: | ------------------------------------------------------------------------------------------- | +| `openrouter/mistralai/mistral-small-3.2-24b-instruct` | 0.075 | 0.20 | Tested only | 55 | 70 | 0/1 | runtime-transport | 274129ms | Passed direct and peer relays, then failed concurrent Tom reply, taskRefs, duplicate check. | +| `openrouter/mistralai/mistral-nemo` | 0.01 | 0.03 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5692ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/cohere/command-r7b-12-2024` | 0.0375 | 0.15 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5463ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/cohere/command-r-08-2024` | 0.15 | 0.60 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5239ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/rekaai/reka-flash-3` | 0.10 | 0.20 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5225ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/google/gemma-4-31b-it` | 0.13 | 0.38 | Tested only | 46 | 15 | 0/1 | runtime-transport | 217938ms | Launched, but timed out already on direct reply. | +| `openrouter/qwen/qwen3-32b` | 0.08 | 0.24 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5639ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: this batch produced no new candidate for recommendation. `mistral-small-3.2-24b-instruct` is close enough to be useful as a negative regression target, but it failed the same concurrent/taskRefs path that matters most for Agent Teams. The cheaper Cohere/Reka/Mistral Nemo/Qwen 32B routes are blocked by OpenCode catalog support, not by model behavior. + +## Small And Mid Budget Routes Single-Run + +Source: another small/mid budget batch across OpenAI OSS, Google Gemma, Qwen, Mistral, Reka, and Nvidia routes. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-small-mid-gauntlet-1777169220" OPENCODE_E2E_MODELS="openrouter/openai/gpt-oss-20b,openrouter/google/gemma-3-27b-it,openrouter/qwen/qwen3-14b,openrouter/qwen/qwen3-8b,openrouter/mistralai/mistral-small-24b-instruct-2501,openrouter/rekaai/reka-edge,openrouter/nvidia/nemotron-3-nano-30b-a3b" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ------------------------------------------------------ | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -------: | ------------------------------------------------------- | +| `openrouter/openai/gpt-oss-20b` | 0.03 | 0.14 | Tested only | 46 | 15 | 0/1 | model-behavior | 218186ms | Launched, but timed out already on direct reply. | +| `openrouter/google/gemma-3-27b-it` | 0.08 | 0.16 | Tested only | 46 | 15 | 0/1 | model-behavior | 219925ms | Launched, but timed out already on direct reply. | +| `openrouter/qwen/qwen3-14b` | 0.06 | 0.24 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5761ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen3-8b` | 0.05 | 0.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5297ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/mistralai/mistral-small-24b-instruct-2501` | 0.05 | 0.08 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5478ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/rekaai/reka-edge` | 0.10 | 0.10 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5452ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/nvidia/nemotron-3-nano-30b-a3b` | 0.05 | 0.20 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5443ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: this batch is a clean negative result. The two routes that did launch, `gpt-oss-20b` and `gemma-3-27b-it`, failed before producing a first visible reply. The remaining cheap routes are unavailable in OpenCode provider scope today. None should be promoted or prioritized for Agent Teams. + +## Legacy Diverse Routes Single-Run + +Source: another diverse batch across Mistral, OpenAI, Qwen, DeepSeek, MiniMax, and Nvidia routes that were not covered by the previous small/mid batches. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-legacy-diverse-gauntlet-1777169838" OPENCODE_E2E_MODELS="openrouter/mistralai/mistral-small-3.1-24b-instruct,openrouter/mistralai/mixtral-8x7b-instruct,openrouter/openai/gpt-4o-mini,openrouter/qwen/qwq-32b,openrouter/deepseek/deepseek-chat,openrouter/minimax/minimax-01,openrouter/nvidia/llama-3.3-nemotron-super-49b-v1.5" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ----------------------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -------: | ------------------------------------------------------------------------------------------ | +| `openrouter/mistralai/mistral-small-3.1-24b-instruct` | 0.35 | 0.56 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 9235ms | Did not reach ready state in OpenCode provider-scope launch. | +| `openrouter/mistralai/mixtral-8x7b-instruct` | 0.54 | 0.54 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5467ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-4o-mini` | 0.15 | 0.60 | Infra blocked | 0 | 70 | 0/1 | provider-infra | 268294ms | Passed direct and peer relays, then failed concurrent delivery, taskRefs, duplicate check. | +| `openrouter/qwen/qwq-32b` | 0.15 | 0.58 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5232ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/deepseek/deepseek-chat` | 0.32 | 0.89 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5145ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/minimax/minimax-01` | 0.20 | 1.10 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8112ms | Did not reach ready state in OpenCode provider-scope launch. | +| `openrouter/nvidia/llama-3.3-nemotron-super-49b-v1.5` | 0.10 | 0.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5220ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no new candidate here. `gpt-4o-mini` is the only model that got far enough to be behaviorally interesting, but it failed exactly the concurrent/taskRefs path and should not be used for Agent Teams. The others are currently blocked before meaningful model behavior can be scored. + +## Alternative Compact Routes Single-Run + +Source: another compact/alternate batch across OpenAI, DeepSeek, Qwen, Google Gemma, and Nvidia routes. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-alt-compact-gauntlet-1777170257" OPENCODE_E2E_MODELS="openrouter/openai/gpt-4o-mini-2024-07-18,openrouter/openai/gpt-4o-mini-search-preview,openrouter/deepseek/deepseek-chat-v3-0324,openrouter/qwen/qwen3-next-80b-a3b-thinking,openrouter/qwen/qwen-turbo,openrouter/google/gemma-4-26b-a4b-it,openrouter/nvidia/nemotron-nano-9b-v2" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ---------------------------------------------- | ---------: | ----------: | ---------------- | --------: | --: | --------: | ---------------- | -------: | ------------------------------------------------------------------------------------------------ | +| `openrouter/openai/gpt-4o-mini-2024-07-18` | 0.15 | 0.60 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5446ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-4o-mini-search-preview` | 0.15 | 0.60 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5345ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/deepseek/deepseek-chat-v3-0324` | 0.20 | 0.77 | Tested only | 54 | 35 | 0/1 | model-behavior | 123504ms | Passed direct reply, then failed peer relay, concurrent delivery, hygiene, and latency. | +| `openrouter/qwen/qwen3-next-80b-a3b-thinking` | 0.0975 | 0.78 | Tested only | 46 | 15 | 0/1 | model-behavior | 218859ms | Launched, but timed out already on direct reply. | +| `openrouter/qwen/qwen-turbo` | 0.0325 | 0.13 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5163ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/google/gemma-4-26b-a4b-it` | 0.06 | 0.33 | Strong candidate | 100 | 100 | 1/1 | none | 175859ms | Passed launch, direct, peer relays, concurrent replies, taskRefs, transcript hygiene. | +| `openrouter/nvidia/nemotron-nano-9b-v2` | 0.04 | 0.16 | Infra blocked | 0 | 35 | 0/1 | provider-infra | 150316ms | Passed direct reply, then failed peer relay and later Agent Teams stages through OpenCode tools. | + +Interpretation: `openrouter/google/gemma-4-26b-a4b-it` is a useful new cheap candidate and is notably better than `gemma-4-31b-it`, which timed out on direct reply. It should be repeated before any promotion because this is only one successful run. `deepseek-chat-v3-0324`, Qwen Next Thinking, and Nemotron Nano are not safe for Agent Teams. + +## OpenAI Codex And Prior Tested Routes Single-Run + +Source: a targeted batch across routes that had older smoke/tested signals in the UI and needed a fresh deeper gauntlet check. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-openai-codex-gauntlet-1777171112" OPENCODE_E2E_MODELS="openrouter/openai/gpt-5.1,openrouter/openai/gpt-5.1-codex-mini,openrouter/openai/gpt-5.1-codex,openrouter/openai/gpt-5.3-codex,openrouter/openai/gpt-5.4-mini,openrouter/qwen/qwen3-max,openrouter/moonshotai/kimi-k2.6" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| -------------------------------------- | ---------: | ----------: | ---------------- | --------: | --: | --------: | ----------------- | -------: | ------------------------------------------------------------------------------------------- | +| `openrouter/openai/gpt-5.1` | 1.25 | 10.00 | Infra blocked | 0 | 70 | 0/1 | provider-infra | 266597ms | Passed direct and peer relays, then failed concurrent Tom reply, taskRefs, duplicate check. | +| `openrouter/openai/gpt-5.1-codex-mini` | 0.25 | 2.00 | Strong candidate | 100 | 100 | 1/1 | none | 83426ms | Passed launch, direct, peer relays, concurrent replies, taskRefs, transcript hygiene. | +| `openrouter/openai/gpt-5.1-codex` | 1.25 | 10.00 | Tested only | 40 | 0 | 0/1 | model-behavior | 65076ms | Failed launch readiness during model verification. | +| `openrouter/openai/gpt-5.3-codex` | 1.75 | 14.00 | Strong candidate | 100 | 100 | 1/1 | none | 111932ms | Passed launch, direct, peer relays, concurrent replies, taskRefs, transcript hygiene. | +| `openrouter/openai/gpt-5.4-mini` | 0.75 | 4.50 | Infra blocked | 0 | 70 | 0/1 | provider-infra | 265504ms | Passed direct and peer relays, then failed concurrent Tom reply, taskRefs, duplicate check. | +| `openrouter/qwen/qwen3-max` | 0.78 | 3.90 | Tested only | 55 | 70 | 0/1 | runtime-transport | 298675ms | Passed direct and peer relays, then failed concurrent Tom reply, taskRefs, duplicate check. | +| `openrouter/moonshotai/kimi-k2.6` | 0.7448 | 4.655 | Strong candidate | 100 | 100 | 1/1 | none | 163496ms | Passed launch, direct, peer relays, concurrent replies, taskRefs, transcript hygiene. | + +Interpretation: this batch materially changes the UI status of several routes. `gpt-5.1-codex-mini`, `gpt-5.3-codex`, and `kimi-k2.6` remain strong repeat-run candidates. `gpt-5.1`, `gpt-5.4-mini`, and `qwen3-max` should no longer be treated as safe just because older smoke evidence existed: each failed the concurrent/taskRefs path. `gpt-5.1-codex` also failed launch readiness and should stay below `Tested` until rerun cleanly. + +## Upper Tier Prior Tested Routes Single-Run + +Source: a targeted upper-tier batch across Anthropic, OpenAI, Google, Mistral, and Xiaomi routes that were still present in the UI tested set or looked like possible production candidates. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-upper-tier-gauntlet-1777172576" OPENCODE_E2E_MODELS="openrouter/anthropic/claude-opus-4.6,openrouter/anthropic/claude-opus-4.7,openrouter/openai/gpt-5.4,openrouter/google/gemini-2.5-flash,openrouter/mistralai/mistral-medium-3.1,openrouter/mistralai/devstral-2512,openrouter/xiaomi/mimo-v2-pro" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ----------------------------------------- | ---------: | ----------: | ---------------- | --------: | --: | --------: | ---------------- | -------: | ------------------------------------------------------------------------------------- | +| `openrouter/anthropic/claude-opus-4.6` | 5.00 | 25.00 | Infra blocked | 0 | 15 | 0/1 | provider-infra | 220945ms | Launched, but timed out already on direct reply. | +| `openrouter/anthropic/claude-opus-4.7` | 5.00 | 25.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 26623ms | Failed OpenCode launch readiness. | +| `openrouter/openai/gpt-5.4` | 2.50 | 15.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 26608ms | Failed OpenCode launch readiness. | +| `openrouter/google/gemini-2.5-flash` | 0.30 | 2.50 | Strong candidate | 100 | 100 | 1/1 | none | 108333ms | Passed launch, direct, peer relays, concurrent replies, taskRefs, transcript hygiene. | +| `openrouter/mistralai/mistral-medium-3.1` | 0.40 | 2.00 | Tested only | 54 | 35 | 0/1 | model-behavior | 227298ms | Passed direct reply, then timed out on peer relay and later Agent Teams stages. | +| `openrouter/mistralai/devstral-2512` | 0.40 | 2.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 23751ms | Failed OpenCode launch readiness. | +| `openrouter/xiaomi/mimo-v2-pro` | 1.00 | 3.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 27870ms | Failed OpenCode launch readiness. | + +Interpretation: `openrouter/google/gemini-2.5-flash` is the only positive result in this upper-tier batch and remains a serious repeat-run candidate. The fresh evidence is negative for `opus-4.6`, `opus-4.7`, `gpt-5.4`, `mistral-medium-3.1`, `devstral-2512`, and `mimo-v2-pro`, so these routes were removed from the UI tested set and marked not recommended until a future clean gauntlet disproves this run. + +## Free Route Variants Single-Run + +Source: a free-route batch across Gemma, Nvidia, OpenAI OSS, and Qwen. These routes are intentionally tracked separately from paid routes because free provider routing can have different capacity, latency, and model behavior. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-free-routes-gauntlet-1777173485" OPENCODE_E2E_MODELS="openrouter/google/gemma-4-26b-a4b-it:free,openrouter/google/gemma-4-31b-it:free,openrouter/nvidia/nemotron-3-super-120b-a12b:free,openrouter/nvidia/nemotron-3-nano-30b-a3b:free,openrouter/openai/gpt-oss-120b:free,openrouter/qwen/qwen3-coder:free,openrouter/qwen/qwen3-next-80b-a3b-instruct:free" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| --------------------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -------: | ---------------------------------------------------------------------------------------- | +| `openrouter/google/gemma-4-26b-a4b-it:free` | 0.00 | 0.00 | Tested only | 40 | 0 | 0/1 | model-behavior | 65701ms | Failed launch readiness during model verification; paid Gemma 4 26B behaved much better. | +| `openrouter/google/gemma-4-31b-it:free` | 0.00 | 0.00 | Tested only | 40 | 0 | 0/1 | model-behavior | 65528ms | Failed launch readiness during model verification. | +| `openrouter/nvidia/nemotron-3-super-120b-a12b:free` | 0.00 | 0.00 | Tested only | 54 | 35 | 0/1 | model-behavior | 127081ms | Passed direct reply, then failed peer relay and later Agent Teams stages. | +| `openrouter/nvidia/nemotron-3-nano-30b-a3b:free` | 0.00 | 0.00 | Tested only | 54 | 35 | 0/1 | model-behavior | 259532ms | Passed direct reply, then timed out on peer relay and later Agent Teams stages. | +| `openrouter/openai/gpt-oss-120b:free` | 0.00 | 0.00 | Infra blocked | 0 | 35 | 0/1 | provider-infra | 253457ms | Passed direct reply, then timed out on peer relay and later Agent Teams stages. | +| `openrouter/qwen/qwen3-coder:free` | 0.00 | 0.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5414ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen3-next-80b-a3b-instruct:free` | 0.00 | 0.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5702ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: none of these free routes should be promoted. The most important result is that `openrouter/google/gemma-4-26b-a4b-it:free` does not inherit the positive signal from the paid `openrouter/google/gemma-4-26b-a4b-it` route; the free route failed launch verification. Existing free-route UI status was tightened accordingly. + +## Next Diverse Routes Single-Run + +Source: another batch across Kimi, DeepSeek, Z.ai, and xAI routes that were not covered by the latest gauntlet batches. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-diverse-gauntlet-1777174654" OPENCODE_E2E_MODELS="openrouter/moonshotai/kimi-k2.5,openrouter/moonshotai/kimi-k2-0905,openrouter/moonshotai/kimi-k2,openrouter/deepseek/deepseek-v3.2-speciale,openrouter/deepseek/deepseek-v3.1-terminus,openrouter/z-ai/glm-4.5,openrouter/x-ai/grok-4.20" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| -------------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | ------: | ------------------------------------------------------------------------- | +| `openrouter/moonshotai/kimi-k2.5` | 0.44 | 2.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 31813ms | Launch was blocked by OpenRouter key/max_tokens allowance for 32k output. | +| `openrouter/moonshotai/kimi-k2-0905` | 0.40 | 2.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 25066ms | Launch was blocked by OpenRouter key/max_tokens allowance for 16k output. | +| `openrouter/moonshotai/kimi-k2` | 0.57 | 2.30 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 24283ms | Launch was blocked by OpenRouter key/max_tokens allowance for 32k output. | +| `openrouter/deepseek/deepseek-v3.2-speciale` | 0.40 | 1.20 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 7401ms | OpenRouter reported no endpoints that support tool use for this route. | +| `openrouter/deepseek/deepseek-v3.1-terminus` | 0.21 | 0.79 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 23360ms | Launch was blocked by OpenRouter key/max_tokens allowance for 32k output. | +| `openrouter/z-ai/glm-4.5` | 0.60 | 2.20 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 7378ms | Production Agent Teams prompt exceeded the route/key token allowance. | +| `openrouter/x-ai/grok-4.20` | 2.00 | 6.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5280ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: this batch produced no new promotion candidate. The Kimi and DeepSeek Terminus results are not useful behavioral judgments yet because they were blocked by the current OpenRouter key/max_tokens allowance before Agent Teams messaging could start. `deepseek-v3.2-speciale` and `grok-4.20` are more actionable negatives for the UI: one lacks tool-use endpoints, the other is not available in the live OpenCode catalog. `glm-4.5` is also not safe under current production prompts because launch fails before the team reaches ready state. + +## Next Mixed Routes Single-Run + +Source: another mixed batch across OpenAI, Gemini, Z.ai, xAI, Qwen, and Mistral routes. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-mixed-gauntlet-1777175031" OPENCODE_E2E_MODELS="openrouter/openai/gpt-5.5,openrouter/google/gemini-3.1-pro-preview-customtools,openrouter/google/gemini-3.1-flash-image-preview,openrouter/z-ai/glm-5v-turbo,openrouter/x-ai/grok-4.20-multi-agent,openrouter/qwen/qwen3.5-plus-02-15,openrouter/mistralai/mistral-small-creative" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ------------------------------------------------------ | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | ------: | ------------------------------------------------------------------------- | +| `openrouter/openai/gpt-5.5` | 5.00 | 30.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 10120ms | Launch was blocked by OpenRouter key/max_tokens allowance even at 512. | +| `openrouter/google/gemini-3.1-pro-preview-customtools` | 2.00 | 12.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8751ms | Production Agent Teams prompt exceeded the route/key token allowance. | +| `openrouter/google/gemini-3.1-flash-image-preview` | 0.50 | 3.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5985ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/z-ai/glm-5v-turbo` | 1.20 | 4.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6234ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/x-ai/grok-4.20-multi-agent` | 2.00 | 6.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5976ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen3.5-plus-02-15` | 0.26 | 1.56 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 29055ms | Launch was blocked by OpenRouter key/max_tokens allowance for 32k output. | +| `openrouter/mistralai/mistral-small-creative` | 0.10 | 0.30 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6181ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: this batch produced no new candidate. The most actionable result is catalog coverage: `gemini-3.1-flash-image-preview`, `glm-5v-turbo`, `grok-4.20-multi-agent`, and `mistral-small-creative` are present in OpenRouter but not usable through the current live OpenCode provider-scoped catalog. `gpt-5.5` and `qwen3.5-plus-02-15` are blocked by the current OpenRouter key/max_tokens allowance, so they need a rerun only if we raise the key limits. `gemini-3.1-pro-preview-customtools` is not viable under current production Agent Teams prompts because the route/key token allowance is too low. + +## Next Cheap/Mid Routes Single-Run + +Source: another cheap/mid batch across OpenAI, MiniMax, Gemma, Mistral Voxtral, and Qwen. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-cheap-mid-gauntlet-1777175221" OPENCODE_E2E_MODELS="openrouter/openai/gpt-5.2,openrouter/openai/gpt-5.3-chat,openrouter/minimax/minimax-m1,openrouter/google/gemma-3n-e4b-it,openrouter/google/gemma-3n-e2b-it:free,openrouter/mistralai/voxtral-small-24b-2507,openrouter/qwen/qwen3.5-397b-a17b" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| --------------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | ------: | ------------------------------------------------------------------------- | +| `openrouter/openai/gpt-5.2` | 1.75 | 14.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 9666ms | Launch was blocked by OpenRouter key/max_tokens allowance even at 512. | +| `openrouter/openai/gpt-5.3-chat` | 1.75 | 14.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6261ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/minimax/minimax-m1` | 0.40 | 2.20 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8741ms | Production Agent Teams prompt exceeded the route/key token allowance. | +| `openrouter/google/gemma-3n-e4b-it` | 0.06 | 0.12 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8583ms | OpenRouter reported no endpoints that support tool use for this route. | +| `openrouter/google/gemma-3n-e2b-it:free` | 0.00 | 0.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8754ms | OpenRouter reported no endpoints that support tool use for this route. | +| `openrouter/mistralai/voxtral-small-24b-2507` | 0.10 | 0.30 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6042ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen3.5-397b-a17b` | 0.39 | 2.34 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 27540ms | Launch was blocked by OpenRouter key/max_tokens allowance for 32k output. | + +Interpretation: this batch produced no new candidate. `gpt-5.3-chat` and `voxtral-small-24b-2507` are OpenRouter-visible but absent from the current OpenCode provider-scoped catalog. Gemma 3n routes are not useful for Agent Teams through this path because OpenRouter reported no tool-use endpoints. `minimax-m1` is not a replacement for the stronger MiniMax M2/M2.1/M2.5 routes: it fails before the team reaches ready state under production prompts. + +## Next Practical Routes Single-Run + +Source: practical/cheap route batch across OpenAI, xAI, Qwen, and DeepSeek Chimera. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-practical-gauntlet-1777175565" OPENCODE_E2E_MODELS="openrouter/openai/gpt-5-mini,openrouter/openai/gpt-5-chat,openrouter/openai/gpt-5-codex,openrouter/x-ai/grok-3-mini-beta,openrouter/qwen/qwen-2.5-coder-32b-instruct,openrouter/qwen/qwen-2.5-72b-instruct,openrouter/tngtech/deepseek-r1t2-chimera" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| --------------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | ------: | ------------------------------------------------------------------------- | +| `openrouter/openai/gpt-5-mini` | 0.25 | 2.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 29354ms | Launch was blocked by OpenRouter key/max_tokens allowance for 32k output. | +| `openrouter/openai/gpt-5-chat` | 1.25 | 10.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6166ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-5-codex` | 1.25 | 10.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8741ms | Production Agent Teams prompt exceeded the route/key token allowance. | +| `openrouter/x-ai/grok-3-mini-beta` | 0.30 | 0.50 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 35543ms | Launch was blocked by OpenRouter key/max_tokens allowance for 8k output. | +| `openrouter/qwen/qwen-2.5-coder-32b-instruct` | 0.66 | 1.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8845ms | OpenRouter reported no endpoints that support tool use for this route. | +| `openrouter/qwen/qwen-2.5-72b-instruct` | 0.12 | 0.39 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6023ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/tngtech/deepseek-r1t2-chimera` | 0.30 | 1.10 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5970ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: this batch produced no new candidate. `gpt-5-mini`, `grok-3-mini-beta`, and `gpt-5-codex` need a rerun only if key/max-token limits are raised enough for production Agent Teams prompts. The Qwen 2.5 and DeepSeek Chimera routes are not useful through the current OpenCode path: either they are absent from the live OpenCode catalog or OpenRouter reports no tool-use endpoints. + +## Legacy And Preview Routes Single-Run + +Source: another batch across legacy Anthropic/Gemini/Mistral routes and Qwen VL. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-legacy-preview-gauntlet-1777175763" OPENCODE_E2E_MODELS="openrouter/google/gemini-2.5-pro-preview,openrouter/google/gemini-2.5-pro-preview-05-06,openrouter/mistralai/mistral-saba,openrouter/mistralai/mistral-large-2411,openrouter/anthropic/claude-3.5-haiku,openrouter/anthropic/claude-3.7-sonnet,openrouter/qwen/qwen3-vl-30b-a3b-instruct" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ------------------------------------------------ | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -----: | ---------------------------------------------------------------------- | +| `openrouter/google/gemini-2.5-pro-preview` | 1.25 | 10.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6854ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/google/gemini-2.5-pro-preview-05-06` | 1.25 | 10.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 9299ms | Launch was blocked by OpenRouter key/max_tokens allowance even at 512. | +| `openrouter/mistralai/mistral-saba` | 0.20 | 0.60 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5764ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/mistralai/mistral-large-2411` | 2.00 | 6.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5928ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/anthropic/claude-3.5-haiku` | 0.80 | 4.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8470ms | Production Agent Teams prompt exceeded the route/key token allowance. | +| `openrouter/anthropic/claude-3.7-sonnet` | 3.00 | 15.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8163ms | Launch was blocked by OpenRouter key/max_tokens allowance even at 512. | +| `openrouter/qwen/qwen3-vl-30b-a3b-instruct` | 0.13 | 0.52 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6018ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: this batch produced no new candidate. The older preview/legacy routes are not better than the already-tested modern winners: they either are missing from the OpenCode catalog or cannot accept production Agent Teams prompts under the current key/token allowance. For practical usage, keep focus on the routes that already passed full gauntlet stages, such as Gemini 3 Flash, Gemini 3.1 Flash Lite, MiniMax M2/M2.1/M2.5, Codestral 2508, GLM 4.7/5.1, Step 3.5 Flash, and Claude Sonnet 4.6 until repeated promotion runs are done. + +## Cheap Tool-Capable Routes Single-Run + +Source: another cheap/diverse batch selected from the live OpenRouter model catalog for long-context routes that advertised tool support. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-cheap-tools-gauntlet-1777176070" OPENCODE_E2E_MODELS="openrouter/inclusionai/ling-2.6-1t:free,openrouter/inclusionai/ling-2.6-flash:free,openrouter/nvidia/nemotron-nano-12b-v2-vl:free,openrouter/meta-llama/llama-3.1-8b-instruct,openrouter/qwen/qwen-2.5-7b-instruct,openrouter/amazon/nova-lite-v1,openrouter/z-ai/glm-4-32b,openrouter/google/gemini-2.5-flash-lite-preview-09-2025" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| --------------------------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | ------: | ------------------------------------------------------------------------- | +| `openrouter/inclusionai/ling-2.6-1t:free` | 0.00 | 0.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6516ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/inclusionai/ling-2.6-flash:free` | 0.00 | 0.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6031ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/nvidia/nemotron-nano-12b-v2-vl:free` | 0.00 | 0.00 | Tested only | 40 | 0 | 0/1 | model-behavior | 65941ms | Failed launch readiness because model verification timed out. | +| `openrouter/meta-llama/llama-3.1-8b-instruct` | 0.02 | 0.05 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6117ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen-2.5-7b-instruct` | 0.04 | 0.10 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5760ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/amazon/nova-lite-v1` | 0.06 | 0.24 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6086ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/z-ai/glm-4-32b` | 0.10 | 0.10 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6226ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/google/gemini-2.5-flash-lite-preview-09-2025` | 0.10 | 0.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 26204ms | Launch was blocked by OpenRouter key/max_tokens allowance for 32k output. | + +Interpretation: this batch produced no new candidate. The important practical signal is that OpenRouter catalog availability plus advertised tool support still does not mean the route is visible to the OpenCode provider-scoped catalog. The only non-catalog failure worth revisiting later is `openrouter/google/gemini-2.5-flash-lite-preview-09-2025`, which reached OpenRouter but was blocked by current key/max-token allowance. + +## Diverse Tool-Capable Routes Single-Run + +Source: another cheap/mid diverse batch selected from the live OpenRouter model catalog for routes that advertised tool support. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-diverse-tools-gauntlet-1777176324" OPENCODE_E2E_MODELS="openrouter/bytedance-seed/seed-1.6-flash,openrouter/meta-llama/llama-4-scout,openrouter/qwen/qwen3-30b-a3b-instruct-2507,openrouter/meta-llama/llama-3.3-70b-instruct,openrouter/bytedance-seed/seed-2.0-mini,openrouter/qwen/qwen3-vl-32b-instruct,openrouter/alibaba/tongyi-deepresearch-30b-a3b,openrouter/arcee-ai/trinity-large-preview" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ------------------------------------------------ | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | ------: | ------------------------------------------------------------------------- | +| `openrouter/bytedance-seed/seed-1.6-flash` | 0.075 | 0.30 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6015ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/meta-llama/llama-4-scout` | 0.08 | 0.30 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5266ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen3-30b-a3b-instruct-2507` | 0.09 | 0.30 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 27361ms | Launch was blocked by OpenRouter key/max_tokens allowance for 32k output. | +| `openrouter/meta-llama/llama-3.3-70b-instruct` | 0.10 | 0.32 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5511ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/bytedance-seed/seed-2.0-mini` | 0.10 | 0.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5502ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen3-vl-32b-instruct` | 0.104 | 0.416 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5737ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/alibaba/tongyi-deepresearch-30b-a3b` | 0.09 | 0.45 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5490ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/arcee-ai/trinity-large-preview` | 0.15 | 0.45 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5505ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: this batch also produced no candidate. Most of these OpenRouter routes are not currently usable through OpenCode's provider-scoped catalog even though OpenRouter advertises tool support. `openrouter/qwen/qwen3-30b-a3b-instruct-2507` reached a different failure mode: it was visible enough to hit OpenRouter, but the production Agent Teams launch was blocked by the key/max-token allowance. + +## Wide Tool-Capable Routes Single-Run + +Source: another low/mid-cost batch selected from the live OpenRouter model catalog for tool-capable routes that had not yet been recorded. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-wide-tools-gauntlet-1777176494" OPENCODE_E2E_MODELS="openrouter/amazon/nova-micro-v1,openrouter/arcee-ai/trinity-mini,openrouter/qwen/qwen3.5-9b,openrouter/essentialai/rnj-1-instruct,openrouter/upstage/solar-pro-3,openrouter/allenai/olmo-3.1-32b-instruct,openrouter/inception/mercury-2,openrouter/qwen/qwen-plus-2025-07-28" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ------------------------------------------ | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -----: | --------------------------------------------------------------------- | +| `openrouter/amazon/nova-micro-v1` | 0.035 | 0.14 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6041ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/arcee-ai/trinity-mini` | 0.045 | 0.15 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5490ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen3.5-9b` | 0.10 | 0.15 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5554ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/essentialai/rnj-1-instruct` | 0.15 | 0.15 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5569ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/upstage/solar-pro-3` | 0.15 | 0.60 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5425ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/allenai/olmo-3.1-32b-instruct` | 0.20 | 0.60 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5458ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/inception/mercury-2` | 0.25 | 0.75 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8281ms | Production Agent Teams prompt exceeded the route/key token allowance. | +| `openrouter/qwen/qwen-plus-2025-07-28` | 0.26 | 0.78 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5495ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no new candidate. The repeated pattern is now clear enough to trust for UI status: many OpenRouter routes are real catalog entries, but unavailable through the OpenCode provider-scoped launch catalog. `openrouter/inception/mercury-2` is visible enough to execute, but cannot fit the production Agent Teams launch prompt under the current route/key allowance. + +## Broad Tool-Capable Routes Single-Run + +Source: another broad batch selected from the live OpenRouter model catalog for cheap or mid-cost tool-capable routes that had not yet been recorded. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-broad-tools-gauntlet-1777176681" OPENCODE_E2E_MODELS="openrouter/tencent/hy3-preview:free,openrouter/nvidia/nemotron-nano-9b-v2:free,openrouter/google/gemma-3-12b-it,openrouter/openai/gpt-oss-safeguard-20b,openrouter/qwen/qwen3-30b-a3b-thinking-2507,openrouter/qwen/qwen3-vl-8b-instruct,openrouter/nex-agi/deepseek-v3.1-nex-n1,openrouter/baidu/ernie-4.5-vl-28b-a3b" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| --------------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -------: | ------------------------------------------------------------------------- | +| `openrouter/tencent/hy3-preview:free` | 0.00 | 0.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6736ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/nvidia/nemotron-nano-9b-v2:free` | 0.00 | 0.00 | Tested only | 54 | 35 | 0/1 | model-behavior | 217470ms | Passed direct reply, then failed peer relay through OpenCode tool usage. | +| `openrouter/google/gemma-3-12b-it` | 0.04 | 0.13 | Infra blocked | 0 | 15 | 0/1 | provider-infra | 222783ms | Launched, but timed out already on direct reply. | +| `openrouter/openai/gpt-oss-safeguard-20b` | 0.075 | 0.30 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 23531ms | Launch was blocked by OpenRouter key/max_tokens allowance for 32k output. | +| `openrouter/qwen/qwen3-30b-a3b-thinking-2507` | 0.08 | 0.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 27877ms | Launch was blocked by OpenRouter key/max_tokens allowance for 32k output. | +| `openrouter/qwen/qwen3-vl-8b-instruct` | 0.08 | 0.50 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5427ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/nex-agi/deepseek-v3.1-nex-n1` | 0.135 | 0.50 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5521ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/baidu/ernie-4.5-vl-28b-a3b` | 0.14 | 0.56 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5306ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no new candidate. The best partial signal was `openrouter/nvidia/nemotron-nano-9b-v2:free`, which launched and answered the direct user message but failed peer relay through OpenCode's tool path. `openrouter/google/gemma-3-12b-it` launched but timed out on the first direct reply, so it is not a practical Agent Teams route. + +## Mid Tool-Capable Routes Single-Run + +Source: another mid-cost batch selected from the live OpenRouter model catalog for tool-capable routes that had not yet been recorded. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-mid-tools-gauntlet-1777177301" OPENCODE_E2E_MODELS="openrouter/thedrummer/rocinante-12b,openrouter/meta-llama/llama-3.1-70b-instruct,openrouter/qwen/qwen-plus-2025-07-28:thinking,openrouter/z-ai/glm-4.6v,openrouter/prime-intellect/intellect-3,openrouter/anthropic/claude-3-haiku,openrouter/openai/gpt-4.1-mini,openrouter/bytedance-seed/seed-2.0-lite" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ----------------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -----: | --------------------------------------------------------------------- | +| `openrouter/thedrummer/rocinante-12b` | 0.17 | 0.43 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5720ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/meta-llama/llama-3.1-70b-instruct` | 0.40 | 0.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5418ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen-plus-2025-07-28:thinking` | 0.26 | 0.78 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5491ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/z-ai/glm-4.6v` | 0.30 | 0.90 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5503ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/prime-intellect/intellect-3` | 0.20 | 1.10 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8182ms | Production Agent Teams prompt exceeded the route/key token allowance. | +| `openrouter/anthropic/claude-3-haiku` | 0.25 | 1.25 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5465ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-4.1-mini` | 0.40 | 1.60 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 7631ms | Production Agent Teams prompt exceeded the route/key token allowance. | +| `openrouter/bytedance-seed/seed-2.0-lite` | 0.25 | 2.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5473ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no new candidate. `openrouter/openai/gpt-4.1-mini` and `openrouter/prime-intellect/intellect-3` are visible enough to hit OpenRouter, but they cannot fit the production Agent Teams bootstrap under the current route/key token allowance. The rest are unavailable through OpenCode's provider-scoped catalog. + +## Upper-Mid Tool-Capable Routes Single-Run + +Source: upper-mid/high-interest routes selected from the live OpenRouter model catalog, including larger Qwen, DeepSeek, Amazon Nova, OpenAI o-series, and Mistral routes. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-upper-mid-gauntlet-1777177431" OPENCODE_E2E_MODELS="openrouter/qwen/qwen3-235b-a22b,openrouter/qwen/qwen3.5-122b-a10b,openrouter/deepseek/deepseek-r1-0528,openrouter/amazon/nova-2-lite-v1,openrouter/openai/o4-mini,openrouter/openai/o3-mini,openrouter/mistralai/mistral-large-2407" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ----------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -----: | ---------------------------------------------------------------------- | +| `openrouter/qwen/qwen3-235b-a22b` | 0.455 | 1.82 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5318ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen3.5-122b-a10b` | 0.26 | 2.08 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5539ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/deepseek/deepseek-r1-0528` | 0.50 | 2.15 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5529ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/amazon/nova-2-lite-v1` | 0.30 | 2.50 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5396ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/o4-mini` | 1.10 | 4.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8177ms | Launch was blocked by OpenRouter key/max_tokens allowance even at 512. | +| `openrouter/openai/o3-mini` | 1.10 | 4.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5454ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/mistralai/mistral-large-2407` | 2.00 | 6.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5584ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no new candidate. These higher-profile routes do not currently improve the Agent Teams OpenCode set: most are absent from OpenCode's provider-scoped catalog, and `openrouter/openai/o4-mini` was blocked by key/max-token allowance even at the reduced 512-token verification cap. + +## Remaining Mid Tool-Capable Routes Single-Run + +Source: another batch from the remaining unrecorded low/mid-cost tool-capable OpenRouter routes. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-remaining-mid-gauntlet-1777177599" OPENCODE_E2E_MODELS="openrouter/thedrummer/unslopnemo-12b,openrouter/arcee-ai/trinity-large-thinking,openrouter/qwen/qwen3-vl-235b-a22b-instruct,openrouter/qwen/qwen3-vl-8b-thinking,openrouter/kwaipilot/kat-coder-pro-v2,openrouter/qwen/qwen3-235b-a22b-thinking-2507,openrouter/xiaomi/mimo-v2-omni,openrouter/deepseek/deepseek-r1" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ----------------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -----: | --------------------------------------------------------------------- | +| `openrouter/thedrummer/unslopnemo-12b` | 0.40 | 0.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6576ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/arcee-ai/trinity-large-thinking` | 0.22 | 0.85 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 9348ms | Production Agent Teams prompt exceeded the route/key token allowance. | +| `openrouter/qwen/qwen3-vl-235b-a22b-instruct` | 0.20 | 0.88 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6038ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen3-vl-8b-thinking` | 0.117 | 1.365 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5979ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/kwaipilot/kat-coder-pro-v2` | 0.30 | 1.20 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6162ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen3-235b-a22b-thinking-2507` | 0.1495 | 1.495 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8808ms | Production Agent Teams prompt exceeded the route/key token allowance. | +| `openrouter/xiaomi/mimo-v2-omni` | 0.40 | 2.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8381ms | Production Agent Teams prompt exceeded the route/key token allowance. | +| `openrouter/deepseek/deepseek-r1` | 0.70 | 2.50 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8272ms | Production Agent Teams prompt exceeded the route/key token allowance. | + +Interpretation: no new candidate. The main useful signal is negative: several reasoning/thinking or VL routes are either absent from OpenCode's provider-scoped catalog or too constrained for production Agent Teams bootstrap prompts under the current key/route allowance. + +## High-Profile Routes Single-Run + +Source: high-profile OpenAI/Cohere routes selected from the remaining unrecorded tool-capable OpenRouter models. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-high-profile-gauntlet-1777177738" OPENCODE_E2E_MODELS="openrouter/openai/o4-mini-high,openrouter/openai/o3-mini-high,openrouter/openai/gpt-4.1,openrouter/openai/gpt-5,openrouter/openai/gpt-4o,openrouter/cohere/command-r-plus-08-2024" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ------------------------------------------ | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -----: | ---------------------------------------------------------------------- | +| `openrouter/openai/o4-mini-high` | 1.10 | 4.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6810ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/o3-mini-high` | 1.10 | 4.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6182ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-4.1` | 2.00 | 8.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 9192ms | Launch was blocked by OpenRouter key/max_tokens allowance even at 512. | +| `openrouter/openai/gpt-5` | 1.25 | 10.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8688ms | Launch was blocked by OpenRouter key/max_tokens allowance even at 512. | +| `openrouter/openai/gpt-4o` | 2.50 | 10.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6099ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/cohere/command-r-plus-08-2024` | 2.50 | 10.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6050ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no new candidate. This confirms that several familiar high-profile OpenRouter IDs are not available through the current OpenCode provider-scoped catalog. The two visible OpenAI routes in this batch, `gpt-4.1` and `gpt-5`, are blocked by current key/max-token allowance before they can prove Agent Teams behavior. + +## Tail Mid Tool-Capable Routes Single-Run + +Source: another tail batch from the remaining unrecorded mid-cost tool-capable OpenRouter routes. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-tail-mid-gauntlet-1777177902" OPENCODE_E2E_MODELS="openrouter/qwen/qwen3-vl-30b-a3b-thinking,openrouter/sao10k/l3.1-euryale-70b,openrouter/qwen/qwen3.5-27b,openrouter/arcee-ai/virtuoso-large,openrouter/openai/gpt-3.5-turbo,openrouter/bytedance-seed/seed-1.6,openrouter/z-ai/glm-4.5v,openrouter/nvidia/llama-3.1-nemotron-70b-instruct" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| --------------------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -----: | --------------------------------------------------------------------- | +| `openrouter/qwen/qwen3-vl-30b-a3b-thinking` | 0.13 | 1.56 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6770ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/sao10k/l3.1-euryale-70b` | 0.85 | 0.85 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6150ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen3.5-27b` | 0.195 | 1.56 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6142ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/arcee-ai/virtuoso-large` | 0.75 | 1.20 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6001ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-3.5-turbo` | 0.50 | 1.50 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5883ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/bytedance-seed/seed-1.6` | 0.25 | 2.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6180ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/z-ai/glm-4.5v` | 0.60 | 1.80 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 9425ms | Production Agent Teams prompt exceeded the route/key token allowance. | +| `openrouter/nvidia/llama-3.1-nemotron-70b-instruct` | 1.20 | 1.20 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6030ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no new candidate. This further confirms that most long-tail OpenRouter routes are not available through the current OpenCode provider-scoped catalog. `openrouter/z-ai/glm-4.5v` is visible enough to reach OpenRouter, but its effective token allowance is too small for production Agent Teams launch prompts. + +## Final Mid Tool-Capable Routes Single-Run + +Source: final mid-cost tail batch selected from the remaining unrecorded OpenRouter routes before the expensive deep-research/pro tier. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-next-final-mid-gauntlet-1777178031" OPENCODE_E2E_MODELS="openrouter/qwen/qwen-vl-max,openrouter/qwen/qwen3-vl-235b-a22b-thinking,openrouter/openai/gpt-audio-mini,openrouter/amazon/nova-pro-v1,openrouter/relace/relace-search,openrouter/qwen/qwen-max,openrouter/mistralai/pixtral-large-2411,openrouter/mistralai/mixtral-8x22b-instruct" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| --------------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -----: | ------------------------------------------------------- | +| `openrouter/qwen/qwen-vl-max` | 0.52 | 2.08 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6496ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen3-vl-235b-a22b-thinking` | 0.26 | 2.60 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5985ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-audio-mini` | 0.60 | 2.40 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6015ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/amazon/nova-pro-v1` | 0.80 | 3.20 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6113ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/relace/relace-search` | 1.00 | 3.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5935ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen-max` | 1.04 | 4.16 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6078ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/mistralai/pixtral-large-2411` | 2.00 | 6.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6114ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/mistralai/mixtral-8x22b-instruct` | 2.00 | 6.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5803ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no new candidate. Every route in this batch exists in OpenRouter but was missing from the current OpenCode provider-scoped launch catalog, so none can be offered as a practical Agent Teams OpenCode choice. + +## More High-Value Routes Single-Run + +Source: additional high-value and legacy/pro OpenRouter routes selected from the remaining unrecorded tool-capable routes. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-more-gauntlet-1777178255" OPENCODE_E2E_MODELS="openrouter/openai/gpt-3.5-turbo-16k,openrouter/mistralai/mistral-large,openrouter/openai/o4-mini-deep-research,openrouter/ai21/jamba-large-1.7,openrouter/openai/o3,openrouter/openai/gpt-5.1-codex-max" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ----------------------------------------- | ---------: | ----------: | --------------- | --------: | --: | --------: | ---------------- | -----: | --------------------------------------------------------------------- | +| `openrouter/openai/gpt-3.5-turbo-16k` | 3.00 | 4.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6940ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/mistralai/mistral-large` | 2.00 | 6.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5846ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/o4-mini-deep-research` | 2.00 | 8.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6124ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/ai21/jamba-large-1.7` | 2.00 | 8.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6067ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/o3` | 2.00 | 8.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6181ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-5.1-codex-max` | 1.25 | 10.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 9088ms | OpenRouter key/max_tokens allowance was too low for launch bootstrap. | + +Interpretation: no new candidate. Five routes are still absent from OpenCode's provider-scoped catalog. `openrouter/openai/gpt-5.1-codex-max` is visible enough to reach OpenRouter, but the current key/max_tokens allowance cannot support even the reduced production launch check. + +## More Legacy Audio And Chat Routes Single-Run + +Source: second additional batch from remaining OpenAI audio/versioned routes, Amazon Nova Premier, and GPT-5.2 Chat. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-more-gauntlet-1777178436" OPENCODE_E2E_MODELS="openrouter/openai/gpt-audio,openrouter/openai/gpt-4o-audio-preview,openrouter/openai/gpt-4o-2024-11-20,openrouter/openai/gpt-4o-2024-08-06,openrouter/amazon/nova-premier-v1,openrouter/openai/gpt-5.2-chat" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ---------------------------------------- | ---------: | ----------: | --------------- | --------: | --: | --------: | ---------------- | -----: | --------------------------------------------------------------------- | +| `openrouter/openai/gpt-audio` | 2.50 | 10.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6425ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-4o-audio-preview` | 2.50 | 10.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6152ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-4o-2024-11-20` | 2.50 | 10.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5901ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-4o-2024-08-06` | 2.50 | 10.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6163ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/amazon/nova-premier-v1` | 2.50 | 12.50 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5831ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-5.2-chat` | 1.75 | 14.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 9207ms | OpenRouter key/max_tokens allowance was too low for launch bootstrap. | + +Interpretation: no new candidate. The audio/versioned OpenAI routes and Nova Premier are unavailable through the current OpenCode provider-scoped launch catalog. `openrouter/openai/gpt-5.2-chat` reached OpenRouter but is not practical with the current key/max_tokens allowance for production Agent Teams prompts. + +## Expensive Remaining Routes Single-Run + +Source: next remaining high-cost OpenRouter routes before the ultra-expensive pro tail. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-more-gauntlet-1777178600" OPENCODE_E2E_MODELS="openrouter/x-ai/grok-3,openrouter/anthropic/claude-sonnet-4,openrouter/x-ai/grok-3-beta,openrouter/anthropic/claude-3.7-sonnet:thinking,openrouter/openai/gpt-4o-2024-05-13,openrouter/~anthropic/claude-opus-latest" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ------------------------------------------------- | ---------: | ----------: | --------------- | --------: | --: | --------: | ---------------- | -----: | --------------------------------------------------------------------- | +| `openrouter/x-ai/grok-3` | 3.00 | 15.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8702ms | OpenRouter key/max_tokens allowance was too low for launch bootstrap. | +| `openrouter/anthropic/claude-sonnet-4` | 3.00 | 15.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 7765ms | OpenRouter key/max_tokens allowance was too low for launch bootstrap. | +| `openrouter/x-ai/grok-3-beta` | 3.00 | 15.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 7590ms | OpenRouter key/max_tokens allowance was too low for launch bootstrap. | +| `openrouter/anthropic/claude-3.7-sonnet:thinking` | 3.00 | 15.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5562ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-4o-2024-05-13` | 5.00 | 15.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5641ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/~anthropic/claude-opus-latest` | 5.00 | 25.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5368ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no new candidate. The visible expensive Grok/Sonnet routes were blocked by key/max_tokens allowance during launch bootstrap. The thinking/legacy alias routes were absent from OpenCode's provider-scoped launch catalog. + +## Turbo And Deep-Research Tail Single-Run + +Source: additional expensive OpenAI legacy Turbo and deep-research routes from the remaining unrecorded tail. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-more-gauntlet-1777178755" OPENCODE_E2E_MODELS="openrouter/openai/gpt-4-turbo,openrouter/openai/gpt-4-turbo-preview,openrouter/openai/gpt-4-1106-preview,openrouter/openai/o3-deep-research,openrouter/openai/o1" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| --------------------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -----: | ------------------------------------------------------- | +| `openrouter/openai/gpt-4-turbo` | 10.00 | 30.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6739ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-4-turbo-preview` | 10.00 | 30.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6031ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-4-1106-preview` | 10.00 | 30.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6093ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/o3-deep-research` | 10.00 | 40.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6252ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/o1` | 15.00 | 60.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6146ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no new candidate. These expensive OpenAI legacy/deep-research routes exist in OpenRouter, but none is currently launchable through OpenCode's provider-scoped catalog. + +## Ultra-Expensive Pro Tail Single-Run + +Source: small ultra-expensive batch from remaining Opus/OpenAI Pro routes. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-more-gauntlet-1777178864" OPENCODE_E2E_MODELS="openrouter/anthropic/claude-opus-4.1,openrouter/anthropic/claude-opus-4,openrouter/openai/o3-pro,openrouter/openai/gpt-5-pro" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| -------------------------------------- | ---------: | ----------: | --------------- | --------: | --: | --------: | ---------------- | -----: | --------------------------------------------------------------------- | +| `openrouter/anthropic/claude-opus-4.1` | 15.00 | 75.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 9975ms | OpenRouter key/max_tokens allowance was too low for launch bootstrap. | +| `openrouter/anthropic/claude-opus-4` | 15.00 | 75.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8442ms | OpenRouter key/max_tokens allowance was too low for launch bootstrap. | +| `openrouter/openai/o3-pro` | 20.00 | 80.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6138ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-5-pro` | 15.00 | 120.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8762ms | OpenRouter key/max_tokens allowance was too low for launch bootstrap. | + +Interpretation: no new candidate. Opus 4.x and GPT-5 Pro reached OpenRouter but were blocked by key/max_tokens allowance before behavioral testing. `o3-pro` was absent from OpenCode's provider-scoped launch catalog. + +## Final Pro Tail Single-Run + +Source: final pro-tail batch, excluding `openrouter/auto` because it is an aggregator route rather than a stable model candidate. Prices were read from `https://openrouter.ai/api/v1/models` before the run. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-more-gauntlet-1777178974" OPENCODE_E2E_MODELS="openrouter/anthropic/claude-opus-4.6-fast,openrouter/openai/gpt-5.2-pro,openrouter/openai/gpt-5.5-pro,openrouter/openai/gpt-5.4-pro" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ------------------------------------------- | ---------: | ----------: | --------------- | --------: | --: | --------: | ---------------- | -----: | --------------------------------------------------------------------- | +| `openrouter/anthropic/claude-opus-4.6-fast` | 30.00 | 150.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 7245ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-5.2-pro` | 21.00 | 168.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 9413ms | OpenRouter key/max_tokens allowance was too low for launch bootstrap. | +| `openrouter/openai/gpt-5.5-pro` | 30.00 | 180.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6177ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openai/gpt-5.4-pro` | 30.00 | 180.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 8702ms | OpenRouter key/max_tokens allowance was too low for launch bootstrap. | + +Interpretation: no new candidate. The only remaining pro-tail routes either are absent from OpenCode's provider-scoped catalog or cannot pass the launch bootstrap with the current OpenRouter key/max_tokens allowance. Further testing here requires a higher key allowance before behavioral conclusions are possible. + +## Auto Router Single-Run + +Source: final unrecorded OpenRouter tool-capable route from the catalog sweep. `openrouter/auto` is an aggregator route rather than a stable model candidate, but it was still checked to avoid leaving an unclassified entry. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-more-gauntlet-1777179110" OPENCODE_E2E_MODELS="openrouter/openrouter/auto" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ---------------------------- | ---------: | ----------: | ------------- | --------: | --: | --------: | ---------------- | -----: | ------------------------------------------------------- | +| `openrouter/openrouter/auto` | -1.00 | -1.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6596ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no candidate. Even the OpenRouter auto-router was not present in OpenCode's provider-scoped launch catalog, and it would be unsuitable for a stable Agent Teams recommendation anyway because routing can vary by request. + +## Edge Free And Non-Tool Routes Single-Run + +Source: first edge batch outside the tool-capable sweep. These routes were selected from unrecorded low-cost/free OpenRouter entries to explicitly classify non-tool, OCR/audio/router-like models that are unlikely to fit Agent Teams. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-edge-gauntlet-1777179178" OPENCODE_E2E_MODELS="openrouter/openrouter/pareto-code,openrouter/openrouter/bodybuilder,openrouter/baidu/qianfan-ocr-fast:free,openrouter/google/lyria-3-pro-preview,openrouter/liquid/lfm-2.5-1.2b-thinking:free,openrouter/liquid/lfm-2.5-1.2b-instruct:free,openrouter/google/gemma-3-4b-it:free,openrouter/meta-llama/llama-3.2-3b-instruct:free" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| -------------------------------------------------- | ---------: | ----------: | --------------- | --------: | --: | --------: | ---------------- | -----: | -------------------------------------------------------- | +| `openrouter/openrouter/pareto-code` | -1.00 | -1.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6615ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/openrouter/bodybuilder` | -1.00 | -1.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6048ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/baidu/qianfan-ocr-fast:free` | 0.00 | 0.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6035ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/google/lyria-3-pro-preview` | 0.00 | 0.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6088ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/liquid/lfm-2.5-1.2b-thinking:free` | 0.00 | 0.00 | Not recommended | 0 | 0 | 0/1 | provider-infra | 8597ms | OpenRouter reported no tool-use endpoint for this route. | +| `openrouter/liquid/lfm-2.5-1.2b-instruct:free` | 0.00 | 0.00 | Not recommended | 0 | 0 | 0/1 | provider-infra | 8563ms | OpenRouter reported no tool-use endpoint for this route. | +| `openrouter/google/gemma-3-4b-it:free` | 0.00 | 0.00 | Not recommended | 0 | 0 | 0/1 | provider-infra | 8759ms | OpenRouter reported no tool-use endpoint for this route. | +| `openrouter/meta-llama/llama-3.2-3b-instruct:free` | 0.00 | 0.00 | Not recommended | 0 | 0 | 0/1 | provider-infra | 7912ms | OpenRouter reported no tool-use endpoint for this route. | + +Interpretation: no new candidate. The router/OCR/audio-like routes were absent from OpenCode's provider-scoped catalog. The small free text routes were visible enough to reach OpenRouter, but lack tool-use endpoints, so they cannot support Agent Teams MCP messaging. + +## Edge Guard Vision And Distill Routes Single-Run + +Source: second edge batch outside the tool-capable sweep, covering free Gemma/Hermes, small Granite/Phi, guard, vision, and distill routes. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-edge-gauntlet-1777179324" OPENCODE_E2E_MODELS="openrouter/google/gemma-3-27b-it:free,openrouter/nousresearch/hermes-3-llama-3.1-405b:free,openrouter/google/gemma-3-4b-it,openrouter/ibm-granite/granite-4.0-h-micro,openrouter/microsoft/phi-4,openrouter/meta-llama/llama-guard-4-12b,openrouter/qwen/qwen-vl-plus,openrouter/deepseek/deepseek-r1-distill-qwen-32b" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ------------------------------------------------------ | ---------: | ----------: | --------------- | --------: | --: | --------: | ---------------- | -----: | -------------------------------------------------------- | +| `openrouter/google/gemma-3-27b-it:free` | 0.00 | 0.00 | Not recommended | 0 | 0 | 0/1 | provider-infra | 9675ms | OpenRouter reported no tool-use endpoint for this route. | +| `openrouter/nousresearch/hermes-3-llama-3.1-405b:free` | 0.00 | 0.00 | Not recommended | 0 | 0 | 0/1 | provider-infra | 8710ms | OpenRouter reported no tool-use endpoint for this route. | +| `openrouter/google/gemma-3-4b-it` | 0.04 | 0.08 | Not recommended | 0 | 0 | 0/1 | provider-infra | 8724ms | OpenRouter reported no tool-use endpoint for this route. | +| `openrouter/ibm-granite/granite-4.0-h-micro` | 0.017 | 0.11 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6065ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/microsoft/phi-4` | 0.065 | 0.14 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6055ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/meta-llama/llama-guard-4-12b` | 0.18 | 0.18 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6048ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen-vl-plus` | 0.1365 | 0.4095 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5995ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/deepseek/deepseek-r1-distill-qwen-32b` | 0.29 | 0.29 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5822ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no new candidate. Free Gemma/Hermes and paid Gemma lacked tool-use endpoints, while Granite/Phi/guard/vision/distill routes were absent from OpenCode's provider-scoped launch catalog. + +## Edge Creative Small And UI Routes Single-Run + +Source: third edge batch outside the tool-capable sweep, covering audio/creative preview, free uncensored/Gemma, small Liquid/Llama, UI-TARS, ERNIE thinking, and Arcee Spotlight. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-edge-gauntlet-1777179503" OPENCODE_E2E_MODELS="openrouter/google/lyria-3-clip-preview,openrouter/cognitivecomputations/dolphin-mistral-24b-venice-edition:free,openrouter/google/gemma-3-12b-it:free,openrouter/liquid/lfm-2-24b-a2b,openrouter/meta-llama/llama-3.2-1b-instruct,openrouter/bytedance/ui-tars-1.5-7b,openrouter/baidu/ernie-4.5-21b-a3b-thinking,openrouter/arcee-ai/spotlight" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| -------------------------------------------------------------------------- | ---------: | ----------: | --------------- | --------: | --: | --------: | ---------------- | -----: | -------------------------------------------------------- | +| `openrouter/google/lyria-3-clip-preview` | 0.00 | 0.00 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6694ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/cognitivecomputations/dolphin-mistral-24b-venice-edition:free` | 0.00 | 0.00 | Not recommended | 0 | 0 | 0/1 | provider-infra | 8886ms | OpenRouter reported no tool-use endpoint for this route. | +| `openrouter/google/gemma-3-12b-it:free` | 0.00 | 0.00 | Not recommended | 0 | 0 | 0/1 | provider-infra | 8542ms | OpenRouter reported no tool-use endpoint for this route. | +| `openrouter/liquid/lfm-2-24b-a2b` | 0.03 | 0.12 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6188ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/meta-llama/llama-3.2-1b-instruct` | 0.027 | 0.20 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6094ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/bytedance/ui-tars-1.5-7b` | 0.10 | 0.20 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5779ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/baidu/ernie-4.5-21b-a3b-thinking` | 0.07 | 0.28 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6056ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/arcee-ai/spotlight` | 0.18 | 0.18 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6007ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no new candidate. The free Dolphin/Gemma routes lacked tool-use endpoints. The creative preview, small Liquid/Llama, UI, ERNIE, and Arcee routes were absent from OpenCode's provider-scoped launch catalog. + +## Edge Llama Hermes Reasoning And Vision Routes Single-Run + +Source: fourth edge batch outside the tool-capable sweep, covering small Llama, Llama vision, Llama Guard, Hermes 70B, Olmo/Tencent reasoning, Llama 4 Maverick, and Nemotron VL. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-edge-gauntlet-1777179653" OPENCODE_E2E_MODELS="openrouter/meta-llama/llama-3.2-3b-instruct,openrouter/meta-llama/llama-3.2-11b-vision-instruct,openrouter/meta-llama/llama-guard-3-8b,openrouter/nousresearch/hermes-3-llama-3.1-70b,openrouter/allenai/olmo-3-32b-think,openrouter/tencent/hunyuan-a13b-instruct,openrouter/meta-llama/llama-4-maverick,openrouter/nvidia/nemotron-nano-12b-v2-vl" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| ----------------------------------------------------- | ---------: | ----------: | --------------- | --------: | --: | --------: | ---------------- | -----: | -------------------------------------------------------- | +| `openrouter/meta-llama/llama-3.2-3b-instruct` | 0.051 | 0.34 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6598ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/meta-llama/llama-3.2-11b-vision-instruct` | 0.245 | 0.245 | Not recommended | 0 | 0 | 0/1 | provider-infra | 9078ms | OpenRouter reported no tool-use endpoint for this route. | +| `openrouter/meta-llama/llama-guard-3-8b` | 0.48 | 0.03 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5728ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/nousresearch/hermes-3-llama-3.1-70b` | 0.30 | 0.30 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6140ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/allenai/olmo-3-32b-think` | 0.15 | 0.50 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5893ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/tencent/hunyuan-a13b-instruct` | 0.14 | 0.57 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6118ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/meta-llama/llama-4-maverick` | 0.15 | 0.60 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6072ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/nvidia/nemotron-nano-12b-v2-vl` | 0.20 | 0.60 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6214ms | Not found in the live OpenCode provider-scoped catalog. | + +Interpretation: no new candidate. Llama vision lacked a tool-use endpoint. The rest of this edge batch was absent from OpenCode's provider-scoped launch catalog. + +## Edge Mid-Cost Creative Coder And Distill Routes Single-Run + +Source: fifth edge batch outside the tool-capable sweep, covering creative/roleplay routes, Qwen VL, WizardLM, Arcee Coder, Baidu ERNIE, Sao10K, and DeepSeek distill. + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_REPORT_DIR="/tmp/opencode-edge-gauntlet-1777179799" OPENCODE_E2E_MODELS="openrouter/thedrummer/cydonia-24b-v4.1,openrouter/qwen/qwen2.5-vl-72b-instruct,openrouter/microsoft/wizardlm-2-8x22b,openrouter/arcee-ai/coder-large,openrouter/thedrummer/skyfall-36b-v2,openrouter/baidu/ernie-4.5-300b-a47b,openrouter/sao10k/l3.3-euryale-70b,openrouter/deepseek/deepseek-r1-distill-llama-70b" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Input $/1M | Output $/1M | Verdict | Readiness | Avg | Pass Runs | Dominant Failure | p50 | Key finding | +| --------------------------------------------------- | ---------: | ----------: | --------------- | --------: | --: | --------: | ---------------- | -----: | -------------------------------------------------------- | +| `openrouter/thedrummer/cydonia-24b-v4.1` | 0.30 | 0.50 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 6797ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/qwen/qwen2.5-vl-72b-instruct` | 0.25 | 0.75 | Not recommended | 0 | 0 | 0/1 | provider-infra | 8233ms | OpenRouter reported no tool-use endpoint for this route. | +| `openrouter/microsoft/wizardlm-2-8x22b` | 0.62 | 0.62 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5419ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/arcee-ai/coder-large` | 0.50 | 0.80 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5508ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/thedrummer/skyfall-36b-v2` | 0.55 | 0.80 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5589ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/baidu/ernie-4.5-300b-a47b` | 0.28 | 1.10 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5441ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/sao10k/l3.3-euryale-70b` | 0.65 | 0.75 | Infra blocked | 0 | 0 | 0/1 | provider-infra | 5581ms | Not found in the live OpenCode provider-scoped catalog. | +| `openrouter/deepseek/deepseek-r1-distill-llama-70b` | 0.70 | 0.80 | Not recommended | 0 | 0 | 0/1 | provider-infra | 7560ms | OpenRouter reported no tool-use endpoint for this route. | + +Interpretation: no new candidate. Qwen VL and DeepSeek distill lacked tool-use endpoints. The creative, WizardLM, coder, ERNIE, and Sao10K routes were absent from OpenCode's provider-scoped launch catalog. + +## Latest Single-Run Format Smoke + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=1 OPENCODE_E2E_MODELS="opencode/minimax-m2.5-free" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Verdict | Confidence | Behavior Avg | Overall Avg | Counted | Pass Runs | Provider Infra | Runtime Transport | Model Fails | p50 | +| ---------------------------- | ----------- | ---------- | -----------: | ----------: | ------: | --------: | -------------: | ----------------: | ----------: | -------: | +| `opencode/minimax-m2.5-free` | Tested only | low | 35 | 35 | 1/1 | 0/1 | 0 | 0 | 1 | 226017ms | + +Per-run details from the latest smoke: + +| Run | Outcome | Category | Score | Counted | Duration | Failed Stages | Slowest Stage | TaskRefs | Protocol | +| --: | --------------- | -------------- | ----: | ------- | -------: | ---------------------------------------------------------------------------------------------- | -------------------- | -------------- | -------- | +| 1 | behavioral-fail | model-behavior | 35 | yes | 226017ms | peerRelayAB, peerRelayBC, concurrentReplies, cleanTranscript, noDuplicateTokens, latencyStable | peerRelayAB:183786ms | directReply:ok | - | + +This result is useful as a smoke signal only. It is not enough to mark the model `Recommended`, and the latest single-run result shows instability in peer relay. Keep it below `Recommended` until it passes repeated counted runs. + +## Repeated Top-3 Run + +Source command: + +```bash +OPENCODE_E2E=1 OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET=1 OPENCODE_E2E_USE_REAL_APP_CREDENTIALS=1 OPENCODE_E2E_GAUNTLET_RUNS=3 OPENCODE_E2E_MODELS="openrouter/minimax/minimax-m2.7,openrouter/anthropic/claude-sonnet-4.6,openrouter/google/gemini-3.1-pro-preview" pnpm vitest run test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts +``` + +| Model | Verdict | Confidence | Behavior Avg | Overall Avg | Counted | Pass Runs | Provider Infra | Runtime Transport | Model Fails | p50 | p95 | +| ------------------------------------------ | ------------------ | ---------- | -----------: | ----------: | ------: | --------: | -------------: | ----------------: | ----------: | -------: | -------: | +| `openrouter/minimax/minimax-m2.7` | Strong candidate | high | 88.3 | 88.3 | 3/3 | 2/3 | 0 | 1 | 0 | 102425ms | 298070ms | +| `openrouter/anthropic/claude-sonnet-4.6` | Recommended | high | 100 | 100 | 3/3 | 3/3 | 0 | 0 | 0 | 107810ms | 109271ms | +| `openrouter/google/gemini-3.1-pro-preview` | Infra/test blocked | blocked | n/a | 33.3 | mixed | 0/3 | multiple | mixed | mixed | 251812ms | 315644ms | + +## Interpretation + +`openrouter/anthropic/claude-sonnet-4.6` historically passed repeated production-path launches, direct delivery, peer relay, chained peer relay, concurrent deliveries, taskRefs, transcript hygiene, and duplicate guard checks. It is still kept as `Tested` in UI until rerun under the current gauntlet and recommendation gate. + +`openrouter/minimax/minimax-m2.7` is a strong candidate, not recommended yet. It passed 2/3 full runs, but one run hit a runtime transport failure during near-concurrent delivery: `tom` attempted `agent-teams_message_send` with correct payload, OpenCode returned `OpenCode tool failed without output`, and the expected visible reply never reached `user.json`. + +Latest 2026-04-26 targeted rerun passed 1/1 with 100/100 readiness and no runtime transport, taskRefs, protocol, duplicate-token, or transcript-hygiene failures. Product UI now treats it as `Tested`, not `Recommended`. + +`opencode/minimax-m2.5-free` has mixed gauntlet evidence: one earlier single-run pass and one latest single-run behavioral failure in peer relay. It should stay below `Recommended` and needs repeated runs before any promotion. + +`openrouter/google/gemini-3.1-pro-preview` is not judged as a clean model failure from this evidence because OpenRouter credit/max-token limits contaminated the run. It needs rerun after provider limits are resolved. diff --git a/src/features/runtime-provider-management/renderer/ui/RuntimeProviderManagementPanelView.tsx b/src/features/runtime-provider-management/renderer/ui/RuntimeProviderManagementPanelView.tsx index eaa85a52..7f1bf6a6 100644 --- a/src/features/runtime-provider-management/renderer/ui/RuntimeProviderManagementPanelView.tsx +++ b/src/features/runtime-provider-management/renderer/ui/RuntimeProviderManagementPanelView.tsx @@ -1126,15 +1126,22 @@ function ModelBadges({ ? 'bg-emerald-400/15 px-1.5 py-0 text-[10px] text-emerald-200' : modelRecommendation.level === 'recommended-with-limits' ? 'bg-amber-400/15 px-1.5 py-0 text-[10px] text-amber-200' - : modelRecommendation.level === 'unavailable-in-opencode' - ? 'bg-slate-400/15 px-1.5 py-0 text-[10px] text-slate-200' - : 'bg-red-400/15 px-1.5 py-0 text-[10px] text-red-200' + : modelRecommendation.level === 'tested' + ? 'bg-sky-400/15 px-1.5 py-0 text-[10px] text-sky-200' + : modelRecommendation.level === 'tested-with-limits' + ? 'bg-cyan-400/15 px-1.5 py-0 text-[10px] text-cyan-200' + : modelRecommendation.level === 'unavailable-in-opencode' + ? 'bg-slate-400/15 px-1.5 py-0 text-[10px] text-slate-200' + : 'bg-red-400/15 px-1.5 py-0 text-[10px] text-red-200' } title={modelRecommendation.reason} > {modelRecommendation.level === 'not-recommended' || modelRecommendation.level === 'unavailable-in-opencode' ? ( + ) : modelRecommendation.level === 'tested' || + modelRecommendation.level === 'tested-with-limits' ? ( + ) : ( )} diff --git a/src/renderer/components/team/dialogs/TeamModelSelector.tsx b/src/renderer/components/team/dialogs/TeamModelSelector.tsx index 25e41aa7..b112b872 100644 --- a/src/renderer/components/team/dialogs/TeamModelSelector.tsx +++ b/src/renderer/components/team/dialogs/TeamModelSelector.tsx @@ -2,6 +2,7 @@ import React, { useEffect, useMemo, useState } from 'react'; import { ProviderBrandLogo } from '@renderer/components/common/ProviderBrandLogo'; import { Checkbox } from '@renderer/components/ui/checkbox'; +import { Input } from '@renderer/components/ui/input'; import { Label } from '@renderer/components/ui/label'; import { Tabs, TabsList, TabsTrigger } from '@renderer/components/ui/tabs'; import { @@ -43,7 +44,7 @@ import { extractProviderScopedBaseModel } from '@renderer/utils/teamModelContext import { resolveAnthropicLaunchModel } from '@shared/utils/anthropicLaunchModel'; import { getAnthropicDefaultTeamModel } from '@shared/utils/anthropicModelDefaults'; import { isTeamProviderId } from '@shared/utils/teamProvider'; -import { AlertTriangle, Info, Star } from 'lucide-react'; +import { AlertTriangle, CheckCircle2, Info, Search, Star } from 'lucide-react'; import type { CliProviderStatus, TeamProviderId } from '@shared/types'; @@ -169,6 +170,7 @@ export const TeamModelSelector: React.FC = ({ }) => { const multimodelEnabled = useStore((s) => s.appConfig?.general?.multimodelEnabled ?? true); const [recommendedOnly, setRecommendedOnly] = useState(false); + const [modelQuery, setModelQuery] = useState(''); const effectiveProviderId = disableGeminiOption && isGeminiUiFrozen() && providerId === 'gemini' ? 'anthropic' : providerId; @@ -322,15 +324,41 @@ export const TeamModelSelector: React.FC = ({ } }, [effectiveProviderId, hasRecommendedOpenCodeModels]); + useEffect(() => { + setModelQuery(''); + }, [effectiveProviderId]); + const visibleModelOptions = useMemo(() => { + const normalizedModelQuery = modelQuery.trim().toLowerCase(); + const matchesModelQuery = (option: (typeof modelOptions)[number]): boolean => { + if (!normalizedModelQuery) { + return true; + } + const modelRecommendation = + effectiveProviderId === 'opencode' + ? getOpenCodeTeamModelRecommendation(option.value) + : null; + return [ + option.value, + option.label, + option.badgeLabel ?? '', + modelRecommendation?.label ?? '', + modelRecommendation?.reason ?? '', + ] + .join(' ') + .toLowerCase() + .includes(normalizedModelQuery); + }; + if (effectiveProviderId !== 'opencode') { - return modelOptions; + return modelOptions.filter(matchesModelQuery); } const concreteOptions = modelOptions .filter((option) => option.value.trim().length > 0) .map((option, index) => ({ option, index })) .filter(({ option }) => !recommendedOnly || isOpenCodeTeamModelRecommended(option.value)) + .filter(({ option }) => matchesModelQuery(option)) .sort((left, right) => { const recommendationOrder = compareOpenCodeTeamModelRecommendations( left.option.value, @@ -347,8 +375,11 @@ export const TeamModelSelector: React.FC = ({ return [ ...modelOptions.filter((option) => option.value.trim().length === 0), ...concreteOptions, - ]; - }, [effectiveProviderId, modelOptions, recommendedOnly]); + ].filter(matchesModelQuery); + }, [effectiveProviderId, modelOptions, modelQuery, recommendedOnly]); + const concreteModelOptionCount = modelOptions.filter((option) => option.value.trim()).length; + const shouldShowModelSearch = concreteModelOptionCount > 8; + const trimmedModelQuery = modelQuery.trim(); const shouldConstrainModelListHeight = visibleModelOptions.length > 8; return ( @@ -433,6 +464,20 @@ export const TeamModelSelector: React.FC = ({ list is syncing.

) : null} + {shouldShowModelSearch ? ( +
+ + setModelQuery(event.target.value)} + placeholder="Search models" + aria-label="Search models" + className="h-9 pr-3 text-sm" + style={{ paddingLeft: 40 }} + /> +
+ ) : null} {hasRecommendedOpenCodeModels ? (
= ({ ? 'bg-emerald-300/12 border-emerald-300/35 text-emerald-200' : modelRecommendation.level === 'recommended-with-limits' ? 'bg-amber-300/12 border-amber-300/35 text-amber-200' - : modelRecommendation.level === 'unavailable-in-opencode' - ? 'border-slate-300/30 bg-slate-400/10 text-slate-200' - : 'border-red-300/35 bg-red-400/10 text-red-200' + : modelRecommendation.level === 'tested' + ? 'bg-sky-300/12 border-sky-300/35 text-sky-200' + : modelRecommendation.level === 'tested-with-limits' + ? 'border-cyan-300/30 bg-cyan-400/10 text-cyan-200' + : modelRecommendation.level === 'unavailable-in-opencode' + ? 'border-slate-300/30 bg-slate-400/10 text-slate-200' + : 'border-red-300/35 bg-red-400/10 text-red-200' )} title={modelRecommendation.reason} > {modelRecommendation.level === 'not-recommended' || modelRecommendation.level === 'unavailable-in-opencode' ? ( + ) : modelRecommendation.level === 'tested' || + modelRecommendation.level === 'tested-with-limits' ? ( + ) : ( )} @@ -626,9 +678,13 @@ export const TeamModelSelector: React.FC = ({ })() )}
- {effectiveProviderId === 'opencode' && visibleModelOptions.length === 0 ? ( + {visibleModelOptions.length === 0 ? (
- No recommended OpenCode models are available in the current runtime list. + {trimmedModelQuery + ? 'No models match this search.' + : effectiveProviderId === 'opencode' && recommendedOnly + ? 'No recommended OpenCode models are available in the current runtime list.' + : 'No models are available in the current runtime list.'}
) : null} diff --git a/src/renderer/utils/openCodeModelRecommendations.ts b/src/renderer/utils/openCodeModelRecommendations.ts index 137b8d7d..c69e3588 100644 --- a/src/renderer/utils/openCodeModelRecommendations.ts +++ b/src/renderer/utils/openCodeModelRecommendations.ts @@ -1,6 +1,8 @@ export type OpenCodeTeamModelRecommendationLevel = | 'recommended' | 'recommended-with-limits' + | 'tested' + | 'tested-with-limits' | 'unavailable-in-opencode' | 'not-recommended'; @@ -11,48 +13,46 @@ export interface OpenCodeTeamModelRecommendation { } const PASSED_REAL_AGENT_TEAMS_E2E_REASON = - 'This exact model route passed real OpenCode Agent Teams E2E: launch, direct reply, and teammate-to-teammate relay.'; + 'This exact model route passed real OpenCode Agent Teams smoke E2E: launch, direct reply, and teammate-to-teammate relay.'; const PASSED_FREE_ROUTE_REAL_AGENT_TEAMS_E2E_REASON = - 'This exact free model route passed real OpenCode Agent Teams E2E, but free routes can still have capacity limits, rate limits, and variable latency.'; + 'This exact free model route passed real OpenCode Agent Teams smoke E2E, but free routes can still have capacity limits, rate limits, and variable latency.'; -const OPENCODE_TEAM_RECOMMENDED_MODELS = new Set([ +const PASSED_GAUNTLET_REAL_AGENT_TEAMS_E2E_REASON = + 'This exact model route passed the deeper OpenCode Agent Teams gauntlet: repeated launches, peer relays, concurrent deliveries, taskRefs, and transcript hygiene.'; + +const PASSED_GAUNTLET_WITH_LIMITS_REASON = + 'This exact model route passed the deeper OpenCode Agent Teams gauntlet, but has a production caveat such as free-route capacity, preview availability, cost, or latency variance.'; + +const OPENCODE_TEAM_RECOMMENDED_MODELS = new Set([]); + +const OPENCODE_TEAM_RECOMMENDED_WITH_LIMITS_MODELS = new Set([]); + +const OPENCODE_TEAM_TESTED_MODELS = new Set([ 'openrouter/anthropic/claude-haiku-4.5', - 'openrouter/anthropic/claude-opus-4.6', - 'openrouter/anthropic/claude-opus-4.7', - 'openrouter/anthropic/claude-sonnet-4.5', 'openrouter/anthropic/claude-sonnet-4.6', 'openrouter/google/gemini-2.5-flash', 'openrouter/google/gemini-3.1-flash-lite-preview', - 'openrouter/google/gemini-3.1-pro-preview', 'openrouter/google/gemini-3-flash-preview', + 'openrouter/google/gemma-4-26b-a4b-it', + 'openrouter/minimax/minimax-m2', + 'openrouter/minimax/minimax-m2.1', 'openrouter/minimax/minimax-m2.5', 'openrouter/minimax/minimax-m2.7', 'openrouter/moonshotai/kimi-k2.6', 'openrouter/mistralai/codestral-2508', - 'openrouter/mistralai/devstral-2512', - 'openrouter/mistralai/mistral-medium-3.1', - 'openrouter/openai/gpt-5.1', - 'openrouter/openai/gpt-5.1-codex', 'openrouter/openai/gpt-5.1-codex-mini', 'openrouter/openai/gpt-5.3-codex', - 'openrouter/openai/gpt-5.4', - 'openrouter/openai/gpt-5.4-mini', - 'openrouter/qwen/qwen3-max', - 'openrouter/qwen/qwen3-coder', - 'openrouter/qwen/qwen3-coder-flash', + 'openrouter/openai/gpt-5-nano', + 'openrouter/stepfun/step-3.5-flash', 'openrouter/x-ai/grok-4.1-fast', - 'openrouter/x-ai/grok-4-fast', 'openrouter/xiaomi/mimo-v2-pro', - 'openrouter/z-ai/glm-4.6', - 'openrouter/z-ai/glm-5', + 'openrouter/z-ai/glm-4.7', + 'openrouter/z-ai/glm-4.7-flash', 'openrouter/z-ai/glm-5.1', ]); -const OPENCODE_TEAM_RECOMMENDED_WITH_LIMITS_MODELS = new Set([ - 'opencode/minimax-m2.5-free', - 'openrouter/openai/gpt-oss-120b:free', -]); +const OPENCODE_TEAM_TESTED_WITH_LIMITS_MODELS = new Set(['opencode/minimax-m2.5-free']); const OPENCODE_TEAM_UNAVAILABLE_MODELS = new Map([ [ @@ -63,6 +63,110 @@ const OPENCODE_TEAM_UNAVAILABLE_MODELS = new Map([ 'openrouter/qwen/qwen3-coder-next', 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', ], + [ + 'openrouter/google/gemini-2.0-flash-lite-001', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-4.1-nano', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-4o-mini-2024-07-18', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-4o-mini-search-preview', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen-plus', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen-turbo', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-coder:free', + 'This free route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-next-80b-a3b-instruct:free', + 'This free route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-235b-a22b-2507', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-32b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-14b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-8b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwq-32b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/deepseek/deepseek-v3.2-exp', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/deepseek/deepseek-chat', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/mistral-nemo', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/mistral-small-24b-instruct-2501', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/mistral-small-3.1-24b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/mixtral-8x7b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/cohere/command-r7b-12-2024', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/cohere/command-r-08-2024', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/rekaai/reka-flash-3', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/rekaai/reka-edge', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/nvidia/nemotron-3-nano-30b-a3b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/minimax/minimax-01', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/nvidia/llama-3.3-nemotron-super-49b-v1.5', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], [ 'openrouter/qwen/qwen3-max-thinking', 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', @@ -71,10 +175,546 @@ const OPENCODE_TEAM_UNAVAILABLE_MODELS = new Map([ 'openrouter/mistralai/devstral-medium', 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', ], + [ + 'openrouter/mistralai/devstral-small', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], [ 'openrouter/mistralai/mistral-large-2512', 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', ], + [ + 'openrouter/mistralai/ministral-14b-2512', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/ministral-8b-2512', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/ministral-3b-2512', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3.5-35b-a3b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-30b-a3b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/baidu/ernie-4.5-21b-a3b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/nousresearch/hermes-4-70b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/minimax/minimax-m2-her', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/xiaomi/mimo-v2.5', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/xiaomi/mimo-v2.5-pro', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/x-ai/grok-4.20', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/google/gemini-3.1-flash-image-preview', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/z-ai/glm-5v-turbo', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/x-ai/grok-4.20-multi-agent', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/mistral-small-creative', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-5.3-chat', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/voxtral-small-24b-2507', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-5-chat', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen-2.5-72b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/tngtech/deepseek-r1t2-chimera', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/google/gemini-2.5-pro-preview', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/mistral-saba', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/mistral-large-2411', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-vl-30b-a3b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/inclusionai/ling-2.6-1t:free', + 'This free route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/inclusionai/ling-2.6-flash:free', + 'This free route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/meta-llama/llama-3.1-8b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen-2.5-7b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/amazon/nova-lite-v1', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/z-ai/glm-4-32b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/bytedance-seed/seed-1.6-flash', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/meta-llama/llama-4-scout', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/meta-llama/llama-3.3-70b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/bytedance-seed/seed-2.0-mini', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-vl-32b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/alibaba/tongyi-deepresearch-30b-a3b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/arcee-ai/trinity-large-preview', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/amazon/nova-micro-v1', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/arcee-ai/trinity-mini', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3.5-9b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/essentialai/rnj-1-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/upstage/solar-pro-3', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/allenai/olmo-3.1-32b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen-plus-2025-07-28', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/tencent/hy3-preview:free', + 'This free route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-vl-8b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/nex-agi/deepseek-v3.1-nex-n1', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/baidu/ernie-4.5-vl-28b-a3b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/thedrummer/rocinante-12b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/meta-llama/llama-3.1-70b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen-plus-2025-07-28:thinking', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/z-ai/glm-4.6v', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/anthropic/claude-3-haiku', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/bytedance-seed/seed-2.0-lite', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-235b-a22b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3.5-122b-a10b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/deepseek/deepseek-r1-0528', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/amazon/nova-2-lite-v1', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/o3-mini', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/mistral-large-2407', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/thedrummer/unslopnemo-12b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-vl-235b-a22b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-vl-8b-thinking', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/kwaipilot/kat-coder-pro-v2', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/o4-mini-high', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/o3-mini-high', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-4o', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/cohere/command-r-plus-08-2024', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-vl-30b-a3b-thinking', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/sao10k/l3.1-euryale-70b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3.5-27b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/arcee-ai/virtuoso-large', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-3.5-turbo', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/bytedance-seed/seed-1.6', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/nvidia/llama-3.1-nemotron-70b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen-vl-max', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen3-vl-235b-a22b-thinking', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-audio-mini', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/amazon/nova-pro-v1', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/relace/relace-search', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen-max', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/pixtral-large-2411', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/mixtral-8x22b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-3.5-turbo-16k', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/mistralai/mistral-large', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/o4-mini-deep-research', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/ai21/jamba-large-1.7', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/o3', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-audio', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-4o-audio-preview', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-4o-2024-11-20', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-4o-2024-08-06', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/amazon/nova-premier-v1', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/anthropic/claude-3.7-sonnet:thinking', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-4o-2024-05-13', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/~anthropic/claude-opus-latest', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-4-turbo', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-4-turbo-preview', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-4-1106-preview', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/o3-deep-research', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/o1', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/o3-pro', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/anthropic/claude-opus-4.6-fast', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openai/gpt-5.5-pro', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openrouter/auto', + 'This aggregator route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openrouter/pareto-code', + 'This aggregator route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/openrouter/bodybuilder', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/baidu/qianfan-ocr-fast:free', + 'This free route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/google/lyria-3-pro-preview', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/ibm-granite/granite-4.0-h-micro', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/microsoft/phi-4', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/meta-llama/llama-guard-4-12b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/qwen/qwen-vl-plus', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/deepseek/deepseek-r1-distill-qwen-32b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/google/lyria-3-clip-preview', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/liquid/lfm-2-24b-a2b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/meta-llama/llama-3.2-1b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/bytedance/ui-tars-1.5-7b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/baidu/ernie-4.5-21b-a3b-thinking', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/arcee-ai/spotlight', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/meta-llama/llama-3.2-3b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/meta-llama/llama-guard-3-8b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/nousresearch/hermes-3-llama-3.1-70b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/allenai/olmo-3-32b-think', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/tencent/hunyuan-a13b-instruct', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/meta-llama/llama-4-maverick', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/nvidia/nemotron-nano-12b-v2-vl', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/thedrummer/cydonia-24b-v4.1', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/microsoft/wizardlm-2-8x22b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/arcee-ai/coder-large', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/thedrummer/skyfall-36b-v2', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/baidu/ernie-4.5-300b-a47b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], + [ + 'openrouter/sao10k/l3.3-euryale-70b', + 'This route exists in OpenRouter, but was not found in the live OpenCode provider catalog used for Agent Teams launch.', + ], ]); const OPENCODE_TEAM_NOT_RECOMMENDED_MODELS = new Map([ @@ -90,6 +730,14 @@ const OPENCODE_TEAM_NOT_RECOMMENDED_MODELS = new Map([ 'openrouter/google/gemini-2.5-pro', 'Real OpenCode Agent Teams E2E passed direct reply but failed peer relay.', ], + [ + 'openrouter/anthropic/claude-opus-4.6', + 'Fresh OpenCode Agent Teams gauntlet launched but timed out already on direct reply.', + ], + [ + 'openrouter/anthropic/claude-opus-4.7', + 'Fresh OpenCode Agent Teams gauntlet failed OpenCode launch readiness.', + ], [ 'openrouter/google/gemini-2.5-flash-lite', 'Real OpenCode Agent Teams E2E passed direct reply but failed peer relay with plain/control-character output instead of MCP message_send.', @@ -98,10 +746,26 @@ const OPENCODE_TEAM_NOT_RECOMMENDED_MODELS = new Map([ 'openrouter/google/gemini-3-pro-preview', 'OpenRouter reported no runnable endpoints for this model during execution verification.', ], + [ + 'openrouter/google/gemini-3.1-pro-preview', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness before producing usable Agent Teams behavior.', + ], + [ + 'openrouter/google/gemini-3.1-pro-preview-customtools', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because production Agent Teams prompts exceeded the route/key token allowance.', + ], + [ + 'openrouter/google/gemini-2.5-pro-preview-05-06', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], [ 'openrouter/deepseek/deepseek-v3.2', 'Real OpenCode Agent Teams E2E passed direct reply but failed peer relay after treating Agent Teams MCP tools as unavailable.', ], + [ + 'openrouter/deepseek/deepseek-v3.2-speciale', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this route.', + ], [ 'openrouter/meta-llama/llama-3.3-70b-instruct:free', 'Execution verification timed out before Agent Teams launch could proceed.', @@ -110,22 +774,110 @@ const OPENCODE_TEAM_NOT_RECOMMENDED_MODELS = new Map([ 'openrouter/minimax/minimax-m2.5:free', 'This OpenRouter free route for MiniMax M2.5 passed direct reply but failed teammate-to-teammate relay. The non-free OpenRouter route and the OpenCode free alias are tracked separately.', ], + [ + 'openrouter/minimax/minimax-m1', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because production Agent Teams prompts exceeded the route/key token allowance.', + ], [ 'openrouter/moonshotai/kimi-k2-thinking', 'Real OpenCode Agent Teams E2E failed during launch reconciliation with an aborted assistant message.', ], + [ + 'openrouter/moonshotai/kimi-k2.5', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/moonshotai/kimi-k2-0905', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/moonshotai/kimi-k2', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], [ 'openrouter/openai/gpt-5.2-codex', 'Real OpenCode Agent Teams E2E failed launch readiness because model verification timed out.', ], + [ + 'openrouter/openai/gpt-5.2', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/openai/gpt-5-mini', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/openai/gpt-5-codex', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because production Agent Teams prompts exceeded the route/key token allowance.', + ], + [ + 'openrouter/openai/gpt-5.5', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], [ 'openrouter/openai/gpt-5.1-chat', 'Real OpenCode Agent Teams E2E passed direct reply but failed peer relay by delegating to the lead instead of messaging the requested teammate.', ], + [ + 'openrouter/openai/gpt-5.1', + 'Fresh OpenCode Agent Teams gauntlet passed direct and peer relays but failed concurrent delivery, taskRefs, and duplicate-token correctness.', + ], + [ + 'openrouter/openai/gpt-5.1-codex', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness during model verification.', + ], + [ + 'openrouter/openai/gpt-5.4-mini', + 'Fresh OpenCode Agent Teams gauntlet passed direct and peer relays but failed concurrent delivery, taskRefs, and duplicate-token correctness.', + ], + [ + 'openrouter/openai/gpt-5.4', + 'Fresh OpenCode Agent Teams gauntlet passed direct and peer relays but failed concurrent delivery, taskRefs, and duplicate-token correctness.', + ], [ 'openrouter/openai/gpt-oss-20b:free', 'Execution verification passed, but real Agent Teams E2E produced fake tool text instead of MCP message_send.', ], + [ + 'openrouter/openai/gpt-oss-120b:free', + 'Fresh OpenCode Agent Teams gauntlet passed direct reply but timed out on peer relay and later Agent Teams stages.', + ], + [ + 'openrouter/google/gemma-4-26b-a4b-it:free', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness during model verification.', + ], + [ + 'openrouter/google/gemma-4-31b-it:free', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness during model verification.', + ], + [ + 'openrouter/nvidia/nemotron-3-super-120b-a12b:free', + 'Fresh OpenCode Agent Teams gauntlet passed direct reply but failed peer relay and later Agent Teams stages.', + ], + [ + 'openrouter/nvidia/nemotron-3-nano-30b-a3b:free', + 'Fresh OpenCode Agent Teams gauntlet passed direct reply but failed peer relay and later Agent Teams stages.', + ], + [ + 'openrouter/nvidia/nemotron-nano-12b-v2-vl:free', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because model verification timed out.', + ], + [ + 'openrouter/nvidia/nemotron-nano-9b-v2:free', + 'Fresh OpenCode Agent Teams gauntlet passed direct reply but failed peer relay through OpenCode tool usage.', + ], + [ + 'openrouter/google/gemma-3-12b-it', + 'Fresh OpenCode Agent Teams gauntlet launched but timed out already on direct reply.', + ], + [ + 'openrouter/openai/gpt-oss-safeguard-20b', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/google/gemini-2.5-flash-lite-preview-09-2025', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], [ 'openrouter/openrouter/free', 'Aggregator routing was unstable in real Agent Teams E2E and timed out during peer relay.', @@ -138,12 +890,332 @@ const OPENCODE_TEAM_NOT_RECOMMENDED_MODELS = new Map([ 'openrouter/z-ai/glm-4.5-air:free', 'Real OpenCode Agent Teams E2E was slow and failed peer relay with empty assistant turns.', ], + [ + 'openrouter/z-ai/glm-4.5', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because production Agent Teams prompts exceeded the route/key token allowance.', + ], + [ + 'openrouter/z-ai/glm-5-turbo', + 'Real OpenCode Agent Teams gauntlet passed direct reply but failed peer relay and concurrent delivery.', + ], + [ + 'openrouter/qwen/qwen3.5-flash-02-23', + 'Real OpenCode Agent Teams gauntlet completed the flow but failed duplicate-token correctness.', + ], + [ + 'openrouter/qwen/qwen3-coder-flash', + 'Fresh OpenCode Agent Teams gauntlet passed direct and first relay, then failed chained and concurrent reply flow.', + ], + [ + 'openrouter/qwen/qwen3-coder', + 'Fresh OpenCode Agent Teams gauntlet failed concurrent delivery, taskRefs, and protocol cleanliness.', + ], + [ + 'openrouter/qwen/qwen3.6-plus', + 'Real OpenCode Agent Teams gauntlet completed most stages but failed taskRefs and duplicate-token correctness.', + ], + [ + 'openrouter/qwen/qwen3.5-plus-02-15', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/qwen/qwen3.5-397b-a17b', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/qwen/qwen3-30b-a3b-instruct-2507', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/qwen/qwen3-30b-a3b-thinking-2507', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/inception/mercury-2', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because production Agent Teams prompts exceeded the route/key token allowance.', + ], + [ + 'openrouter/prime-intellect/intellect-3', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because production Agent Teams prompts exceeded the route/key token allowance.', + ], + [ + 'openrouter/openai/gpt-4.1-mini', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because production Agent Teams prompts exceeded the route/key token allowance.', + ], + [ + 'openrouter/openai/o4-mini', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance even at the reduced verification cap.', + ], + [ + 'openrouter/arcee-ai/trinity-large-thinking', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because production Agent Teams prompts exceeded the route/key token allowance.', + ], + [ + 'openrouter/qwen/qwen3-235b-a22b-thinking-2507', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because production Agent Teams prompts exceeded the route/key token allowance.', + ], + [ + 'openrouter/xiaomi/mimo-v2-omni', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because production Agent Teams prompts exceeded the route/key token allowance.', + ], + [ + 'openrouter/deepseek/deepseek-r1', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because production Agent Teams prompts exceeded the route/key token allowance.', + ], + [ + 'openrouter/openai/gpt-4.1', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance even at the reduced verification cap.', + ], + [ + 'openrouter/openai/gpt-5', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance even at the reduced verification cap.', + ], + [ + 'openrouter/openai/gpt-5.1-codex-max', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/openai/gpt-5.2-chat', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/x-ai/grok-3', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/anthropic/claude-sonnet-4', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/x-ai/grok-3-beta', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/anthropic/claude-opus-4.1', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/anthropic/claude-opus-4', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/openai/gpt-5-pro', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/openai/gpt-5.2-pro', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/openai/gpt-5.4-pro', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/liquid/lfm-2.5-1.2b-thinking:free', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this free route.', + ], + [ + 'openrouter/liquid/lfm-2.5-1.2b-instruct:free', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this free route.', + ], + [ + 'openrouter/google/gemma-3-4b-it:free', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this free route.', + ], + [ + 'openrouter/meta-llama/llama-3.2-3b-instruct:free', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this free route.', + ], + [ + 'openrouter/google/gemma-3-27b-it:free', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this free route.', + ], + [ + 'openrouter/nousresearch/hermes-3-llama-3.1-405b:free', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this free route.', + ], + [ + 'openrouter/google/gemma-3-4b-it', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this route.', + ], + [ + 'openrouter/cognitivecomputations/dolphin-mistral-24b-venice-edition:free', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this free route.', + ], + [ + 'openrouter/google/gemma-3-12b-it:free', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this free route.', + ], + [ + 'openrouter/meta-llama/llama-3.2-11b-vision-instruct', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this route.', + ], + [ + 'openrouter/qwen/qwen2.5-vl-72b-instruct', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this route.', + ], + [ + 'openrouter/deepseek/deepseek-r1-distill-llama-70b', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this route.', + ], + [ + 'openrouter/z-ai/glm-4.5v', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because production Agent Teams prompts exceeded the route/key token allowance.', + ], + [ + 'openrouter/qwen/qwen3-max', + 'Fresh OpenCode Agent Teams gauntlet passed direct and peer relays but failed concurrent delivery, taskRefs, and duplicate-token correctness.', + ], + [ + 'openrouter/xiaomi/mimo-v2-flash', + 'Real OpenCode Agent Teams gauntlet failed concurrent reply delivery and taskRefs for one teammate.', + ], + [ + 'openrouter/mistralai/mistral-small-2603', + 'Real OpenCode Agent Teams gauntlet passed direct and first relay but failed chained/concurrent relay.', + ], + [ + 'openrouter/deepseek/deepseek-v4-pro', + 'Real OpenCode Agent Teams gauntlet passed direct reply but timed out on peer relay and concurrent delivery.', + ], + [ + 'openrouter/x-ai/grok-4', + 'Real OpenCode Agent Teams gauntlet passed direct reply but failed peer relay with an empty assistant turn.', + ], + [ + 'openrouter/google/gemini-2.0-flash-001', + 'Real OpenCode Agent Teams gauntlet passed direct and first relay but failed chained/concurrent relay.', + ], + [ + 'openrouter/google/gemma-3n-e4b-it', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this route.', + ], + [ + 'openrouter/google/gemma-3n-e2b-it:free', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this free route.', + ], + [ + 'openrouter/deepseek/deepseek-chat-v3.1', + 'Real OpenCode Agent Teams gauntlet failed concurrent replies, taskRefs, and duplicate-token correctness.', + ], + [ + 'openrouter/z-ai/glm-4.5-air', + 'Real OpenCode Agent Teams gauntlet passed direct reply but failed peer relay through OpenCode tool handling.', + ], + [ + 'openrouter/anthropic/claude-sonnet-4.5', + 'Fresh OpenCode Agent Teams gauntlet hit concurrent reply, taskRefs, and protocol failure; keep below Tested until rerun passes.', + ], + [ + 'openrouter/anthropic/claude-opus-4.5', + 'Fresh OpenCode Agent Teams gauntlet hit concurrent reply, taskRefs, and protocol failure; keep below Tested until rerun passes.', + ], + [ + 'openrouter/anthropic/claude-3.5-haiku', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because production Agent Teams prompts exceeded the route/key token allowance.', + ], + [ + 'openrouter/anthropic/claude-3.7-sonnet', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/mistralai/mistral-medium-3', + 'Real OpenCode Agent Teams gauntlet timed out already on direct reply.', + ], + [ + 'openrouter/mistralai/mistral-medium-3.1', + 'Fresh OpenCode Agent Teams gauntlet passed direct reply but timed out on peer relay and later Agent Teams stages.', + ], + [ + 'openrouter/mistralai/devstral-2512', + 'Fresh OpenCode Agent Teams gauntlet failed OpenCode launch readiness.', + ], + [ + 'openrouter/x-ai/grok-3-mini', + 'Real OpenCode Agent Teams gauntlet passed direct reply but failed peer relay and concurrent delivery.', + ], + [ + 'openrouter/x-ai/grok-3-mini-beta', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/qwen/qwen3-next-80b-a3b-instruct', + 'Real OpenCode Agent Teams gauntlet passed direct reply but timed out on peer relay through OpenCode.', + ], + [ + 'openrouter/z-ai/glm-4.6', + 'Fresh OpenCode Agent Teams gauntlet regressed versus older smoke: direct reply passed, but peer relay failed.', + ], + [ + 'openrouter/openai/gpt-oss-120b', + 'Real OpenCode Agent Teams gauntlet passed direct and first relay but failed chained relay, concurrent delivery, transcript hygiene, and duplicate-token correctness.', + ], + [ + 'openrouter/qwen/qwen3-coder-30b-a3b-instruct', + 'Real OpenCode Agent Teams gauntlet passed direct and peer relays but failed concurrent delivery, taskRefs, and duplicate-token correctness.', + ], + [ + 'openrouter/qwen/qwen-2.5-coder-32b-instruct', + 'Fresh OpenCode Agent Teams gauntlet failed launch readiness because OpenRouter reported no tool-use endpoint for this route.', + ], + [ + 'openrouter/x-ai/grok-4-fast', + 'Fresh OpenCode Agent Teams gauntlet passed direct and peer relays but failed concurrent delivery, taskRefs, transcript hygiene, and duplicate-token correctness.', + ], + [ + 'openrouter/openai/gpt-5.4-nano', + 'Real OpenCode Agent Teams gauntlet passed direct and peer relays but failed concurrent delivery, taskRefs, and duplicate-token correctness.', + ], + [ + 'openrouter/nvidia/nemotron-3-super-120b-a12b', + 'Real OpenCode Agent Teams gauntlet passed direct reply but failed peer relay, concurrent delivery, transcript hygiene, and latency stability.', + ], + [ + 'openrouter/mistralai/mistral-small-3.2-24b-instruct', + 'Real OpenCode Agent Teams gauntlet passed direct and peer relays but failed concurrent delivery, taskRefs, and duplicate-token correctness.', + ], + [ + 'openrouter/google/gemma-4-31b-it', + 'Real OpenCode Agent Teams gauntlet launched but timed out already on direct reply.', + ], + [ + 'openrouter/openai/gpt-oss-20b', + 'Real OpenCode Agent Teams gauntlet launched but timed out already on direct reply.', + ], + [ + 'openrouter/google/gemma-3-27b-it', + 'Real OpenCode Agent Teams gauntlet launched but timed out already on direct reply.', + ], + [ + 'openrouter/openai/gpt-4o-mini', + 'Fresh OpenCode Agent Teams gauntlet passed direct and peer relays but failed concurrent delivery, taskRefs, and duplicate-token correctness.', + ], + [ + 'openrouter/deepseek/deepseek-chat-v3-0324', + 'Real OpenCode Agent Teams gauntlet passed direct reply but failed peer relay, concurrent delivery, transcript hygiene, and latency stability.', + ], + [ + 'openrouter/deepseek/deepseek-v3.1-terminus', + 'Fresh OpenCode Agent Teams gauntlet was blocked during launch by OpenRouter key/max_tokens allowance under production Agent Teams prompts.', + ], + [ + 'openrouter/qwen/qwen3-next-80b-a3b-thinking', + 'Real OpenCode Agent Teams gauntlet launched but timed out already on direct reply.', + ], + [ + 'openrouter/nvidia/nemotron-nano-9b-v2', + 'Real OpenCode Agent Teams gauntlet passed direct reply but failed peer relay, concurrent delivery, transcript hygiene, and latency stability.', + ], ]); function normalizeOpenCodeTeamModelId(modelId: string | null | undefined): string { return modelId?.trim().toLowerCase() ?? ''; } +function isOpenCodeProviderAllowanceReason(reason: string): boolean { + return /(?:allowance|max_tokens|max-token)/i.test(reason); +} + export function getOpenCodeTeamModelRecommendation( modelId: string | null | undefined ): OpenCodeTeamModelRecommendation | null { @@ -156,7 +1228,7 @@ export function getOpenCodeTeamModelRecommendation( return { level: 'recommended', label: 'Recommended', - reason: PASSED_REAL_AGENT_TEAMS_E2E_REASON, + reason: PASSED_GAUNTLET_REAL_AGENT_TEAMS_E2E_REASON, }; } @@ -164,6 +1236,22 @@ export function getOpenCodeTeamModelRecommendation( return { level: 'recommended-with-limits', label: 'Recommended with limits', + reason: PASSED_GAUNTLET_WITH_LIMITS_REASON, + }; + } + + if (OPENCODE_TEAM_TESTED_MODELS.has(normalizedModelId)) { + return { + level: 'tested', + label: 'Tested', + reason: PASSED_REAL_AGENT_TEAMS_E2E_REASON, + }; + } + + if (OPENCODE_TEAM_TESTED_WITH_LIMITS_MODELS.has(normalizedModelId)) { + return { + level: 'tested-with-limits', + label: 'Tested with limits', reason: PASSED_FREE_ROUTE_REAL_AGENT_TEAMS_E2E_REASON, }; } @@ -179,6 +1267,10 @@ export function getOpenCodeTeamModelRecommendation( const notRecommendedReason = OPENCODE_TEAM_NOT_RECOMMENDED_MODELS.get(normalizedModelId); if (notRecommendedReason) { + if (isOpenCodeProviderAllowanceReason(notRecommendedReason)) { + return null; + } + return { level: 'not-recommended', label: 'Not recommended', @@ -206,13 +1298,19 @@ export function getOpenCodeTeamModelRecommendationSortRank( if (recommendation?.level === 'recommended-with-limits') { return 1; } - if (recommendation?.level === 'unavailable-in-opencode') { + if (recommendation?.level === 'tested') { + return 2; + } + if (recommendation?.level === 'tested-with-limits') { return 3; } - if (recommendation?.level === 'not-recommended') { - return 4; + if (recommendation?.level === 'unavailable-in-opencode') { + return 5; } - return 2; + if (recommendation?.level === 'not-recommended') { + return 6; + } + return 4; } export function compareOpenCodeTeamModelRecommendations( diff --git a/test/main/services/team/OpenCodeSemanticMessaging.live.test.ts b/test/main/services/team/OpenCodeSemanticMessaging.live.test.ts index 546ccc7d..61df3548 100644 --- a/test/main/services/team/OpenCodeSemanticMessaging.live.test.ts +++ b/test/main/services/team/OpenCodeSemanticMessaging.live.test.ts @@ -48,6 +48,7 @@ liveDescribe('OpenCode semantic messaging live e2e', () => { const { bridgeClient, selectedModel, svc, dispose } = await createOpenCodeLiveHarness({ tempDir, selectedModel: process.env.OPENCODE_E2E_MODEL?.trim() || DEFAULT_MODEL, + projectPath: PROJECT_PATH, }); const teamName = `opencode-semantic-message-${Date.now()}`; @@ -163,6 +164,7 @@ liveDescribe('OpenCode semantic messaging live e2e', () => { const { bridgeClient, selectedModel, svc, dispose } = await createOpenCodeLiveHarness({ tempDir, selectedModel: process.env.OPENCODE_E2E_MODEL?.trim() || DEFAULT_MODEL, + projectPath: PROJECT_PATH, }); const teamName = `opencode-peer-message-${Date.now()}`; diff --git a/test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts b/test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts new file mode 100644 index 00000000..59566fd9 --- /dev/null +++ b/test/main/services/team/OpenCodeSemanticModelGauntlet.live.test.ts @@ -0,0 +1,2113 @@ +import { promises as fs } from 'fs'; +import * as os from 'os'; +import * as path from 'path'; + +import { describe, expect, it, vi } from 'vitest'; + +import { + getTeamsBasePath, + setClaudeBasePathOverride, +} from '../../../../src/main/utils/pathDecoder'; +import { + buildOpenCodeScenarioTeamRequest, + loadOpenCodeSemanticScenario, + materializeOpenCodeScenarioProject, + materializeOpenCodeScenarioTasks, + parseOpenCodeE2EModelList, + taskRefForScenario, + type OpenCodeSemanticScenario, +} from './openCodeSemanticScenarioHarness'; +import { + createOpenCodeLiveHarness, + getRuntimeTranscript, + readInboxMessages, + waitForMemberInboxMessage, + waitForOpenCodeLanesStopped, + waitForOpenCodeMemberIdle, + waitForOpenCodePeerRelay, + waitForUserInboxReply, + type InboxMessage, +} from './openCodeLiveTestHarness'; + +import type { + AgentActionMode, + TaskRef, + TeamProvisioningProgress, +} from '../../../../src/shared/types'; + +const liveDescribe = + process.env.OPENCODE_E2E === '1' && process.env.OPENCODE_E2E_SEMANTIC_MODEL_GAUNTLET === '1' + ? describe + : describe.skip; + +interface GauntletReport { + generatedAt: string; + runsPerModel: number; + qualification: { + minimumAverageScore: number; + minimumSuccessfulRuns: number; + minimumConsistencyScore: number; + requireNoHardFailures: boolean; + }; + models: ModelGauntletResult[]; +} + +interface ModelGauntletResult { + model: string; + verdict: ModelGauntletVerdict; + confidence: ModelGauntletConfidence; + qualified: boolean; + readinessScore: number; + consistencyScore: number; + averageScore: number; + behavioralAverageScore: number | null; + minScore: number; + successfulRuns: number; + countedRuns: number; + hardFailures: number; + providerInfraFailures: number; + runtimeTransportFailures: number; + modelBehaviorFailures: number; + harnessFailures: number; + p50DurationMs: number; + p95DurationMs: number; + stagePassRates: Record; + taskRefPassRates: Record; + protocolViolationTotals: ProtocolViolationTotals; + stageFailureImpact: StageFailureImpactSummary[]; + scoreStability: ScoreStabilitySummary; + dominantFailureCategory: RunFailureCategory | 'mixed'; + recommendationBlockers: string[]; + runs: RunGauntletResult[]; +} + +interface RatioSummary { + passed: number; + total: number; + rate: number | null; +} + +interface ProtocolViolationTotals { + badMessages: number; + duplicateOrMissingTokens: number; + affectedRuns: number; +} + +interface StageFailureImpactSummary { + stage: GauntletStageName; + failedRuns: number; + weightedLoss: number; + passRate: RatioSummary; +} + +interface ScoreStabilitySummary { + sampleSize: number; + minScore: number; + maxScore: number; + spread: number; + standardDeviation: number; + consistencyScore: number; +} + +type ModelGauntletVerdict = + | 'recommended' + | 'strong-candidate' + | 'tested-only' + | 'infra-blocked' + | 'inconclusive'; + +type ModelGauntletConfidence = 'high' | 'medium' | 'low' | 'blocked'; + +type RunFailureCategory = + | 'none' + | 'model-behavior' + | 'runtime-transport' + | 'provider-infra' + | 'harness-error'; + +type RunOutcome = + | 'passed' + | 'behavioral-fail' + | 'runtime-transport-fail' + | 'provider-infra-blocked' + | 'harness-error'; + +interface RunGauntletResult { + runIndex: number; + passed: boolean; + score: number; + countedForRecommendation: boolean; + outcome: RunOutcome; + failureCategory: RunFailureCategory; + primaryFailure: string | null; + durationMs: number; + hardFailure: boolean; + stageDurationsMs: Record; + stageFailures: Record; + taskRefChecks: Record; + protocolViolations: { + badMessages: number; + duplicateOrMissingTokens: string[]; + }; + stages: { + launchBootstrap: boolean; + directReply: boolean; + peerRelayAB: boolean; + peerRelayBC: boolean; + concurrentReplies: boolean; + taskRefs: boolean; + cleanTranscript: boolean; + noDuplicateTokens: boolean; + latencyStable: boolean; + }; + diagnostics: string[]; +} + +interface DeliveryExpectation { + memberName: string; + messageId: string; + replyRecipient: string; + actionMode: AgentActionMode; + taskIndex: number; + text: string; + expectedReplyToken: string; +} + +interface PeerExpectation { + senderName: string; + recipientName: string; + messageId: string; + taskIndex: number; + peerToken: string; + expectedReplyToken: string; + text: string; +} + +type GauntletStageName = keyof RunGauntletResult['stages']; + +const SCORE_WEIGHTS: Record = { + launchBootstrap: 15, + directReply: 10, + peerRelayAB: 15, + peerRelayBC: 15, + concurrentReplies: 15, + taskRefs: 10, + cleanTranscript: 10, + noDuplicateTokens: 5, + latencyStable: 5, +}; + +const GAUNTLET_STAGE_NAMES = Object.keys(SCORE_WEIGHTS) as GauntletStageName[]; + +const TASK_REF_CHECK_NAMES = [ + 'directReply', + 'peerRelayAB', + 'peerRelayBC', + 'concurrentBob', + 'concurrentTom', +] as const; + +liveDescribe('OpenCode semantic model gauntlet live e2e', () => { + it( + 'stress-tests selected OpenCode models and writes a markdown scorecard', + async () => { + const scenario = buildGauntletScenario(await loadOpenCodeSemanticScenario()); + const models = parseOpenCodeE2EModelList(); + const runsPerModel = parsePositiveInteger(process.env.OPENCODE_E2E_GAUNTLET_RUNS, 3); + const minimumAverageScore = parsePositiveInteger( + process.env.OPENCODE_E2E_GAUNTLET_MIN_AVERAGE_SCORE, + 90 + ); + const minimumSuccessfulRuns = parsePositiveInteger( + process.env.OPENCODE_E2E_GAUNTLET_MIN_SUCCESSFUL_RUNS, + Math.max(3, runsPerModel) + ); + const minimumConsistencyScore = parsePositiveInteger( + process.env.OPENCODE_E2E_GAUNTLET_MIN_CONSISTENCY_SCORE, + 85 + ); + const report: GauntletReport = { + generatedAt: new Date().toISOString(), + runsPerModel, + qualification: { + minimumAverageScore, + minimumSuccessfulRuns, + minimumConsistencyScore, + requireNoHardFailures: true, + }, + models: [], + }; + + for (const model of models) { + report.models.push( + await runModelGauntlet({ + scenario, + model, + runsPerModel, + minimumAverageScore, + minimumSuccessfulRuns, + minimumConsistencyScore, + }) + ); + await writeGauntletReport(report); + } + + await writeGauntletReport(report); + clearExpectedOpenCodeGauntletConsoleNoise(); + + if (process.env.OPENCODE_E2E_GAUNTLET_REQUIRE_RECOMMENDED === '1') { + const notQualified = report.models.filter((model) => !model.qualified); + expect(notQualified, JSON.stringify(report, null, 2)).toEqual([]); + } else { + expect(report.models.length).toBeGreaterThan(0); + } + }, + Math.max( + 900_000, + parseOpenCodeE2EModelList().length * + parsePositiveInteger(process.env.OPENCODE_E2E_GAUNTLET_RUNS, 3) * + 600_000 + ) + ); +}); + +describe('OpenCode semantic model gauntlet report helpers', () => { + it('separates provider infrastructure failures from model behavior', () => { + const stages = createPassingStages({ launchBootstrap: false }); + const diagnostics = ['openrouter/google/gemini-3.1-pro-preview: key total limit exceeded']; + const category = classifyGauntletFailure({ diagnostics, stages }); + + expect(category).toBe('provider-infra'); + expect(isCountedForRecommendation(category)).toBe(false); + expect(getRunOutcome(category)).toBe('provider-infra-blocked'); + }); + + it('does not promote a single perfect run to Recommended', () => { + const qualified = isModelQualified({ + averageScore: 100, + successfulRuns: 1, + minimumAverageScore: 90, + minimumSuccessfulRuns: 3, + minimumConsistencyScore: 85, + hardFailures: 0, + providerInfraFailures: 0, + harnessFailures: 0, + consistencyScore: 100, + }); + const verdict = getModelGauntletVerdict({ + qualified, + behavioralAverageScore: 100, + successfulRuns: 1, + countedRuns: 1, + totalRuns: 1, + providerInfraFailures: 0, + runtimeTransportFailures: 0, + modelBehaviorFailures: 0, + harnessFailures: 0, + }); + const confidence = getModelGauntletConfidence({ + countedRuns: 1, + providerInfraFailures: 0, + harnessFailures: 0, + }); + + expect(qualified).toBe(false); + expect(verdict).toBe('strong-candidate'); + expect(confidence).toBe('low'); + }); + + it('does not mark failed high-score runs as strong candidates', () => { + const verdict = getModelGauntletVerdict({ + qualified: false, + behavioralAverageScore: 85, + successfulRuns: 0, + countedRuns: 1, + totalRuns: 1, + providerInfraFailures: 0, + runtimeTransportFailures: 0, + modelBehaviorFailures: 1, + harnessFailures: 0, + }); + + expect(verdict).toBe('tested-only'); + }); + + it('renders weakest stages, taskRef rates, and protocol totals', () => { + const run = createTestGauntletRun({ + stages: createPassingStages({ + peerRelayAB: false, + taskRefs: false, + }), + taskRefChecks: { + directReply: true, + peerRelayAB: false, + peerRelayBC: true, + concurrentBob: true, + concurrentTom: null, + }, + protocolViolations: { + badMessages: 1, + duplicateOrMissingTokens: ['GAUNTLET_JACK_USER_OK_1'], + }, + primaryFailure: 'peer relay timed out', + }); + const markdown = renderGauntletMarkdown({ + generatedAt: '2026-04-26T00:00:00.000Z', + runsPerModel: 1, + qualification: { + minimumAverageScore: 90, + minimumSuccessfulRuns: 3, + minimumConsistencyScore: 85, + requireNoHardFailures: true, + }, + models: [createTestGauntletModel({ runs: [run] })], + }); + + expect(markdown).toContain('Weakest Stage'); + expect(markdown).toContain('Weakest TaskRef'); + expect(markdown).toContain('Provider-infra runs are reported separately'); + expect(markdown).toContain('Scoring weights: launchBootstrap=15'); + expect(markdown).toContain('Readiness score:'); + expect(markdown).toContain('Recommendation blockers:'); + expect(markdown).toContain('Weighted stage impact:'); + expect(markdown).toContain('Stage pass rates:'); + expect(markdown).toContain('TaskRef pass rates:'); + expect(markdown).toContain('peerRelayAB 0/1 (0%)'); + expect(markdown).toContain('Protocol totals: badMessages=1'); + }); + + it('ranks failure impact by lost weighted score instead of raw failure count only', () => { + const model = createTestGauntletModel({ + runs: [ + createTestGauntletRun({ + stages: createPassingStages({ + peerRelayAB: false, + noDuplicateTokens: false, + }), + }), + createTestGauntletRun({ + stages: createPassingStages({ + noDuplicateTokens: false, + }), + }), + ], + }); + + expect(model.stageFailureImpact[0]).toMatchObject({ + stage: 'peerRelayAB', + failedRuns: 1, + weightedLoss: 15, + }); + expect(model.stageFailureImpact[1]).toMatchObject({ + stage: 'noDuplicateTokens', + failedRuns: 2, + weightedLoss: 10, + }); + expect(model.recommendationBlockers).toContain('highest weighted stage loss peerRelayAB=15'); + }); + + it('keeps infra-only runs out of readiness ranking', () => { + const model = createTestGauntletModel({ + runs: [ + createTestGauntletRun({ + failureCategory: 'provider-infra', + countedForRecommendation: false, + outcome: 'provider-infra-blocked', + hardFailure: false, + passed: false, + stages: createPassingStages({ launchBootstrap: false }), + diagnostics: ['OpenRouter key total limit exceeded'], + }), + ], + }); + + expect(model.verdict).toBe('infra-blocked'); + expect(model.confidence).toBe('blocked'); + expect(model.readinessScore).toBe(0); + expect(model.recommendationBlockers).toContain('provider-infra failures 1'); + }); + + it('blocks recommendation when repeated runs are too inconsistent', () => { + const perfectRun = createTestGauntletRun({ + runIndex: 1, + stages: createPassingStages(), + score: 100, + failureCategory: 'none', + passed: true, + hardFailure: false, + }); + const unstableButNonHardRun = createTestGauntletRun({ + runIndex: 2, + stages: createPassingStages({ + latencyStable: false, + }), + score: 80, + failureCategory: 'none', + passed: true, + hardFailure: false, + }); + const model = createTestGauntletModel({ + runs: [perfectRun, perfectRun, unstableButNonHardRun], + }); + + expect(model.successfulRuns).toBe(3); + expect(model.averageScore).toBeGreaterThanOrEqual(90); + expect(model.qualified).toBe(false); + expect(model.consistencyScore).toBeLessThan(85); + expect(model.recommendationBlockers).toContain( + `consistency score ${model.consistencyScore} < 85` + ); + }); +}); + +function clearExpectedOpenCodeGauntletConsoleNoise(): void { + const warn = vi.mocked(console.warn); + const expectedWarning = + 'OpenCode prompt delivery watchdog job failed: OpenCode prompt delivery record not found'; + if (warn.mock) { + removeMatchingMockCalls(warn.mock.calls, [expectedWarning, 'OpenCode inbox relay failed']); + } + + const error = vi.mocked(console.error); + if (error.mock) { + removeMatchingMockCalls(error.mock.calls, [ + 'Rejected runtime evidence without current run: delivery_call', + 'Rejected runtime evidence without current run: heartbeat', + 'Rejected stale runtime evidence: bootstrap_checkin', + ]); + } +} + +function removeMatchingMockCalls(calls: unknown[][], expectedSubstrings: string[]): void { + for (let index = calls.length - 1; index >= 0; index -= 1) { + const rendered = calls[index]?.map((arg) => String(arg)).join(' ') ?? ''; + if (expectedSubstrings.some((substring) => rendered.includes(substring))) { + calls.splice(index, 1); + } + } +} + +async function runModelGauntlet(input: { + scenario: OpenCodeSemanticScenario; + model: string; + runsPerModel: number; + minimumAverageScore: number; + minimumSuccessfulRuns: number; + minimumConsistencyScore: number; +}): Promise { + const runs: RunGauntletResult[] = []; + for (let runIndex = 1; runIndex <= input.runsPerModel; runIndex += 1) { + runs.push( + await runGauntletOnce({ + scenario: input.scenario, + model: input.model, + runIndex, + }) + ); + } + + const scores = runs.map((run) => run.score); + const countedRuns = runs.filter((run) => run.countedForRecommendation); + const countedScores = countedRuns.map((run) => run.score); + const scoreStability = summarizeScoreStability(countedScores); + const durations = runs.map((run) => run.durationMs).sort((left, right) => left - right); + const averageScore = round( + scores.reduce((sum, score) => sum + score, 0) / Math.max(scores.length, 1), + 1 + ); + const behavioralAverageScore = + countedScores.length > 0 + ? round( + countedScores.reduce((sum, score) => sum + score, 0) / Math.max(countedScores.length, 1), + 1 + ) + : null; + const hardFailures = runs.filter((run) => run.hardFailure).length; + const successfulRuns = runs.filter((run) => run.passed).length; + const stagePassRates = summarizeStagePassRates(runs); + const taskRefPassRates = summarizeTaskRefPassRates(runs); + const protocolViolationTotals = summarizeProtocolViolations(runs); + const stageFailureImpact = summarizeStageFailureImpact(runs, stagePassRates); + const providerInfraFailures = runs.filter( + (run) => run.failureCategory === 'provider-infra' + ).length; + const runtimeTransportFailures = runs.filter( + (run) => run.failureCategory === 'runtime-transport' + ).length; + const modelBehaviorFailures = runs.filter( + (run) => run.failureCategory === 'model-behavior' + ).length; + const harnessFailures = runs.filter((run) => run.failureCategory === 'harness-error').length; + const qualified = isModelQualified({ + averageScore, + successfulRuns, + minimumAverageScore: input.minimumAverageScore, + minimumSuccessfulRuns: input.minimumSuccessfulRuns, + minimumConsistencyScore: input.minimumConsistencyScore, + hardFailures, + providerInfraFailures, + harnessFailures, + consistencyScore: scoreStability.consistencyScore, + }); + const dominantFailureCategory = getDominantFailureCategory(runs); + const recommendationBlockers = buildRecommendationBlockers({ + qualified, + averageScore, + behavioralAverageScore, + successfulRuns, + minimumAverageScore: input.minimumAverageScore, + minimumSuccessfulRuns: input.minimumSuccessfulRuns, + minimumConsistencyScore: input.minimumConsistencyScore, + hardFailures, + providerInfraFailures, + runtimeTransportFailures, + modelBehaviorFailures, + harnessFailures, + stageFailureImpact, + taskRefPassRates, + protocolViolationTotals, + }); + return { + model: input.model, + verdict: getModelGauntletVerdict({ + qualified, + behavioralAverageScore, + successfulRuns, + countedRuns: countedRuns.length, + totalRuns: runs.length, + providerInfraFailures, + runtimeTransportFailures, + modelBehaviorFailures, + harnessFailures, + }), + confidence: getModelGauntletConfidence({ + countedRuns: countedRuns.length, + providerInfraFailures, + harnessFailures, + }), + qualified, + readinessScore: calculateReadinessScore({ + behavioralAverageScore, + successfulRuns, + countedRuns: countedRuns.length, + providerInfraFailures, + totalRuns: runs.length, + taskRefPassRates, + protocolViolationTotals, + consistencyScore: scoreStability.consistencyScore, + }), + averageScore, + consistencyScore: scoreStability.consistencyScore, + behavioralAverageScore, + minScore: scores.length > 0 ? Math.min(...scores) : 0, + successfulRuns, + countedRuns: countedRuns.length, + hardFailures, + providerInfraFailures, + runtimeTransportFailures, + modelBehaviorFailures, + harnessFailures, + p50DurationMs: percentile(durations, 0.5), + p95DurationMs: percentile(durations, 0.95), + stagePassRates, + taskRefPassRates, + protocolViolationTotals, + stageFailureImpact, + scoreStability, + dominantFailureCategory, + recommendationBlockers, + runs, + }; +} + +async function runGauntletOnce(input: { + scenario: OpenCodeSemanticScenario; + model: string; + runIndex: number; +}): Promise { + const startedAt = Date.now(); + const stages: RunGauntletResult['stages'] = { + launchBootstrap: false, + directReply: false, + peerRelayAB: false, + peerRelayBC: false, + concurrentReplies: false, + taskRefs: false, + cleanTranscript: false, + noDuplicateTokens: false, + latencyStable: false, + }; + const stageDurationsMs: Record = {}; + const stageFailures: Record = {}; + const taskRefChecks: Record = { + directReply: null, + peerRelayAB: null, + peerRelayBC: null, + concurrentBob: null, + concurrentTom: null, + }; + let protocolViolations: RunGauntletResult['protocolViolations'] = { + badMessages: 0, + duplicateOrMissingTokens: [], + }; + const diagnostics: string[] = []; + const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), 'opencode-semantic-gauntlet-')); + const tempClaudeRoot = path.join(tempDir, '.claude'); + const projectPath = path.join(tempDir, 'project'); + const teamName = `${input.scenario.teamNamePrefix}-gauntlet-${sanitizeModelForTeamName( + input.model + )}-${Date.now()}-${input.runIndex}`; + let harness: Awaited> | null = null; + let keepTempDir = false; + const measure = async (stageName: string, run: () => Promise): Promise => { + const stageStartedAt = Date.now(); + try { + return await run(); + } catch (error) { + const message = getGauntletErrorMessage(error); + stageFailures[stageName] = message; + diagnostics.push(`${stageName}: ${message}`); + throw error; + } finally { + stageDurationsMs[stageName] = + (stageDurationsMs[stageName] ?? 0) + Date.now() - stageStartedAt; + } + }; + + try { + await measure('setup', async () => { + await fs.mkdir(tempClaudeRoot, { recursive: true }); + await fs.mkdir(projectPath, { recursive: true }); + setClaudeBasePathOverride(tempClaudeRoot); + await materializeOpenCodeScenarioProject(input.scenario, projectPath); + harness = await createOpenCodeLiveHarness({ + tempDir, + selectedModel: input.model, + projectPath, + }); + }); + + const progressEvents: TeamProvisioningProgress[] = []; + const createStartedAt = Date.now(); + const { runId } = await measure('launchBootstrap', async () => + harness!.svc.createTeam( + buildOpenCodeScenarioTeamRequest({ + scenario: input.scenario, + teamName, + projectPath, + model: harness!.selectedModel, + }), + (progress) => progressEvents.push(progress) + ) + ); + diagnostics.push(`runId=${runId}`); + await measure('materializeTasks', async () => + materializeOpenCodeScenarioTasks({ scenario: input.scenario, teamName, projectPath }) + ); + assertLaunchReady(progressEvents); + stages.launchBootstrap = true; + + const direct = await measure('directReply', async () => + sendDirectAndWait({ + scenario: input.scenario, + harness: harness!, + teamName, + projectPath, + runIndex: input.runIndex, + expectation: { + memberName: 'bob', + messageId: `gauntlet-direct-${input.runIndex}-${Date.now()}`, + replyRecipient: 'user', + actionMode: 'ask', + taskIndex: 0, + expectedReplyToken: `GAUNTLET_DIRECT_BOB_OK_${input.runIndex}`, + text: [ + `Investigate task #${input.scenario.tasks[0].displayId} under realistic context pressure.`, + `Reply to the app user with GAUNTLET_DIRECT_BOB_OK_${input.runIndex}.`, + 'Use agent-teams_message_send to user, preserve taskRefs, and do not use SendMessage.', + ].join('\n'), + }, + }) + ); + stages.directReply = direct.replyOk; + stages.taskRefs = direct.taskRefsOk; + taskRefChecks.directReply = direct.taskRefsOk; + await waitForOpenCodeMemberIdle({ + bridgeClient: harness!.bridgeClient, + teamName, + memberName: 'bob', + projectPath, + timeoutMs: 90_000, + }); + + const peerAB = await measure('peerRelayAB', async () => + sendPeerAndWait({ + scenario: input.scenario, + harness: harness!, + teamName, + projectPath, + runIndex: input.runIndex, + expectation: { + senderName: 'bob', + recipientName: 'jack', + messageId: `gauntlet-peer-ab-${input.runIndex}-${Date.now()}`, + taskIndex: 1, + peerToken: `GAUNTLET_PEER_BOB_TO_JACK_OK_${input.runIndex}`, + expectedReplyToken: `GAUNTLET_JACK_USER_OK_${input.runIndex}`, + text: [ + `Send one team message to jack about task #${input.scenario.tasks[1].displayId}.`, + `The teammate message must include GAUNTLET_PEER_BOB_TO_JACK_OK_${input.runIndex}.`, + `Ask jack to reply to the app user with GAUNTLET_JACK_USER_OK_${input.runIndex}.`, + 'Use agent-teams_message_send to jack, not a plain assistant answer.', + ].join('\n'), + }, + }) + ); + stages.peerRelayAB = peerAB.replyOk; + stages.taskRefs = stages.taskRefs && peerAB.taskRefsOk; + taskRefChecks.peerRelayAB = peerAB.taskRefsOk; + await waitForOpenCodeMemberIdle({ + bridgeClient: harness!.bridgeClient, + teamName, + memberName: 'jack', + projectPath, + timeoutMs: 90_000, + }); + + const peerBC = await measure('peerRelayBC', async () => + sendPeerAndWait({ + scenario: input.scenario, + harness: harness!, + teamName, + projectPath, + runIndex: input.runIndex, + expectation: { + senderName: 'jack', + recipientName: 'tom', + messageId: `gauntlet-peer-bc-${input.runIndex}-${Date.now()}`, + taskIndex: 2, + peerToken: `GAUNTLET_PEER_JACK_TO_TOM_OK_${input.runIndex}`, + expectedReplyToken: `GAUNTLET_TOM_USER_OK_${input.runIndex}`, + text: [ + `Send one team message to tom about task #${input.scenario.tasks[2].displayId}.`, + `The teammate message must include GAUNTLET_PEER_JACK_TO_TOM_OK_${input.runIndex}.`, + `Ask tom to reply to the app user with GAUNTLET_TOM_USER_OK_${input.runIndex}.`, + 'Use agent-teams_message_send to tom and keep the taskRefs metadata.', + ].join('\n'), + }, + }) + ); + stages.peerRelayBC = peerBC.replyOk; + stages.taskRefs = stages.taskRefs && peerBC.taskRefsOk; + taskRefChecks.peerRelayBC = peerBC.taskRefsOk; + await waitForOpenCodeMemberIdle({ + bridgeClient: harness!.bridgeClient, + teamName, + memberName: 'tom', + projectPath, + timeoutMs: 90_000, + }); + + const concurrent = await measure('concurrentReplies', async () => + Promise.allSettled([ + sendDirectAndWait({ + scenario: input.scenario, + harness: harness!, + teamName, + projectPath, + runIndex: input.runIndex, + expectation: { + memberName: 'bob', + messageId: `gauntlet-concurrent-bob-${input.runIndex}-${Date.now()}`, + replyRecipient: 'user', + actionMode: 'ask', + taskIndex: 3, + expectedReplyToken: `GAUNTLET_CONCURRENT_BOB_OK_${input.runIndex}`, + text: [ + `Concurrent check for task #${input.scenario.tasks[3].displayId}.`, + `Reply to user with GAUNTLET_CONCURRENT_BOB_OK_${input.runIndex}.`, + 'This message is intentionally sent near another teammate delivery.', + ].join('\n'), + }, + }), + sendDirectAndWait({ + scenario: input.scenario, + harness: harness!, + teamName, + projectPath, + runIndex: input.runIndex, + expectation: { + memberName: 'tom', + messageId: `gauntlet-concurrent-tom-${input.runIndex}-${Date.now()}`, + replyRecipient: 'user', + actionMode: 'ask', + taskIndex: 4, + expectedReplyToken: `GAUNTLET_CONCURRENT_TOM_OK_${input.runIndex}`, + text: [ + `Concurrent check for task #${input.scenario.tasks[4].displayId}.`, + `Reply to user with GAUNTLET_CONCURRENT_TOM_OK_${input.runIndex}.`, + 'Do not produce an idle acknowledgement or duplicate reply.', + ].join('\n'), + }, + }), + ]) + ); + const concurrentBob = concurrent[0]; + const concurrentTom = concurrent[1]; + if (concurrentBob?.status === 'rejected') { + const message = getGauntletErrorMessage(concurrentBob.reason); + stageFailures.concurrentBob = message; + diagnostics.push(`concurrentBob: ${message}`); + } + if (concurrentTom?.status === 'rejected') { + const message = getGauntletErrorMessage(concurrentTom.reason); + stageFailures.concurrentTom = message; + diagnostics.push(`concurrentTom: ${message}`); + } + const concurrentResults = concurrent.flatMap((result) => + result.status === 'fulfilled' ? [result.value] : [] + ); + stages.concurrentReplies = + concurrent.length === 2 && + concurrent.every((result) => result.status === 'fulfilled' && result.value.replyOk); + stages.taskRefs = + stages.taskRefs && + concurrent.length === 2 && + concurrent.every((result) => result.status === 'fulfilled' && result.value.taskRefsOk); + taskRefChecks.concurrentBob = + concurrentBob?.status === 'fulfilled' ? concurrentBob.value.taskRefsOk : false; + taskRefChecks.concurrentTom = + concurrentTom?.status === 'fulfilled' ? concurrentTom.value.taskRefsOk : false; + if (concurrentResults.length !== concurrent.length) { + stageFailures.concurrentReplies = 'one_or_more_concurrent_deliveries_failed'; + } + + const expectedUserReplyTokens = [ + `GAUNTLET_DIRECT_BOB_OK_${input.runIndex}`, + `GAUNTLET_JACK_USER_OK_${input.runIndex}`, + `GAUNTLET_TOM_USER_OK_${input.runIndex}`, + `GAUNTLET_CONCURRENT_BOB_OK_${input.runIndex}`, + `GAUNTLET_CONCURRENT_TOM_OK_${input.runIndex}`, + ]; + const hygiene = await measure('hygiene', async () => + inspectMessageHygiene({ + teamName, + members: input.scenario.members.map((member) => member.name), + expectedUserReplyTokens, + }) + ); + stages.cleanTranscript = hygiene.clean; + stages.noDuplicateTokens = hygiene.noDuplicateTokens; + protocolViolations = { + badMessages: hygiene.badMessages, + duplicateOrMissingTokens: hygiene.duplicateOrMissingTokens, + }; + diagnostics.push(...hygiene.diagnostics); + stages.latencyStable = Date.now() - createStartedAt < 420_000; + + const score = scoreRun(stages); + const failureCategory = + score >= 90 && !isHardProtocolFailure(stages) + ? 'none' + : classifyGauntletFailure({ diagnostics, stages }); + return { + runIndex: input.runIndex, + passed: failureCategory === 'none', + score, + countedForRecommendation: isCountedForRecommendation(failureCategory), + outcome: getRunOutcome(failureCategory), + failureCategory, + primaryFailure: getPrimaryFailure(diagnostics), + durationMs: Date.now() - startedAt, + hardFailure: failureCategory !== 'none' && failureCategory !== 'provider-infra', + stageDurationsMs, + stageFailures, + taskRefChecks, + protocolViolations, + stages, + diagnostics, + }; + } catch (error) { + if (process.env.OPENCODE_E2E_KEEP_FAILED === '1') { + keepTempDir = true; + diagnostics.push(`tempDir=${tempDir}`); + } + diagnostics.push(error instanceof Error ? error.message : String(error)); + const failureCategory = classifyGauntletFailure({ diagnostics, stages }); + return { + runIndex: input.runIndex, + passed: false, + score: scoreRun(stages), + countedForRecommendation: isCountedForRecommendation(failureCategory), + outcome: getRunOutcome(failureCategory), + failureCategory, + primaryFailure: getPrimaryFailure(diagnostics), + durationMs: Date.now() - startedAt, + hardFailure: failureCategory !== 'provider-infra', + stageDurationsMs, + stageFailures, + taskRefChecks, + protocolViolations, + stages, + diagnostics, + }; + } finally { + if (harness) { + await harness.svc.stopTeam(teamName).catch(() => undefined); + await harness.dispose().catch(() => undefined); + await waitForOpenCodeLanesStopped(teamName).catch(() => undefined); + } + setClaudeBasePathOverride(null); + if (!keepTempDir) { + await fs.rm(tempDir, { recursive: true, force: true }).catch(() => undefined); + } + } +} + +async function sendDirectAndWait(input: { + scenario: OpenCodeSemanticScenario; + harness: Awaited>; + teamName: string; + projectPath: string; + runIndex: number; + expectation: DeliveryExpectation; +}): Promise<{ replyOk: boolean; taskRefsOk: boolean }> { + const taskRef = taskRefForScenario(input.scenario, input.expectation.taskIndex, input.teamName); + const delivery = await input.harness.svc.deliverOpenCodeMemberMessage(input.teamName, { + memberName: input.expectation.memberName, + messageId: input.expectation.messageId, + replyRecipient: input.expectation.replyRecipient, + actionMode: input.expectation.actionMode, + taskRefs: [taskRef], + source: 'manual', + text: input.expectation.text, + }); + if (!delivery.delivered) { + throw new Error(`Direct delivery failed: ${JSON.stringify(delivery, null, 2)}`); + } + const reply = await waitForReplyWithTranscript({ + bridgeClient: input.harness.bridgeClient, + teamName: input.teamName, + memberName: input.expectation.memberName, + projectPath: input.projectPath, + expectedToken: input.expectation.expectedReplyToken, + timeoutMs: 180_000, + }); + assertVisibleReplyContract(reply, { + expectedFrom: input.expectation.memberName, + expectedTo: 'user', + }); + return { + replyOk: true, + taskRefsOk: hasTaskRef(reply, taskRef), + }; +} + +async function sendPeerAndWait(input: { + scenario: OpenCodeSemanticScenario; + harness: Awaited>; + teamName: string; + projectPath: string; + runIndex: number; + expectation: PeerExpectation; +}): Promise<{ replyOk: boolean; taskRefsOk: boolean }> { + const taskRef = taskRefForScenario(input.scenario, input.expectation.taskIndex, input.teamName); + const delivery = await input.harness.svc.deliverOpenCodeMemberMessage(input.teamName, { + memberName: input.expectation.senderName, + messageId: input.expectation.messageId, + replyRecipient: input.expectation.recipientName, + actionMode: 'delegate', + taskRefs: [taskRef], + source: 'manual', + text: input.expectation.text, + }); + if (!delivery.delivered) { + throw new Error(`Peer delivery failed: ${JSON.stringify(delivery, null, 2)}`); + } + const memberMessage = await waitForMemberInboxMessage( + input.teamName, + input.expectation.recipientName, + input.expectation.senderName, + input.expectation.peerToken, + 180_000 + ); + assertVisibleReplyContract(memberMessage, { + expectedFrom: input.expectation.senderName, + expectedTo: input.expectation.recipientName, + }); + await waitForOpenCodePeerRelay( + input.harness.svc, + input.teamName, + input.expectation.recipientName, + memberMessage.messageId, + 180_000 + ); + const reply = await waitForReplyWithTranscript({ + bridgeClient: input.harness.bridgeClient, + teamName: input.teamName, + memberName: input.expectation.recipientName, + projectPath: input.projectPath, + expectedToken: input.expectation.expectedReplyToken, + timeoutMs: 180_000, + }); + assertVisibleReplyContract(reply, { + expectedFrom: input.expectation.recipientName, + expectedTo: 'user', + }); + return { + replyOk: true, + taskRefsOk: hasTaskRef(memberMessage, taskRef), + }; +} + +async function waitForReplyWithTranscript(input: { + bridgeClient: Parameters[0]['bridgeClient']; + teamName: string; + memberName: string; + projectPath: string; + expectedToken: string; + timeoutMs: number; +}): Promise { + try { + return await waitForUserInboxReply( + input.teamName, + input.memberName, + input.expectedToken, + input.timeoutMs + ); + } catch (error) { + const transcript = await getRuntimeTranscript({ + bridgeClient: input.bridgeClient, + teamName: input.teamName, + memberName: input.memberName, + projectPath: input.projectPath, + }); + throw new Error( + `${error instanceof Error ? error.message : String(error)}\nTranscript: ${JSON.stringify( + transcript, + null, + 2 + )}` + ); + } +} + +function assertVisibleReplyContract( + message: InboxMessage, + input: { + expectedFrom: string; + expectedTo: string; + expectedTaskRef?: TaskRef; + } +): void { + expect(message).toMatchObject({ + from: input.expectedFrom, + to: input.expectedTo, + }); + const text = message.text ?? ''; + expect(text).not.toContain('SendMessage'); + expect(text).not.toContain('runtime_deliver_message'); + expect(text).not.toContain('#00000000'); + expect(text.trim()).not.toBe('\u041f\u043e\u043d\u044f\u043b'); + if (input.expectedTaskRef) { + expect(hasTaskRef(message, input.expectedTaskRef)).toBe(true); + } +} + +function hasTaskRef(message: InboxMessage, expected: TaskRef): boolean { + return Boolean( + message.taskRefs?.some( + (taskRef) => + taskRef.teamName === expected.teamName && + taskRef.taskId === expected.taskId && + taskRef.displayId === expected.displayId + ) + ); +} + +async function inspectMessageHygiene(input: { + teamName: string; + members: string[]; + expectedUserReplyTokens: string[]; +}): Promise<{ + clean: boolean; + noDuplicateTokens: boolean; + badMessages: number; + duplicateOrMissingTokens: string[]; + diagnostics: string[]; +}> { + const userInboxPath = path.join(getTeamsBasePath(), input.teamName, 'inboxes', 'user.json'); + const inboxPaths = [ + userInboxPath, + ...input.members.map((member) => + path.join(getTeamsBasePath(), input.teamName, 'inboxes', `${member}.json`) + ), + ]; + const userMessages = await readInboxMessages(userInboxPath); + const messages = ( + await Promise.all(inboxPaths.map((inboxPath) => readInboxMessages(inboxPath))) + ).flat(); + const diagnostics: string[] = []; + const badMessages = messages.filter((message) => { + const text = message.text ?? ''; + return ( + text.includes('SendMessage') || + text.includes('runtime_deliver_message') || + text.includes('#00000000') || + text.trim() === '\u041f\u043e\u043d\u044f\u043b' + ); + }); + if (badMessages.length > 0) { + diagnostics.push(`badMessages=${JSON.stringify(badMessages.slice(0, 5))}`); + } + const duplicateTokens = input.expectedUserReplyTokens.filter((token) => { + const count = userMessages.filter((message) => message.text?.includes(token)).length; + return count !== 1; + }); + if (duplicateTokens.length > 0) { + diagnostics.push(`duplicateOrMissingTokens=${duplicateTokens.join(',')}`); + } + return { + clean: badMessages.length === 0, + noDuplicateTokens: duplicateTokens.length === 0, + badMessages: badMessages.length, + duplicateOrMissingTokens: duplicateTokens, + diagnostics, + }; +} + +function buildGauntletScenario(base: OpenCodeSemanticScenario): OpenCodeSemanticScenario { + const extraProjectFiles = Array.from({ length: 24 }, (_, index) => ({ + path: `src/gauntlet/context-${String(index + 1).padStart(2, '0')}.ts`, + contentLines: [ + `export const GAUNTLET_CONTEXT_${index + 1} = {`, + ` area: 'opencode-agent-teams-${index + 1}',`, + " constraint: 'preserve routing, taskRefs, and visible MCP replies',", + '};', + ], + })); + return { + ...base, + teamPromptLines: [ + ...base.teamPromptLines, + 'Gauntlet mode: maintain correctness across repeated deliveries, near-concurrent messages, and multi-hop teammate relays.', + 'A model is only strong if it repeatedly uses Agent Teams MCP message_send, keeps recipients exact, preserves taskRefs, and avoids duplicate visible replies.', + ...Array.from( + { length: 12 }, + (_, index) => + `Gauntlet context note ${index + 1}: stale acknowledgements, fake tool text, wrong recipients, missing taskRefs, or duplicate tokens are production failures.` + ), + ], + members: [ + ...base.members, + { + name: 'tom', + role: 'Developer', + workflowLines: [ + 'Own concurrent delivery checks and multi-hop peer relay replies.', + 'Reply to the app user only when explicitly instructed by a relayed teammate message.', + 'Preserve taskRefs and never emit fake task ids.', + ], + }, + ], + projectFiles: [...base.projectFiles, ...extraProjectFiles], + tasks: [ + ...base.tasks, + { + taskId: 'task-82ad912c-multihop-relay', + displayId: '82ad912c', + subject: 'Verify multi-hop OpenCode peer relay reaches a third teammate', + owner: 'tom', + comment: + 'Jack must send a relayed message to tom and tom must reply visibly to the app user.', + }, + { + taskId: 'task-9e2f74aa-concurrent-bob', + displayId: '9e2f74aa', + subject: 'Verify concurrent delivery to bob does not duplicate replies', + owner: 'bob', + comment: + 'Bob receives a near-concurrent app message and must produce exactly one visible reply.', + }, + { + taskId: 'task-1b4c8afd-concurrent-tom', + displayId: '1b4c8afd', + subject: 'Verify concurrent delivery to tom preserves taskRefs', + owner: 'tom', + comment: + 'Tom receives a near-concurrent app message and must preserve the taskRefs metadata.', + }, + ], + }; +} + +function assertLaunchReady(progressEvents: TeamProvisioningProgress[]): void { + if ( + !progressEvents.some((progress) => progress.message.includes('OpenCode team launch is ready')) + ) { + throw new Error( + `OpenCode launch did not reach ready state.\n${formatProgressDump(progressEvents)}` + ); + } +} + +function scoreRun(stages: RunGauntletResult['stages']): number { + return Object.entries(stages).reduce( + (score, [stage, passed]) => + score + (passed ? SCORE_WEIGHTS[stage as keyof RunGauntletResult['stages']] : 0), + 0 + ); +} + +function isHardProtocolFailure(stages: RunGauntletResult['stages']): boolean { + return ( + !stages.launchBootstrap || + !stages.directReply || + !stages.peerRelayAB || + !stages.peerRelayBC || + !stages.concurrentReplies || + !stages.taskRefs || + !stages.cleanTranscript || + !stages.noDuplicateTokens + ); +} + +function classifyGauntletFailure(input: { + diagnostics: readonly string[]; + stages: RunGauntletResult['stages']; +}): RunFailureCategory { + const text = input.diagnostics.join('\n').toLowerCase(); + if (!text && !isHardProtocolFailure(input.stages)) { + return 'none'; + } + if ( + [ + 'key limit exceeded', + 'total limit exceeded', + 'requires more credits', + 'can only afford', + 'rate limit', + '429', + '402', + 'no endpoints found', + 'selected model', + 'not found in the live provider catalog', + 'unable to connect', + 'provider unavailable', + 'insufficient credits', + ].some((pattern) => text.includes(pattern)) + ) { + return 'provider-infra'; + } + if ( + [ + 'opencode tool failed without output', + 'not connected to message transport', + 'opencode prompt delivery record not found', + '"responsestate": "session_error"', + 'session_error', + 'transport', + ].some((pattern) => text.includes(pattern)) + ) { + return 'runtime-transport'; + } + if (text.includes('transcript:') || isHardProtocolFailure(input.stages)) { + return 'model-behavior'; + } + return 'harness-error'; +} + +function isCountedForRecommendation(category: RunFailureCategory): boolean { + return category === 'none' || category === 'model-behavior' || category === 'runtime-transport'; +} + +function getRunOutcome(category: RunFailureCategory): RunOutcome { + if (category === 'none') { + return 'passed'; + } + if (category === 'provider-infra') { + return 'provider-infra-blocked'; + } + if (category === 'runtime-transport') { + return 'runtime-transport-fail'; + } + if (category === 'harness-error') { + return 'harness-error'; + } + return 'behavioral-fail'; +} + +function getPrimaryFailure(diagnostics: readonly string[]): string | null { + const diagnostic = diagnostics.find((item) => item.trim() && !item.startsWith('runId=')); + if (!diagnostic) { + return null; + } + return diagnostic.replace(/\s+/g, ' ').trim().slice(0, 240); +} + +function getGauntletErrorMessage(error: unknown): string { + return (error instanceof Error ? error.message : String(error)).replace(/\s+/g, ' ').trim(); +} + +function getModelGauntletVerdict(input: { + qualified: boolean; + behavioralAverageScore: number | null; + successfulRuns: number; + countedRuns: number; + totalRuns: number; + providerInfraFailures: number; + runtimeTransportFailures: number; + modelBehaviorFailures: number; + harnessFailures: number; +}): ModelGauntletVerdict { + if (input.qualified) { + return 'recommended'; + } + if (input.countedRuns === 0 && input.providerInfraFailures > 0) { + return 'infra-blocked'; + } + if (input.countedRuns === 0 || input.harnessFailures === input.totalRuns) { + return 'inconclusive'; + } + if ( + input.behavioralAverageScore !== null && + input.behavioralAverageScore >= 85 && + input.successfulRuns > 0 && + input.providerInfraFailures === 0 && + input.harnessFailures === 0 && + input.runtimeTransportFailures <= 1 && + input.modelBehaviorFailures <= 1 + ) { + return 'strong-candidate'; + } + return 'tested-only'; +} + +function isModelQualified(input: { + averageScore: number; + successfulRuns: number; + minimumAverageScore: number; + minimumSuccessfulRuns: number; + minimumConsistencyScore: number; + hardFailures: number; + providerInfraFailures: number; + harnessFailures: number; + consistencyScore: number; +}): boolean { + return ( + input.averageScore >= input.minimumAverageScore && + input.successfulRuns >= input.minimumSuccessfulRuns && + input.consistencyScore >= input.minimumConsistencyScore && + input.hardFailures === 0 && + input.providerInfraFailures === 0 && + input.harnessFailures === 0 + ); +} + +function getModelGauntletConfidence(input: { + countedRuns: number; + providerInfraFailures: number; + harnessFailures: number; +}): ModelGauntletConfidence { + if (input.countedRuns === 0 || input.harnessFailures > 0) { + return 'blocked'; + } + if (input.countedRuns >= 3 && input.providerInfraFailures === 0) { + return 'high'; + } + if (input.countedRuns >= 2) { + return 'medium'; + } + return 'low'; +} + +function summarizeStagePassRates( + runs: readonly RunGauntletResult[] +): Record { + return Object.fromEntries( + GAUNTLET_STAGE_NAMES.map((stage) => { + const passed = runs.filter((run) => run.stages[stage]).length; + return [stage, buildRatioSummary(passed, runs.length)]; + }) + ) as Record; +} + +function summarizeTaskRefPassRates( + runs: readonly RunGauntletResult[] +): Record { + return Object.fromEntries( + TASK_REF_CHECK_NAMES.map((name) => { + const values = runs + .map((run) => run.taskRefChecks[name]) + .filter((value): value is boolean => typeof value === 'boolean'); + const passed = values.filter(Boolean).length; + return [name, buildRatioSummary(passed, values.length)]; + }) + ); +} + +function summarizeProtocolViolations(runs: readonly RunGauntletResult[]): ProtocolViolationTotals { + return runs.reduce( + (totals, run) => { + const duplicateOrMissingTokens = run.protocolViolations.duplicateOrMissingTokens.length; + const affected = run.protocolViolations.badMessages > 0 || duplicateOrMissingTokens > 0; + return { + badMessages: totals.badMessages + run.protocolViolations.badMessages, + duplicateOrMissingTokens: totals.duplicateOrMissingTokens + duplicateOrMissingTokens, + affectedRuns: totals.affectedRuns + (affected ? 1 : 0), + }; + }, + { badMessages: 0, duplicateOrMissingTokens: 0, affectedRuns: 0 } + ); +} + +function summarizeScoreStability(scores: readonly number[]): ScoreStabilitySummary { + if (scores.length === 0) { + return { + sampleSize: 0, + minScore: 0, + maxScore: 0, + spread: 0, + standardDeviation: 0, + consistencyScore: 0, + }; + } + const minScore = Math.min(...scores); + const maxScore = Math.max(...scores); + const average = scores.reduce((sum, score) => sum + score, 0) / scores.length; + const variance = scores.reduce((sum, score) => sum + (score - average) ** 2, 0) / scores.length; + const standardDeviation = round(Math.sqrt(variance), 1); + const spread = maxScore - minScore; + return { + sampleSize: scores.length, + minScore, + maxScore, + spread, + standardDeviation, + consistencyScore: Math.max(0, round(100 - spread * 0.8 - standardDeviation * 1.2, 1)), + }; +} + +function summarizeStageFailureImpact( + runs: readonly RunGauntletResult[], + stagePassRates: Record +): StageFailureImpactSummary[] { + return GAUNTLET_STAGE_NAMES.map((stage) => { + const failedRuns = runs.filter((run) => !run.stages[stage]).length; + return { + stage, + failedRuns, + weightedLoss: failedRuns * SCORE_WEIGHTS[stage], + passRate: stagePassRates[stage], + }; + }).sort((left, right) => { + if (left.weightedLoss !== right.weightedLoss) { + return right.weightedLoss - left.weightedLoss; + } + return left.stage.localeCompare(right.stage); + }); +} + +function getDominantFailureCategory( + runs: readonly RunGauntletResult[] +): RunFailureCategory | 'mixed' { + const failedRuns = runs.filter((run) => run.failureCategory !== 'none'); + if (failedRuns.length === 0) { + return 'none'; + } + const counts = failedRuns.reduce>( + (totals, run) => ({ + ...totals, + [run.failureCategory]: totals[run.failureCategory] + 1, + }), + { + none: 0, + 'model-behavior': 0, + 'runtime-transport': 0, + 'provider-infra': 0, + 'harness-error': 0, + } + ); + const ranked = (Object.entries(counts) as Array<[RunFailureCategory, number]>) + .filter(([category, count]) => category !== 'none' && count > 0) + .sort((left, right) => right[1] - left[1]); + if (ranked.length === 0) { + return 'none'; + } + if (ranked.length > 1 && ranked[0][1] === ranked[1][1]) { + return 'mixed'; + } + return ranked[0][0]; +} + +function calculateReadinessScore(input: { + behavioralAverageScore: number | null; + successfulRuns: number; + countedRuns: number; + providerInfraFailures: number; + totalRuns: number; + taskRefPassRates: Record; + protocolViolationTotals: ProtocolViolationTotals; + consistencyScore: number; +}): number { + if (input.countedRuns === 0) { + return 0; + } + const behavioralScore = input.behavioralAverageScore ?? 0; + const passRate = + input.countedRuns > 0 ? round((input.successfulRuns / input.countedRuns) * 100, 1) : 0; + const taskRefRate = averageRatioRate(input.taskRefPassRates); + const protocolCleanRate = + input.totalRuns > 0 + ? round( + ((input.totalRuns - input.protocolViolationTotals.affectedRuns) / input.totalRuns) * 100, + 1 + ) + : 0; + const infraCleanRate = + input.totalRuns > 0 + ? round(((input.totalRuns - input.providerInfraFailures) / input.totalRuns) * 100, 1) + : 0; + return round( + behavioralScore * 0.4 + + passRate * 0.2 + + taskRefRate * 0.15 + + protocolCleanRate * 0.1 + + input.consistencyScore * 0.1 + + infraCleanRate * 0.05, + 1 + ); +} + +function averageRatioRate(ratios: Record): number { + const rates = Object.values(ratios) + .map((summary) => summary.rate) + .filter((rate): rate is number => typeof rate === 'number'); + if (rates.length === 0) { + return 100; + } + return round(rates.reduce((sum, rate) => sum + rate, 0) / rates.length, 1); +} + +function buildRecommendationBlockers(input: { + qualified: boolean; + averageScore: number; + behavioralAverageScore: number | null; + successfulRuns: number; + minimumAverageScore: number; + minimumSuccessfulRuns: number; + minimumConsistencyScore: number; + hardFailures: number; + providerInfraFailures: number; + runtimeTransportFailures: number; + modelBehaviorFailures: number; + harnessFailures: number; + consistencyScore: number; + stageFailureImpact: readonly StageFailureImpactSummary[]; + taskRefPassRates: Record; + protocolViolationTotals: ProtocolViolationTotals; +}): string[] { + if (input.qualified) { + return []; + } + const blockers: string[] = []; + if (input.averageScore < input.minimumAverageScore) { + blockers.push(`overall average ${input.averageScore} < ${input.minimumAverageScore}`); + } + if ( + input.behavioralAverageScore !== null && + input.behavioralAverageScore < input.minimumAverageScore + ) { + blockers.push( + `behavioral average ${input.behavioralAverageScore} < ${input.minimumAverageScore}` + ); + } + if (input.successfulRuns < input.minimumSuccessfulRuns) { + blockers.push(`successful runs ${input.successfulRuns} < ${input.minimumSuccessfulRuns}`); + } + if (input.consistencyScore < input.minimumConsistencyScore) { + blockers.push(`consistency score ${input.consistencyScore} < ${input.minimumConsistencyScore}`); + } + if (input.hardFailures > 0) { + blockers.push(`hard failures ${input.hardFailures}`); + } + if (input.providerInfraFailures > 0) { + blockers.push(`provider-infra failures ${input.providerInfraFailures}`); + } + if (input.harnessFailures > 0) { + blockers.push(`harness failures ${input.harnessFailures}`); + } + if (input.runtimeTransportFailures > 0) { + blockers.push(`runtime-transport failures ${input.runtimeTransportFailures}`); + } + if (input.modelBehaviorFailures > 0) { + blockers.push(`model-behavior failures ${input.modelBehaviorFailures}`); + } + const highestImpact = input.stageFailureImpact.find((stage) => stage.failedRuns > 0); + if (highestImpact) { + blockers.push( + `highest weighted stage loss ${highestImpact.stage}=${highestImpact.weightedLoss}` + ); + } + const weakestTaskRef = getWeakestRatioEntry(input.taskRefPassRates); + if (weakestTaskRef && weakestTaskRef[1].rate !== null && weakestTaskRef[1].rate < 100) { + blockers.push(`weakest taskRefs ${weakestTaskRef[0]}=${renderRatio(weakestTaskRef[1])}`); + } + if (input.protocolViolationTotals.affectedRuns > 0) { + blockers.push(`protocol violations in ${input.protocolViolationTotals.affectedRuns} runs`); + } + return blockers; +} + +function buildRatioSummary(passed: number, total: number): RatioSummary { + return { + passed, + total, + rate: total > 0 ? round((passed / total) * 100, 1) : null, + }; +} + +async function writeGauntletReport(report: GauntletReport): Promise { + const outputDir = process.env.OPENCODE_E2E_REPORT_DIR?.trim() + ? path.resolve(process.env.OPENCODE_E2E_REPORT_DIR.trim()) + : path.join(process.cwd(), 'test-results', 'opencode-semantic-model-gauntlet'); + await fs.mkdir(outputDir, { recursive: true }); + await fs.writeFile( + path.join(outputDir, 'model-gauntlet-results.json'), + `${JSON.stringify(report, null, 2)}\n`, + 'utf8' + ); + await fs.writeFile( + path.join(outputDir, 'model-gauntlet-results.md'), + renderGauntletMarkdown(report), + 'utf8' + ); +} + +function renderGauntletMarkdown(report: GauntletReport): string { + const lines = [ + '# OpenCode Model Gauntlet Results', + '', + `Generated: ${report.generatedAt}`, + '', + `Runs per model: ${report.runsPerModel}`, + `Recommended threshold: average >= ${report.qualification.minimumAverageScore}, successful runs >= ${report.qualification.minimumSuccessfulRuns}, consistency >= ${report.qualification.minimumConsistencyScore}, hard failures = 0`, + '', + 'Provider-infra runs are reported separately and are not counted as model behavior. They still block a Recommended verdict until rerun succeeds.', + '', + `Scoring weights: ${renderScoreWeights()}.`, + '', + '## Model Summary', + '', + '| Model | Verdict | Confidence | Readiness | Consistency | Score Spread | Behavior Avg | Overall Avg | Counted | Pass Runs | Weakest Stage | Weakest TaskRef | Dominant Failure | Blockers | Provider Infra | Runtime Transport | Model Fails | Protocol Runs | p50 | p95 |', + '| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |', + ]; + for (const model of report.models) { + lines.push( + `| \`${model.model}\` | ${renderModelVerdict(model.verdict)} | ${model.confidence} | ${ + model.readinessScore + } | ${model.consistencyScore} | ${model.scoreStability.spread} | ${ + model.behavioralAverageScore ?? 'n/a' + } | ${model.averageScore} | ${model.countedRuns}/${model.runs.length} | ${ + model.successfulRuns + }/${model.runs.length} | ${renderWeakestRatio(model.stagePassRates)} | ${renderWeakestRatio( + model.taskRefPassRates + )} | ${model.dominantFailureCategory} | ${escapeMarkdownTableCell( + renderBlockers(model.recommendationBlockers) + )} | ${model.providerInfraFailures} | ${model.runtimeTransportFailures} | ${ + model.modelBehaviorFailures + } | ${model.protocolViolationTotals.affectedRuns} | ${model.p50DurationMs}ms | ${ + model.p95DurationMs + }ms |` + ); + } + for (const model of report.models) { + lines.push('', `## ${model.model}`, ''); + lines.push( + `Readiness score: ${model.readinessScore}.`, + '', + `Score stability: ${renderScoreStability(model.scoreStability)}.`, + '', + `Recommendation blockers: ${renderBlockers(model.recommendationBlockers)}.`, + '', + `Weighted stage impact: ${renderStageFailureImpact(model.stageFailureImpact)}.`, + '', + `Stage pass rates: ${renderRatioMap(model.stagePassRates)}.`, + '', + `TaskRef pass rates: ${renderRatioMap(model.taskRefPassRates)}.`, + '', + `Protocol totals: badMessages=${model.protocolViolationTotals.badMessages}, duplicateOrMissingTokens=${model.protocolViolationTotals.duplicateOrMissingTokens}, affectedRuns=${model.protocolViolationTotals.affectedRuns}.`, + '' + ); + lines.push( + '| Run | Outcome | Category | Score | Counted | Duration | Failed Stages | Slowest Stage | TaskRefs | Protocol | Diagnostics |' + ); + lines.push('| ---: | --- | --- | ---: | --- | ---: | --- | --- | --- | --- | --- |'); + for (const run of model.runs) { + const failedStages = Object.entries(run.stages) + .filter(([, passed]) => !passed) + .map(([stage]) => stage) + .join(', '); + const slowestStage = renderSlowestStage(run.stageDurationsMs); + lines.push( + `| ${run.runIndex} | ${run.outcome} | ${run.failureCategory} | ${run.score} | ${ + run.countedForRecommendation ? 'yes' : 'no' + } | ${run.durationMs}ms | ${failedStages || '-'} | ${slowestStage} | ${renderTaskRefChecks( + run.taskRefChecks + )} | ${renderProtocolViolations(run.protocolViolations)} | ${escapeMarkdownTableCell( + (run.primaryFailure ?? + renderStageFailures(run.stageFailures) ?? + run.diagnostics.join('; ').slice(0, 600)) || + '-' + )} |` + ); + } + } + lines.push(''); + return `${lines.join('\n')}\n`; +} + +function renderModelVerdict(verdict: ModelGauntletVerdict): string { + switch (verdict) { + case 'recommended': + return 'Recommended'; + case 'strong-candidate': + return 'Strong candidate'; + case 'infra-blocked': + return 'Infra blocked'; + case 'inconclusive': + return 'Inconclusive'; + case 'tested-only': + default: + return 'Tested only'; + } +} + +function renderScoreWeights(): string { + return GAUNTLET_STAGE_NAMES.map((stage) => `${stage}=${SCORE_WEIGHTS[stage]}`).join(', '); +} + +function renderWeakestRatio(ratios: Record): string { + const weakest = getWeakestRatioEntry(ratios); + if (!weakest) { + return '-'; + } + const [name, summary] = weakest; + return `${name} ${renderRatio(summary)}`; +} + +function getWeakestRatioEntry(ratios: Record): [string, RatioSummary] | null { + const entries = Object.entries(ratios).filter(([, summary]) => summary.total > 0); + if (entries.length === 0) { + return null; + } + return entries.sort((left, right) => { + const leftRate = left[1].rate ?? Number.POSITIVE_INFINITY; + const rightRate = right[1].rate ?? Number.POSITIVE_INFINITY; + if (leftRate !== rightRate) { + return leftRate - rightRate; + } + return left[0].localeCompare(right[0]); + })[0]; +} + +function renderRatioMap(ratios: Record): string { + const entries = Object.entries(ratios); + if (entries.length === 0) { + return '-'; + } + return entries.map(([name, summary]) => `${name}:${renderRatio(summary)}`).join(', '); +} + +function renderRatio(summary: RatioSummary): string { + if (summary.total === 0 || summary.rate === null) { + return 'n/a'; + } + return `${summary.passed}/${summary.total} (${summary.rate}%)`; +} + +function renderBlockers(blockers: readonly string[]): string { + return blockers.length > 0 ? blockers.join('; ') : '-'; +} + +function renderScoreStability(stability: ScoreStabilitySummary): string { + if (stability.sampleSize === 0) { + return 'n/a'; + } + return [ + `consistency=${stability.consistencyScore}`, + `min=${stability.minScore}`, + `max=${stability.maxScore}`, + `spread=${stability.spread}`, + `stdDev=${stability.standardDeviation}`, + `samples=${stability.sampleSize}`, + ].join(', '); +} + +function renderStageFailureImpact(impact: readonly StageFailureImpactSummary[]): string { + const failed = impact.filter((stage) => stage.failedRuns > 0); + if (failed.length === 0) { + return '-'; + } + return failed + .slice(0, 5) + .map( + (stage) => + `${stage.stage}:loss=${stage.weightedLoss}, failed=${stage.failedRuns}, pass=${renderRatio( + stage.passRate + )}` + ) + .join('; '); +} + +function renderSlowestStage(stageDurationsMs: Record): string { + const [stage, durationMs] = + Object.entries(stageDurationsMs).sort((left, right) => right[1] - left[1])[0] ?? []; + if (!stage || typeof durationMs !== 'number') { + return '-'; + } + return `${stage}:${durationMs}ms`; +} + +function renderTaskRefChecks(checks: Record): string { + const entries = Object.entries(checks).filter(([, value]) => value !== null); + if (entries.length === 0) { + return '-'; + } + return entries.map(([stage, passed]) => `${stage}:${passed ? 'ok' : 'fail'}`).join(', '); +} + +function renderProtocolViolations(violations: RunGauntletResult['protocolViolations']): string { + const parts = []; + if (violations.badMessages > 0) { + parts.push(`bad=${violations.badMessages}`); + } + if (violations.duplicateOrMissingTokens.length > 0) { + parts.push(`token=${violations.duplicateOrMissingTokens.join('+')}`); + } + return parts.length > 0 ? parts.join(', ') : '-'; +} + +function renderStageFailures(stageFailures: Record): string | null { + const entries = Object.entries(stageFailures); + if (entries.length === 0) { + return null; + } + return entries + .map(([stage, message]) => `${stage}: ${message}`) + .join('; ') + .slice(0, 600); +} + +function formatProgressDump(progressEvents: TeamProvisioningProgress[]): string { + return progressEvents + .map((progress) => + [ + progress.state, + progress.message, + progress.messageSeverity, + progress.error, + progress.cliLogsTail, + ] + .filter(Boolean) + .join(' | ') + ) + .join('\n'); +} + +function parsePositiveInteger(value: string | undefined, fallback: number): number { + const parsed = Number.parseInt(value ?? '', 10); + return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback; +} + +function percentile(sortedValues: number[], percentileValue: number): number { + if (sortedValues.length === 0) { + return 0; + } + const index = Math.min( + sortedValues.length - 1, + Math.max(0, Math.ceil(sortedValues.length * percentileValue) - 1) + ); + return sortedValues[index]; +} + +function round(value: number, digits: number): number { + const multiplier = 10 ** digits; + return Math.round(value * multiplier) / multiplier; +} + +function escapeMarkdownTableCell(value: string): string { + return value.replaceAll('|', '\\|').replace(/\s+/g, ' ').trim(); +} + +function sanitizeModelForTeamName(model: string): string { + return model + .toLowerCase() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-+|-+$/g, '') + .slice(0, 48); +} + +function createPassingStages( + overrides: Partial = {} +): RunGauntletResult['stages'] { + return { + launchBootstrap: true, + directReply: true, + peerRelayAB: true, + peerRelayBC: true, + concurrentReplies: true, + taskRefs: true, + cleanTranscript: true, + noDuplicateTokens: true, + latencyStable: true, + ...overrides, + }; +} + +function createTestGauntletRun(overrides: Partial = {}): RunGauntletResult { + const stages = overrides.stages ?? createPassingStages(); + const score = overrides.score ?? scoreRun(stages); + const failureCategory = overrides.failureCategory ?? (score === 100 ? 'none' : 'model-behavior'); + return { + runIndex: 1, + passed: failureCategory === 'none', + score, + countedForRecommendation: isCountedForRecommendation(failureCategory), + outcome: getRunOutcome(failureCategory), + failureCategory, + primaryFailure: null, + durationMs: 1_000, + hardFailure: failureCategory !== 'none' && failureCategory !== 'provider-infra', + stageDurationsMs: { + launchBootstrap: 100, + directReply: 100, + }, + stageFailures: {}, + taskRefChecks: { + directReply: true, + peerRelayAB: true, + peerRelayBC: true, + concurrentBob: true, + concurrentTom: true, + }, + protocolViolations: { + badMessages: 0, + duplicateOrMissingTokens: [], + }, + stages, + diagnostics: [], + ...overrides, + }; +} + +function createTestGauntletModel(input: { runs: RunGauntletResult[] }): ModelGauntletResult { + const scores = input.runs.map((run) => run.score); + const countedRuns = input.runs.filter((run) => run.countedForRecommendation); + const countedScores = countedRuns.map((run) => run.score); + const durations = input.runs.map((run) => run.durationMs).sort((left, right) => left - right); + const behavioralAverageScore = + countedScores.length > 0 + ? round(countedScores.reduce((sum, score) => sum + score, 0) / countedScores.length, 1) + : null; + const averageScore = + scores.length > 0 ? round(scores.reduce((sum, score) => sum + score, 0) / scores.length, 1) : 0; + const providerInfraFailures = input.runs.filter( + (run) => run.failureCategory === 'provider-infra' + ).length; + const runtimeTransportFailures = input.runs.filter( + (run) => run.failureCategory === 'runtime-transport' + ).length; + const modelBehaviorFailures = input.runs.filter( + (run) => run.failureCategory === 'model-behavior' + ).length; + const harnessFailures = input.runs.filter( + (run) => run.failureCategory === 'harness-error' + ).length; + const successfulRuns = input.runs.filter((run) => run.passed).length; + const hardFailures = input.runs.filter((run) => run.hardFailure).length; + const stagePassRates = summarizeStagePassRates(input.runs); + const taskRefPassRates = summarizeTaskRefPassRates(input.runs); + const protocolViolationTotals = summarizeProtocolViolations(input.runs); + const stageFailureImpact = summarizeStageFailureImpact(input.runs, stagePassRates); + const scoreStability = summarizeScoreStability(countedScores); + const qualified = isModelQualified({ + averageScore, + successfulRuns, + minimumAverageScore: 90, + minimumSuccessfulRuns: 3, + minimumConsistencyScore: 85, + hardFailures, + providerInfraFailures, + harnessFailures, + consistencyScore: scoreStability.consistencyScore, + }); + return { + model: 'openrouter/test/model', + verdict: getModelGauntletVerdict({ + qualified, + behavioralAverageScore, + successfulRuns, + countedRuns: countedRuns.length, + totalRuns: input.runs.length, + providerInfraFailures, + runtimeTransportFailures, + modelBehaviorFailures, + harnessFailures, + }), + confidence: getModelGauntletConfidence({ + countedRuns: countedRuns.length, + providerInfraFailures, + harnessFailures, + }), + qualified, + readinessScore: calculateReadinessScore({ + behavioralAverageScore, + successfulRuns, + countedRuns: countedRuns.length, + providerInfraFailures, + totalRuns: input.runs.length, + taskRefPassRates, + protocolViolationTotals, + consistencyScore: scoreStability.consistencyScore, + }), + consistencyScore: scoreStability.consistencyScore, + averageScore, + behavioralAverageScore, + minScore: scores.length > 0 ? Math.min(...scores) : 0, + successfulRuns, + countedRuns: countedRuns.length, + hardFailures, + providerInfraFailures, + runtimeTransportFailures, + modelBehaviorFailures, + harnessFailures, + p50DurationMs: percentile(durations, 0.5), + p95DurationMs: percentile(durations, 0.95), + stagePassRates, + taskRefPassRates, + protocolViolationTotals, + stageFailureImpact, + scoreStability, + dominantFailureCategory: getDominantFailureCategory(input.runs), + recommendationBlockers: buildRecommendationBlockers({ + qualified, + averageScore, + behavioralAverageScore, + successfulRuns, + minimumAverageScore: 90, + minimumSuccessfulRuns: 3, + minimumConsistencyScore: 85, + hardFailures, + providerInfraFailures, + runtimeTransportFailures, + modelBehaviorFailures, + harnessFailures, + consistencyScore: scoreStability.consistencyScore, + stageFailureImpact, + taskRefPassRates, + protocolViolationTotals, + }), + runs: input.runs, + }; +} diff --git a/test/main/services/team/OpenCodeSemanticModelMatrix.live.test.ts b/test/main/services/team/OpenCodeSemanticModelMatrix.live.test.ts index c9a14492..cbfb115e 100644 --- a/test/main/services/team/OpenCodeSemanticModelMatrix.live.test.ts +++ b/test/main/services/team/OpenCodeSemanticModelMatrix.live.test.ts @@ -106,6 +106,7 @@ async function runModelScenario(input: { harness = await createOpenCodeLiveHarness({ tempDir, selectedModel: input.model, + projectPath, }); const progressEvents: TeamProvisioningProgress[] = []; diff --git a/test/main/services/team/openCodeLiveTestHarness.ts b/test/main/services/team/openCodeLiveTestHarness.ts index 82e15ead..3f450222 100644 --- a/test/main/services/team/openCodeLiveTestHarness.ts +++ b/test/main/services/team/openCodeLiveTestHarness.ts @@ -54,6 +54,7 @@ export interface OpenCodeLiveHarness { export async function createOpenCodeLiveHarness(input: { tempDir: string; selectedModel: string; + projectPath?: string; }): Promise { const orchestratorCli = process.env.CLAUDE_AGENT_TEAMS_ORCHESTRATOR_CLI_PATH?.trim() || DEFAULT_ORCHESTRATOR_CLI; @@ -104,6 +105,17 @@ export async function createOpenCodeLiveHarness(input: { svc, dispose: async () => { svc.setControlApiBaseUrlResolver(null); + if (input.projectPath?.trim()) { + await readinessBridge + .cleanupOpenCodeHosts({ + reason: 'test-harness-dispose', + mode: 'force', + projectPath: input.projectPath, + staleAgeMs: null, + leaseStaleAgeMs: null, + }) + .catch(() => undefined); + } await controlApi.close(); }, }; diff --git a/test/renderer/components/team/TeamModelSelectorDisabledState.test.ts b/test/renderer/components/team/TeamModelSelectorDisabledState.test.ts index 45f85136..9a6095aa 100644 --- a/test/renderer/components/team/TeamModelSelectorDisabledState.test.ts +++ b/test/renderer/components/team/TeamModelSelectorDisabledState.test.ts @@ -253,8 +253,10 @@ describe('TeamModelSelector disabled Codex models', () => { 'openrouter/openai/gpt-oss-20b:free', 'openrouter/qwen/qwen3-coder-plus', 'opencode/big-pickle', + 'opencode/minimax-m2.5-free', 'openrouter/openai/gpt-oss-120b:free', - 'openrouter/qwen/qwen3-coder-flash', + 'openrouter/mistralai/codestral-2508', + 'openrouter/anthropic/claude-sonnet-4.6', ], modelVerificationState: 'idle', modelAvailability: [], @@ -278,11 +280,13 @@ describe('TeamModelSelector disabled Codex models', () => { await Promise.resolve(); }); - expect(host.textContent).toContain('Recommended only'); - expect(host.textContent).toContain('qwen/qwen3-coder-flash'); - expect(host.textContent).toContain('Recommended'); + expect(host.textContent).toContain('anthropic/claude-sonnet-4.6'); + expect(host.textContent).toContain('Tested'); + expect(host.textContent).toContain('mistralai/codestral-2508'); + expect(host.textContent).toContain('Tested'); + expect(host.textContent).toContain('minimax-m2.5-free'); + expect(host.textContent).toContain('Tested with limits'); expect(host.textContent).toContain('openai/gpt-oss-120b:free'); - expect(host.textContent).toContain('Recommended with limits'); expect(host.textContent).toContain('big-pickle'); expect(host.textContent).toContain('qwen/qwen3-coder-plus'); expect(host.textContent).toContain('Unavailable in OpenCode'); @@ -292,38 +296,26 @@ describe('TeamModelSelector disabled Codex models', () => { const buttonTexts = Array.from(host.querySelectorAll('button')).map( (button) => button.textContent ?? '' ); - const recommendedIndex = buttonTexts.findIndex((text) => - text.includes('qwen/qwen3-coder-flash') + const sonnetIndex = buttonTexts.findIndex((text) => + text.includes('anthropic/claude-sonnet-4.6') ); + const testedIndex = buttonTexts.findIndex((text) => text.includes('mistralai/codestral-2508')); const neutralIndex = buttonTexts.findIndex((text) => text.includes('big-pickle')); - const limitedIndex = buttonTexts.findIndex((text) => - text.includes('openai/gpt-oss-120b:free') - ); + const limitedIndex = buttonTexts.findIndex((text) => text.includes('minimax-m2.5-free')); const notRecommendedIndex = buttonTexts.findIndex((text) => text.includes('openai/gpt-oss-20b:free') ); const unavailableIndex = buttonTexts.findIndex((text) => text.includes('qwen/qwen3-coder-plus') ); - expect(recommendedIndex).toBeGreaterThanOrEqual(0); - expect(limitedIndex).toBeGreaterThan(recommendedIndex); + expect(sonnetIndex).toBeGreaterThanOrEqual(0); + expect(testedIndex).toBeGreaterThanOrEqual(0); + expect(limitedIndex).toBeGreaterThan(testedIndex); expect(neutralIndex).toBeGreaterThan(limitedIndex); expect(unavailableIndex).toBeGreaterThan(neutralIndex); expect(notRecommendedIndex).toBeGreaterThan(unavailableIndex); - await act(async () => { - const checkbox = Array.from(host.querySelectorAll('button')).find( - (button) => button.getAttribute('role') === 'checkbox' - ); - checkbox?.click(); - await Promise.resolve(); - }); - - expect(host.textContent).toContain('qwen/qwen3-coder-flash'); - expect(host.textContent).toContain('openai/gpt-oss-120b:free'); - expect(host.textContent).not.toContain('big-pickle'); - expect(host.textContent).not.toContain('qwen/qwen3-coder-plus'); - expect(host.textContent).not.toContain('openai/gpt-oss-20b:free'); + expect(host.textContent).not.toContain('Recommended only'); await act(async () => { root.unmount(); @@ -375,6 +367,23 @@ describe('TeamModelSelector disabled Codex models', () => { expect(modelGrid).toBeTruthy(); expect(modelGrid?.style.maxHeight).toBe('400px'); expect(modelGrid?.className).toContain('overflow-y-auto'); + const searchInput = host.querySelector( + '[data-testid="team-model-selector-model-search"]' + ) as HTMLInputElement | null; + expect(searchInput).toBeTruthy(); + + await act(async () => { + const setValue = Object.getOwnPropertyDescriptor( + window.HTMLInputElement.prototype, + 'value' + )?.set; + setValue?.call(searchInput, '5.3'); + searchInput?.dispatchEvent(new Event('input', { bubbles: true })); + await Promise.resolve(); + }); + + expect(host.textContent).toContain('5.3 Codex'); + expect(host.textContent).not.toContain('5.4 Mini'); await act(async () => { root.unmount(); @@ -413,6 +422,7 @@ describe('TeamModelSelector disabled Codex models', () => { expect(host.textContent).toContain('5.4'); expect(host.textContent).toContain('5.3 Codex'); expect(host.textContent).not.toContain('Explicit models load from the current runtime'); + expect(host.querySelector('[data-testid="team-model-selector-model-search"]')).toBeNull(); await act(async () => { root.unmount(); diff --git a/test/renderer/features/runtime-provider-management/RuntimeProviderManagementPanelView.test.ts b/test/renderer/features/runtime-provider-management/RuntimeProviderManagementPanelView.test.ts index a128face..121308b4 100644 --- a/test/renderer/features/runtime-provider-management/RuntimeProviderManagementPanelView.test.ts +++ b/test/renderer/features/runtime-provider-management/RuntimeProviderManagementPanelView.test.ts @@ -586,8 +586,26 @@ describe('RuntimeProviderManagementPanelView', () => { }, { providerId: 'openrouter', - modelId: 'openrouter/qwen/qwen3-coder-flash', - displayName: 'qwen/qwen3-coder-flash', + modelId: 'opencode/minimax-m2.5-free', + displayName: 'minimax-m2.5-free', + sourceLabel: 'OpenCode', + free: true, + default: false, + availability: 'untested', + }, + { + providerId: 'openrouter', + modelId: 'openrouter/mistralai/codestral-2508', + displayName: 'mistralai/codestral-2508', + sourceLabel: 'OpenRouter', + free: false, + default: false, + availability: 'untested', + }, + { + providerId: 'openrouter', + modelId: 'openrouter/anthropic/claude-sonnet-4.6', + displayName: 'anthropic/claude-sonnet-4.6', sourceLabel: 'OpenRouter', free: false, default: false, @@ -621,11 +639,12 @@ describe('RuntimeProviderManagementPanelView', () => { expect(host.textContent).toContain('openrouter/openai/gpt-oss-20b:free'); expect(host.textContent).toContain('Used for new teams'); expect(host.textContent).toContain('Model probe passed'); + expect(host.textContent).toContain('Recommended'); expect(host.textContent).toContain('Not recommended'); expect(host.textContent).toContain('Unavailable in OpenCode'); - expect(host.textContent).toContain('Recommended only'); - expect(host.textContent).toContain('Recommended'); - expect(host.textContent).toContain('Recommended with limits'); + expect(host.textContent).toContain('Tested'); + expect(host.textContent).toContain('Tested with limits'); + expect(host.textContent).not.toContain('Recommended only'); expect(host.textContent).not.toContain('Set OpenCode default'); expect( Array.from(host.querySelectorAll('button')).some( @@ -640,11 +659,8 @@ describe('RuntimeProviderManagementPanelView', () => { ) as HTMLElement | undefined; expect(connectedBadge?.style.color).toBeTruthy(); expect( - ( - host.querySelector( - '[data-testid="runtime-provider-model-search"]' - ) as HTMLElement | null - )?.style.paddingLeft + (host.querySelector('[data-testid="runtime-provider-model-search"]') as HTMLElement | null) + ?.style.paddingLeft ).toBe('42px'); expect( (host.querySelector('[data-testid="runtime-provider-model-list"]') as HTMLElement | null) @@ -659,10 +675,13 @@ describe('RuntimeProviderManagementPanelView', () => { '[data-testid="runtime-provider-model-result-openrouter/openai/gpt-oss-20b:free"]' ) as HTMLElement | null; expect(modelResult?.style.color).toBe('#86efac'); - expect((host.textContent ?? '').indexOf('qwen/qwen3-coder-flash')).toBeLessThan( - (host.textContent ?? '').indexOf('openai/gpt-oss-120b:free') + expect((host.textContent ?? '').indexOf('mistralai/codestral-2508')).toBeLessThan( + (host.textContent ?? '').indexOf('anthropic/claude-sonnet-4.6') ); - expect((host.textContent ?? '').indexOf('openai/gpt-oss-120b:free')).toBeLessThan( + expect((host.textContent ?? '').indexOf('anthropic/claude-sonnet-4.6')).toBeLessThan( + (host.textContent ?? '').indexOf('minimax-m2.5-free') + ); + expect((host.textContent ?? '').indexOf('minimax-m2.5-free')).toBeLessThan( (host.textContent ?? '').indexOf('opencode/big-pickle') ); expect((host.textContent ?? '').indexOf('opencode/big-pickle')).toBeLessThan( @@ -671,29 +690,6 @@ describe('RuntimeProviderManagementPanelView', () => { expect((host.textContent ?? '').indexOf('qwen/qwen3-coder-plus')).toBeLessThan( (host.textContent ?? '').indexOf('openrouter/openai/gpt-oss-20b:free') ); - - await act(async () => { - const checkbox = Array.from(host.querySelectorAll('button')).find( - (button) => button.getAttribute('role') === 'checkbox' - ); - checkbox?.click(); - await Promise.resolve(); - }); - - expect(host.textContent).toContain('qwen/qwen3-coder-flash'); - expect(host.textContent).toContain('openai/gpt-oss-120b:free'); - expect(host.textContent).not.toContain('opencode/big-pickle'); - expect(host.textContent).not.toContain('qwen/qwen3-coder-plus'); - expect(host.textContent).not.toContain('openrouter/openai/gpt-oss-20b:free'); - - await act(async () => { - const checkbox = Array.from(host.querySelectorAll('button')).find( - (button) => button.getAttribute('role') === 'checkbox' - ); - checkbox?.click(); - await Promise.resolve(); - }); - await act(async () => { host .querySelector( @@ -854,14 +850,10 @@ describe('RuntimeProviderManagementPanelView', () => { expect(logo).not.toBeNull(); expect(logo?.className).toContain('runtime-provider-brand-icon'); expect(logo?.querySelector('svg,img')).not.toBeNull(); - expect(logo?.getAttribute('style')).toContain( - '--runtime-provider-brand-fallback-background' - ); + expect(logo?.getAttribute('style')).toContain('--runtime-provider-brand-fallback-background'); expect(logo?.getAttribute('style')).toContain('--runtime-provider-brand-fallback-border'); if (logo?.querySelector('svg')) { - expect(logo.getAttribute('style')).toContain( - '--runtime-provider-brand-fallback-color' - ); + expect(logo.getAttribute('style')).toContain('--runtime-provider-brand-fallback-color'); } } }); diff --git a/test/renderer/utils/openCodeModelRecommendations.test.ts b/test/renderer/utils/openCodeModelRecommendations.test.ts index 73c86344..95440e57 100644 --- a/test/renderer/utils/openCodeModelRecommendations.test.ts +++ b/test/renderer/utils/openCodeModelRecommendations.test.ts @@ -7,104 +7,94 @@ import { } from '@renderer/utils/openCodeModelRecommendations'; describe('getOpenCodeTeamModelRecommendation', () => { - it('marks models that passed real OpenCode Agent Teams E2E as recommended', () => { - expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-coder-flash')).toMatchObject({ - level: 'recommended', - label: 'Recommended', + it('keeps Claude Sonnet 4.6 as tested while recommendations are disabled', () => { + expect( + getOpenCodeTeamModelRecommendation('openrouter/anthropic/claude-sonnet-4.6') + ).toMatchObject({ + level: 'tested', + label: 'Tested', + }); + expect(isOpenCodeTeamModelRecommended('openrouter/anthropic/claude-sonnet-4.6')).toBe(false); + }); + + it('marks models that passed real OpenCode Agent Teams smoke E2E as tested', () => { + expect(getOpenCodeTeamModelRecommendation('openrouter/mistralai/codestral-2508')).toMatchObject({ + level: 'tested', + label: 'Tested', }); expect( getOpenCodeTeamModelRecommendation(' OPENROUTER/GOOGLE/GEMINI-3-FLASH-PREVIEW ') ).toMatchObject({ - level: 'recommended', - label: 'Recommended', + level: 'tested', + label: 'Tested', }); expect(getOpenCodeTeamModelRecommendation('openrouter/moonshotai/kimi-k2.6')).toMatchObject({ - level: 'recommended', - label: 'Recommended', + level: 'tested', + label: 'Tested', }); expect(getOpenCodeTeamModelRecommendation('openrouter/z-ai/glm-5.1')).toMatchObject({ - level: 'recommended', - label: 'Recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/z-ai/glm-5')).toMatchObject({ - level: 'recommended', - label: 'Recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/minimax/minimax-m2.7')).toMatchObject({ - level: 'recommended', - label: 'Recommended', - }); - expect( - getOpenCodeTeamModelRecommendation('openrouter/google/gemini-3.1-pro-preview') - ).toMatchObject({ - level: 'recommended', - label: 'Recommended', - }); - expect( - getOpenCodeTeamModelRecommendation('openrouter/anthropic/claude-sonnet-4.6') - ).toMatchObject({ - level: 'recommended', - label: 'Recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/anthropic/claude-opus-4.6')).toMatchObject({ - level: 'recommended', - label: 'Recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/anthropic/claude-opus-4.7')).toMatchObject({ - level: 'recommended', - label: 'Recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/mistralai/devstral-2512')).toMatchObject({ - level: 'recommended', - label: 'Recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-5.4')).toMatchObject({ - level: 'recommended', - label: 'Recommended', + level: 'tested', + label: 'Tested', }); expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-5.3-codex')).toMatchObject({ - level: 'recommended', - label: 'Recommended', + level: 'tested', + label: 'Tested', }); - expect(getOpenCodeTeamModelRecommendation('openrouter/x-ai/grok-4-fast')).toMatchObject({ - level: 'recommended', - label: 'Recommended', + expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-5-nano')).toMatchObject({ + level: 'tested', + label: 'Tested', }); - expect(getOpenCodeTeamModelRecommendation('openrouter/x-ai/grok-4.1-fast')).toMatchObject({ - level: 'recommended', - label: 'Recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/xiaomi/mimo-v2-pro')).toMatchObject({ - level: 'recommended', - label: 'Recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-5.1-codex')).toMatchObject({ - level: 'recommended', - label: 'Recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-max')).toMatchObject({ - level: 'recommended', - label: 'Recommended', + expect(getOpenCodeTeamModelRecommendation('openrouter/minimax/minimax-m2')).toMatchObject({ + level: 'tested', + label: 'Tested', }); expect( - getOpenCodeTeamModelRecommendation('openrouter/mistralai/mistral-medium-3.1') + getOpenCodeTeamModelRecommendation('openrouter/google/gemma-4-26b-a4b-it') ).toMatchObject({ - level: 'recommended', - label: 'Recommended', + level: 'tested', + label: 'Tested', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/x-ai/grok-4.1-fast')).toMatchObject({ + level: 'tested', + label: 'Tested', }); expect( getOpenCodeTeamModelRecommendation('openrouter/google/gemini-3.1-flash-lite-preview') ).toMatchObject({ - level: 'recommended', - label: 'Recommended', + level: 'tested', + label: 'Tested', }); - expect(isOpenCodeTeamModelRecommended('openrouter/qwen/qwen3-coder-flash')).toBe(true); + expect(getOpenCodeTeamModelRecommendation('openrouter/z-ai/glm-4.7-flash')).toMatchObject({ + level: 'tested', + label: 'Tested', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/z-ai/glm-4.7')).toMatchObject({ + level: 'tested', + label: 'Tested', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/stepfun/step-3.5-flash')).toMatchObject({ + level: 'tested', + label: 'Tested', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/minimax/minimax-m2.1')).toMatchObject({ + level: 'tested', + label: 'Tested', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/minimax/minimax-m2.7')).toMatchObject({ + level: 'tested', + label: 'Tested', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/xiaomi/mimo-v2-pro')).toMatchObject({ + level: 'tested', + label: 'Tested', + }); + expect(isOpenCodeTeamModelRecommended('openrouter/mistralai/codestral-2508')).toBe(false); }); it('keeps similarly named models distinct when real E2E disagreed', () => { expect(getOpenCodeTeamModelRecommendation('opencode/minimax-m2.5-free')).toMatchObject({ - level: 'recommended-with-limits', - label: 'Recommended with limits', + level: 'tested-with-limits', + label: 'Tested with limits', }); expect( getOpenCodeTeamModelRecommendation('openrouter/minimax/minimax-m2.5:free') @@ -113,53 +103,50 @@ describe('getOpenCodeTeamModelRecommendation', () => { }); }); - it('marks passing free routes as recommended with limits', () => { - expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-oss-120b:free')).toMatchObject( - { - level: 'recommended-with-limits', - label: 'Recommended with limits', - } - ); - expect(isOpenCodeTeamModelRecommended('openrouter/openai/gpt-oss-120b:free')).toBe(true); + it('marks models with real launch or messaging failures as not recommended', () => { + for (const modelId of [ + 'openrouter/openai/gpt-oss-20b:free', + 'openrouter/openai/gpt-oss-120b:free', + 'openrouter/google/gemini-3-pro-preview', + 'openrouter/google/gemini-2.5-flash-lite', + 'openrouter/deepseek/deepseek-v3.2', + 'openrouter/x-ai/grok-code-fast-1', + 'openrouter/openai/gpt-5.1', + 'openrouter/openai/gpt-5.4', + 'openrouter/z-ai/glm-5-turbo', + 'openrouter/qwen/qwen3.6-plus', + 'openrouter/qwen/qwen3-coder-flash', + 'openrouter/qwen/qwen3-coder', + 'openrouter/google/gemini-3.1-pro-preview', + 'openrouter/anthropic/claude-opus-4.6', + 'openrouter/mistralai/mistral-medium-3', + 'openrouter/nvidia/nemotron-nano-9b-v2', + 'openrouter/liquid/lfm-2.5-1.2b-thinking:free', + 'openrouter/deepseek/deepseek-r1-distill-llama-70b', + ]) { + expect(getOpenCodeTeamModelRecommendation(modelId)).toMatchObject({ + level: 'not-recommended', + label: 'Not recommended', + }); + } }); - it('marks models with real launch or messaging failures as not recommended', () => { - expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-oss-20b:free')).toMatchObject({ - level: 'not-recommended', - label: 'Not recommended', - }); - expect( - getOpenCodeTeamModelRecommendation('openrouter/google/gemini-3-pro-preview') - ).toMatchObject({ - level: 'not-recommended', - label: 'Not recommended', - }); - expect( - getOpenCodeTeamModelRecommendation('openrouter/google/gemini-2.5-flash-lite') - ).toMatchObject({ - level: 'not-recommended', - label: 'Not recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/deepseek/deepseek-v3.2')).toMatchObject({ - level: 'not-recommended', - label: 'Not recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/x-ai/grok-code-fast-1')).toMatchObject({ - level: 'not-recommended', - label: 'Not recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-5.2-codex')).toMatchObject({ - level: 'not-recommended', - label: 'Not recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/moonshotai/kimi-k2-thinking')).toMatchObject({ - level: 'not-recommended', - label: 'Not recommended', - }); - expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-5.1-chat')).toMatchObject({ - level: 'not-recommended', - label: 'Not recommended', - }); + it('does not mark provider allowance or depleted-credit failures as model verdicts', () => { + for (const modelId of [ + 'openrouter/openai/gpt-5.1-codex-max', + 'openrouter/openai/gpt-5.2-chat', + 'openrouter/x-ai/grok-3', + 'openrouter/anthropic/claude-sonnet-4', + 'openrouter/anthropic/claude-opus-4.1', + 'openrouter/openai/gpt-5-pro', + 'openrouter/openai/gpt-5.2-pro', + 'openrouter/openai/gpt-5.4-pro', + 'openrouter/moonshotai/kimi-k2.5', + 'openrouter/openai/gpt-5-mini', + 'openrouter/google/gemini-2.5-flash-lite-preview-09-2025', + ]) { + expect(getOpenCodeTeamModelRecommendation(modelId)).toBeNull(); + } }); it('marks OpenRouter routes missing from the OpenCode catalog as unavailable, not bad', () => { @@ -171,6 +158,124 @@ describe('getOpenCodeTeamModelRecommendation', () => { level: 'unavailable-in-opencode', label: 'Unavailable in OpenCode', }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-coder:free')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-next-80b-a3b-instruct:free') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/google/gemini-2.0-flash-lite-001') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-4.1-nano')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-4o-mini-2024-07-18') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-4o-mini-search-preview') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen-plus')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen-turbo')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-235b-a22b-2507') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/deepseek/deepseek-v3.2-exp') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-32b')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-14b')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-8b')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwq-32b')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/deepseek/deepseek-chat')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/mistralai/mistral-nemo')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/mistralai/mistral-small-24b-instruct-2501') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/cohere/command-r7b-12-2024') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/cohere/command-r-08-2024')).toMatchObject( + { + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + } + ); + expect(getOpenCodeTeamModelRecommendation('openrouter/rekaai/reka-flash-3')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/rekaai/reka-edge')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/nvidia/nemotron-3-nano-30b-a3b') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/minimax/minimax-01')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/nvidia/llama-3.3-nemotron-super-49b-v1.5') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-max-thinking')).toMatchObject({ level: 'unavailable-in-opencode', label: 'Unavailable in OpenCode', @@ -181,35 +286,528 @@ describe('getOpenCodeTeamModelRecommendation', () => { level: 'unavailable-in-opencode', label: 'Unavailable in OpenCode', }); - expect(getOpenCodeTeamModelRecommendation('openrouter/mistralai/devstral-medium')).toMatchObject( + expect( + getOpenCodeTeamModelRecommendation('openrouter/mistralai/devstral-medium') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/mistralai/devstral-small')).toMatchObject( { level: 'unavailable-in-opencode', label: 'Unavailable in OpenCode', } ); + expect( + getOpenCodeTeamModelRecommendation('openrouter/mistralai/ministral-14b-2512') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/mistralai/ministral-8b-2512') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/mistralai/ministral-3b-2512') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/minimax/minimax-m2-her')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/xiaomi/mimo-v2.5')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/xiaomi/mimo-v2.5-pro')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/x-ai/grok-4.20')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/google/gemini-3.1-flash-image-preview') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/z-ai/glm-5v-turbo')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/x-ai/grok-4.20-multi-agent') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/mistralai/mistral-small-creative') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-5.3-chat')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/mistralai/voxtral-small-24b-2507') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-5-chat')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen-2.5-72b-instruct') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/tngtech/deepseek-r1t2-chimera') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/google/gemini-2.5-pro-preview') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/mistralai/mistral-saba')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/mistralai/mistral-large-2411') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-vl-30b-a3b-instruct') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/inclusionai/ling-2.6-1t:free') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/inclusionai/ling-2.6-flash:free') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/meta-llama/llama-3.1-8b-instruct') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen-2.5-7b-instruct') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/amazon/nova-lite-v1')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/z-ai/glm-4-32b')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/bytedance-seed/seed-1.6-flash') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/meta-llama/llama-4-scout')).toMatchObject( + { + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + } + ); + expect( + getOpenCodeTeamModelRecommendation('openrouter/meta-llama/llama-3.3-70b-instruct') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/bytedance-seed/seed-2.0-mini') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-vl-32b-instruct') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/alibaba/tongyi-deepresearch-30b-a3b') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/arcee-ai/trinity-large-preview') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/amazon/nova-micro-v1')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/arcee-ai/trinity-mini')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3.5-9b')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/essentialai/rnj-1-instruct') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/upstage/solar-pro-3')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/allenai/olmo-3.1-32b-instruct') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen-plus-2025-07-28') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/tencent/hy3-preview:free')).toMatchObject( + { + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + } + ); + expect( + getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-vl-8b-instruct') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/nex-agi/deepseek-v3.1-nex-n1') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/baidu/ernie-4.5-vl-28b-a3b') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/thedrummer/rocinante-12b')).toMatchObject( + { + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + } + ); + expect( + getOpenCodeTeamModelRecommendation('openrouter/meta-llama/llama-3.1-70b-instruct') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen-plus-2025-07-28:thinking') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/z-ai/glm-4.6v')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/anthropic/claude-3-haiku')).toMatchObject( + { + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + } + ); + expect( + getOpenCodeTeamModelRecommendation('openrouter/bytedance-seed/seed-2.0-lite') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-235b-a22b')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3.5-122b-a10b')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/deepseek/deepseek-r1-0528') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/amazon/nova-2-lite-v1')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/openai/o3-mini')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/mistralai/mistral-large-2407') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/thedrummer/unslopnemo-12b') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-vl-235b-a22b-instruct') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-vl-8b-thinking') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/kwaipilot/kat-coder-pro-v2') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/openai/o4-mini-high')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/openai/o3-mini-high')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-4o')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/cohere/command-r-plus-08-2024') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-vl-30b-a3b-thinking') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/sao10k/l3.1-euryale-70b')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3.5-27b')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/arcee-ai/virtuoso-large')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-3.5-turbo')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/bytedance-seed/seed-1.6')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/nvidia/llama-3.1-nemotron-70b-instruct') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen-vl-max')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-vl-235b-a22b-thinking') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/openai/gpt-audio-mini')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/amazon/nova-pro-v1')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/relace/relace-search')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen-max')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/mistralai/pixtral-large-2411') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/mistralai/mixtral-8x22b-instruct') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3.5-35b-a3b')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/qwen/qwen3-30b-a3b')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect(getOpenCodeTeamModelRecommendation('openrouter/baidu/ernie-4.5-21b-a3b')).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + expect( + getOpenCodeTeamModelRecommendation('openrouter/nousresearch/hermes-4-70b') + ).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + for (const modelId of [ + 'openrouter/openai/gpt-3.5-turbo-16k', + 'openrouter/mistralai/mistral-large', + 'openrouter/openai/o4-mini-deep-research', + 'openrouter/ai21/jamba-large-1.7', + 'openrouter/openai/o3', + 'openrouter/openai/gpt-audio', + 'openrouter/openai/gpt-4o-audio-preview', + 'openrouter/openai/gpt-4o-2024-11-20', + 'openrouter/openai/gpt-4o-2024-08-06', + 'openrouter/amazon/nova-premier-v1', + 'openrouter/anthropic/claude-3.7-sonnet:thinking', + 'openrouter/openai/gpt-4o-2024-05-13', + 'openrouter/~anthropic/claude-opus-latest', + 'openrouter/openai/gpt-4-turbo', + 'openrouter/openai/gpt-4-turbo-preview', + 'openrouter/openai/gpt-4-1106-preview', + 'openrouter/openai/o3-deep-research', + 'openrouter/openai/o1', + 'openrouter/openai/o3-pro', + 'openrouter/anthropic/claude-opus-4.6-fast', + 'openrouter/openai/gpt-5.5-pro', + 'openrouter/openrouter/auto', + 'openrouter/openrouter/pareto-code', + 'openrouter/openrouter/bodybuilder', + 'openrouter/baidu/qianfan-ocr-fast:free', + 'openrouter/google/lyria-3-pro-preview', + 'openrouter/ibm-granite/granite-4.0-h-micro', + 'openrouter/microsoft/phi-4', + 'openrouter/meta-llama/llama-guard-4-12b', + 'openrouter/qwen/qwen-vl-plus', + 'openrouter/deepseek/deepseek-r1-distill-qwen-32b', + 'openrouter/google/lyria-3-clip-preview', + 'openrouter/liquid/lfm-2-24b-a2b', + 'openrouter/meta-llama/llama-3.2-1b-instruct', + 'openrouter/bytedance/ui-tars-1.5-7b', + 'openrouter/baidu/ernie-4.5-21b-a3b-thinking', + 'openrouter/arcee-ai/spotlight', + 'openrouter/meta-llama/llama-3.2-3b-instruct', + 'openrouter/meta-llama/llama-guard-3-8b', + 'openrouter/nousresearch/hermes-3-llama-3.1-70b', + 'openrouter/allenai/olmo-3-32b-think', + 'openrouter/tencent/hunyuan-a13b-instruct', + 'openrouter/meta-llama/llama-4-maverick', + 'openrouter/nvidia/nemotron-nano-12b-v2-vl', + 'openrouter/thedrummer/cydonia-24b-v4.1', + 'openrouter/microsoft/wizardlm-2-8x22b', + 'openrouter/arcee-ai/coder-large', + 'openrouter/thedrummer/skyfall-36b-v2', + 'openrouter/baidu/ernie-4.5-300b-a47b', + 'openrouter/sao10k/l3.3-euryale-70b', + ]) { + expect(getOpenCodeTeamModelRecommendation(modelId)).toMatchObject({ + level: 'unavailable-in-opencode', + label: 'Unavailable in OpenCode', + }); + } expect(isOpenCodeTeamModelRecommended('openrouter/qwen/qwen3-coder-plus')).toBe(false); }); it('does not label noisy or unproven models as good or bad', () => { expect(getOpenCodeTeamModelRecommendation('opencode/big-pickle')).toBeNull(); - expect(getOpenCodeTeamModelRecommendation('openrouter/x-ai/grok-4.20')).toBeNull(); + expect(getOpenCodeTeamModelRecommendation('openrouter/x-ai/grok-4.20-unknown')).toBeNull(); expect(getOpenCodeTeamModelRecommendation('')).toBeNull(); }); - it('sorts recommended, limited, neutral, unavailable, and not-recommended routes by status', () => { + it('sorts tested, tested-with-limits, neutral, unavailable, and not-recommended routes by status', () => { const models = [ 'openrouter/openai/gpt-oss-20b:free', 'openrouter/qwen/qwen3-coder-plus', 'opencode/big-pickle', - 'openrouter/openai/gpt-oss-120b:free', - 'openrouter/qwen/qwen3-coder-flash', + 'opencode/minimax-m2.5-free', + 'openrouter/mistralai/codestral-2508', + 'openrouter/anthropic/claude-sonnet-4.6', ]; expect( [...models].sort((left, right) => compareOpenCodeTeamModelRecommendations(left, right)) ).toEqual([ - 'openrouter/qwen/qwen3-coder-flash', - 'openrouter/openai/gpt-oss-120b:free', + 'openrouter/mistralai/codestral-2508', + 'openrouter/anthropic/claude-sonnet-4.6', + 'opencode/minimax-m2.5-free', 'opencode/big-pickle', 'openrouter/qwen/qwen3-coder-plus', 'openrouter/openai/gpt-oss-20b:free',