docs(team): checkpoint attachment plans and gauntlet results
This commit is contained in:
parent
c79333a668
commit
0146c2b600
8 changed files with 4590 additions and 107 deletions
1022
docs/team-management/agent-attachments-architecture-plan.md
Normal file
1022
docs/team-management/agent-attachments-architecture-plan.md
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,911 @@
|
|||
# Phase 1 - Attachment normalization, image optimization, budgets, and UI warnings
|
||||
|
||||
## Summary
|
||||
|
||||
Goal: make attachment intake safe before changing provider delivery paths.
|
||||
|
||||
Chosen approach: **new agent-attachments feature skeleton + renderer pica optimizer + backend budget validator + capability warnings**, with current runtime delivery behavior preserved.
|
||||
|
||||
🎯 9.4 🛡️ 9.3 🧠 5.8
|
||||
Estimated change size: `260-420` LOC.
|
||||
|
||||
This phase is intentionally conservative. It reduces crash risk from oversized image payloads without changing Claude/Codex/OpenCode runtime launch or delivery semantics.
|
||||
|
||||
## Why this phase first
|
||||
|
||||
Current attachment handling stores images as base64 in renderer and validates decoded file size only. This misses the real risk:
|
||||
|
||||
```text
|
||||
image bytes -> base64 expands by ~33% -> JSON wrapper -> stream-json stdin line
|
||||
```
|
||||
|
||||
A 20MB decoded total can become a much larger single-line JSON payload and can destabilize a long-lived lead process.
|
||||
|
||||
Phase 1 creates the safety foundation:
|
||||
|
||||
- normalize attachments;
|
||||
- optimize screenshots;
|
||||
- calculate estimated serialized payload size;
|
||||
- block too-large sends before stdin write;
|
||||
- show clear UI warnings;
|
||||
- do not change runtime adapter logic yet.
|
||||
|
||||
## Scope
|
||||
|
||||
In scope:
|
||||
|
||||
- new `src/features/agent-attachments` contracts/core shell;
|
||||
- renderer image optimization using `pica@9.0.1`;
|
||||
- new normalized attachment DTOs;
|
||||
- backend validation for image dimensions, bytes, base64 size, and estimated serialized payload;
|
||||
- UI warnings in composer;
|
||||
- tests for optimizer decisions and validation.
|
||||
|
||||
Out of scope:
|
||||
|
||||
- Codex `--image` wiring;
|
||||
- OpenCode file parts;
|
||||
- model capability catalog beyond basic warnings;
|
||||
- document/PDF optimization;
|
||||
- live provider calls.
|
||||
|
||||
## Dependency decision
|
||||
|
||||
Add:
|
||||
|
||||
```bash
|
||||
pnpm add pica@9.0.1
|
||||
```
|
||||
|
||||
Rationale:
|
||||
|
||||
- pure browser-side high-quality resize;
|
||||
- no native Electron packaging risk;
|
||||
- good quality for screenshots and UI text;
|
||||
- safer before release than `sharp` in Electron main.
|
||||
|
||||
Do not add:
|
||||
|
||||
- `sharp` in Electron main in this phase;
|
||||
- `@squoosh/lib` due staleness/complexity;
|
||||
- `jimp` due lower quality/performance for screenshots.
|
||||
|
||||
## New feature layout
|
||||
|
||||
```text
|
||||
src/features/agent-attachments/
|
||||
contracts/
|
||||
api.ts
|
||||
dto.ts
|
||||
channels.ts
|
||||
core/
|
||||
domain/
|
||||
AttachmentBudget.ts
|
||||
AttachmentModel.ts
|
||||
AttachmentValidation.ts
|
||||
application/
|
||||
AttachmentIntakePolicy.ts
|
||||
AttachmentBudgetEstimator.ts
|
||||
main/
|
||||
composition/
|
||||
createAgentAttachmentsFeature.ts
|
||||
adapters/
|
||||
input/ipc/registerAgentAttachmentIpc.ts
|
||||
infrastructure/
|
||||
ServerAttachmentValidator.ts
|
||||
preload/
|
||||
createAgentAttachmentsBridge.ts
|
||||
renderer/
|
||||
hooks/useAttachmentPreparation.ts
|
||||
ui/AttachmentCapabilityNotice.tsx
|
||||
utils/picaImageOptimizer.ts
|
||||
```
|
||||
|
||||
If this feels too much for phase 1, contracts/domain/application can be created first and IPC can be deferred. But the boundaries should be established now.
|
||||
|
||||
## Contract DTOs
|
||||
|
||||
```ts
|
||||
export type AgentAttachmentKind = 'image' | 'document' | 'text' | 'unsupported';
|
||||
|
||||
export interface AgentAttachmentDraftDto {
|
||||
id: string;
|
||||
filename: string;
|
||||
mimeType: string;
|
||||
kind: AgentAttachmentKind;
|
||||
originalBytes: number;
|
||||
dataBase64: string;
|
||||
width?: number;
|
||||
height?: number;
|
||||
optimized?: AgentAttachmentOptimizedVariantDto;
|
||||
warnings: AgentAttachmentWarningDto[];
|
||||
}
|
||||
|
||||
export interface AgentAttachmentOptimizedVariantDto {
|
||||
mimeType: 'image/jpeg' | 'image/png' | 'image/webp';
|
||||
dataBase64: string;
|
||||
bytes: number;
|
||||
width: number;
|
||||
height: number;
|
||||
quality?: number;
|
||||
strategy: 'unchanged' | 'resized' | 'converted' | 'resized-and-converted';
|
||||
}
|
||||
|
||||
export interface AgentAttachmentWarningDto {
|
||||
code:
|
||||
| 'image_resized'
|
||||
| 'image_quality_reduced'
|
||||
| 'image_too_large'
|
||||
| 'animated_gif_unchanged'
|
||||
| 'unsupported_mime_type'
|
||||
| 'serialized_payload_too_large';
|
||||
severity: 'info' | 'warning' | 'error';
|
||||
message: string;
|
||||
}
|
||||
```
|
||||
|
||||
## Budget constants
|
||||
|
||||
Start conservative. These can be tuned after e2e.
|
||||
|
||||
```ts
|
||||
export const AGENT_ATTACHMENT_BUDGETS = {
|
||||
maxFiles: 5,
|
||||
maxOriginalFileBytes: 10 * 1024 * 1024,
|
||||
maxTotalOriginalBytes: 20 * 1024 * 1024,
|
||||
maxOptimizedImageBytes: 1_500_000,
|
||||
maxTotalOptimizedBytes: 4_000_000,
|
||||
maxEstimatedStreamJsonPayloadBytes: 7_500_000,
|
||||
maxDecodedMegapixels: 24,
|
||||
maxLongEdgePx: 2000,
|
||||
minJpegQuality: 0.72,
|
||||
initialJpegQuality: 0.88,
|
||||
} as const;
|
||||
```
|
||||
|
||||
Rationale:
|
||||
|
||||
- Claude Code docs mention 10MB stdin limit for headless input modes. Use `7.5MB` app budget to leave JSON/base64 overhead headroom.
|
||||
- Multiple images need a total optimized budget, not only per-image limits.
|
||||
- Screenshots need enough resolution to read text, so do not crush quality below `0.72` silently.
|
||||
|
||||
## Renderer optimizer policy
|
||||
|
||||
Use `pica` only for images where this is safe.
|
||||
|
||||
```ts
|
||||
export async function optimizeImageForAgentAttachment(
|
||||
input: BrowserImageInput,
|
||||
policy = DEFAULT_IMAGE_OPTIMIZATION_POLICY,
|
||||
): Promise<AgentAttachmentOptimizedVariantDto> {
|
||||
if (input.mimeType === 'image/gif') {
|
||||
return keepOriginalWithWarning('animated_gif_unchanged');
|
||||
}
|
||||
|
||||
if (input.hasAlpha) {
|
||||
return resizePngPreservingAlpha(input, policy);
|
||||
}
|
||||
|
||||
return resizeRgbScreenshotToJpeg(input, policy);
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Preserve aspect ratio.
|
||||
- Preserve alpha by staying PNG unless output exceeds budget and user must choose a lower-fidelity conversion explicitly later.
|
||||
- Do not silently convert animated GIF to a still image.
|
||||
- Prefer JPEG for large RGB screenshots.
|
||||
- Try qualities in bounded steps: `0.88`, `0.82`, `0.76`, `0.72`.
|
||||
- If still too large, show error instead of making unreadable images.
|
||||
|
||||
## Payload size estimator
|
||||
|
||||
Do not rely only on decoded bytes.
|
||||
|
||||
```ts
|
||||
export function estimateStreamJsonPayloadBytes(input: {
|
||||
text: string;
|
||||
attachments: AgentAttachmentDraftDto[];
|
||||
}): number {
|
||||
const contentBlocks = input.attachments.map(attachment => ({
|
||||
type: attachment.kind === 'image' ? 'image' : 'document',
|
||||
source: {
|
||||
type: 'base64',
|
||||
media_type: attachment.optimized?.mimeType ?? attachment.mimeType,
|
||||
data: attachment.optimized?.dataBase64 ?? attachment.dataBase64,
|
||||
},
|
||||
}));
|
||||
|
||||
return Buffer.byteLength(JSON.stringify({
|
||||
type: 'user',
|
||||
message: {
|
||||
role: 'user',
|
||||
content: [{ type: 'text', text: input.text }, ...contentBlocks],
|
||||
},
|
||||
}), 'utf8');
|
||||
}
|
||||
```
|
||||
|
||||
This estimator lives in shared/core if it avoids Node-only APIs, or duplicated as pure helper with `TextEncoder` for renderer and `Buffer.byteLength` for main. Prefer pure `TextEncoder` for cross-process reuse.
|
||||
|
||||
## Backend validation
|
||||
|
||||
The backend must revalidate everything because renderer optimization is not a security boundary.
|
||||
|
||||
```ts
|
||||
export function validateAgentAttachmentsForSend(input: {
|
||||
text: string;
|
||||
attachments: AgentAttachmentDraftDto[];
|
||||
runtimeHint: RuntimeAttachmentHint;
|
||||
}): ValidationResult {
|
||||
if (input.attachments.length > AGENT_ATTACHMENT_BUDGETS.maxFiles) {
|
||||
return error('Too many attachments.');
|
||||
}
|
||||
|
||||
const estimatedBytes = estimateStreamJsonPayloadBytes(input);
|
||||
if (estimatedBytes > AGENT_ATTACHMENT_BUDGETS.maxEstimatedStreamJsonPayloadBytes) {
|
||||
return error(
|
||||
`Attachments are too large after optimization (${formatBytes(estimatedBytes)} serialized). ` +
|
||||
`Remove an image or reduce screenshot size.`,
|
||||
);
|
||||
}
|
||||
|
||||
return ok();
|
||||
}
|
||||
```
|
||||
|
||||
For phase 1, wire this into existing `validateAttachments` before `sendMessageToTeam` accepts attachments.
|
||||
|
||||
## Composer UI behavior
|
||||
|
||||
Add a small notice near attachment previews.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
Screenshot optimized to 1920x1080 JPEG, 612 KB.
|
||||
```
|
||||
|
||||
```text
|
||||
Attachments are too large after optimization. Remove one image or use a smaller screenshot.
|
||||
```
|
||||
|
||||
```text
|
||||
Animated GIFs are not optimized yet and may be too large for agent delivery.
|
||||
```
|
||||
|
||||
Do not mention provider-specific capability in Phase 1 unless the target runtime is already known in composer state. The main blocker in Phase 1 is size/budget safety.
|
||||
|
||||
## Integration points
|
||||
|
||||
Existing code to adjust carefully:
|
||||
|
||||
```text
|
||||
src/renderer/utils/attachmentUtils.ts
|
||||
src/renderer/hooks/useComposerDraft.ts
|
||||
src/main/ipc/teams.ts
|
||||
src/main/services/team/TeamProvisioningService.ts
|
||||
```
|
||||
|
||||
Do not move all logic at once. Add wrappers and leave current API shape compatible.
|
||||
|
||||
## Edge cases
|
||||
|
||||
### Multiple high-resolution screenshots
|
||||
|
||||
Expected behavior:
|
||||
|
||||
- optimize each image;
|
||||
- if total serialized payload still too large, block send with clear error;
|
||||
- do not partially send only some images.
|
||||
|
||||
### Transparent PNG
|
||||
|
||||
Expected behavior:
|
||||
|
||||
- preserve PNG/alpha;
|
||||
- if too large, ask user to reduce or confirm future lossy conversion in a later phase;
|
||||
- do not silently flatten transparency.
|
||||
|
||||
### Animated GIF
|
||||
|
||||
Expected behavior:
|
||||
|
||||
- keep original if within budget;
|
||||
- otherwise block with clear message;
|
||||
- do not silently first-frame it.
|
||||
|
||||
### Corrupt image
|
||||
|
||||
Expected behavior:
|
||||
|
||||
- show `Cannot read image file`;
|
||||
- do not pass corrupt base64 to runtime.
|
||||
|
||||
### Old draft with base64-only attachment
|
||||
|
||||
Expected behavior:
|
||||
|
||||
- load draft;
|
||||
- if no optimized variant exists, optimize on send;
|
||||
- if optimization fails, block send.
|
||||
|
||||
### Unsupported file type
|
||||
|
||||
Expected behavior:
|
||||
|
||||
- existing path fallback for local files can remain;
|
||||
- unsupported binary file is not converted to base64 attachment.
|
||||
|
||||
## Test plan
|
||||
|
||||
### Unit
|
||||
|
||||
- `estimateStreamJsonPayloadBytes` includes base64 and JSON overhead.
|
||||
- RGB PNG screenshot converts/resizes to JPEG under budget.
|
||||
- Small PNG remains unchanged if already safe.
|
||||
- Alpha PNG does not become JPEG silently.
|
||||
- Animated GIF is not converted silently.
|
||||
- Corrupt image returns error.
|
||||
- Total optimized bytes over budget blocks send.
|
||||
|
||||
### Renderer
|
||||
|
||||
- composer shows optimization notice;
|
||||
- composer shows too-large error;
|
||||
- removing an attachment clears budget error;
|
||||
- old drafts trigger optimization before send.
|
||||
|
||||
### Main/IPС
|
||||
|
||||
- IPC rejects too many attachments;
|
||||
- IPC rejects payload above serialized budget;
|
||||
- IPC accepts safe optimized image;
|
||||
- error messages are user-readable and do not include base64 data.
|
||||
|
||||
Suggested focused checks:
|
||||
|
||||
```bash
|
||||
pnpm vitest run src/features/agent-attachments/**/*.test.ts test/main/ipc/teams.test.ts test/renderer/components/team/messages/MessageComposer.test.tsx
|
||||
pnpm typecheck --pretty false
|
||||
```
|
||||
|
||||
## Safety checklist
|
||||
|
||||
- No provider runtime path changed.
|
||||
- No launch/provisioning path changed.
|
||||
- Text-only messages still use old path.
|
||||
- Attachments are blocked before send if unsafe.
|
||||
- Backend validation cannot be bypassed by renderer state.
|
||||
- No secrets or base64 blobs in diagnostics.
|
||||
|
||||
## Deep implementation details
|
||||
|
||||
### Step-by-step implementation sequence
|
||||
|
||||
1. Add feature contracts and pure budget estimator.
|
||||
2. Add renderer-only `picaImageOptimizer` with no imports from main.
|
||||
3. Add backend `ServerAttachmentValidator` that can validate legacy payloads.
|
||||
4. Wire backend validator into existing IPC send path before `TeamProvisioningService.sendMessageToTeam()`.
|
||||
5. Add composer warnings from renderer optimization state.
|
||||
6. Add tests for estimator and validator.
|
||||
|
||||
This order avoids changing provider delivery until validation is proven.
|
||||
|
||||
### Pure byte estimator
|
||||
|
||||
Use a runtime-neutral helper so both renderer and main can compute comparable values.
|
||||
|
||||
```ts
|
||||
export function utf8Bytes(value: string): number {
|
||||
return new TextEncoder().encode(value).byteLength;
|
||||
}
|
||||
|
||||
export function estimateBase64JsonStringBytes(base64: string): number {
|
||||
// JSON string escaping is normally small for base64, but include quotes.
|
||||
return utf8Bytes(JSON.stringify(base64));
|
||||
}
|
||||
|
||||
export function estimateClaudeStreamJsonPayloadBytes(input: {
|
||||
text: string;
|
||||
attachments: Array<{ mimeType: string; base64: string; kind: 'image' | 'document' }>;
|
||||
}): number {
|
||||
const payload = {
|
||||
type: 'user',
|
||||
message: {
|
||||
role: 'user',
|
||||
content: [
|
||||
{ type: 'text', text: input.text },
|
||||
...input.attachments.map(att => ({
|
||||
type: att.kind === 'image' ? 'image' : 'document',
|
||||
source: {
|
||||
type: 'base64',
|
||||
media_type: att.mimeType,
|
||||
data: att.base64,
|
||||
},
|
||||
})),
|
||||
],
|
||||
},
|
||||
};
|
||||
return utf8Bytes(JSON.stringify(payload));
|
||||
}
|
||||
```
|
||||
|
||||
Avoid using `Buffer` in shared/renderer code.
|
||||
|
||||
### Renderer optimizer pseudo-code
|
||||
|
||||
```ts
|
||||
export async function prepareImageAttachmentDraft(file: File): Promise<AgentAttachmentDraftDto> {
|
||||
const originalBase64 = await readFileAsBase64(file);
|
||||
const metadata = await readImageMetadata(file);
|
||||
|
||||
if (metadata.megapixels > AGENT_ATTACHMENT_BUDGETS.maxDecodedMegapixels) {
|
||||
return errorDraft(file, 'Image resolution is too large to process safely.');
|
||||
}
|
||||
|
||||
const optimized = await optimizeImageForAgent(file, metadata);
|
||||
const warnings = buildOptimizationWarnings(file, optimized);
|
||||
|
||||
return {
|
||||
id: stableBrowserDraftId(file, originalBase64),
|
||||
filename: file.name,
|
||||
mimeType: file.type,
|
||||
kind: 'image',
|
||||
originalBytes: file.size,
|
||||
dataBase64: originalBase64,
|
||||
width: metadata.width,
|
||||
height: metadata.height,
|
||||
optimized,
|
||||
warnings,
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### Pica resize pseudo-code
|
||||
|
||||
```ts
|
||||
async function resizeRgbToJpeg(input: ImageBitmap, policy: ImagePolicy) {
|
||||
const { width, height } = fitWithinLongEdge(input.width, input.height, policy.maxLongEdgePx);
|
||||
const canvas = new OffscreenCanvas(width, height);
|
||||
await pica().resize(input, canvas, {
|
||||
quality: 3,
|
||||
alpha: false,
|
||||
unsharpAmount: 80,
|
||||
unsharpRadius: 0.6,
|
||||
unsharpThreshold: 2,
|
||||
});
|
||||
|
||||
for (const quality of [0.88, 0.82, 0.76, 0.72]) {
|
||||
const blob = await canvas.convertToBlob({ type: 'image/jpeg', quality });
|
||||
if (blob.size <= policy.maxOptimizedImageBytes) {
|
||||
return toVariant(blob, { width, height, quality, strategy: 'resized-and-converted' });
|
||||
}
|
||||
}
|
||||
|
||||
throw new AttachmentTooLargeError('Image is still too large after resizing.');
|
||||
}
|
||||
```
|
||||
|
||||
Fallback if `OffscreenCanvas` is unavailable:
|
||||
|
||||
```ts
|
||||
const canvas = document.createElement('canvas');
|
||||
canvas.width = width;
|
||||
canvas.height = height;
|
||||
await pica().resize(sourceCanvasOrImage, canvas);
|
||||
```
|
||||
|
||||
### Alpha detection
|
||||
|
||||
Do not decode full huge images on main thread just to check alpha. In renderer, after image bitmap decode and drawing to a small sampling canvas:
|
||||
|
||||
```ts
|
||||
function likelyHasAlpha(ctx: CanvasRenderingContext2D, width: number, height: number): boolean {
|
||||
const sampleWidth = Math.min(width, 256);
|
||||
const sampleHeight = Math.min(height, 256);
|
||||
const data = ctx.getImageData(0, 0, sampleWidth, sampleHeight).data;
|
||||
for (let i = 3; i < data.length; i += 4) {
|
||||
if (data[i] !== 255) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
```
|
||||
|
||||
If uncertain, prefer PNG and warn rather than silently flattening.
|
||||
|
||||
### Backend legacy payload normalization
|
||||
|
||||
```ts
|
||||
export function normalizeLegacyAttachmentPayload(input: {
|
||||
data: string;
|
||||
mimeType: string;
|
||||
filename?: string;
|
||||
}): NormalizedLegacyAttachment {
|
||||
const decodedBytes = estimateDecodedBase64Bytes(input.data);
|
||||
const kind = classifyMimeType(input.mimeType);
|
||||
|
||||
if (decodedBytes > AGENT_ATTACHMENT_BUDGETS.maxOriginalFileBytes) {
|
||||
throw new AttachmentValidationError({
|
||||
code: 'attachment_too_large_original',
|
||||
userMessage: `${input.filename ?? 'Attachment'} is too large.`,
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
id: stableAttachmentId(input),
|
||||
filename: sanitizeAttachmentFilename(input.filename),
|
||||
mimeType: input.mimeType,
|
||||
kind,
|
||||
decodedBytes,
|
||||
base64: input.data,
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### Filename sanitization
|
||||
|
||||
Never use attachment filenames directly as filesystem paths.
|
||||
|
||||
```ts
|
||||
export function sanitizeAttachmentFilename(name: string | undefined): string {
|
||||
const fallback = 'attachment';
|
||||
const base = (name ?? fallback)
|
||||
.replace(/[\\/\0\r\n\t]/g, '_')
|
||||
.replace(/^\.+$/, fallback)
|
||||
.slice(0, 120)
|
||||
.trim();
|
||||
return base || fallback;
|
||||
}
|
||||
```
|
||||
|
||||
### More edge cases
|
||||
|
||||
| Edge case | Expected behavior |
|
||||
|---|---|
|
||||
| Browser cannot decode HEIC pasted from iPhone | show unsupported image format, suggest PNG/JPEG screenshot |
|
||||
| User attaches 5 images each individually under budget but combined over budget | block whole send, show combined payload size |
|
||||
| Image has huge dimensions but tiny compressed bytes | block before decode if dimensions exceed safe megapixels |
|
||||
| File extension says `.jpg` but MIME says PNG | trust detected MIME if available, otherwise validate magic bytes in backend later |
|
||||
| Renderer optimization fails due memory pressure | keep draft but mark send-blocked with retry/remove action |
|
||||
| User edits message text after optimization | do not recompress image, only recompute serialized payload estimate |
|
||||
| User removes image | revoke object URLs and release ImageBitmap/canvas refs |
|
||||
| User switches team while optimization running | cancel or ignore stale optimization result by draft id |
|
||||
| SVG image | treat as unsupported in v1 unless converted explicitly later |
|
||||
| WebP | allow if runtime supports, otherwise convert to JPEG/PNG if safe |
|
||||
|
||||
### Bug-prevention checklist
|
||||
|
||||
- All async optimizer results must check current draft id before writing state.
|
||||
- Object URLs must be revoked on unmount/remove.
|
||||
- Do not store huge base64 in React error messages.
|
||||
- Do not include base64 in Zustand dev logs if avoidable.
|
||||
- Do not throw raw DOMException to user.
|
||||
- Backend validation must run even if renderer says optimized.
|
||||
- Tests should include both `data.length` and decoded byte calculations.
|
||||
|
||||
## File-by-file implementation plan
|
||||
|
||||
### 1. Contracts
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
src/features/agent-attachments/contracts/dto.ts
|
||||
src/features/agent-attachments/contracts/api.ts
|
||||
src/features/agent-attachments/contracts/index.ts
|
||||
```
|
||||
|
||||
Keep contracts serializable. Do not expose classes or functions that require DOM/Node.
|
||||
|
||||
Example:
|
||||
|
||||
```ts
|
||||
export interface AgentAttachmentBudgetDto {
|
||||
maxFiles: number;
|
||||
maxOriginalFileBytes: number;
|
||||
maxTotalOriginalBytes: number;
|
||||
maxOptimizedImageBytes: number;
|
||||
maxEstimatedSerializedBytes: number;
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Core domain
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
src/features/agent-attachments/core/domain/AttachmentBudget.ts
|
||||
src/features/agent-attachments/core/domain/AttachmentMime.ts
|
||||
src/features/agent-attachments/core/domain/AttachmentErrors.ts
|
||||
```
|
||||
|
||||
This layer must be pure. No `fs`, no `Electron`, no `React`, no `Buffer` if it needs renderer reuse.
|
||||
|
||||
### 3. Renderer optimizer
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
src/features/agent-attachments/renderer/utils/picaImageOptimizer.ts
|
||||
```
|
||||
|
||||
This file may import `pica`, DOM APIs, and browser canvas APIs. It must not import main process modules.
|
||||
|
||||
### 4. Existing renderer integration
|
||||
|
||||
Update carefully:
|
||||
|
||||
```text
|
||||
src/renderer/utils/attachmentUtils.ts
|
||||
src/renderer/hooks/useComposerDraft.ts
|
||||
```
|
||||
|
||||
Do not replace the whole draft flow. Add a narrow call:
|
||||
|
||||
```ts
|
||||
const prepared = await prepareAgentAttachmentDraft(file);
|
||||
```
|
||||
|
||||
### 5. Main validation
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
src/features/agent-attachments/main/infrastructure/ServerAttachmentValidator.ts
|
||||
```
|
||||
|
||||
Then call it from existing IPC validation. Do not move all IPC into the new feature in Phase 1 unless it is trivial.
|
||||
|
||||
### 6. UI warnings
|
||||
|
||||
Add small rendering components only if existing composer can consume warnings without a broad refactor.
|
||||
|
||||
Potential target:
|
||||
|
||||
```text
|
||||
src/renderer/components/team/messages/MessageComposer.tsx
|
||||
```
|
||||
|
||||
Keep UI changes minimal.
|
||||
|
||||
## Additional code examples
|
||||
|
||||
### Domain error class
|
||||
|
||||
```ts
|
||||
export class AgentAttachmentError extends Error {
|
||||
constructor(readonly failure: AttachmentFailure) {
|
||||
super(failure.userMessage);
|
||||
this.name = 'AgentAttachmentError';
|
||||
}
|
||||
}
|
||||
|
||||
export function isAgentAttachmentError(error: unknown): error is AgentAttachmentError {
|
||||
return error instanceof AgentAttachmentError;
|
||||
}
|
||||
```
|
||||
|
||||
### MIME classifier
|
||||
|
||||
```ts
|
||||
export function classifyAttachmentMimeType(mimeType: string): AgentAttachmentKind {
|
||||
const normalized = mimeType.toLowerCase();
|
||||
if (['image/png', 'image/jpeg', 'image/webp', 'image/gif'].includes(normalized)) return 'image';
|
||||
if (normalized === 'application/pdf') return 'document';
|
||||
if (normalized.startsWith('text/')) return 'text';
|
||||
return 'unsupported';
|
||||
}
|
||||
```
|
||||
|
||||
### Base64 decoded byte estimator
|
||||
|
||||
```ts
|
||||
export function estimateDecodedBase64Bytes(base64: string): number {
|
||||
const clean = base64.replace(/\s/g, '');
|
||||
const padding = clean.endsWith('==') ? 2 : clean.endsWith('=') ? 1 : 0;
|
||||
return Math.floor((clean.length * 3) / 4) - padding;
|
||||
}
|
||||
```
|
||||
|
||||
Do not decode huge base64 just to estimate size.
|
||||
|
||||
### Safe async draft update pattern
|
||||
|
||||
```ts
|
||||
const generation = ++attachmentPreparationGenerationRef.current;
|
||||
const result = await prepareAttachment(file);
|
||||
if (generation !== attachmentPreparationGenerationRef.current) {
|
||||
return; // stale result after team/message switch
|
||||
}
|
||||
setDraftAttachments(prev => [...prev, result]);
|
||||
```
|
||||
|
||||
## More detailed test cases
|
||||
|
||||
### Budget estimator table
|
||||
|
||||
| Input | Expected |
|
||||
|---|---|
|
||||
| no attachments, short text | under budget |
|
||||
| one 1MB base64 image | serialized estimate greater than decoded bytes |
|
||||
| five 1MB images | total serialized limit can fail |
|
||||
| base64 with whitespace | decoded byte estimator handles it |
|
||||
| empty base64 | invalid attachment error |
|
||||
|
||||
### Optimizer table
|
||||
|
||||
| Input | Expected |
|
||||
|---|---|
|
||||
| 320x240 PNG under budget | unchanged or tiny optimized variant |
|
||||
| 6000x4000 screenshot | resized to max long edge |
|
||||
| transparent PNG | stays PNG |
|
||||
| animated GIF | not converted, warning |
|
||||
| corrupt PNG | error draft |
|
||||
| WebP | accepted if browser decodes, otherwise unsupported |
|
||||
|
||||
### UI state table
|
||||
|
||||
| Action | Expected |
|
||||
|---|---|
|
||||
| attach image then remove | warning disappears, object URL revoked |
|
||||
| attach too-large image | send disabled with specific reason |
|
||||
| edit text after attach | only serialized estimate recalculated |
|
||||
| switch team during optimization | stale result ignored |
|
||||
| attach unsupported binary | existing path/link fallback or blocked, no base64 blob |
|
||||
|
||||
## Extra risk controls
|
||||
|
||||
- Keep old constants temporarily and map them to new budget constants to avoid conflicting limits.
|
||||
- If `pica` import increases renderer bundle unexpectedly, keep it lazy-loaded only when image attachment is selected.
|
||||
- If optimization fails unexpectedly, fail closed for attachments but do not affect text-only sends.
|
||||
- Add analytics/log event only with counts/bytes, never filenames if privacy-sensitive.
|
||||
|
||||
## Phase 1 exit criteria
|
||||
|
||||
Phase 1 is complete only when:
|
||||
|
||||
- text-only composer send is unchanged;
|
||||
- image drafts show optimized size or clear error;
|
||||
- backend rejects oversized serialized payloads;
|
||||
- renderer and backend use consistent budget constants;
|
||||
- no runtime provider delivery code is changed;
|
||||
- old legacy payload shape still works;
|
||||
- no base64/data URL appears in UI errors or logs.
|
||||
|
||||
## Migration seam from existing code
|
||||
|
||||
Existing code should be wrapped, not replaced wholesale.
|
||||
|
||||
Current likely call chain:
|
||||
|
||||
```text
|
||||
MessageComposer -> useComposerDraft -> attachmentUtils.fileToAttachmentPayload -> teams IPC -> validateAttachments -> sendMessageToTeam
|
||||
```
|
||||
|
||||
Phase 1 seam:
|
||||
|
||||
```text
|
||||
attachmentUtils.fileToAttachmentPayload
|
||||
-> prepareAgentAttachmentDraft
|
||||
-> returns legacy-compatible payload plus metadata/warnings
|
||||
|
||||
main validateAttachments
|
||||
-> ServerAttachmentValidator.validateLegacyPayloads
|
||||
```
|
||||
|
||||
Do not change `sendMessageToTeam` signature in Phase 1.
|
||||
|
||||
## More concrete backend validator
|
||||
|
||||
```ts
|
||||
export interface ServerAttachmentValidationInput {
|
||||
messageText: string;
|
||||
attachments: Array<{ data: string; mimeType: string; filename?: string }>;
|
||||
budget?: Partial<AgentAttachmentBudget>;
|
||||
}
|
||||
|
||||
export interface ServerAttachmentValidationOutput {
|
||||
ok: true;
|
||||
normalized: NormalizedLegacyAttachment[];
|
||||
estimatedSerializedBytes: number;
|
||||
warnings: AttachmentWarning[];
|
||||
} | {
|
||||
ok: false;
|
||||
failure: AttachmentFailure;
|
||||
};
|
||||
```
|
||||
|
||||
Usage:
|
||||
|
||||
```ts
|
||||
const validation = serverAttachmentValidator.validateLegacyPayloads({
|
||||
messageText,
|
||||
attachments,
|
||||
});
|
||||
if (!validation.ok) {
|
||||
throw new Error(validation.failure.userMessage);
|
||||
}
|
||||
```
|
||||
|
||||
### Validation order
|
||||
|
||||
Order matters for predictable user errors.
|
||||
|
||||
1. attachment count;
|
||||
2. base64 validity;
|
||||
3. decoded bytes per file;
|
||||
4. total decoded bytes;
|
||||
5. MIME support;
|
||||
6. estimated serialized payload bytes;
|
||||
7. warning collection.
|
||||
|
||||
Do not compute JSON payload with unbounded decoded buffers.
|
||||
|
||||
## Renderer optimizer cancellation
|
||||
|
||||
```ts
|
||||
export interface AttachmentPreparationJob {
|
||||
id: string;
|
||||
cancel(): void;
|
||||
promise: Promise<AgentAttachmentDraftDto>;
|
||||
}
|
||||
```
|
||||
|
||||
If using AbortController:
|
||||
|
||||
```ts
|
||||
const controller = new AbortController();
|
||||
const promise = prepareAgentAttachmentDraft(file, { signal: controller.signal });
|
||||
return { id, cancel: () => controller.abort(), promise };
|
||||
```
|
||||
|
||||
If pica cannot fully abort, still ignore stale results by generation id.
|
||||
|
||||
## Memory safety
|
||||
|
||||
Large images can pressure renderer memory. Keep rules strict.
|
||||
|
||||
- Reject dimensions above max megapixels before full resize when possible.
|
||||
- Release `ImageBitmap` with `imageBitmap.close()` after resize.
|
||||
- Revoke object URLs.
|
||||
- Avoid storing duplicate base64 strings if optimized variant replaces original for send.
|
||||
- Do not put raw base64 in React component props beyond draft state if avoidable.
|
||||
|
||||
## Phase 1 bug traps and prevention
|
||||
|
||||
| Trap | Prevention |
|
||||
|---|---|
|
||||
| Backend accepts unsafe payload because renderer already warned | backend validator is mandatory |
|
||||
| UI warning says optimized but send uses original huge base64 | send path chooses optimized variant or blocks |
|
||||
| GIF silently becomes static image | explicit GIF policy, test it |
|
||||
| transparent PNG becomes white/black JPEG | alpha test and PNG preservation |
|
||||
| stale optimization adds attachment to wrong team draft | generation id check |
|
||||
| file name path traversal appears in future artifact path | sanitize filenames now |
|
||||
| tests rely on browser-only APIs in Node | keep optimizer tests in jsdom/browser-compatible environment or mock pica |
|
||||
|
||||
## Extra test skeletons
|
||||
|
||||
```ts
|
||||
describe('ServerAttachmentValidator', () => {
|
||||
it('rejects payload by serialized size even when decoded bytes are under old limit', () => {
|
||||
const image = makeBase64OfSize(6_000_000);
|
||||
const result = validator.validateLegacyPayloads({
|
||||
messageText: 'x',
|
||||
attachments: [{ data: image, mimeType: 'image/png', filename: 'large.png' }],
|
||||
});
|
||||
expect(result.ok).toBe(false);
|
||||
if (!result.ok) expect(result.failure.code).toBe('attachment_serialized_payload_too_large');
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
```ts
|
||||
describe('picaImageOptimizer', () => {
|
||||
it('does not flatten transparent PNG to JPEG', async () => {
|
||||
const result = await optimizeImageForAgentAttachment(transparentPngFile);
|
||||
expect(result.mimeType).toBe('image/png');
|
||||
});
|
||||
});
|
||||
```
|
||||
|
|
@ -0,0 +1,615 @@
|
|||
# Phase 2 - Claude stream-json attachment delivery adapter
|
||||
|
||||
## Summary
|
||||
|
||||
Goal: route existing Claude lead attachment delivery through the new attachment planner, preserving current stream-json content block behavior while adding deterministic budgets and diagnostics.
|
||||
|
||||
Chosen approach: **extract current Claude serialization into `ClaudeStreamJsonAttachmentAdapter` and call it from `TeamProvisioningService.sendMessageToRun()`**.
|
||||
|
||||
🎯 9.0 🛡️ 8.8 🧠 5.8
|
||||
Estimated change size: `180-320` LOC.
|
||||
|
||||
This phase should not change launch, bootstrap, provider auth, or teammate liveness. It only replaces ad-hoc attachment block assembly with a tested adapter.
|
||||
|
||||
## Current behavior to preserve
|
||||
|
||||
Current path in `TeamProvisioningService.sendMessageToRun()` builds content blocks:
|
||||
|
||||
```ts
|
||||
const contentBlocks: Record<string, unknown>[] = [{ type: 'text', text: message }];
|
||||
|
||||
if (att.mimeType === 'application/pdf') {
|
||||
contentBlocks.push({
|
||||
type: 'document',
|
||||
source: {
|
||||
type: 'base64',
|
||||
media_type: 'application/pdf',
|
||||
data: att.data,
|
||||
},
|
||||
title: att.filename,
|
||||
});
|
||||
} else if (att.mimeType === 'text/plain') {
|
||||
// text or base64 document
|
||||
} else {
|
||||
contentBlocks.push({
|
||||
type: 'image',
|
||||
source: {
|
||||
type: 'base64',
|
||||
media_type: att.mimeType,
|
||||
data: att.data,
|
||||
},
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
Keep the same Claude content block shape.
|
||||
|
||||
## Why use adapter
|
||||
|
||||
`TeamProvisioningService` should not know image optimization or provider-specific attachment serialization details. Its responsibility is team lifecycle and message routing.
|
||||
|
||||
The adapter gives:
|
||||
|
||||
- unit-testable serialization;
|
||||
- budget diagnostics before stdin write;
|
||||
- future support for variant selection;
|
||||
- less risk when adding Codex/OpenCode adapters.
|
||||
|
||||
## New adapter sketch
|
||||
|
||||
```ts
|
||||
export class ClaudeStreamJsonAttachmentAdapter implements AttachmentDeliveryAdapter {
|
||||
readonly runtimeKind = 'claude-stream-json' as const;
|
||||
|
||||
canDeliver(
|
||||
ctx: AttachmentRuntimeContext,
|
||||
attachment: NormalizedAgentAttachment,
|
||||
): AttachmentCapabilityDecision {
|
||||
if (attachment.kind === 'image') {
|
||||
return allowIfMime(attachment, ['image/png', 'image/jpeg', 'image/gif', 'image/webp']);
|
||||
}
|
||||
|
||||
if (attachment.kind === 'document' || attachment.kind === 'text') {
|
||||
return allow();
|
||||
}
|
||||
|
||||
return block('This attachment type is not supported by Claude.');
|
||||
}
|
||||
|
||||
async prepare(
|
||||
ctx: AttachmentRuntimeContext,
|
||||
attachment: NormalizedAgentAttachment,
|
||||
): Promise<PreparedAttachmentPart> {
|
||||
const variant = selectClaudeVariant(attachment);
|
||||
return {
|
||||
runtimeKind: this.runtimeKind,
|
||||
attachmentId: attachment.id,
|
||||
part: {
|
||||
kind: 'claude-content-block',
|
||||
value: toClaudeContentBlock(attachment, variant),
|
||||
},
|
||||
diagnostics: [`prepared ${attachment.kind} for Claude stream-json`],
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Serialization helpers
|
||||
|
||||
```ts
|
||||
function toClaudeContentBlock(
|
||||
attachment: NormalizedAgentAttachment,
|
||||
variant: AgentAttachmentVariant,
|
||||
): Record<string, unknown> {
|
||||
if (attachment.kind === 'image') {
|
||||
return {
|
||||
type: 'image',
|
||||
source: {
|
||||
type: 'base64',
|
||||
media_type: variant.mimeType,
|
||||
data: readBase64Variant(variant),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
if (attachment.kind === 'text') {
|
||||
return {
|
||||
type: 'document',
|
||||
source: {
|
||||
type: 'text',
|
||||
media_type: 'text/plain',
|
||||
data: readTextVariant(variant),
|
||||
},
|
||||
title: attachment.originalName,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
type: 'document',
|
||||
source: {
|
||||
type: 'base64',
|
||||
media_type: attachment.mimeType,
|
||||
data: readBase64Variant(variant),
|
||||
},
|
||||
title: attachment.originalName,
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
## `sendMessageToRun` target shape
|
||||
|
||||
Before:
|
||||
|
||||
```ts
|
||||
const contentBlocks = buildInlineInService(message, attachments);
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```ts
|
||||
const contentBlocks: Record<string, unknown>[] = [{ type: 'text', text: message }];
|
||||
|
||||
if (attachments?.length) {
|
||||
const prepared = await this.attachmentDeliveryPlanner.prepareAll(
|
||||
{
|
||||
teamName: run.teamName,
|
||||
providerId: run.providerId,
|
||||
modelId: run.model,
|
||||
runtimeKind: 'claude-stream-json',
|
||||
deliveryTarget: 'lead',
|
||||
},
|
||||
await this.attachmentNormalizer.normalizeLegacyPayloads(attachments),
|
||||
);
|
||||
|
||||
for (const part of prepared) {
|
||||
if (part.part.kind !== 'claude-content-block') {
|
||||
throw new Error('Internal attachment planner returned non-Claude part for Claude runtime');
|
||||
}
|
||||
contentBlocks.push(part.part.value);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Payload write safety
|
||||
|
||||
Before writing stdin:
|
||||
|
||||
```ts
|
||||
const payload = JSON.stringify({
|
||||
type: 'user',
|
||||
message: {
|
||||
role: 'user',
|
||||
content: contentBlocks,
|
||||
},
|
||||
});
|
||||
|
||||
this.attachmentBudgetValidator.assertSerializedPayloadWithinBudget(payload);
|
||||
```
|
||||
|
||||
If blocked, return actionable error:
|
||||
|
||||
```text
|
||||
Attachments are too large for Claude stream-json input after optimization. Remove one image or send a smaller screenshot.
|
||||
```
|
||||
|
||||
## Edge cases
|
||||
|
||||
### Existing text-only sends
|
||||
|
||||
No change. If `attachments` is empty, the planner is not called.
|
||||
|
||||
### Existing PDF support
|
||||
|
||||
Keep current content block shape. Do not optimize PDFs in this phase.
|
||||
|
||||
### Non-UTF text files
|
||||
|
||||
Keep current behavior: try UTF-8, fallback to base64 document if replacement characters appear.
|
||||
|
||||
### Runtime process exits after send
|
||||
|
||||
Do not attribute exit to attachment unless the error path can prove stdin write/payload size failure. This phase should only make pre-send failures visible.
|
||||
|
||||
### Claude image support in wrong mode
|
||||
|
||||
Team lead is long-lived stream-json, so supported. Do not use `claude -p` as e2e validation for this path.
|
||||
|
||||
### Multiple images
|
||||
|
||||
Send all if under budget. If over budget, send none.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
Add bounded diagnostics only:
|
||||
|
||||
```text
|
||||
Prepared 2 attachments for Claude stream-json: image/jpeg 612KB, image/png 124KB.
|
||||
```
|
||||
|
||||
Never log:
|
||||
|
||||
- base64 content;
|
||||
- full file paths unless already user-selected and safe;
|
||||
- API keys;
|
||||
- raw JSON payload.
|
||||
|
||||
## Test plan
|
||||
|
||||
### Unit
|
||||
|
||||
- image attachment serializes to Claude `image` block;
|
||||
- PDF serializes to Claude `document` block;
|
||||
- UTF-8 text serializes to `document` text source;
|
||||
- non-UTF text falls back to base64 document;
|
||||
- planner rejects unsupported mime;
|
||||
- serialized payload over budget rejects before stdin write.
|
||||
|
||||
### Service tests
|
||||
|
||||
- text-only `sendMessageToRun` does not call planner;
|
||||
- safe image calls planner and writes stream-json with image block;
|
||||
- over-budget image throws user-visible error and does not write stdin;
|
||||
- failure does not mark team offline by itself.
|
||||
|
||||
Suggested focused checks:
|
||||
|
||||
```bash
|
||||
pnpm vitest run src/features/agent-attachments/**/*.test.ts test/main/services/team/TeamProvisioningService.test.ts test/main/ipc/teams.test.ts
|
||||
pnpm typecheck --pretty false
|
||||
```
|
||||
|
||||
## Safety checklist
|
||||
|
||||
- Current Claude content block schema preserved.
|
||||
- No Codex/OpenCode paths touched.
|
||||
- No launch/provisioning path touched.
|
||||
- No live provider calls in unit tests.
|
||||
- Existing UI attachment workflow remains compatible.
|
||||
|
||||
## Deep implementation details
|
||||
|
||||
### Refactor target
|
||||
|
||||
The desired refactor is small and reversible.
|
||||
|
||||
Before:
|
||||
|
||||
```ts
|
||||
private async sendMessageToRun(run, message, attachments) {
|
||||
const contentBlocks = [{ type: 'text', text: message }];
|
||||
// inline attachment serialization here
|
||||
stdin.write(JSON.stringify({ ...contentBlocks }) + '\n');
|
||||
}
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```ts
|
||||
private async sendMessageToRun(run, message, attachments) {
|
||||
const contentBlocks = await this.buildClaudeLeadContentBlocks(run, message, attachments);
|
||||
const payload = this.buildClaudeStreamJsonUserPayload(contentBlocks);
|
||||
this.agentAttachments.assertPayloadBudget(payload, { runtime: 'claude-stream-json' });
|
||||
await this.writeToLeadStdin(run, payload);
|
||||
}
|
||||
```
|
||||
|
||||
This keeps `sendMessageToRun()` readable and moves serialization into testable helpers.
|
||||
|
||||
### Helper extraction plan
|
||||
|
||||
```ts
|
||||
private async buildClaudeLeadContentBlocks(
|
||||
run: ProvisioningRun,
|
||||
message: string,
|
||||
attachments?: LegacyAttachmentPayload[],
|
||||
): Promise<Record<string, unknown>[]> {
|
||||
const blocks: Record<string, unknown>[] = [{ type: 'text', text: message }];
|
||||
if (!attachments?.length) return blocks;
|
||||
|
||||
const prepared = await this.agentAttachments.prepareForRuntime({
|
||||
teamName: run.teamName,
|
||||
providerId: run.providerId,
|
||||
modelId: run.model,
|
||||
runtimeKind: 'claude-stream-json',
|
||||
deliveryTarget: 'lead',
|
||||
}, attachments);
|
||||
|
||||
for (const item of prepared) {
|
||||
assertPreparedPartKind(item, 'claude-content-block');
|
||||
blocks.push(item.part.value);
|
||||
}
|
||||
return blocks;
|
||||
}
|
||||
```
|
||||
|
||||
### Content block compatibility tests
|
||||
|
||||
Snapshot the exact old shape.
|
||||
|
||||
```ts
|
||||
expect(toClaudeContentBlock(imageAttachment)).toEqual({
|
||||
type: 'image',
|
||||
source: {
|
||||
type: 'base64',
|
||||
media_type: 'image/png',
|
||||
data: '...',
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
For text:
|
||||
|
||||
```ts
|
||||
expect(toClaudeContentBlock(textAttachment)).toEqual({
|
||||
type: 'document',
|
||||
source: {
|
||||
type: 'text',
|
||||
media_type: 'text/plain',
|
||||
data: 'hello',
|
||||
},
|
||||
title: 'notes.txt',
|
||||
});
|
||||
```
|
||||
|
||||
### Error handling
|
||||
|
||||
Use typed attachment errors and convert at IPC boundary.
|
||||
|
||||
```ts
|
||||
try {
|
||||
await service.sendMessageToTeam(teamName, message, attachments);
|
||||
} catch (error) {
|
||||
if (isAttachmentValidationError(error)) {
|
||||
throw new Error(error.userMessage);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
```
|
||||
|
||||
Do not catch and convert provider/runtime errors here.
|
||||
|
||||
### More edge cases
|
||||
|
||||
| Edge case | Expected behavior |
|
||||
|---|---|
|
||||
| Claude lead is alive but stdin not writable | existing `process stdin is not writable` error wins |
|
||||
| Payload over budget | no stdin write, no message marked delivered |
|
||||
| Attachment adapter throws unsupported mime | user-visible attachment error, team remains alive |
|
||||
| Claude process exits after successful stdin write | existing runtime process close handling owns it |
|
||||
| PDF title contains slash/newline | sanitized title in content block |
|
||||
| Text file is empty | send empty text document or block? Prefer send with warning `empty text file` |
|
||||
| Message text empty but image present | allow if composer supports image-only send; text block can be empty or omitted consistently |
|
||||
| Multiple attachments include one invalid | block all, do not partial-send |
|
||||
| Optimized variant missing | rebuild from legacy base64 or block with retryable local error |
|
||||
|
||||
### Why not change delivery proof
|
||||
|
||||
Claude lead message delivery currently depends on process stdin write and subsequent assistant stream/result. This phase does not add proof. It only makes payload construction safe.
|
||||
|
||||
Do not add new notifications like “image delivered” because it would imply semantic understanding.
|
||||
|
||||
### Regression traps
|
||||
|
||||
- Accidentally using optimized JPEG for transparent PNG without user-visible warning.
|
||||
- Forgetting to include `title` for documents.
|
||||
- Throwing generic `Internal attachment planner returned...` to user instead of diagnostics.
|
||||
- Double-validating text-only messages and blocking them due missing attachment metadata.
|
||||
- Logging full stream-json payload in debug output.
|
||||
|
||||
## File-by-file implementation plan
|
||||
|
||||
### 1. Add adapter
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
src/features/agent-attachments/main/adapters/output/ClaudeStreamJsonAttachmentAdapter.ts
|
||||
```
|
||||
|
||||
This file should depend only on feature contracts/core and small shared helpers.
|
||||
|
||||
### 2. Add facade method
|
||||
|
||||
In feature composition, expose:
|
||||
|
||||
```ts
|
||||
prepareClaudeStreamJsonContentBlocks(input): Promise<Record<string, unknown>[]>
|
||||
```
|
||||
|
||||
or a generic:
|
||||
|
||||
```ts
|
||||
prepareForRuntime(ctx, attachments): Promise<PreparedAttachmentPart[]>
|
||||
```
|
||||
|
||||
Prefer generic if Phase 3/4 will reuse it soon. Prefer Claude-specific if generic abstraction becomes too abstract too early. The plan's recommendation remains generic, but keep the public facade small.
|
||||
|
||||
### 3. Update TeamProvisioningService
|
||||
|
||||
Change only the attachment serialization part of `sendMessageToRun()`.
|
||||
|
||||
Do not change:
|
||||
|
||||
- run tracking;
|
||||
- process liveness checks;
|
||||
- stdin writable checks;
|
||||
- lead activity updates;
|
||||
- close/error handling.
|
||||
|
||||
### 4. Add focused tests
|
||||
|
||||
Update existing `TeamProvisioningService.test.ts` only around send message attachment cases. Add adapter unit tests under feature tests.
|
||||
|
||||
## Compatibility shim
|
||||
|
||||
Because Phase 1 may still use legacy payloads, adapter should accept normalized attachments from a shim.
|
||||
|
||||
```ts
|
||||
async function normalizeForClaudeAdapter(
|
||||
legacy: LegacyTeamMessageAttachment[],
|
||||
): Promise<NormalizedAgentAttachment[]> {
|
||||
return this.normalizer.normalizeLegacyPayloads(legacy, {
|
||||
preferredRuntime: 'claude-stream-json',
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
## Detailed failure cases and expected messages
|
||||
|
||||
| Failure | User message | Internal diagnostic |
|
||||
|---|---|---|
|
||||
| payload over serialized budget | `Attachments are too large for Claude input after optimization.` | include estimated bytes and limit |
|
||||
| unsupported MIME | `This attachment type is not supported by Claude.` | include MIME and filename sanitized |
|
||||
| corrupt image missed by renderer | `Cannot send image because it could not be decoded.` | include attachment id only |
|
||||
| stdin not writable | existing `Team process stdin is not writable` | not attachment diagnostic |
|
||||
| Claude API says image invalid | preserve provider error | not rewritten as optimizer error |
|
||||
|
||||
## Review checklist
|
||||
|
||||
- Adapter output equals previous content block shape for same input.
|
||||
- Payload budget check happens before `stdin.write`.
|
||||
- Error handling does not mark team offline.
|
||||
- No base64 in thrown error message.
|
||||
- No tests require Claude live auth.
|
||||
- Text-only send test still passes without creating feature attachments.
|
||||
|
||||
## More examples
|
||||
|
||||
### Image block
|
||||
|
||||
```ts
|
||||
const block = adapter.toClaudeContentBlock(imageAttachment);
|
||||
expect(block).toMatchObject({
|
||||
type: 'image',
|
||||
source: {
|
||||
type: 'base64',
|
||||
media_type: 'image/jpeg',
|
||||
},
|
||||
});
|
||||
expect(String((block.source as any).data)).toHaveLength(imageBase64.length);
|
||||
```
|
||||
|
||||
### Full payload budget assertion
|
||||
|
||||
```ts
|
||||
const payload = buildClaudeStreamJsonPayload([{ type: 'text', text }, imageBlock]);
|
||||
expect(() => validator.assertWithinBudget(payload)).not.toThrow();
|
||||
```
|
||||
|
||||
### Negative payload budget assertion
|
||||
|
||||
```ts
|
||||
const huge = makeFakeBase64(8_000_000);
|
||||
expect(() => buildAndValidatePayload(huge)).toThrowAgentAttachmentError(
|
||||
'attachment_serialized_payload_too_large',
|
||||
);
|
||||
```
|
||||
|
||||
## Phase 2 exit criteria
|
||||
|
||||
Phase 2 is complete only when:
|
||||
|
||||
- old Claude image/PDF/text content block shapes are preserved;
|
||||
- text-only sends bypass attachment adapter;
|
||||
- oversized attachment blocks before stdin write;
|
||||
- adapter errors do not mark team offline;
|
||||
- copied diagnostics include attachment summary but no base64;
|
||||
- no Codex/OpenCode path changes are included.
|
||||
|
||||
## Migration seam
|
||||
|
||||
Replace only this concern in `TeamProvisioningService`:
|
||||
|
||||
```text
|
||||
legacy attachments -> Claude content blocks
|
||||
```
|
||||
|
||||
Do not touch:
|
||||
|
||||
```text
|
||||
run selection
|
||||
stdin lifecycle
|
||||
process close handling
|
||||
lead activity state
|
||||
message persistence
|
||||
```
|
||||
|
||||
## Claude adapter detailed API
|
||||
|
||||
```ts
|
||||
export interface ClaudeContentBlockBuildInput {
|
||||
messageText: string;
|
||||
attachments: NormalizedAgentAttachment[];
|
||||
budget: AgentAttachmentBudget;
|
||||
}
|
||||
|
||||
export interface ClaudeContentBlockBuildOutput {
|
||||
contentBlocks: Record<string, unknown>[];
|
||||
estimatedSerializedBytes: number;
|
||||
diagnostics: string[];
|
||||
}
|
||||
```
|
||||
|
||||
This allows tests to assert payload size without writing to stdin.
|
||||
|
||||
## Safe payload builder
|
||||
|
||||
```ts
|
||||
export function buildClaudeStreamJsonUserPayload(
|
||||
contentBlocks: Record<string, unknown>[],
|
||||
): string {
|
||||
return JSON.stringify({
|
||||
type: 'user',
|
||||
message: {
|
||||
role: 'user',
|
||||
content: contentBlocks,
|
||||
},
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
Keep this helper tiny and deterministic.
|
||||
|
||||
## Stdin write failure handling
|
||||
|
||||
Attachment errors happen before write. Stdin write errors are runtime errors.
|
||||
|
||||
```ts
|
||||
try {
|
||||
const payload = buildClaudeStreamJsonUserPayload(blocks);
|
||||
this.agentAttachments.assertPayloadBudget(payload);
|
||||
await writeLine(stdin, payload);
|
||||
} catch (error) {
|
||||
if (isAgentAttachmentError(error)) throw error;
|
||||
throw new Error(`Team "${run.teamName}" process stdin is not writable`);
|
||||
}
|
||||
```
|
||||
|
||||
Do not wrap provider/runtime errors as attachment errors.
|
||||
|
||||
## More Claude-specific edge cases
|
||||
|
||||
| Edge case | Expected behavior |
|
||||
|---|---|
|
||||
| `image/webp` sent to Claude | allow only if current existing path allowed it; otherwise block consistently |
|
||||
| `image/gif` animated | preserve existing behavior if under budget, but warn in Phase 1 |
|
||||
| empty message with image | allow only if current composer allows it; otherwise composer-level validation |
|
||||
| PDF over budget | block with attachment size message |
|
||||
| text file with invalid UTF-8 | fallback base64 document as current code did |
|
||||
| Claude returns `Could not process image` | show provider error, do not blame optimizer unless image validation failed locally |
|
||||
| CLI output includes image processing error | include bounded stderr tail in diagnostics through existing runtime mechanisms |
|
||||
|
||||
## Test skeleton for no stdin write on budget failure
|
||||
|
||||
```ts
|
||||
it('does not write to stdin when attachment payload exceeds Claude budget', async () => {
|
||||
const stdin = fakeWritable();
|
||||
await expect(service.sendMessageToRun(runWithStdin(stdin), 'x', [hugeImage]))
|
||||
.rejects.toThrow(/too large/i);
|
||||
expect(stdin.write).not.toHaveBeenCalled();
|
||||
});
|
||||
```
|
||||
|
||||
## Code review notes
|
||||
|
||||
If the diff shows a new `if (mimeType)` ladder inside `TeamProvisioningService`, the refactor failed. That logic belongs in adapter/helper tests.
|
||||
|
|
@ -0,0 +1,626 @@
|
|||
# Phase 3 - Codex native image attachment delivery
|
||||
|
||||
## Summary
|
||||
|
||||
Goal: support image attachments for Codex native teammates by using Codex CLI's supported `--image <FILE>` transport rather than embedding base64 in prompt text.
|
||||
|
||||
Chosen approach: **optimized image artifact files + Codex native exec args extension + text-only fallback errors for unsupported attachment kinds**.
|
||||
|
||||
🎯 8.6 🛡️ 8.4 🧠 6.6
|
||||
Estimated change size: `260-440` LOC across two repos.
|
||||
|
||||
Repos:
|
||||
|
||||
- `/Users/belief/dev/projects/claude/claude_team`
|
||||
- `/Users/belief/dev/projects/claude/agent_teams_orchestrator`
|
||||
|
||||
## Live proof
|
||||
|
||||
Validated manually:
|
||||
|
||||
```bash
|
||||
printf '%s\n' 'Look at the attached image. Reply with exactly one word: red, green, or blue.' \
|
||||
| codex exec --json --skip-git-repo-check -C /tmp \
|
||||
--model gpt-5.4-mini \
|
||||
--image /tmp/agent-attachment-prototypes/red-card-valid.png \
|
||||
--output-last-message /tmp/agent-attachment-prototypes/codex-last.txt \
|
||||
-
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```text
|
||||
red
|
||||
```
|
||||
|
||||
Therefore Codex adapter should pass file paths.
|
||||
|
||||
## Current blocker
|
||||
|
||||
`agent_teams_orchestrator` currently rejects non-text prompts in Codex native:
|
||||
|
||||
```text
|
||||
Codex native phase 0 only supports text-only prompts. Images, documents, and structured input are not wired yet.
|
||||
```
|
||||
|
||||
Likely locations:
|
||||
|
||||
```text
|
||||
agent_teams_orchestrator/src/services/codexNative/turnExecutor.ts
|
||||
agent_teams_orchestrator/src/services/codexNative/execRunner.ts
|
||||
```
|
||||
|
||||
Do not remove this guard globally. Replace it with structured extraction for supported image content only.
|
||||
|
||||
## Data contract
|
||||
|
||||
Add a Codex native input shape that can represent text plus image files.
|
||||
|
||||
```ts
|
||||
export interface CodexNativeTurnInput {
|
||||
promptText: string;
|
||||
imagePaths: string[];
|
||||
}
|
||||
```
|
||||
|
||||
If the current internal API only accepts text, introduce a narrow overload or adapter:
|
||||
|
||||
```ts
|
||||
export type CodexNativePromptInput =
|
||||
| { kind: 'text'; text: string }
|
||||
| { kind: 'text-with-images'; text: string; imagePaths: string[] };
|
||||
```
|
||||
|
||||
Do not pass base64 into Codex prompt text.
|
||||
|
||||
## Claude-team side adapter
|
||||
|
||||
`CodexNativeAttachmentAdapter` should prepare image artifacts.
|
||||
|
||||
```ts
|
||||
export class CodexNativeAttachmentAdapter implements AttachmentDeliveryAdapter {
|
||||
readonly runtimeKind = 'codex-native' as const;
|
||||
|
||||
canDeliver(ctx: AttachmentRuntimeContext, attachment: NormalizedAgentAttachment) {
|
||||
if (attachment.kind !== 'image') {
|
||||
return block('Codex native currently supports image attachments only.');
|
||||
}
|
||||
return allowIfMime(attachment, ['image/png', 'image/jpeg', 'image/webp']);
|
||||
}
|
||||
|
||||
async prepare(ctx: AttachmentRuntimeContext, attachment: NormalizedAgentAttachment) {
|
||||
const variant = selectCodexImageFileVariant(attachment);
|
||||
const path = await this.artifactStore.materializeFileVariant(variant, {
|
||||
teamName: ctx.teamName,
|
||||
runtime: 'codex-native',
|
||||
});
|
||||
|
||||
return {
|
||||
runtimeKind: this.runtimeKind,
|
||||
attachmentId: attachment.id,
|
||||
part: { kind: 'codex-image-arg', path },
|
||||
diagnostics: [`prepared image file for Codex native: ${formatBytes(variant.byteSize)}`],
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Artifact directory should be app-owned and not user-editable:
|
||||
|
||||
```text
|
||||
~/.claude/teams/<team>/attachments/<message-id>/<attachment-id>/<variant-id>.<ext>
|
||||
```
|
||||
|
||||
If existing team data conventions prefer another base path, use that. The key is deterministic metadata and cleanup safety.
|
||||
|
||||
## Orchestrator changes
|
||||
|
||||
### Extract Codex image paths from content blocks
|
||||
|
||||
```ts
|
||||
export function toCodexNativeTurnInput(input: string | ContentBlockParam[]): CodexNativeTurnInput {
|
||||
if (typeof input === 'string') {
|
||||
return { promptText: input, imagePaths: [] };
|
||||
}
|
||||
|
||||
const textParts: string[] = [];
|
||||
const imagePaths: string[] = [];
|
||||
|
||||
for (const block of input) {
|
||||
if (block.type === 'text') {
|
||||
textParts.push(block.text);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (block.type === 'image') {
|
||||
const path = materializeCodexImageBlockToTempFile(block);
|
||||
imagePaths.push(path);
|
||||
continue;
|
||||
}
|
||||
|
||||
throw new Error(`Codex native does not support ${block.type} attachments yet.`);
|
||||
}
|
||||
|
||||
return {
|
||||
promptText: textParts.join('\n\n').trim(),
|
||||
imagePaths,
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
Preferred path: desktop already materializes artifacts and passes paths, so orchestrator should not need to decode base64 except for compatibility with direct SDK/fork calls.
|
||||
|
||||
### Extend exec args
|
||||
|
||||
```ts
|
||||
export function buildCodexNativeExecArgs(options: CodexNativeExecOptions): string[] {
|
||||
return [
|
||||
'exec',
|
||||
'--json',
|
||||
'--skip-git-repo-check',
|
||||
'-C', options.cwd,
|
||||
...options.imagePaths.flatMap(path => ['--image', path]),
|
||||
'-',
|
||||
];
|
||||
}
|
||||
```
|
||||
|
||||
### Preserve stdin prompt behavior
|
||||
|
||||
Keep:
|
||||
|
||||
```ts
|
||||
child.stdin.end(options.prompt);
|
||||
```
|
||||
|
||||
Do not switch to putting the prompt in argv if it can be long.
|
||||
|
||||
## Edge cases
|
||||
|
||||
### Image file path missing before Codex starts
|
||||
|
||||
Expected behavior:
|
||||
|
||||
- fail before spawn with a clear error;
|
||||
- do not start Codex with missing `--image` path.
|
||||
|
||||
### Multiple images
|
||||
|
||||
Codex CLI supports repeatable `--image <FILE>`. Pass one arg pair per image.
|
||||
|
||||
### Unsupported document/PDF
|
||||
|
||||
Expected behavior:
|
||||
|
||||
```text
|
||||
Codex native does not support PDF attachments yet. Send text or images only.
|
||||
```
|
||||
|
||||
Do not silently convert PDF to text in this phase.
|
||||
|
||||
### OpenAI account/session issue
|
||||
|
||||
Attachment code must not mask auth errors. If Codex says login required, show Codex auth error unchanged.
|
||||
|
||||
### Artifact cleanup
|
||||
|
||||
Do not delete image files immediately after spawn. Codex may read after process start. Keep artifacts with message/team data and clean with team cleanup or retention policy.
|
||||
|
||||
### Project path sandbox
|
||||
|
||||
Codex gets `--image` absolute paths outside project. Confirm current Codex CLI accepts this. Live test used `/tmp`, so it does. If future sandbox blocks, copy artifacts into an app-owned allowed directory.
|
||||
|
||||
## Test plan
|
||||
|
||||
### Orchestrator unit
|
||||
|
||||
- text-only input produces no `--image` args;
|
||||
- text plus one image produces one `--image` arg;
|
||||
- multiple images produce repeated args in order;
|
||||
- unsupported document block throws clear error;
|
||||
- missing image path throws before spawn;
|
||||
- prompt still goes to stdin.
|
||||
|
||||
### Desktop unit/service
|
||||
|
||||
- Codex adapter chooses file variant;
|
||||
- artifact materialization writes expected file;
|
||||
- planner blocks PDF for Codex;
|
||||
- error messages do not include base64.
|
||||
|
||||
### Live e2e
|
||||
|
||||
Only when explicitly requested:
|
||||
|
||||
```bash
|
||||
codex exec --json --skip-git-repo-check -C /tmp --model gpt-5.4-mini --image red-card-valid.png -
|
||||
```
|
||||
|
||||
Expected final message:
|
||||
|
||||
```text
|
||||
red
|
||||
```
|
||||
|
||||
Suggested focused checks:
|
||||
|
||||
```bash
|
||||
# claude_team
|
||||
pnpm vitest run src/features/agent-attachments/**/*.test.ts test/main/services/team/TeamProvisioningService.test.ts
|
||||
pnpm typecheck --pretty false
|
||||
|
||||
# agent_teams_orchestrator
|
||||
bun test src/services/codexNative/*.test.ts
|
||||
```
|
||||
|
||||
## Safety checklist
|
||||
|
||||
- Text-only Codex path unchanged.
|
||||
- Auth/session errors preserved.
|
||||
- No base64 in prompt text.
|
||||
- No immediate cleanup of image files after spawn.
|
||||
- Unsupported files fail before model call.
|
||||
|
||||
## Deep implementation details
|
||||
|
||||
### Two-repo boundary
|
||||
|
||||
Desktop should decide and materialize attachment artifacts. Orchestrator should execute Codex with prepared input.
|
||||
|
||||
```text
|
||||
claude_team:
|
||||
normalize/optimize/store image
|
||||
decide Codex supports image
|
||||
pass prepared prompt + image artifact refs into runtime handoff
|
||||
|
||||
agent_teams_orchestrator:
|
||||
accept text + imagePaths
|
||||
validate files exist/readable
|
||||
append --image args
|
||||
keep prompt on stdin
|
||||
```
|
||||
|
||||
Avoid making orchestrator depend on desktop feature internals.
|
||||
|
||||
### Minimal orchestrator type extension
|
||||
|
||||
```ts
|
||||
export interface CodexNativeExecOptions {
|
||||
cwd: string;
|
||||
prompt: string;
|
||||
model?: string;
|
||||
imagePaths?: string[];
|
||||
env?: NodeJS.ProcessEnv;
|
||||
}
|
||||
```
|
||||
|
||||
Default `imagePaths = []` preserves existing callers.
|
||||
|
||||
### Args builder exact behavior
|
||||
|
||||
```ts
|
||||
export function buildCodexNativeExecArgs(options: CodexNativeExecOptions): string[] {
|
||||
const args = [
|
||||
'exec',
|
||||
'--json',
|
||||
'--skip-git-repo-check',
|
||||
'-C',
|
||||
options.cwd,
|
||||
];
|
||||
|
||||
if (options.model) {
|
||||
args.push('--model', options.model);
|
||||
}
|
||||
|
||||
for (const imagePath of options.imagePaths ?? []) {
|
||||
args.push('--image', imagePath);
|
||||
}
|
||||
|
||||
args.push('-');
|
||||
return args;
|
||||
}
|
||||
```
|
||||
|
||||
Order matters. Keep `-` last so stdin is prompt.
|
||||
|
||||
### File validation before spawn
|
||||
|
||||
```ts
|
||||
async function assertCodexImageFilesReady(paths: string[]): Promise<void> {
|
||||
for (const imagePath of paths) {
|
||||
const stat = await fs.promises.stat(imagePath).catch(() => null);
|
||||
if (!stat?.isFile()) {
|
||||
throw new Error(`Codex image attachment is missing: ${path.basename(imagePath)}`);
|
||||
}
|
||||
if (stat.size <= 0) {
|
||||
throw new Error(`Codex image attachment is empty: ${path.basename(imagePath)}`);
|
||||
}
|
||||
if (stat.size > CODEX_IMAGE_FILE_MAX_BYTES) {
|
||||
throw new Error(`Codex image attachment is too large: ${path.basename(imagePath)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Do not include full absolute paths in user messages unless copied diagnostics need them and they are redacted/safe.
|
||||
|
||||
### Desktop artifact store contract
|
||||
|
||||
```ts
|
||||
export interface AttachmentArtifactStore {
|
||||
materializeVariantFile(input: {
|
||||
teamName: string;
|
||||
messageId: string;
|
||||
attachmentId: string;
|
||||
variantId: string;
|
||||
filename: string;
|
||||
base64: string;
|
||||
expectedSha256: string;
|
||||
}): Promise<{ path: string; bytes: number; sha256: string }>;
|
||||
}
|
||||
```
|
||||
|
||||
Validation:
|
||||
|
||||
- directory created with recursive mkdir;
|
||||
- filename sanitized;
|
||||
- write to temp file then rename;
|
||||
- sha256 verified after write;
|
||||
- if existing file has same sha256, reuse;
|
||||
- if existing file mismatch, rewrite from original.
|
||||
|
||||
### Artifact write pattern
|
||||
|
||||
```ts
|
||||
const tmp = `${target}.${process.pid}.${Date.now()}.tmp`;
|
||||
await fs.promises.writeFile(tmp, bytes, { flag: 'wx' }).catch(async error => {
|
||||
if (error.code === 'EEXIST') {
|
||||
await fs.promises.rm(tmp, { force: true });
|
||||
await fs.promises.writeFile(tmp, bytes, { flag: 'wx' });
|
||||
return;
|
||||
}
|
||||
throw error;
|
||||
});
|
||||
await fs.promises.rename(tmp, target);
|
||||
```
|
||||
|
||||
Prefer a shared atomic write helper if one already exists.
|
||||
|
||||
### More edge cases
|
||||
|
||||
| Edge case | Expected behavior |
|
||||
|---|---|
|
||||
| Codex login expires | Codex auth error shown unchanged |
|
||||
| Image path contains spaces | args array handles it, no shell quoting needed |
|
||||
| Artifact deleted between validation and spawn | Codex may fail; surface exact stderr, but pre-spawn validation reduces probability |
|
||||
| Multiple Codex members use same attachment | artifact store can reuse same variant path by hash |
|
||||
| User sends image to Codex lead while lead busy | existing lead busy/message delivery semantics remain unchanged |
|
||||
| Codex model selected is text-only in future | capability gate should block when catalog knows; otherwise live model may error, preserve exact error |
|
||||
| Image is WebP | if Codex accepts through `--image`, allow; otherwise convert to PNG/JPEG in Phase 1/adapter policy |
|
||||
| PDF attached to Codex | block in v1 with clear message |
|
||||
|
||||
### Test additions in orchestrator
|
||||
|
||||
```ts
|
||||
test('adds repeated --image args before stdin marker', () => {
|
||||
expect(buildCodexNativeExecArgs({ cwd: '/tmp', prompt: 'x', imagePaths: ['/a.png', '/b.jpg'] }))
|
||||
.toContainSequence(['--image', '/a.png', '--image', '/b.jpg', '-']);
|
||||
});
|
||||
```
|
||||
|
||||
```ts
|
||||
test('keeps text-only args unchanged', () => {
|
||||
expect(buildCodexNativeExecArgs({ cwd: '/tmp', prompt: 'x' }))
|
||||
.not.toContain('--image');
|
||||
});
|
||||
```
|
||||
|
||||
### Regression traps
|
||||
|
||||
- Passing prompt as argv and accidentally truncating/escaping long prompts.
|
||||
- Deleting artifact file in finally before Codex has read it.
|
||||
- Allowing arbitrary renderer-supplied paths into `--image`.
|
||||
- Hiding Codex auth errors behind `Attachment failed`.
|
||||
- Treating Codex CLI `turn.completed` as image-understood proof without response content.
|
||||
|
||||
## File-by-file implementation plan
|
||||
|
||||
### claude_team
|
||||
|
||||
Potential files:
|
||||
|
||||
```text
|
||||
src/features/agent-attachments/main/adapters/output/CodexNativeAttachmentAdapter.ts
|
||||
src/features/agent-attachments/main/infrastructure/AttachmentArtifactStore.ts
|
||||
src/main/services/team/TeamProvisioningService.ts
|
||||
src/main/ipc/teams.ts
|
||||
```
|
||||
|
||||
Keep the desktop side responsible for app-owned artifact paths.
|
||||
|
||||
### agent_teams_orchestrator
|
||||
|
||||
Potential files:
|
||||
|
||||
```text
|
||||
src/services/codexNative/turnExecutor.ts
|
||||
src/services/codexNative/execRunner.ts
|
||||
src/services/codexNative/*.test.ts
|
||||
```
|
||||
|
||||
Make the orchestrator change backward compatible by defaulting `imagePaths` to `[]`.
|
||||
|
||||
## Integration contract between repos
|
||||
|
||||
If the desktop already invokes orchestrator through a structured prompt, add image paths explicitly rather than hiding them in text.
|
||||
|
||||
Preferred:
|
||||
|
||||
```ts
|
||||
interface NativeRuntimePromptEnvelope {
|
||||
text: string;
|
||||
attachments?: Array<{
|
||||
kind: 'image-file';
|
||||
path: string;
|
||||
mimeType: 'image/png' | 'image/jpeg' | 'image/webp';
|
||||
sha256: string;
|
||||
}>;
|
||||
}
|
||||
```
|
||||
|
||||
Avoid:
|
||||
|
||||
```text
|
||||
Here is an image: /tmp/foo.png
|
||||
```
|
||||
|
||||
unless the runtime explicitly cannot accept images and user chose a textual fallback.
|
||||
|
||||
## Artifact security rules
|
||||
|
||||
- Renderer never supplies final file path.
|
||||
- Backend chooses artifact path under app-owned directory.
|
||||
- Path traversal in filename is sanitized.
|
||||
- Artifact path passed to Codex is absolute.
|
||||
- Artifact checksum is verified after write.
|
||||
- Artifact metadata does not include API keys or user prompt text.
|
||||
|
||||
## More detailed edge cases
|
||||
|
||||
| Edge case | Expected behavior |
|
||||
|---|---|
|
||||
| Codex process starts but exits before reading image | surface exact Codex stderr/exit code |
|
||||
| Artifact file exists but unreadable due permissions | fail before spawn if detectable |
|
||||
| Two sends with same image and same message id | reuse same artifact variant |
|
||||
| Two sends with same image but different message id | allow separate metadata, optionally same content-addressed blob |
|
||||
| Image path has non-ASCII filename | store sanitized ASCII filename plus metadata originalName |
|
||||
| User cancels send during artifact write | abort write if supported, cleanup temp file |
|
||||
| Codex CLI changes `--image` flag | tests fail at args builder/live smoke before release |
|
||||
|
||||
## Test code skeleton
|
||||
|
||||
```ts
|
||||
describe('CodexNativeAttachmentAdapter', () => {
|
||||
it('materializes image variant and returns codex image arg', async () => {
|
||||
const adapter = new CodexNativeAttachmentAdapter(fakeArtifactStore);
|
||||
const part = await adapter.prepare(ctx, imageAttachment);
|
||||
expect(part.part).toEqual({ kind: 'codex-image-arg', path: '/tmp/app/att/red.png' });
|
||||
});
|
||||
|
||||
it('blocks PDF attachments', () => {
|
||||
const decision = adapter.canDeliver(ctx, pdfAttachment);
|
||||
expect(decision.allowed).toBe(false);
|
||||
expect(decision.blockers[0].code).toBe('attachment_runtime_unsupported');
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
```ts
|
||||
describe('buildCodexNativeExecArgs', () => {
|
||||
it('keeps stdin marker last with image args before it', () => {
|
||||
expect(buildCodexNativeExecArgs({ cwd: '/tmp', prompt: 'x', imagePaths: ['/a.png'] }))
|
||||
.toEqual(expect.arrayContaining(['--image', '/a.png', '-']));
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
## Review checklist
|
||||
|
||||
- Existing text-only Codex tests still pass.
|
||||
- `imagePaths` default is empty.
|
||||
- No shell string command building for image paths.
|
||||
- Missing image file fails before spawn where possible.
|
||||
- Auth errors are not converted to attachment errors.
|
||||
- The feature works with Codex subscription auth, not only API key.
|
||||
|
||||
## Phase 3 exit criteria
|
||||
|
||||
Phase 3 is complete only when:
|
||||
|
||||
- text-only Codex native still uses the same exec path;
|
||||
- Codex image send uses `--image <path>` and stdin prompt;
|
||||
- image paths come only from app-owned artifacts;
|
||||
- missing artifact fails before spawn;
|
||||
- unsupported PDFs/documents are blocked before Codex call;
|
||||
- Codex auth errors remain exact;
|
||||
- no OpenCode/Claude code changes are included except shared interfaces.
|
||||
|
||||
## Cross-repo sequencing
|
||||
|
||||
Recommended order:
|
||||
|
||||
1. Orchestrator: add optional `imagePaths` to Codex exec runner with tests.
|
||||
2. Orchestrator: keep turn executor text-only behavior unless image paths are explicitly supplied.
|
||||
3. Desktop: add Codex adapter that materializes image files.
|
||||
4. Desktop: wire Codex adapter only for Codex native send path.
|
||||
5. Live smoke Codex with `gpt-5.4-mini`.
|
||||
|
||||
Do not wire desktop before orchestrator can safely accept `imagePaths`.
|
||||
|
||||
## Backward compatibility in orchestrator
|
||||
|
||||
```ts
|
||||
function normalizeExecOptions(options: CodexNativeExecOptions): Required<Pick<CodexNativeExecOptions, 'imagePaths'>> {
|
||||
return {
|
||||
imagePaths: options.imagePaths ?? [],
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
Existing tests should not need imagePaths.
|
||||
|
||||
## Handling structured content blocks
|
||||
|
||||
If orchestrator receives Anthropic-style content blocks, only support image blocks when they already point to app-owned artifacts or can be materialized safely.
|
||||
|
||||
V1 preference:
|
||||
|
||||
```text
|
||||
Desktop passes file paths, not base64 blocks, to Codex native.
|
||||
```
|
||||
|
||||
If a direct orchestrator caller passes base64 image blocks, fail with clear TODO unless implementing materialization there too.
|
||||
|
||||
```ts
|
||||
throw new Error('Codex native image blocks must be materialized to file paths before execution.');
|
||||
```
|
||||
|
||||
This prevents duplicate artifact stores across repos.
|
||||
|
||||
## Codex path validation nuance
|
||||
|
||||
Do not require image file to be inside project cwd. Live test showed `/tmp` works. Requiring cwd-only would break app-owned artifact store. Instead require:
|
||||
|
||||
- absolute path;
|
||||
- file exists;
|
||||
- file extension/MIME allowed;
|
||||
- size under budget;
|
||||
- path was produced by trusted desktop adapter or trusted test input.
|
||||
|
||||
## More Codex tests
|
||||
|
||||
```ts
|
||||
it('rejects relative image paths', async () => {
|
||||
await expect(runCodexNativeExec({ prompt: 'x', imagePaths: ['foo.png'] }))
|
||||
.rejects.toThrow(/absolute/i);
|
||||
});
|
||||
```
|
||||
|
||||
```ts
|
||||
it('does not include image path in prompt stdin', async () => {
|
||||
const child = fakeCodexChild();
|
||||
await runner.run({ prompt: 'describe', imagePaths: ['/tmp/a.png'] });
|
||||
expect(child.stdin.end).toHaveBeenCalledWith('describe');
|
||||
});
|
||||
```
|
||||
|
||||
## More Codex bug traps
|
||||
|
||||
| Trap | Prevention |
|
||||
|---|---|
|
||||
| prompt accidentally becomes argv | assert `-` remains final arg |
|
||||
| image path included twice | test exact args |
|
||||
| artifact path deleted too early | keep artifacts with message retention |
|
||||
| base64 path from renderer | backend-only artifact store |
|
||||
| Codex auth failure hidden | do not catch provider errors as attachment errors |
|
||||
| unsupported PDF converted to prompt text silently | block explicitly |
|
||||
|
|
@ -0,0 +1,694 @@
|
|||
# Phase 4 - OpenCode file parts and model vision capability gate
|
||||
|
||||
## Summary
|
||||
|
||||
Goal: support image attachments for OpenCode teammates only when the selected model/runtime can actually accept image parts, and show clear UI errors for text-only models.
|
||||
|
||||
Chosen approach: **OpenCode file-part adapter + curated model vision capability catalog + unknown capability fail-safe + optional live smoke tooling**.
|
||||
|
||||
🎯 8.3 🛡️ 8.0 🧠 7.2
|
||||
Estimated change size: `320-560` LOC across two repos.
|
||||
|
||||
Repos:
|
||||
|
||||
- `/Users/belief/dev/projects/claude/claude_team`
|
||||
- `/Users/belief/dev/projects/claude/agent_teams_orchestrator`
|
||||
|
||||
## Live proof
|
||||
|
||||
Validated manually:
|
||||
|
||||
```text
|
||||
opencode run --model openai/gpt-5.4-mini -f red-card-valid.png -> red
|
||||
opencode run --model openrouter/moonshotai/kimi-k2.6 -f red-card-valid.png -> red
|
||||
opencode run --model openrouter/z-ai/glm-4.5v -f red-card-valid.png -> red
|
||||
opencode run --model openrouter/z-ai/glm-5.1 -f red-card-valid.png -> model says it cannot view images
|
||||
```
|
||||
|
||||
OpenCode session export shows file part shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "file",
|
||||
"mime": "image/png",
|
||||
"url": "data:image/png;base64,...",
|
||||
"filename": "red-card-valid.png"
|
||||
}
|
||||
```
|
||||
|
||||
Therefore transport can carry images, but model capability must gate usage.
|
||||
|
||||
## Current behavior
|
||||
|
||||
Current desktop delivery blocks OpenCode secondary attachments:
|
||||
|
||||
```text
|
||||
opencode_attachments_not_supported_for_secondary_runtime
|
||||
```
|
||||
|
||||
This was safe. Phase 4 replaces the blanket block with a capability-aware path.
|
||||
|
||||
## Capability catalog
|
||||
|
||||
Create a pure catalog in the attachment feature or shared OpenCode model utilities.
|
||||
|
||||
```ts
|
||||
export type VisionCapability =
|
||||
| { kind: 'supported'; source: 'curated' | 'provider-metadata' | 'live-probe' }
|
||||
| { kind: 'unsupported'; source: 'curated' | 'model-response' | 'provider-metadata'; reason: string }
|
||||
| { kind: 'unknown'; reason: string };
|
||||
|
||||
export function resolveOpenCodeVisionCapability(modelId: string): VisionCapability {
|
||||
const normalized = normalizeOpenCodeModelId(modelId);
|
||||
|
||||
if (normalized === 'openrouter/z-ai/glm-5.1') {
|
||||
return {
|
||||
kind: 'unsupported',
|
||||
source: 'curated',
|
||||
reason: 'GLM 5.1 did not accept image input in live OpenCode/OpenRouter smoke test.',
|
||||
};
|
||||
}
|
||||
|
||||
if (
|
||||
normalized === 'openai/gpt-5.4-mini' ||
|
||||
normalized === 'openrouter/moonshotai/kimi-k2.6' ||
|
||||
normalized === 'openrouter/z-ai/glm-4.5v'
|
||||
) {
|
||||
return { kind: 'supported', source: 'curated' };
|
||||
}
|
||||
|
||||
if (/\b(vl|vision|image)\b/i.test(normalized)) {
|
||||
return { kind: 'supported', source: 'provider-metadata' };
|
||||
}
|
||||
|
||||
return {
|
||||
kind: 'unknown',
|
||||
reason: 'Image capability for this OpenCode model has not been verified.',
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
Policy decision:
|
||||
|
||||
- `supported`: allow image delivery.
|
||||
- `unsupported`: block with clear error.
|
||||
- `unknown`: block by default for production sends, but allow future manual override only if explicitly designed.
|
||||
|
||||
Before release, do not allow unknown models to receive images silently.
|
||||
|
||||
## OpenCode adapter
|
||||
|
||||
```ts
|
||||
export class OpenCodeAttachmentAdapter implements AttachmentDeliveryAdapter {
|
||||
readonly runtimeKind = 'opencode' as const;
|
||||
|
||||
canDeliver(ctx: AttachmentRuntimeContext, attachment: NormalizedAgentAttachment) {
|
||||
if (attachment.kind !== 'image') {
|
||||
return block('OpenCode currently supports image attachments only.');
|
||||
}
|
||||
|
||||
const capability = resolveOpenCodeVisionCapability(ctx.modelId);
|
||||
if (capability.kind === 'unsupported') {
|
||||
return block(`${ctx.modelId} does not support image input. ${capability.reason}`);
|
||||
}
|
||||
if (capability.kind === 'unknown') {
|
||||
return block(`Image input support for ${ctx.modelId} is unknown. Choose a verified vision model.`);
|
||||
}
|
||||
|
||||
return allowIfMime(attachment, ['image/png', 'image/jpeg', 'image/webp']);
|
||||
}
|
||||
|
||||
async prepare(ctx: AttachmentRuntimeContext, attachment: NormalizedAgentAttachment) {
|
||||
const variant = selectOpenCodeFilePartVariant(attachment);
|
||||
return {
|
||||
runtimeKind: 'opencode',
|
||||
attachmentId: attachment.id,
|
||||
part: {
|
||||
kind: 'opencode-file-part',
|
||||
value: {
|
||||
type: 'file',
|
||||
mime: variant.mimeType,
|
||||
url: toDataUrl(variant),
|
||||
filename: attachment.originalName,
|
||||
},
|
||||
},
|
||||
diagnostics: [`prepared OpenCode file part for ${ctx.modelId}`],
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Orchestrator OpenCode bridge changes
|
||||
|
||||
Current bridge sends text-only parts. Extend to accept file parts.
|
||||
|
||||
```ts
|
||||
export interface OpenCodePromptPart {
|
||||
type: 'text' | 'file';
|
||||
text?: string;
|
||||
mime?: string;
|
||||
url?: string;
|
||||
filename?: string;
|
||||
}
|
||||
|
||||
export interface SendOpenCodePromptInput {
|
||||
sessionId: string;
|
||||
parts: OpenCodePromptPart[];
|
||||
}
|
||||
```
|
||||
|
||||
When no attachments:
|
||||
|
||||
```ts
|
||||
parts: [{ type: 'text', text: params.text }]
|
||||
```
|
||||
|
||||
When attachments:
|
||||
|
||||
```ts
|
||||
parts: [
|
||||
{ type: 'text', text: params.text },
|
||||
{ type: 'file', mime: 'image/png', url: 'data:image/png;base64,...', filename: 'screenshot.png' },
|
||||
]
|
||||
```
|
||||
|
||||
Do not change proof gates. OpenCode delivery success still requires existing response proof:
|
||||
|
||||
- visible reply;
|
||||
- safe plain-text materialization;
|
||||
- `message_send` with correct relay id;
|
||||
- work-sync report;
|
||||
- task/progress evidence.
|
||||
|
||||
Attachment accepted by OpenCode is not response proof.
|
||||
|
||||
## UI capability behavior
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
GLM 5.1 does not support image input. Choose GLM 4.5V, Kimi K2.6, GPT-5.4-mini, Claude, or Codex.
|
||||
```
|
||||
|
||||
```text
|
||||
Image input support for this OpenCode model is not verified. Choose a verified vision model before sending screenshots.
|
||||
```
|
||||
|
||||
```text
|
||||
This image will be sent to Kimi K2.6 through OpenCode/OpenRouter.
|
||||
```
|
||||
|
||||
Keep this next to attachment previews and also validate at send time.
|
||||
|
||||
## Edge cases
|
||||
|
||||
### OpenRouter provider missing
|
||||
|
||||
If OpenCode has no OpenRouter key/provider, model list may not include `openrouter/*`. This is provider auth/config issue, not attachment issue.
|
||||
|
||||
Expected error:
|
||||
|
||||
```text
|
||||
OpenRouter is not connected in OpenCode. Connect OpenRouter before using this model.
|
||||
```
|
||||
|
||||
### OpenRouter credits exhausted
|
||||
|
||||
Preserve exact provider error. Do not rewrite it as attachment failure.
|
||||
|
||||
### Model accepts file part but says cannot see image
|
||||
|
||||
Treat this as model capability failure and update curated deny list only after repeated confirmed cases.
|
||||
|
||||
Do not mark delivery transport failed if OpenCode accepted the prompt and model responded.
|
||||
|
||||
### Unknown model
|
||||
|
||||
Block image attachments by default. Text-only messages still work.
|
||||
|
||||
### Multiple images
|
||||
|
||||
Allow only if total optimized budget is safe. Do not send partial images.
|
||||
|
||||
### File part size
|
||||
|
||||
Use same optimized variants as Claude/Codex. OpenCode data URL still has base64 overhead, so backend serialized budget applies.
|
||||
|
||||
### Retry
|
||||
|
||||
Retry must reuse the same original/variant metadata. Do not recompress differently on every retry unless original variant is missing.
|
||||
|
||||
## Tests
|
||||
|
||||
### Unit
|
||||
|
||||
- `openrouter/moonshotai/kimi-k2.6` is supported;
|
||||
- `openrouter/z-ai/glm-4.5v` is supported;
|
||||
- `openrouter/z-ai/glm-5.1` is unsupported;
|
||||
- unknown OpenRouter model is blocked for image;
|
||||
- OpenCode adapter emits `file` part with data URL;
|
||||
- no base64 appears in diagnostics;
|
||||
- text-only OpenCode messages remain unchanged.
|
||||
|
||||
### Bridge tests
|
||||
|
||||
- OpenCode bridge accepts text-only parts;
|
||||
- OpenCode bridge accepts text plus file parts;
|
||||
- unsupported part type rejects before API call;
|
||||
- response observer/proof semantics unchanged.
|
||||
|
||||
### Desktop service tests
|
||||
|
||||
- direct OpenCode secondary image send blocks unsupported model;
|
||||
- direct OpenCode secondary image send prepares file part for supported model;
|
||||
- provider/API error is preserved;
|
||||
- attachment accepted does not set delivery success without response proof.
|
||||
|
||||
### Live e2e
|
||||
|
||||
Only on explicit request:
|
||||
|
||||
```bash
|
||||
OPENROUTER_API_KEY=... opencode run --pure --format json --dir /tmp --model openrouter/moonshotai/kimi-k2.6 "..." -f red-card-valid.png
|
||||
OPENROUTER_API_KEY=... opencode run --pure --format json --dir /tmp --model openrouter/z-ai/glm-4.5v "..." -f red-card-valid.png
|
||||
OPENROUTER_API_KEY=... opencode run --pure --format json --dir /tmp --model openrouter/z-ai/glm-5.1 "..." -f red-card-valid.png
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```text
|
||||
Kimi K2.6 -> red
|
||||
GLM 4.5V -> red
|
||||
GLM 5.1 -> unsupported/refusal
|
||||
```
|
||||
|
||||
## Safety checklist
|
||||
|
||||
- OpenCode proof gates unchanged.
|
||||
- Unsupported models blocked before send.
|
||||
- Provider auth errors preserved.
|
||||
- No silent fallback to text base64.
|
||||
- No model-specific prompt hacks.
|
||||
- Capability catalog is pure and unit-tested.
|
||||
|
||||
## Deep implementation details
|
||||
|
||||
### Capability resolver should be pure
|
||||
|
||||
Do not call OpenCode or OpenRouter inside the send hot path.
|
||||
|
||||
```ts
|
||||
export interface OpenCodeVisionCapabilityResolver {
|
||||
resolve(modelId: string): VisionCapability;
|
||||
}
|
||||
```
|
||||
|
||||
Use static curated rules plus model id heuristics. Live probing belongs in Phase 5 tooling, not every send.
|
||||
|
||||
### Capability result examples
|
||||
|
||||
```ts
|
||||
resolve('openrouter/moonshotai/kimi-k2.6')
|
||||
// { kind: 'supported', source: 'curated' }
|
||||
|
||||
resolve('openrouter/z-ai/glm-4.5v')
|
||||
// { kind: 'supported', source: 'curated' }
|
||||
|
||||
resolve('openrouter/z-ai/glm-5.1')
|
||||
// { kind: 'unsupported', source: 'curated', reason: 'Live smoke returned text-only refusal.' }
|
||||
|
||||
resolve('openrouter/qwen/qwen2.5-vl-72b-instruct')
|
||||
// { kind: 'supported', source: 'provider-metadata' }
|
||||
|
||||
resolve('openrouter/minimax/minimax-m2.5')
|
||||
// { kind: 'unknown', reason: 'No verified vision capability.' }
|
||||
```
|
||||
|
||||
### Model normalization
|
||||
|
||||
OpenCode may expose ids in different forms:
|
||||
|
||||
```text
|
||||
openrouter/moonshotai/kimi-k2.6
|
||||
moonshotai/kimi-k2.6 via openrouter provider context
|
||||
z-ai/glm-4.5v
|
||||
```
|
||||
|
||||
Normalize consistently:
|
||||
|
||||
```ts
|
||||
export function normalizeOpenCodeModelRef(input: string, providerId?: string): string {
|
||||
const trimmed = input.trim().toLowerCase();
|
||||
if (trimmed.startsWith('openrouter/')) return trimmed;
|
||||
if (providerId === 'openrouter') return `openrouter/${trimmed}`;
|
||||
return trimmed;
|
||||
}
|
||||
```
|
||||
|
||||
### OpenCode bridge part schema
|
||||
|
||||
Use narrow supported schema. Do not allow arbitrary JSON parts from renderer/main.
|
||||
|
||||
```ts
|
||||
export type OpenCodeBridgePromptPart =
|
||||
| { type: 'text'; text: string }
|
||||
| { type: 'file'; mime: 'image/png' | 'image/jpeg' | 'image/webp'; url: string; filename: string };
|
||||
|
||||
function assertOpenCodeBridgePromptPart(part: OpenCodeBridgePromptPart): void {
|
||||
if (part.type === 'file') {
|
||||
if (!part.url.startsWith(`data:${part.mime};base64,`)) {
|
||||
throw new Error('Invalid OpenCode file part data URL.');
|
||||
}
|
||||
if (part.url.length > OPENCODE_FILE_PART_DATA_URL_MAX_CHARS) {
|
||||
throw new Error('OpenCode file part is too large.');
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Delivery result semantics
|
||||
|
||||
Do not change existing OpenCode delivery proof. Attachment support is input preparation only.
|
||||
|
||||
```text
|
||||
file part accepted != response delivered
|
||||
response text exists != relay proof for peer messages
|
||||
empty assistant turn != success
|
||||
provider auth failure != retryable prompt repair
|
||||
```
|
||||
|
||||
### UI capability copy
|
||||
|
||||
Keep model-specific text concise:
|
||||
|
||||
```text
|
||||
Images are not supported by GLM 5.1 in OpenCode. Choose GLM 4.5V, Kimi K2.6, GPT-5.4-mini, Claude, or Codex.
|
||||
```
|
||||
|
||||
For unknown:
|
||||
|
||||
```text
|
||||
Image support for this OpenCode model is not verified. Send text only or choose a verified vision model.
|
||||
```
|
||||
|
||||
For missing provider auth:
|
||||
|
||||
```text
|
||||
OpenRouter is not connected for OpenCode. Connect OpenRouter before sending images to this model.
|
||||
```
|
||||
|
||||
### More edge cases
|
||||
|
||||
| Edge case | Expected behavior |
|
||||
|---|---|
|
||||
| OpenCode provider exists only through env key during tests | live smoke works, app-managed config still needs explicit provider setup |
|
||||
| User pastes image to OpenCode text-only model | composer blocks send before creating ledger record |
|
||||
| User sends text-only to GLM 5.1 | allowed, no image capability warning blocking send |
|
||||
| Kimi accepts image but returns low-quality answer | delivery success if response proof exists; not attachment bug |
|
||||
| OpenRouter key has quota limit | preserve exact provider error and show advisory/notification if runtime error path triggers |
|
||||
| OpenCode returns `ProviderModelNotFoundError` | provider/model config error, not attachment serialization |
|
||||
| File part accepted but no assistant response | existing OpenCode repair policy handles no response, not attachment feature |
|
||||
| Unsupported image MIME like SVG | block before OpenCode bridge |
|
||||
| Data URL too large | block before OpenCode API call |
|
||||
|
||||
### Negative test for GLM 5.1
|
||||
|
||||
```ts
|
||||
test('blocks GLM 5.1 image send before OpenCode prompt', async () => {
|
||||
const result = planner.canPrepare({ runtimeKind: 'opencode', modelId: 'openrouter/z-ai/glm-5.1' }, [image]);
|
||||
expect(result.allowed).toBe(false);
|
||||
expect(result.blockers[0].code).toBe('attachment_model_vision_unsupported');
|
||||
});
|
||||
```
|
||||
|
||||
### Positive test for Kimi
|
||||
|
||||
```ts
|
||||
test('allows Kimi K2.6 image send', async () => {
|
||||
const result = planner.canPrepare({ runtimeKind: 'opencode', modelId: 'openrouter/moonshotai/kimi-k2.6' }, [image]);
|
||||
expect(result.allowed).toBe(true);
|
||||
});
|
||||
```
|
||||
|
||||
### Regression traps
|
||||
|
||||
- Making unknown OpenCode models permissive by default.
|
||||
- Marking ledger `delivered` just because OpenCode accepted a file part.
|
||||
- Adding model-specific prompt text like “you can see image” instead of capability gating.
|
||||
- Writing OpenRouter API key into logs while running live smoke.
|
||||
- Updating app-managed OpenCode auth store from ephemeral env without explicit user action.
|
||||
|
||||
## File-by-file implementation plan
|
||||
|
||||
### claude_team
|
||||
|
||||
Potential files:
|
||||
|
||||
```text
|
||||
src/features/agent-attachments/core/domain/OpenCodeVisionCapability.ts
|
||||
src/features/agent-attachments/main/adapters/output/OpenCodeAttachmentAdapter.ts
|
||||
src/main/services/team/TeamProvisioningService.ts
|
||||
src/renderer/utils/openCodeRuntimeDeliveryDiagnostics.ts
|
||||
src/renderer/components/team/messages/MessageComposer.tsx
|
||||
```
|
||||
|
||||
Do not place the capability map inside `TeamProvisioningService`.
|
||||
|
||||
### agent_teams_orchestrator
|
||||
|
||||
Potential files:
|
||||
|
||||
```text
|
||||
src/services/opencode/OpenCodeSessionBridge.ts
|
||||
src/services/opencode/OpenCodeBridgeCommandHandler.ts
|
||||
src/services/opencode/OpenCodeDeliveryResponseObserver.ts
|
||||
src/services/opencode/*.test.ts
|
||||
```
|
||||
|
||||
Bridge accepts structured parts. Observer/proof logic should remain unchanged except diagnostics may include attachment context.
|
||||
|
||||
## Capability map structure
|
||||
|
||||
Use a compact data file or pure module.
|
||||
|
||||
```ts
|
||||
const OPENCODE_VISION_CAPABILITY_OVERRIDES: Record<string, VisionCapability> = {
|
||||
'openai/gpt-5.4-mini': { kind: 'supported', source: 'curated' },
|
||||
'openrouter/moonshotai/kimi-k2.6': { kind: 'supported', source: 'curated' },
|
||||
'openrouter/z-ai/glm-4.5v': { kind: 'supported', source: 'curated' },
|
||||
'openrouter/z-ai/glm-5.1': {
|
||||
kind: 'unsupported',
|
||||
source: 'curated',
|
||||
reason: 'Live smoke returned model response saying image input is unsupported.',
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
Heuristics only after exact overrides:
|
||||
|
||||
```ts
|
||||
if (/\b(vl|vision|image)\b/i.test(modelId)) {
|
||||
return { kind: 'supported', source: 'provider-metadata' };
|
||||
}
|
||||
```
|
||||
|
||||
If no exact/heuristic match:
|
||||
|
||||
```ts
|
||||
return { kind: 'unknown', reason: 'No verified vision capability for this model.' };
|
||||
```
|
||||
|
||||
## UI state transitions
|
||||
|
||||
| State | UI |
|
||||
|---|---|
|
||||
| supported | normal send enabled |
|
||||
| unsupported | send disabled with model-specific message |
|
||||
| unknown | send disabled with `not verified` message |
|
||||
| provider missing | send disabled with provider connect message |
|
||||
| provider quota/auth error after send | runtime error notification/advisory, not preflight warning |
|
||||
|
||||
## Direct teammate delivery with attachments
|
||||
|
||||
For OpenCode secondary direct user message:
|
||||
|
||||
```text
|
||||
user -> OpenCode member
|
||||
```
|
||||
|
||||
The send path must:
|
||||
|
||||
1. resolve member provider/model from live config/meta;
|
||||
2. run attachment capability gate;
|
||||
3. prepare OpenCode file parts;
|
||||
4. call bridge with text + file parts;
|
||||
5. keep existing ledger proof semantics.
|
||||
|
||||
Do not let UI-provided model labels drive capability. Backend must resolve actual member model.
|
||||
|
||||
## Peer/lead relay with attachments
|
||||
|
||||
For v1, be conservative.
|
||||
|
||||
If user sends attachment to lead and lead delegates to OpenCode member, do not automatically forward original binary attachment unless a later phase explicitly implements attachment relay. The lead can describe or ask user to send directly.
|
||||
|
||||
This avoids accidental broad binary propagation across agents.
|
||||
|
||||
## More detailed tests
|
||||
|
||||
### Capability resolver
|
||||
|
||||
```ts
|
||||
it.each([
|
||||
['openrouter/moonshotai/kimi-k2.6', 'supported'],
|
||||
['openrouter/z-ai/glm-4.5v', 'supported'],
|
||||
['openrouter/z-ai/glm-5.1', 'unsupported'],
|
||||
['openrouter/minimax/minimax-m2.5', 'unknown'],
|
||||
])('resolves %s', (modelId, expected) => {
|
||||
expect(resolveOpenCodeVisionCapability(modelId).kind).toBe(expected);
|
||||
});
|
||||
```
|
||||
|
||||
### Bridge part validation
|
||||
|
||||
```ts
|
||||
expect(() => assertOpenCodeBridgePromptPart({
|
||||
type: 'file',
|
||||
mime: 'image/png',
|
||||
url: 'not-a-data-url',
|
||||
filename: 'x.png',
|
||||
})).toThrow('Invalid OpenCode file part data URL');
|
||||
```
|
||||
|
||||
### Ledger preservation
|
||||
|
||||
```ts
|
||||
it('does not mark delivery successful only because file part was accepted', async () => {
|
||||
const result = await delivery.sendWithAttachment(...);
|
||||
expect(result.delivered).toBe(false);
|
||||
expect(result.responsePending).toBe(true);
|
||||
});
|
||||
```
|
||||
|
||||
## Review checklist
|
||||
|
||||
- Unknown OpenCode model does not receive image by default.
|
||||
- GLM 5.1 image send is blocked before API call.
|
||||
- Kimi K2.6 and GLM 4.5V are allowed.
|
||||
- Text-only sends to all models still work.
|
||||
- OpenCode provider errors are preserved exactly.
|
||||
- Ledger/proof semantics unchanged.
|
||||
- No OpenRouter key printed in live test logs.
|
||||
|
||||
## Phase 4 exit criteria
|
||||
|
||||
Phase 4 is complete only when:
|
||||
|
||||
- text-only OpenCode delivery path is unchanged;
|
||||
- OpenCode image send is allowed only for supported/verified vision models;
|
||||
- GLM 5.1 image send is blocked before OpenCode call;
|
||||
- Kimi K2.6 and GLM 4.5V image sends serialize to file parts in tests;
|
||||
- OpenCode delivery proof semantics are unchanged;
|
||||
- provider auth/quota/model errors remain exact;
|
||||
- no automatic live probe runs on normal send.
|
||||
|
||||
## Cross-repo sequencing
|
||||
|
||||
Recommended order:
|
||||
|
||||
1. Orchestrator: extend OpenCode bridge to accept typed `parts` while preserving text-only API.
|
||||
2. Orchestrator: test file part serialization with fake client.
|
||||
3. Desktop: add OpenCode capability resolver.
|
||||
4. Desktop: add OpenCode adapter.
|
||||
5. Desktop: wire direct OpenCode secondary attachment path.
|
||||
6. Add UI warnings/blockers for unsupported/unknown models.
|
||||
7. Run live smoke only after unit/fixture tests pass.
|
||||
|
||||
## Backward-compatible bridge API
|
||||
|
||||
Keep old text API as wrapper.
|
||||
|
||||
```ts
|
||||
async sendPromptText(input: { sessionId: string; text: string }) {
|
||||
return this.sendPromptParts({
|
||||
sessionId: input.sessionId,
|
||||
parts: [{ type: 'text', text: input.text }],
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
New API:
|
||||
|
||||
```ts
|
||||
async sendPromptParts(input: { sessionId: string; parts: OpenCodeBridgePromptPart[] }) {
|
||||
validateParts(input.parts);
|
||||
return this.client.sendSessionMessage(input.sessionId, { parts: input.parts });
|
||||
}
|
||||
```
|
||||
|
||||
## Capability resolver detail
|
||||
|
||||
Return both machine code and user copy.
|
||||
|
||||
```ts
|
||||
export interface OpenCodeVisionCapabilityDecision {
|
||||
kind: 'supported' | 'unsupported' | 'unknown';
|
||||
code:
|
||||
| 'opencode_vision_supported'
|
||||
| 'opencode_model_text_only'
|
||||
| 'opencode_model_vision_unknown';
|
||||
source: 'curated' | 'heuristic' | 'provider-metadata';
|
||||
userMessage?: string;
|
||||
diagnostic: string;
|
||||
}
|
||||
```
|
||||
|
||||
This prevents UI from string-matching diagnostics.
|
||||
|
||||
## Provider auth vs capability
|
||||
|
||||
Do not require provider auth to decide static capability. A model can be known vision-capable even if auth is missing. Send still fails/preflights on auth separately.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
openrouter/moonshotai/kimi-k2.6 capability = supported
|
||||
OpenRouter auth missing = provider setup blocker
|
||||
```
|
||||
|
||||
Both may appear in UI, but auth blocker is operational and capability is model-level.
|
||||
|
||||
## More OpenCode tests
|
||||
|
||||
```ts
|
||||
it('keeps text-only API as wrapper around parts API', async () => {
|
||||
await bridge.sendPromptText({ sessionId: 's', text: 'hello' });
|
||||
expect(client.sendSessionMessage).toHaveBeenCalledWith('s', {
|
||||
parts: [{ type: 'text', text: 'hello' }],
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
```ts
|
||||
it('serializes image file part without changing response observer', async () => {
|
||||
await bridge.sendPromptParts({
|
||||
sessionId: 's',
|
||||
parts: [
|
||||
{ type: 'text', text: 'what color?' },
|
||||
{ type: 'file', mime: 'image/png', url: 'data:image/png;base64,AAAA', filename: 'x.png' },
|
||||
],
|
||||
});
|
||||
expect(observer).not.toHaveNewSuccessRule();
|
||||
});
|
||||
```
|
||||
|
||||
## More OpenCode bug traps
|
||||
|
||||
| Trap | Prevention |
|
||||
|---|---|
|
||||
| unknown model allowed and silently fails | unknown blocks by default |
|
||||
| bridge accepts arbitrary part JSON | strict union validation |
|
||||
| file part accepted marks ledger delivered | tests assert proof unchanged |
|
||||
| provider key logged in smoke | redactor in smoke script |
|
||||
| model capability string hardcoded in UI only | backend resolver is source of truth |
|
||||
| OpenRouter model id normalization inconsistent | shared `normalizeOpenCodeModelRef` tests |
|
||||
|
|
@ -0,0 +1,631 @@
|
|||
# Phase 5 - Cross-runtime attachment E2E, diagnostics, docs, and polish
|
||||
|
||||
## Summary
|
||||
|
||||
Goal: make the completed attachment system observable, testable, and understandable for users before release.
|
||||
|
||||
Chosen approach: **small live smoke harness + deterministic diagnostics + UI copy polish + documentation**, with no new runtime semantics.
|
||||
|
||||
🎯 8.8 🛡️ 8.7 🧠 5.4
|
||||
Estimated change size: `180-320` LOC plus tests/docs.
|
||||
|
||||
This phase should happen after Claude, Codex, and OpenCode adapters are implemented. It should not introduce new delivery behavior.
|
||||
|
||||
## Deliverables
|
||||
|
||||
- live attachment smoke script;
|
||||
- reusable test fixture image generator;
|
||||
- user-visible diagnostics for unsupported models and oversized images;
|
||||
- docs for supported runtimes/models;
|
||||
- release checklist.
|
||||
|
||||
## Live smoke harness
|
||||
|
||||
Create a script that generates a deterministic image and runs each supported runtime.
|
||||
|
||||
Suggested location:
|
||||
|
||||
```text
|
||||
scripts/smoke/agent-attachments-smoke.mjs
|
||||
```
|
||||
|
||||
Sketch:
|
||||
|
||||
```ts
|
||||
const cases = [
|
||||
{
|
||||
id: 'claude-subscription-streaming',
|
||||
runtime: 'claude',
|
||||
model: 'claude-haiku-4-5',
|
||||
expected: /red/i,
|
||||
},
|
||||
{
|
||||
id: 'codex-native-gpt-5-4-mini',
|
||||
runtime: 'codex',
|
||||
model: 'gpt-5.4-mini',
|
||||
expected: /red/i,
|
||||
},
|
||||
{
|
||||
id: 'opencode-openai-gpt-5-4-mini',
|
||||
runtime: 'opencode',
|
||||
model: 'openai/gpt-5.4-mini',
|
||||
expected: /red/i,
|
||||
},
|
||||
{
|
||||
id: 'opencode-openrouter-kimi-k2-6',
|
||||
runtime: 'opencode',
|
||||
model: 'openrouter/moonshotai/kimi-k2.6',
|
||||
envRequired: ['OPENROUTER_API_KEY'],
|
||||
expected: /red/i,
|
||||
},
|
||||
{
|
||||
id: 'opencode-openrouter-glm-4-5v',
|
||||
runtime: 'opencode',
|
||||
model: 'openrouter/z-ai/glm-4.5v',
|
||||
envRequired: ['OPENROUTER_API_KEY'],
|
||||
expected: /red/i,
|
||||
},
|
||||
{
|
||||
id: 'opencode-openrouter-glm-5-1-negative',
|
||||
runtime: 'opencode',
|
||||
model: 'openrouter/z-ai/glm-5.1',
|
||||
envRequired: ['OPENROUTER_API_KEY'],
|
||||
expectedUnsupported: true,
|
||||
},
|
||||
];
|
||||
```
|
||||
|
||||
The harness must:
|
||||
|
||||
- redact keys;
|
||||
- use timeouts;
|
||||
- kill child processes on timeout;
|
||||
- write structured JSON result;
|
||||
- skip cases when required auth/env is missing;
|
||||
- never print base64 image content.
|
||||
|
||||
## Deterministic fixture image
|
||||
|
||||
Do not depend on external image files.
|
||||
|
||||
Generate a small valid PNG with Node `zlib` and CRC32, like the prototype did.
|
||||
|
||||
```ts
|
||||
export function writeRedCardPng(path: string): void {
|
||||
// 320x240 red card with white center marker.
|
||||
}
|
||||
```
|
||||
|
||||
This avoids flaky fixtures and keeps smoke tests self-contained.
|
||||
|
||||
## Diagnostics UX
|
||||
|
||||
Add compact diagnostics wherever attachments are shown or rejected.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
Sent 1 optimized image: screenshot.jpg, 1920x1080, 612 KB.
|
||||
```
|
||||
|
||||
```text
|
||||
Images are not supported by openrouter/z-ai/glm-5.1. Choose GLM 4.5V, Kimi K2.6, GPT-5.4-mini, Claude, or Codex.
|
||||
```
|
||||
|
||||
```text
|
||||
Attachment payload is too large after optimization: 8.4 MB serialized. Limit is 7.5 MB.
|
||||
```
|
||||
|
||||
```text
|
||||
OpenRouter is not connected in OpenCode. Connect OpenRouter before using this model.
|
||||
```
|
||||
|
||||
## Copy diagnostics
|
||||
|
||||
When user copies diagnostics for a failed send, include:
|
||||
|
||||
```text
|
||||
Attachment summary:
|
||||
- files: 2
|
||||
- optimized bytes: 1.2 MB
|
||||
- estimated serialized payload: 1.7 MB
|
||||
- target runtime: opencode
|
||||
- target model: openrouter/z-ai/glm-5.1
|
||||
- capability decision: unsupported image input
|
||||
```
|
||||
|
||||
Do not include:
|
||||
|
||||
- base64;
|
||||
- full API keys;
|
||||
- bearer tokens;
|
||||
- raw data URLs.
|
||||
|
||||
## Documentation
|
||||
|
||||
Add docs under:
|
||||
|
||||
```text
|
||||
docs/team-management/agent-attachments.md
|
||||
```
|
||||
|
||||
Contents:
|
||||
|
||||
- supported runtimes;
|
||||
- supported model examples;
|
||||
- unsupported model examples;
|
||||
- why images may be resized;
|
||||
- why some models cannot receive screenshots;
|
||||
- troubleshooting auth/provider issues;
|
||||
- how to run smoke tests.
|
||||
|
||||
## Release checklist
|
||||
|
||||
Before release:
|
||||
|
||||
- text-only messages still work for Claude/Codex/OpenCode;
|
||||
- oversized image blocked before send;
|
||||
- Claude image send works;
|
||||
- Codex image send works;
|
||||
- OpenCode OpenAI image send works;
|
||||
- OpenCode OpenRouter Kimi works if key configured;
|
||||
- OpenCode GLM 5.1 image is blocked or clearly marked unsupported;
|
||||
- no base64 appears in logs, copied diagnostics, or UI error text;
|
||||
- retry with attachments reuses artifacts or fails loudly;
|
||||
- removing attachments clears warnings;
|
||||
- unsupported model warning updates when model changes.
|
||||
|
||||
## E2E scenarios
|
||||
|
||||
### Scenario 1 - Claude lead screenshot
|
||||
|
||||
```text
|
||||
Create/launch Claude team -> send screenshot to lead -> lead answers about image.
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- no process crash;
|
||||
- message visible;
|
||||
- optimized attachment notice visible;
|
||||
- lead response received.
|
||||
|
||||
### Scenario 2 - Codex lead screenshot
|
||||
|
||||
```text
|
||||
Create/launch Codex team -> send screenshot -> Codex sees image via --image.
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- artifact file created;
|
||||
- Codex args include `--image`;
|
||||
- no base64 in prompt text;
|
||||
- response received.
|
||||
|
||||
### Scenario 3 - OpenCode supported model
|
||||
|
||||
```text
|
||||
OpenCode Kimi K2.6 secondary -> direct user message with screenshot.
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- file part delivered;
|
||||
- delivery proof still required;
|
||||
- response visible.
|
||||
|
||||
### Scenario 4 - OpenCode unsupported model
|
||||
|
||||
```text
|
||||
OpenCode GLM 5.1 secondary -> attempt screenshot send.
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- send blocked before model call;
|
||||
- message explains model does not support image input;
|
||||
- no fake queued/pending delivery;
|
||||
- text-only send still works.
|
||||
|
||||
### Scenario 5 - Oversized multi-image send
|
||||
|
||||
```text
|
||||
Attach 5 large screenshots.
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- optimizer reduces where safe;
|
||||
- if still too large, send blocked;
|
||||
- no partial delivery.
|
||||
|
||||
## Test plan
|
||||
|
||||
Suggested focused checks:
|
||||
|
||||
```bash
|
||||
pnpm vitest run src/features/agent-attachments/**/*.test.ts test/main/ipc/teams.test.ts test/renderer/components/team/messages/MessageComposer.test.tsx
|
||||
pnpm vitest run test/main/services/team/TeamProvisioningService.test.ts test/main/services/team/OpenCodePromptDeliveryLedger.test.ts
|
||||
pnpm typecheck --pretty false
|
||||
```
|
||||
|
||||
Live smoke only when requested:
|
||||
|
||||
```bash
|
||||
node scripts/smoke/agent-attachments-smoke.mjs --case claude-subscription-streaming
|
||||
node scripts/smoke/agent-attachments-smoke.mjs --case codex-native-gpt-5-4-mini
|
||||
OPENROUTER_API_KEY=... node scripts/smoke/agent-attachments-smoke.mjs --case opencode-openrouter-kimi-k2-6
|
||||
```
|
||||
|
||||
## Safety checklist
|
||||
|
||||
- Smoke harness redacts secrets.
|
||||
- Live tests have timeouts and cleanup.
|
||||
- Docs clearly separate transport support from model vision support.
|
||||
- No new runtime behavior is introduced in this phase.
|
||||
|
||||
## Deep implementation details
|
||||
|
||||
### Live smoke output contract
|
||||
|
||||
The smoke script should write machine-readable JSON and concise console output.
|
||||
|
||||
```ts
|
||||
export interface AttachmentSmokeResult {
|
||||
id: string;
|
||||
runtime: 'claude' | 'codex' | 'opencode';
|
||||
model: string;
|
||||
status: 'passed' | 'failed' | 'skipped';
|
||||
reason?: string;
|
||||
responseText?: string;
|
||||
durationMs: number;
|
||||
diagnostics: string[];
|
||||
}
|
||||
```
|
||||
|
||||
Console output example:
|
||||
|
||||
```text
|
||||
PASS claude-subscription-streaming -> red
|
||||
PASS codex-native-gpt-5-4-mini -> red
|
||||
SKIP opencode-openrouter-kimi-k2-6 -> OPENROUTER_API_KEY not set
|
||||
FAIL opencode-openrouter-glm-5-1-negative -> expected unsupported but got red
|
||||
```
|
||||
|
||||
Never print secrets.
|
||||
|
||||
### Timeout wrapper
|
||||
|
||||
```ts
|
||||
async function runWithTimeout<T>(label: string, timeoutMs: number, run: (signal: AbortSignal) => Promise<T>): Promise<T> {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(new Error(`${label} timed out`)), timeoutMs);
|
||||
try {
|
||||
return await run(controller.signal);
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For child processes, abort must kill process group when possible.
|
||||
|
||||
### Redaction helper
|
||||
|
||||
```ts
|
||||
export function redactAttachmentSmokeLog(input: string): string {
|
||||
return input
|
||||
.replace(/sk-or-v1-[A-Za-z0-9_-]+/g, 'sk-or-v1-[REDACTED]')
|
||||
.replace(/Bearer\s+[A-Za-z0-9._-]+/gi, 'Bearer [REDACTED]')
|
||||
.replace(/data:image\/[a-z0-9.+-]+;base64,[A-Za-z0-9+/=]+/gi, 'data:image/[REDACTED];base64,[REDACTED]');
|
||||
}
|
||||
```
|
||||
|
||||
### Docs structure
|
||||
|
||||
`docs/team-management/agent-attachments.md` should include:
|
||||
|
||||
```text
|
||||
# Agent attachments
|
||||
|
||||
## Supported runtimes
|
||||
## Supported image models
|
||||
## Unsupported or unverified models
|
||||
## Why screenshots are optimized
|
||||
## Troubleshooting
|
||||
## Running smoke tests
|
||||
## Security and privacy notes
|
||||
```
|
||||
|
||||
### UI polish details
|
||||
|
||||
Attachment preview should show:
|
||||
|
||||
```text
|
||||
screenshot.jpg
|
||||
1920x1080 - 612 KB - optimized
|
||||
```
|
||||
|
||||
Unsupported model warning should include direct action:
|
||||
|
||||
```text
|
||||
Change model
|
||||
Remove image
|
||||
```
|
||||
|
||||
Do not show internal provider ids only. Use friendly label when available:
|
||||
|
||||
```text
|
||||
GLM 5.1 via OpenRouter
|
||||
```
|
||||
|
||||
But copied diagnostics should include exact model id:
|
||||
|
||||
```text
|
||||
modelId=openrouter/z-ai/glm-5.1
|
||||
```
|
||||
|
||||
### More e2e cases
|
||||
|
||||
| Scenario | Expected |
|
||||
|---|---|
|
||||
| Text-only message after failed image send | succeeds normally |
|
||||
| User removes unsupported image and sends text | no stale warning blocks send |
|
||||
| User switches from GLM 5.1 to GLM 4.5V | warning clears and send allowed |
|
||||
| User switches from OpenCode to Claude | OpenCode model warning disappears, Claude budget warning remains if oversized |
|
||||
| OpenRouter key missing | OpenRouter smoke skipped, not failed |
|
||||
| OpenRouter quota exhausted | smoke failed with provider quota diagnostic, no secret printed |
|
||||
| Codex auth expired | Codex smoke failed with auth diagnostic, attachment system not blamed |
|
||||
| Claude subscription over limit | Claude smoke failed with provider limit diagnostic, attachment system not blamed |
|
||||
|
||||
### Release readiness scoring
|
||||
|
||||
Before shipping, score each area:
|
||||
|
||||
| Area | Target score |
|
||||
|---|---:|
|
||||
| Text-only regression confidence | 9/10 |
|
||||
| Oversized image protection | 9/10 |
|
||||
| Claude image path | 8.5/10 |
|
||||
| Codex image path | 8/10 |
|
||||
| OpenCode OpenAI image path | 8/10 |
|
||||
| OpenCode OpenRouter model gating | 7.5/10 |
|
||||
| User-facing errors | 8.5/10 |
|
||||
|
||||
If any score is below target, do not release the whole attachment feature. Ship only earlier phases.
|
||||
|
||||
### Regression traps
|
||||
|
||||
- Smoke tests accidentally depend on local user secrets and fail in CI.
|
||||
- UI says “image sent” when only optimization happened.
|
||||
- Diagnostics copy includes data URL.
|
||||
- Docs overpromise unknown OpenRouter models.
|
||||
- Negative model smoke becomes flaky because provider upgrades model capability. If GLM 5.1 starts supporting images, update catalog and test expectation.
|
||||
|
||||
## File-by-file implementation plan
|
||||
|
||||
### Smoke script
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
scripts/smoke/agent-attachments-smoke.mjs
|
||||
```
|
||||
|
||||
Optional helper:
|
||||
|
||||
```text
|
||||
scripts/smoke/lib/write-red-card-png.mjs
|
||||
scripts/smoke/lib/redact-smoke-log.mjs
|
||||
```
|
||||
|
||||
Do not put live smoke in normal test suite by default.
|
||||
|
||||
### Documentation
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
docs/team-management/agent-attachments.md
|
||||
```
|
||||
|
||||
Link it from:
|
||||
|
||||
```text
|
||||
docs/team-management/debugging-agent-teams.md
|
||||
```
|
||||
|
||||
only if it helps support/debugging.
|
||||
|
||||
### UI polish tests
|
||||
|
||||
Potential tests:
|
||||
|
||||
```text
|
||||
test/renderer/components/team/messages/MessageComposer.test.tsx
|
||||
test/renderer/utils/attachmentUtils.test.ts
|
||||
src/features/agent-attachments/**/*.test.ts
|
||||
```
|
||||
|
||||
## Smoke script behavior details
|
||||
|
||||
### CLI options
|
||||
|
||||
```bash
|
||||
node scripts/smoke/agent-attachments-smoke.mjs --all
|
||||
node scripts/smoke/agent-attachments-smoke.mjs --case codex-native-gpt-5-4-mini
|
||||
node scripts/smoke/agent-attachments-smoke.mjs --json /tmp/attachment-smoke.json
|
||||
```
|
||||
|
||||
### Skip logic
|
||||
|
||||
```ts
|
||||
if (case.envRequired?.some(name => !process.env[name])) {
|
||||
return { status: 'skipped', reason: `${name} not set` };
|
||||
}
|
||||
```
|
||||
|
||||
Missing auth should be `failed` if the runtime is expected to be locally logged in, but OpenRouter env cases can be `skipped` if key absent.
|
||||
|
||||
### Child process cleanup
|
||||
|
||||
```ts
|
||||
const child = spawn(command, args, { detached: true });
|
||||
try {
|
||||
return await waitForResult(child, timeoutMs);
|
||||
} finally {
|
||||
if (!child.killed) {
|
||||
try { process.kill(-child.pid!, 'SIGTERM'); } catch {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Be careful on macOS where process groups may differ. If not detached, kill child pid only.
|
||||
|
||||
## Docs examples
|
||||
|
||||
### Supported model section
|
||||
|
||||
```md
|
||||
## Verified image-capable models
|
||||
|
||||
- Claude subscription via stream-json
|
||||
- Codex native GPT-5.4-mini via `--image`
|
||||
- OpenCode OpenAI GPT-5.4-mini
|
||||
- OpenCode OpenRouter Kimi K2.6
|
||||
- OpenCode OpenRouter GLM 4.5V
|
||||
```
|
||||
|
||||
### Unsupported model section
|
||||
|
||||
```md
|
||||
## Known unsupported or text-only models
|
||||
|
||||
- OpenCode OpenRouter GLM 5.1: accepts text but does not support image input in live smoke.
|
||||
```
|
||||
|
||||
### Troubleshooting section
|
||||
|
||||
```md
|
||||
If OpenCode says `Provider not found: openrouter`, connect OpenRouter in provider management or provide `OPENROUTER_API_KEY` for smoke tests.
|
||||
```
|
||||
|
||||
## More polish edge cases
|
||||
|
||||
| Edge case | UI/docs behavior |
|
||||
|---|---|
|
||||
| User sees “not verified” for a model they know supports vision | docs explain conservative default and how to request/verify model |
|
||||
| Live smoke passes for a previously unknown model | update capability catalog in separate commit |
|
||||
| Provider changes model behavior | negative smoke catches mismatch, catalog updated deliberately |
|
||||
| User reports model saw image but UI blocked | add override only after reproducing or provider metadata confirms |
|
||||
| User reports image too blurry | adjust Phase 1 quality policy, not provider adapters |
|
||||
| User reports process crashed with image | diagnostics should include payload bytes and runtime stderr tail, not base64 |
|
||||
|
||||
## Final release decision tree
|
||||
|
||||
```text
|
||||
If Phase 1 is green but Phase 2 is risky -> ship safer budget validation only.
|
||||
If Claude is green but Codex is flaky -> ship Claude only, keep Codex blocked.
|
||||
If Codex is green but OpenCode model gate is incomplete -> ship Claude+Codex, keep OpenCode blocked.
|
||||
If OpenCode OpenAI is green but OpenRouter is unstable -> allow OpenAI, block OpenRouter unknowns.
|
||||
```
|
||||
|
||||
Do not hold safer early phases hostage to later dynamic OpenRouter model risk.
|
||||
|
||||
## Phase 5 exit criteria
|
||||
|
||||
Phase 5 is complete only when:
|
||||
|
||||
- smoke harness can run selected cases independently;
|
||||
- smoke harness redacts secrets and data URLs;
|
||||
- docs list verified and unsupported models separately;
|
||||
- UI copy does not overpromise unknown models;
|
||||
- copied diagnostics include enough metadata to debug without leaking payload;
|
||||
- release checklist is green or explicitly scoped down.
|
||||
|
||||
## Smoke harness case definitions
|
||||
|
||||
```ts
|
||||
const cases: AttachmentSmokeCase[] = [
|
||||
{
|
||||
id: 'claude-streaming-haiku',
|
||||
runtime: 'claude',
|
||||
command: 'node',
|
||||
args: ['scripts/smoke/runners/claude-sdk-image.mjs'],
|
||||
expected: /\bred\b/i,
|
||||
timeoutMs: 60_000,
|
||||
},
|
||||
{
|
||||
id: 'codex-native-gpt-5-4-mini',
|
||||
runtime: 'codex',
|
||||
command: 'codex',
|
||||
args: ['exec', '--json', '--skip-git-repo-check', '-C', '/tmp', '--model', 'gpt-5.4-mini', '--image', '$IMAGE', '-'],
|
||||
stdin: 'Look at the attached image. Reply with exactly one word: red, green, or blue.',
|
||||
expected: /\bred\b/i,
|
||||
timeoutMs: 90_000,
|
||||
},
|
||||
{
|
||||
id: 'opencode-openrouter-glm-5-1-negative',
|
||||
runtime: 'opencode',
|
||||
envRequired: ['OPENROUTER_API_KEY'],
|
||||
expectCapabilityBlocked: true,
|
||||
},
|
||||
];
|
||||
```
|
||||
|
||||
For negative cases after Phase 4, prefer testing the app capability gate rather than spending OpenRouter tokens calling known unsupported models.
|
||||
|
||||
## Diagnostics copy example
|
||||
|
||||
```text
|
||||
Attachment delivery diagnostic
|
||||
team: atlas-hq
|
||||
recipient: jack
|
||||
runtime: opencode
|
||||
model: openrouter/z-ai/glm-5.1
|
||||
attachments: 1 image
|
||||
optimized bytes: 612 KB
|
||||
estimated serialized bytes: 842 KB
|
||||
capability: unsupported
|
||||
reason: GLM 5.1 is text-only for image input in verified OpenCode/OpenRouter smoke.
|
||||
```
|
||||
|
||||
No base64, no data URL, no API key.
|
||||
|
||||
## Documentation warnings
|
||||
|
||||
Docs must say:
|
||||
|
||||
```text
|
||||
Verified model support can change. If a model starts or stops accepting images, update the capability catalog and smoke expectations in a separate commit.
|
||||
```
|
||||
|
||||
Docs must not say:
|
||||
|
||||
```text
|
||||
All OpenRouter models support screenshots.
|
||||
```
|
||||
|
||||
## Final pre-release manual checklist
|
||||
|
||||
- Send text-only message to Claude lead.
|
||||
- Send optimized image to Claude lead.
|
||||
- Send text-only message to Codex lead.
|
||||
- Send image to Codex lead.
|
||||
- Send text-only direct message to OpenCode member.
|
||||
- Send image to OpenCode OpenAI member.
|
||||
- Send image to OpenCode Kimi K2.6 member if OpenRouter configured.
|
||||
- Attempt image to OpenCode GLM 5.1 and confirm it blocks before send.
|
||||
- Attempt oversized image and confirm it blocks before send.
|
||||
- Copy diagnostics and confirm no data URL/base64/key.
|
||||
|
||||
## Phase 5 bug traps
|
||||
|
||||
| Trap | Prevention |
|
||||
|---|---|
|
||||
| live smoke consumes tokens in normal CI | not part of default test command |
|
||||
| smoke fails due missing auth and blocks release | missing optional env is skipped, not failed |
|
||||
| docs become stale | capability catalog references live smoke date |
|
||||
| unsupported negative model changes behavior | update catalog/test explicitly |
|
||||
| copied diagnostics leak image data | redaction unit tests |
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -1,6 +1,6 @@
|
|||
# OpenCode Model Gauntlet Results
|
||||
|
||||
Generated: 2026-05-08T21:13:58.089Z
|
||||
Generated: 2026-05-08T21:28:15.433Z
|
||||
|
||||
Runs per model: 1
|
||||
Recommended threshold: average >= 80, successful runs >= 1, consistency >= 85, hard failures = 0
|
||||
|
|
@ -13,25 +13,25 @@ Scoring weights: launchBootstrap=15, directReply=10, peerRelayAB=15, peerRelayBC
|
|||
|
||||
| Model | Verdict | Confidence | Readiness | Consistency | Score Spread | Behavior Avg | Overall Avg | Counted | Pass Runs | Weakest Stage | Weakest TaskRef | Dominant Failure | Blockers | Provider Infra | Runtime Transport | Model Fails | Protocol Runs | p50 | p95 |
|
||||
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| `opencode/big-pickle` | Infra blocked | blocked | 0 | 0 | 0 | n/a | 70 | 0/1 | 0/1 | concurrentReplies 0/1 (0%) | concurrentBob 0/1 (0%) | provider-infra | overall average 70 < 80; successful runs 0 < 1; consistency score 0 < 85; provider-infra failures 1; highest weighted stage loss concurrentReplies=15; weakest taskRefs concurrentBob=0/1 (0%); protocol violations in 1 runs | 1 | 0 | 0 | 1 | 281016ms | 281016ms |
|
||||
| `opencode/big-pickle` | Recommended | low | 100 | 100 | 0 | 100 | 100 | 1/1 | 1/1 | cleanTranscript 1/1 (100%) | concurrentBob 1/1 (100%) | none | - | 0 | 0 | 0 | 0 | 173403ms | 173403ms |
|
||||
|
||||
## opencode/big-pickle
|
||||
|
||||
Readiness score: 0.
|
||||
Readiness score: 100.
|
||||
|
||||
Score stability: n/a.
|
||||
Score stability: consistency=100, min=100, max=100, spread=0, stdDev=0, samples=1.
|
||||
|
||||
Recommendation blockers: overall average 70 < 80; successful runs 0 < 1; consistency score 0 < 85; provider-infra failures 1; highest weighted stage loss concurrentReplies=15; weakest taskRefs concurrentBob=0/1 (0%); protocol violations in 1 runs.
|
||||
Recommendation blockers: -.
|
||||
|
||||
Weighted stage impact: concurrentReplies:loss=15, failed=1, pass=0/1 (0%); taskRefs:loss=10, failed=1, pass=0/1 (0%); noDuplicateTokens:loss=5, failed=1, pass=0/1 (0%).
|
||||
Weighted stage impact: -.
|
||||
|
||||
Stage pass rates: launchBootstrap:1/1 (100%), directReply:1/1 (100%), peerRelayAB:1/1 (100%), peerRelayBC:1/1 (100%), concurrentReplies:0/1 (0%), taskRefs:0/1 (0%), cleanTranscript:1/1 (100%), noDuplicateTokens:0/1 (0%), latencyStable:1/1 (100%).
|
||||
Stage pass rates: launchBootstrap:1/1 (100%), directReply:1/1 (100%), peerRelayAB:1/1 (100%), peerRelayBC:1/1 (100%), concurrentReplies:1/1 (100%), taskRefs:1/1 (100%), cleanTranscript:1/1 (100%), noDuplicateTokens:1/1 (100%), latencyStable:1/1 (100%).
|
||||
|
||||
TaskRef pass rates: directReply:1/1 (100%), peerRelayAB:1/1 (100%), peerRelayBC:1/1 (100%), concurrentBob:0/1 (0%), concurrentTom:1/1 (100%).
|
||||
TaskRef pass rates: directReply:1/1 (100%), peerRelayAB:1/1 (100%), peerRelayBC:1/1 (100%), concurrentBob:1/1 (100%), concurrentTom:1/1 (100%).
|
||||
|
||||
Protocol totals: badMessages=0, duplicateOrMissingTokens=2, affectedRuns=1.
|
||||
Protocol totals: badMessages=0, duplicateOrMissingTokens=0, affectedRuns=0.
|
||||
|
||||
| Run | Outcome | Category | Score | Counted | Duration | Failed Stages | Slowest Stage | TaskRefs | Protocol | Diagnostics |
|
||||
| ---: | --- | --- | ---: | --- | ---: | --- | --- | --- | --- | --- |
|
||||
| 1 | provider-infra-blocked | provider-infra | 70 | no | 281016ms | concurrentReplies, taskRefs, noDuplicateTokens | concurrentReplies:189928ms | directReply:ok, peerRelayAB:ok, peerRelayBC:ok, concurrentBob:fail, concurrentTom:ok | token=GAUNTLET_CONCURRENT_BOB_OK_1+GAUNTLET_CONCURRENT_TOM_OK_1 | concurrentBob: Timed out waiting for OpenCode reply in /var/folders/7b/ydmc_b0n251bc4hss4tz8y880000gn/T/opencode-semantic-gauntlet-ZwZPyq/.claude/teams/opencode-semantic-realistic-gauntlet-opencode-big-pickle-1778274838090-1/inboxes/user.js |
|
||||
| 1 | passed | none | 100 | yes | 173403ms | - | concurrentReplies:77988ms | directReply:ok, peerRelayAB:ok, peerRelayBC:ok, concurrentBob:ok, concurrentTom:ok | - | runId=55b87bf5-abe4-4c21-9b98-db1ed8c22cfb |
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue