vision agent

# valoswiss-vision-agent (34°)

**Macro-categoria**: 👁️ INFRA/AI/META (integrazione trasversale vision-language)
**Scope**: Vision-language model (VLM) per interpretazione screenshot broker, UI automation, layout complesso documenti, valutazione foto immobili/asset fisici
**Born**: 2026-05-03 (W1 sidecar Python VLM + W2 NestJS module + W3 frontend upload + W4 computer-use automation + W5 real-estate photo + W6 chart extraction pipeline)
**Owner downstream**: ADVISOR (lettura output chart/doc) · SUPERVISOR/ADMIN (queue + cost ledger + computer-use sessions) · DOC-OPS (documento layout enhancement)
**Last aligned**: 2026-05-03 V20

---

## §0 · Pre-flight check (entry rituale dell'agente)

Prima di ogni intervento, verifica in quest'ordine:

1. **Branch + working tree**
   ```bash
   cd ~/git/valoswiss && git status --short && git log -3 --oneline
   ```
2. **Sidecar Python health**
   ```bash
   curl -s http://127.0.0.1:8893/healthz | jq .
   ```
   Deve ritornare `{"status":"ok","version":"...","defaultModel":"claude-opus-4-7-vision","computerUseEnabled":true|false}`. Se 502/connection refused → sidecar PM2 down: `pm2 list | grep vision-agent-py`.
3. **NestJS proxy health**
   ```bash
   curl -s http://127.0.0.1:4010/api/vision-agent/health -H "Cookie: valo_token=<dev-token>"
   ```
   Deve ritornare `{ sidecar:{status:'ok'}, circuitBreaker:{state:'closed', failures:0}, modelsAvailable:['claude-opus-4-7-vision','gpt-5-vision','gemini-3-1-ultra'] }`.
4. **Prisma schema sync**
   ```bash
   cd apps/api && npx prisma migrate status
   ```
   Verifica che le 2 model `VisionAnalysis` + `VisionScreenshot` + enum `VisionAnalysisStatus` + enum `VisionAnalysisType` siano applicati (migration `vision_agent_w2`).
5. **Tenant configs**: `tenants/ws.json` e `tenants/az.json` devono avere `"visionAgent": true` subito dopo `docIntelligence`.
6. **Persona pack**: `apps/api/src/common/persona-packs/persona-packs.constants.ts` deve avere `'visionAgent'` in `defaultModules` per `ADVISOR` + `SUPERVISOR` + `ADMIN` (NON in PROSPECT/RETAIL_CLIENT/AFFLUENT_CLIENT/UHNW_CLIENT/FAMILY_OFFICE_PRINCIPAL — PII foto).
7. **Module registry**: `apps/web/src/lib/module-registry.ts` deve esporre entry `visionAgent` con `sidebarSection: 'OPERARE'`, `requiredRole: 'ADVISOR'`, `personaHint: 'visual'`, icon `👁️`.
8. **R-Audit gate**: prima di qualsiasi commit su file CRITICAL (vedi §3), eseguire `npx tsx scripts/r-audit.ts <file> --validate-business-logic`.

Se uno qualunque dei 7 punti fallisce, **fermati e annota la deviazione** prima di procedere — la 3-Point Registration V16 è invariante non negoziabile (vedi `feedback_new_module_registration.md`).

---

## §1 · Aree di competenza

### 1.1 Reference repos

| Repo | License | Stars | Ruolo in ValoSwiss |
|---|---|---|---|
| **niuzaisheng/ScreenAgent** | Apache 2.0 | 2.1k | VLM computer control — agent loop screenshot→action→feedback |
| **GLM-4.6V** (THUDM/GLM-4) | Apache 2.0 | 17k | Vision tool use + UI screenshot understanding |
| **Anthropic computer-use** | MIT | — | Browser/desktop automation via Claude Opus tool use |
| **OpenAI o5 vision** | proprietary | — | GPT-5 vision multi-image reasoning |
| **Gemini 3.1 Vision Ultra** | proprietary | — | Google multimodal 3.1 Ultra vision endpoint |
| **LLaVA** (haotian-liu/LLaVA) | Apache 2.0 | 20k | Open-source VLM locale fallback Ollama |
| **IDEFICS** (HuggingFace) | Apache 2.0 | — | Open-source multimodal local fallback |

### 1.2 Use cases ValoSwiss (4 domini)

#### A. Chart screenshot interpretation
- Input: screenshot PNG/JPEG da broker (Bloomberg Terminal, Refinitiv, Interactive Brokers, Degiro, Fineco, SIX Swiss Exchange)
- Output: JSON strutturato con OHLCV series estratta, indicatori tecnici visibili (RSI, MACD, Bollinger), supporti/resistenze annotati, timeframe rilevato
- Modello primario: Claude Opus 4.7 vision (instruction-tuned chart reading)
- Fallback: GPT-5 vision → Gemini 3.1 Ultra → LLaVA locale (degraded accuracy)
- Integrazione: `strategic-watch` consuma output per executive memo auto-populated

#### B. UI automation (computer-use Anthropic)
- Input: task description in lingua naturale + URL/app target
- Output: sequenza di action primitives (click, type, scroll, screenshot) con screenshot evidence step-by-step
- Engine: Anthropic computer-use tool via Claude Opus 4.7 (tools: `computer`, `text_editor`, `bash`)
- Use case: login banche private svizzere senza PSD2 (fallback browser-agent), form filling KYC, report download
- Session isolata: Docker container VM ephemeral — NEVER persistent state tra sessioni
- Integrazione: `banking-integration` fallback scraping CH private banks

#### C. Document layout understanding
- Input: PDF/immagine con layout complesso (multi-colonna, tabelle nested, headers floating, footnotes interleaved)
- Output: layout map JSON `{regions: [{type, bbox, text, confidence}]}` che doc-intelligence usa per migliorare extraction accuracy
- Complementare a: `doc-intelligence` (Docling primario, VisionAgent enhancement per layout anomali)
- Trigger: doc-intelligence confidence < 0.75 su regione → chiama VisionAgent per re-parse visuale

#### D. Photo evidence (real estate + asset valuation)
- Input: foto smartphone HEIC/JPEG di immobili, opere d'arte, auto di lusso, orologi, barche
- Output: valuation evidence JSON (stato conservazione, tipologia asset, comparabili stimati, flag anomalie)
- Uso: UHNW family office asset register, collateral valuation, due diligence in-loco
- Integrazione: `real-estate` module (mapping `RealEstateAsset.photoEvidenceId → VisionAnalysis.id`)
- VISION-PII-REDACTION: face/person detection su foto → blur automatico prima di persistence (GDPR)

### 1.3 Cascade model selection

```
Priority waterfall per task type:
┌─────────────────────────────────────────────────────────┐
│  CHART_EXTRACTION     → Claude Opus 4.7 vision          │
│                         → GPT-5 vision                  │
│                         → Gemini 3.1 Ultra              │
│                         → LLaVA (local, degraded)       │
├─────────────────────────────────────────────────────────┤
│  COMPUTER_USE         → Claude Opus 4.7 (solo Anthropic)│
│                         NO fallback (feature exclusive) │
├─────────────────────────────────────────────────────────┤
│  DOCUMENT_LAYOUT      → Gemini 3.1 Ultra (long context) │
│                         → Claude Opus 4.7 vision        │
│                         → GPT-5 vision                  │
│                         → IDEFICS (local, degraded)     │
├─────────────────────────────────────────────────────────┤
│  PHOTO_EVIDENCE       → GPT-5 vision (object detail)    │
│                         → Gemini 3.1 Ultra              │
│                         → Claude Opus 4.7 vision        │
│                         → LLaVA (local, degraded)       │
└─────────────────────────────────────────────────────────┘
```

### 1.4 Persona visibility

- **ADVISOR** (ws+az): submit analisi su propri clienti/asset; read output chart extraction + doc layout
- **SUPERVISOR/ADMIN**: cross-tenant + computer-use session management + cost ledger view + PII redaction audit log
- **DOC-OPS**: trigger layout enhancement su documenti in coda bassa-confidence
- **CLIENT/PROSPECT/RETAIL_CLIENT/AFFLUENT_CLIENT/UHNW_CLIENT/FAMILY_OFFICE_PRINCIPAL**: NEGATO — foto clienti PII privacy + output VLM non auditabile per cliente finale

---

## §2 · Pattern di codice

### 2.1 Prisma schema (migration `vision_agent_w2`)

```prisma
enum VisionAnalysisStatus {
  PENDING
  RUNNING
  SUCCESS
  FAILED
  PII_REDACTION_PENDING
}

enum VisionAnalysisType {
  CHART_EXTRACTION
  COMPUTER_USE
  DOCUMENT_LAYOUT
  PHOTO_EVIDENCE
}

model VisionAnalysis {
  id              String               @id @default(uuid())
  tenantSlug      String
  analysisType    VisionAnalysisType
  status          VisionAnalysisStatus @default(PENDING)
  requestedByUserId String
  inputRef        String               // documentVaultId | URL | uploadPath
  outputJson      Json?                // structured result
  modelUsed       String?
  tokensIn        Int?
  tokensOut       Int?
  costUsd         Decimal?             @db.Decimal(10, 6)
  piiRedacted     Boolean              @default(false)
  jobId           String?              @unique
  createdAt       DateTime             @default(now())
  updatedAt       DateTime             @updatedAt
  screenshots     VisionScreenshot[]

  @@index([tenantSlug, analysisType, createdAt(sort: Desc)])
  @@index([tenantSlug, requestedByUserId])
  @@index([status, createdAt])
}

model VisionScreenshot {
  id               String         @id @default(uuid())
  visionAnalysisId String
  analysis         VisionAnalysis @relation(fields: [visionAnalysisId], references: [id], onDelete: Cascade)
  stepIndex        Int
  storagePath      String         // vault path post-redaction
  width            Int
  height           Int
  capturedAt       DateTime       @default(now())

  @@index([visionAnalysisId, stepIndex])
}
```

**Wave 1.6 — getter espliciti** su `TenantPrismaService`:
```typescript
// apps/api/src/common/tenant-prisma/tenant-prisma.service.ts
get visionAnalysis() { return this.client.visionAnalysis; }
get visionScreenshot() { return this.client.visionScreenshot; }
```
NON usare il legacy cast as-any su prisma — pre-commit triage blocca.

### 2.2 NestJS service — VisionAnalysisService

```typescript
// apps/api/src/modules/vision-agent/vision-analysis.service.ts
@Injectable()
export class VisionAnalysisService {
  private readonly logger = new Logger(VisionAnalysisService.name);

  constructor(
    @Optional() private readonly prisma: TenantPrismaService,
    private readonly sidecar: VisionSidecarClient,
    private readonly costLedger: AiCostLedgerService,
    private readonly piiRedaction: PiiRedactionService,
  ) {}

  async submitAnalysis(dto: SubmitVisionAnalysisDto, userId: string): Promise<VisionAnalysisEntity> {
    const record = await this.prisma.visionAnalysis.create({
      data: {
        tenantSlug: dto.tenantSlug,
        analysisType: dto.analysisType,
        status: 'PENDING',
        requestedByUserId: userId,  // NON field variant legacy — vedi §7
        inputRef: dto.inputRef,
      },
    });

    // enqueue async al sidecar Python :8893
    const { jobId } = await this.sidecar.enqueueAsync({
      analysisId: record.id,
      analysisType: dto.analysisType,
      inputRef: dto.inputRef,
      tenantSlug: dto.tenantSlug,
      modelOverride: dto.modelOverride,
    });

    return this.prisma.visionAnalysis.update({
      where: { id: record.id },
      data: { jobId, status: 'RUNNING' },
    });
  }

  async persistResult(analysisId: string, payload: VisionResultPayload): Promise<void> {
    // PII redaction BEFORE persist se photo evidence
    let outputJson = payload.outputJson;
    if (payload.analysisType === 'PHOTO_EVIDENCE' && payload.facesDetected) {
      outputJson = await this.piiRedaction.redactFaces(outputJson);
      await this.prisma.visionAnalysis.update({
        where: { id: analysisId },
        data: { status: 'PII_REDACTION_PENDING' },
      });
    }

    await this.prisma.visionAnalysis.update({
      where: { id: analysisId },
      data: {
        status: 'SUCCESS',
        outputJson,
        modelUsed: payload.modelUsed,
        tokensIn: payload.tokensIn,
        tokensOut: payload.tokensOut,
        costUsd: payload.costUsd,
        piiRedacted: payload.facesDetected ?? false,
      },
    });

    // cost ledger non-blocking
    try {
      await this.costLedger.record({
        taskId: `vision-agent:${analysisId}`,
        model: payload.modelUsed,
        tokensIn: payload.tokensIn,
        tokensOut: payload.tokensOut,
        costUsd: payload.costUsd,
        tenantSlug: payload.tenantSlug,
      });
    } catch (e) {
      this.logger.warn('cost-ledger-record-failed', e);
    }
  }
}
```

### 2.3 REST endpoints — VisionAgentController

```typescript
// apps/api/src/modules/vision-agen

…[truncato — apri il file MD per testo completo]