Simulation QA agents (Platform owner)

Last updated: 2026-05-25 Owner: Product Writer Audience: Platform owner (super_admin, ops_admin)

Purpose

Explain how the team of autonomous AI agents works. Against a dedicated simulation tenant ("Tecnolife", slug tecnolife-sim), the agents perform realistic product actions, capture errors and regressions, and surface them in the /workspace/platform/simulation dashboard for Platform owner review.

Key concepts

• Dedicated simulation tenant: tecnolife-sim (UUID c07c899f-dcad-4227-8758-4cddc085c99e). Separate from the real paying tenant called "Tecnolife". Hard-coded guard against cross-tenant contamination.
• Agent = role + scenarios: each agent impersonates a product role (HR admin, manager, user, owner) and runs an action sequence via API.
• Findings = classified NDJSON events: every step of every scenario emits an event (request/response/latency/error). A deterministic classifier assigns finding_hint (bug, regression, performance, ux_friction, suggestion).
• Persistence: opt-in. Run + findings persisted into platform_simulation_runs + platform_simulation_findings. RLS blocked, accessible only by Platform owner via UI or service-role.

Prerequisites

• Platform owner role (super_admin / ops_admin / billing_admin)
• migrations 0080_hr_candidates_soft_delete.sql + 0081_platform_simulation_findings.sql applied
• env SUPABASE_SERVICE_ROLE_KEY to run the bootstrap script (one-shot)

Quick access

• Findings dashboard: /workspace/platform/simulation
• Run agents (CLI): node horis-saas/apps/web/scripts/simulation/orchestrator.mjs
• Bootstrap tenant: node horis-saas/apps/web/scripts/bootstrap-tecnolife-sim.mjs

Operating steps

1. Bootstrap the tenant (one-shot, already done)

Creates the tenant + 30 seed users + workforce blueprint. Idempotent.

```bash cd horis-saas node apps/web/scripts/bootstrap-tecnolife-sim.mjs ```

Result: tenant tecnolife-sim with 1 owner + 1 admin + 1 billing_admin + 1 hr_admin + 2 managers + 30 users, 4 projects, ~100 timesheet rows, ~700 presence entries, 51 skill catalog.

2. Run a simulation

```bash cd horis-saas HORIS_SIMULATION_ENABLED=true \ HORIS_SIMULATION_PERSIST=true \ node apps/web/scripts/simulation/orchestrator.mjs ```

The orchestrator:

1. Pre-flight: kill-switch check + simulation-tenant targeting
2. Login as the HR admin of the simulation tenant
3. Selects the due scenarios (cadence + profile), then runs them: isolate ones sequentially first, the rest in parallel (concurrency cap)
4. Prints NDJSON on stdout + saves to tmp/simulation-runs/run-<uuid>.ndjson
5. (If PERSIST=true) ingests into DB → run + findings visible in dashboard

Scheduling env (HORIS_SIMULATION_CONCURRENCY_01):

• HORIS_SIMULATION_PROFILE = full (default, applies cadence) | all (everything, ignores cadence) | canary (canary only, for a high-frequency cron)
• HORIS_SIMULATION_CONCURRENCY = N parallel workers (default 4; 1 = sequential)
• HORIS_SIMULATION_DAILY_HOUR = UTC hour for daily scenarios (default 3)

3. Read the dashboard

/workspace/platform/simulation:

• KPI strip: total runs, events, PASS/FAIL scenarios aggregate
• Recent runs table (last 20)
• Click on a run → findings table with hint filter
• Per-hint aggregates shown as colored chips

4. Export findings (CSV / JSON / MD)

"Export findings" card below the KPI strip:

• Format: JSON (complete data with raw_event and payload_summary), CSV (Excel/Numbers, flat), Markdown (human-readable, copy-paste)
• Scope: "Selected run" (only the currently expanded one) or "Last 7 days" (max 100 runs)
• Respects the current hint filter: if you selected bug, only bugs are exported
• Endpoint: GET /api/platform/simulation/export?format={json|csv|md}&run_id?&hint?&since?&limit?

5. Read the trends (heatmap + latency)

"Trend & Heatmap" section below the KPI strip (HORIS_TREND_ANALYTICS_01):

• Scenario × day heatmap: one row per scenario, one column per day. Cell color = worst severity seen that day (red = bug/regression, amber = performance, sky = ux_friction/suggestion, green = healthy, grey = no run). Hover for event/findings/hint counts.
• P50 / P95 latency trend: two line charts (overall + Talent Partner canary) with per-day percentiles. P50 dark line, P95 amber line.
• Window selector: last 7 / 14 / 30 days.
• Aggregated from platform_simulation_findings (healthy events count for the baseline too).
• Endpoint: GET /api/platform/simulation/trends?days={1..90}

Scenarios currently covered

| Scenario | Role | What it does | |---|---|---| | hr-screen-candidate | hr_admin | Create candidate + 2 status transitions (transactional email triggers) + soft-delete | | hr-upload-extract-cv | hr_admin | Create candidate + upload PDF + AI extract + verify draft + cleanup | | hr-trash-cycle | hr_admin | Create + soft-delete + verify trash view + restore + cleanup | | hr-certification-cycle | hr_admin | Create cert for another user + list verify + update status + delete (SCENARIOS_02 W01) | | hr-ccnl-assignment | hr_admin | GET CCNL catalog + PATCH ccnlId/level/classification + verify persistence (SCENARIOS_02 W01) | | hr-employment-classification | hr_admin | Apprenticeship happy + freelance without vat = expected 500 + HR vat = 403 PII boundary + cleanup (SCENARIOS_02 W01) | | boundary-permissions | multi (manager+user+hr) | 8 negative-path steps: manager blocked HR, user blocked other-user certs, HR blocked PII self-only (SCENARIOS_02 W02) | | hr-talent-partner-search | hr_admin | POST /api/hr-chat with realistic consulting question + verify non-empty answer + latency check (SCENARIOS_02 W03) | | user-timesheet-month | user | GET /api/timesheet/month — own month read (OPERATIONAL_EXCELLENCE_01 W02) | | user-leave-requests | user | GET /api/requests — own leave/travel list (OPERATIONAL_EXCELLENCE_01 W02) | | manager-team-capacity | manager | GET /api/team-capacity — team scope read (OPERATIONAL_EXCELLENCE_01 W02) | | manager-timesheet-reviews | manager | GET /api/timesheet/reviews?status=submitted — approval queue (OPERATIONAL_EXCELLENCE_01 W02) | | manager-ops-calendar-team | manager | GET /api/ops-calendar/team — team daily ops (OPERATIONAL_EXCELLENCE_01 W02) | | platform-tenants-overview | hr_admin+super_admin | HR blocked 403 + super_admin reads tenants 200 (OPERATIONAL_EXCELLENCE_01 W02) | | edge-race-condition-patch | hr_admin | 2 parallel PATCH → last-write-wins verify (OPERATIONAL_EXCELLENCE_01 W03) | | edge-storage-upload-burst | hr_admin | 5 consecutive CV → versioning is_current=1 verify (OPERATIONAL_EXCELLENCE_01 W03) | | edge-token-invalid | anon | 4 token boundary steps: malformed/expired/empty/unknown-route (OPERATIONAL_EXCELLENCE_01 W03) | | canary-talent-partner-latency | hr_admin | 5-shot /api/hr-chat with P50/P95 calc + classifies performance/regression/bug per 25s/45s/55s thresholds (OPERATIONAL_EXCELLENCE_01 W04) | | llm-explorer | hr_admin | LLM agent that observes state and decides actions from a safe catalog (LLM_DRIVEN_SCENARIOS_01). OFF by default, self-skips | | canary-skill-expansion | hr_admin | Daily probe on /api/hr-chat: asks for skills adjacent to Kubernetes and checks the answer cites the family siblings → regression if the expansion does not happen (SKILL_EXPANSION_HARDENING_01) |

Email alert on fail_count > 0 is sent automatically to pabloliuzzi@gmail.com via Resend (OPERATIONAL_EXCELLENCE_01 W01). Alert killswitch: env HORIS_ALERT_EMAIL_ENABLED=false.

Cadence & concurrency (HORIS_SIMULATION_CONCURRENCY_01)

The scenarios/registry.mjs registry assigns each scenario a cadence and an isolate flag. The orchestrator runs the isolate ones (latency sensitive) sequentially, then everything else in parallel with a cap.

| Cadence | When it runs (hourly cron) | Scenarios | |---|---|---| | hourly | every run | core HR + manager/user reads + boundary + canary | | every_3h | hour % 3 == 0 | CV upload + consulting quick-wins (cert/ccnl/employment) | | every_6h | hour % 6 == 0 | talent-partner-search (expensive LLM, isolate) | | daily | hour == 3 UTC | edge case stress (race / storage burst / token) |

• Isolate: canary-talent-partner-latency + hr-talent-partner-search run alone → clean latency, not contaminated by self-induced load.
• GitHub Actions: two crons — 0 * * * * profile full, 30 * * * * profile canary (high-frequency latency monitor). Default concurrency 4.

Browser-level smoke (HORIS_BROWSER_LEVEL_TESTING_01)

Complementary to the API-level harness: Playwright against the deployed app to catch UI bugs APIs do not reveal (runtime JS errors, broken layouts, unwired forms). Sources in apps/web/e2e-browser/.

• Core: helpers/console-guard.ts collects console.error + pageerror

(tight allowlist) and detects horizontal overflow (broken layout).

• 3 smoke specs: login (public: form + wired submit), workspace-dashboard

(authed: nav+main, 0 JS errors), chat-hr (authed: Talent Partner composer).

• Auth: auth.setup.ts logs in once and saves the storageState; without

PLAYWRIGHT_USER_EMAIL/PASSWORD the authed specs skip.

• Run: cd apps/web && npm install && npx playwright install chromium && npm run test:e2e
• CI: workflow browser-smoke.yml (nightly 02:15 UTC + manual dispatch),

installs the browser and runs against production; report + traces as artifacts.

Kill-switch and guard (safety)

• env `HORIS_SIMULATION_ENABLED`: must be true or 1, otherwise the orchestrator aborts. Default off.
• Tenant guard: assertSimTenant checks orgId, slug, email domain. Refuse-and-exit if any of them does not match tecnolife-sim / tecnolife-sim.test.
• Hardcoded forbidden slugs: tecnolife, acme-trial, acme-uat, nbn → impossible to mistakenly hit production tenants.

Finding hint classification

| Hint | Severity | When emitted (classifier rule) | |---|---|---| | bug | high | error non-null or HTTP 5xx or 0 | | regression | high | HTTP 4xx other than 404/409 (expected outcomes for missing resources / conflicts) | | performance | medium | response.latencyMs > 3000 | | ux_friction | low | explicit override from the scenario (e.g. fail-soft warning on extract) | | suggestion | low | explicit override (not auto-emitted yet) | | null / info | info | all healthy |

All events land in platform_simulation_findings even when healthy — needed for trend and baseline.

API endpoints

| Method | URL | Description | |---|---|---| | GET | /api/platform/simulation/runs | Paginated list of recent runs | | GET | /api/platform/simulation/findings?run_id=...&hint=... | Findings per run, filterable by hint/severity/scenario, with aggregates | | GET | /api/platform/simulation/trends?days=... | Scenario × day heatmap + P50/P95 latency trend over the window (HORIS_TREND_ANALYTICS_01) |

All gated by PLATFORM_PERMISSIONS.TENANTS_READ.

Security & GDPR

• The sim tenant data is fake (users @tecnolife-sim.test, candidates sim-*@tecnolife-sim.test). No real PII.
• Tables platform_simulation_* live at Platform scope, RLS active with no policy: no client sees their data in the findings (and they couldn't — the sim tenant is separate).
• Cleanup: scenarios soft-delete the candidates they create. For a full wipe of the sim tenant, run SQL manually via Studio.

LLM-driven exploration (HORIS_LLM_DRIVEN_SCENARIOS_01)

Beyond the scripted scenarios, the llm-explorer agent uses a Claude model (reusing the ai + @ai-sdk/anthropic stack) to decide actions based on observed state, surfacing edge cases the scripts do not foresee.

• Constrained catalog: only safe tools (list/get/create/patch/soft-delete

candidate + finish). No arbitrary URLs; candidate emails forced onto @tecnolife-sim.test. Auto-cleanup of created candidates.

• OFF by default: requires HORIS_SIMULATION_LLM_ENABLED=true +

ANTHROPIC_API_KEY. When disabled it emits an info event and returns ok (zero LLM cost). Cadence daily, isolate (clean latency).

• Findings: every action goes through the instrumented apiCall → classified

like the other scenarios (a 500 during exploration = bug). Anomalies the agent reports in its summary → suggestion hint (human review).

• Env: HORIS_SIMULATION_LLM_MODEL (default claude-haiku-4-5-20251001),

HORIS_SIMULATION_LLM_MAX_STEPS (default 8). Enable in CI via repo variable HORIS_SIMULATION_LLM_ENABLED.

Origin program

• HORIS_AGENT_SIMULATION_01 (2026-05-25) — 4 waves: Tecnolife bootstrap + agents harness + findings pipeline + dashboard
• HORIS_SIMULATION_CRON_01 (2026-05-25) — GitHub Actions hourly cron
• HORIS_SIMULATION_SCENARIOS_02 (2026-05-25) — HR consulting + boundary + Talent Partner coverage (8 scenarios)
• HORIS_SIMULATION_EXPORT_01 (2026-05-27) — findings CSV/JSON/MD export
• HORIS_OPERATIONAL_EXCELLENCE_01 (2026-05-27) — fail email alert + manager/user/owner scenarios + edge stress + Talent Partner P50/P95 canary (18 scenarios)
• HORIS_STORAGE_RESILIENCE_01 (2026-05-28) — retry on CV upload burst (fix for bug caught by cron)
• HORIS_TREND_ANALYTICS_01 (2026-06-03) — scenario × day heatmap + P50/P95 latency trend in the dashboard
• HORIS_SIMULATION_CONCURRENCY_01 (2026-06-03) — parallel orchestrator (concurrency cap) + per-scenario scheduling (high-frequency canary)
• HORIS_BROWSER_LEVEL_TESTING_01 (2026-06-03) — Playwright smoke (login + dashboard + chat-hr) for UI bugs APIs do not reveal
• HORIS_LLM_DRIVEN_SCENARIOS_01 (2026-06-03) — llm-explorer agent that decides actions from observed state (OFF by default)
• HORIS_SKILL_EXPANSION_HARDENING_01 (2026-06-10) — canary-skill-expansion canary watching the Talent Partner latent skill discovery