How do you evaluate an AI underwriting platform for a P&C carrier or MGA?

Evaluating an AI underwriting platform requires asking five questions in order. If a vendor fails the first question, the rest don't matter.

1. Audit trail — is every LLM call logged?

• Every prompt, every model name, every output, every trace id, every human decision, every downstream notification must log to an append-only table.
• The artifact must be exportable as a single document for a state DOI data request, a NAIC market-conduct exam, or a Lloyd's coverholder quarterly review.
• Ask for a sample audit pack from a recent decision. If the vendor can't produce one in the demo, walk.

2. Data fabric — what's the platform grounded on?

• A platform that only calls an LLM is a chat wrapper, not an insurance tool.
• The minimum data fabric for US commercial property: FEMA NFIP / NFHL, NOAA HURDAT2, OFAC SDN, ZIP / Census, OS Postcodes (for UK).
• Add ISO ClaimSearch + NICB Forewarn for claims.
• For specialty: Verisk, D&B, SEC EDGAR.
• Test: ask for a postcode and have the platform return the FEMA flood zone with a citation. If it can't, the fabric is missing.

3. Architecture — multi-agent or single chatbot?

• Single-agent platforms collapse at the third specialist task. You'll hit the context window or the prompt complexity ceiling.
• Multi-agent platforms route each task (parse, risk, flood, pricing, compliance, treaty, portfolio, memo) to a dedicated model bracket. Each agent is replaceable.
• Bonus: ask if agents run in parallel where independent. Sequential chains are 5–10× slower without an architectural reason.

4. Deployment — where does our data go?

• Multi-tenant cloud. Easiest to deploy, hardest to procure.
• VPC / dedicated tenant. Required for most large carriers.
• On-premise. Required for some Lloyd's syndicates and government-adjacent carriers.
• Bring-your-own-LLM. Your data routes through your Bedrock / Vertex / Anthropic-direct account. The vendor is the orchestration layer; the model perimeter stays inside your VPC.

If the vendor's only deployment is multi-tenant cloud on their LLM keys, procurement will not approve at any enterprise carrier.

5. Configurability — does it match our filed product?

• Appetite filters per state, per line, per occupancy. Editable without a code release.
• Decision authorities: who can bind up to what TSI / premium / loss reserve. Audit-logged when over-authority pays trigger refer-to-senior.
• Statutory letters per state with cited regulation. UCSPA-compliant for the four largest states minimum (CA, FL, NY, TX).
• STP rules — auto-bind, auto-decline, refer thresholds — owned and edited by the carrier's CUO.

A platform that meets all five gets you to a 90-day pilot. The 91st-day question — is it actually changing your loss ratio — answers itself by month three.