From System Cards to Continuous Verification

From System Cards to Continuous Verification

System cards and model documentation remain useful. They explain intended capabilities, known limitations and the results of defined evaluations. But they are a snapshot of a system that is increasingly dynamic.

Production agents change as models are updated, prompts evolve, tools are added, permissions shift and real-world workflows expose failure modes that controlled testing did not cover. A system card can describe what was evaluated at a point in time. It cannot, by itself, show whether an agent remains reliable and appropriately constrained after deployment.

The operational implication is clear: AI assurance is moving from a release activity to a continuous production capability.

The signal: verification is becoming more independent and persistent

Anthropic’s September 12 proposal to invite embedded third-party evaluators—with ongoing, employee-like access and publication rights—marks a significant shift in how frontier AI risks may be assessed. OpenAI’s CEO publicly supported the embedded-evaluator idea, according to the reporting summarised in the intelligence brief.

The important development is not simply the involvement of another reviewer. It is the proposed continuity and access. Independent evaluators would be able to inspect systems and processes over time rather than relying solely on provider-selected evidence published after a major release.

That model has a direct parallel in enterprise AI. Product teams should not expect a model provider’s documentation to answer whether their own agent is safe and dependable in a specific workflow. They need evidence from the deployed system: what it attempted, which tools it called, what authority it had, what approvals occurred, where it failed and how the failure was handled.

Independent evaluation does not replace engineering ownership. It makes ownership more credible by creating observable evidence that can be checked by people beyond the team that built the system.

What continuous verification means in practice

Continuous verification is not a single dashboard or a larger test set. It is a set of controls that connects pre-release evaluation with runtime evidence and incident response.

A practical assurance loop includes:

1. Define the behaviours that matter. Specify acceptable outcomes, prohibited actions, escalation conditions and performance expectations for each workflow. “The agent should be helpful” is too broad to verify. “The agent must not submit a payment without an approved transaction record” is testable.

1. Evaluate representative tasks. Build a test set from real workflow cases, including ambiguous requests, missing information, permission conflicts, tool failures and adversarial inputs. Track both task success and unsafe or unauthorised behaviour.

1. Instrument every meaningful action. Record the request, model and prompt versions, retrieved context, tool calls, permissions, approvals, outputs and final state. Logs should support investigation without exposing more sensitive data than necessary.

1. Make authority explicit. Separate what the agent may read, recommend, prepare and execute. High-impact actions should require appropriate approval, and the control should be enforced by the application or tool layer—not merely requested in a prompt.

1. Re-run evaluations after change. Changes to the model, system prompt, tools, data sources, policy rules or workflow should trigger relevant tests. A passing result from last month is not evidence that today’s configuration behaves the same way.

1. Test recovery as well as reasoning. Agents that use external tools are distributed systems. Test timeouts, unavailable files, duplicate requests, partial completion and interrupted runs. Verify that work can resume from a durable checkpoint and that retries do not create duplicate side effects.

1. Review incidents and feed them back into the evaluation set. A production failure should become a new test case, policy rule, alert or workflow change where appropriate. Otherwise the organisation is repeatedly paying for the same lesson.

Why workflow context matters

The move toward continuous verification is happening alongside a shift from general-purpose chatbots to vertical workflow systems. Anthropic’s Claude for Financial Advisors launch connects Claude with investment analytics and wealth-management software for tasks such as meeting preparation, portfolio review and follow-up work. The reported integrations include Charles Schwab and Addepar; Charles Schwab said its authenticated connection includes administrative audit logs.

That example illustrates why assurance cannot be evaluated at the model layer alone. A model may produce a plausible answer, but a production workflow also depends on:

• whether the information came from an authorised system of record;
• whether the user had permission to access it;
• whether the agent created an accurate and reviewable artifact;
• whether a human approved an action when required; and
• whether the resulting activity can be reconstructed later.

The more useful unit of evaluation is therefore the complete workflow: model, data, connectors, permissions, policies, human review and operational recovery.

Microsoft’s provisional AI code of conduct points in a similar direction by emphasising explicit prohibited uses and human control. For product teams, principles only become operational when they compile into enforceable runtime controls: tool permissions, approval gates, blocked actions, escalation paths and audit records.

Reliability is part of assurance

The reported September 14 Work Mode incident, which affected task start or resume and access to workspace tools and files for some users before mitigation and recovery, is a reminder that agent reliability is not separate from AI governance.

A highly capable agent can still fail because a dependency is unavailable, state was not persisted or a retry repeated an action. If a workflow cannot be resumed safely, the team may not know what has already happened or what the agent will do next.

Production agent design should therefore include:

• durable state and checkpoints;
• idempotent tool actions where possible;
• bounded retries with clear failure states;
• timeouts and dependency health checks;
• compensation or rollback procedures for partial completion;
• human escalation for high-impact failures; and
• user-visible status showing what was completed, paused or rejected.

These are familiar distributed-systems practices. Their relevance increases when the system can interpret changing instructions and select tools dynamically.

A useful readiness test

Ask whether an independent engineer, auditor or incident responder could answer these questions from your system’s evidence:

• What did the agent intend to do?
• What information did it use?
• Which tools did it call, and with what authority?
• What approvals or policy checks occurred?
• What changed in the external system?
• Where did the run stop or fail?
• Can the task be resumed without repeating a side effect?
• Has this failure mode been added to the evaluation suite?

If the answer depends on the model’s explanation or a user’s memory, verification is incomplete.

The next step: run a production agent assurance review

Choose one agent workflow with real business consequences and map it end to end. Document its systems of record, data access, tool permissions, approval points, durable state, logs, failure modes and evaluation cases.

Then run a small review using three scenarios: a normal task, an ambiguous or unauthorised request, and an interrupted run caused by a failed dependency. Capture the evidence needed to reconstruct each outcome.

The result should be a prioritised backlog—not a compliance document. Strengthen the highest-risk authority boundary, add the most important missing evaluation, and make one failure mode safely resumable. That is the practical transition from a system card describing intended behaviour to continuous verification of the system you actually operate.