Daviyon DanielsAn AI inventory spreadsheet is a useful start. It rarely answers the next question: what can each system see, produce, retain, or do, and what evidence supports the answer?
Whether you are adding an AI feature to a product, deploying an agent internally, or reviewing a vendor, an assessment is more useful when its scope and evidence are clear. You do not need perfect documentation before beginning. You do need to distinguish facts you have verified from assumptions and areas you have not tested.
This checklist helps you assemble a reviewable record. It is a preparation guide, not a universal compliance requirement or a substitute for a scoped assessment.
1. Name the systems and their purposes
For each in-scope system, record the owner, users, purpose, deployment environment, models, providers, integrations, and current version. State whether it summarizes, recommends, generates content, changes records, invokes tools, or makes an external decision or action. Identify where a person reviews the output and what that review can actually prevent.
Evidence to collect: system inventory, architecture diagram, approved use cases, feature documentation, and change history. Separate production behavior from a proposed roadmap.
2. Draw the sensitive-data path
Trace inputs from source to prompt or API call, through retrieval, model processing, tool invocation, output, logs, storage, backups, support access, and deletion. Label data categories and trust boundaries. The relevant category may be patient information, financial records, legal client material, student information, customer data, source code, or credentials; the applicable requirements depend on the use case and jurisdiction.
Evidence to collect: data-flow diagram, data inventory, retention and deletion settings, contracts and provider terms, subprocessor list, and a redacted example transaction. Verify any claim about provider training or retention against the specific contract and configuration.
3. Show access and authority
Describe which people and service identities can use the system or view its data. For an agent, list its tools, credentials, permitted actions, approval gates, and ability to affect external systems. Document tenant or workspace boundaries and how they are enforced.
Evidence to collect: access matrix, identity and authorization design, representative configuration exports, approval records, tenant-isolation results, and audit events. A diagram of intended permissions is useful, but a reviewer may still need evidence of the permissions actually deployed.
4. Document dependencies and changes
List model and API providers, retrieval sources, plugins, vector stores, and other components that shape the workflow. Explain who can change models, prompts, tool permissions, data connections, and deployment configuration; how changes are reviewed; and what happens when a provider changes its terms or suffers an incident.
Evidence to collect: provider inventory, contract and security-review records, change approvals, release notes, rollback process, and incident notification terms.
5. Bring testing records with boundaries
Identify plausible failure and abuse paths for this system. Depending on the workflow, these may include indirect prompt injection, disclosure of sensitive information, cross-tenant retrieval, excessive agent authority, unsafe outputs, and misuse of connected tools. A risk list alone is not evidence that safeguards work.
Evidence to collect: threat model, test procedures, system version and date, results, findings, remediation and retest records. State what was not tested. OWASP's LLM application guidance can inform a threat review, but it does not replace system-specific evidence. OWASP Top 10 for LLM Applications
An ordinary Ayliea assessment does not automatically include penetration testing, source-code review, model-performance evaluation, red teaming, or continuous monitoring; those require separate written scope. Ayliea assessment services
6. Explain monitoring and response
Document what activity is logged, who reviews it, how unusual behavior is detected, and who can restrict access or disable the workflow. Identify an incident owner and a path for investigating provider incidents, data exposure, and agent actions. Keep the evidence sufficient for investigation while controlling exposure of sensitive content in logs.
Evidence to collect: monitoring coverage, alert ownership, incident playbooks, example redacted events, escalation path, and rollback or containment procedure. NIST's AI RMF organizes AI risk management around Govern, Map, Measure, and Manage; it can help structure this work across sectors. It is voluntary guidance, not a certification. NIST AI RMF
7. Mark evidence quality and limitations
Date each artifact and record the system, environment, and period it covers. Separate direct observation from documents supplied by a vendor or team, unverified stakeholder statements, and items not tested. List missing evidence, conflicting evidence, open risks, owners, and the next review date. Avoid applying an organization-level assurance report to a new AI feature without checking the report's boundaries.
The result is a concise evidence index: what the system does; where sensitive data goes; what people, agents, and vendors can access; what has been tested; how it is monitored; and what remains uncertain. That index gives internal decision makers and external reviewers a way to interrogate the actual deployment.
Ayliea can independently evaluate an agreed AI environment and its evidence, document findings and limitations, prioritize remediation, and deliver a point-in-time report with accountable human review. The procedures and industry mappings depend on the written scope. See the methodology, view a sample report, or request a scoping call. Do not send sensitive assessment materials through the initial booking form.