Your Compliance Evidence Is Getting Better. Your Controls Aren't.

‍

AI can now collect compliance evidence, map it to controls, identify gaps, draft policies, and write the narrative explaining how a company meets a requirement.

‍AI didn't create the problem I want to talk about. It just makes the evidence cleaner, faster, and easier to produce.

I've spent a large part of my career assessing security programs, working through compliance requirements, and helping companies fix what happens after the assessment is over. Long before AI entered the conversation, I saw the same issue over and over again. A company could show evidence that a control existed without necessarily proving that the control worked the way everyone thought it did.

I don't think better evidence is a bad thing. I think we're going to see a lot more of it. The problem is what happens when we start confusing better evidence with better security.

And that changes what we should be asking.

What did compliance automation actually change?

Tools like Vanta and Drata have improved a part of compliance that needed to improve.

Anyone who has been through enough audits remembers the old process. Someone creates an evidence request list. Teams start taking screenshots. People chase control owners. Evidence gets dropped into folders. Spreadsheets track what is missing. Then everyone does it again during the next audit cycle.

Compliance automation changed a lot of that. Platforms can integrate directly with cloud environments, identity providers, endpoint tools, code repositories, HR systems, and other parts of the technology stack. Instead of proving once a year that a setting was enabled, organizations can continuously collect information about their environment and identify when expected configurations are missing.

That's progress.

Now AI is taking the next step. Instead of only collecting evidence, AI can help interpret it, map it against requirements, summarize gaps, draft policies and control descriptions, and prepare information for an assessment.

We should automate work that doesn't require someone spending hours doing it manually. But there is an important distinction we can't lose along the way.

‍

Continuous evidence collection is not the same thing as continuous control validation.

‍

A system can tell you what is configured. That doesn't always tell you whether the control is effective.

‍

Can you trust AI-generated compliance evidence?

Yes, but you need to understand what you're trusting it to tell you.

Take MFA. A compliance platform can connect to Microsoft 365 or another identity provider and determine that MFA is configured. It can collect that information, map it to a control, and provide evidence showing the control is in place. AI can then take that evidence and write a clear description of how the organization protects access.

Everything can be accurate.

Then I come in and start asking different questions.

• Who is excluded from MFA?
• Are there service accounts that aren't covered?
• Are there break-glass accounts? How are privileged accounts handled?
• Are there old Conditional Access exceptions from a migration? Are there authentication methods that provide another path into the environment?

Suddenly, "MFA is enabled" doesn't tell us everything we need to know.

The evidence wasn't wrong. The conclusion we drew from it was incomplete. That's an important difference.

Automation is very good at answering, "What does the system say is configured?" Security still has to answer, "Does this control actually work the way we think it does?"

What does a passing control actually prove?

This problem goes well beyond MFA.

Backups are one of my favorite examples because almost everyone understands them. A company can produce months of reports showing successful backup jobs. An automated platform can collect those reports. AI can review them and summarize backup success rates. An auditor can verify that the organization has a backup process.

But when was the last time someone restored something?

I have seen organizations with backups that looked fine until you started asking about recovery. Nobody had performed a meaningful restore. Recovery time hadn't been validated. Dependencies weren't understood. In some cases, the organization didn't know whether the backup would actually support the recovery they expected.

‍

A successful backup job and a successful recovery are two different things.

‍

Logging works the same way. A company might have a SIEM. Logs are being collected. Screenshots and reports prove the platform is operating.

But which systems are actually sending logs? Did a critical source stop reporting three months ago? Would anyone notice? Are the right alerts configured? Who receives them? What happens when a high-severity alert fires at two in the morning?

The existence of the technology isn't the control. The outcome is the control.

Endpoint security is another example. A dashboard might show EDR deployed to 98 percent of endpoints. That sounds good until you ask what is in the missing two percent. Then you need to ask another question: how do we know the asset inventory we're comparing it against is complete?

‍

You can have 100 percent coverage of 90 percent of your actual environment.

‍

That looks great on a dashboard. It isn't great security.

‍

Why does AI make this more important?

Because we're starting to automate more than evidence collection.

If AI reads configuration data, maps it against a framework, identifies whether a requirement appears to be satisfied, and then drafts the control narrative, AI is participating in the interpretation of the evidence.

NIST is already exploring this direction. Its draft SP 1353, published August 19, 2026, with public comment open through October 15, includes use cases where AI helps analyze artifacts, interview notes, and other information to develop CSF profiles and reporting.

At the same time, the criteria are moving to meet it. The AICPA has begun folding AI governance into the Trust Services Criteria, which means auditors are starting to ask what your organization uses AI for and what proves those controls operate. So AI is increasingly producing the evidence at the same moment AI use is becoming the thing being examined.

That's probably where the industry is going. The question is where human judgment remains necessary.

If AI collects the evidence, interprets the evidence, and writes the explanation of why the evidence satisfies the control, somebody still needs to validate the underlying assumption. Otherwise, we risk building a very efficient system for proving that controls exist without spending enough time determining whether they work.

The question isn't really, "Can I trust AI-generated compliance evidence?"

A better question is: who tested the control behind it?

What happens after the audit?

Security environments don't stay still after an audit.

People leave. Administrators create exceptions. Companies acquire businesses. Cloud environments change. Applications migrate. New SaaS platforms appear. Service accounts get created. Security tools stop reporting. Someone temporarily disables a control to fix a production problem and forgets to turn it back on.

Identity is the common one. MFA might be configured correctly during an assessment, but exceptions accumulate. Privileged access expands. Users change roles and keep permissions. Quarterly access reviews continue to happen, but nobody really challenges whether the access still makes sense.

Then there is incident response and recovery. The organization has a plan. The auditor saw it. There is a backup policy and the jobs are running.

But has anyone run the plan?

Do the people named in it still have those roles? Who calls cyber insurance? Who contacts outside counsel? Who has the authority to shut down production? Has anyone tested whether the company can restore its most important systems within the time the business expects?

A document doesn't answer those questions. Testing does.

Haven't we seen this before?

AI makes this feel new, but the underlying shift isn't.

I spent roughly 15 years working around payment security and watched PCI change alongside technologies like tokenization and point-to-point encryption. I also watched payment technology move into the cloud and more responsibility shift between merchants and their providers.

Those technologies were good things. They reduced exposure and made certain controls easier for merchants to operate. But they didn't eliminate responsibility. They changed where responsibility lived.

If a provider tokenized cardholder data, the merchant still needed to understand what remained in scope. If a payment environment moved to the cloud, someone still needed to understand the shared responsibility model. If a provider handled part of PCI compliance, the merchant still needed to know where the provider's responsibility ended and theirs began.

AI is creating a similar shift in compliance. As more evidence collection and analysis moves to platforms and AI, the job of the security team and the assessor should move too. We should spend less time collecting evidence and more time challenging it.

What should companies actually do?

Test the control, not just the evidence.

• If MFA is a critical control, don't stop when the dashboard turns green. Look for ways around it.
• If backups are important, don't just review the successful backup report. Restore something.
• If EDR is supposed to cover the environment, reconcile it against the asset inventory and generate a test detection.
• If your SIEM is supposed to identify suspicious activity, verify that critical systems are reporting and test whether an alert reaches the right person.
• If privileged access is reviewed quarterly, select accounts and validate whether the access actually makes sense.
• If you have an incident response plan, run a tabletop.

None of this means abandoning automation. I think we should do the opposite. Use Vanta, Drata, compliance automation, AI, and whatever comes next to eliminate as much repetitive compliance work as possible. Stop spending expensive security resources taking screenshots and moving evidence between folders when technology can do it better.

Then take the time you get back and put it toward the part that matters: validating whether the security program works.

The goal shouldn't be to make compliance harder again because technology made it easier. The goal should be to use that efficiency to make security better.

AI can collect the evidence. It can organize it. It can map it. It can even write a pretty good explanation of what it means. But someone still needs to ask the question that matters:

Does the control actually work?

‍

If you're not sure where your program sits between "the evidence passes" and "the controls have been tested," that's the gap worth measuring. Our security maturity self-assessment walks through it in a few minutes.

‍