You can’t firewall a conversation: How AI red-teaming became mission-critical

Industry Trends | April 16, 2026

Jessica BrennanSenior Product Marketing Manager | F5

Allan HealySenior Solutions Engineer | F5

The explosion of AI usage since 2020 is unprecedented. In terms of adoption, AI is moving faster than cloud, faster than mobile, and certainly faster than the Internet did. Gartner predicts that more than 80% of enterprises will deploy AI this year.

When we classify an enterprise’s journey through AI adoption, we see maturity falling into four categories, which often run in parallel rather than in sequence:

Category 1 is general purpose AI and productivity—think employees using ChatGPT, Gemini, Copilot, etc.
Category 2 is when organizations have internal use cases, building custom chatbots for functions such as HR or IT.
Category 3 includes external use cases like building public-facing generative AI applications, including customer service chatbots.
Category 4 is agentic workflows, which are made up of complex systems that take actions autonomously on behalf of users.

It is in these last three categories that security becomes critical. That’s because organizations are building complex software on top of non-deterministic AI models, creating vulnerabilities that traditional firewalls simply cannot see.

“It’s clear that enterprises deploying AI need rapid automated testing against known vulnerabilities just to establish a baseline.”

Security is always a priority for business but, with AI, the concern is different—it’s a blind spot. Security leaders have spent 20 years deploying and configuring web application firewalls (WAFs) to protect their web applications, websites, and even APIs. But WAFs, while extremely useful, have limitations when it comes to AI. The natural language processing used in AI can be problematic for traditional WAF detection; put simply, you can't firewall a conversation. Enterprises need a new approach to protect their apps and APIs anywhere and secure their AI everywhere.

That’s why 75% of CISOs are reporting AI security incidents; their existing shields simply aren’t designed to catch these threats. It’s also why 91% have already detected attempted attacks on their AI infrastructure and why a whopping 94% are now prioritizing testing of their AI systems.

New categories of cognitive attacks

In the AI era, organizations aren’t just dealing with code vulnerabilities. They are facing entirely new categories of cognitive attacks, including:

Prompt injection, both direct and indirect
Data poisoning during the training phase
Sophisticated jailbreak techniques like symbolic language attacks
Token compression, where attackers hide malicious instructions in formats that the AI model(s) can read but humans can’t

Traditional security guardrails handle deterministic input, but prompt injection and other natural language attacks are semantic problems, not pattern-matching ones. These aren’t isolated bugs; they are systemic business risks introduced by AI-driven architectures.

There are plenty of real-world examples of how AI is changing the threat model. A breach at an international software provider last summer stemmed from a tenant-isolation logic flaw in the MCP server that allowed cross-organization data exposure. That’s a classic multi-tenant bug, but it’s more dangerous in LLM systems because leaked data appears as fluent language, which makes it much more difficult to detect.

Meanwhile, an incident at a global technology company reflected a different failure: broken trust boundaries. Prompt injection redefined a chatbot’s role, and the back-end systems trusted its tool requests without enforcing server-side authorization. The issue wasn’t the model ignoring rules but authorization being inappropriately delegated to it.

These are just two examples that map to a much broader emerging risk landscape. The industry is racing to categorize these AI vulnerabilities. There are frameworks emerging like the OWASP Top 10 for Large Language Model (LLM) Applications, the OWASP Top 10 for Agentic Applications , Mitre Atlas and the NIST AI Risk Management Framework,but we don’t have a definitive database or unified standard for what secure actually looks like.

The old approach can’t keep up

The pressure on industry right now to deploy AI is existential. Developers are using AI to write code 10 times faster than ever before; organizations are literally shipping new features—and even products—overnight.

At the same time, regulation is accelerating matters on the compliance side. The EU AI Act, for example, explicitly calls for adversarial testing for high-risk and general-purpose AI systems. In practice, that means that purpose-built red-teaming—testing AI systems with simulated adversarial attacks—must now be considered a core component of the AI security stack, and in a way that addresses the real-world challenges these systems face.

As CISOs and security teams attempt to secure changes happening at machine speed, it can feel like trying to stop a tsunami with a bucket. The math doesn’t work. The speed doesn’t work. The AI attack surface is fundamentally different, and the old approaches can’t keep up.

It’s clear that traditional red-teaming is ineffective and AI red-teaming is needed to resolve the tension point of speed versus control. From speaking to customers, helping them to secure their AI systems, there are four key areas we need to consider:

Firstly, threat evolution: AI attacks evolve faster than static test suites. As soon as checks are automated, the AI model or the attack changes, and security teams end up maintaining tests instead of reducing risk.

Secondly, agent complexity: because AI agents aren’t deterministic systems, once you add retrieval, tools, and memory, there are almost infinite permutations. You are no longer testing code; you’re testing a conversation that changes based on context.

Thirdly, automation and scale: manual red-teaming does not scale for these systems. One chatbot may be manageable. Hundreds or thousands of chatbots are not. You can’t rely on humans to replay thousands of adversarial conversations every time the model or the system prompt is updated.

Finally, actionable reporting: findings must be reproduceable and actionable. “The bot behaved badly” is not actionable. Engineers need the conversation parameters and trigger conditions; otherwise, the fixes and the remediation will stall.

Ensuring AI behaves as intended, even under attack

Let’s ground-truth this with some examples. One of our customers is a global bank, operating in a highly regulated environment. When we first engaged with this customer, it had over 50 AI use cases across HR, procurement, and cybersecurity, but couldn’t ship any of them because the bank couldn’t prove safety to its internal auditors.

AI red-teaming with F5 AI Red Team gave the bank the evidence it needed to understand how its AI systems actually behaved—where data could leak, how prompts could be abused, and where controls broke down in its environment.

This customer is taking the findings from AI Red Team to improve its defensive posture with custom security controls using F5 AI Guardrails. This combination allows the bank to scale AI across the business with confidence in its security posture and governance program.

In the public sector, the imperative shifts from voluntary testing to mandatory. Guided by agencies including NIST and CISA, public sector agencies are taking action, such as adversarial stress tests to identify mission-critical risks like the weaponization of biological data. Here, AI red-teaming isn’t just about reducing risk; it’s about maintaining mission continuity and authority to operate.

Whether you’re protecting customer data or public services, the requirement is the same—continuous, evidence-backed assurance that AI systems behave as intended, even when someone is trying to break them.

Closing the gaps on compliance

It’s clear that enterprises deploying AI need rapid automated testing against known vulnerabilities just to establish a baseline. Context is the new attack surface; static defenses fail against agentic attacks so organizations must test workloads, not just models.

These are the real-world gaps that security teams are trying to close right now, and the reasons why AI red-teaming is coming to the forefront. In this environment, compliance is a competitive advantage. With the right reporting, security stops being a blocker and becomes the enabler that gets an enterprise’s AI to market faster.

In that world, the large numbers of enterprises that plan to deploy AI this year can do so with confidence rather than fear, whatever stage of their journey they’re on.

Learn more about how to test your AI systems here.

Watch F5 experts show how AI Red Team accelerates continuous adversarial testing here.