What’s the first thing you think of when you hear about ai security threats and vulnerabilities?
If you’re like most people, your mind probably jumps to Large Language Model (LLM) vulnerabilities—system prompt disclosures, jailbreaks, or prompt injections that trick ai chatbots into revealing sensitive information or behaving in unintended ways. These ai security risks have dominated headlines and security discussions, and for good reason: they’re novel, they’re interesting, and they affect some of the most visible ai applications today. The development of ai has brought new forms of ai that many ai systems must address.
But here’s what keeps us up at night: while security teams worry about chatbots leaking secrets, attackers could be dropping payloads targeting the production infrastructure itself. The complexity of ai systems makes it challenging to secure their ai implementations effectively.
The attack surface for artificial intelligence systems extends far beyond conversational interfaces. The platforms that host, train, and deploy machine learning models—the infrastructure that powers everything from fraud detection to recommendation engines—present an entirely different class of security threats. These are the platforms that enable businesses to build, test, and deploy ai models at scale, often with self-service capabilities that prioritize ease of use over security isolation. The ai training process and ai model training phases can introduce potential risks that organizations must address.
During a recent red team engagement, our security team encountered one of these platforms in the wild, and what we discovered illustrates why the security community needs to expand its thinking about ai security risks. The emerging threats in this space require organizations to examine how ai systems are deployed and accessed.
Key Takeaways
AI security risks in ML platform security extend far beyond prompt injection — the infrastructure itself is the attack surface.
Using only a self-registered trial account, we deployed a malicious ai model that achieved remote code execution on the provider’s cloud infrastructure.
Weak network segmentation allowed our C2 beacon to reach internal services, databases, and resources that should have been completely inaccessible to external users.
The core problem: ML platforms must execute arbitrary code to function, making them inherently harder to sandbox than traditional web applications. This creates various security threats that organizations must understand when implementing ai use cases.
The Setup: How Modern AI-Powered MLOps Platforms Create Attack Surface
Modern MLOps providers are looking to solve the problem of how they can streamline the adoption of ai across business units. Some platforms are opting to provide developers with managed infrastructure to train ai models and deploy ai applications quickly. The appeal is obvious—developers can go from idea to production in hours rather than weeks. The role of ai in accelerating development cycles has made this approach increasingly popular.
From a business perspective, this is exactly what you want: low friction, rapid onboarding, and quick time-to-value. Users can self-register, receive their own isolated environment in a cloud provider, and begin experimenting immediately with ai tools and generative ai models. Once an ai system is built and trained, the platform handles deployment, exposing it through an authenticated API endpoint that can be called from anywhere.
It’s elegant. It’s modern. And as we discovered, it’s also a potential gateway into your internal network. The responsible use of ai requires understanding these risks and implementing appropriate security measures.
What is an MLOps Platform? MLOps (Machine Learning Operations) platforms provide managed infrastructure for building, training, and deploying ai models at scale. They typically offer self-service environments, automated provisioning, and API-based model serving — designed to reduce friction for developers but often creating security blind spots in the process. These platforms represent a significant evolution in how organizations use ai and machine learning technologies.
The Attack: Turning AI Infrastructure Against Itself — Remote Code Execution via Model Deployment
Our objective was simple: What could an adversary do with nothing more than a self-registered trial account?
The answer turned out to be: Quite a lot.
The attack chain we developed exploited a fundamental characteristic of ai platforms—they need to execute code. Machine learning models aren’t just static files; they’re executable artifacts that process inputs and generate outputs. In this case, we leveraged that execution capability to deploy what appeared to be a legitimate machine learning model but was actually a malicious payload designed to evade detection.
Here’s how it worked:
The platform allowed us to create and deploy an ai model that accepted custom parameters as part of its API requests. We designed our learning model to accept a specific parameter: a URL pointing to a malicious code snippet. When the ai system received an API request containing this URL, it would retrieve the content from that snippet and execute the code within the container where the model was deployed. This poisoning attack technique allowed us to compromise the ai system’s integrity.
From the platform’s perspective, this appeared to be normal model behavior—processing an input and generating an output. But in reality, we had just created a remote code execution capability in the provider’s infrastructure, all through a self-registered trial account. The ai algorithms were manipulated to make ai systems execute our malicious code.
We used this capability to deploy command-and-control (C2) infrastructure directly onto the platform’s underlying cloud environment. A “random person” on the internet—with no legitimate business relationship, no pre-existing access, and no stolen credentials—now had a foothold in the provider’s infrastructure. This demonstrates how attackers can exploit ai systems provided by third-party vendors.
The Real Problem: Cybersecurity Network Access and Trust Boundary Failures
Deploying malicious code is concerning, but what made this situation critical was what came next: network access. The risks of attack escalate significantly when network isolation fails.
The containers hosting deployed ai models weren’t entirely isolated from the provider’s internal resources. Our C2 beacon could reach internal services, databases, and infrastructure that should have been completely inaccessible to external users. The trust boundary that should have existed between “customer/client infrastructure” and “internal corporate resources” was either poorly implemented or non-existent. This represents one of many security incidents that could have been prevented with proper security measures in place.
This is where a security issue becomes a business-critical risk. An attacker who gains this level of access could:
Exfiltrate sensitive data from internal databases, APIs, and services that trusted the ML platform’s network space
Establish command-and-control infrastructure that survives account deletion (by deploying additional backdoors to the underlying cloud infrastructure before the account is terminated)
Use the compromised container as a trusted insider to pivot deeper into the network and discover additional high-value targets. This type of attack demonstrates how ai is changing the threat landscape and creating new risks for organizations.
All of this from a free trial account that took minutes to create. The ai has transformed how quickly attackers can gain access to ai systems and infrastructure.
Why This Matters: The Convergence of AI Security and Cloud Security
This type of vulnerability sits at the intersection of multiple security domains: application security, cloud infrastructure security, ML operations (MLOps), and red teaming. It’s a blind spot that emerges when organizations focus heavily on one type of security threat (like LLM prompt injection) while underestimating the security implications of the platforms themselves. The applications of ai continue to expand, creating new attack surfaces that security teams must address.
Several factors make these cybersecurity threats particularly challenging:
First, the self-service model prioritizes user experience over security isolation. Organizations want to reduce barriers to adoption, which often means automating environment provisioning and minimizing security friction. This is a business requirement, but it creates opportunities for abuse if trust boundaries aren’t carefully designed. The eu ai act and other regulations are beginning to address some of these concerns around responsible ai deployment.
Second, ai applications and ML workloads require significant compute and execution capabilities. Unlike traditional web applications that might serve static content or make database queries, ai platforms must execute arbitrary code as part of their core functionality. This makes them inherently more difficult to sandbox and isolate. Organizations must build an ai security system that can handle these unique requirements.
Third, the rapid evolution of ai tools means security practices often lag behind deployment. Organizations are under pressure to deliver ai capabilities quickly, and security reviews may not fully account for the unique risks these platforms present. Many ai applications are deployed without adequate security and privacy controls in place.
The training of ai systems introduces additional vulnerabilities. Data fed into an ai system during the training process can be manipulated by attackers. When malicious data is fed into the ai system, it can compromise the entire machine learning model. Organizations need to implement security measures in place to protect against these threats and secure ai systems effectively.
Secure AI: What Organizations Should Do
If your organization operates a customer-facing ai platform, especially those with self-service capabilities, consider these security principles:
Assume all users are adversaries. Design your isolation boundaries with the assumption that some percentage of accounts will attempt malicious actions. This isn’t about being paranoid—it’s about realistic threat modeling. Organizations must mitigate ai security risks by implementing appropriate controls.
Implement strict network segmentation. Customer environments should have no connectivity to internal resources by default. Any necessary communication should flow through well-defined, heavily monitored interfaces with explicit allowlisting of permitted connections. This helps prevent attacks on ai systems from spreading to other parts of the infrastructure.
Monitor model behavior and resource access. Unusual API patterns, unexpected network connections, or abnormal resource consumption from deployed models should trigger alerts and investigation. Advanced detection systems can help identify when attackers attempt to use ai systems maliciously.
Limit execution capabilities within model containers. Consider what level of code execution is actually necessary for legitimate model operation, and restrict everything beyond that scope. Sandboxing and containerization are starting points, not complete solutions for securing ai systems.
Implement robust authentication and authorization for all model APIs. Even though these are user-deployed models, access should be tightly controlled and auditable. Organizations need to control access to ai systems and ensure only authorized users can interact with sensitive ai applications.
Conduct regular red team assessments specifically focused on your ai infrastructure. Traditional penetration testing may not uncover the unique attack paths available through ai platforms. Security teams should examine how ai systems can be exploited and implement appropriate countermeasures.
Establish human oversight for ai deployments. The use of ai should include human review processes to detect anomalous behavior and ensure that generative ai applications operate within expected parameters.
Generative AI Security: The Broader Lesson
The security community’s focus on LLM vulnerabilities is understandable and important. Prompt injection, model poisoning, and adversarial attacks on ai systems are real risks that deserve attention and resources. However, generative ai has become increasingly sophisticated, creating new types of ai-powered threats that organizations must address.
But as this engagement demonstrated, the attack surface for ai extends far beyond the models themselves. The platforms that enable ML development and deployment create their own set of risks—risks that can provide direct access to an organization’s most sensitive resources. The output of the model is just one concern; the security posture of the entire ai system must be evaluated.
As businesses continue to embrace ai and machine learning capabilities, they need to think holistically about security. That means not just protecting against prompt injection and model manipulation, but also securing the infrastructure that makes modern ai possible. Organizations must see how ai is changing their attack surface and adapt their security strategy accordingly.
Attackers already know this. They’re not debating prompt engineering techniques—they’re registering user accounts and looking for what those accounts can reach. The question isn’t just whether your ai models are vulnerable. It’s whether your ai platform is accidentally functioning as a VPN into your internal network. Organizations need to examine how ai systems connect to their broader infrastructure and ensure appropriate isolation.
The type of ai deployed, whether it’s generative ai models or traditional machine learning systems, doesn’t matter as much as ensuring that the underlying infrastructure is secure. Organizations should focus on securing ai systems comprehensively, not just addressing individual ai techniques or specific risks of using particular ai tools.
AI in Cybersecurity: MITRE ATT&CK TTP Mapping
Understanding how these ai security threats map to established frameworks helps organizations better detect and respond to threats. The integration of data and ai in modern attack chains requires updated threat detection capabilities.
Initial Access can occur through self-service registration on ai platforms, where attackers gain legitimate access to ai systems provided by vendors. Execution happens when malicious code is embedded within ai models and executed during model inference. Persistence can be achieved by deploying multiple backdoors across different ai applications within the same environment.
The ai is expected to process user inputs, making it difficult to distinguish between legitimate ai use and malicious exploitation. This creates unique challenges for threat detection systems that must understand the difference between normal ai behavior and attacks.
Frequently Asked Questions
What are the main security risks associated with AI?
The primary AI security risks include adversarial attacks that manipulate AI algorithms to produce incorrect outputs, data poisoning during AI training phases, and model theft where attackers steal proprietary machine learning models. AI systems are also vulnerable to prompt injection attacks in generative AI applications, where malicious inputs can cause the AI to behave unexpectedly or reveal sensitive information. Additionally, AI-powered systems can amplify existing cybersecurity vulnerabilities and create new attack vectors that traditional security measures may fail to detect.
How can AI systems be exploited in cyberattacks?
Cybercriminals can exploit AI systems through adversarial examples that fool machine learning models into misclassifying threats, allowing malicious content to evade detection. Attackers may also target the AI training data to poison learning models, causing them to make incorrect decisions in production environments. Furthermore, AI applications can be compromised through injection attacks that manipulate generative AI models to produce harmful outputs or leak confidential information.
How can AI systems be used to enhance cyberattacks?
Malicious actors leverage AI tools to automate and scale cyberattacks, using machine learning algorithms to identify vulnerabilities faster than human analysts. AI-powered attacks can generate sophisticated phishing emails, create deepfake content for social engineering, and develop polymorphic malware that constantly evolves to evade detection. Generative AI models enable attackers to produce convincing fake identities and documents, making it increasingly difficult for traditional cyber security measures to distinguish between legitimate and malicious activities.
How can organizations mitigate AI security risks?
Organizations should implement robust AI security frameworks that include regular testing for adversarial attacks, secure AI training pipelines with data validation, and continuous monitoring of AI model behavior in production. Essential mitigation strategies involve maintaining human oversight over critical AI applications, implementing input validation to prevent injection attacks, and establishing incident response procedures specifically for AI security threats. Additionally, companies must ensure proper access controls for AI systems and regularly update their AI tools to address emerging threats and vulnerabilities.
What is AI Security?
AI security encompasses the protection of artificial intelligence systems, data, and infrastructure from cyber threats while ensuring AI applications operate safely and reliably. It involves securing AI algorithms against adversarial attacks, protecting machine learning models from data poisoning, and preventing unauthorized access to AI training data. AI security also includes implementing safeguards in generative AI security to prevent misuse and ensuring that AI use in cybersecurity doesn’t introduce new vulnerabilities to organizational systems.
