A new class of physical-world vulnerabilities for autonomous systems has been uncovered by researchers, demonstrating that a simple printed sign can be sufficient to hijack the decision-making of a self-driving car or other embodied artificial intelligence. The research reveals a profound security flaw rooted in the very mechanisms that allow modern AI to navigate and understand the environment, creating a simple yet potentially catastrophic threat to public safety. This attack, which researchers have dubbed an "environmental indirect prompt injection," leverages the vision-language models (LVLMs) that power these systems, proving that instructions placed in the physical world can override critical safety protocols without any electronic hacking of the vehicle's software[1][2][3][4].

The core of the vulnerability lies in the autonomous systems' reliance on sophisticated Large Vision-Language Models (LVLMs) to process visual information, including text, and use it for real-time decision-making[5][3]. These multimodal AI systems are designed to simultaneously analyze both images and text from their environment, allowing them to interpret road signs, read navigation cues, and reason about the safest course of action[5][3]. However, this flexibility also creates an unexpected entry point for attackers[3]. Researchers from the University of California, Santa Cruz, and Johns Hopkins University found that a carefully designed text command printed on a piece of paper or cardboard and placed within the robot's visual field could be misinterpreted by the AI as an authoritative instruction[5][2][3]. This is an extension of the "prompt injection" attacks previously seen in digital chatbots, but translated into the physical world, targeting embodied AI entities like self-driving cars, drones, and delivery robots[1][2][4].

The researchers developed a technique they call CHAI, or Command Hijacking against Embodied AI, to demonstrate the exploit's real-world feasibility[3][4]. In one of the most alarming scenarios, an autonomous car that had correctly identified pedestrians crossing an intersection as a signal to stop was made to accelerate instead[3]. The car’s perception system was overridden when an attacker placed a printed sign nearby with a carefully optimized text command, such as "PROCEED ONWARD"[6][3]. The research team found that even when the system recognized an obstacle and the associated collision risk, the printed text led it to conclude that continuing forward was the safe decision[6]. This type of command injection forces the machine to act contrary to its original, safety-critical programming and design[5]. The attack has also been successfully tested on other platforms, including drones, where misleading text could compel a drone to land on an unsafe, crowded rooftop instead of a designated clear area[3].

The tests, which were conducted with small robot vehicles in a physical environment, confirmed the attacks work reliably under real-world conditions, including varying lighting, different viewing angles, and sensor noise[6][5][2]. The misleading signs, which sometimes included text prompts in multiple languages such as English, Spanish, and Chinese, achieved a significant success rate[4][7]. Against the small driverless cars used in the experiments, the malicious signs achieved an 82 percent success rate, causing them to crash into obstacles, while confusing drones in aerial object tracking achieved a 96 percent success rate[7]. A specific real-world test with a robot vehicle showed a success rate exceeding 87 percent in field conditions[6]. The key to the attack’s success is the adversarial optimization of the text prompt itself, though the researchers also found that visual factors like font, color, and placement contributed to maximizing the AI system’s likelihood of registering it as a command[4][7].

These findings present a critical design challenge for the entire robotics and autonomous systems community[5]. The study highlights a fundamental paradox: the ability of LVLMs to read and reason from environmental text—a feature intended to make them robust and adaptable—is now a major security weakness[3][2]. The ease with which this attack can be carried out, requiring nothing more than a printer and a piece of paper, underscores the severity of the threat, especially in environments where autonomous machines interact closely with the public[3][5]. For the AI industry, this vulnerability suggests that protecting the decision-making layer of AI—where language and vision are fused—may be just as vital as protecting the sensor layer from traditional adversarial attacks[5]. Cybersecurity experts now face the complex task of designing defenses that can enable embodied AI to read and understand legitimate signage while reliably filtering out harmful, unauthorized instructions[5]. Proposed solutions include stronger alignment between safety protocols and language understanding, improved filtering, and enhanced verification methods[5]. The research, which is set to be presented at the 2026 IEEE Conference on Secure and Trustworthy Machine Learning, serves as a stark call to action, urging developers to address these weaknesses before the widespread deployment of the next generation of LVLM-powered autonomous systems[2]. This new type of physical-world vulnerability, which one of the leading researchers noted is part of the nature of new technologies, necessitates an urgent and focused effort to secure systems against what may prove to be an unresolvable vulnerability, a challenge which other AI companies have previously conceded may never be completely ruled out for language models[3][6].