City University of Hong Kong × Information Security Research Center Engineering a Resilient Ecosystem for AI Security

As AI and electric vehicle (EV) technologies advance at lightning speed, cybersecurity threats have spread from the virtual into the physical world. To confront increasingly complex attack methods, the theoretical depth of academia must be tightly integrated with the real-world practice of industry.

Against this backdrop, the Information Security Research Center and the Department of Computer Science at City University of Hong Kong are engaged in close collaboration. As a world-class technology R&D institution, Hon Hai offers its internally developed large language models, such as FoxBrain, as a practical platform, enabling academic research to directly enter real-world industrial applications.

To address the pressing issue of Generative AI security, the joint research team has put forward a Red and Blue Teaming framework. The Red Team uses the intrinsically motivated red teaming methods developed by CityU to identify a target model’s vulnerabilities. This approach employs deep reinforcement learning to give the system self-evaluation capabilities, allowing it to explore previously unknown attack spaces and significantly expanding attack surface coverage. On the defensive side, the joint research team developed a suffix-based defense module for multimodal (cross-modal) attacks, pioneering the use of reinforcement learning to train defensive models. The result is a plug-and-play effect without altering the original model. In addition, the team has further optimized detection latency to extremely low levels, ensuring that FoxBrain can effectively resist jailbreak attacks while achieving a 28% improvement in security against specific attack methods.

Deepening Security Resilience

The research team also identified new types of threats in the physical world. For instance, in online HD map construction models, researchers demonstrated that attackers need only use a flashlight or small physical patches to create disturbances. The disturbances trigger amplification effects in deep learning networks, causing the model to misinterpret traffic lights or stop signs. This study indicates that future sensor-fusion designs for intelligent electric vehicles (EVs) cannot rely solely on visual models; they must proactively address physical-world attack vectors.

In testing standards, the joint team identified significant flaws in collision detection within existing autonomous driving simulators. By developing debugging techniques targeted at the simulators themselves, the team exposed corner cases where collisions went unreported. These findings were shared with suppliers, enabling virtual testing to more accurately reflect the complexities of real-world physical environments.

Beyond technical achievements, the collaboration has also yielded vital digital assets, such as the high topic-diversity red team dataset. For academia, the dataset provides a framework for fair competition; for industry, it can be used to align internal models (such as FoxBrain) for security, making them more resilient against adversarial or malicious inputs.

Looking ahead, cybersecurity will be defined by AI versus AI engagements. The joint team has already begun deploying post-quantum cryptography to develop long-term defense mechanisms for multi-robot collaboration systems, laying a secure foundation for future smart factories.

As we transition to the next phase of agent security, Dr. Xiang Zheng, postdoctoral fellow at CityU, notes that AI tools will significantly lower the threshold for attackers. Consequently, the collaboration has expanded into agent privacy protection, aiming to prevent the unlawful extraction of proprietary prompts or corporate memory from core systems.

Professor Cong Wang of the CityU Department of Computer Science emphasized that academic research risks remaining purely theoretical if it does not engage with industry practice. The Information Security Research Center provides the real-world application scenarios, performance benchmarks, and rigorous testing requirements necessary for research teams to innovate while managing cost and risk. This collaboration has not only yielded security breakthroughs but has also fostered a responsible, evolutionary AI security ecosystem.