摘要
Industrial Control Systems (ICS) serve as the backbone of critical infrastructure and face increasingly severe cybersecurity threats driven by the convergence of Information Technology (IT) and Operational Technology (OT). Current ICS penetration testing practices are constrained by a shortage of skilled professionals, insufficient attack vector coverage, and a lack of intelligent decision-making capabilities, making it difficult to systematically address the challenges posed by heterogeneous multi-vendor device environments.
This study proposes ICS-LLMPenTest, an automated penetration testing framework that integrates Large Language Model (LLM) agents with a finite state machine specifically designed for ICS attack chains—the ICS Penetration Testing State Machine (IPSM). The framework realizes a complete automated attack chain encompassing reconnaissance, protocol analysis, attack execution, and impact verification. IPSM defines four primary states (S1 Reconnaissance, S2 Protocol Identification, S3 Attack Execution, S4 Impact Verification) and constrains the LLM's decision space through state transition mechanisms, effectively preventing invalid attack attempts caused by hallucination while enabling vendor-adaptive attack module selection.
In particular, the dynamic protocol inference mechanism in the S2 stage constitutes the core innovation of this work: without any pre-written vendor scripts, the LLM agent autonomously reverse-engineers unknown or proprietary protocols and generates attacks, achieving template-free, cross-vendor adaptive penetration.
For LLM integration, a unified API direct-connect strategy is adopted to integrate six representative models: Sonnet 4.6, Opus 4.7, GPT-4o, GPT-5.4, Qwen 3.5, and Gemma 4. The attack tool library includes reverse-engineering reference scripts for several common PLC vendors, all implemented in the ICS simulation testbed of the Critical Infrastructure Information Security (CIIS) program at National Cheng ii Kung University.
Experiments evaluate six LLMs using four metrics: Task Completion Rate (TCR), Vendor Identification Accuracy (VIA), Average Attack Time (AAT), and Physical Impact Success Rate (PISR). Experimental results demonstrate that the ICS-LLMPenTest framework successfully guides LLM agents to autonomously complete multi-stage penetration testing against multi-vendor ICS devices, validating the effectiveness of IPSM in enhancing attack chain completeness and vendor identification accuracy.
This work provides a practically deployable technical solution for automated ICS security assessment and opens new research directions for LLM applications in OT cybersecurity.