Enhanced Interactive Generative Model Agent for CTF Challenges

Talor Abramovich¹, Meet Udeshi², Minghao Shao², Kilian Lieret³, Haoran Xi², Kimberly Milner², Sofija Jancheska², John Yang⁴, Carlos E. Jimenez³, Farshad Khorrami², Prashanth Krishnamurthy², Brendan Dolan-Gavitt², Muhammad Shafique⁵, Karthik Narasimhan³, Ramesh Karri², and Ofir Press³

Although language model (LM) agents are demonstrating growing potential in many domains, their success in cybersecurity has been limited due to simplistic design and the lack of fundamental features for this domain. We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges. EnIGMA introduces new Agent-Computer Interfaces (ACIs) to improve the success rate on CTF challenges. We establish the novel Interactive Agent Tools concept, which enables LM agents to run interactive command-line utilities essential for these challenges. Empirical analysis of EnIGMA on over 350 CTF challenges from three different benchmarks indicates that providing a robust set of new tools with demonstration of their usage helps the LM solve complex problems and achieves state-of-the-art results on the NYU CTF and Intercode-CTF benchmarks, managing to solve more than three times more challenges of NYU CTF benchmark compared to previous best agent (the NYU CTF agent).

Want to try it yourself and explore our new agent? We are completely open-source! You can try it out in the SWE-agent repository ! GitHub Repo stars, read our documentation and explore more about the research work in our paper. Please use SWE-agent 0.7 while we update EnIGMA for 1.0.

Results

Benchmark	Model	% Solved	Date	Trajectories
NYU CTF	EnIGMA w/ Claude 3.5 Sonnet	13.5	2024-09-24
	EnIGMA w/ GPT-4 Turbo (1106)	7.0	2024-09-24
	EnIGMA w/ GPT-4o	9.0	2024-09-24
	NYU CTF agent w/ GPT-4 Turbo	4.0	2024-08-21
InterCode-CTF	EnIGMA w/ Claude 3.5 Sonnet	67.0	2024-09-24
	EnIGMA w/ GPT-4 Turbo (1106)	72.0	2024-09-24
	EnIGMA w/ GPT-4o	69.0	2024-09-24
	InterCode-CTF Agent	40.0	2023-11-14	N/A
	Google DeepMind Agent w/ Gemini 1.5 Pro	43.0	2024-08-08	N/A
CyBench	EnIGMA w/ Claude 3.5 Sonnet	20.0	2024-12-05
	EnIGMA w/ GPT-4 Turbo (1106)	17.5	2024-12-05
	EnIGMA w/ GPT-4o	12.5	2024-12-05
	EnIGMA w/ Llama 3.1 405B Instruct	10.0	2024-12-05
	CyBench agent w/ Claude 3.5 Sonnet	17.5	2024-08-15
	CyBench agent w/ Llama 3.1 405B Instruct	7.5	2024-08-15
HackTheBox	EnIGMA w/ Claude 3.5 Sonnet	26.0	2024-09-24
	EnIGMA w/ GPT-4 Turbo (1106)	18.0	2024-09-24
	EnIGMA w/ GPT-4o	16.0	2024-09-24
	NYU CTF agent w/ GPT-4 Turbo	20.0	2024-08-21

How it Works

Interactive Agent Tools In Action

BibTeX

If you found this work helpful, please consider using the following citation:

@misc{abramovich2024enigmaenhancedinteractivegenerative,
      title={EnIGMA: Enhanced Interactive Generative Model Agent for CTF Challenges},
      author={Talor Abramovich and Meet Udeshi and Minghao Shao and Kilian Lieret and Haoran Xi and Kimberly Milner and Sofija Jancheska and John Yang and Carlos E. Jimenez and Farshad Khorrami and Prashanth Krishnamurthy and Brendan Dolan-Gavitt and Muhammad Shafique and Karthik Narasimhan and Ramesh Karri and Ofir Press},
      year={2024},
      eprint={2409.16165},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2409.16165},
}

1. Tel-Aviv University ↩︎

2. New York University ↩︎ ↩︎² ↩︎³ ↩︎⁴ ↩︎⁵ ↩︎⁶ ↩︎⁷ ↩︎⁸ ↩︎⁹

3. Princeton Language and Intelligence, Princeton University ↩︎ ↩︎² ↩︎³ ↩︎⁴

4. Stanford University ↩︎

5. New York University Abu Dhabi ↩︎