Hello Hackers!

AI agents are all the rage these days, from articles to technical reports. But theory is one thing, and practice is quite another. So, I decided to conduct a small experiment, using a new CAI tool to test whether it could handle popular pentesting training tasks.

In this article, I will talk about my small test of the open-source CAI (Cybersecurity AI) (https://github.com/aliasrobotics/cai) project from Alias Robotics , announced on May 28, 2025 .

In short, CAI is an open-source platform positioned as an AI assistant for vulnerability detection and offensive (and defensive) security tasks. It features multi-agent orchestration, a comprehensive toolset, and support for local models.

The installation was almost hassle-free. I did have a bit of trouble working through a proxy (sanction restrictions are taking their toll) and setting up OpenAI tokens. I had to top up my account by $5 and install the gpt-3.5-turbo model, as it's significantly more efficient with token usage than the gpt-4o-mini.

To test my capabilities, I chose two scenarios: OWASP Juice Shop and several CTFs on TryHackMe.

At Juice Shop, I tested CAI on an application intentionally filled with vulnerabilities. The agent quickly scanned accessible pages, automatically found SQL Injection in the search parameters, and used it to bypass authentication, gaining access to the admin panel. It also pointed out XSS vulnerabilities in the feedback form and even offered working payloads, and I was able to extract user lists and passwords in cleartext from the database. The scenario turned out to be straightforward, and CAI proved itself a convenient assistant for web pentesting, taking over the routine search for SQL injection and XSS.

The next test was on TryHackMe, I tried to do three CTFs: Offensive Security Intro, Basic Pentesting and Mr Robot CTF.

In Offensive Security Intro, the goal was to access the web application's internal files and find hidden keys. CAI began with reconnaissance: scanning ports with nmap, discovering a web server on port 80 and the "Bank" application. Next, the agent noticed a parameter vulnerable to Local File Inclusion and, through injection, was able to read the contents of /etc/passwd and then traverse the /home/ and /var/www/html/ directories. Everything went smoothly, and CAI again performed completely autonomously, from LFI detection to extracting the necessary files.

The next CTF was Basic Pentesting, where the goal was to gain access to a system through weak credentials.

CAI started with basic reconnaissance: running ping and nmap -sV -A. The host was alive, with Apache running on port 80 (the HTML comment on the main page immediately hinted at this: “Check our dev note section…”), Samba on 139/445, and Tomcat 9.0.7 on port 8080, plus AJP on 8009.

Then I ran a set of commands in one script: checking typical paths, bruteforcing directories, basic SMB enumeration.
CAI detected /development/ and inside dev.txt,

I checked Tomcat /manager/html (responded with a 401 error) and also collected SMB information without smbclient. At this point, it highlighted two important clues: first, SMB is indeed configured and provides an attack surface; second, Struts 2.5.12 is mentioned, and Tomcat is running on port 8080.

Then everything followed the "classic" pattern of this room: SMB → users → password → SSH. First, I pulled everything from /development/, then hints appeared about the users jan and kay. j.txt directly indicated J's weak password. In such scenarios, jan's password is usually armando.

I tried connecting via SSH "ssh jan@10.10.222.131 (password: armando)". When I entered the password, CAI froze and stopped responding (even Ctrl+C didn't help), so I had to restart it. On the next launch, I went a more reliable route.

sshpass -p "armando" ssh -o StrictHostKeyChecking=no jan@10.10.222.131 "whoami && id && hostname"

Explain the code with

So CAI immediately executed the command on behalf of jan, without manual input, and as a result I got SSH access jan@10.10.222.131.

In this room, you need to get to the second user kay and get the final key.
I used an RSA key for this. I found that copying the key output to a file via nano inside CAI was a bad idea (it's interactive and often freezes). The solution turned out to be simple: do everything non-interactively, with a single command (a script). After that, the key was saved, I removed the passphrase (using ssh-keygen -p) and connected to kay.

The final key was extracted. Overall, the scenario is well automated; password brute-force, SSH access, and subsequent key usage are easily integrated into CAI.

The final challenge was Mr Robot CTF, where the goal was to find three keys.

During reconnaissance, CAI discovered a WordPress site and was able to log in using the account elliot/ER28-0652. The first key was extracted directly through the web interface, a task the agent completed without issue.

However, the second and third keys were located deeper in the file system (/home/elliot/ and /root/), and obtaining them required a reverse shell and subsequent privilege escalation to root.

Here, CAI proved powerless. After several attempts, it became clear that CAI couldn't automatically load PHP reverse shell, launch a listener, or execute root privileges. Such steps still require a live penetration tester or a combination of full-fledged exploitation frameworks.

This experience clearly demonstrated the limits of CAI's capabilities. Such minimal tests, of course, don't allow for definitive conclusions, but they do provide a general idea of what this product is and where it fits. There's no talk yet of it replacing tools like Burp Suite. Nevertheless, CAI left a very interesting impression; it's a useful "assistant" for reconnaissance and simple attacks, and I'll definitely continue testing it, but for me personally, it remains more of an experiment than a permanent tool.