The s1ngularity attack: When attackers prompt your AI agents to do their bidding

In mid-September, a software supply chain incident - commonly dubbed the "s1ngularity attack" - marked a turning point in adversary use of Artificial Intelligence.

While the initial access vector was familiar - a phishing attack targeting npm package maintainers - the resulting malware was novel. This is one of the first attacks in which local AI agents were weaponized to aid in credential theft.

This technique represents a foreboding escalation in attacker ingenuity, moving beyond simple custom scripts to leverage the problem-solving capabilities of AI tools already available on a target's machine.

I sat down with Paul McCarty, Head of Research at Safety Cybersecurity - who recently published a detailed analysis of the attack - to discuss his findings, including the AI prompts used to search for secrets and bypass guardrails, and how this threat could evolve in the future. Watch the video above, or continue reading for an overview of what we talked about.

The anatomy of a post-install attack

During the attack, malicious updates were pushed to popular npm packages including nx, ultimately resulting in the theft of thousands of developer credentials. A key component of the malware was a post-install script the attacker named telemetry.js.

The main function of this script was to execute immediately upon installation and search the compromised host for sensitive files and secrets. Crucially, the attacker introduced a new discovery method, by checking for the presence of local CLI agents for popular Large Language Models (LLMs) - specifically Claude, Gemini, and AWS Q.

McCarty describes this as a clear evolution of the "living off the land" approach, where attackers abuse pre-installed, trusted tools. Developers often have these CLI agents installed to streamline coding workflows, making them a high-value, readily available target for an attacker.

Living off the local prompts

The malicious telemetry.js script didn't just passively look for files; it used AI agents to do the heavy lifting via a prompt.

• Initial check: The script would detect if a supported AI agent (CLI app) was installed and on which platform (the malware primarily targeted Linux and macOS hosts).

• The prompt: The malware then executed a specially-crafted prompt through the installed agent. The prompt's core objective was file enumeration: asking the AI to recursively search the host's filesystem and generate a comprehensive array of files that might contain secrets.

• Targeted secrets: This wasn't a blind search. The prompt focused on "juicy files", such as .env and .config files, that would commonly hold secrets, such as:

  • GitHub and npm tokens

  • Cloud platform credentials (e.g., AWS tokens)

  • SSH keys

The AI agent's role was to use the vast training data of the LLM and contextual awareness of the target's environment to determine the best locations to search. The AI agent effectively provided the attacker a discovery tool that could dynamically adapt to the target environment.

AI agents are unpredictable, but reliable enough at scale

McCarty's analysis of the attack payload revealed a fascinating glimpse into the attacker's thought process. Multiple successive updates to the poisoned packages reveal at least four different versions of the AI prompt.

This rapid iteration was an active battle against the non-determinism and guardrails inherent to AI applications. The attacker had to continuously experiment to find the perfect prompt that would:

1. Bypass guardrails: Avoid the agent's built-in safeguards that reject obvious malicious requests or requests to access file systems resources

2. Maximize results: Generate the most comprehensive list of sensitive file paths

3. Optimize speed: Narrow the focus to exclude common source code files, speeding up the overall data collection process

This iterative process demonstrates one of the shortcomings of AI tools: users often have to ask the same question multiple times, in different ways, to get closer to a successful outcome. It also highlights a growing risk for defenders: there are no guarantees that AI agents will adhere to guardrails. As McCarty's own testing showed, AI agents sometimes "ignored the guardrails and just went into YOLO mode."

This incident confirms that agentic AI represents a significant new attack surface. When wired up to developer workstations, they can be weaponized as a powerful discovery engine for the harvesting of credentials and other sensitive information.

Recommended reading

For an in-depth, technical breakdown of the four malicious AI prompt versions and their varying success rates against Claude and Gemini, we strongly recommend reading the full analysis by Paul McCarty and the Safety team: Analysing the AI used in the NX Attack.