Comet Gains Fame for Outwitting AI’s Hacking Attempt

"I just thought they were inexperienced hackers," he said.

Demir had been working on an inventory-management system that day to build his portfolio and gain experience. He didn't think much more about the incident.

But a week later, on Aug. 4, he received an email from the AI Security Institute (AISI), a U.K. government research agency. AISI had been testing an AI agent using cybersecurity challenges and had given the agent internet access during the test.

The institute thanked Demir for flagging the code and delivered a surprise: The users who had argued with him were not humans.

The AI agent not only tried to submit the malicious code on GitHub, it also created fake identities to challenge Demir's warning. Neither of those actions were sanctioned by AISI, according to a report issued by the agency.

GitHub is a web-based platform where users host, share and collaborate on software projects. Many projects on GitHub are open source, meaning their source code is publicly available for others to inspect, use, modify and share.

AISI said in the report: "A human maintainer caught and refused to approve the malicious code."

Demir said, "I read the technical report and realized the person in the report was me."

He told his mother, who was sitting next to him at their home in Austin, Texas, when he learned about the incident.

"She thinks that AI will control our lives. She said, 'You know, I told you this many times, and you didn't believe me,'" Demir said. "I said 'You're probably right.'"

The incident illustrates how AI could make social engineering attacks more sophisticated, dangerous and difficult to detect, said Dr. Kevin Hamlen, the Louis Beecherl Jr. Distinguished Professor of computer science and executive director of the UT Dallas Cyber Security Research and Education Institute.

Social engineering attacks involve psychological deception to trick people into providing sensitive information or breaking security rules.

"Criminals can now mass-produce thousands of highly convincing AI-generated emails and posts to convince open-source software maintainers to accept malicious code changes to their projects," Hamlen said. "Open-source components get included within most modern software projects, so this allows criminals to infect the digital world if they succeed."

After the AISI released its report, the Erik Jonsson School of Engineering and Computer Science student was fielding interview requests from national and international news outlets. Demir was featured in an Aug. 20 Reuters article about the incident with the headline: How a Texas student blew the whistle on a rogue AI hacking attempt.

"I realized the attempt was a bigger deal than I thought," Demir said.

Demir moved with his family from Turkey to the U.S. in 2022. He transferred from Austin Community College to UT Dallas in January.

He said the attention was a pleasant surprise during a summer in which he had hoped to find an internship.

"Even though getting attention wasn't my intention when I stopped the attack, the recognition afterward made me feel truly seen, especially after months of sending applications and hearing nothing back," Demir said.

Hamlen said Demir's actions reflect the need for the next generation of computer scientists to withstand AI social engineering attacks.

"It goes beyond mere technical know-how; they must be able to stand up to social pressure, ridicule and doubt - all of which could be the result of a social engineering campaign crafted by a malicious AI - to safeguard the integrity of the software systems they maintain," Hamlen said. "Sinan's courage in resisting one of those attacks showcases the value of that kind of training."

Demir said he wasn't particularly interested in cybersecurity before the incident. Now, he wants to specialize in the field. But that day, he was simply trying to be helpful on a site where users collaborate to identify problems and propose fixes.

"I was just trying to do what I felt was right," he said.