AI bots swarm UDSpace for information

Long gone are the days of slowly sifting through dense pages of information for answers to mind-bending questions or remaining up-to-date with scientific discoveries. Now, artificial-intelligence bots are doing the sorting and summarizing for us by swarming scholarly content as if it were mounds of honey.

And UDSpace, the University of Delaware's digital repository, is dripping with honey. This valuable resource is managed by UD's Library, Museum and Press, to preserve and provide long-term free and open access to scholarly research outputs produced by UD faculty, students and staff, which includes articles, white papers, reports, and theses and dissertations.

In the last year, AI bots caused a significant spike in the volume of online information queries to UDSpace, which strained the capacity of the system and resulted in responses being delayed or not available at all. UDSpace was dealing with an unforeseen consequence of its own success: By making UD research freely available to anyone, it had actually made it less accessible to everyone.

"It's all about the data," said Annie Johnson, associate university librarian for research, teaching and technology at the Library, Museum and Press. "AI companies are constantly looking for more data to improve their Large Language Models (LLMs). UDSpace has massive amounts of high-quality data that is attractive to these companies."

This phenomenon isn't unique to UDSpace - other institutional repositories are also dealing with this challenge, said Johnson. In the past year, AI bots have also made their presence known at Duke University and the University of North Carolina at Chapel Hill, for example. Similar institutions throughout North America, Europe and other regions are experiencing the same challenges, according to a recent survey by the GLAM-E Lab, a joint research center at the University of Exeter in England and the New York University School of Law that focuses on galleries, libraries, archives and museums.

"At the start of a normal day, we're used to receiving less than 10 simultaneous queries a second to access information," said Mark Grabowski, head of Library Information Technology. "As you can imagine, we were surprised to have 80 to 100 simultaneous queries with users complaining about UDSpace being inaccessible."

The Library's IT team diligently investigated the issue, finding that AI bots submitted multiple complex requests, including a researcher's name, a topic, date and title, in about half a second. Grabowski said that there are rules to AI bots making simple requests, "but the bots were mostly ignoring these rules, and the team noticed that many of the searches were randomly, and quickly, moving through UDSpace."

With searches coming almost nonstop and from almost everywhere around the world, Colleen Estes, assistant head of Library IT, said that finding a resolution was a matter of trial and error. One of the initial ideas was to block IP addresses.

"Our goal is always to enable, improve and increase access to information," said Estes. "We knew that a permanent resolution couldn't be to block IP addresses because that would lead to us blocking entire countries, and we don't want to block real people from accessing information."

Another temporary response was for the system to tell the AI bots that the information did not exist.

"When we did this, it was like poking a hornet's nest, and the bots tried harder," Grabowski said. "The unusually high 80-100 simultaneous searches jumped to 250 to 300 simultaneous searches."

Since May 2025, UDSpace's message to the AI bots has changed: We have the information that you're looking for, but you're forbidden to access it.

Grabowski said that UDSpace has three layers: the website that takes in a request, the application that processes the request and an underlying database that is queried for information based on the request. Now, there's a "forbidden" message that appears at the website level when the request comes from a bot, allowing the server to receive traffic, monitoring AI bots, but without requests getting to, and slowing down, the application.

"There was one round of bot queries that ran for an entire weekend, determined to gather as much information as possible when no one was looking," he said. "Now that we have 'forbidden' in place, they receive so many rejections, they stop making the request."

AI is quickly changing reality, but human intelligence is still a potent force in solving challenges like AI bots.

"This is all a testament to the ingenuity and the great lengths that our team has gone to to maintain access to information," Johnson said. "AI is changing the rules, and this is an excellent example of UD Library's IT team quickly responding with an incredible solution that doesn't limit our students, faculty, staff, and the global community from accessing the information that UDSpace holds."