Web-scraping AI bots cause disruption for scientific databases and journals

 

Automated programs gathering training data for artificial-intelligence tools are overwhelming academic websites.

In February, the online image repository DiscoverLife, which contains nearly three million photographs of different species, started to receive millions of hits to its website every day — a much higher volume than normal. At times, this spike in traffic was so high that it slowed the site down to the point that it became unusable. The culprit? Bots.

These automated programs, which attempt to ‘scrape’ large amounts of content from websites, are increasingly becoming a headache for scholarly publishers and researchers who run sites hosting journal papers, databases and other resources.

volume of requests” to access a website, “which is causing strain on their systems. It costs money and causes disruption to genuine users.”






A flood of bots


Internet bots have been around for decades, and some have been useful. For example, Google and other search engines have bots that scan millions of web pages to identify and retrieve content. But the rise of generative AI has led to a deluge of bots, including many ‘bad’ ones that scrape without permission.


This year, the BMJ, a publisher of medical journals based in London, has seen bot traffic to its websites surpass that of real users. The aggressive behaviour of these bots overloaded the publisher’s servers and led to interruptions in services for legitimate customers, says Ian Mulvany, BMJ’s chief technology officer.


which roughly two-thirds had experienced service disruptions as a result. “Repositories are open access, so in a sense, we welcome the reuse of the contents,” says Kathleen Shearer, COAR’s executive director. “But some of these bots are super aggressive, and it’s leading to service outages and significant operational problems.”


More info: 

Website Link: https://databasescientist.org/ 

Contact Us: contact@databasescientist.org

_______________________________________________________________________________________________________________   

Social Media:








Comments

Post a Comment

Popular posts from this blog

Large Language Models and Vector Databases for News Recommendations

Image
  Large Language Models   (LLMs) generated a global buzz in the machine learning community with recent releases of generative AI tools such as Chat-GPT, Bard, and others alike. One of the core ideas behind these solutions is to compute a numerical representation of unstructured data (such as texts and images) and find similarities between these representations. However, taking all of these concepts into a production environment has its own set of machine learning engineering challenges: How to generate these representations quickly? How to store them in a proper database? How to quickly compute similarities for production environments? In this article, I introduce two open-source solutions that aim to solve these questions: Sentence Transformers  [1] : an embedding generation technique based on textual information and; Qdrant : a vector database capable of storing embeddings and providing an easy interface to query them. These tools are applied to NPR [2], a News Portal R...
Read more

Is Palantir creating a national database of US citizens?

Image
According to its website, Palantir makes "products for human-driven analysis of real-world data." On May 30, 2025,  The New York Times published an article   titled "Trump Taps Palantir to Compile Data on Americans," detailing a supposed combined effort between the U.S. federal government and the data software company Palantir to centralize data on American citizens. Data privacy advocates did not take the news well, calling it "dystopian" and a massive invasion of privacy. Four days later, Palantir posted a statement to X responding to the article, calling the reporting " blatantly untrue" because "Palantir never collects data to unlawfully surveil Americans, and our Foundry platform employs granular security protections." On June 9, 2025, the company followed up with a long blog post on X titled " Correcting the Record: Responses to the May 30, 2025 New York Times Article on Palantir," which aimed to fact-check the article....
Read more

NIH autism database announcement raises concerns among researchers

Image
A new research initiative from the U.S. National Institutes of Health (NIH) seeks to gather and analyze multiple autism-related datasets to better understand the underlying causes of the condition. Letters of intent are due tomorrow, but the project’s atypical call for applications has autism scientists divided on whether or not to participate. The Autism Data Science Initiative provides $50 million for an anticipated 10 to 25 projects, according to the 27 May announcement. The funding opportunity uses an Other Transaction (OT) funding mechanism, which tends to be implemented when, among other reasons, a project is expected to evolve over time, more collaboration is required from the NIH or nontraditional research groups are involved, according to an NIH website. For a field that has recently seen budget cuts, some researchers are welcoming the opportunity for funding, says   Helen Tager-Flusberg , director of the Center for Autism Research Excellence at Bosto...
Read more