In recent times, companies like OpenAI and Perplexity have made bold assertions regarding their AI-powered search engines, which scrape information from the web to generate summarized answers. They claim that these innovations will create new revenue streams for publishers by driving increased reader traffic to their websites. However, the findings from a recent report by content licensing platform TollBit present a starkly different reality. According to the report, AI search engines send a staggering 96% less referral traffic to news sites and blogs compared to traditional search engines like Google.
The report highlights that the activity of AI developers scraping websites has more than doubled in recent months. Notably, OpenAI, Perplexity, Meta, and other AI companies scraped websites an average of 2 million times in the fourth quarter of last year. This analysis covered 160 websites, including national and local news outlets, consumer tech sites, and shopping blogs, over the last three months of 2024. Each page was scraped approximately seven times on average. CEO Toshit Panigrahi of TollBit remarked, “We are seeing an influx of bots that are hammering these sites every time a user asks a question. The amount of demand for publisher content is nontrivial.”
TollBit plays a crucial role in this landscape by integrating with publishers to track scraping activities and charge AI companies each time they scrape content. The data collected from publishers signed up on its platform provides valuable insights into traffic and scraping behaviors on their websites. Unfortunately, responses from major players like OpenAI and Meta have been lacking, as both companies did not comment on the findings. While a spokesperson for Perplexity acknowledged the importance of “robots.txt” directives, which guide web crawlers on site access, they did not address the specific claims made in the report.
As the landscape shifts, concerns among businesses relying on search traffic are escalating. Chegg, an edtech company, has taken legal action against Google, alleging that the search giant's AI-generated summaries included content from its site without proper attribution. This practice has reportedly led to a significant 49% decline in traffic for Chegg year-over-year, prompting discussions about the possibility of going private or seeking acquisition. Nathan Schultz, CEO of Chegg, expressed frustration, stating, “It’s time to say no.” He emphasized that a long-standing social contract exists between Google and publishers to direct users to high-quality content rather than retaining that traffic solely on Google.
Research firm Gartner has predicted a 25% drop in traffic from traditional search engines by 2026, largely attributed to the rise of AI chatbots and virtual agents. As AI technologies evolve, their impact on search traffic could be profound. Legal actions have also been initiated against both OpenAI and Perplexity by news publishers, claiming infringement of intellectual property rights. These companies are now grappling with the legal implications of their scraping practices while they continue to innovate.
Another issue complicating the situation is that many AI developers utilize user agents to crawl the web and gather data, but they often fail to properly identify or disclose their scraping bots. This lack of transparency makes it challenging for website owners to understand how their content is being accessed. Panigrahi noted that some companies, including Google, use the same bots for multiple purposes, which further blurs the lines between indexing and data scraping.
Despite attempts by publishers to block Perplexity from accessing their sites, the AI startup reportedly continues to send referral traffic, suggesting it may be scraping content covertly. In one instance, it scraped a publisher’s website 500 times while sending over 10,000 referrals. This raises significant concerns about the ethical implications of such practices. Last year, Perplexity faced backlash for scraping and republishing paywalled articles from major news outlets without sufficient attribution.
As the challenges of rampant AI scraping continue to affect publishers' traffic and revenue, many are exploring alternative economic models. Licensing articles directly has emerged as a viable solution, with organizations like the Associated Press, Axel Springer, and the Financial Times forging content deals with AI companies like OpenAI. Additionally, TollBit has developed a model that charges AI firms each time they scrape content from a publisher's site, working with over 500 publishers including TIME, Hearst, and Adweek.
In conclusion, the landscape of AI-powered search engines poses significant challenges for publishers, as they navigate the complexities of scraping, traffic decline, and legal repercussions. As AI technology continues to evolve, finding sustainable economic models will be essential for publishers to thrive in this new digital age.
