Neil Vogel, the CEO of People, Inc., the largest digital and print publisher in the United States, has made serious allegations against Google, claiming that the tech giant is acting unethically by crawling its websites to enhance its AI products. This accusation comes amidst an ongoing debate about the role of large tech companies in the content creation and publishing landscape. People, Inc., which operates over 40 popular brands including People, Food & Wine, Travel + Leisure, Better Homes & Gardens, and Allrecipes, has experienced a significant decline in traffic attributed to Google.
At the recent Fortune Brainstorm Tech conference, Vogel revealed that three years ago, Google Search was responsible for approximately 65% of the company’s web traffic. However, this figure has now plummeted to the "high 20s." In a previous discussion with AdExchanger, he noted that at one point, Google provided as much as 90% of People Inc.'s traffic from the open web. Despite the decline, Vogel stated, “I’m not complaining. We’ve grown our audience. We’ve grown our revenue,” emphasizing that the real issue lies in Google’s use of its web crawler.
Vogel voiced his concerns about Google’s single crawler system, which is utilized for both indexing websites for search results and training its AI systems. “Google has one crawler, which means they use the same crawler for their search, where they still send us traffic, as they do for their AI products, where they steal our content,” he explained. This situation, Vogel argues, creates an unfair advantage for Google and diminishes the leverage that publishers have in the evolving digital landscape.
In response to the challenges posed by AI crawlers—automated programs that scan websites for content—Vogel’s company has begun implementing solutions from web infrastructure company Cloudflare to block these bots unless they enter into content agreements. He described the collaboration with OpenAI as a “good actor,” suggesting that such partnerships are necessary for publishers to thrive in the AI era. Although Vogel did not disclose specific names, he mentioned that “large LLM providers” have approached People Inc. regarding potential content deals. While no agreements have been finalized, he noted significant progress since adopting the crawler-blocking measures.
Despite the success in blocking some AI crawlers, Vogel pointed out that Google’s crawler cannot be obstructed without risking the loss of indexing on Google Search, which still provides around 20% of their traffic. “They know this, and they’re not splitting their crawler. So they are an intentional bad actor here,” Vogel asserted, highlighting the challenges publishers face in navigating their relationship with Google.
Janice Min, the editor-in-chief and CEO of Ankler Media, echoed Vogel’s sentiments, labeling major tech companies like Google and Meta as “content kleptomaniacs.” She expressed skepticism about the benefits of partnering with AI companies, stating that her organization actively blocks AI crawlers as well. Meanwhile, Cloudflare CEO Matthew Prince, who also participated in the conference, suggested that future changes in the AI landscape might be influenced by new regulations. He questioned whether traditional legal frameworks, such as copyright law, would effectively address the challenges posed by AI companies.
Prince also remarked on the detrimental impact Google has had on the publishing industry by teaching publishers to prioritize traffic over original content creation. He noted that this shift has led to content strategies focused on click-baiting, as seen with companies like BuzzFeed. However, he acknowledged the tough position Google currently finds itself in and predicted that by this time next year, the search giant might be compelled to pay content creators for the use of their material in AI models.
As the conversation around AI and content ownership continues to evolve, the publishing industry is at a crossroads, navigating the complex relationship between technology and content creation.
