Large Retailers Block Blend's AI Shopping Agent More Often Than Smaller Ones
TLDR
Bot protection blocked Blend's AI shopping agent at 29 of 200 retailers, with large retailers refusing it far more often than emerging ones.
Optimixed Overview
Optimixed's original report on this story.
PPC Land reports that 29 of 200 retailers blocked Blend's AI shopping agent, with large retailers refusing it at a 20% rate and emerging retailers at 2.5%. All 24 refusing retailers with readable robots.txt files still allow AI crawlers, which raises the question of who set the blocks.
How we got here: earlier coverage
- Nov 1Resemble AI Test Finds Common Bot Detection Tools Caught 3.1% to 14.8% of Disguised AI Agents
A Resemble AI test found that common bot detection tools caught only 3.1% to 14.8% of disguised AI agents, while its own RAD-S1 tool flagged at least 98.8% of the same agents.
- Oct 31Google and Bing will honor a shared robots.txt signal to block AI training
Google and Bing said they will honor a shared robots.txt signal that blocks AI training, and a study put AI Overviews click-through at 0.9%.
- Oct 20European regulators tell AI developers to respect robots.txt and six other data minimization steps
European data regulators issued guidelines telling AI developers to respect robots.txt blocks and adopt six other data minimization practices, with public comment closing October 30.
- Sep 29Amazon Blocked Meta’s Muse, And Robots.txt Had Nothing To Say via @sejournal, @slobodanmanic
Amazon Blocked Meta’s Muse, And Robots.txt Had Nothing To Say via @sejournal, @slobodanmanic