Being Afraid of Amazon’s Spiders

Some operators of “associate sites” that link to Amazon.com are less than happy about the online retailing giant’s search BOTs , which they claim are “spidering” their sites excessively, taking up bandwidth, time and money.

Seattle-based Amazon , which claims to be “the online affiliate marketing leader,” has long offered associate deals with smaller Web site operators to, in effect, manage specialized book, music and video stores on those sites via links to Amazon.

The company advertises that if you place links on your site to Amazon.com, you can earn up to 15 percent in referral fees when your visitors make a purchase at Amazon. Amazon says it has more than 900,000 such associates.

But some of them are less than happy about the webcrawlers or spiders that Amazon’s Alexa Internet subsidiary sends their way. One associate site operator complained that the BOT visited his site “13,406 times over a period of 17 hours, consuming approximately 200 megabytes of bandwidth.”

Another associate posted this in a discussion about the matter on slashdot.org:

• “I’m an Amazon associate and I’ve been following this problem. Amazon’s web-BOTs are looking for outdated links to books that don’t exist, etc. The reasoning is that if the associate fixes the dead links, then Amazon (and the associate) will presumably make more money.
• “The problem is that the BOTs are way too diligent. They go to every single link on every single page, even if the page is dynamically generated. Many sites have an infinite combination of URLs, and as a result, the BOTs sit on them trying to download every single variation of query. That means that Joe Amazon Associate’s web site is hammered with requests and his bandwidth fees go through the roof.”

Amazon said it is working to rectify the problem but also added that it will continue to rely on search BOT technology.

“We are trying to make sure the links to Amazon products are actually working links,” Amazon.com spokesperson Patty Smith told internetnews.com. “This is a feature that we officially launched about a year ago for top performing sites and we have started rolling it out to other associate sites.”

She emphasized that the BOT search will be a “once-a-month service.” She added that “we take any complaint seriously” and “we want to make sure the service works well for everyone. We want to make sure associates are compensated appropriately.”

According to a posting on its associates discussion board, Amazon informed associates early this month about its crawler.

The retailer halted use of the Alexa BOT while it looked into some of the complaints, but then said it will resume use next month. Amazon acquired Alexa in April of 1999.

“As a reminder, Alexa’s Broken Link crawler will only execute once per month,” Amazon said in a posting. “We do anticipate restarting this tool beginning next month, but with sole intention of identifying broken and invalid Amazon.com Associate links.”

Amazon also updated its associate agreement language to state explicitly that: “you acknowledge that we (and our corporate affiliates, such as Alexa Internet Inc.) may crawl or otherwise monitor your site for the purpose of ensuring the quality and reliability of Special Links on your site (for example, to detect links that are broken or non-functional, links to products that are out of stock or otherwise unavailable, etc.). Therefore, you agree that we and our corporate affiliates may take such actions and that you will not seek to block or otherwise interfere with such crawling or monitoring …”

The complete text of the associates agreement is available here.

This isn’t the first time that Amazon has experienced trouble with its Alexa unit. In May 2001, the company narrowly dodged an FTC probe that alleged the online retailer likely engaged in some deceptive business practices regarding the privacy of users of its Alexa service. However, no enforcement action was taken at the time.