Spiders and Robots Are Ghouls and Goblins
Page 1 of 1
In a move to improve online ad campaign measurement, auditing specialist ABCi Monday pledged to create and maintain a master list of spiders and robots, which are widely acknowledged as the main culprit for inaccurate measurements, for the Interactive Advertising Bureau (IAB).
Why is a list important? Spiders and robots automatically search Web content for information and are used to: retrieve data about content for search engines, gather content pages for personal offline viewing and track advertising activity by Web site and by advertiser and identification of "What's new information". Because there may be multiple robots or spiders on a site, sites may have a problem distinguishing between non-human spider and robot activity and consumer interactions, which throws off the metrics reported to advertisers.
Robin Webster, president and chief executive officer of the IAB, extolled the virtues his organization's plan, claiming the robot/spider list will cut back "inconsistent counts."
"The primary benefits of the list will be to reduce the time, effort and cost to publishers, third party ad servers and agencies associated with report reconciliation, and to provide advertisers with accurate measurement metrics," Webster said. "By utilizing this approved and constantly updated master list, publishers will be able to filter out robot traffic from their reporting to advertisers, instilling greater confidence in the medium."
Once the robot/spider list is in place, IAB said the next step will be for ABCi to examine the effect of spiders and robots on the two different methods of ad serving: client-initiated and server-initiated, followed by the release of measurement guidelines by year-end.
Having maintained a robot list compile from over 8,000 audits and 200 trillion transactions since May 1995, it is logical that ABCi is qualified to compile the list and conduct further examinations of the ad measurement process. ABCi invites other organizations to suggest updates to the list, so long as proper documentation is supplied. On the other hand, ABCi also welcomes challenges to the list, also accompanied by proper documentation.
Available for free to IAB members and ABCi clients, the robot/spider list will be updated monthly, and serves as the third step in a slew of recommendations the IAB will post by year's end.