RealTime IT News

Search Leaders, Bloggers Band to Fight Comment Spam

MSN, Yahoo and blogging software vendor Six Apart said they would adopt a tag proposed by Google and join the effort to fight the comment spam that can skew search results.

Most search engines use popularity as one way of evaluating a site's worth. If lots of other sites link to it, the reasoning goes, it must have valuable content. This was the insight that made Google a search leader, and it's now used to some extent by the other major search players.

However, with the rise of search engine optimization and affiliate marketing, some people began to game the system, trying to insert links to their own sites wherever they could. The comment sections of blogs were a top choice for posting comments like, "Great site! Check out mine," along with a link to the writer's own site.

Google's proposed tag, (rel="nofollow") tells search engine spiders to ignore the link when indexing.

On the Google corporate blog, Matt Cutts, a Google software engineer, and Jason Shellen, Blogger program manager, wrote, "From now on, when Google sees the attribute (rel="nofollow") on hyperlinks, those links won't get any credit when we rank Web sites in our search results."

"The problem they're addressing is comment spam," said Kevin Lee, president of Did-It, a search engine marketing company. "People will write robots that clutter up your comments area with hotlinks that strip your page rank and shoot it off toward the spam site. Google's spider sees the links and says, 'This blog thinks that site is good.'"

Lee said spammers can potentially add their own links to tens of thousands of blogs. "It gives them an amazing advantage in spamming [search engines]," he said.

It's notable, said Danny Sullivan, editor of SearchEngineWatch, that Google proposed an industry standard and was joined by its rivals in making the announcement. (SearchEngineWatch and internetnews.com both are owned by Jupitermedia.)

"They had already decided to do it and talked to some vendors about the idea," Sullivan said. Google engineers evidently contacted MSN and Yahoo at the beginning of the week.

"One thing that helped is that both companies were already considering ideas like this. Google had a solution that looked like it would work and some vendors who were already on board with it. That shouldn't diminish the fact that they did come out in a unified voice," Sullivan added.

The last time that happened, he said, was in 1996, when a spidering work group composed of employees of the different search engines came together to decide on a meta robot standard to block spidering.

Jeremy Zawodny, a Yahoo marketing executive, said "Every company in this business has been thinking about this, and some of us have been writing about possible ways to combat comment spam. We had to start with something, and we started with this."

He said that since Yahoo doesn't host blogs itself, the use of the no-follow tag would take time to make its presence felt in Yahoo's index.

"It's not a difficult thing to do," he said, "but it's not like flipping a switch. It's a question of how long it takes for those changes to propagate through our index. People should start to notice the effects in the next few weeks."

MSN product manager Justin Osmer pointed out that MSN Spaces already lets bloggers limit access, require authentication before posting comments and delete comments. He said the company's "Search Champs," a group of search industry advisors, had suggested something similar.

"MSN supports the idea of cooperative standards to combat spam and will support 'no follow' ... in the near future," he said.

Six Apart, a provider of hosted and licensed blogging software, said it would turn on the tag for comments automatically for subscribers to the hosted TypePad service by the end of day Wednesday. Users of its MoveableType software can download a plug-in to add this functionality.

Six Apart's Anil Dash, vice president of the professional network, said TypePad users won't be able to turn off the no-follow tag. "We try to err on the side of security. If you're an advanced user and can edit your HTML, you can disable it," Dash said, adding that Six Apart plans to offer more granular controls in future releases. "Some people want to be able to share page rank or link credit with the people who leave comments," he said.

A Google spokesman refused an interview request and referred inquiries to the company's blog, which said that previously posted links on blogs would be "transformed" to include the no-follow tag. An FAQ said that individual bloggers "probably" don't need to do anything. "Updating the software that generates these pages will ensure that most bloggers get these changes automatically," the FAQ said.

"Google is trying to ensure they continue to improve the user experience," said Ellen Siminoff, CEO of Efficient Frontier, a company that specializes in paid search. "That will help marketers as well. We're thrilled that Google and Yahoo/Overture are continuing to evolve their natural results, so the channel continues to grow and be useful for people."

Sullivan pointed out that the tag is not useful only to bloggers; it can be used on Web sites as well. "For the first time, page authors have been given the ability to control individual elements of the page," he said, rather than just telling spiders not to index the entire page.

But it was unclear from the various announcements whether the no-follow tag could be automatically implemented across the universe of hosted blogs. That's an important issue, Sullivan said, because most comment spam takes place on abandoned blogs, not active ones. "On blogs that have been started and abandoned, you left the door wide open, and now people are ransacking it," he said.

Yahoo's Zawodny said, "This is a technique that requires effort on the part of the blog owner. If a blog is abandoned, they probably won't make a decision like that. But it may prompt some of the weblog hosting companies to think more about their policies for hosting abandoned blogs."

Said Did-It's Lee, "If the tag requires an opt-in by the blog owner to implement it, and abandoned blogs are the most heavily spammed, it barely makes an impact."

But Six Apart's Dash disagreed that abandoned blogs were the major source of comment spam. "They do become cesspools and do get attacked, but there's not a lot of value in them, and spammers know this," he said. "Spammers attack high page-ranked blogs."

And those authors likely will quickly follow the no-follow rule.