Why Your Site Wasn’t Indexed

Do you Yahoo? Personally, I try to take off evenings and weekends.

But at Jupitermedia’s recent Search Engine Strategies 2004 Conference & Expo in New York, I ran into people thinking 24/7 about how to help you and your site grab more mindshare on the search listings pages that — let’s face it — are going to drive nearly all of your incremental Web traffic.

For those who couldn’t attend the conference, did you know that the best site architecture is one that designs a site just to appeal to the bot-like constituency of Web crawlers? Though it has the catchiest marketing phrase, Yahoo is in fact only number two among search sites. Google, according to comScore, is number one, followed by AOL, MSN, AskJeeves, Infospace, AltaVista (my personal search destination of choice) and Lycos.

Of course I no longer think that if you build a site, people will come. (It’s not 1994 anymore.) Silly me. I believed that a Web site captured eyeballs because of the quality of its contents, not the glitziness of its home page.

But you’d be amazed at how many people don’t know that there’s more to the job of building traffic than just building a flashy home page. (In fact, Flash is actually a detriment to traffic. More on that later.)

Indeed, a whole new discipline has sprung up among marketing types who explain the design techniques that make it easy for search engines — which get confounded a lot more quickly than you’d think — to successfully capture your site’s listings.

Most of my education came at the show’s panel on Successful Site
Architecture. Barbara Coll, chief executive officer of WebMama, angled her talk with the intention of bridging a perceived divide between the people who sell things (marketing types like herself) and the people who build them (us IT folks, naturally).

“Everything you, your programmers, and your IT department does affects search-engine rankings,” she said, pointing out that decisions by omission often have major, unintended consequences.

Take the aforementioned Flash example. Using the graphics application to create a splashy (and to my mind, slow) home page won’t impress search-engine crawlers. They can’t recognize Flash pages. Accordingly, if your navigation is Flash-based, you’ll have to jump through hoops to get indexed, Coll pointed out. Or you may not get indexed at all.

Indeed, lots of whiz-bang effects we’re used to seeing are also an
impediment to getting indexed by search engines. Crawlers aren’t good at recognizing Javascript-based pages, frames, dynamic URLs, and URLs with too many parameters.

So what can search crawlers do and see? Well, they’re best at recognizing old-fashioned, text-based links. So it’s back to the future if you want to optimize your Web site to boost its odds in the search listings sweepstakes.

Turning to URLs, Coll pegged as bad one of my personal sticking points: pages that have 15 hyphens and special characters embedded in the path listing. Such destinations aren’t an invitation to traffic.

However, one trick that will help searchers and search engines alike find you is to use file names that have something to do with what the site is about. So if you’re selling shoes, a directory called “/spikedheels” might actually be appropriate.

“Having directory names that look like ‘file.asp’ is not necessarily the best thing,” quipped Coll.

Another important tool is the user-sensitive error page. Customizing “404 error” pages so they link back to the main site helps humans and crawlers. Better to say “The document you were looking for could not be found. Try searching for it.” Then segue to helpful search links. (One nontraditional approach I did enjoy was the White House’s error page, which helpfully advised that “many files associated with the previous administration have been removed from this server.”)

Summing up, Coll explained the thinking that drives her site-design advice.

“The real power of search engine marketing is to acquire new people,” she advised, “not just to serve the people who already know you.”


Alexander Wolfe is a senior editor at internetnews.com

Get the Free Newsletter!

Subscribe to our newsletter.

Subscribe to Daily Tech Insider for top news, trends & analysis

News Around the Web