Data on Billions of Web Pages Made Available

Amazon.com’s Alexa Internet division has launched a service for
developers today that offers over a hundred terabytes of
Web site information. Alexa Internet, which has been crawling the Web since
1996, said it has data on some 4.5 billion Web pages covering 16 million
sites.

The company’s Alexa Web Information Service (AWIS) is being released
after a beta test period of one year, during which over 3,000 developers used it.

Examples of Web site information in AWIS that developers can draw on are:

— URL Information about millions of Web sites, including
traffic rank, load speed, related links, site owner contact information
and adult content identification.

— Browse Category, everything that is needed to build a site browse
tree, including lists of sub-categories for a top-level category, most
popular sites in a category and all sites within the category.

— Web search based on Alexa’s crawl. This includes support for advanced
queries, so developers can construct difficult queries and incorporate the
answers into their applications.

— Crawl meta data for specific pages found in the Alexa
crawl, including size, checksum, frames, images and links.

— Web Map, a topographic representation of Alexa’s Web crawl displaying
all links in and out for specific pages on the Web.

“The service is
ready to roll,” Geoff Mack, Alexa product manager, told
internetnews.com. “I think we’re speaking to a whole new class of
developers based on the Web 2.0 concept, where Alexa does some of the hard
work for them, so they can simply pull from our databases and focus more
energy on the things they’re good at.”

While you may not connect e-commerce giant Amazon.com
to developer tools, the company launched its Web Services platform in July
2002, and it says over 140,000 developers have signed up since to use it. There
is no charge for the first 10,000 AWIS requests made by a developer each
month, and each additional request is only $0.00015.

The new AWIS is designed to let developers get information about the Web
and programmatically incorporate those answers directly into their
applications, which could be an anything from an e-commerce site or
storefront affiliated with Amazon.com to a specialty search service.

“Undoubtedly, some of this will feed back to the Amazon eco-retail system,
but we’ve already seen many independent developers take advantage of this
for their own private services,” Adam Selipsky, vice president of Web services
at Amazon.com, told internetnews.com.

One such example is AdBrite, which used the beta version of AWIS to
enhance its existing service and is now a paying customer. AdBrite is a
marketplace for buying and selling advertising space on thousands of Web
sites.

“At AdBrite, we have 150 to 200 publications sign up every day,” explained
Philip Kaplan, CEO of AdBrite, said in a statement. “One of the major statistics we can provide
is their Alexa ranking to help determine their popularity versus other
Web sites. The fact that AWIS is a Web service means that we dedicate minimal
software development resources while offering a service to our customers
that is invaluable in their decision-making process.”

Get the Free Newsletter!

Subscribe to our newsletter.

Subscribe to Daily Tech Insider for top news, trends & analysis

News Around the Web