Data on Billions of Web Pages Made Available | Internet News

Data on Billions of Web Pages Made Available

Written By
David Needle
David Needle
Oct 12, 2005
3 minute read

Amazon.com’s Alexa Internet division has launched a service for
developers today that offers over a hundred terabytes of
Web site information. Alexa Internet, which has been crawling the Web since
1996, said it has data on some 4.5 billion Web pages covering 16 million
sites.

The company’s Alexa Web Information Service (AWIS) is being released
after a beta test period of one year, during which over 3,000 developers used it.

Examples of Web site information in AWIS that developers can draw on are:

— URL Information about millions of Web sites, including
traffic rank, load speed, related links, site owner contact information
and adult content identification.

— Browse Category, everything that is needed to build a site browse
tree, including lists of sub-categories for a top-level category, most
popular sites in a category and all sites within the category.

— Web search based on Alexa’s crawl. This includes support for advanced
queries, so developers can construct difficult queries and incorporate the
answers into their applications.

— Crawl meta data for specific pages found in the Alexa
crawl, including size, checksum, frames, images and links.

— Web Map, a topographic representation of Alexa’s Web crawl displaying
all links in and out for specific pages on the Web.

“The service is
ready to roll,” Geoff Mack, Alexa product manager, told
internetnews.com. “I think we’re speaking to a whole new class of
developers based on the Web 2.0 concept, where Alexa does some of the hard
work for them, so they can simply pull from our databases and focus more
energy on the things they’re good at.”

While you may not connect e-commerce giant Amazon.com
to developer tools, the company launched its Web Services platform in July
2002, and it says over 140,000 developers have signed up since to use it. There
is no charge for the first 10,000 AWIS requests made by a developer each
month, and each additional request is only $0.00015.

The new AWIS is designed to let developers get information about the Web
and programmatically incorporate those answers directly into their
applications, which could be an anything from an e-commerce site or
storefront affiliated with Amazon.com to a specialty search service.

“Undoubtedly, some of this will feed back to the Amazon eco-retail system,
but we’ve already seen many independent developers take advantage of this
for their own private services,” Adam Selipsky, vice president of Web services
at Amazon.com, told internetnews.com.

One such example is AdBrite, which used the beta version of AWIS to
enhance its existing service and is now a paying customer. AdBrite is a
marketplace for buying and selling advertising space on thousands of Web
sites.

“At AdBrite, we have 150 to 200 publications sign up every day,” explained
Philip Kaplan, CEO of AdBrite, said in a statement. “One of the major statistics we can provide
is their Alexa ranking to help determine their popularity versus other
Web sites. The fact that AWIS is a Web service means that we dedicate minimal
software development resources while offering a service to our customers
that is invaluable in their decision-making process.”

Internet News Logo

InternetNews is a source of industry news and intelligence for IT professionals from all branches of the technology world. InternetNews focuses on helping professionals grow their knowledge base and authority in their field with the top news and trends in Software, IT Management, Networking & Communications, and Small Business.

Property of TechnologyAdvice. © 2026 TechnologyAdvice. All Rights Reserved

Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.