Cleversafe Expands Open Source Cloud Storage

To get seven copies of a particular piece of content distributed online, typically it has required a network to have seven physical copies of the data. But what if you could have seven copies available while having less than two copies’ worth of data on the network?

That’s been the promise of open source storage vendor Cleversafe, which this week is unveiling new technology to further streamline delivery — and potentially improving the process of making content available at massive scale.

Cleversafe claims its distributed delivery solution decreases storage requirements while improving availability. With its new smart client feature, the company said its technology gets even faster and more efficient, and it can take its dispersed storage network technology to the cloud — an approach that could have profound impact on content delivery networks, theoretically providing availability of content at a scale that is not possible today.

“We’ve been building this dispersed storage technology for 4.5 years, and what’s happening now is this big interest in cloud computing creates interest in cloud storage,” Cleversafe CEO Chris Gladwin told InternetNews.com. “It has certain characteristics, massive scalability, virtualization of resources and location with limitless reliability.”

Cleversafe first emerged out of stealth mode in October 2007 with its Dispersed Storage Network (dsNet) technology. dsNet takes slices of data and distributes them across multiple nodes. Users are able to specify how many locations the data will be sliced across and how many active nodes a downloader would need in order to recreate the data.

In its new release, Cleversafe is now adding a new layer to the dsNet experience: the smart client, which will determine at the client side the location from which to pull data to ensure downloaders the best possible performance.

“Dispersal has properties that enable it to revolutionize content delivery,” Gladwin said. “Instead of just storing a copy of the data, we mathematically transform the data into new numbers and you can store a virtual copy of the data that is comprised of many small slices.”

Those small slices require only a faction of the storage typically needed to host a number of identical copies under traditional distribution systems. Gladwin noted that the total storage required by slices is typically only 1.1 to 1.6x times the data a user started with.

In addition to saving a tremendous amount of space, the system also promises significant improvements in availability. In one scenario, a user could decide to divide their content into 16 data slices, with only 10 required to reassemble the data. That means that up to six hard drives can fail and the network would still be able to deliver the content. Gladwin argued that the approach means there are more copies available, increasing overall availability.

“If I want to read data off a dispersed storage network, there are many combinations,” Gladwin explained. “In the example of 16 slices needing any 10, there are over 8,000 combinations, and as a result, a user has thousands of choices on how to read data for performance.”

Gladwin also said that a non-dsNet solution would require 8,000 copies of the data to achieve similar flexibility.

Cleversafe’s approach is only getting more streamlined with the introduction of the smart client, according to Gladwin. He said the enhancement dynamically analyzes the dsNet to determine where the best performance can be had, optimizing delivery in real time. As a result, the dsNet can make content delivery faster than any one server, he said.

For example, if one million clients were trying to access content from a typical content delivery network today, they would be routed via a central DNS server to a particular download location. The clients would then be at the mercy of the download location’s connection speed and quality for the duration of the download.

“If I had a million clients using smart client on the dsNet, each one can dynamically reassess how to get the data and each independently can do it for themselves,” Gladwin said.

Open source

Cleversafe sells hardware appliances to access a dsNet (dubbed the “Accesser”) and an appliance to create the slices (called “Slicestor”). At the heart of both appliances is the open source code for the dsNet itself. It’s code that could also be extended beyond Cleversafe’s appliances, Gladwin said.

“You can take the dsNet client itself and embed into a device directly,” he said. “We have a Java SDK that will allow you to do that, so you could put it in your phone, etc.”

While Gladwin sees the smart client dsNet approach as being valuable to content delivery networks, he admitted that so far, he doesn’t have any public content delivery networks as customers, yet.

However, he argued that that Cleversafe’s focus has chiefly been on content storage clouds.

“What we’re putting in smart client today has profound implications, so we’ll see over time that we will be getting into the content delivery network realm and announcing partnerships,” Gladwin said. “How we bring technology to market is more of an open source philosophy instead of a pure commercial philosophy. Since the guts of our technology is open source, we first present the technology in a public way and then we follow-up with customer and partners.”