Showing a convergence between open source and mainstream enterprise technology, HP unveiled a new file system that uses the company's hardware and Linux to deliver up to 100 times more bandwidth than traditional clusters.

The HP StorageWorks Scalable File Share (HP SFS), which includes the company's ProLiant servers and StorageWorks disk arrays, allows bandwidth to be shared by distributing files in parallel across clusters of servers and storage devices. The system was designed to bust bandwidth constraints in high-performance computing environments.

Simon Towers, a technical director in the office of the CTO at HP, said the system is based on HP's "storage grid" architecture, allowing applications to see a single file system image regardless of the number of servers or storage devices connected to it.

The system also protects against hardware failures with redundant hardware and built-in fail-over and recovery. SFS can span dozens to thousands of clustered Linux servers, making it ideal to run distributed applications for science and engineering projects.

Towers told internetnews.com products like SFS are important to HP as a storage vendor because of the drastic increase in the amount of unstructured data -- e-mail, video clips and PDFs -- has grown. File systems are needed to manage this data.

"In a way, HP has sort of leveraged all the open source programmers out there to rearchitect how file systems are built and lay that on top of storage grid architecture," Towers said.

Companies such as IBM and HP, along with software makers like Red Hat and SuSE, have been working to broaden the sphere of the Linux operating system in the commercial enterprise, as well as scale the software out for HPC projects.

The companies want to push the envelope and prove that Linux can no longer be banished to the back rooms of research labs. However, they recognize Linux as a viable alternative to find the crack in Windows' armor.

That the SFS is powered by Linux is a statement from HP that Linux is viable for large-scale computing. SFS is the first commercial product to use Lustre, a new Linux clustering technology developed by HP, the U.S. Department of Energy (DoE) and Cluster File Systems.

The Lustre protocol used in SFS already powers some of the world's largest HPC environments, including the DoE Pacific Northwest National Laboratory (PNNL), where it eliminates bandwidth bottlenecks and saves users hours of time copying files across distributed file systems.

One of the 10 largest Linux clusters in the world according to the Top500 supercomputing rankings, PNNL's HP Linux super cluster, clocks in at more than 11 teraflops (one trillion floating point operations per second) and sustains more than 3.2 gigabytes per second of bandwidth running production loads on a single 53-terabyte, Lustre-based file share.

IBM makes a file system in its storage product line, called TotalStorage SAN File System, that runs Red Hat Linux Enterprise Server 3.0 and Sun Solaris 9. HP's SFS is based on StorageWorks grid architecture, which allows storage services to be delivered across a centrally managed system.

The SFS news comes just two days after the Top500 project posted its latest list of leading supercomputers. HP was second to IBM in total share of systems to make the cut, grabbing a number two ranking with 28 percent of all systems.