RealTime IT News

VMware Goes Benchmarking

Some companies are using virtualization to create efficiencies in their datacenters by running multiple operating systems on fewer machines. But how do they know it works?

Until now there hasn't been a proper gauge to check a virtual machine's speed or performance.

But VMware, which fancies itself as the lead driver in virtualization technologies, has created VMmark, a benchmark system to help customers measure the performance of various applications running in virtual machines.

While traditional benchmarks were developed to measure the performance of a single workload on a server, they don't effectively gauge the ability of a server to support multiple, simultaneous workloads on the same server.

VMmark is the answer, said Andrea Eubanks, senior director of enterprise and technical marketing at VMware.

Eubanks said customers who have been running production workloads have long been asking VMware for the typical virtual machine density policy for certain enterprise applications, or the right number of virtual machines to run on a certain piece of hardware. Until now, server vendors could only help them roughly estimate.

With VMmark, server vendors can publish a score that provides scaling information about the workloads each product can support, as well as on the overall performance of virtual machines running on a server. This helps customers determine how much hardware they need to buy based on their plans to deploy certain enterprise workloads.

Eubanks said VMmark presently runs six basic workloads as virtual appliances, including a file server, mail server, Web server, a standby server, an OLTP database and a Java order-entry system.

Half run on Windows, half run on Linux. Those six workloads represent what VMware calls a "tile." Scoring is determined by how many tiles a machine can scale up to.

"Unfortunately, because of Microsoft's licensing restrictions, we're not able to package the Exchange e-mail server and the other workloads on Windows on virtual appliances, although it would massively simplify the benchmark tool if we could," Eubanks said.

Dell  and Sun Microsystems  have published benchmarks with the beta version of VMmark, but VMware is working with IBM, HP Fujitsu-Siemens and other hardware vendors on VMmark.

VMmark is a VMware benchmark, but Eubanks said the company has lent VMmark to performance benchmark group Standard Performance Evaluation Corporation (SPEC) to be used as the basis for a standard benchmark for measuring virtualization performance.

SPEC includes a variety of high-tech leaders, including Apple , IBM  and Microsoft .

But don't expect SPEC's spec to be VMmark on steroids.

Eubanks claimed the open standard virtualization benchmark will have a more narrow scope than VMmark in two areas: VMware defines the operating system configuration and application configuration in VMmark; Eubanks said SPEC's may not cover both.

VMmark is the latest release from VMware, which is cycling for its initial public offering. But the EMC subsidiary is no stranger to testing.

The company in April rolled out the TCO Calculator, a Web-based tool that gauges the return on investment of VMware Infrastructure and provides a report of potential cost savings.