Epic Week for VROOM!

I know that I'm a bit biased towards the VMware Performance team blog VROOM!, but the amount and quality of posts that went up last week was really great.  If you are looking for something to read over the holidays - you should be sure to include VROOM!.  All of the new posts include new performance results that have not been published anywhere else before AND have full color graphs (and even a few diagrams as well!)

The week started with a set of tests that illustrate how Load-Based Teaming (LBT) performs in ESX 4.1.  This is a new traffic-aware dynamic load-balancing option for teamed NICs that was added with 4.1.

On Tuesday the first performance results with the newly released multi-host VMmark 2.0 benchmark were published. It also includes a great overview of VMmark two with a description of the workload, infrastructure operations, and scoring methodology for the benchmark.

After taking a day off, VROOM! continued a furious pace with perhaps the article of the week on Thursday (which was of course the one that I did!) on the Performance of Oracle RAC on vSphere 4.1.  Performance tests showed that a virtualized Oracle RAC database was within 11 to 13 percent of native.  This used a test that allowed for the comparison of virtual and physical to be as fair as possible.

Just for fun on Friday a post went up that used VMmark 2.0 to do performance testing in a new way.  It looks at the performance of low-cost entry level hosts combined into a vSphere cluster.  Each individual entry-level host would not have been able to run even one tile for VMmark, but combined they can support multiple tiles.

I guarantee that you will learn a few things about vSphere performance that you didn't know before if you have a chance to get some reading between trips to the mall to do stuff like getting to see Santa.

Todd

VMmark 2.0 Rocks Your Multi-Host World

That's right - VMmark 2.0 - a new version of an established benchmark - ROCKS!  It breaks new ground for virtualization benchmarks because it was designed and built as a multi-host benchmark.  The minimum config for VMmark 2.0 is two physical servers because a core component of the new benchmark are what is referred to as "Infrastructure Operations."  This is stuff like vMotion, Storage vMotion, and DRS.  These are some of the key reasons that vSphere is used and they only happen when there are more than one physical host involved.

These infrastructure operations are combined with an application workload to make VMmark 2.0. In addition the workload that is driven varies throughout the course of the test run.  This isn't just a steady state test, the workload levels change throughout the test.  This combination of factors is what makes VMmark 2.0 so compelling.  It includes more than just the application performance and should more closely reflect the work that is being done in real environments.

There are two key reasons why it is great that the benchmark more closely models real environments. The first is that as everybody competes to improve their VMmark 2.0 scores, the changes and enhancements that they make are more likely to be useful for real environments.  The second is that the results can be more directly applied in terms of sizing estimates.

This is good stuff that is a big improvement for benchmarking virtualized environments.

Todd

Android Platform vs Apple Control

I am the happy owner of an HTC EVO Android based phone.  I really like the big bright screen, the flexibility and power of the interface, the great and growing number of apps, and the service plan from Sprint.  The only thing that I didn't like was that the battery life was a little bit short.  I was able to make it through a normal full day, but around 9 or 10 pm the battery would be completely spent.  So I bought an extended battery from Seidio.  It is a big battery so it comes with a new back for the phone which ends up making it almost twice as thick for a portion of the back of the phone.  Now it is perfect for me.  I like that the battery lasts longer, more than I do an extremely thin phone.

The iPhone is also a great phone.  Many people are very happy with it.  As a matter of fact, if it had been available on Sprint a couple of years ago I probably would have gotten one.  With the iPhone you have to have AT&T as your carrier.  You also have to use iTunes, the Apple App store, and your interface is a grid of icons and a single button.  If you like this then great.  If you want something a bit different, then Apple doesn't have a solution for you.

The great thing about Android is that there is a wide range of devices offered by different carriers with different plans and different prices.  Everybody should be able to find a phone that fits their wants best, and still have access to a great set of applications and games.

The iPhone is not going to go away, but it also isn't currently a broad enough platform to keep the market share it has.  If the rumors are true this time, and the iPhone does become available on Verizon then this is a step in broadening their platform.  But it doesn't open up the whole market in the way the wide variety and availability of Android phones has done over the past year or two.

I like my phone. Although I think Apple would tell me that it is too big, the interface is too complicated, and the Android Market Place app store is uncontrolled and dangerous!  All of those things are exactly what I want.

Crazy Stuff Does Come To Pass

When I first started working on ESX about five years ago I went through the same light bulb process as everybody else.  I remember a conversation I that I had in the lab one day -

"Wouldn't it be cool if you put Oracle RAC on top of ESX? Then you could have complete flexibility. A database spread across multiple VMs and those VMs could be moved around as needed."

"It would be cool. But - Who would do that? People use RAC because their database needs more than one server. That would be crazy."

It becomes less crazy the more your light bulbs come on about the benefits of virtualization.  It's not just a consolidation story. It allows you to manage and run your infrastructure differently.  

With recent changes in both Oracle and SAP support statements it is now possible and supported to run RAC with SAP on vSphere.  I believe that the key reason why Oracle made the change to it's support statement was in response to customers.  Customers were asking for this support because they wanted to run RAC on vSphere and in some cases they were already RAC on vSphere.

This is what makes it fun to have these crazy conversations about what would be cool.  Sometimes crazy happens.

Todd

SAP Note for Oracle on VMware Now Includes RAC

Last week SAP note 1173954 (Support for Oracle on VMware) was updated to include RAC.  You can view the SAP note directly at https://service.sap.com/sap/support/notes/1173954 but you will need to have an account on the SAP site.

This update basically makes the same change in support that Oracle made with their metalink support statement for Oracle RAC on vSphere earlier this month.  It all boils down to a change in the list of prerequistes for running SAP on Oracle on vSphere.  In the previous version of the SAP note there was a bullet point that read -Only Oracle single instance, no support for Oracle RAC.  This has now been replaced with - RAC 11.2.0.2 and higher.

Before only single instance Oracle was supported and now RAC 11.2.0.2 or higher can be used.

So it is now supported to run SAP on Oracle RAC on vSphere (provided you follow the guidance and use the version levels called out in the SAP note and Oracle Metalink docs).

Todd

SQL Pass 2010 Trip Report

In addition to the two key things I picked up on at SQL Pass I also wanted to put together a more detailed trip report with a bit more detail.

The SQL Pass Summit 2010 conference was well run with an audience that is focused on Microsoft SQL Server.  The attendees were mostly SQL Server DBAs and all of the sessions were pretty technical from what I saw.  There were about 3000 attendees which is much smaller than the 40,000ish that were at Oracle Open World - but this conference is just for a single product, where Oracle Open World is actually the conference for Oracle DB, Oracle Apps, PeopleSoft, JD Edwards, Java, and the many others that Oracle has in it's portfolio.

I arrived on Sunday evening so that I would be there early Monday morning when the conference started.  I found out when I checked in a registration that Monday was a pre-conference day.  This meant that while there were sessions going on all day, you had to pay extra to attend them.  So I ended up working from my hotel room for the morning and then working from the very nice hotel lobby lounge in the afternoon.  The moral of the story is that SQL Pass has a pre-conference to the conference.  (There was also a post-conference day as well!)

On Tuesday I attended the opening keynote address which was very interesting.  Ted Kimmert from Microsoft talked about the next version of SQL Server - codename Denali.  Skip to about the 80% point in the keynote and watch Microsoft Distinguished Engineer Amir Netz.  Very good demo and presentation.  Amir manages to make managing a billion rows fun.

I then ended up spending the rest of the day in the VMware booth talking almost non-stop about virtualizing SQL Server.  Just about everybody was already doing some amount of virtualizing on vSphere and many had virtualized most of their SQL Server databases.  So the conversation usually centered on what SQL Server databases had not been virtualized yet.  These tended to be big and performance sensitive database instances.  This is where Storage I/O Control comes into play to be able to give these high performance databases a higher priority in terms of storage.  This is a big deal and gives a virtualization admin the ability to configure a VM to be able to support performance intensive database the storage performance it needs.

I also did quite a bit of talking about how important it is to understand exactly how the storage is configured and deployed for the SQL Server VM.  Mis-configured or under sized storage is the primary reason that I see for poor performance with SQL Server.  After some analysis of the storage configuration it usually tuns out that the SQL Server data is on the same physical disk spindles as a bunch of other stuff (file servers, web servers, dns servers, etc).  While this sharing of disks works out great in many scenarios, it can lead to a disaster with databases if there are not enough IOPS to go around.  The solution to this is to get to know your SAN and/or virtualization admins better.  Get them to understand that your storage requirements are more than just capacity in terms of gigabytes.  The number of IOPS and disk latency are important to ensuring good performance.

On Wednesday morning at 6:45am I was on panel with Brent Ozar, Donny, and Wanda He to answer questions on virtualizing SQL Server on vSphere.  Brent acted as the panel moderator and also answered most of the questions and generally did a great job (even though he was wearing a kilt!).

I spent some more time in the booth on Wednesday.  There was some discussion even at SQL Pass about Oracle change in their support stance for Oracle RAC.  I also talked to several DBAs about the perfmon counters that are added by VMtools to allow for more transparency in CPU and Memory usage for the VM from the vSphere ESX server level.  These counters give information that will indicate if they are in contention for resources with other VMs on the same host.  When having a discussion with the virtualization admin (or analyzing the performance if you are the virtualization admin) these perfmon counters are very useful.

I also went running every morning before the conference.  I ran down Pike street to the famous Pike Street Market.  One day I turned north and ran next to the water on sidewalks and pedestrian/bike lanes.  There was an incredible view of the mountains through a few clouds as the sun was coming up.  But it was kinda cold and windy running right next to the water.  So the next day I turned south at the market and ran down to the where the Seattle Seahawks play on Sundays.  Cool to see the stadium.  Even though it was cold (and just a tiny bit rainy one day) I would highly recommend running down in this area of Seattle if you get the chance.

Todd

Two Things from SQL Pass

I attended SQL Pass in Seattle last week and found the attendees to be a bunch of knowledgeable SQL Server DBAs.  While this may not be a surprising thing for me to say about THE conference for SQL Server, I did learn two things from talking and interacting with everybody I met.

One thing that didn't surprise me was that most SQL Server DBAs were already on board with running, at least some, production level SQL Server databases on VMware vSphere VMs.  What a lot of them wanted was a better understanding of the vSphere virtualized infrastructure that they were running on to be able to have more productive conversations with their vSphere and SAN administrators.  This leads to two things: special VM performance counters in the guest and Storage I/O Control (SIOC).

When VMtools is installed in a Windows VM it installs several things including a set of performance counters that are specific to that VMs performance.  These counters are populated with information from the ESX level for VM processor and VM memory usage.  These counters give an accurate look at how these resources are being used or consumed by the VM and not affected by the VM being virtualized.  They are under VM Processor and VM Memory in windows performance monitor (aka perfmon).  DBAs can use this performance information when talking to their vSphere admins to have a more productive conversation.

SIOC is a new feature with vSphere 4.1 that allows for VMs to be assigned different levels of priority to a given LUN.  It is coordinated across all of the vSphere hosts so that the priority is enforced even if the VMs are on different physical hosts.  For DBAs this means that if you have some SQL Server VMs that really need to be guaranteed higher levels of storage performance, you can use SIOC to ensure that a poorly written query on an unimportant DB doesn't take all of your I/O performance.

Todd

Support for Oracle RAC on vSphere - Removing the Barriers

I've been excited for a couple of days now.  Two days ago Oracle modified their official support stance for their products on VMware vSphere to include Oracle Real Application Clusters (RAC).  Previously RAC was the only product that Oracle had specifically called out as unsupported.  With the change of a couple of days ago - everything is supported (with a couple of caveats - like a specific version number and a few other things).

The reason this is exciting is that it removes a major barrier to the perception of vSphere as a platform for high performance enterprise applications.  Oracle RAC is a high performance cluster solution that is often used by customers for their most critical Oracle databases to ensure high availability and provide scale-out performance. The fact that RAC is now supported by Oracle running on VMware vSphere illustrates that vSphere is capable of running the most demanding and intensive applications.  Previously when a customer would hear that RAC wasn't supported on vSphere it would put plant a seed of doubt, but that is now gone.

This was the biggest unsupported statement from one of the biggest software companies and getting it reversed is a removal of a barrier to move to a more flexible, more efficient, and easily managed datacenter.

There is still plenty of work to be done to improve the support and licensing aspects of Oracle software on virtualized environments, but I think that this change with RAC support is a very positive step.

Todd

No Problem with More Cores for vSphere

I read a great blog a couple of weeks ago by Brent Ozar called SQL Server Virtualization: Mo Cores, Mo Problems.  I thought that it was cool because it talked about virtualization and performance and databases and, well that's just three of my favorite things.  Only problem was that the title made it sound like it wasn't a good idea to virtualize SQL Server on a system with lots of cores.

But that's not really what the blog says when you read it.  Brent is an expert on virtualizing SQL Server and what he actually says in the blog is a more detailed discussion.  He concludes that he prefers smaller servers over larger servers for SQL Server virtualization.  Despite the title I recommend you read his blog for the discussion.

In my opinion, which I admit is biased, the title should be something along the lines of "SQL Server Virtualization: vSphere is the best hypervisor to use when you have lots of cores because it does a good job of scheduling VMs on servers with NUMA architecture."  Of course this is a bit long, but I like it overall much better.  :-)

Todd

JDRF Walk This Sunday

On Sunday I'm going to be walking on my 9-year old's team in the Juvenile Diabetes Research Foundation (JDRF) Walk for a Cure.  Since his diagnosis about 2 months ago I have learned alot.

The short summary is that my son, and all other type-1 diabetics, have to check their blood glucose level throughout the day and inject insulin.  For him that means at least 4 injections and 6 finger pricks a day.  Some days it is much more than that.  It also means that he has to closely watch and measure everything he eats.

He has not complained.  He has not asked why me.  He has not said that it isn't fair.

The support that we have received from our friends and family have been amazing. Thank you.  We're all looking forward to the walk.

Todd

The Difficultly of Comparing Physical and Virtual Performance

I have the best car, it can go zero to sixty in under 4 seconds.

No, no my car is the best - It can carry seven people.

Wait, mine is better than both of yours - Its gets 40+ MPG.

As a performance guy at VMware I get involved in lots of discussions comparing physical and virtual performance.  What might seem like a simple thing to do can actually be quite difficult to get right, and in many cases it really is like comparing apples and oranges.

In order to get performance tests configured so that the physcial or virtual configurations match, it almost always involves using some settings that you would not normally use if you weren't doing a performance comparison.

Current generation servers commonly have more than eight cores (and the with hyperthreading its double that number for logical cores).  The maximum number of vCPUs that a VM can have today is eight.  This means that the physical system has to be somehow limited to 8 or less to enable a  comparison.  This can be done in BIOS or in the OS or via physical reconfiguration of hardware (removing processors).  It gets more complex when you have to also factor in the architecture of the server.  All new x86 based servers are using NUMA architectures where performance can be affected depending on how an application or VM is or isn't spread across the NUMA nodes.  When you limit a system to a subset of it's processors, you have to consider how this is done in respect to the NUMA nodes.

It also goes the other way as well.  The size of the VM is increased to 8 or 4 vCPUs to be able to compare the performance against physical, but the VM when run in production will be a 1 or 2 vCPU system.  The scalability of the application itself can become the gating factor if it doesn't do as well with more cores.  It is also common to run a single VM on a host for these comparisons.  A single VM on a host doesn't really make a lot of sense in most cases, so this becomes another aspect of the test that isn't reflective of how things will be run in production.

After all of this, it is possible to get a test that allows for a decent comparison if things were done carefully.  The problem is that this isn't how it would actually be run.  While these tests are great for getting a basic idea of performance differences, they don't really take into account the bigger picture of why virtualize in the first place.

It would probably be much better to run a test with your applications in an environment that is as close as possible to how you would run it in production. Base the results and analysis on end user experience or response time measurements.  I know that this is usually not practical to do, which is why we end up doing these other tests.  Just remember to take the constraints of a test into account when looking at results.

If you were buying a car to be able to carry your family of five, with luggage, on vacations every summer then your definition of performance probably won't rely too heavily on 0-60 time (Although we would all probably still want to know what that time was).

Is This Thing Still On?

I'm tapping on the microphone.  Dust is flying off.  Individual specs picked up in the light coming in from the partially open door.  A faint thud thud.  "Is this thing still on?"

After a very long break of about 3 years, I've decided to pick up blogging here again.  I was blogging only on my work blogs (Virtualization Frontier on delltechcenter.com,Virtualization Edge on VMware communities, and VROOM! on VMware.com)  for the past few years, but that has become too limited.  I'll still be posting on VROOM! which is a great blog, but for things that are less formal or not directly work related I need a different blog.

So here I'm going to continue to mostly blog about virtualization, computers, and geek stuff.  This platform will provide me with better information about traffic, much more more freedom in terms of design and layout, and more flexibility in terms of tools available to post content, than what was possible on the VMware community site.