Understanding %Ready with Monster VMs

I was doing some benchmark testing with a "Monster VM" of 24 vCPUs and 85 GB of RAM and was concerned that the %Ready CPU time for the VM was very high at 30% to 40%.  The strange thing was that performance of the VM seemed to be pretty good, with no real issues (I was of course trying to find another few percent of performance for the benchmark - but is another story).

Our guidance with %Ready has been that it should be below 5%, but this needs to be reconsidered when using large VMs.  The reason is that the %Ready you see in esxtop is the sum of %Ready for all of the individual vCPUs for that VM.  In versions prior to vSphere 5, this meant that only up to 8 vCPUs could be involved in reaching the %Ready.  With vSphere 5 and "Monster VMs" this is now up to 32 vCPUs and even small %Ready times or each individual vCPU can easily push the overall %Ready over 5%.

In general esxtop uses this addition method when showing the usage level of CPUs and it makes sense that it should be carried over to %Ready as well.  For example, when esxtop shows the CPU usage of an eight vCPU VM the maximum is 800, and for a 32 vCPU VM is is 3200.

When looking at %Ready it is important to also understand how many vCPUs are in that VM and take that into consideration.  If you expand out the stats for the VM (press "e" and then enter the GID for the VM when in esxtop) you will be able to see the %Ready for all of the individual vCPUs which should help to put things into context very quickly.

Returning to my example, it turns out that each of my 24 vCPUs was around 1 for %Ready and it did not represent a performance problem with the %Ready for the entire VM to be at 30%.

Todd

VMworld 2011 Trip Report

I'm an engineer at VMware and sometimes get to know about things that we are working on before they come out, it is great to see how it all comes together.  That happened for me this year at VMworld.

I think that Steve Herrod's keynote really did the best job of showing where this is all going.  The scenario of the new insurance claims adjuster worked really well. If you are only going to watch one session from VMworld - this would the one I would recommend.

I have a specific interest in the new vSphere Storage Appliance (VSA) and it was great to see it covered in both keynotes.  The VSA session that I presented with the VSA product manager went really well and we had some great discussion and Q and A after the session.  The official tally was 451 people and I didn't really see many leave during the session, and nobody threw anything at us!  VSA is a great feature to bring shared storage to small environments that previously couldn't afford it or lacked the technical ability to install and configure it.  A whitepaper is in the works and I will post more on VSA in the future.

The million IOPS on a single vSphere 5 hosts gots lots of coverage.  It's really cool to see the hero numbers that Chethan produces and he did a great job working with EMC on this number.  They were able to get to 1 million IOPS very quickly because they really didn't have to do any tweaking or tunning.  They just had to get the server and the storage needed to support 1 million IOPS in place, then get the VMs setup, and run through the series of tests.  No benchmark special settings required.

I was invited to speak at the Dell TechCenter Users Group meeting that happened on Tues night over at the Wynn.  I gave a 15 minute strictly technical overview of VSA and had a great time getting to see many of my twitter friends there.  I heard that there was some video of the event.

I attended sessions throughout the week, and all of them were pretty good.  There were a couple that really stood out.  Chad Sakac gave a great session on all of the new things that EMC was doing with vSphere 5 to enable new features and better performance.  He also presented part of EMC's super session and covered lots of ground including the new vCloud Data Director Database as a Service.  Chad is a great presenter and I would recommend that you catch his sessions when possible.  In particular he has this great ability to use recorded video demos and make you almost think it is a live demo.  He really has great energy and passion and it comes across in his sessions.  He practically got a standing ovation at the end (or maybe people were just trying to get to lunch, hard to tell with geeks).

The most important part of the show is getting to meet and talk with so many people from around the industry.   There were lots of people that I only see once at year - at VMworld.  It was great to see and talk to everybody. I can't wait until next year.

Todd

A Small Part in 1 Million IOPS

VMware demonstrated the impressive storage I/O capabilities of vSphere 5 recently when a single host achieved 1 MILLION IOPS.  We can do a whole bunch of IOPS - which is really cool.  This is way more IOPS than just about any workload needs and demonstrates that storage throughput is not a problem for vSphere.

I had a small part to play in this effort. I got an email in June asking if I had a large server that I could loan out for a few weeks.  It turned out that I did have a four socket server with 512GB of RAM that we had already purchased, and had shipped, but had not yet arrived.  So as soon as it arrived at our lab, I had it immediately shipped right back out to an EMC lab the same day.  They had a deadline of just a few weeks to get everything done.

Three weeks later I got an email that it was ready to ship back, I provided our address and received the 1 million IOPS server back in my lab another week later.  I'm now using this famous 1 million IOPS server for some other tests and it will reappear in future blogs and white papers.  It kinda feels like working with a rockstar.

Todd

Up to Speed with vFabric Data Director

When I read the press release announcing VMware vFabric Data Director, I wasn't sure this was what we had called project Aurora internally until I read paragraph eight.  It was at this point in the press release that we get into enough technical specifics that I could be sure.

This made me fear that the coolness of this new database from VMware would get lost.  This was most certainly was not the case.  Chad Sakac wrote  a great post giving some of the background around project Aurora and lots on exactly why this is such a powerful new capability.  Brent Ozar's post was from a completely different angle than you would expect (or maybe you would if you've seen him in his kilt at PASS) and first describes it as a new feature of Microsoft SQL Server Denali.  Scott Drummonds wrote another great post that I think puts the new VMware database into perspective for the database market.

The common thread among all of this is that the new vFabric Data Director is a new approach to using databases.  It opens up new ways to manage and use databases in a more cloud or service oriented way.  If you missed it from the initial announcement or it didn't make it onto your radar screen due to all of the other stuff going on during VMworld, a good way to get up to speed is to read these blog posts.

Todd

Getting Started with VSA

If you are going to be at VMworld this year, please come to the session I'm presenting with Edward Hsu on VSA.  We are to cover some basics and some in-depth goodies.  If you thinking about using VSA, this session will help you to understand exactly what VSA is and some details to help you understand how to deploy and manage it.  It is on Monday at 12:30 in Palazzo Ballroom G - VSP3299.

Here's a very fast overview of VSA:  It stands for vSphere Storage Appliance.  It takes the local disks on two or three vSphere 5 hosts and creates a "virtual SAN".  This virtual SAN enables all the advanced featrues of vSphere - vMotion, DRS, HA - to be possible with only local storage.

I've done lots of testing with it over the past four months and now that the product is finished, I can start talking about.  This session at VMworld is the starting point and it will be followed by a whitepaper and several blog articles.

Todd

HP TouchPad Conspiracy Theory #1

I've always wanted to be the author of a conspiracy theory and I think that HP has given me the perfect opportunity.  The whole TouchPad / WebOS apparent dismantling is just a trick.  HP has actually just pulled off one of the greatest stunts of all times.  What follows is my theory of what is really going on.

HP acquired Palm and WebOS last year and began to work on a strategy to create devices that were competitive with Apple's iPhone and iPad.  The WebOS is a technically superior OS to iOS in many ways, which is an advantage, but would not guarantee success simply due to technical superiority.  HP also had the ability to basically copy the same general tablet form factor and technical specs that all the other tablets were using.  In basic terms HP was able to create a device that was similar technically to the iPad and in some ways even better.

HP realized that they also had to compete with the Apple appstore and development community.  This is not something that can be easily solved, so a few high placed executives must of come up with the secret plan that we now see unfolding.

They planned to release the Touchpad as a high end tablet that "Worked Like Nothing Else" with celebrity spokespeople and lots of ads.  This would give consumers the impression of the high value of their new tablet.    In order to spur demand and quickly sell ALL the Touchpads they could make, they would then slash prices to $99 and announce that they were discontinuing the Touchpad.  All Touchpads would sell out in days.

This part of the secret plan has been executed already, and we have seen that it was successful.  The next steps should come in the next few days.

It is my theory that HP's secret plan next calls for HP to announce that in the wake of unprecedented demand and the need to support all of it's new cusomers - They are going to keep working on tablets and WebOS.

Maybe this is just a crazy consipiracy theory, but I thnk that it makes sense.  We'll all just have to wait and see what happens over the next couple of weeks :)

What Are You Waiting For?

I'm feeling bold today, so here it is. There are no remaining technical barriers to virtualizing your entire data-center.  The benefits are real.  What are you waiting for?

Many many customers have virtualized all the most business critical and performance intensive apps found in datacenters.  Exchange is now commonly virtualized.  SQL Server and Oracle databases are also run as VMs on vSphere in data centers all over the world.  Even bigger more complex stuff like SAP and Oracle RAC can be run virtualized with no technical issues.  There are proven examples for just about every application type out there.  It works.

The benefits are big and from what I have seen, the more that you virtualize the bigger the benefits become.  I think that everybody is aware of the clear consolidation benefit for legacy and underused servers.  Once you have completed a consolidation effort you begin to see that the operational benefits are much bigger than what you realized before starting.  In fact I would say that these benefits are much bigger than the pure consolidation gains.

The number of workloads that cannot be virtualized due to technical barriers is very small.  This leaves issues like application licensing uncertainty, organizational politics, out dated conceptions of what virtualization can do, support questions, fear of change, and others.  Some of these have simple answers and others are more difficult, but even though they are not technical reasons they are still very real reasons.

If you were to purchase a new Intel or AMD based server today with either two or four sockets, you would be hard pressed to find an application that truly needs the whole host.  The only practical way to really utilize these nice new servers is to use virtualization to be able to run multiple workloads on it at the same time.  Otherwise you are looking at a system that is probably less than 30% utilized almost all of the time.

Todd