No More Talk About Old Stuff at Oracle Open World

Oracle Open World 2011 starts next week, and I'll be there again this year (I think this will be the 7th one for me).  I've done lots of testing with Oracle  RAC on vSphere 5 over the last year and look forward to getting the chance to discuss it with customers, partners, and general database geeks.

One thing that has changed since OOW last year is that Oracle changed their support stance and RAC has been supported on vSphere since last November.  We still get some questions about support for Oracle on vSphere, but not nearly as many as last year.  If you want to discuss it or get your questions answered you can come by and talk to us at the VMware booth or you check the website.

An even bigger thing that is different is that VMware has begun to enter the database world.  With the announcement vFabric Data Director we now have the capability to add database as a service to your vSphere environment.  This changes and challenges many of the traditional notions about how to provision, manage, and use databases.  I think this is an incredibly interesting new development that is an example of how things are going to be changing in the future.

I think it's going to be a fun year at OOW with lots of the discussion revolving around virtualization and cloud  and hopefully less talk about support and licensing.

Todd

Five Things You Might Not Know About ESXTOP

I use ESXTOP almost daily in my job to do analysis of performance on ESX / vSphere.  In working with partners and customers I often hear them say that they didn't know ESXTOP could do that.  Here is list of five most popular:

1.  What is this "ESXTOP" that you speak of?  Many have just not heard of ESXTOP and either have not been concerned with performance previously or only used the information from the viClient's performance graphs.  To directly answer the question, ESXTOP or rESXTOP is a character based utility that runs on the ESX or ESXi host that can be used to monitor, display, and log virtually all the performance information of the ESX host.  You can simply enable the ability to ssh into the host from the troubleshooting options on the ESX host local console, and then ssh into the system and run "esxtop" to get started.  You can also use "resxtop" to run it remotely against an ESX host. Either way, ESXTOP is a tool often used at VMware to diagnose and identify performance problems.

2. There's more to see than just CPU data.  The initial screen for esxtop is very similiar to top from linux showing CPU usage information.  This is only a small part of what can be seen with esxtop.  By prssing different keys, different sets of data are displayed.  The "d" key will open the disk screen, "m" will show memory related information, "n" will show the virtual networking stats, "v" will show the virtual disks. To get a complete list of available screens just press "?" or "h".  You can also add more fields or columns of data by pressing the "f" key and then selecting the additional fields that you want to be displayed.

3. You can capture ESXTOP data in batch mode.  It seems that most people run esxtop top in it's interactive mode, where it displays all of it's great performance info in real time.  This is a fun way to run ESXTOP and and can give valuable insights in real time.  But it can be even more useful to be able to capture all of this data and be able to analyze it at a more detailed level later AND use it to create cool graphs.  By starting esxtop with the -b option and redirecting the output to a text file with a pipe, ESXTOP will create a CSV file with all of the performance data.

4. View ESXTOP data with windows perfmon and impress your friends and co-workers.  The CSV file that is output by ESXTOP can be imported into Windows perfmon.exe (Windows Performance Monitor).  This provides a good graphical interface to be able to quickly select counters and view them.  In order to import the file into perfmon, you will first need to copy it onto the windows system where you will be running perfmon.  You can use winscp or other utility of your choice to get it copied off the ESX host.  Start perfmon and then right click in the middle of the graph area and select Properties.  Then select the Source tab and click on the Add button.  Browse to the saved .CSV file and open it.  You will now to be able to add counters in perfmon that are from the data for your ESX host.

5. Carve out a small set of data from an ESXTOP data file with esxplot to make your job easier.  The CSV files that esxtop produces can be massively wide with thousands of columns.  Too wide for even the latest versions of Excel in many cases.  This can make it difficult to quickly pull out a specific performance counter to put into your spreadsheet for analysis or cool graph creation.  The easiest solution I know is to use the VMware labs fling esxplot to export a subset of the data.  You simply import the .CSV file from ESXTOP into esxplot, select and view data for counters in esxplot, and then when you ready export the data.  The result is a much smaller .CSV file that just has the columns of data that you want.

So now you are armed with lots of ESXTOP tips and tricks that will make it easier for you to analyze and view performance of ESX and it's VMs.   

Todd

Understanding %Ready with Monster VMs

I was doing some benchmark testing with a "Monster VM" of 24 vCPUs and 85 GB of RAM and was concerned that the %Ready CPU time for the VM was very high at 30% to 40%.  The strange thing was that performance of the VM seemed to be pretty good, with no real issues (I was of course trying to find another few percent of performance for the benchmark - but is another story).

Our guidance with %Ready has been that it should be below 5%, but this needs to be reconsidered when using large VMs.  The reason is that the %Ready you see in esxtop is the sum of %Ready for all of the individual vCPUs for that VM.  In versions prior to vSphere 5, this meant that only up to 8 vCPUs could be involved in reaching the %Ready.  With vSphere 5 and "Monster VMs" this is now up to 32 vCPUs and even small %Ready times or each individual vCPU can easily push the overall %Ready over 5%.

In general esxtop uses this addition method when showing the usage level of CPUs and it makes sense that it should be carried over to %Ready as well.  For example, when esxtop shows the CPU usage of an eight vCPU VM the maximum is 800, and for a 32 vCPU VM is is 3200.

When looking at %Ready it is important to also understand how many vCPUs are in that VM and take that into consideration.  If you expand out the stats for the VM (press "e" and then enter the GID for the VM when in esxtop) you will be able to see the %Ready for all of the individual vCPUs which should help to put things into context very quickly.

Returning to my example, it turns out that each of my 24 vCPUs was around 1 for %Ready and it did not represent a performance problem with the %Ready for the entire VM to be at 30%.

Todd

VMworld 2011 Trip Report

I'm an engineer at VMware and sometimes get to know about things that we are working on before they come out, it is great to see how it all comes together.  That happened for me this year at VMworld.

I think that Steve Herrod's keynote really did the best job of showing where this is all going.  The scenario of the new insurance claims adjuster worked really well. If you are only going to watch one session from VMworld - this would the one I would recommend.

I have a specific interest in the new vSphere Storage Appliance (VSA) and it was great to see it covered in both keynotes.  The VSA session that I presented with the VSA product manager went really well and we had some great discussion and Q and A after the session.  The official tally was 451 people and I didn't really see many leave during the session, and nobody threw anything at us!  VSA is a great feature to bring shared storage to small environments that previously couldn't afford it or lacked the technical ability to install and configure it.  A whitepaper is in the works and I will post more on VSA in the future.

The million IOPS on a single vSphere 5 hosts gots lots of coverage.  It's really cool to see the hero numbers that Chethan produces and he did a great job working with EMC on this number.  They were able to get to 1 million IOPS very quickly because they really didn't have to do any tweaking or tunning.  They just had to get the server and the storage needed to support 1 million IOPS in place, then get the VMs setup, and run through the series of tests.  No benchmark special settings required.

I was invited to speak at the Dell TechCenter Users Group meeting that happened on Tues night over at the Wynn.  I gave a 15 minute strictly technical overview of VSA and had a great time getting to see many of my twitter friends there.  I heard that there was some video of the event.

I attended sessions throughout the week, and all of them were pretty good.  There were a couple that really stood out.  Chad Sakac gave a great session on all of the new things that EMC was doing with vSphere 5 to enable new features and better performance.  He also presented part of EMC's super session and covered lots of ground including the new vCloud Data Director Database as a Service.  Chad is a great presenter and I would recommend that you catch his sessions when possible.  In particular he has this great ability to use recorded video demos and make you almost think it is a live demo.  He really has great energy and passion and it comes across in his sessions.  He practically got a standing ovation at the end (or maybe people were just trying to get to lunch, hard to tell with geeks).

The most important part of the show is getting to meet and talk with so many people from around the industry.   There were lots of people that I only see once at year - at VMworld.  It was great to see and talk to everybody. I can't wait until next year.

Todd

A Small Part in 1 Million IOPS

VMware demonstrated the impressive storage I/O capabilities of vSphere 5 recently when a single host achieved 1 MILLION IOPS.  We can do a whole bunch of IOPS - which is really cool.  This is way more IOPS than just about any workload needs and demonstrates that storage throughput is not a problem for vSphere.

I had a small part to play in this effort. I got an email in June asking if I had a large server that I could loan out for a few weeks.  It turned out that I did have a four socket server with 512GB of RAM that we had already purchased, and had shipped, but had not yet arrived.  So as soon as it arrived at our lab, I had it immediately shipped right back out to an EMC lab the same day.  They had a deadline of just a few weeks to get everything done.

Three weeks later I got an email that it was ready to ship back, I provided our address and received the 1 million IOPS server back in my lab another week later.  I'm now using this famous 1 million IOPS server for some other tests and it will reappear in future blogs and white papers.  It kinda feels like working with a rockstar.

Todd

Up to Speed with vFabric Data Director

When I read the press release announcing VMware vFabric Data Director, I wasn't sure this was what we had called project Aurora internally until I read paragraph eight.  It was at this point in the press release that we get into enough technical specifics that I could be sure.

This made me fear that the coolness of this new database from VMware would get lost.  This was most certainly was not the case.  Chad Sakac wrote  a great post giving some of the background around project Aurora and lots on exactly why this is such a powerful new capability.  Brent Ozar's post was from a completely different angle than you would expect (or maybe you would if you've seen him in his kilt at PASS) and first describes it as a new feature of Microsoft SQL Server Denali.  Scott Drummonds wrote another great post that I think puts the new VMware database into perspective for the database market.

The common thread among all of this is that the new vFabric Data Director is a new approach to using databases.  It opens up new ways to manage and use databases in a more cloud or service oriented way.  If you missed it from the initial announcement or it didn't make it onto your radar screen due to all of the other stuff going on during VMworld, a good way to get up to speed is to read these blog posts.

Todd