Beginners Guide to Copenhagen Train System

I attended VMworld Europe last week, in Copenhagen, Denmark and had a great time.  Part of this was using the excellent system of trains to get around.  In the process I learned a few key insights that I wanted to share for others that might be using the trains in Copenhagen for the first time.  If others reading this are more knowledgeable please add comments (or if you find errors let me know about that too).

There are actually three different types of trains that operate in Copenhagen.  If you get a city pass, you can ride any of them while remaining in the Copenhagen area.  The three train types are the Metro, S-Train, and Regional Trains.  The Metro is run by a different company than the S-Train and regional trains, and hence they have separate websites.

The Metro currently has two lines - M1 and M2 - which both share Vanlose as the end of the line to the west, but diverge at the Christianshaven stop to end at Vestamager for M1 and Lufthavnen for M2.

The S-Train (or S-Tog in Danish) is the other commuter train system in Copenhagen.  It has what appears to be at least seven different lines, that are color coded.  There is a big "S" sign that marks which platforms are for the S-Train.

The regional trains go to areas beyond Copenhagen and is what you take to go over into Sweden or up to Helsingor for example.  This is also what I rode from the Airport to the Kobenhaven H station, and later from Kobenhaven H to Malmo C station in Sweden.

This map is the best one I found. It shows all three trains and stations.  It's from a travel blog site.  I think that it is the best because the Metro is run by a different company than the S-Tog and regional trains.  So it took an independent 3rd party to put everything together.

You can transfer between the different train types, you just need to make sure you have the proper ticket.  They do check people for tickets and they do hand out fines to people that do not have a ticket - I saw this happen twice in the week I was in Copenhagen.

There were always very helpful people in the ticket offices and on the platforms, at the big stations, that were happy to help me understand which platform and which train to take.  It was also very nice that they all spoke English.  If you aren't sure what you are doing - just ask somebody.

Todd

Taking a Different Look at Performance for vSphere Storage Appliance

I've been working with and doing performance testing with the new vSphere Storage Appliance (VSA) for most of this year.  The white paper was published last week and I encourage you to read it if you are interested in getting a deeper understanding of VSA.

I've also done a lot of performance work with Oracle RAC, SAP, and SQL Server this year, and there is a big difference between that and VSA testing.

When testing with big databases and the applications that go along with them, the storage requirements to achieve the needed performance levels are usually pretty high.  I've used big EMC, NetApp, HP Lefthand, and Dell Equallogic storage to support thousands of IOPs while maintaining low latency.  In these tests we are analyzing the storage usage in each test to ensure that we get the performance that we need. These big storage systems are necessary to meet the requirements of the applications being tested.

VSA is designed to be a simple to install, simple to use, and simple to manage storage solution for small environments.  It can be the first "SAN" for environments where it was not possible to have a SAN before.  It is not designed to support or be used with the types of workloads at the stress levels that I normally test with.  So approaching a performance study of VSA required something different.

Instead of trying to push as many IOPS as possible or getting as many users as possible, I focused on providing some insights into the key factors of VSA performance.  This includes a couple of test scenarios designed to show two different things.

The first used the VMmark2 workload to show that enterprise apps can run with good performance on VSA.  Specifically, it showed that Exchange, a webstore front and backend, and a Java based interactive website could all run very well, at the same time, while supporting the workload equivalent of over 1000 users.

The second test used IOBlazer (a cool and easy to use IO generator tool) to show how the VSA's Network RAID (or replication) traffic impacts the performance across the cluster as load is spread across the VSA cluster. This test illustrates how as you put load one datastore the network RAID that is ensuring the data is always available causes some load to also occur on a secondary node in the VSA cluster.  The graphs are cool in this section.

I think that most customers will probably not need the additional performance information that I have included with these scenarios because of how easy VSA is to use and manage, but it will help those that are interested in getting a more advanced understanding or want to get into a more detailed level of performance monitoring.  Most will not have need for this additional detail, but many geeks like me will be interested anyway.

So this approach is a bit different from the usual type of performance work that you see.  This isn't the million IOPS test or an Oracle RAC performance study.  It's meant for a different environment.  One that is getting it's first taste of shared storage and cool things like vMotion, DRS, and HA.

Todd

VMware Was Everywhere and Nowhere At Oracle Open World 2011

There was a running joke with everybody that came by the VMware booth at Oracle Open World this year.

"Wow, Oracle really doesn't like you guys!""What did you do to get back here?"  "Are they trying to hide you?" "You guys have been literally put in the corner!"

The VMware booth was literally in the back corner of the expo floor at OOW 2011.  Some of this was our fault for deciding to go with a smaller booth this year, but I think some of it may have to do with something else.

There were still lots of customers that came by and we had tons of discussion around running Oracle products on vSphere.  This year it seemed like there were more customers who were already running some or all of their Oracle databases on vSphere which was great.  I also spent some time explaining the concepts of virtualization and specifics about vSphere to customers who were new to the ideas which was also great.

From what I heard, talking to various speakers at OOW that I know, Oracle requested that all references to VMware be removed from their presentations and replaced with generic references.  But the speakers would go ahead and say "VMware" during the talk.

The final VMware highlight for me came during Larry Ellison's keynote on Wednesday afternoon.  He announced the new Oracle Cloud and gave a lengthy demo.  While announcing the Oracle Cloud he spent a fair amount of time describing why it was good.  You could take most of the statements that he made about the Oracle Cloud and put them directly into a keynote from VMware's Paul Maritz.  The Oracle vision for cloud and the VMware vision for cloud were similar in many ways.  Larry even used Paul's line about the Hotel California for proprietary clouds - VMs can check in, but they can never leave.  Larry, of course, never directly referenced VMware in his keynote, but the similar hybrid cloud strategy was definite.

So while Oracle preferred that VMware not be mentioned or seen at OOW, in reality VMware was everywhere.

Todd

No More Talk About Old Stuff at Oracle Open World

Oracle Open World 2011 starts next week, and I'll be there again this year (I think this will be the 7th one for me).  I've done lots of testing with Oracle  RAC on vSphere 5 over the last year and look forward to getting the chance to discuss it with customers, partners, and general database geeks.

One thing that has changed since OOW last year is that Oracle changed their support stance and RAC has been supported on vSphere since last November.  We still get some questions about support for Oracle on vSphere, but not nearly as many as last year.  If you want to discuss it or get your questions answered you can come by and talk to us at the VMware booth or you check the website.

An even bigger thing that is different is that VMware has begun to enter the database world.  With the announcement vFabric Data Director we now have the capability to add database as a service to your vSphere environment.  This changes and challenges many of the traditional notions about how to provision, manage, and use databases.  I think this is an incredibly interesting new development that is an example of how things are going to be changing in the future.

I think it's going to be a fun year at OOW with lots of the discussion revolving around virtualization and cloud  and hopefully less talk about support and licensing.

Todd

Five Things You Might Not Know About ESXTOP

I use ESXTOP almost daily in my job to do analysis of performance on ESX / vSphere.  In working with partners and customers I often hear them say that they didn't know ESXTOP could do that.  Here is list of five most popular:

1.  What is this "ESXTOP" that you speak of?  Many have just not heard of ESXTOP and either have not been concerned with performance previously or only used the information from the viClient's performance graphs.  To directly answer the question, ESXTOP or rESXTOP is a character based utility that runs on the ESX or ESXi host that can be used to monitor, display, and log virtually all the performance information of the ESX host.  You can simply enable the ability to ssh into the host from the troubleshooting options on the ESX host local console, and then ssh into the system and run "esxtop" to get started.  You can also use "resxtop" to run it remotely against an ESX host. Either way, ESXTOP is a tool often used at VMware to diagnose and identify performance problems.

2. There's more to see than just CPU data.  The initial screen for esxtop is very similiar to top from linux showing CPU usage information.  This is only a small part of what can be seen with esxtop.  By prssing different keys, different sets of data are displayed.  The "d" key will open the disk screen, "m" will show memory related information, "n" will show the virtual networking stats, "v" will show the virtual disks. To get a complete list of available screens just press "?" or "h".  You can also add more fields or columns of data by pressing the "f" key and then selecting the additional fields that you want to be displayed.

3. You can capture ESXTOP data in batch mode.  It seems that most people run esxtop top in it's interactive mode, where it displays all of it's great performance info in real time.  This is a fun way to run ESXTOP and and can give valuable insights in real time.  But it can be even more useful to be able to capture all of this data and be able to analyze it at a more detailed level later AND use it to create cool graphs.  By starting esxtop with the -b option and redirecting the output to a text file with a pipe, ESXTOP will create a CSV file with all of the performance data.

4. View ESXTOP data with windows perfmon and impress your friends and co-workers.  The CSV file that is output by ESXTOP can be imported into Windows perfmon.exe (Windows Performance Monitor).  This provides a good graphical interface to be able to quickly select counters and view them.  In order to import the file into perfmon, you will first need to copy it onto the windows system where you will be running perfmon.  You can use winscp or other utility of your choice to get it copied off the ESX host.  Start perfmon and then right click in the middle of the graph area and select Properties.  Then select the Source tab and click on the Add button.  Browse to the saved .CSV file and open it.  You will now to be able to add counters in perfmon that are from the data for your ESX host.

5. Carve out a small set of data from an ESXTOP data file with esxplot to make your job easier.  The CSV files that esxtop produces can be massively wide with thousands of columns.  Too wide for even the latest versions of Excel in many cases.  This can make it difficult to quickly pull out a specific performance counter to put into your spreadsheet for analysis or cool graph creation.  The easiest solution I know is to use the VMware labs fling esxplot to export a subset of the data.  You simply import the .CSV file from ESXTOP into esxplot, select and view data for counters in esxplot, and then when you ready export the data.  The result is a much smaller .CSV file that just has the columns of data that you want.

So now you are armed with lots of ESXTOP tips and tricks that will make it easier for you to analyze and view performance of ESX and it's VMs.   

Todd

Understanding %Ready with Monster VMs

I was doing some benchmark testing with a "Monster VM" of 24 vCPUs and 85 GB of RAM and was concerned that the %Ready CPU time for the VM was very high at 30% to 40%.  The strange thing was that performance of the VM seemed to be pretty good, with no real issues (I was of course trying to find another few percent of performance for the benchmark - but is another story).

Our guidance with %Ready has been that it should be below 5%, but this needs to be reconsidered when using large VMs.  The reason is that the %Ready you see in esxtop is the sum of %Ready for all of the individual vCPUs for that VM.  In versions prior to vSphere 5, this meant that only up to 8 vCPUs could be involved in reaching the %Ready.  With vSphere 5 and "Monster VMs" this is now up to 32 vCPUs and even small %Ready times or each individual vCPU can easily push the overall %Ready over 5%.

In general esxtop uses this addition method when showing the usage level of CPUs and it makes sense that it should be carried over to %Ready as well.  For example, when esxtop shows the CPU usage of an eight vCPU VM the maximum is 800, and for a 32 vCPU VM is is 3200.

When looking at %Ready it is important to also understand how many vCPUs are in that VM and take that into consideration.  If you expand out the stats for the VM (press "e" and then enter the GID for the VM when in esxtop) you will be able to see the %Ready for all of the individual vCPUs which should help to put things into context very quickly.

Returning to my example, it turns out that each of my 24 vCPUs was around 1 for %Ready and it did not represent a performance problem with the %Ready for the entire VM to be at 30%.

Todd

VMworld 2011 Trip Report

I'm an engineer at VMware and sometimes get to know about things that we are working on before they come out, it is great to see how it all comes together.  That happened for me this year at VMworld.

I think that Steve Herrod's keynote really did the best job of showing where this is all going.  The scenario of the new insurance claims adjuster worked really well. If you are only going to watch one session from VMworld - this would the one I would recommend.

I have a specific interest in the new vSphere Storage Appliance (VSA) and it was great to see it covered in both keynotes.  The VSA session that I presented with the VSA product manager went really well and we had some great discussion and Q and A after the session.  The official tally was 451 people and I didn't really see many leave during the session, and nobody threw anything at us!  VSA is a great feature to bring shared storage to small environments that previously couldn't afford it or lacked the technical ability to install and configure it.  A whitepaper is in the works and I will post more on VSA in the future.

The million IOPS on a single vSphere 5 hosts gots lots of coverage.  It's really cool to see the hero numbers that Chethan produces and he did a great job working with EMC on this number.  They were able to get to 1 million IOPS very quickly because they really didn't have to do any tweaking or tunning.  They just had to get the server and the storage needed to support 1 million IOPS in place, then get the VMs setup, and run through the series of tests.  No benchmark special settings required.

I was invited to speak at the Dell TechCenter Users Group meeting that happened on Tues night over at the Wynn.  I gave a 15 minute strictly technical overview of VSA and had a great time getting to see many of my twitter friends there.  I heard that there was some video of the event.

I attended sessions throughout the week, and all of them were pretty good.  There were a couple that really stood out.  Chad Sakac gave a great session on all of the new things that EMC was doing with vSphere 5 to enable new features and better performance.  He also presented part of EMC's super session and covered lots of ground including the new vCloud Data Director Database as a Service.  Chad is a great presenter and I would recommend that you catch his sessions when possible.  In particular he has this great ability to use recorded video demos and make you almost think it is a live demo.  He really has great energy and passion and it comes across in his sessions.  He practically got a standing ovation at the end (or maybe people were just trying to get to lunch, hard to tell with geeks).

The most important part of the show is getting to meet and talk with so many people from around the industry.   There were lots of people that I only see once at year - at VMworld.  It was great to see and talk to everybody. I can't wait until next year.

Todd