Saturday, 5 April 2014

vCenter 5.5 U1 vs Server 2012 Upgrade to R2 MU1

Thought I'd snapshot my Lab's vCenter running 5.5 U1 and upgrade the O/S from 2012 to 2012 R2 MU1 which was just released. Don't try this on Production folks!

I took a snapshot and kicked off the upgrade. After the final reboot, all the services started up except the vSphere Web Client. I tried the C# Client but that wouldn't connect either. I turned off the Firewall to see if that helped as the main vCenter service had started and I still got a connection failure. Then I noticed that the IP Address had changed, it was using DHCP. I changed back to the original static address, enabled the Firewall again and rebooted. Now the C# Client worked but the web client service was still not responding.

I did a repair install (remove/install) of the Web Client to see if that would help. This got the Web Client Service into a running state finally and after a few minutes I could connect via the Web Client Interface.

I had the same trouble as this with 5.5.0b and was hoping 5.5 U1 would get around it but obviously not. For Production Servers you have the choice of trying the steps above with good backups in place (!) or build a fresh vCenter on 2012 R2 MU1 and migrate the Database, SSL keys etc.

Note: By MU1 I mean the maintenance update released by Microsoft for Server 2012 R2 in April 2014. There is a set of 6 patches for existing 2012 R2 systems or you can give it the integrated ISO which is the method I used above to upgrade from 2012.

Thursday, 3 April 2014

vSphere 5.5 observations

Well, as I mentioned in another post, I've finished a vSphere 5.5 deployment. Just thought I'd jot down a few observations. Disclaimer: these are based on my own opinion and do not represent the views of any Company I've worked for past, present or future.

I found the new web interface a little daunting at first, I tried to do everything possible with the it rather than drop back to the C# client.So in no particular order:

The refresh rate is a little slow which is annoying, when you power off a VM it doesn't instantly update and I find myself forcing a refresh several times a day.

Drag and drop works nicely but you have to wait until it's finished as it removes the object temporarily and then puts it back which can screw up your next selection!

Great working with the new SSO, no Database required, didn't give any trouble at all.

Tried using the VCSA but it gave a LOT of trouble so after two rebuilds I called time and reverted to the Windows version. It's just not ready for the big time outside of Labs / Test / Development, again, my opinion only. The web client was more unstable with the VCSA than with the windows one.

The Notes field doesn't show custom attribute data so it's either input what the VM does or have Veeam input the latest backup job information. Tags don't cut it for dynamic data like that. You can use Custom Attributes but the web client won't display them. You can get a community written plugin that does however.

Use the latest vCenter build you can get provided your plugins work with it. There's many, many bugfixes in the newer versions, especially over the last year and multiple vCenter releases. Bit concerning but at least they are trying to address it. One outstanding issue is you can't deploy OVFs with the web client, you have to drop back to the C# Client.

Tip: Deploy vCenter and SQL VMs with Hardware version 9 and keep a standard switch handy with no physical uplinks to allow you to restore vCenter back onto the network where you are using Distributed switches exclusively. Remember the C# Client can't edit VMs with a hardware version of 10, hope your PowerCLI skills are good in that case when vCenter is down and you've to restore from backup!

The console plugin doesn't launch great in Firefox, it stalls the browser completely for about 10 seconds. When doing testing using multiple VMs, drop back to the C# Client, console is a little better there. Or try a different browser?!

All the third party plugins are pretty Cool, I've used HP Insight Control v 7.3 (it Rocks! Especially for Storage visibility), Veeam is good also - last 7 days of backup history on tap.

Sometimes you right click on a VM and the options you expect to be available aren't. Try the Action menu at the top instead. Annoying especially when it's due to a refresh lag.

Filtering Advanced ESXi Host settings is very handy, looking for words like "Scratch", "BPDU" or "PDL" makes it very easy to configure or check settings.

Not using Tags to date but found Datastore Clusters interesting. When they overlap with Storage features however they are less useful.

Still not getting LLDP exposed through HP Virtual Connect but one day.....

Don't forget to backup your DVS configuration! Handy feature but could be placed better or allow automation?





Veeam & 3PAR Integration

I've recently finished a VMware 5.5 Deployment that included HP Blades, 3PAR 7400 and Veeam Backup & Replication 7.0. I was curious how this would work from a SAN backup point of view and I got good experience configuring the integration. One thing I found lacking was a good guide from either Veeam or HP to tie these together. I knew in Theory what needed to happen but it's not listed anywhere I could find so here's some pointers and how I got it to work.

I'm using Datastore based backups, i.e. 1 Veeam Backup job per Datastore so I'm not taking multiple 3PAR snapshots of each Datastore as it sifts through the grouping of VMs. The solution has two 3PAR 7400's, a virtual Veeam Enterprise Manager, two physical Veeam Proxy Servers all running 7.0 with Patch 3. The Veeam Proxy Servers have 10Gb Nics and 8Gb Fibre. The backup storage is iSCSI, the Fibre is used to connect to the 3PAR Storage where the VMs reside and a Tape Library for good measure. So far, so fun, let's see where configuration is needed:
  • SAN Switch Zoning - I examined one of the existing Blade Server Zones and used that as a template. There are two SAN fabrics, each needed two Zones set up for each Proxy Server. Proxy Server HBA #1 is zoned with one SAN Controller and then separately to the second SAN's Controller. Then HBA #2 then zoned to the same way to the other controller on each SAN. 
  • 3PAR - The Proxy Server can now log in to the SAN Fabric and you should see the WWIDs appear in 3PAR. Define a Host using both sets of WWIDs. You need to do this on each 3PAR.
  • Double check the Proxy O/S is not set to automount. Veeam disables this by default but recheck.
  • Now you can configure a test backup job, you need Veeam B&R Enterprise Plus to do this remember, check your license. Enable SAN Based Backups on the test job.
During the Backup window you'll see temporary snapshot volumes being presented to the Proxy Server in Disk Manager, these disappear after a time. At no stage do you need to export the live Production LUNs used for VMware to the proxy server. 3PAR snapshots the LUN and presents this so it's a safe configuration and fast.

Restores have to be carried out over the network. I've seen backups processing 100's of MB/s but restores typically are in the region of 50 MB/s even with 10Gb network speeds. So a 200GB VM could take 2 hours to restore but that's typical of most backup / restore comparisons (it takes twice as long to restore than to backup).

I had one issue where some Datastores would use SAN snapshots and others wouldn't. When I spent some time looking at it I found it was working for SAN #1 but not SAN #2. Turned out I had defined the Host on SAN #1 but forgotten to do it also on SAN #2. Easy fix!

Get your SAN administrator to do the Zoning & 3PAR piece if you're not comfortable as it can be tricky. The Server zoning was already done by an expert so I just copied their naming convention and config to extend it to the Proxy Servers which made it handy!

Tape Integration is next, so far so good, basic but it works nicely. I tested instant restore also which worked like a charm and used Storage vMotion to relocate the VM afterwards.

I really like Veeam, they deliver great software and it's very usable from an Engineer's point of view. Probably one of the last product out there I can say that about! 

Friday, 4 October 2013

Securing vCenter to SQL communication - event id 36870

Hi,

Well, I'm rebuilding my Lab currently using Server 2012. This gives me a chance to use SQL 2012 and vCenter 5.5 (Note to self: 2012 R2 is fine EXCEPT for vCenter! It gets stuck at "Installing Directory Services" and is not supported anyway). One thing I'd read about was configuring SQL communication encryption (as opposed to Database encryption) and I wanted to use that. I held off installing vCenter until I could get the certificate installed and SQL services running but ran into a roadblock every time I applied the Certificate and started the SQL Server Service:

I thought it was my Certificate but I'd generated it with Openssl and the AD certificate services just like the other VMware Certs I'd deployed previously. I came back at the issue a week later and found this post which solved my problem:
http://sqlblog.com/blogs/greg_low/archive/2013/05/30/sql-server-service-won-t-start-after-changing-service-account-service-specific-error-2146885628.aspx
Now I'm no SQL guru but a permission problem wasn't what I expected. What I did remember was I tried using Managed Service Accounts to test how these work and that was the account the SQL server service was using and subsequently didn't have sufficient permissions.
Here is SQL configuration manager showing the accounts I'm running the services under:
And here is the Certificate I had previously selected:
So once I had changed the permissions on the folder C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys I could start the SQL server service just fine. I did get an error applying permissions for the msvc_Labsql55 account at the folder level, I had 2 files in the folder, one would not let me edit permissions but the other accepted the update fine. I think one of the files corresponds to my SSL cert but the other is a system key and shouldn't be meddled with! Test in a Lab yourself to make sure!
So now I can create an ODBC connection for the vCenter 5.5 Database and tick the box to encrypt the connection:
 
And it tests successfully:
Job Done! Now I just have to get building!
 
 
 
 
 




Saturday, 14 September 2013

SRM Lab - No Array Pair in Protection Group

Well, this was a weird one, I spent a day labbing an SRM configuration and after fighting my way through certificates and domain controller issues I ended up not being able to create a protection group. This taken from the SRM Logs:

2013-09-06T13:30:09.385+01:00 [05420 warning 'DatastoreGroupManager' opID=33174abf] No read-write devices found for array pair 'array-pair-2038'
2013-09-06T13:30:09.385+01:00 [05420 verbose 'DatastoreGroupManager' opID=33174abf] Recomputed datastore groups for array pair 'array-pair-2038': 0 replicated datastores, 0 replicated RDMs, 0 free devices, 0 datastore groups

It's just not detecting anything it can work with even though everything looks fine and is replicating at the VSA end...!

I got talking to a great guy in HP Support about it and he came back and said to check the IQN for upper case characters. In the HP StoreVirtual Centralized Management Console I had added the VSA's using upper case characters and this was reflected in the IQN seen in VMware.

I unexported the volumes, deleted the servers in the HP console and recreated them, refreshed the ESX iSCSI config and rebooted the nested ESXi hosts for good measure. The screen shots below show what the configuration I got working looks like and how it SHOULD have been configured!

 
There should be no upper case in the Name shown below:
 
 
Then after rescanning the SRA and Devices I was able to create my first protection group (My issue was I was shown NO array pairs to choose from):


So that's one I'll remember! Thanks to HP Support for spotting my mistake!

Mike

Friday, 6 September 2013

Upgrading DC to 2012 throws VMware SSO a loop

Hi Folks,

Well, I decided to prep my Lab for vSphere 5.5 and get server 2012 and sql 2012 out there so I can have fun figuring out where everything is moved to! I just started with my single Lab DC which was on Server 2008 R2 and installed a second DC with Server 2012. Darn, I just remembered 2012 R2 is around the corner, could've waited, oh well! I know there are some weird things about changing the name of your DC affecting vCenter and you've to edit the database but I wanted to basically replace my Lab DC with a new one and end up keeping the same IP and Name.

Note: None of this may be supported so follow at your own risk, this is my Lab, not a Production environment!

My problem was when I booted up my full lab and the new DC was running away fine with the old name and IP, the vCenter Service on my two linked mode vCenter VMs would not start. The vpxd.log showed the following errors:

2013-09-06T09:11:58.472+01:00 [04896 warning 'Default'] Warning, existence of group "LAB\Domain Admins" unknown, permission may not be effective until it is resolved.
2013-09-06T09:11:58.534+01:00 [04896 error 'Default'] The group account "LAB\Domain Admins" could not be successfully resolved.  Check network connectivity to domain controllers and domain membership.  Users may not be able to log in until connectivity is restored.

I also got netlogon errors in the system event logs. To start with I thought the vCenter VMs were having some difficulty talking to the new DC even though it is using the same name and IP address. I use netdom to rejoin them to the domain:

netdom.exe remove IEDUBDC1VC01 /Domain:lab.local /userd:lab\administrator /passwordd:XXXXXXX

netdom.exe join IEDUBDC1VC01 /Domain:lab.local /userd:lab\administrator /passwordd:XXXXXXX

I was able to enter both commands one after the other and then reboot saving some time.
This fixed the logon errors but the vCenter service still had the same issue.

So maybe SSO was the issue - the errors above were wrapped by SSO entries:
[SSO][SsoAdminFacadeImpl]'] [FindGroup]

I found a great post here that would deal with the issue:
http://www.gabesvirtualworld.com/vcenter-sso-changes-when-demoting-domain-controller/
except when I went to launch the web client this is what I got:
 
Catch 22?!! Looked like it. I knew SSO had command line options but I'd never used them. I was able to list the current SSO configuration and from there tried removing and re-adding my Domain source, once I'd done that vCenter service started up fine. Here are the commands:
Note: the account to use appears to be just "admin", I'd tried admin@System-Domain and other variations before I figured this out!
 
This lists the current configuration:
 
C:\Program Files\VMware\Infrastructure\SSOServer\utils>rsautil manage-identity-sources -a list
Super Administrator's name: admin
Super Administrator's Password: ********
External identity sources:
Name: lab.local
ID: ims.8dd6788d2b0aa8c05a7226e3edcef61b
Type: Active Directory
Primary URL: ldaps://LABDC.lab.local:3269
Failover URL:
Domain: lab.local
Alias: LAB
Principal Base DN: DC=lab,DC=local
Group Base DN: DC=lab,DC=local
Successfully executed action: 'list'
 
This removes the current configuration, copy and paste the ID from the output above:
 
C:\Program Files\VMware\Infrastructure\SSOServer\utils>rsautil manage-identity-sources -a delete
Super Administrator's name: admin
Super Administrator's Password: ********
ID of the identity source to delete. Use list action to retrieve the ID: ims.8dd
6788d2b0aa8c05a7226e3edcef61b
Successfully executed action: 'delete'
 
This one adds back the Domain information, I've used the domain rather than specify a particular domain controller:
 
C:\Program Files\VMware\Infrastructure\SSOServer\utils>rsautil manage-identity-sources -a create
Super Administrator's name: admin
Super Administrator's Password: ********
Primary URL: ldap://lab.local
Secondary failover URL:
Domain name: lab.local
Domain alias: LAB
Base DN for users: cn=Users,dc=lab,dc=local
Base DN for groups: cn=Users,dc=lab,dc=local
LDAP administrative account user (read operations): lab\administrator
LDAP administrative account password: ********
Identity Source added. ID: ims.32fa4eb12b0aa8c0073ec51929b46c33
Successfully executed action: 'create'
 
This confirms the new settings:
 
C:\Program Files\VMware\Infrastructure\SSOServer\utils>rsautil manage-identity-sources -a list
Super Administrator's name: admin
Super Administrator's Password: ********
External identity sources:
Name: lab.local
ID: ims.32fa4eb12b0aa8c0073ec51929b46c33
Type: Active Directory
Primary URL: ldap://lab.local
Failover URL:
Domain: lab.local
Alias: LAB
Principal Base DN: cn=Users,dc=lab,dc=local
Group Base DN: cn=Users,dc=lab,dc=local
Successfully executed action: 'list'
 
vCenter functioned perfectly at this point and the service loaded up fine. I had to manually reselect the Update Manager service account and re-enter the password but then that service worked fine also, even before the SSO steps above.
 
I'm looking forward to the new overhaul of SSO in 5.5, hoping it makes life a little easier but we'll see! This might help those of you stuck with a Wintel team changing AD on you without notification and the next time you bounce vCenter you're stuck! Make sure you warn them about how VMware components are tied to AD intimately so they give you plenty of time to plan ahead and test after the upgrade.
 



Thursday, 20 June 2013

Snapshot Conundrum
VMware have a KB article dealing with understanding Snapshots:
In it they provide a diagram that shows how a VM with 3 snapshots against it operates. I've copied the diagram below.

I just completed reading a VMware book and they made a statement which tweaked my curiosity. "If you have three snapshots, any new data is written to all three". This conflicted with what I had learned and assumed from my VCP days. The diagram above appears to support this statement (apart from the VM write to the Parent Disk which the Authors have issued an errata for) but the KB article itself makes no similar assertion.

My interest was around performance. If I create multiple snapshots how much does it degrade performance? I understand that Reads may have to traverse the whole string of snapshots back to the Parent disk to retrieve a file and ensure its the most up to date version, but writes?

My own understanding was Writes only happen to the active Snapshot disk, the others are essentially frozen. Being confused by this contradiction I wondered if it would be possible to test this is my lab? If I could use ESXTOP to view the disk statistics for the parent file and each snapshot disk I could tell straightaway where the reads and writes were occurring. Easy? Goes to show I need to use ESXTOP more!

The disk statistics in VMware's ESXTOP aggregates all disks for the VM under one heading. Even if stored on different Datastores. This means I can't see "inside" each disk file to tell where the writes are occurring based on performance counters in ESXi. I can see individual disk latency in the ViClient but nothing more granular than that.
So, maybe we can use the modified date & timestamp and view the disk size after a change to determine where the writes are happening. We can view the directory in Putty and use the "ls -alh" to see what the directory looks like for a single VM with one virtual disk and 3 snapshots against it.

This is the VM's folder view in Putty with no snapshots, our baseline:
This is the VM's folder view in Putty after 3 Snapshots have been created, a few minutes apart:
Now, we're interested in which disk gets modified and grows if we drop a file into the VM. I'm going to copy the vSphere Client, a 110MB file onto the C: drive and run the ls -alh command again to see what happens. We will either see the 000003 disk modified time change and the size grow accordingly, or we'll see all three snapshot disks change the same way. The results are as follows:
And the result? Only the most recent Snapshot disk in the chain is affected by the file copy. The Modified Time and Size adjust appropriately but there is no associated change to the previous snapshots in the chain. The VM Parent Disk "Snapshot-flat.vmdk" is also unaffected. 

[Edit 21/6/13: Note the Snapshot file grows in 16MB chunks, hence the size is multiples of 16MB and may be larger than the file copied into the VM - Thanks to VMware's Rick Blyths Post on Snapshots for that nugget!]

A ViClient Datastore Browser view of the same thing is here:

I must admit I love reading something which challenges your preconceptions and motivates you to investigate. I found the rest of the book a great read, almost too short as the case studies and experiences relayed are rarely found outside Blogs and Live events. As writing a book and getting it past editors and published is a significant achievement I've nothing but respect for the Authors. I'm going to deliberately hold off naming anyone as my interest is from a technology perspective but if the Authors leave a comment on my Blog I'm more than happy to publish it.

So, that's it for my very first blog post! I'll be posting again as soon as I find interesting topics to share. If you have any suggestions on the article above, can suggest alternative tests or spot obvious faults in my methodology please let me know. I'm still learning!

Mike

[Update: 21/6/13] Well, I found a way to monitor which snapshot files are being touched when a write occurs. How did I do it? Windows Server 2012! I enabled NFS and was able to view the contents of the folder with Windows Explorer and, more importantly, view the individual file writes with Resource Monitor. I copied the ViClient as before and then copied the vCenter install folder (1.2GB) to get a longer reading and captured the results. This is the Windows Folder "E:\NFS\Snapshot" Explorer view after both file copies:
This is the output from Windows Resource Monitor, Disk View, scoped to the System process. The NFS Root Share was created in E:\NFS and the VM name was called "Snapshot" as shown in the path below:
 
So that's it, Myth-Busted as they say! There are NO writes to the Parent or first two snapshot files, only the most recent Snapshot file (000003). If I do a command line directory read of the Windows Folder files & subdirectories (dir /s) which should all come from the parent I get the following result:
The Read traverses through the third and first snapshots before hitting the Parent disk where the files are. So there is a read overhead but no write overhead. Again, the only writes occur on the 00003 Snapshot disk. 

Note: I initially tried Windows 2012 iSCSI first but it uses a VHD file which you can't get inside with Resource Monitor. It had been a while since I'd attempted to use UNIX services for Windows, and I'm glad its come a long way since then! Now that's over I can get some sleep!