VMware Snapshots

Friday, 5 August 2016

vSphere Upgrade 5.0 to 6.0 - Part 6

So, back to our test install. I've deployed two VMs as follows:

testpsc.lab.local - Platform Services Controller 6.0 U2 (192.168.10.105 - you use this later)
testvc2.lab.local - vCenter 6.0 U2 (192.168.10.104)

I've installed Update Manager and confirmed the self signed certificates are in place. When using the legacy C# client you'll see the warning below, click Ignore for now. Once we correctly replace these certs we should get no warning which is a good sign we've been successful!

Back to our original article:

Understanding and using vSphere 6.0 Certificate Manager (2097936)
https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2097936

So this will be somewhat similar to my last post but in a distributed setup the steps are a little trickier! I'm going to take a powered down snapshot of my two VMs and I'd strongly suggest you do the same as I've found running the certificate utility multiple times breaks something and I've been unable to fix it - basically you can try any option and NO services are updated anymore. There's some cleanup step I'm probably missing. I'd also recommend setting up a Lab for this if you get time as a typo could cost you a day!

The Certificate Manager interface

I edit the command prompt by launching it separately here, right click on the "start button":

I've chosen to use Width 150 & Height 900:

Edit the certool.cfg file to preload some defaults:
C:\Program Files\VMware\vCenter Server\vmcad\certool.cfg

I've setup mine as follows for the testpsc VM:

#
# Template file for a CSR request
#

# Country is needed and has to be 2 characters
Country = IE
Name = vmca-editme
Organization = Lab
OrgUnit = IT
State = Leinster
Locality = Dublin
IPAddress =
Email = administrator@lab.local
Hostname = testpsc.lab.local

And the following for the testvc2 VM:

#
# Template file for a CSR request
#

# Country is needed and has to be 2 characters
Country = IE
Name = vmca-editme
Organization = Lab
OrgUnit = IT
State = Leinster
Locality = Dublin
IPAddress =
Email = administrator@lab.local
Hostname = testvc2.lab.local

Now launch the utility.
Note: Run the utility as administrator (update drive path to utility as required):
"C:\Program Files\VMware\vCenter Server\vmcad\certificate-manager.bat"

Now you might well ask from which VM do you run this?!! We want to execute option 2 and if we try this from the vCenter VM we'll get the following warning:

The VMCA is on the Platform Services Controller VM testpsc.lab.local and that's where we need to replace the self-signed root certificate with an intermediate certificate issued by our Lab / Production CA. So far so squeezy:

It then asks "do you wish to generate all certificates using configuration file" - say yes to this. We now get to input the options for each of the certificates it will generate, they have to be unique in some way however, so I've used the following Name and Hostname variations:

On testpsc.lab.local (Option 2):

MACHINE_SSL_CERT.cfg - vmca-machine-ssl - testpsc.lab.local

machine.cfg - vmca-machine - testpsc.lab.local

vsphere-webclient.cfg - vmca-webclient - testpsc.lab.local

certool.cfg - vmca-certool - testpsc.lab.local

Once this completes on vCenter VM testvc2.lab.local execute the following:

"C:\Program Files\VMware\vCenter Server\bin\service-control" --stop --all

"C:\Program Files\VMware\vCenter Server\bin\service-control" --start --all

Now switch VMs from the PSC to vCenter and run the Certificate Management Tool there and execute the following options:

On testvc2.lab.local (Option 3):
(Don't forget to edit the certool.cfg defaults!)

MACHINE_SSL_CERT.cfg - vmca-vc-machine-ssl - testvc2.lab.local

On testvc2.lab.local (Option 6):

machine.cfg - vmca-vc-machine - testvc2.lab.local

vsphere-webclient.cfg - vmca-vc-webclient - testvc2.lab.local

vpxd.cfg - vmca-vc-vpxd - testvc2.lab.local

vpxd-extension.cfg - vmca-vc-vpxd-ext - testvc2.lab.local

I tried duplicating the values in the Name field between the PSC 7 vCenter responses as the hostnames are different. However, I got an error loading the web interface, the logs reported "Error occurred looking for solution user : More than one solution user found". Stick to unique Name fields above to avoid this problem!

The error is covered in this KB:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2144086

"To avoid this issue, ensure the config files are unique for each solution user by modifying the Name.

For example, use vSphere-vpxd, vSphere-vpxd-extension, etc, instead of vSphere for each config file."

Just stick to the Names I've listed above and you will be find. I reverted to my snapshot after encountering the error and once I use the names above I was fine again and the web interface worked ok.

Also, I've dumped a full copy of the Certificate Manager output at the bottom of this post to avoid making things too boring! You can compare your output against it there. In that example I've used unique Name & hostname values as listed above. If you get the error above, rerun the Certificate Manager on the vCenter Server and options 3 & 6 once more and the problem will resolve itself or revert to snapshot if none of the services update anymore (until I figure out why someday!).

After doing this more than once but backing out you may need to do a clearout:

C:\ProgramData\VMware\vCenterServer\runtime\tmp\vmware

Make sure you clear out all the file from this folder except the "cis-license" subfolder - it's important to make sure you remove the old_machine_ssl.crt file in addition to the cfg files to reset the Certificate Manager again. This isn't always enough however so revert to snapshot if the services stop updating.

Note: If using Firefox which maintains it's own Root CA list, be aware that the Active Directory one isn't referenced. You can import the chain.cer into Firefox directly and restart the browser to eliminate the error you get otherwise:

PSC Certificate Utility

VMware vSphere 6.0 Update 1 brought us a web interface for the PSC. The features are covered here:

https://blogs.vmware.com/vsphere/2015/09/introducing-the-platform-services-controller-interface-in-vcenter-server-6-0-update-1.html

So, here is the web interface on the PSC to manage certificates (there since 6.0 U1):

I checked both of my PSCs and they are using different root certificates so that means whatever steps I find work will need to be carried out on both PSCs and they in turn will issue certs for the other vCenter components and ESXi hosts in their respective sites.

Note: be careful of PSC browser bookmarks as they often capture SAMLRequest guids which are invalid when reused later - just edit the bookmark and cut off anything after /psc/

You appear to be able to replace the Root Certificate but you need to manually set one up and upload it as shown below. However when I tried this it did nothing, no errors, zip. Maybe I need to restart services or something but I've not found any resources for this approach online so gave up in the end!

Please ignore this section as it's a completely worthless waste of both of our time!

VMCA Subordinate Certificate

The vCenter Web Interface now has a trusted certificate subordinated to my Lab CA, see the full chain below.

As before replace the Update Manager Certificate and you'll also need to update SRM but you're otherwise done! These certs need to be manually generated the old way, they will reference the CA Root directly threfore, the subordinate cert doesn't extend to these solutions at this time.

Troubleshooting

The main trouble I encountered is after trying to replace the PSC certs using option 2 and it updates 0 services which is not good! I tried cleaning down the tmp folder: C:\ProgramData\VMware\vCenterServer\runtime\tmp\vmware

and using option 4 to regenerate the VMCA Root Certificate, and tried option 8 to reset all certificates but all fail to resolve this issue, it ends up stating "Updated 0 service(s)". This might be down to a faulty old ssl certificate thumbprint which when it can't match, it can't replace it. I've found some detail below:

Validating the Trust Anchors is covered in the following article but it's meant to be fixed in vCenter 6.0 U1:

https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2121701

My advice - log a support call! (Or revert to snapshot, or practice in a lab first!)

Certificate Manager Output

This is the full command prompt dump from VM Testpsc.lab.local:

C:\Windows\system32>"C:\Program Files\VMware\vCenter Server\vmcad\certificate-manager.bat"
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.0 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-Z and hit Enter to exit.
Option[1 to 8]: 2
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : y

Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:

Please configure MACHINE_SSL_CERT.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-testpsc] : vmca-machine-ssl

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testpsc.lab.local

Please configure machine.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-testpsc] : vmca-machine

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local

Please configure vsphere-webclient.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-testpsc] : vmca-webclient

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local
1. Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate

2. Import custom certificate(s) and key(s) to replace existing VMCA Root Signing certificate

Option [1 or 2]: 1

Please provide a directory location to write the CSR(s) and PrivateKey(s) to:
Output directory path: c:\temp

Please configure certool.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-testpsc] : vmca-certool

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testpsc.lab.local
2016-08-02T10:25:37.548Z Running command: ['C:\\Program Files\\VMware\\vCenter Server\\vmcad\\certool.exe', '--genkey', '--privkey', 'c:\\temp\\vmca
_issued_key.key', '--pubkey', 'c:\\users\\admini~1.lab\\appdata\\local\\temp\\2\\pubkey.pub']
2016-08-02T10:25:37.918Z Done running command
2016-08-02T10:25:37.920Z Running command: ['C:\\Program Files\\VMware\\vCenter Server\\vmcad\\certool.exe', '--gencsr', '--privkey', 'c:\\temp\\vmca
_issued_key.key', '--pubkey', 'c:\\users\\admini~1.lab\\appdata\\local\\temp\\2\\pubkey.pub', '--config', 'C:\\ProgramData\\VMware\\vCenterServer\\run
time\\tmp\\vmware\\certool.cfg', '--csrfile', 'c:\\temp\\vmca_issued_csr.csr']
2016-08-02T10:25:38.172Z Done running command

CSR generated at: c:\temp\vmca_issued_csr.csr
1. Continue to importing Custom certificate(s) and key(s) for VMCA Root Signing certificate

2. Exit certificate-manager

Option [1 or 2]: 1

Please provide valid custom certificate for Root.
File : c:\temp\chain.cer

Please provide valid custom key for Root.
File : c:\temp\vmca_issued_key.key

You are going to replace Root Certificate with custom certificate and regenerate all other certificates
Continue operation : Option[Y/N] ? : y
Get site nameCompleted [Replacing Machine SSL Cert...]
dublin
Lookup all services
Get service dublin:5fd6bca2-cea8-45f7-80b5-d02c23b88b41
Update service dublin:5fd6bca2-cea8-45f7-80b5-d02c23b88b41; spec:
---------------
blah,blah
---------------
Get service 3672c8ff-6af9-4bf3-8f23-9d78c1620b03
Don't update service 3672c8ff-6af9-4bf3-8f23-9d78c1620b03
Updated 7 service(s)
Status : 100% Completed [All tasks completed successfully]

Please restart all services in associated vCenter Server/s for changes made in Platform Service Controller machine to reflect

Perform restart operation on the vCenter Server/s by using 'service-control --stop --all' and 'service-control --start --all'

This is the full command prompt dump from VM Testvc2.lab.local:

C:\Windows\system32>"e:\Program Files\VMware\vCenter Server\bin\service-control" --stop --all
C:\Windows\system32>"e:\Program Files\VMware\vCenter Server\bin\service-control" --start --all
-------------------
note: above done AFTER finishing option 2 on the PSC vm testpsc.lab.local
-------------------

C:\Windows\system32>"E:\Program Files\VMware\vCenter Server\vmcad\certificate-manager.bat"
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.0 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-Z and hit Enter to exit.
Option[1 to 8]: 3

Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:

Performing operation on distributed setup, Please provide valid Infrastructure Server IP.
Server : 192.168.10.105

Please configure MACHINE_SSL_CERT.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-testvc2] : vmca-vc-machine-ssl

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local

You are going to regenerate Machine SSL cert using VMCA
Continue operation : Option[Y/N] ? : y
Get site nameompleted [Replacing Machine SSL Cert...]
dublin
Lookup all services
Get service dublin:5fd6bca2-cea8-45f7-80b5-d02c23b88b41
Don't update service dublin:5fd6bca2-cea8-45f7-80b5-d02c23b88b41
-------------------
blah,blah
-------------------
Update service 3672c8ff-6af9-4bf3-8f23-9d78c1620b03; spec: c:\users\admini~1.lab\appdata\local\temp\2\svcspec_tfp3r3
Updated 19 service(s)
Status : 100% Completed [All tasks completed successfully]

C:\Windows\system32>"E:\Program Files\VMware\vCenter Server\vmcad\certificate-manager.bat"
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.0 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-Z and hit Enter to exit.
Option[1 to 8]: 6
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : y

Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:

Performing operation on distributed setup, Please provide valid Infrastructure Server IP.
Server : 192.168.10.105

Please configure machine.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-testvc2] : vmca-vc-machine

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local

Please configure vsphere-webclient.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-testvc2] : vmca-vc-webclient

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local

Please configure vpxd.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-testvc2] : vmca-vc-vpxd

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local

Please configure vpxd-extension.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-testvc2] : vmca-vc-vpxd-ext

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local

You are going to regenerate Solution User Certificates using VMCA
Continue operation : Option[Y/N] ? : y
Status : 100% Completed [All tasks completed successfully]

Well, I hope you enjoyed the series, I'll try one more post on Site Recovery Manager 6 & SSL if I get my certificate problem fixed and that will be that!

Wednesday, 3 August 2016

vSphere Upgrade 5.0 to 6.0 - Part 5

This post deals with deploying signed SSL certificates to vSphere 6.0 using a subordinate CA and the VMware VMCA. Ideally I'd like to use the new PSC UI in 6.0 U1 but there's scant resources on the exact steps and when I tried importing the generated Root Certificate it did nothing!

I've been trying various approaches for a while to get this to work and not having much luck! I've a dual datacenter setup simulated in my lab and an external PSC. The first two VMs I need to get this to work on are:
lab50psc.lab.local - this is the external Platform Services Controller
lab50vcc.lab.local - this is the VMware vCenter Server

The Certificate Tool VMware provides the following 8 options:

1. Replace Machine SSL certificate with Custom Certificate
2. Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates
3. Replace Machine SSL certificate with VMCA Certificate
4. Regenerate a new VMCA Root Certificate and replace all certificates
5. Replace Solution user certificates with Custom Certificate
6. Replace Solution user certificates with VMCA certificates
7. Revert last performed operation by re-publishing old certificates
8. Reset all Certificates

The best article to read first is this one that explains the 3 approaches:
Understanding and using vSphere 6.0 Certificate Manager (2097936)
https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2097936

So we have the following approaches:

Regenerate all the default certificates - option 4 (Maybe they are about to expire etc)
Use Delegated Certificates with the VMCA - option 2 (This one is for me which uses subordinate certificates and saves you a lot of trouble later!)
Replace all Certificates with custom signed ones - option 5 (This is where you generate multiple custom certificates yourself using your Microsoft CA)

I'm going to use option 2 and Delegated Certificates. The best place to start is to practice with an embedded PSC/vCenter installation, you don't necessarily need any attached ESXi hosts but it's provides a good foundation. The steps for my test of "testvc.lab.local" are shown below. After this was completed I connected to the server without error. There's still update manager left but this isn't a big task.

(Note: Update the path to your VMware installation to find the command below)

C:\Windows\system32>"E:\Program Files\VMware\vCenter Server\vmcad\certificate-manager.bat"
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.0 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-Z and hit Enter to exit.
Option[1 to 8]: 2
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : y

Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:

Please configure MACHINE_SSL_CERT.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-] : vmca-machine-ssl

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc.lab.local

Please configure machine.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-] : vmca-machine

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc.lab.local

Please configure vsphere-webclient.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-] : vmca-webclient

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc.lab.local

Please configure vpxd.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-] : vmca-vpxd

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc.lab.local

Please configure vpxd-extension.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-] : vmca-vpxd-ext

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc.lab.local
1. Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate

2. Import custom certificate(s) and key(s) to replace existing VMCA Root Signing certificate

Option [1 or 2]: 1

Please provide a directory location to write the CSR(s) and PrivateKey(s) to:
Output directory path: c:\temp

Please configure certool.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : IE] :

Enter proper value for 'Name' [Default value : vmca-] : vmca-certool

Enter proper value for 'Organization' [Default value : Lab] :

Enter proper value for 'OrgUnit' [Default value : IT] :

Enter proper value for 'State' [Default value : Leinster] :

Enter proper value for 'Locality' [Default value : Dublin] :

Enter proper value for 'IPAddress' [optional] :

Enter proper value for 'Email' [Default value : administrator@lab.local] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc.lab.local
2016-07-29T14:25:45.410Z Running command: ['E:\\Program Files\\VMware\\vCenter Server\\vmcad\\certool.exe', '--genkey', '--privkey', 'c:\\temp\\vmca_issued_ke
y.key', '--pubkey', 'c:\\users\\admini~1.lab\\appdata\\local\\temp\\2\\pubkey.pub']
2016-07-29T14:25:45.712Z Done running command
2016-07-29T14:25:45.713Z Running command: ['E:\\Program Files\\VMware\\vCenter Server\\vmcad\\certool.exe', '--gencsr', '--privkey', 'c:\\temp\\vmca_issued_ke
y.key', '--pubkey', 'c:\\users\\admini~1.lab\\appdata\\local\\temp\\2\\pubkey.pub', '--config', 'E:\\ProgramData\\VMware\\vCenterServer\\runtime\\tmp\\vmware\\c
ertool.cfg', '--csrfile', 'c:\\temp\\vmca_issued_csr.csr']
2016-07-29T14:25:46.146Z Done running command

CSR generated at: c:\temp\vmca_issued_csr.csr
1. Continue to importing Custom certificate(s) and key(s) for VMCA Root Signing certificate

2. Exit certificate-manager

Now at this stage you use the CSR and the "vSphere 6.0 VMCA" CA Template (covered in later section below) to generate the required certificate. Download the cert and chain in base 64, open the chain and export the root ca cert in base 64. Next create a chain.cer file with the cert contents at the top and the root ca cert content below it with no spaces:

Chain.cer

-----BEGIN CERTIFICATE-----
MIIFnDCCBISgAwIBAgITKQAAAFhlR+fgdiITFAABAAAAWDANBgkqhkiG9w0BAQsF
-------------
blah.blah - this is the vmca-certool certificate using the vSphere 6.0 VMCA Template
-------------
W3a6NgBlkZpBWULSItO0SvSfzyZAKKPOfaYBQxf6APIUCxYGyGf0IFNxVjsuDfwk
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDojCCAoqgAwIBAgIQFwovcSg587lHjAqGSoM7KTANBgkqhkiG9w0BAQsFADBD
------------
blah,blah - this is the root CA certificate exported from the p7b chain certificate
------------
T/EU8xHmMqi5bXmQvte3y0Z/joIjAg==
-----END CERTIFICATE-----

Then proceed. You will find these steps described in many of the KBs and other Blog articles and after doing them 30 times I got bored so I'm not covering the steps here!

Option [1 or 2]: 1

Please provide valid custom certificate for Root.
File : c:\temp\chain.cer

Please provide valid custom key for Root.
File : c:\temp\vmca_issued_key.key

You are going to replace Root Certificate with custom certificate and regenerate all other certificates
Continue operation : Option[Y/N] ? : y
Get site nameCompleted [Replacing Machine SSL Cert...]
default-first-site
Lookup all services
Get service default-first-site:f2adacfb-7533-4370-ac88-9189c89acdda
Update service default-first-site:f2adacfb-7533-4370-ac88-9189c89acdda; spec: c:\users\admini~1.lab\appdata\local\temp\2\svcspec_slmoug
---------------
Ditto
---------------
Updated 24 service(s)
Status : 100% Completed [All tasks completed successfully]

The steps above show a successful SSL cert replacement on an embedded vCenter installation, i.e. the Platform Services Controller and all vCenter services on a single VM and I used the postgres database to make life easier as this was just a lab.

VMware Update Manager

Update Manager is an extra step. You use Openssl to generate a CSR and the vSphere 6.0 template to issue it. Then create a PFX file and copy the crt/key/pfx file to the update manager ssl folder, stop the service, run the update manager utility and start the service again and that's it! The key thnig when installing Update Manager is to ALWAYS select the FQDN on the following 2 screens if you intend to use SSL:

Change the IP Address to a FQDN here:

And also use the drop down here to select the FQDN:

This is the Update Manager Utility:
E:\Program Files (x86)\VMware\Infrastructure\Update Manager\VMwareUpdateManagerUtility.exe

It outlines the steps and I used the following commands on my Openssl config to generate the CSR and PFX:

openssl req -new -nodes -out rui.csr -keyout rui.key -config openssl.cfg

openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx

I used both domain names in the openssl.cfg file:

[alt_names]

DNS.1 = testvc.lab.local

DNS.2 = testvc

In the next section I'll describe the process of creating the two CA Templates.

CA Template

The first step is to configure two new Certificate Templates on your Microsoft CA server

Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 6.0 (2112009)
https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2112009

Firstly, I'm keen to ensure the certificates that will withstand scrutiny and avoid currently broken cyphers (one's now considered too weak for production). Ideally I'd like RSA with 2048 bits and SHA512 for good measure. Note - ideally you select this when setting up your CA so I checked mine, just view any of the certificates you've already issued and you can see the Signature Algorithm and Public Key fields:

I'm not far off it - note if you're using a legacy 2003 CA (?!!), you have bigger problems!

There is a good blog post on changing the signing algorithm here:
https://blogs.technet.microsoft.com/askpfeplat/2015/03/15/sha-1-deprecation-and-changing-the-root-cas-hash-algorithm/

The two CA interfaces used are as follows:

Click Start > Run, type certtmpl.msc, and click OK
Click Start > Run, type certsrv.msc, and click OK

Here are the properties of the vSphere 6.0 Template used for Update Manager, Machine SSL and Solution User certificates (Duplicated from the default Web Server template):

Check the Extensions Tab and Application Policies should be blank, remove any listed inside it

Check the Key Usage matches the following:

Check the Subject Name Tab matches the following:

The vSphere 6.0 VMCA Template for the subordinate CA should be copied from the Subordinate Certificate Authority Template and match the following:

Ensure it is published to AD:

And check the Key Usage policies match the ones shown below:

Note: The Certificate issued later to the PSC will take 24 hours before ESXi Hosts can be joined successfully so time this carefully! You can generate the request and issue the Certificate but hold off applying it until 24 hours have passed if you want to reconnect your Hosts immediately.

Now we can publish these templates:

There they are!

So we'll leave this post as is. The next step is to apply SSL certs to a distributed setup / external PSC which is a LOT of fun I can tell you! As they say, easy when you know how!!

Tuesday, 2 August 2016

vSphere Upgrade 5.0 to 6.0 - Part 4

So, now we've upgrade both legacy vCenter servers to 6.0U2 where do we go from here?! Update Manager has also been installed so we know all vCenter and Update Manager databases are on 6.0 but the underlying OS is still 2008R2 and the SQL backend is 2008R2.

I would be inclined to move vCenter and Update Manager at this point and their databases, as if we point SRM as them now, we're only going to have to change it later when their server name moves.

There is a VMware Kb on moving SRM to run on a different host:

Migrating an SRM server to run on a different host (1008426)
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008426

So again this only covers up to SRM 5.5.x but not 6.x!

Are we going to keep the same server name and ip address for vCenter? This is a key decision point as if we go with new the ESXi hosts have to be reconnected and what about the SSL certificates? We'll still have to deal with the PSC SSL cert so we may as well do the whole management platform while we're at it....

So where are we overall. We've actually only covered the vCenter & PSC servers as marked in orange below. Not much, huh?!! But we're in a strong position to re-platform vCenter now before returning to look at SRM.

Apart from credentials there is nothing we need off the old vCenter VMs. I'm going to use new Hostnames and IPs and see how I get on. I'll begin by powering down the old vCenter VMs after one last check they are working ok. I also powered down the whole lab and took a snapshot of each so I can return to this point. Watch your disk space!

We'll use the method of backing up and restoring the vCenter and Update Manager Databases to relocate them to SQL 2014 and upgrade their compatibility mode again. Of course this time they are 6.0 databases and not 5.0 so we should be fine. I've created a SQL database backup maintenance plan as it's quicker that way to backup all 3 databases in one go.
Note: You can use a free product like: https://sqlbackupandftp.com but it's limited to two databases. It's an easy product to use though if you're not familiar with SQL maintenance plans.

So, we'll proceed as before, choose to install an External Deployment of a vCenter Server with a new System Name. We point it at the "local site" Platform Services Controller. Choose the DSN and give it the Database Credentials and you might want to say no to the following:

The usual next, next stuff and Click Install....! Wait for this:

Now install the C# Client, Update Manager Server and then we can check things are heading in the right direction....remember the Update Manager frickin' DSN is still 32-bit, not 64-bit. See screenshot below:

When you connect with the right 32-bit DSN the default choice is to NOT overwrite the database:

Ensure you change the server to use the FQDN and NOT the IP Address:

Now I'll load the C# client and install the Update Manager Plugin. There is an SRM one but I'm not going to load it as it's not compatible until after it's upgraded. I'm just ignoring the certificates for now. I want the warnings to remain however for when I fix them up.

Now we can fix the Database Rollup Jobs in SQL 2014 which are currently missing:

Updating rollup jobs after the error: Performance data is currently not available for this entity (1004382)

https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1004382

So, as far as I can tell you copy the following files from C:\Program Files\VMware\vCenter Server\vpxd\sql to the SQL Server and execute their contents as SQL queries.

The filenames in the article are slightly different as these files have changed name it seems over the various versions of vCenter.

Log into SQL with the same account your vCenter DSN uses to connect to the vCenter Database
Double click each in the following order
Ensure the vCenter Database is selected and not the MASTER DB (You can help by selecting the vCenter Database in the list of databases on the left before double clicking the SQL file below)
Click Execute:

job_schedule1_mssql.sql

job_schedule2_mssql.sql

job_schedule3_mssql.sql

job_dbm_performance_data_mssql.sql

job_cleanup_events_mssql.sql

job_topn_past_day_mssql.sql

job_topn_past_month_mssql.sql

job_topn_past_week_mssql.sql

job_topn_past_year_mssql.sql

Back to the C# client - so the host is disconnected. We could reconnect it now but I want to fix the SSL certificates first in vCenter and the PSC before doing that. Then we can upgrade SRM. Connect to the Web Client next. I'm going to use Firefox as the administrator account won't launch IE in Server 2012R2 and I'm not bothered about circumventing it. I've also installed flash although this is a Lab, I wouldn't do this on a Production server....

Note, I'm got an error in the web client:

To resolve follow this article:
VMware vSphere Web Client displays the error: Failed to verify the SSL certificate for one or more vCenter Server Systems (2050273)
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2050273

So, I've a long output file but if you take your time you'll see a number of distinct sections, one per vCenter Server - I've my old and new ones listed so I now need to pull the "Service ID" for thw two legacy ones and purge these from the PSC. I assume the change will replicate but I'll rerun the query on both PSC's to be sure.

This is the query command run from an administrative command prompt:
"%VMWARE_PYTHON_BIN%" "%VMWARE_CIS_HOME%\VMware Identity Services\lstool\scripts\lstool.py" list --url http://localhost:7080/lookupservice/sdk --type vcenterserver > c:\psc_services.txt

This is the command to purge my first legacy vCenter Service ID:
"%VMWARE_PYTHON_BIN%" "%VMWARE_CIS_HOME%\VMware Identity Services\lstool\scripts\lstool.py" unregister --url http://localhost:7080/lookupservice/sdk --id C7BDC2DE-9016-4C01-8921-BCE2DF0ECF38 --user "administrator@vsphere.local" --pasd "YOUR PASS HERE" --no-check-cert

Update the password and run it again with the other Service ID and you should be ok. My query when I ran it again on both PSC's showed that the legacy vCenter's were both purged.

That's it for this post!

Wednesday, 27 July 2016

vSphere Upgrade 5.0 to 6.0 - Part 3

This is the list of services I found on 5.0 from vCenter (top) and SRM (below):

Check you've stopped them all - then we can proceed to SQL. Start by backing up the three SQL Databases to a handy location (I've used C:\Temp):

Next, configure the backup for each as follows, giving each a different name:

Then you should end up with the following files:

Copy them to the new SQL 2014 server:

Restore the Databases one at a time by choosing the "Device" option and browsing to the file:

Do this operation separately for each Database:

Then when they are restored you can edit each to change the compatibility from 100 to 120:

Finally recreate the SQL logins if you are using those or grant permissions to the domain service accoutns if not. Assign DB ownership etc as before. We have to install vCenter before we can recreate the rollup jobs for SQL Agent.

Next go to the new vCenter and configure the DSN's for vCenter and Update Manager and test the connection for each of them:

Both components are still not 64-bit in this edition of vCenter, see below. Note that the Native Driver is used NOT the ODBC one for SQL 2014:

Now, let's try installing vCenter 6.0U2 and pointing it at an existing database and see where we get!

So - this is new VM, Server 2012R2, DSN pointing at SQL 2014 SP1 but with restored copy of vCenter 5.0 Database, just to be clear!!

Note: the vCenter 6.0 installer checks there are TWO vCPUs for vCenter, make any necessary adjustments to your lab.

Here we go, install External vCenter:

Next confirm the vCenter system name:

Give it the Platform Services Controller FQDN:

Click OK:

If you're using the right native driver you'll be able to pick the DSN and enter the credentials below:

Now see what mess I've gotten myself into...!!!!

SO......this approach will not work. I'll have to perform a vCenter upgrade in place on Server 2008R2 and SQL 2008R2 and THEN perform the replatform and resql tasks afterwards.....darn it! Time to startup all those services again...! I'll cheat and just reboot the 4 legacy VMs..! Don't forget to clean up the databases on SQL 2014.

Don't forget vCenter 6.0 installer checks for TWO vCPU so if you're using a lab increase them now. I've also increased the RAM to 8GB to ensure smooth sailing....

So here we are again....this time performing an in place vCenter upgrade on the old 2008R2 Server VMs:

Once you run the installer it immediately recognises it's performing an upgrade:

You select the external model and we've already our PSC's ready to rock and roll:

Enter the Administrator password and Click OK regarding the linked mode warning shown below:

Enter the PSC FQDN and password:

Click OK on the certificate validation warning:

Make any changes you need to the destination paths, I'm in a Lab so I don't care...!!

Tick the box and sign your life away......

Click Upgrade to process the VM and transform it to vCenter 6.0. This is a clean and pristine install of vCenter 5.0 so I'm not expecting issues but I've read about a few production systems that ran into difficulty so ensure you have good backups of EVERYTHING if you're doing this live.

So we have a web interface we can log into and let's compare views before we upgrade our second vCenter 5.0:

There is no sign of the other vCenter anymore in either view so linked mode is down until I upgrade the second vCenter.

So no sign of SRM plugin in the web interface of course, I checked the C# client after running the 6.0 client installer, it's still there but doesn't work. One thing I found was trying to login with "lab\administrator" fails but just using "administrator" worked fine. This was in both the Web and C# client. Probably something I need to fix in the SSO. Log in as administrator@vsphere.local to resolve. As you see there is no lab.local domain listed:

We need to define our AD domain here:

And here we add in the Domain Administrator account as a vsphere.local Administrator:

now, log out and test!

We can also upgrade the Update Manager at this point.

It knows this is an upgrade

Use the FQDN always for SSL certificate use

Enter the DSN password

The default here is to NOT upgrade so change it

Ensure the FQDN is selected. The Download Service is an optional component for DMZ or air gapped situations, you're not missing anything with just the core Update Manager install here.

If we take a look now in the Web and C# client we can see if both Sites are back again. This is using the PSC enhanced linked mode feature:

The Web Client is ok but the C# client no longer shows the other Site as legacy ADAM linked mode is gone in 6.0 and you have to use the lovely, lovely web client (!) to manage both sides from here on in!!

Note: The configuration change to add in the domain admin user to SSO is only required once as we are using a unified SSO domain (!), so when you check it on Site B you'll see it already is there!