vSphere Upgrade 5.0 to 6.0 - Part 6
So, back to our test install. I've deployed two VMs as follows:testpsc.lab.local - Platform Services Controller 6.0 U2 (192.168.10.105 - you use this later)
testvc2.lab.local - vCenter 6.0 U2 (192.168.10.104)
I've installed Update Manager and confirmed the self signed certificates are in place. When using the legacy C# client you'll see the warning below, click Ignore for now. Once we correctly replace these certs we should get no warning which is a good sign we've been successful!
Understanding and using vSphere 6.0 Certificate Manager (2097936)
https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2097936
https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2097936
So this will be somewhat similar to my last post but in a distributed setup the steps are a little trickier! I'm going to take a powered down snapshot of my two VMs and I'd strongly suggest you do the same as I've found running the certificate utility multiple times breaks something and I've been unable to fix it - basically you can try any option and NO services are updated anymore. There's some cleanup step I'm probably missing. I'd also recommend setting up a Lab for this if you get time as a typo could cost you a day!
The Certificate Manager interface
I edit the command prompt by launching it separately here, right click on the "start button":
I've chosen to use Width 150 & Height 900:
C:\Program Files\VMware\vCenter Server\vmcad\certool.cfg
I've setup mine as follows for the testpsc VM:
#
# Template file for a CSR request
#
# Country is needed and has to be 2 characters
Country = IE
Name = vmca-editme
Organization = Lab
OrgUnit = IT
State = Leinster
Locality = Dublin
IPAddress =
Email = administrator@lab.local
Hostname = testpsc.lab.local
And the following for the testvc2 VM:
#
# Template file for a CSR request
#
# Country is needed and has to be 2 characters
Country = IE
Name = vmca-editme
Organization = Lab
OrgUnit = IT
State = Leinster
Locality = Dublin
IPAddress =
Email = administrator@lab.local
Hostname = testvc2.lab.local
Now launch the utility.
Note: Run the utility as administrator (update drive path to utility as required):
"C:\Program Files\VMware\vCenter Server\vmcad\certificate-manager.bat"
Now you might well ask from which VM do you run this?!! We want to execute option 2 and if we try this from the vCenter VM we'll get the following warning:
It then asks "do you wish to generate all certificates using configuration file" - say yes to this. We now get to input the options for each of the certificates it will generate, they have to be unique in some way however, so I've used the following Name and Hostname variations:
On testpsc.lab.local (Option 2):
MACHINE_SSL_CERT.cfg - vmca-machine-ssl - testpsc.lab.local
machine.cfg - vmca-machine - testpsc.lab.local
vsphere-webclient.cfg - vmca-webclient - testpsc.lab.local
certool.cfg - vmca-certool - testpsc.lab.local
Once this completes on vCenter VM testvc2.lab.local execute the following:
"C:\Program Files\VMware\vCenter Server\bin\service-control" --stop --all
"C:\Program Files\VMware\vCenter Server\bin\service-control" --start --all
Now switch VMs from the PSC to vCenter and run the Certificate Management Tool there and execute the following options:
(Don't forget to edit the certool.cfg defaults!)
MACHINE_SSL_CERT.cfg - vmca-vc-machine-ssl - testvc2.lab.local
On testvc2.lab.local (Option 6):
machine.cfg - vmca-vc-machine - testvc2.lab.local
vsphere-webclient.cfg - vmca-vc-webclient - testvc2.lab.local
vpxd.cfg - vmca-vc-vpxd - testvc2.lab.local
vpxd-extension.cfg - vmca-vc-vpxd-ext - testvc2.lab.local
I tried duplicating the values in the Name field between the PSC 7 vCenter responses as the hostnames are different. However, I got an error loading the web interface, the logs reported "Error occurred looking for solution user : More than one solution user found". Stick to unique Name fields above to avoid this problem!
The error is covered in this KB:
"To avoid this issue, ensure the config files are unique for each solution user by modifying the Name.
For example, use vSphere-vpxd, vSphere-vpxd-extension, etc, instead of vSphere for each config file."
Just stick to the Names I've listed above and you will be find. I reverted to my snapshot after encountering the error and once I use the names above I was fine again and the web interface worked ok.
Also, I've dumped a full copy of the Certificate Manager output at the bottom of this post to avoid making things too boring! You can compare your output against it there. In that example I've used unique Name & hostname values as listed above. If you get the error above, rerun the Certificate Manager on the vCenter Server and options 3 & 6 once more and the problem will resolve itself or revert to snapshot if none of the services update anymore (until I figure out why someday!).
After doing this more than once but backing out you may need to do a clearout:
C:\ProgramData\VMware\vCenterServer\runtime\tmp\vmware
Make sure you clear out all the file from this folder except the "cis-license" subfolder - it's important to make sure you remove the old_machine_ssl.crt file in addition to the cfg files to reset the Certificate Manager again. This isn't always enough however so revert to snapshot if the services stop updating.
Note: If using Firefox which maintains it's own Root CA list, be aware that the Active Directory one isn't referenced. You can import the chain.cer into Firefox directly and restart the browser to eliminate the error you get otherwise:
PSC Certificate Utility
VMware vSphere 6.0 Update 1 brought us a web interface for the PSC. The features are covered here:
So, here is the web interface on the PSC to manage certificates (there since 6.0 U1):
I checked both of my PSCs and they are using different root certificates so that means whatever steps I find work will need to be carried out on both PSCs and they in turn will issue certs for the other vCenter components and ESXi hosts in their respective sites.
Note: be careful of PSC browser bookmarks as they often capture SAMLRequest guids which are invalid when reused later - just edit the bookmark and cut off anything after /psc/
You appear to be able to replace the Root Certificate but you need to manually set one up and upload it as shown below. However when I tried this it did nothing, no errors, zip. Maybe I need to restart services or something but I've not found any resources for this approach online so gave up in the end!
Please ignore this section as it's a completely worthless waste of both of our time!
VMCA Subordinate Certificate
The vCenter Web Interface now has a trusted certificate subordinated to my Lab CA, see the full chain below.
Troubleshooting
The main trouble I encountered is after trying to replace the PSC certs using option 2 and it updates 0 services which is not good! I tried cleaning down the tmp folder: C:\ProgramData\VMware\vCenterServer\runtime\tmp\vmware
and using option 4 to regenerate the VMCA Root Certificate, and tried option 8 to reset all certificates but all fail to resolve this issue, it ends up stating "Updated 0 service(s)". This might be down to a faulty old ssl certificate thumbprint which when it can't match, it can't replace it. I've found some detail below:
Validating the Trust Anchors is covered in the following article but it's meant to be fixed in vCenter 6.0 U1:
My advice - log a support call! (Or revert to snapshot, or practice in a lab first!)
Certificate Manager Output
This is the full command prompt dump from VM Testpsc.lab.local:
C:\Windows\system32>"C:\Program Files\VMware\vCenter Server\vmcad\certificate-manager.bat"
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.0 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-Z and hit Enter to exit.
Option[1 to 8]: 2
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : y
Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:
Please configure MACHINE_SSL_CERT.cfg with proper values before proceeding to next step.
Press Enter key to skip optional parameters or use Default value.
Enter proper value for 'Country' [Default value : IE] :
Enter proper value for 'Name' [Default value : vmca-testpsc] : vmca-machine-ssl
Enter proper value for 'Organization' [Default value : Lab] :
Enter proper value for 'OrgUnit' [Default value : IT] :
Enter proper value for 'State' [Default value : Leinster] :
Enter proper value for 'Locality' [Default value : Dublin] :
Enter proper value for 'IPAddress' [optional] :
Enter proper value for 'Email' [Default value : administrator@lab.local] :
Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testpsc.lab.local
Please configure machine.cfg with proper values before proceeding to next step.
Press Enter key to skip optional parameters or use Default value.
Enter proper value for 'Country' [Default value : IE] :
Enter proper value for 'Name' [Default value : vmca-testpsc] : vmca-machine
Enter proper value for 'Organization' [Default value : Lab] :
Enter proper value for 'OrgUnit' [Default value : IT] :
Enter proper value for 'State' [Default value : Leinster] :
Enter proper value for 'Locality' [Default value : Dublin] :
Enter proper value for 'IPAddress' [optional] :
Enter proper value for 'Email' [Default value : administrator@lab.local] :
Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local
Please configure vsphere-webclient.cfg with proper values before proceeding to next step.
Press Enter key to skip optional parameters or use Default value.
Enter proper value for 'Country' [Default value : IE] :
Enter proper value for 'Name' [Default value : vmca-testpsc] : vmca-webclient
Enter proper value for 'Organization' [Default value : Lab] :
Enter proper value for 'OrgUnit' [Default value : IT] :
Enter proper value for 'State' [Default value : Leinster] :
Enter proper value for 'Locality' [Default value : Dublin] :
Enter proper value for 'IPAddress' [optional] :
Enter proper value for 'Email' [Default value : administrator@lab.local] :
Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local
1. Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate
2. Import custom certificate(s) and key(s) to replace existing VMCA Root Signing certificate
Option [1 or 2]: 1
Please provide a directory location to write the CSR(s) and PrivateKey(s) to:
Output directory path: c:\temp
Please configure certool.cfg with proper values before proceeding to next step.
Press Enter key to skip optional parameters or use Default value.
Enter proper value for 'Country' [Default value : IE] :
Enter proper value for 'Name' [Default value : vmca-testpsc] : vmca-certool
Enter proper value for 'Organization' [Default value : Lab] :
Enter proper value for 'OrgUnit' [Default value : IT] :
Enter proper value for 'State' [Default value : Leinster] :
Enter proper value for 'Locality' [Default value : Dublin] :
Enter proper value for 'IPAddress' [optional] :
Enter proper value for 'Email' [Default value : administrator@lab.local] :
Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testpsc.lab.local
2016-08-02T10:25:37.548Z Running command: ['C:\\Program Files\\VMware\\vCenter Server\\vmcad\\certool.exe', '--genkey', '--privkey', 'c:\\temp\\vmca
_issued_key.key', '--pubkey', 'c:\\users\\admini~1.lab\\appdata\\local\\temp\\2\\pubkey.pub']
2016-08-02T10:25:37.918Z Done running command
2016-08-02T10:25:37.920Z Running command: ['C:\\Program Files\\VMware\\vCenter Server\\vmcad\\certool.exe', '--gencsr', '--privkey', 'c:\\temp\\vmca
_issued_key.key', '--pubkey', 'c:\\users\\admini~1.lab\\appdata\\local\\temp\\2\\pubkey.pub', '--config', 'C:\\ProgramData\\VMware\\vCenterServer\\run
time\\tmp\\vmware\\certool.cfg', '--csrfile', 'c:\\temp\\vmca_issued_csr.csr']
2016-08-02T10:25:38.172Z Done running command
CSR generated at: c:\temp\vmca_issued_csr.csr
1. Continue to importing Custom certificate(s) and key(s) for VMCA Root Signing certificate
2. Exit certificate-manager
Option [1 or 2]: 1
Please provide valid custom certificate for Root.
File : c:\temp\chain.cer
Please provide valid custom key for Root.
File : c:\temp\vmca_issued_key.key
You are going to replace Root Certificate with custom certificate and regenerate all other certificates
Continue operation : Option[Y/N] ? : y
Get site nameCompleted [Replacing Machine SSL Cert...]
dublin
Lookup all services
Get service dublin:5fd6bca2-cea8-45f7-80b5-d02c23b88b41
Update service dublin:5fd6bca2-cea8-45f7-80b5-d02c23b88b41; spec:
---------------
blah,blah
---------------
Get service 3672c8ff-6af9-4bf3-8f23-9d78c1620b03
Don't update service 3672c8ff-6af9-4bf3-8f23-9d78c1620b03
Updated 7 service(s)
Status : 100% Completed [All tasks completed successfully]
Please restart all services in associated vCenter Server/s for changes made in Platform Service Controller machine to reflect
Perform restart operation on the vCenter Server/s by using 'service-control --stop --all' and 'service-control --start --all'
This is the full command prompt dump from VM Testvc2.lab.local:
C:\Windows\system32>"e:\Program Files\VMware\vCenter Server\bin\service-control" --stop --all
C:\Windows\system32>"e:\Program Files\VMware\vCenter Server\bin\service-control" --start --all
-------------------
note: above done AFTER finishing option 2 on the PSC vm testpsc.lab.local
-------------------
C:\Windows\system32>"E:\Program Files\VMware\vCenter Server\vmcad\certificate-manager.bat"
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.0 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-Z and hit Enter to exit.
Option[1 to 8]: 3
Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:
Performing operation on distributed setup, Please provide valid Infrastructure Server IP.
Server : 192.168.10.105
Please configure MACHINE_SSL_CERT.cfg with proper values before proceeding to next step.
Press Enter key to skip optional parameters or use Default value.
Enter proper value for 'Country' [Default value : IE] :
Enter proper value for 'Name' [Default value : vmca-testvc2] : vmca-vc-machine-ssl
Enter proper value for 'Organization' [Default value : Lab] :
Enter proper value for 'OrgUnit' [Default value : IT] :
Enter proper value for 'State' [Default value : Leinster] :
Enter proper value for 'Locality' [Default value : Dublin] :
Enter proper value for 'IPAddress' [optional] :
Enter proper value for 'Email' [Default value : administrator@lab.local] :
Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local
You are going to regenerate Machine SSL cert using VMCA
Continue operation : Option[Y/N] ? : y
Get site nameompleted [Replacing Machine SSL Cert...]
dublin
Lookup all services
Get service dublin:5fd6bca2-cea8-45f7-80b5-d02c23b88b41
Don't update service dublin:5fd6bca2-cea8-45f7-80b5-d02c23b88b41
-------------------
blah,blah
-------------------
Update service 3672c8ff-6af9-4bf3-8f23-9d78c1620b03; spec: c:\users\admini~1.lab\appdata\local\temp\2\svcspec_tfp3r3
Updated 19 service(s)
Status : 100% Completed [All tasks completed successfully]
C:\Windows\system32>"E:\Program Files\VMware\vCenter Server\vmcad\certificate-manager.bat"
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.0 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-Z and hit Enter to exit.
Option[1 to 8]: 6
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : y
Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:
Performing operation on distributed setup, Please provide valid Infrastructure Server IP.
Server : 192.168.10.105
Please configure machine.cfg with proper values before proceeding to next step.
Press Enter key to skip optional parameters or use Default value.
Enter proper value for 'Country' [Default value : IE] :
Enter proper value for 'Name' [Default value : vmca-testvc2] : vmca-vc-machine
Enter proper value for 'Organization' [Default value : Lab] :
Enter proper value for 'OrgUnit' [Default value : IT] :
Enter proper value for 'State' [Default value : Leinster] :
Enter proper value for 'Locality' [Default value : Dublin] :
Enter proper value for 'IPAddress' [optional] :
Enter proper value for 'Email' [Default value : administrator@lab.local] :
Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local
Please configure vsphere-webclient.cfg with proper values before proceeding to next step.
Press Enter key to skip optional parameters or use Default value.
Enter proper value for 'Country' [Default value : IE] :
Enter proper value for 'Name' [Default value : vmca-testvc2] : vmca-vc-webclient
Enter proper value for 'Organization' [Default value : Lab] :
Enter proper value for 'OrgUnit' [Default value : IT] :
Enter proper value for 'State' [Default value : Leinster] :
Enter proper value for 'Locality' [Default value : Dublin] :
Enter proper value for 'IPAddress' [optional] :
Enter proper value for 'Email' [Default value : administrator@lab.local] :
Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local
Please configure vpxd.cfg with proper values before proceeding to next step.
Press Enter key to skip optional parameters or use Default value.
Enter proper value for 'Country' [Default value : IE] :
Enter proper value for 'Name' [Default value : vmca-testvc2] : vmca-vc-vpxd
Enter proper value for 'Organization' [Default value : Lab] :
Enter proper value for 'OrgUnit' [Default value : IT] :
Enter proper value for 'State' [Default value : Leinster] :
Enter proper value for 'Locality' [Default value : Dublin] :
Enter proper value for 'IPAddress' [optional] :
Enter proper value for 'Email' [Default value : administrator@lab.local] :
Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local
Please configure vpxd-extension.cfg with proper values before proceeding to next step.
Press Enter key to skip optional parameters or use Default value.
Enter proper value for 'Country' [Default value : IE] :
Enter proper value for 'Name' [Default value : vmca-testvc2] : vmca-vc-vpxd-ext
Enter proper value for 'Organization' [Default value : Lab] :
Enter proper value for 'OrgUnit' [Default value : IT] :
Enter proper value for 'State' [Default value : Leinster] :
Enter proper value for 'Locality' [Default value : Dublin] :
Enter proper value for 'IPAddress' [optional] :
Enter proper value for 'Email' [Default value : administrator@lab.local] :
Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : testvc2.lab.local
You are going to regenerate Solution User Certificates using VMCA
Continue operation : Option[Y/N] ? : y
Status : 100% Completed [All tasks completed successfully]
Well, I hope you enjoyed the series, I'll try one more post on Site Recovery Manager 6 & SSL if I get my certificate problem fixed and that will be that!
