Friday 26 July 2019

SSL Certificates - vCenter 6.5 with integrated PSC

This will hopefully be a series of posts about deploying signed certificates from a Server 2016 Enterprise CA to a variety of VMware Solutions. I've been installing such certs in a number of VMware Products recently and thought it might be useful to capture the process a bit here.

With vCenter, there are the default self signed certificates of course. VMCA issues self signed certificates to the hosts it manages and you can control the renewal of these.

For signed certificates, you have three main routes:

  • Use the CA to issue certificates to each of the vCenter components and to each ESXi host
  • Use the CA to issue a subordinate certificate to vCenter VMCA and let it sort out the rest!
  • Use the CA to issue certificates to user facing vCenter components and let VMCA handle the rest - HYBRID *VMware's recommended option*
My preference is the Hybrid approach as it's less work, less hair pulling and really deals with the issue of man in the middle head on - it's the people facing admin interfaces which you want to protect most of all. The other interfaces are either not used by admins ever or in the case of the ESXi web interface, rarely. Like when you power down vCenter and have to go searching for which host it was last seen alive on...! 

A lot of customers who have security officers won't be happy with the subordinate CA approach as it gives the VMCA (vCenter's integrated certificate authority) a LOT of power to issue any certificates it like under the Root/Intermediate branches. That can be abused. They usually just say NO! 

They also say no to any wildcard certificates being used. They are handy as you can issue the same cert to multiple products and various common names but it's a fudge and bad security practise. Great for a Lab maybe but not Development and definitely not Production. 

So, assuming Hybrid is the way you want to go, here is the procedure. I'm using vCenter 6.5 update 1 here as I happened to have a nested lab with this version ready to go.

By the way the Blog about Hybrid is here:
Also a walk through is here:

Now, here is my current vCenter SSL certificate (out of the box). Note the 10 year expiration and the path says the issuer can't be found. This is because in a proper CA environment, every time you browse to a secure site you should be able to check in with the CA to see if that particular certificate has been revoked, very important in a security context if the certificate has been compromised. A self signed doesn't do that. 
 

Now we need a new template on the 2016 Server CA for VMware. Duplicate the Web Server one as shown. Click start, run, type in MMC and add in the"Certificate Templates" snap-in to get this UI.
A little bit of editing is required. Set the Certification Authority to Windows Server 2008 and Update the template display name. 



Edit Key Usage and set "Signature is proof of origin (nonrepudiation)



Click Ok and this time edit Application Policies to remove Server Authentication entirely. Check Supply in the request is set in the Subject Name tab. Click OK to save.

 

Now we need to issue the template. Bring up server manager and from the Tools drop down menu choose Certification Authority. Right Click on Certificate Templates, Click New, Certificate Template to Issue. Select your template and Click Ok.


Now, the handiest way I'm used to issuing certificates is with the old web interface on Server 2016. When you install a CA role it should give you this web enrollment option.

When you browse to http://localhost/certsrv or to this URL remotely and authenticate it will provide you the familiar UI to request and issue certificates. The current setup doesn't wait for approval although this is certainly the way a Production CA would operate so build in some time in your project to allow for certificates to be signed and issued and check for typos before you send in the request or you will have wasted some time in the process. A certificate request can be decoded via openssl or most commercial CA authorities have a web tool so you can check its all spelled correctly. 

Click on Download a CA certificate, certificate chain, or CRL Choose Base 64 and Click Download CA certificate. You now have certnew.cer - keep this somewhere handy. 
Enable SSH to your vCenter - if you haven't already done so browse to 
https://<vCenter FQDN>:5480
You can enable SSH and the Bash Shell in here. 


Putty to the vCenter IP / FQDN and we can start to interact with the VMCA. 


I've changed to the shell interface, set the default to shell so we can use WinSCP later to upload certificates and started up the certificate-manager. We're going to replace the Machine SSL certificate. Option 1 looks like this:

Now load up a copy of WinSCP and connect to vCenter and grab /tmp/vmca_issued_csr.csr and the /tmp/vmca_issued_key.key just in case! 

Back to the certificate web server request page. Click Home and then Request a Certificate. Click advanced certificate request. 

 


Click the second link, I'm not typing that in here....!!
 
 Edit the vmca_issued_csr.csr in Notepad++ (Get this, it's worth it!) or just use old Notepad from windows. Paste the entire contents into the saved request field below and choose you template we made earlier.
 At this stage in a lab we get our certificate immediately. Change to Base 64 and Click Download Certificate and rename it to something after you vCenter, in my case I called it Labvc2.cer.

If you did this directly on your CA Server, don't start looking for the cert on your local desktop, copy it off the CA server first!!!! Now, remember that certnew.cer root cert from earlier, we need to add that to the signed certificate as the VMCA doesn't have a clue about the CA, this way it's told to trust the certificate and here's the CA cert to prove it's legit. 
Edit the Labvc2.cer and paste the contents of certnew.cer at the bottom, no spaces please....
Upload the file to vCenter along with certnew.cer with WinSCP and we're back to Putty again to install them. I've called my chained certificate Labvc2_SSL.cer. 


Now I got an error first time which was because I'd already registered NSX and vCloud Director with this vCenter, ideally you would replace certificates BEFORE deploying other solutions. The error was in /var/log/vmware/vmcad/certificate-manager.log and stated:
ERROR certificate-manager 'lstool get' failed: 1
Fix:
https://kb.vmware.com/s/article/2150057
https://kb.vmware.com/s/article/1025360

After removing those plugins I tried again:


Success!! That's it. You can verify the certificate is signed for both the Flash and HTML5 interfaces. The appliance :5480 one is not part of the Machine SSL so don't worry about that one....
 









Posted by at