Migrating to VCSA 6.5 U1

Migration to VCSA 6.5 U1

I've been working with a customer migrating vCenter 5.5 U3 to VCSA 6.5 U1. They had a lab to test against and the upgrade there went fine but when we got to the first Production vCenter it turned out to be a very different story!

I'm writing this post to gather my thoughts and offer tips that will help you when it comes to your turn!!

The first stumbling block was that the vCenter SSL certs had expired. These were the ones replacing the original self signed ones. We tried a number of methods but ran into problems re-registering the inventory service with SSO. Time to reinstall and keep the old database. We backed up the license keys and permissions just in case and uninstalled ALL the vCenter components. The simple install failed. The advanced install also failed on the SSO. It just kept rolling back. We logged a call with support and they guided us through cleaning out the %temp% folder as it turns out VMware likes to reuse the crap that's in there. We then install the Pre-Req's by hand - see KB2059481 for info. There were also 2 x CIS folders to delete, see the same KB. Once we'd done that SSO installed fine and we ran though the other services up to Update Manager. Oh, how I wished we'd stopped before that though!!

We ended up re-running the migration EIGHT times in total at 1-2 hours a pop before we finally got it to work all the way through. One of THOSE migrations!! The first issue was when we could see it zipping up some vum files on the old vCenter where the migration assistant runs. Shortly after that it would fail with "The compressed (zipped) folder is invalid or corrupted". A google search found lots of others with similar issues but no relevant KB article. The guidance was uninstall Update Manager and take the database offline and try again. That didn't work. During the next migration it still found & zipped Update Manager components from somewhere despite it being uninstalled and the database no longer being online!! Turns out the migration assistant works off another directory and keeps the Update Manager components there and in future attempts grab the old files and tries to use them where it fails....again!!!! Crapola.

Here's the process we worked out to get you the best chance of pulling off the migration. Repeat these steps between each migration attempt as necessary.

1. Uninstall Update Manager, get rid of it - take notes or whatever but seriously, kill the damn thing
2. Take the database offline
3. Delete all files in the %temp% folder - go up a level if it drops you into %temp%/2 or somewhere like that - the users temp folder would be somewhere like "C:\Users\Bozo\AppData\Local\Temp" - remove everything BELOW this directory
4. Delete the C:\Users\Michael\AppData\VMware folder and everything below it - this is where the migration-assistant folder lives and its contents are going to screw you everytime until you castrate 'em! 
5. Create a local admin account with admin privs to get you out of a hole if you've to rollback as the server's computer account is transferred to the VCSA and you will have to rejoin AD. If you know your local admin account password, great, but TEST it!! Or you'll get locked out. Good luck with that. 
6. You need a VCSA port group that is configured for ephemeral ports, get this ready or you won't see any valid port groups listed in the migration wizard!
7. Configure the security settings "MAC Address Changes" and "Forged Transmits" to Accept on this Port Group. You can move the VM to another port group once the migration is finished but if you run the migration process enough times your feckin' VCSA network port will end up BLOCKED as it keeps appearing with a new MAC addresses each time you try and if the port gets blocked, guess what, you've to start all over again as it fails the migration process!! 
8. Disable HA/DRS on your clusters temporarily if you can
9. Expect to sail through on your first attempt or face hours and hours of pain and frustration, it's gonna go one way or the other! 

If you reinstall vCenter like we did, once you remove everything from add/remove programs, probably a good idea to remove VMware folders from both program files folders and programdata for good measure. You'll need to reconnect your ESXi Hosts after the reinstall, there's a way to script this out there somewhere. 

You'll find once this works (the migration) your hosts are still connected and your vCenter license is set to expire 59 days later so get to that. Check your SSO Admin password expiry is appropriate. Move onto your next vCenter and hope it gets easier!! Best of luck and did I mentioned that if the starting PortGres database message takes a few hours, just wait a bit longer!! Good Luck!!