vCenter 6 Enhanced Linked Mode

vCenter 6 Enhanced Linked Mode

I was studying for my VCP-DCV6 and was trying to get my head around Enhanced Linked Mode. I used two VCSA's to see how this operated. I wanted to find out how do you configure the appliance settings after they are built and how do you configure linked mode. Both answers surprised me a little.

The last time I used the VCSA was with 5.5 and I had nothing but trouble with it. It held up a project I was working on for a week before I reverted to the Windows version. I guess it was early days, over a year and a half ago so I stuck to using it in Labs and POCs since then. With 6.0 it's gotten an overhaul so today I tried using it to see what's changed.

The 5.5 appliance had a specific URL to configure management settings, NTP, AD, IP information etc but it was nowhere to be found! I pulled up the documentation and to my surprise it's now configurable from within vCenter:

Now I thought for a second someone was pulling my leg but sure enough all the settings I would normally look for are listed above. I'm sure you'll find lots of other posts about deploying VCSA so I won't cover that here. There is also a CLI interface for when you mess up!

The next task was to deploy a second one and see how to "join" them in Enhanced Linked Mode. The Platform Services Controller can contain an instance of SSO and this is key. If during install you point at the SAME SSO installation you get Enhanced Linked Mode automatically. Unlike legacy linked mode you can't turn it on afterwards so get your design right and plan for HA of SSO! So I've two VCSA's, both with the Platform Services running but only one has SSO on it - both use it and share information as shown below.

You may need to log out and back in to the first VCSA to see the second one's resources after you add them. As you can see above you can browse through to either VCSA and see Hosts, VMs and lots more besides!

So for planning it's worth it to take the time and review the recommended vCenter architectures as once you upgrade (and make your choices) you can't go back!

SR-IOV and nested ESXi

SR-IOV and nested ESXi

I was always curious about some VMware options that I never had the hardware to replicate. One in particular was SR-IOV where you can expose a physical Intel X540 10Gb network card to guest VMs. I wanted to find out for curiosity if this would work with nested ESXi Hosts?!


As mentioned in a previous post you need to enable this in the BIOS before it will work in VMware.

As I don't have a 10Gb switch I've connected both of the ports to a 1Gb switch to enable me to use the feature. I wasn't allowed to use SR-IOV when I selected ESXi 6.0 as the OS, and existing nested VMs only let me switch between E1000 and E1000E as the network driver. When I created a new nested ESXi I could select VMXNET3 which appears to work ok. I really wanted to see if the SR-IOV ports would work so I cheated and created a Windows 2012 VM, then edited it to make it an ESXi one. You can see the results below. It does warn you SR-IOV is NOT compatible with ESXi.

So, sure enough when I'd finished this is the result I found (Only the VMXNET3 Driver Nic listed):

And so my curiosity was satisfied, it's not currently possible or compatible. There may be a hack but I'll stick with VMXNET3 drivers for now...!! I could always turn off SR-IOV and see what speeds I get internally from nested ESXi connected to the same port group on a 10Gb chipset.....!

Note: I did get a Windows 2012 R2 VM to use the SR-IOV ports fine and could configure the NIC advanced settings without issue, was interesting to see it all work! 

Home Lab - Upgrade

Home Lab - Upgrade

Well, after a few months of researching I finally upgraded my home Lab. I was primarily aiming to get rid of the 32GB limit and also get a few more cores and I'm well past that now and everything is running away merrily. I've also got an Intel NUC as a compute target when using Cloud/VDI configurations which is nice. The only issue was a Noctua Heatsink blocking the use of two memory slots, I changed this for a Supermicro one and apart from the fan noise, I'm now running with all 8 memory slots fully populated.

Lessons learnt:

• EATX has two sizes so it's NOT a standard as such, check your case manufacturer's website for their definition against the motherboard manufacturers actual size! 
• Heatsinks may say narrow ILM compatible but you may find they don't allow for the way some motherboards place the memory slots beside the CPU socket and therefore you will run into problems. I bought two expensive Noctua narrow ILM compatible heatsinks to find they are not suitable. The Supermicro heatsinks are fine but I've to replace the fan with a quieter 60mm version. 
• Power requirements are not as bad as I feared. I was running < 100 kWH with my old Baby Dragon rig and now I'm running at around 150kWH, I was budgeting for up to 300! 50% extra for 4 times the Ram and 2 times the # cores! 

Heatsink issue :

Noctua with Fan Removed (Left two memory slots blocked by angled heatpipe):

(Noctua NH-U12DX i4 Xeon Cooler)

Supermicro Designed for 2U Servers, lower profile & noisier 60mm fan):
(SuperO SNK-P0048AP4 2U LGA2011)

EATX:

I won't show you what I did to mount the motherboard, I'd be embarrassed because the case was not compatible, I could use about 3 screw holes and that was it! I used plastic standoffs and cable ties for the rest and hope that the PCI cards take some of the strain! I love the case though, it's a Corsair Carbide 540 High Airflow ATX Cube Case and very easy to work with. I had a Lian Li for the Micro-ATX build before but it's SO difficult to work with the SSDs I gave up and went large. I didn't want a full tower case so a wide one was great until I found it didn't take the new motherboard as well as I'd assumed! 

Kit List:

Motherboard:

SUPERMICRO MBD-X10DRI-T-O 

Got this from Newegg in USA. Had a LOT of difficulty with this part. At first I ordered an open box one saving me $100 then they had issues with payment. When I went back to reorder the open box one was gone or stuck in the system. I ended up paying full whack. I get it sent to a shipping forwarded and they too took a cut, they charge by "dimension weight" so even though it's not that heavy an item, it cost a LOT to send on to Europe. The other problem was they couldn't get hold of an export code classifying the type of computer item. Supermicro in the Netherlands came back to me within an hour with the code when I asked them and we were all set! More happened but it eventually turned up and I got hold of it!  

ECCN Notes: 

Export Code for all Supermicro Motherboards with IPMI - ECCN=5A002

Export Code for all Supermicro Motherboards without IPMI - ECCN=5A992

support@supermicro.nl

I went for this board as it was only an extra $60 for the 10Gb ports for future expansion and while I planned on using one CPU initially and 8 slots of ram, I could grow later. I was lucky to get a great price on the CPU via ebay, I stayed away from Engineering Samples...! No problems with the board, just made sure none of the standoffs could cause a short and it powered on first time. The IPMI and Motherboard Bios were all bang up to date too which was a bonus!

Heatsinks:

Noctua NH-U12DX i4 Xeon Cooler

replaced by 

Supermicro SNK-P0048AP4 2U LGA2011

and

GE4084 ITR-XR-1 Noiseblocker BlackSilent Case Fan Fan XR1-60mm

Power supply:

XFX 650w P1-650B-BEFX GOLD PSU

This has connectivity for BOTH 8 pin motherboard power supplies to be attached together in addition to the main 24 pin one. So if I run some heavy CPUs in the future, this should cope fine. I'm not getting the best efficiency currently as I'm well below 300 Watts, the optimum but so be it! 

Memory:

Crucial 64GB Kit (16GBx4) DDR4 2133 MT/s (PC4-2133) CL15 DR

(CT4K16G4RFD4213)

I can go to 256GB of ram without a sweat (except to my pocket!) when I add a second CPU someday. By then 32GB modules may be feasible. 

CPU:

I spent most of time time looking for the ideal CPU. I got hold of an Intel E5-2618L v3 thanks to a brilliant deal on E-Bay. This is a 75W CPU but has 8 cores and high enough boost clock speeds (2.3GHz base vs 3.4GHz boost). My previous CPU was faster per core but this one is more energy efficient and gives me room to deploy larger VMs. A 2630L or 2650L would have been nice but none appeared in my price threshold, after a year or two they may be more of them around....

http://ark.intel.com/products/83351/Intel-Xeon-Processor-E5-2618L-v3-20M-Cache-2_30-GHz

SR-IOV Notes: 

FYI: if you want to play with SR-IOV you need to enable it specifically in the Bios here:

Then you can enable in the Host Physical Network Properties in VMware and assign a port to the VM after a reboot. I've connected the 10Gb ports to a 1Gb switch and was able to ping out from the VM fine, letting me see how this works for the first time! 

Intel NUC:

 

Not much I can say, it works fine and doubles as a windows media player by removing the USB with ESXi on it. I've a mSATA drive with windows, USB for ESXi and 2.5" SSD for VMware Storage all sitting in a nice 2014 generation NUC! I wanted to see what these were like to work with and so far so good. It does get into a state currently after booting it up saying the host is not able to power on VMs, disconnecting the Host and reconnecting it fixes this issue. Otherwise grand, might be a power saving issue or something creeping in. The UEFI Bios is nice too! I got this from the USA on E-Bay at a nice price. Just remember you're getting an empty box, you need to get a suitable power cable and memory/ storage before you get going. Don't forget a mouse/keyboard for the initial configuration. This model doesn't come with Intel vPRO for remote configuration. 

The NUC is maxed out with 16GB Ram but this is perfect for a few VMs when doing VDI/Cloud Labs. There is only a single 1Gb NIC but so far this hasn't been an issue for me. I may well replace it eventually with an Intel Xeon-D 1520/1540 down the road if I need more horsepower but for now it's a versatile and tidy, power lean package! It worked with distributed vSwitches which I can't risk on my main server as any changes of hardware etc and you're asking for trouble! 

So, that's it, enough to bankrupt me for the rest of the year and hopefully give me loads of scale to work with as I go. I'm doing my VCP-DCV6-Delta next week so glad to have this all behind me so I can put it to good use! 

Why a Home Lab?

Why a Home Lab?

I've been lucky enough to have a home lab to play with for the last few years. It helps to get to know different aspects of technology and software to help me in my job. Rather than get into the exact hardware and options in my Lab I thought I'd take a stab at why it's been useful to me, so if you're considering one you might find some of my uses resonate?:

• You're responsible for the hardware. Both in terms of selection and price. You can spend as much or little as you want. If it breaks, you're on the spot to fix it. This can give good experience in troubleshooting but sometimes leave you scratching your head and looking for help. There a certain satisfaction when you configure & get something working for the first time! Also seeking out useful upgrades and how to best put them to use.
• You have exclusive use of the hardware. Unlike work labs which can be unrecognisable by the time you come back to it a few months after being "reconfigured" by well meaning colleagues, you spend as much time putting it back together as you do using it for what you intended in the first place. Yes, the employer pays the electricity and it can be left switched on but few employers see any value in it as an investment. You're probably going to be left with old unwanted hardware and ram or cpu limits you can't do anything about! There are some good scroungers out there that can always add value but it's an uphill struggle.
• Exams. Probably the main reason I went with a home lab is to study for exams. It gives me a chance to break things, and mess about without worrying about annoyed colleagues or customers giving out! I get to play around in advance of a project with new software and make sure I avoid the pitfalls when I'm onsite.
• Blogging. I've started blogging about my experiences using my Home Lab to build different technology solutions. It's at a very basic level but I've often used the posts myself down the road to refer to doing something I don't do often enough to become second nature. Some customers want LDAP & SSL, some don't etc.
• Faster Upgrade Cycles. You can upgrade as soon as new software comes out, you're not limited to being tied into compatibility issues or breaking backups. You know the risks going in and revert if it fails or wait it out until a patch arrives. Be bleeding edge at home, not at work! Want to see what all the fuss is about, stick it in your home lab!
• Looking at new versions of Products before I see them in future Projects. So at least I'm not installing something for the first time onsite....! I can test out particular use cases and find out what works, what breaks and what to avoid.

Disadvantages:

• Expense. You probably can't afford 3 x €20,000 servers and a SAN plus a 3KVa power upgrade to your house unless you're deriving direct revenue from your home lab, are you hosting a cloud or something?!! While I would love to go 10Gb Ethernet as an example, the PCI cards are €300+, a switch €1 to 4,000 which is crazy. I would love a faster processor and more Ram but where does it end? You don't get tax back unless you're self employed. So, pick your features carefully - does the new hardware do something you can't do currently? Will this provide the ability to play with features you can't access currently and see them in operation? Would nested hypervisors work at all? What other options do you have? Boat loads of 1Gb?! Know your limits...you might not be able to replicate every technology in your Lab but if you're smart you'll manage on the job....
• Scale out - the model where you scale out to multiple servers increases expense - you need shared storage and you've probably tripled the costs of your host hardware. If you want to upgrade, you need 3 of everything, think about that. Amazing if you have the money, but for me not the way to go. I prefer to Scale Up at home!

You can use Cloud to do a lot and it probably works out cheaper, you pay as you go. There are certainly months that go by and my lab doesn't get powered on which feels like a poor ROI. But....I'm only paying for electricity when I do power it on at that point. The thing I don't like with cloud is you pay for every vCPU, I'm not sure nested would work great or at all, when I own pCPU I can get major overcommit without paying extra. It's only where a particular appliance needs 8 vCPU and I've only 4 pCPU I run into trouble. Or if it needs more Ram than I can afford, another issue!

So, plan carefully, take your time, look for bargains on ebay. Do you want a second hand server at a bargain or build your own? Get a parts list off another blog and see what you would go with or change. Watch out for risks regarding power supply and other key compatibilities. You don't want to hook everything up like I did and power on to discover the motherboard needs a BIOS upgrade to recognize your CPU model! I had to buy a cheap CPU to perform the Bios flash!

My own Lab is based on a 4 core CPU and 32GB Ram. That's the max it can support. Right now I'm in the process of scaling up to remove these limits so I can drop in more Ram or CPU as needed. It's going to be the most expensive home lab investment I've made yet but I can start with a wide foundation and build up over the next year to something I can throw nearly anything at. Having to start a whole new system just to break one barrier (# cores, # ram) makes me think bigger. So consider what the physical limits are and when you're likely to hit them.

I think I'll leave it there. I'm amazed by the variety of home labs other people have blogged about and if it works, go for it. Get a balance between cost and functionality, pay attention to power and noise to keep the partner happy and see if you can access it remotely (smart plug to power it on, auto boot, auto VM startup, RAS solution - Bridged router, DDNS etc) so you can refer to it at work to look at something and not be limited to just using it when you're at home......and enjoy! You can always switch it off and walk away to catch up on some on demand TV! Life is outside the door so stick your head out once in a while to remind yourself what the sky looks like!