Home Lab - Upgrade
Well, after a few months of researching I finally upgraded my home Lab. I was primarily aiming to get rid of the 32GB limit and also get a few more cores and I'm well past that now and everything is running away merrily. I've also got an Intel NUC as a compute target when using Cloud/VDI configurations which is nice. The only issue was a Noctua Heatsink blocking the use of two memory slots, I changed this for a Supermicro one and apart from the fan noise, I'm now running with all 8 memory slots fully populated.
Lessons learnt:
- EATX has two sizes so it's NOT a standard as such, check your case manufacturer's website for their definition against the motherboard manufacturers actual size!
- Heatsinks may say narrow ILM compatible but you may find they don't allow for the way some motherboards place the memory slots beside the CPU socket and therefore you will run into problems. I bought two expensive Noctua narrow ILM compatible heatsinks to find they are not suitable. The Supermicro heatsinks are fine but I've to replace the fan with a quieter 60mm version.
- Power requirements are not as bad as I feared. I was running < 100 kWH with my old Baby Dragon rig and now I'm running at around 150kWH, I was budgeting for up to 300! 50% extra for 4 times the Ram and 2 times the # cores!
Heatsink issue :
Noctua with Fan Removed (Left two memory slots blocked by angled heatpipe):
(Noctua NH-U12DX i4 Xeon Cooler)
(SuperO SNK-P0048AP4 2U LGA2011)
EATX:
I won't show you what I did to mount the motherboard, I'd be embarrassed because the case was not compatible, I could use about 3 screw holes and that was it! I used plastic standoffs and cable ties for the rest and hope that the PCI cards take some of the strain! I love the case though, it's a Corsair Carbide 540 High Airflow ATX Cube Case and very easy to work with. I had a Lian Li for the Micro-ATX build before but it's SO difficult to work with the SSDs I gave up and went large. I didn't want a full tower case so a wide one was great until I found it didn't take the new motherboard as well as I'd assumed!
Kit List:
Motherboard:
SUPERMICRO MBD-X10DRI-T-O
Got this from Newegg in USA. Had a LOT of difficulty with this part. At first I ordered an open box one saving me $100 then they had issues with payment. When I went back to reorder the open box one was gone or stuck in the system. I ended up paying full whack. I get it sent to a shipping forwarded and they too took a cut, they charge by "dimension weight" so even though it's not that heavy an item, it cost a LOT to send on to Europe. The other problem was they couldn't get hold of an export code classifying the type of computer item. Supermicro in the Netherlands came back to me within an hour with the code when I asked them and we were all set! More happened but it eventually turned up and I got hold of it!
ECCN Notes:
Export Code for all Supermicro Motherboards with IPMI - ECCN=5A002
Export Code for all Supermicro Motherboards without IPMI - ECCN=5A992
support@supermicro.nl
I went for this board as it was only an extra $60 for the 10Gb ports for future expansion and while I planned on using one CPU initially and 8 slots of ram, I could grow later. I was lucky to get a great price on the CPU via ebay, I stayed away from Engineering Samples...! No problems with the board, just made sure none of the standoffs could cause a short and it powered on first time. The IPMI and Motherboard Bios were all bang up to date too which was a bonus!
Heatsinks:
Noctua NH-U12DX i4 Xeon Cooler
replaced by
Supermicro SNK-P0048AP4 2U LGA2011
and
GE4084 ITR-XR-1 Noiseblocker BlackSilent Case Fan Fan XR1-60mm
Power supply:
XFX 650w P1-650B-BEFX GOLD PSU
This has connectivity for BOTH 8 pin motherboard power supplies to be attached together in addition to the main 24 pin one. So if I run some heavy CPUs in the future, this should cope fine. I'm not getting the best efficiency currently as I'm well below 300 Watts, the optimum but so be it!
Memory:
Crucial 64GB Kit (16GBx4) DDR4 2133 MT/s (PC4-2133) CL15 DR
(CT4K16G4RFD4213)
I can go to 256GB of ram without a sweat (except to my pocket!) when I add a second CPU someday. By then 32GB modules may be feasible.
CPU:
I spent most of time time looking for the ideal CPU. I got hold of an Intel E5-2618L v3 thanks to a brilliant deal on E-Bay. This is a 75W CPU but has 8 cores and high enough boost clock speeds (2.3GHz base vs 3.4GHz boost). My previous CPU was faster per core but this one is more energy efficient and gives me room to deploy larger VMs. A 2630L or 2650L would have been nice but none appeared in my price threshold, after a year or two they may be more of them around....
http://ark.intel.com/products/83351/Intel-Xeon-Processor-E5-2618L-v3-20M-Cache-2_30-GHz
SR-IOV Notes:
FYI: if you want to play with SR-IOV you need to enable it specifically in the Bios here:
Then you can enable in the Host Physical Network Properties in VMware and assign a port to the VM after a reboot. I've connected the 10Gb ports to a 1Gb switch and was able to ping out from the VM fine, letting me see how this works for the first time!
Intel NUC:
Not much I can say, it works fine and doubles as a windows media player by removing the USB with ESXi on it. I've a mSATA drive with windows, USB for ESXi and 2.5" SSD for VMware Storage all sitting in a nice 2014 generation NUC! I wanted to see what these were like to work with and so far so good. It does get into a state currently after booting it up saying the host is not able to power on VMs, disconnecting the Host and reconnecting it fixes this issue. Otherwise grand, might be a power saving issue or something creeping in. The UEFI Bios is nice too! I got this from the USA on E-Bay at a nice price. Just remember you're getting an empty box, you need to get a suitable power cable and memory/ storage before you get going. Don't forget a mouse/keyboard for the initial configuration. This model doesn't come with Intel vPRO for remote configuration.
The NUC is maxed out with 16GB Ram but this is perfect for a few VMs when doing VDI/Cloud Labs. There is only a single 1Gb NIC but so far this hasn't been an issue for me. I may well replace it eventually with an Intel Xeon-D 1520/1540 down the road if I need more horsepower but for now it's a versatile and tidy, power lean package! It worked with distributed vSwitches which I can't risk on my main server as any changes of hardware etc and you're asking for trouble!
So, that's it, enough to bankrupt me for the rest of the year and hopefully give me loads of scale to work with as I go. I'm doing my VCP-DCV6-Delta next week so glad to have this all behind me so I can put it to good use!
