VMware vSphere Syslog options - Part 1
The troubleshooting facilities in VMware vCenter and vSphere are fairly good in my opinion. Through my work I can access a support dump analyzer for ESXi Hosts that provides very useful information. I can crack open the vCenter dump to access specific logs and see what's been going on. I can tail the vmkernel.logs to monitor realtime activity and with the introduction of the Syslog collector service in vCenter look back in time - just add the logs or vCenter VM to the backup schedule and see if a particular problem existed previous to a patch or upgrade.
Where I don't get involved is in deep investigation of issues. Support typically go deeper in their analysis of particular problems and the in built syslog server uses flat logfiles which make tracing a particular fault more difficult.
I heard about a few options in this area and wanted to explore them here and try them out.
Pay versions:
VMware vRealize Log Insight
https://www.vmware.com/products/vrealize-log-insight/
Commercial, free for up to 25 OSI (Operating System Instances) when you own a supported vCenter license but $$ beyond that. This might just do some smaller businesses and from what I've seen is a great product with contents packs to extend monitoring beyond VMware Products. They don't charge for the storage of large amounts of log data. The product is deployed as a virtual appliance.
Splunk
http://www.splunk.com/en_us/products/splunk-enterprise.html
This is the main competitor to VMware's product and has been out there a while with a good knowledge base. They have cloud options and a Splunk Lite. The pricing is per GB of logs, hence you need to know the volume of logs generated but this could spike when experiencing an issue which is not so good! It installs on Windows, Linux, Solaris and Mac OS.
Kiwi Syslog Server
http://www.kiwisyslog.com/products/kiwi-syslog-server/product-overview.aspx
I've used this myself in the past with their CATTools to backup Cisco switches. Cost is a flat fee of €240 and installs on windows only.
Free/Pay versions:
PRTG Free Syslog Server
https://www.paessler.com/free_syslog_server
Well, it's free up to 100 sensors, they estimate each device will use 5-10 sensors but I'm going to test it with just syslog on an ESXi host to see if I could get 100 ESXi hosts out of it. You want more of course you pay more and it's not free at that point. But it could be just enough for you.....
Free versions:
Syslog-NG
https://www.balabit.com/network-security/syslog-ng
https://syslog-ng.org
https://github.com/balabit/syslog-ng
Well, this is completely free and open source but requires Linux. You can use the VMware virtual management appliance (vMA) to get this up and running. I'm interested to see how much work there is to get it capturing logs from multiple ESXi hosts and then query the product as with most Open Source solutions, they are no easy to use / get working out of the box. But if you're up for a challenge then so am I!!
My ideal is an open source, free product, easy to set up, Kibana web front end with Elastic Search and little to no configuring for vSphere logs!! I can dream right?!
PRTG Free Syslog Server
So this one is interesting. Once installed and loaded up it began discovering my network and picked up everything running on the same subnet. This included vCenter, ESXi, network devices etc. All I was interested in was the syslog server but it's not enabled by default. You have to add a new probe device to the local probe called "Syslog Receiver" and choose the settings if you want them different from the default, mine worked fine as is. Once I configured ESXi from KB2003322 I got a few messages through and could see how it worked.
So it's getting message but how easy is it to retrieve specific message levels. There is a message tab to list all recent ones.
Then you can select a specific severity as shown below. There is some rudimentary text searching.
The historic data tab allows exporting of specific errors to html or csv.
That's as exciting as it gets!
The trial license stays for 30 days even after putting in the free key, then limiting you to 100 devices. I was able to remove a lot of the discovered entries and sensors to reduce the amount in use. Windows Firewall Rules were added automatically. There was a web interface but I didn't spend too much time with it.
Kiwi Syslog Server
This one is also Windows based and required .NET 3.5 so add that Feature into Server 2012 R2 if required to allow the install to proceed. After install I pointed ESXi at the new syslog IP and off it went. You can see immediately the new logs hitting the server.
There is no search or filter really in the main screen. You can schedule archive options as shown to deal with historical records but it just stores things in a flat file much like the VMware equivalent. You can do more filtering of course over what it captures and output it to different "screens" but it's no better than when I last looked at it over 10 years ago....there web interface failed on install so didn't get a look at that.
Splunk Enterprise
Simple to install. I went with the Enterprise version to see what features it provided. There is a VMware App you can add in but after 10 minutes while this was heading in the direction I wanted I gave up as it appears complex but very powerful, not so easy to get to grips with. Very well worth checking out though....
VMware vRealize Log Insight
Erm, I downloaded the appliance and found that you need a supported version of vCenter to get a free 25 OSI license, but they then say go to the downloads page and click on Read More to get the 5.5 and lower version key which is listed right there?! The OVA deployment was standard, I let DHCP do everything as I was only going to test it for a few minutes.
Blog entry here from VMware:
Now, how does it stack up? Well I couldn't connect to it at first with the browser, after a few minutes it rebooted and was doing some configuration by itself so I left it alone. Wait until you see this in the console.
Now you'll be able to connect with the web interface
I clicked New Deployment. Once you finish all the initial setup questions you can configure integration. As it's a VMware product for monitoring VMware primarily I know this is going to be easy!
To start with I just configured my ESXi Host to sent logs to the Log Insight appliance and didn't configure vCenter integration. Just to see what happened. There are content packs for vSAN and more.
I was definitely getting logs in from the host so I decided to configure vCenter integration next.
The VMware specific dashboards are interesting.
And you can drill into these to get the interactive view which is absent in the other products where I tested it.
So, I'd be fairly happy to give HPE Support access to VMware's Log Insight and be sure it would help them out. PRTG would also tick my box but I'm less sure about Kiwi Syslog Server. Splunk with effort in setting it up would most likely beat all of these but cost is a factor and the slight complexity which when you've limited time for a Lab session isn't workable. I'm sure there's good Blogs out there you can find that would help you set it up correctly and to test it out.
Syslog-NG
This one troubles me as there doesn't appear to be any web interface or search engines so what's the point!?! I did find some good articles using other open source software so I'm going to give them a try, document the results and add them as my next post so stay tuned. If it works I just need to find a suitable Linux distribution to deploy and see how manageable that becomes over time.
