Yes, even Windows needs love... I understand that most startups or cool kids use linux, but there are a heck of a lot of businesses that run IIS / C# / SQL Server to get things done. I must admit I have previously setup logstash monitoring on Centos and Ubuntu and it's pretty easy... so easy that I don't need to write about it. Installing logstash on every server to collect and send logs through to elastic can cause a bunch of unnecessary overhead. You might come across the term "logstash shipper", which effectively sends logs to a centralized logstash server, so... that's what we plan to do, or better yet skip logstash and go straight to Elastic. There are 5 millions ways to skin a cat (send logs to Elastic), but I'm going to go straight Elastics direct offerings and have a play around with Beats and winlogbeat, not to be confused with the overly expensive headphones by Dr Dre that sound like poop and only 16 year olds and recruiters wear. #1 DOWNLOAD https://www.elastic.co/downloads/beats/winlogbeat Some reference install documentation https://www.elastic.co/guide/en/beats/winlogbeat/current/index.html #2 INSTALLING I'm going to be the first to admit it, I don't know what powershell is (I have been too linuxed for too long). Google tells me: Alright, now all the windows fan boys can scoff at my lack of knowledge of powershell (I'm actually MCP certified for Win server 2003, but that was over a decade ago):
#3 CONFIGURATION So it seems to me that the service is now installed. Before we kick this thing off we need to configure some stuff, eg: like where to send all of the glorious logs. (I'm assuming you already have an elastic instance running some where). Most of your configuration for your server is located in "C:\Program Files\winlogbeat\winlogbeat.yml".. if you don't know what YAML is, it stands for "yet another markup language" which is slightly confusing as it's more data oriented than markup. Let's move on.. The great thing about the config YAML file is that the developers at Elastic have thankfully made some very clear comments in the file, so it shouldn't be too easy to screw up. I cut all the comments out of the winlogbeat.yml file so it was easier to make sense of for this blog. Below is a small explanation for each section:
Now to figure out what "event_logs" are available to you, in power shell run "Get-EventLog *" this will return a list of what can be gathered from this server, see below (I'm running this on my win 7 desktop initially): Pro-tip: Before you save your own config file, make a backup of the old one. If you're not used to YAML, chances are you'll format something wrong. Try www.yamllint.com to validate your formatting. If you're happy with the config file, save it! #4 CREATE INDEX AND MAPPING IN ELASTIC In Elastics documentation they ask me to run the following command in power shell (replacing the localhost section with your hostname: PS C:\Program Files\Winlogbeat> Invoke-WebRequest -Method Put -InFile winlogbeat.template.json -Uri http://localhost:9200/_template/winlogbeat?pretty I get the following error: When I look at the file winlogbeat.template.json it's a standard mapping file. If you want more information to fix this then check out this article: https://matt40k.uk/2014/07/the-term-invoke-webrequest-is-not-recognized/ . I will just grab the winlogbeat.template.json file and run it a linux terminal (the same way I created all my other mappings) to get passed the errors above. (insert lazyness here). #5 START WINLOGBEAT SERVICE In powershell, startup winlogbeat by running: "Start-Service winlogbeat". You can see in my screenshots below that winlogbeat has started and collecting files. I added a new index-pattern in kibana called "winlogbeat-*" and as you can see below it has started ingesting data. #6 FINISHING UP
A couple of notes to tidy up.
0 Comments
Over a year ago I moved from Canada back to New Zealand. My wife didn't want to directly live in Wellington due to the weather being sub-par, so we moved up the coast to a small town called Raumati. Unfortunately for me there isn't a lot of data work in Raumati, so I commute into Wellington 3 days a week to help out my clients. On the weekend you can travel this stretch of road in around 40-50 minutes, in 2015 during peak traffic I was averaging 60-70 minutes, 2016 struck and some days I'm up to 120 minutes. Now there have a been a lot of road works on my journey that started in 2016 and it's easy to blame the NZ traffic authority, but I have another hypothesis (that isn't ground breaking) but I think it's due to the petrol prices being so low. The New Zealand government is pretty open with its data, so I started to collect traffic stats on State highway 1 (SH1) and historical petrol prices to see if there is a co-relation between the two. So to stop my babbling, I will share my findings: The data you're looking covers 2012 through to 2015 (4 years). NZTA hasn't provided data for 2016 yet. The Y Axis represents the cost of gas & X covers the amount of vehicles. NZTA provided data based on many points on SH1, I used the collection from Pukerua bay as that's pretty close to my place. Using the same data (2012-2015), you can see a much stronger co-relation between gas prices and number of vehicles on the road (graph below). Since I don't have vehicle data for 2016, I can only make assumptions on the volumes, you can see that we are still quite a few cents down from December 2015. So what drives people to drop public transport and take the petrol powered horse and carriage? I drive a honda fit to work (don't laugh) and it averages around 14km per litre (which is pretty good). My total costs for driving into work 5 days a week is:
I will let you come to your on conclusions on the analysis, I feel like my original hypothesis is correct and more people are driving in based on the price of fuel. Personally I value time more than money and losing 2 hours a day isn't worth it which is causing me to get pretty close to getting that train pass. Side note: It's also valid to point out the December has the highest traffic numbers of any month for every year (christmas and holidays etc). If you're hell bent on driving into work, you might want to pray that gas prices go back up.. otherwise suck it up. Data Sources: Train costs Vehicle Data Petrol Data |
AuthorNew Zealand big data nerd, facial hair sculptor and classic car fanatic. Owner of needles.io, freelance big data consultant, ex Activision. Archives
April 2016
Categories |