Wednesday, 20 April 2016

Installing and working with ELK Stack - Part 1 (Environment Setup)

I recently got to work with the ELK Stack. It can be considered as an open-source replacement for Splunk (which btw is pretty impressive in itself if you have a good amount of extra cash in your accounts). The ELK Stack is used for better analysis and visualization of your log files.


ELK Stack Infrastructure (Pic courtesy Mitchell Anicas's tutorial on ELK Stack )


In this Part-1 of the tutorial, I'll go over the Environment Setup for the ELK Stack. ELK Stack comprises of the following three technologies.

Elasticsearch (E): The amazing ElasticSearch datastore server. Its a search server based on Lucene which provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. For more information please visit ElasticSearch Wiki.

Logstash (L): It is an open source tool for collecting, parsing, and storing logs for future use. For more information please visit LogStash Wiki.

Kibana (K): Its an open source data visualization plugin for ElasticSearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. For more information please visit Kibana Wiki.

Question: If we already have data in the logs then why do we need this??

Answer: That's a very good question. Although, you gather a lot of logs - syslogs, user logs, etc., but still its very difficult to go through them and make any scheduled analysis/report. Here's where the ELK stack helps you. You can load your logs into ElasticSearch using LogStash and then use Kibana for querying and generating report dashboards from this data. Pretty Awesome!!!!


Pre-requisites


Minimum System Requirements (The More The Merrier!)

  • Ubuntu 14.04 LTS
  • 1 GB RAM
  • 50 GB HDD
  • 2 CPUs


Preliminary Steps

Lets begin with some ground work which will help us in future.


#update and upgrade all installed packages
$ sudo apt-get update && sudo apt-get upgrade

#install java
$ sudo apt-get install openjdk-7-jre

#create a "Downloads" folder so as to keep all downloads at one place
$ cd ~
$ mkdir Downloads



Installing ElasticSearch


Although it can be installed using a package manager but I still like to do it the old school way, i.e., by downloading the debian package and then installing it.


$ cd ~/Downloads

#Download and unpack ElasticSearch. Latest release at the time of this writing is v2.3.1
$ wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.3.1/elasticsearch-2.3.1.deb
$ sudo dpkg -i elasticsearch-2.3.1.deb

#enable elastic search to start automatically at boot
$ sudo update-rc.d elasticsearch defaults 95 10

#start service
$ sudo service elasticsearch start




Installing LogStash


Now lets install LogStash in the same way - by downloading and installing the debian package


$ cd ~/Downloads

#Download and unpack LogStash. Latest release at the time of this writing is v2.3.1
$ wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_2.3.1-1_all.deb
$ sudo dpkg -i logstash_2.3.1-1_all.deb




Testing LogStash


After installation its a good idea to verify the LogStash install. We can do it using the following.


$ cd ~
$ /opt/logstash/bin/logstash agent -e "input {stdin { } } output { stdout { codec => rubydebug } }"


Type Hello, Logstash!
You should see an output somewhat like this


{
"message" => "Hello, Logstash!",
"@version" => "1",
"@timestamp" => "2014-07-28T01:27:27.231Z",
"host" => "elkstack"
}


You can exit LogStash using CTRL+D



Installing Kibana



Now lets install the final part of the stack, i.e., Kibana.


$ cd ~/Downloads

#Add the Kibana repo to the sources. Latest release at the time of this writing is v4.5
$ wget https://download.elastic.co/kibana/kibana/kibana_4.5.0_amd64.deb
$ sudo dpkg -i kibana_4.5.0_amd64.deb


We need to configure the server host name in the Kibana settings. For this, we will open Kibana's yaml file to replace and uncomment the server.host value with "localhost"


$ sudo editor /opt/kibana/config/kibana.yml


Also, lets allow Kibana to start automatically at boot


$ sudo update-rc.d kibana defaults 96 9
$ sudo service kibana start #start kibana service




Installing Nginx


Lets also install Nginx so that we can access our Kibana Dashboard from browser


#install nginx
$ sudo apt-get install nginx apache2-utils

#create a new user in htpasswd for kibana login
$ sudo htpasswd -c /etc/nginx/htpasswd.users <YourUsername>

#edit the nginx config file
$ sudo mv /etc/nginx/sites-available/default /etc/nginx/sites-available/default.old #create backup
$ sudo editor /etc/nginx/sites-available/default #create new default


Add the following code to the "default" file and save it


server {
  #remember to check that no other service is running on port 80. Apache(if installed) should be stopped.
  listen 80;
  server_name <YourServerName>;

  auth_basic "Restricted Access";
  auth_basic_user_file /etc/nginx/htpasswd.users; #use the htpasswd.users to match the credentials

  location / {
    proxy_pass http://localhost:5601;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
  }
}

#finally lets restart the Nginx service
sudo service nginx restart


If all works well you should be able to access your Kibana Dashboard through your browser. Just browse the host name of your machine and enter the username and password for Kibana.

This completes the Part-1 of the ELK Stack, i.e., the setup part.

Coming Soon: 
  • Part-2, in which I'll discuss about how to work with the ELK Stack and load data into ElasticSearch from a remote database using custom LogStash config. I'll also show how to work with Kibana dashboards to create simple visualizations.
  • Youtube Video tutorials for both parts.
Enjoy Analyzing your logs!!! 

Stay tuned by subscribing to this blog.

5 comments:

  1. we are offering best Obiee online training with work support and job assistance and high quality training facilities and well expert faculty
    for other details and register your demo contact
    obiee training in hyderabad

    ReplyDelete
  2. we are offering best splunk online training with job support and high quality training facilities and well expert faculty . to Register you free demo please visit ,splunk training in hyderabad

    ReplyDelete
  3. Hi,
    You have performed a great job. I will definitely digg it and for my part recommend to my friends. I'm sure they will be benefited from this site. Oracle EBS training
    Thank you.

    ReplyDelete