Install and configure Elastic Stack on ubuntu

I have been exploring the Elastic Stack over the past few days, as a means of collecting and analysing data exported from edge machines.

The project is an on-premise solution on a highly secure network, so there is no possibility of making use of the cloud services offered by ElasticSearch, so to evaluate the solution, I needed to get my hands dirty and install Elastic stack.

This guide will walk through the process of installing the Elastic Stack on a fresh ubuntu server 16.04.

Install Java

ElasticSearch requires at least Java 8. For the purpose of this guide we'll install Oracle JDK Version 1.8.0_121 .
Check out Elasticsearch Reference [5.2] Â» Getting Started to find out more.

To install Java (if not already installed) first update your package index

 sudo apt update

Then we need to add Oracles PPA to our package repository

sudo add-apt-repository ppa:webupd8team/java
    sudo apt-get update

Now we can install the latest Java 8

sudo apt-get install oracle-java8-installer

Once installed we need to set the JAVA_HOME environment variable. This will be used later by both ElasticSearch and Kibana.

Open /etc/environment using nano text editor.

sudo nano /etc/environment

At the end of this file, add the following line, making sure you use the correct path in your environment.

JAVA_HOME="/usr/lib/jvm/java-8-oracle"

Save and exit the file and then ensure it is reloaded

source /etc/environment

now test to ensure the environment variable has been set

echo $JAVA_HOME

Install ElasticSearch

We'll install elastic search using the Debian package. The instructions below are basically a trimmed down version of the instructions from Install ElasticSearch with Debian Package

Import the ElasticSearch PGP key

	wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Installing from the APT repository

You may need to install the apt-transport-https package on Debian before proceeding:

sudo apt-get install apt-transport-https

Add the repository definition

echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

We can now install ElasticSearch with the Debian package

sudo apt-get update && sudo apt-get install elasticsearch

Configure the daemon to run

sudo /bin/systemctl daemon-reload sudo /bin/systemctl enable elasticsearch.service

We can now start and stop the service as required as follows

sudo systemctl start elasticsearch.service sudo systemctl stop elasticsearch.service

Install Kibana

update your package index then install Kibana, once complete check your Kibana configuration file

sudo apt udpate
sudo apt-get -y install kibana 
sudo nano /etc/kibana/kibana.yml

I didn't change anything in this file, but I just had a quick look through to familiarise myself with the settings available.

Enable the Kibana Service and start it.

sudo systemctl daemon-reload sudo systemctl enable kibana sudo systemctl start kibana

Install Nginx

Kibana is packaged with its own webserver which by default is served on http://localhost:5601 , however because we are on an internet facing server, we need to configure a reverse proxy to allow external access to it. We'll need to install Nginx or apache as a web server. In my case I will be using Nginx.

Use apt to install Nginx

Use openssl to create an admin user, call it anything you want i.e. kibadmin, this will be the user that can access the Kibana web interface.

You'll be prompted to enter and confirm a password for this user.

Configure Nginx server block. First, backup the default server block and rename it then create a new file.

sudo apt-get -y install nginx
 sudo -v
 echo "kibadmin:<code>{{EJS0}}</code>" | sudo tee -a /etc/nginx/htpasswd.users
 sudo mv /etc/nginx/sites-available/default   /etc/nginx/sites-available/default.old 
 sudo nano /etc/nginx/sites-available/default

Then copy the following information into the file

server {
        listen 80;
 
        server_name [Your IP or URL address here];
 
        auth_basic "Restricted Access";
        auth_basic_user_file /etc/nginx/htpasswd.users;
 
        location / {
            proxy_pass http://localhost:5601;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;        
        }
    }

Save and exit. This configures Nginx to direct your server's HTTP traffic to the Kibana application, which is listening on http://localhost:5601. Also, Nginx will use the htpasswd.users file, that we created earlier, and require basic authentication.

check the config for syntax errors and restart Nginx if none are found:

sudo nginx -t
sudo systemctl restart nginx

You should now be able to navigate to your website and and be prompted for the secure login

Install Logstash

Update the repository index again and install logstash

sudo apt update
sudo apt install logstash

logstash is now installed but not yet configured.

Generate SSL certificates

We're going to use FileBeats to ship logs from our client servers to ElasticStack server, so we will need to create SSL certificate and key pairs. The certificate will be used by FileBeats to verify and identify the server.

Create the directories to store the certs and private keys

sudo mkdir -p /etc/pki/tls/certs
sudo mkdir /etc/pki/tls/private

In my particular case I don't have DNS set up for my POC so I will be using IP address to resolve my server. So lets open the OpenSSL configuration file

sudo nano /etc/ssl/openssl.cnf

Find the [ v3_ca ] section in the file, and add this line under it

 subjectAltName = IP: server__IP

Now generate the SSL certificate and private key in the appropriate locations (/etc/pki/tls/...), with the following commands:

cd /etc/pki/tls
    sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

The logstash-forwarder.crt file will be copied to all of the servers that will send logs to Logstash but we will do that a little later. Let's complete our Logstash configuration.

Configure Logstash

Logstash configuration files are in the JSON-format, and reside in /etc/logstash/conf.d. The configuration consists of three sections: inputs, filters, and outputs.
create a configuration file called 02-beats-input.conf and set up our "filebeat" input:

sudo nano /etc/logstash/conf.d/02-beats-input.conf

Insert the following input configuration:

input {
      beats {
        port => 5044
        ssl => true
        ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
        ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
      }
    }

Save and quit. This specifies a beats input that will listen on TCP port 5044, and it will use the SSL certificate and private key that we created earlier.

sudo ufw allow 5044

create a configuration file called 10-syslog-filter.conf, where we will add a filter for syslog messages:

sudo nano /etc/logstash/conf.d/10-syslog-filter.conf

Insert the following syslog filter configuration:

 filter {
      if [type] == "syslog" {
        grok {
          match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
          add_field => [ "received_at", "%{@timestamp}" ]
          add_field => [ "received_from", "%{host}" ]
        }
        syslog_pri { }
        date {
          match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
      }
    }

Save and quit. This filter looks for logs that are labeled as "syslog" type (by Filebeat), and it will try to use grok to parse incoming syslog logs to make it structured and query-able.

Lastly, we will create a configuration file called 30-elasticsearch-output.conf:

sudo nano /etc/logstash/conf.d/30-elasticsearch-output.conf 

 output {
      elasticsearch {
        hosts => ["localhost:9200"]
        sniffing => true
        manage_template => false
        index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
      }
    }

Save and exit. This output basically configures Logstash to store the beats data in Elasticsearch which is running at http://localhost:9200, in an index named after the beat used (filebeat, in our case).

sudo /etc/logstash --configtest -f /etc/logstash/conf.d/

About
Latest Posts

Gary Woodfine

Technical Director at threenine.co.uk

Gary is Technical Director at threenine.co.uk, an independent software vendor specialising in IoT, Field Service and associated managed services,enabling customers to be efficient, productive, secure and scale-able.