In this tutorial, we are going to get a basic elasticsearch setup up and running by following some basic steps

1. - Install Elasticsearch
2. - Install Kibana
3. - Basic configurations
4. - Install Xpack
5. - Install Logstash
6. - Testing

 

 

The latest elasticsearch requires minimum java version of  1.8 to function well. In order to continue with this tutorial, we need to verify that our java version is atleast 1.8. 

$ java -version

we should get something like this

If not, we need to install java 1.8

$ sudo add-apt-repository ppa:openjdk-r/ppa
$ sudo apt-get update
$ sudo apt-get install openjdk-8-jdk

Check the java version again to confirm.

Elasticsearch is a distributed, JSON-based search and analytics engine designed for horizontal scalability, maximum reliability, and easy management. There are different ways of installing elasticsearch. For this tutorial, we are going to install elasticsearch using the DEB package. For this, we download the DEB package and install

$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.3.0.deb
$ sudo dpkg -i elasticsearch-5.3.0.deb

 

 

Kibana is an extensible user web interface for the Elastic Stack. Kibana has an extensive use case and these includes configuration, management, development, etc . Just like elasticsearch, there are different ways of installing kibana. For this tutorial, we are going to install kibana using the DEB package. For this, we download the DEB package and install

$ wget https://artifacts.elastic.co/downloads/kibana/kibana-5.3.0-amd64.deb
$ sudo dpkg -i kibana-5.3.0-amd64.deb

Download the packages according to your system architechture!! visit https://www.elastic.co/downloads/kibana for the different packages

---

Hopefully everything went well up to this point

Let's start up our elasticsearch and make a test query to verify our installation.

$ sudo service elasticsearch start
$ curl -X GET 'http://localhost:9200'

we should get a response like this

 

If you did, congratulations! else, google and stackoverflow to the rescue!! also contact me if neccessary..

Now, we comfirmed all is working well, lets stop the elasticsearch service and continue.

$ sudo service elasticsearch stop

 

 

During the installation, elasticsearch is usually installed in /usr/share/elasticsearch/ with its configuration files placed in /etc/elasticsearch and its init script added in /etc/init.d/elasticsearch. 

To make sure elasticsearch and kibana starts with the server,

$ sudo update-rc.d elasticsearch defaults
$ sudo update-rc.d kibana defaults

Now to the basic configuration. However, before we get into that, lets first get a quick introduction to the configuration files. The Elasticsearch configuration files are located in the /etc/elasticsearch directory. The configuration files are :

• elasticsearch.yml — This is the "kinda" the most important configuration file for the elasticsearch server. Almost all options, except those for logging, are stored here.

• log4j2.properties — This is where the configuration for logging is stored.

Now we are done with the introduction, lets start configuring our server. The first file we would like to modify is the elastcsearch.yml.

$ sudo vi /etc/elasticsearch/elasticsearch.yml

First, we want to name our cluster. By default, the cluster is named elasticsearch. To change it, we uncomment or add the line

cluster.name: tutorial-cluster

setting the cluster name to the name you want. Make sure that you don’t reuse the same cluster names in different environments, otherwise you might end up with nodes joining the wrong cluster. You can also change the node name (name of the current server).

Depending on the size of the data you want to host on your server, you might want to change the location of your data and log files which are defaulted to /var/data/elasticsearch and /var/log/elasticsearch respectively. To do this uncomment or add the following.

path.data: /media/different_media
path.log: /media/different_media

Note: Make sure elasticsearch has access to the media!! It is best to set elasticsearch as the owner of the directory

Other important options are index.number_of_shards , index.number_of_replicas and bootstrap.memory_lock. index.number_of_shards determines into how many pieces (shards) the index will be split into. index.number_of_replicas is the number of replicas which will be spread across the cluster. More replicas makes searching faster while more shards enhances the indexing performance. By setting bootstrap.memory_lock to true, none of the JVM is ever swapped out to disk. However, you will need to make some additional changes see https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html#mlockall for the additional changes needed. For this tutorial, we will leave that for now and set the following

bootstrap.memory_lock: true
index.number_of_shards: 1
index.number_of_replicas: 0

Now, to set up some basic security, you should consider binding elasticsearch to an ip not accessable from outside or an ip address you can trust. This is to prevent unauthorized access to it and the server/virtual machine on which it is running. In this tutorial we will bind it to the localhost. For this, you need to uncomment or add this line

network.bind_host: localhost

Also, for additional security you can disable dynamic scripts which are used to evaluate custom expressions. To do this, uncomment or add this line

script.inline: false
script.stored: false

Now we are throught with the basic setup, we can save the file and exit. To exit from vim, press escape and type

:wq!

And then enter.

For more about the extra configurations. see https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html Now we can continue with  installing the other packages.

 

 

X-Pack is an extension that integrates handy features you can trust across the Elastic Stack. This includes additional security, monitoring and reporting. We need to install x-pack extension unto elasticsearch and kibana.

$ /usr/share/elasticsearch/bin/elasticsearch-plugin install -b x-pack
$ /usr/share/kibana/bin/kibana-plugin install x-pack

The later might take some time.

After installation, x-pack would add some restrictions to elasticsearch by adding users and password permission. Add new user or modify the default user in kibana (will be discussed further below). The default user and password is:

user: elastic
password: changeme

 

 

Logstash is a dynamic data collection pipeline with an extensible plugin ecosystem and strong Elasticsearch synergy. This plugin is useful for moving data across different platforms (e.g. From file to elasticsearch, from database to elasticsearch, e.t.c) by using the logstash config files. I will add a tutorial on how to transfer data from postgresql to elasticsearch. We will install logstash using the DEB file:

$ wget https://artifacts.elastic.co/downloads/logstash/logstash-5.3.0.deb
$ sudo dpkg -i logstash-5.3.0.deb

Congratulations! This should complete the basic setup. Now we need to verify that our elasticsearch service works well.

 

 

Now let's start up the full elasticsearch service.

$ sudo service elasticsearch start
$ sudo service kibana start
$ sudo service logstash start

now if we try

curl -X GET 'http://localhost:9200'

we should get a response like this.

This is because of the additional security settings added by the xpack plugin. This is to restrict access to elasticsearch in order to prevent unauthorized access. In order to access elasticsearch we need to add the user and password.

$ curl -u elastic -XGET 'http://localhost:9200'

you will be asked to enter host password for user 'elastic'. Put in the default password "changeme". You should get a response like this.

Now do not forget to change your elastic user password and protect the credentials as it has full access to the cluster as a superuser. To change the built in kibana, logstash_system and elastic users:

curl -XPUT -u elastic 'localhost:9200/_xpack/security/user/elastic/_password'-H "Content-Type: application/json"-d '{
  "password" : "elasticpassword"
}'

curl -XPUT -u elastic 'localhost:9200/_xpack/security/user/kibana/_password' -H "Content-Type: application/json"-d '{
  "password" : "kibanapassword"
}'

curl -XPUT -u elastic 'localhost:9200/_xpack/security/user/logstash_system/_password' -H "Content-Type: application/json" -d '{
  "password" : "logstashpassword"
}'

Or do it using the web interface of kibana.  Visit https://www.elastic.co/guide/en/x-pack/current/security-getting-started.html for more on using x-pack to secure your data.

---

---

To use the kibana web interface, go to your browser and put in

localhost:9200

This should ask for the a username and password. Insert elastic as user and "changeme" as password if you haven't changed it or your new password otherwise. This will open up a web interface which you can explore. You can use the management section to change/add users and roles. For further information on using kibana, visit https://www.elastic.co/guide/en/kibana/current/getting-started.html

 

 

Using this tutorial, you should get your elasticsearch service running. For additional information, please visit the elasticsearch website. I will try to keep adding some tutorial. Contact me if you have any questions or requests. 

ENJOY YOU SERVICE !!