Install Kibana4, Logstash, Logstash-forwarder and Elasticsearch on Debian 8 Jessie
Here is a powerful solution to centralize logs:
Each client will securely forward their logs with logstash-forwarder to the logstash server. Logstash will process logs into Elasticsearch database.
Finally Kibana will be the web interface to check logs and display graphics.

In this example we will forward Apache logs.

Install java:

apt-get install openjdk-7-jre-headless

Add the IP server in hosts:

nano /etc/hosts
server_IP   elkserver elkserver

Install Elasticsearch:

Add elasticsearch to trusted repos and install:

wget -O - | apt-key add -
echo 'deb stable main' | tee /etc/apt/sources.list.d/elasticsearch.list
apt-get update
apt-get install elasticsearch

Configure Elasticsearch

nano /etc/default/elasticsearch
# Run Elasticsearch as this user ID and group ID

# Heap Size (defaults to 256m min, 1g max)

# Heap new generation

# max direct memory

# Maximum number of open files, defaults to 65535.

# Maximum locked memory size. Set to "unlimited" if you use the
# bootstrap.mlockall option in elasticsearch.yml. You must also set

# Maximum number of VMA (Virtual Memory Areas) a process can own

# Elasticsearch log directory

# Elasticsearch data directory

# Elasticsearch work directory

# Elasticsearch configuration directory

# Elasticsearch configuration file (elasticsearch.yml)

# Additional Java OPTS

# Configure restart on package upgrade (true, every other setting will lead to not restarting)
nano /etc/elasticsearch/elasticsearch.yml elasticsearch_clustername
systemctl enable elasticsearch
systemctl restart elasticsearch

check logs:

tail /var/log/elasticsearch/elasticsearch_clustername.log

check ports 9200 and 9300:

netstat -ltap | egrep -e "9200|9300"

Install Logstash

Create keys for lumberjack:

mkdir /var/lib/logstash/{certs,private}
chmod 700 /var/lib/logstash/private
chown logstash:logstash /var/lib/logstash/{certs,private}
cd /var/lib/logstash/
openssl req -subj '/CN=elkserver/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

Add repo and install:

echo 'deb stable main' | tee /etc/apt/sources.list.d/logstash.list
apt-get update
apt-get install logstash
nano /etc/logstash/conf.d/logstash.conf
input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/var/lib/logstash/certs/logstash-forwarder.crt"
    ssl_key => "/var/lib/logstash/private/logstash-forwarder.key"

filter {
  if [type] == "apache" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }

output {
  if !("_grokparsefailure" in [tags]) {
    elasticsearch {
      cluster   => "elasticsearch_clustername"
systemctl enable logstash
systemctl restart logstash
tail -f /var/log/logstash/logstash.log

check ports 5000:

netstat -ltap | grep 5000

Install logstash-forwarder on your clients

nano /etc/ssl/certs/logstash-forwarder.crt

paste output of /var/lib/logstash/certs/logstash-forwarder.crt from elkserver

Add repo and install:

echo 'deb stable main' | tee /etc/apt/sources.list.d/logstashforwarder.list
apt-get update
apt-get install logstash-forwarder
nano /etc/logstash-forwarder.conf
"network": {
"servers": [ "elkserver:5000" ],
"ssl ca": "/etc/ssl/certs/logstash-forwarder.crt",
"timeout": 15
"files": [
     "paths": [ "/var/log/apache2/*.log" ],
     "fields": { "type": "apache" }
systemctl enable logstash-forwarder
systemctl restart logstash-forwarder
tail /var/log/logstash-forwarder/logstash-forwarder.err
 2015/05/19 18:14:04.558895 Registrar: processing 8 events

Back to the server elkserver
Install Kibana

Download and uncompress:

cd /opt &&
wget &&
tar xvf kibana-4.0.2-linux-x64.tar.gz &&
mv kibana-4.0.2-linux-x64 kibana &&
rm kibana-4.0.2-linux-x64.tar.gz

Get init scripts

cd /etc/init.d &&
wget &&
chmod +x kibana4
nano /opt/kibana/config/kibana.yml
host: "elkserver"
systemctl enable kibana4
systemctl restart kibana4

check ports 5601:

netstat -ltap | grep 5601

Go to http://elkserver:5601/
Untick Index contains time-based events

Click on Discover
Add some fileds Fields:
Now we starting to have something!

One thought on “Install Kibana4, Logstash, Logstash-forwarder and Elasticsearch on Debian 8 Jessie

  1. Daztingo

    Hello, and thanks for your tutorial (and sorry for my poor english)

    I followed your whole tutorial except for the part with logstash forwarder, because i’m trying to make it works only in local for moment. So i modified my logstash.conf as followed :
    input {
    file {
    path => “/home/adminsi/syslog”
    type => “syslog”
    start_position => “beginning”
    syslog { }
    output {
    elasticsearch { host => localhost }
    stdout { codec => rubydebug }

    But when i restarted logstash, nothing happened, ES seems to get no logs. I tried this :
    curl 0:9200/_cat/indices

    And i got this as answer :
    yellow open .kibana 1 1 2 0 4.7kb 4.7kb

    I don’t know what went wrong, if u got any idea it might be really helpfull !

    Thank you again for this tutorial !

Leave a Reply

Your email address will not be published. Required fields are marked *