Send Operating System Data into Elastic Stack (ELK Stack)

1. Overview

In this quick tutorial, we’ll discuss how to send OS-level metrics into Elastic Stack. As a reference, we’re going to be using an Ubuntu server here.

We’ll use Metricbeat to collect data from the Operating System and send them periodically to Elasticsearch.

If you’re interested in sending other types of data into an ES instance, we discussed JMX data and Application Logs before.

2. Install Metricbeat

First, we need to download and install the standard Metricbeat agent – on our Ubuntu machine:

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-6.0.1-amd64.deb
sudo dpkg -i metricbeat-6.0.1-amd64.deb

After installation, we need to configure Metricbeat to send data to Elasticsearch by modifying metricbeat.yml found at “/etc/metricbeat/” (on Ubuntu):

output.elasticsearch:
  hosts: ["localhost:9200"]

Then, we can customize the metrics we want to track by modifying /etc/metricbeat/modules.d/system.yml:

- module: system
  period: 10s
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary

Finally, we’ll start our Metricbeat service:

sudo service metricbeat start

3. Quick Check

To make sure Metricbeat is sending data to Elasticsearch, do a quick check of the indices:

curl -X GET 'http://localhost:9200/_cat/indices'

Here’s what you should get:

yellow open metricbeat-6.0.1-2017.12.11 1 1  2185 0   1.7mb   1.7mb

Now, we’ll create new index from ‘Settings’ tab with pattern ‘metricbeat-*‘

4. Visualize OS Metrics

Now, we’ll visualize our memory usage over time.

First, we’ll create a new search – to separate our memory metrics – on our ‘metricbeat-*‘ index with the following query with the name ‘System Memory’:

metricset.name:memory

Finally, we can create a simple visualization of our memory data:

• Navigate to ‘Visualize’ tab
• Choose ‘Line Chart’
• Choose ‘From Saved Search’
• Choose ‘System Memory’ search we just created

For Y-axis, choose:

• Aggregation: Average
• Field: system.memory.used.pct

For X-axis, choose Aggregation: Date Histogram

5. Conclusion

In this quick and to-the-point article, we learned how to send OS-level data into an Elastic Stack instance, using Metricbeat.