Fantastic Four – ELK With A Beat
Brace yourselves, winter is coming! Santa is thinking about replacing his reindeer with their bigger cousins - elks. Humor aside, if you are in search of handling large amounts of data for your application there is a very popular solution, and it's called ELK.
ELK is an acronym for three free open-source projects Elasticsearch, Logstash, and Kibana. What most people are not aware of is that ELK has added one more project to its stack, and it's called Beats. So, do we call it BELK or ELKB? Putting a new letter to the existing ELK acronym just didn’t feel right, so creators called it simply Elastic Stack.
Elastic Stack refers to Elasticsearch, Kibana, and all other possible integrations which are leveraging storing, searching, and analyzing data with enviable speed.
The heart of Elastic Stack. Scalable, distributed, RESTful search and analytics engine. It stores your data for fast and perfectly calibrated search, as well as powerful analytics.
Elastic welcomes all data types, text, numbers, geo, you name it. Full-text search is just one of many challenges which Elasticsearch helps you overcome.
If Elasticsearch is the heart, Kibana is definitely the eyes of Elastic Stack. It's a beautiful user interface that helps you navigate your Elastic Stack with ease. You can visualize your search data in many different ways while managing all members of your Elastic Stack without the need for a console.
Relying on body metaphors, Logstash would probably be the digestive system. Maybe too much? Well, it actually ingests and transforms data from logs, data stores, web apps, AWS services, etc.
With a rich library of filters, it can parse, identify and transform all sorts of data to a more common format for easier analysis.
Fortunately, this author ran out of body metaphors, so let's just get to what is Beats. Beats represent data shippers. Data shippers send data from a number of machines or systems to Logstash (if we need more muscle for processing data) or Elasticsearch.
Not getting into too much detail, it’s interesting to mention the creative names of all available Beats. There are seven official Beats: Filebeat, Metricbeat, Packetbeat, Heartbeat, Winlogbeat, Functionbeat, and Auditbeat. Besides them, here are a lot of community beats for users to download, install, and modify if needed. For creating customized beats for any type of data you’d like to send, you can use the Libbeat framework.
Running multiple applications across a number of servers result in a lot of data to analyze. One of the most critical but often overlooked data sources are logs. Log files are usually unstructured and difficult to read. Using Elastic Stack you can quickly analyze and identify crucial parts of the log data.
Having the opportunity to quickly locate needed information in a massive log history, is crucial for preventing and solving errors, as well as improving your system.
Elastic Stack is not limited to only Beats and Logstash as integrations, there are tons of other integrations available.
If you want to make your web content searchable, you can set up a native Elastic web crawler to index your website.
If you want to write solutions in your preferred language, you can use Elastic language clients.
If you want to send data to Elastic Cloud, you can do so with integrations on AWS, Microsoft Azure, and Google Cloud.
If you want to fix your car, well, you probably can’t do that with any of the integrations.
Logstash requires Java Virtual Machine (JVM) to run. Which can be the main cause of serious memory consumption, especially when running multiple pipelines. To prevent performance issues or data loss you can:
- Add a Buffer – queueing layer before Logstash (Kafka, Redis, RabbitMQ)
- Enable Persistent Queues – a built-in option in Logstash that allows you to store data in an internal queue on a disk, disabled by default.
- Enable Dead Letter Queues – a built-in option in Logstash for storing events that could not be processed on disk, disabled by default.
This is probably the longest song ever. But if you wish to find out more you can read about Elasticsearch here: When to use Elasticsearch. See you in our next blog! :)