All About Elastic Stack 6
ELK Stack or Elastic Stack 6 commonly known as (Elasticsearch, Logstash, Kibana) is a powerful tool not only for driving search on big websites, but also for analyzing big data sets in a matter of milliseconds! It is a progressively popular technology, and a valuable skill to have in today’s job market.
Elastic Stack 6 was released last November, and now it is a good time to assess whether to upgrade. To help you folks make that call, we are going to take a look at some of the key changes included in the different components in the stack and review the main breaking changes.
The main changes to Elastic Stack are intended to boost performance, improve consistency, and make retrieval easier. The most significant changes in Logstash are focused on allowing the users to run multiple self-contained pipelines on the same JVM. There are no main changes to Kibana, but a comparatively large amount of minor usability improvements was added.
What You Need to Know
Elasticsearch 6
Changes to Elasticsearch are typically internal and should not require most organizations to tailor how they configure the Elasticsearch, with the big exception being the change to mapping types.
Sparse Doc Values
A sparse values situation results in the use of a huge amount of disk space and file-system cache. The revolution to Lucene 7 lets Elasticsearch support sparse doc values, a new encoding format that minimizes the disk space and improves query throughput.
Upgrades
Updating to the new Elasticsearch version is made easier with a series of upgrade improvements that target at tackling some of the traditional sprints facing upgrade procedures.
A new restart feature denies the need for a full cluster restart and thus reduces downtime. Elasticsearch 5.x indices will be able to be searched using cross-cluster searching: a new approach to cross-cluster operations that change the traditional tribe-node based approach. Deprecation logs have been reinforced with important information on breaking changes.
Logstash 6
Logstash was initially proposed to handle one type of event per instance, and previous to this version, each Logstash instance supported only a single event pipeline. Users can avoid this restriction using conditional statements in the configuration, which often leads to a new set of problems.
Logstash now supports native support for multiple pipelines. These pipelines are defined in a pipeline.yml file, which is loaded by default.
A new pipeline viewer now lets users monitor and analyze Logstash pipelines in Kibana. Pipelines are displayed on a graph where each component is accompanied by relevant metrics. I explored the pipeline viewer in this article.
Kibana 6
The major changes to the UI include a new CSV export option, new colors for better contrast, and enhanced screen reading and keyboard navigation.
Dashboarding in Kibana has two new features. A new full-screen mode was added when viewing dashboards to the users. In addition, the new dashboard also allows administrators to share dashboards securely.
We are going to end this part with a summary that needs to be considered before upgrading. Keep in mind this is a partial list of breaking changes only.
Elasticsearch
The main breaking change in this version is the slow removal of mapping types from indices, so Elasticsearch 6.0 will be able to read indices created in version 5.0 or above only. Elasticsearch 6.0 requires a re-indexing afore full functionality can be achieved; this is because of a number of changes to the Elasticsearch indices. These are accompanied by changes to the CAT, Document, and Java APIs and the Search and Query DSL, as well as REST changes. A result is a number of key changes that affect Elasticsearch deeply, tailoring everything from the structure of queries and scripts to how internal components communicate.
Logstash
Changes in Logstash 6 include several breaking changes, with a number of changes to the input and output options. There are also several plugin changes and a change to the config.reload.interval configuration option, which now uses time value strings instead of millisecond time values.
Kibana
To transfer existing Kibana installations to Kibana 6.0, the Kibana index needs to be re-indexed.
The Verdict
There are pretty a large number of changes in Elastic Stack 6.0. Many of these changes stem from the upgrade to the Lucene 7 database engine, but just as many are part of a general push towards increased proficiency and performance.
Elastic Stack 6.0 also brings significant security changes and improvements. This follows the needs of users deploying Elastic Stack in production environments, where complex security requirements are increasingly the standard.
The major version of the stack comes with a need to re-index, changes to the index structure and a number of configuration modifications to various plugins should come as no surprise. Migration from former versions will need to be planned carefully and, above all, tested.