Elastic News

• 

  Elastic Cloud Enterprise 2.1.0 Released
  • sorangutan  

  1
  1
  Posts
  66
  Views

  

  Today we are proud to unveil the second release in the 2.X series of Elastic Cloud Enterprise (ECE) - ECE 2.1. This release doubles down on the flexible deployment configurations introduced within the 2.X series and puts even more power in your team’s hands with the following features:

  • Ability to deploy and manage Elastic APM Server as part of your deployment templates
  • Securely search across multiple deployments with Cross Cluster Search
  • More security for your deployments with Traffic Management and Keystore support
  • Support for additional operating systems with SLES 12

  Elastic APM Server for all your deployments

  Today, our customers and the community use Elastic APM to get rich insights into application performance, and bring visibility into their distributed workloads across a number of supported languages. ECE 2.1 now makes the journey of adding APM insights even easier by allowing you to manage Elastic APM Server as part of any deployment of version 6.3 and higher (including the latest Elastic APM 6.6 version).

  

  This new management, paired with ECE’s powerful ability to create hot-warm architecture with automated index curation, means it’s now super easy to centralize your APM, logs and metrics data within ECE 2.1.

  Search across multiple deployments with Cross Cluster Search (CCS)

  Having a single, large deployment for a large team encompassing several use cases can be problematic as there may be differences in team SLA’s, upgrade cycles, security requirements, and backup needs. These challenges are also heightened by an inability to understand fair resource usage for multi-use cases in a single cluster. The solution - split this monolithic deployment into smaller, more manageable deployments that you can independently scale, manage, and monitor with ECE.

  With Cross Cluster Search in ECE you no longer have to worry about losing visibility across multiple deployments, and can now designate a single deployment in an ECE environment to use Cross Cluster Search on top of multiple other deployments.

  More security for your deployments with Traffic Management and Keystore support

  Every deployment inside Elastic Cloud Enterprise already comes with important security features such as node to node encryption, RBAC, field level security, and integration with LDAP/AD/SAML. With ECE 2.1, we’ve added Traffic Management capabilities to each deployment, allowing you to specify trusted IP addresses that can access your deployments. Traffic Management also comes with viewable, enhanced auditing available when deployment access is blocked from non trusted IPs.

  

  In addition to Traffic Management, ECE 2.1 also includes Elasticsearch Keystore support that gives you more flexibility when specifying sensitive settings.

  Support for SLES 12

  As customers continue to offer Elasticsearch as a Service with ECE, a common request has been to support SLES. We are happy to report that SLES 12 is now officially supported with ECE 2.1. A full list of supported operating systems can be found on our support matrix.

  There are more features to check out in ECE 2.1 so be sure to check out the release notes, and if you want to try it out yourself, get started by downloading a 30-day free trial.

  
  
  https://www.elastic.co/blog/elastic-cloud-enterprise-2-1-0-released
• 

  Logstash 6.6.0 Released
  • sorangutan  

  1
  1
  Posts
  54
  Views

  

  Logstash 6.6.0 has been released with new features and enhancements. This release introduces the HTTP filter plugin, the Memcached filter plugin and the experimental Java Plugin API.
  

  
  

  Java Execution Engine

  The Java execution engine is GA in this release after months of being experimental. Since we last blogged about this feature we have had feedback from the field that confirms higher performance, reduced memory usage, and lower config startup and reload times. You still need to opt-in to (turn on) Java execution and we have updated the Logstash documentation to explain this better. Java execution will become the default engine in 7.0.

  
  

  Index Lifecycle Management  (ILM)

  In 6.6.0, Elasticsearch introduces the Index Lifecycle Management feature.  You can control how indices are managed as they age, and specify, for example, a maximum size or age before an index should be automatically rolled over into a new index.
  We updated the Elasticsearch output to suit. It can automatically create a default lifecycle policy that will automatically rollover an index after it reaches 50GB or is 30 days old. It can also work with more complex policies that have been previously created, which can perform more advanced lifecycle operations, such as deleting the index when it is no longer needed for retention, or moving the index to less performant hardware after a certain amount of time.

  
  

  Worker Thread Naming

  We fixed a regression whereby the worker thread names were not being set in the same way that they were in 6.3. This made interpretation of "hot threads" and stalled shutdown reports really difficult, with this bug fix it is much easier to see that workers could be "hot" or stalled on shutdown.

  
  

  Native support for Java plugins

  This long awaited feature is released as experimental. Dan Hermann has written a detailed blog post on this feature detailing how, as a Java developer, you can create Logstash plugins without needing to learn Ruby. With this API, Java developers should be able to write Logstash plugins using idiomatic Java code free of any dependencies on Ruby and its toolchain and ecosystem that seamlessly inter-operate with Ruby plugins and yield high performance in the Java execution engine.

  
  

  HTTP Filter

  This is an enrichment filter, it allows you to invoke HTTP calls with parameters from an event to external APIs and have the response inserted into that event. Thanks go to Lucas Henning who created the logstash-filter-rest, upon which this filter is based.

  Here is an example calling the convert function on the neutrinoapi service.

  
  

   http {
      url => "https://neutrinoapi.com/convert"
      body => {
        "user-id" => "auser"
        "api-key" => "anapikey"
        "from-value" => "%{millimeters}"
        "from-type" => "mm"
        "to-type" => "inch"
      }
      verb => "POST"
      body_format => "json"
      target_body => "neutrinoapi_convert_mm_to_in"
      add_tag => ["from_api"]
      id => "http-neutrino-lookup"
    }
    mutate {
      # needed because the neutrino response JSON object includes unneeded extra key/values
      copy => {"[neutrinoapi_convert_mm_to_in][result]" => "[inches]"}
      remove_field => ["neutrinoapi_convert_mm_to_in"]
      id => "copy-to-inches"
    }

  For more details on the plugin and usage instructions, check the documentation page.
  

  
  

  Memcached Filter

  This is the enrichment filter you have all been waiting for. Why? It enables K/V set as well as get, meaning, for example, you can have one pipeline extracting and loading memcached with key/values and another pipeline enriching events via the same keys.

  An example of setting:

   memcached {
      hosts => ["localhost"]
      namespace => "neutrinoapi_convert"
      set => {
        "[neutrinoapi_convert_mm_to_in][result]" => "%{millimeters}mm"
      }
      id => "memcached-set"
    }

  
  

  An example of getting:

   memcached {
      hosts => ["localhost"]
      namespace => "neutrinoapi_convert"
      get => {
        "%{millimeters}mm" => "[inches]"
      }
      add_tag => ["from_cache"]
      id => "memcached-get"
    }

  
  

  For more details on the plugin and usage instructions, check the documentation page.

  
  

  Other Plugin Improvements

  Logstash 6.6.0 includes a number of improvements to some of our bundled plugins. Some highlights:

  
  

  • The Aggregate filter has some improvements such as custom timeouts, metrics, metadata additions in the code block.
  • The Elasticsearch filter has improvements in the client connection.
  • The Gelf input has had a general overhaul around shutdown, socket handling json parsing and code quality.
  • The Kafka input and output have version 2.1.0 of the Kafka Client and a new setting "ssl.endpoint.identification.algorithm".
  • The Elasticsearch output now supports Index Lifecycle Management for Elasticsearch versions 6.6.0 and above running with a Basic License(or higher). 
  • The Tcp output no longer needs a certificate/key pair when enabling ssl

  
  Download and Release Notes
  

  You can find the full list of the changes in the release notes.

  
  

  • Download 6.6.0
  • Release Notes for 6.6.0

  
  

  Feedback

  If you want to try the new features added in Logstash 6.6.0, please download it, install it, and let us know what you think on Twitter (@elastic) or in our forum. Enjoy!
  
  

  
  
  https://www.elastic.co/blog/logstash-6-6-0-released
• 

  Elastic APM 6.6.0 Released
  • sorangutan  

  1
  1
  Posts
  88
  Views

  

  Elastic APM 6.6 is here, and it's brought a lot of enhancements. Let's get right into it...

  Metrics

  Infrastructure and application metrics are important sources of information when debugging production systems. As part of the 6.6 release, the latest APM agents will automatically pick up basic metrics, such as system and process-level CPU and memory metrics. The metrics will be visualized in the new service Metrics tab in the APM UI.

  

  Metrics is supported by all of our backend agents: Java, Node.js, Go, Python and Ruby:

  Using the search bar, you can easily filter down the metrics shown for specific hosts or containers. There are already additional metrics sent by some agents that you can graph using the “Visualize” tools in Kibana. For example, the Java agent sends JVM metrics and the Go agent sends metrics on the Go runtime. Make sure to use the apm-*-metric-* index pattern to graph these.

  Distributed tracing is now generally available

  We shipped support for distributed tracing in beta 6.5 release. Distributed tracing is becoming a key feature in modern application performance monitoring as application architectures are shifting from monolithic to more distributed and service-based.

  Distributed tracing allows APM users to automatically trace requests all the way through the service architecture, and visualize those traces in one single view in the APM UI. In the 6.6 release, we've polished the UI and are now removing the beta label and marking it as generally available. Distributed tracing is supported by all APM agents, including the JavaScript RUM agent.

  Supporting OpenTracing and Trace Context

  Elastic APM supports the W3C Trace Context working group's effort to standardize the HTTP header format for distributed tracing. Our agent implementations closely follow the Trace Context draft specification, and we intend to fully support the final specification. Additionally, all agents now have OpenTracing compatible bridges.

  Integrations with Infrastructure and Logging

  We recently shipped two new exciting products in early Beta: Logging and Infrastructure. These are dedicated apps for the logging and infrastructure metrics use cases. The logging app enables you to live stream log events and the Infrastructure app provides you with a point in time snapshot of your infrastructure. For more details, check out this Logs and Infrastructure UI blog post.

  Having logs, metrics, and APM traces stored and indexed in Elasticsearch is super powerful. For example, a trace might be slow due to issues with the container it was served from, not the code itself. Being able to quickly correlate data sources like infrastructure metrics, logs and traces enables you to debug the root cause much faster.

  In the APM UI, we've just released the first integration between the APM, Logging and Infrastructure apps. When looking at a trace, you can now quickly jump to the host or container metrics and logs by clicking the Actions menu. Here's a screenshot of the integration points:

  

  Java agent updates

  The Java agent has released two great new features: wildcard tracing and easier agent attachment.

  Wildcard method tracing

  The Java agent doesn't automatically trace methods. You can manually annotate methods in the source code to have them instrumented, but now we're adding a new way of tracing methods.

  trace_methods takes a list of methods to automatically instrument. The syntax is modifier fully.qualified.class.Name#methodName(fully.qualified.parameter.Type). You can use wildcards for the class name, the method name and the parameter types. The * wildcard matches zero or more characters. Specifying the parameter types is optional. The modifier can be omitted or one of public, protected, private or *.

  A few examples:

  • org.example.MyClass#myMethod
  • org.example.MyClass#myMethod()
  • org.example.MyClass#myMethod(java.lang.String)
  • org.example.MyClass#myMe*od(java.lang.String, int)
  • private org.example.MyClass#myMe*od(java.lang.String, *)
  • * org.example.MyClas*#myMe*od(*.String, int[])
  • public org.example.services.*Service#*

  We're planning to post an in-depth blog post on method level tracing with Java in the near future. Until then, check out the trace_methods documentation.

  Easier agent attachment

  This experimental installation method does not require you to alter the configuration of your application server and can be used to conveniently instrument all JVMs on a particular host. This works by running a small background program which attaches the Elastic APM Java agent to a specific JVM or to all running and starting JVMs. This means there is a lot less you have to configure and you don't have to touch the configuration files of your application server at all.

  For more details, check out the automatic agent attacher documentation.

  Try out APM on Elasticsearch Service

  We are also excited to announce that Elastic APM is now supported on Elasticsearch Service and Elastic Cloud Enterprise, bringing you more options on how deploy and manage APM Server. Be sure to try out all the new features for Elastic APM by trying it out on our Elasticsearch Service. Start now with a 14-day Elasticsearch Service trial and a free 512 MB APM Server as part of every deployment.
  
  https://www.elastic.co/blog/elastic-apm-6-6-0-released
• 

  Importing CSV and Log Data into Elasticsearch with File Data Visualizer
  • sorangutan  

  1
  1
  Posts
  126
  Views

  

  Introduced in Elastic Stack 6.5 is the new File Data Visualizer feature. This new feature allows a user to upload a file containing delimited (e.g. CSV), NDJSON or semi-structured text (e.g. log files) where the new Elastic machine learning find_file_structure endpoint will analyse it and report back its findings about the data. This includes a suggested ingest pipeline and mappings which can be used to import the file into elasticsearch from the UI.

  The aim of this feature is to enable users who wish to explore their data with Kibana or Machine Learning to easily get small amounts of data into Elasticsearch without having to learn the intricacies of the ingest process.

  A great recent example is this blog post by a member of Elastic’s marketing team, who doesn’t have a development background. Using the File Data Visualizer he was easily able to import earthquake data into Elasticsearch to help him explore and analyse earthquake locations using geo_point visualizations in Kibana.

  Example: Importing a CSV File into Elasticsearch

  The best way to demonstrate this functionality is to step through an example. The following example will use data from a CSV file containing imaginary data from a flight booking website. Here we’re showing only the first five lines of the file to give you an idea of what the data looks like:

  time,airline,responsetime
  2014-06-23 00:00:00Z,AAL,132.2046
  2014-06-23 00:00:00Z,JZA,990.4628
  2014-06-23 00:00:00Z,JBU,877.5927
  2014-06-23 00:00:00Z,KLM,1355.4812
  

  Configure the CSV Import within File Data Visualizer

  The File Data Visualizer feature can be found in Kibana under the Machine Learning > Data Visualizer section. The user is presented with a page which allows them to select or drag and drop a file. As of 6.5, we’re limited to a maximum file size of 100MB.

  

  Underneath this Summary section is the fields section, this should be familiar with people who have used the original Data Visualizer feature. 

  We can see that the types of the three fields have been correctly identified, with some high level statistics listed for each one. The top 10 occuring values are listed for each field. For responsetime, which has been identified as being a numerical field the min, median and max values are also displayed.

  

  The user will probably have some domain knowledge and may want to rename these fields to something better, they can do this by using the Override settings button.
  

  

  Once we’re happy with these settings, we can hit the Import button located at the bottom left of the page.

  Import the CSV Data into Elasticsearch

  This takes us to the Import page where we can import the data into Elasticsearch. Note, this feature is not intended for use as part of a repeated production process, but just for initial exploration of the data. The main reason being the lack of automation options but also because this feature is currently experimental.

  There are two modes for importing. Simple, where all the user need supply is a new unique index name and choose whether you want to also create an index pattern.

  

  • Index settings - By default, no additional settings are needed for the creation of the index and the import, but the option to customise the index settings is still made available.
  • Mappings - The find_file_structure provides a mappings object based on the fields and the types it has identified. For a list of possible mappings, see our Elasticsearch mapping documentation.
  • Ingest pipeline - The find_file_structure provides a default ingest pipeline object. This will used when ingesting the data and can be used for uploading any additional data.

  Within 6.5, we only allow the creation of new indices and not the ability to add data to an existing index to reduce the risk of damaging the index.

  Clicking the import button starts the import process. This consists of various numbered steps:

  1. Processing file - Turning the data into NDJSON documents so they can be ingested using the bulk api
  2. Creating index - Creating the index using the settings and mappings objects
  3. Creating ingest pipeline - Creating the ingest pipeline using the ingest pipeline object
  4. Uploading the data - Loading data into the new Elasticsearch index
  5. Creating index pattern - Create a Kibana index pattern (if the user has opted to)

  

  Example: Importing Log Files and other Semi-Structured Text into Elasticsearch

  So far this has covered CSV data — and NDJSON is even simpler with little processing needed for import —  but what about semi-structured text? Let’s look at how the analysis of CSV data differs from typical log file data, aka semi-structured text.

  Below are three lines from a log file generated by a router.

  <190>38377: GOW45-AR002: Apr 18 08:44:02.434 GMT: %JHG_MS-6-ROUTE_EVENT_INFO: Route changed Prefix 10.156.26.0/23, BR 10.123.11.255, i/f Ki0/0/0.849, Reason None, OOP Reason Timer Expired
  <190>38378: GOW45-AR002: Apr 18 08:44:07.538 GMT: %JHG_MS-6-ROUTE_EVENT_INFO: Route changed Prefix 10.156.72.0/23, BR 10.123.11.255, i/f Ki0/0/0.849, Reason None, OOP Reason Timer Expired
  <190>38379: GOW45-AR002: Apr 18 08:44:08.818 GMT: %JHG_MS-6-ROUTE_EVENT_INFO: Route changed Prefix 10.156.55.0/24, BR 10.123.11.255, i/f Ki0/0/0.849, Reason None, OOP Reason Timer Expired
  

  When analysed by the find_file_structure endpoint, it correctly recognizes that the format is semi-structured text and creates a grok pattern to extract the fields and their types from each line. Of these fields, it also recognises which one is the time field and its format.

  

  However, this grok pattern is editable via the Override settings menu, and so we are able to correct the field names and their types.

  

  These corrected field names will then be displayed in the File stats section. Note they are ordered alphabetically, and so have changed order slightly.

  

  When imported, these new field names are added to the mappings object and the grok pattern is added to the processors list in the ingest pipeline.

  
  
  https://www.elastic.co/blog/importing-csv-and-log-data-into-elasticsearch-with-file-data-visualizer

• 

  Improving Quality Assurance Automation at Ramsey Solutions with the Elastic Stack
  • sorangutan  

  1
  1
  Posts
  67
  Views

  

  Ensuring extraordinary quality for customers while reducing risk is a passion of the Quality Assurance Team at Ramsey Solutions, and it’s my privilege to lead our Test Automation efforts here. My role includes identifying areas of opportunity where test automation can bring its most significant benefits. While still a reasonably young engineering field, I’ve been working in some form of testing and automation for over 35 years. During this adventure, I’ve recognized many challenges that surface in the craft of quality, regardless of technology or development philosophy.

  One of the most common challenges for Quality Engineers is making the mental shift from merely pursuing pass or fail test results to understanding the characteristics and trends of tests. How often does it pass or fail? What's the duration aspects of a test that executes tirelessly? How can we create tests that are accurate and actionable?

  In 2016, Ramsey Solutions sent me to attend the Google Test Automation Conference (GTAC) in Mountain View, California, to learn from some of the top automation engineers in the world. At the conclusion of that conference, Google Test Engineers imparted the following challenge to us. They said, "find a way to measure everything." These simple words planted a seed, leading to a journey of pursuing and expecting more from our testing data than ever before. You can imagine my excitement when I returned from the GTAC conference, to learn that our architecture team was working on something called a "data lake" built on top of the Elastic Stack.

  As I discovered more about what a data lake was and the opportunities it offered, the dots began to connect. Would this "data lake" allow us to start our journey to measuring everything as the Google Test Engineers had encouraged? For a little background, Ramsey Solutions first looked at the Elastic Stack to help understand customer engagement details. It was determined that Elasticsearch could outperform a database solution, and it shipped with Kibana for data visualization. But what caught my attention about Elastic was that it could be a great place to capture and analyze test automation metrics. I wanted to push our automation technology efforts beyond the temporal knowledge of did the test pass or fail and instead, look at metrics such as duration of test execution over time. 

  With the help of our team and some experimentation, our small crew of two Test Automation Engineers began populating test metrics into the data lake. These metrics included; the business unit name, application id's, test case names, the execution environment, test duration times, test results, and the exception details of why a test failed. Altogether, over a dozen metrics related to each test case were ingested into the data lake. We were now gathering meaningful data into a place that would enable us to gain new insights across our software development lifecycle. (Not to get too far ahead of myself, but having access to the Elastic Stack has enabled my team to capture nearly 2 million automated test case results in the past year, equipping our leaders and test engineers with amazing metrics related to the quality of our products and the quality of our processes.)

  Visualizing QA Test Data in Kibana

  With test data now going into the data lake, it was time to build our first visualization using Kibana. Even at this early stage of visualizing simple test duration over time, I knew we were on to something. Let’s face it, a test merely passing or failing is only one of many attributes that might give insight into the health of a software application. As an example of how pass/fails can be misleading, I discovered that a test which completes within a few seconds one moment and nearly a minute longer the next does not necessarily mean both had passed with flying colors. Technically, they both produced green (passing) test results, but the duration of the long-running test was an indicator that something was up.

  Let's take a look at some of the dashboards I've been able to create:

  Test Run Overview Dashboard

  

  The "Project - Overview" dashboard, created for our Quality Assurance Leadership, was developed to give an overview of the health of individual products by business unit. The yellow and green circles reset to black each morning and reflect (first yellow then green) the usage of test automation. These insights allow leaders to know team members are leveraging the test automation tools developed to assist them. The breakdown summaries related to automation types, result details, and exceptions encountered make this a very concise and invaluable tool for our leaders.

  Environment Overview Dashboard

  

  The "Custom Reporting - Overview" dashboard provides an easy way to give specific insights to a specific product team. This dashboard coupled with custom filters allows product teams to have a simple overview of automated test metrics, trends, and results.

  Mastering Kibana with Training

  As a quick note to the reader, I am a big believer of "use a tool as designed not as discovered" (I've even dedicated a chapter in my book — Test Automation In The Real World — to this topic because it ensures you get the maximum benefits from most tools). In early 2018, I took the Kibana Data Analysis virtual training class which was very helpful in getting my dashboards to the point where they are today. 
  

  Information Visualization is a Communications Art

  From these early learnings, we developed some basic philosophies about visualizing test data and how to obtain fine-grain, more informative data in the future. (Some of these philosophies were based on the teachings of Edward Tufte, among others.) 

  Build effective communication

  Regardless of its form, build your communication upon these two, essential principals:

  • Know your audience: Are you communicating your information to business executives or factory workers? Will they be consuming your content in a boardroom or a dashboard mounted on the lunchroom wall? Does your audience understand the dialect of your information or does it need to be translated into something more understandable?
  • What to take away: What should your audience take away from the information you’re presenting? Too many dashboards and visualizations may be nothing more than flashy wall art with no real actionable information. Ensure your audience can understand the information and respond appropriately. The take away could include areas that are performing well and others that need specific attention.

  An example of these principles in action is the "Project - Overview" Kibana dashboard (shown above) that was created for our Quality Assurance Leaders. It gives them quick insight into our test automation usage with the results broken down by product name and business channel in a language they can understand. Here's a quick zoom in: 

  
  
  https://www.elastic.co/blog/improving-quality-assurance-automation-at-ramsey-solutions-with-the-elastic-stack

• 

  This Week in Elasticsearch and Apache Lucene - 2019-01-19
  • sorangutan  

  1
  1
  Posts
  72
  Views

  

  Elasticsearch Highlights

  Leniency in request handling

  We are trying to remove leniency throughout Elasticsearch. It makes for a more predictable experience for our users, where Elasticsearch does what the user thinks they are trying to do, or otherwise tells them when something seems amiss. An example we have recently decided to address is HTTP requests with a body, where the endpoint doesn’t actually consume the body. We have recently merged a change to reject HTTP requests with a body when the endpoint doesn’t use the body. This is a breaking change that will ship in 7.0.

  Watcher

  We added a whitelist to the HttpClient that is used by Watcher. This allows admins to allow only specific URL's to be called from Watcher. For passivity this defaults to all, but in future versions will require this to be explicitly set. (#36817)

  Index management

  We worked on a couple of PRs adding support for various index states to the index management UI. One added marking indices with a "Rollup" badge when they are created by a rollup job. We also added support for frozen indexes to index management, including freeze/unfreeze actions in the context menu and a "Frozen" badge in the index list.

  Index Lifecycle Management

  We submitted PR for the first system level index, .watch-history, to be managed by ILM. This lays the groundwork for a future system level index to be managed by ILM. The general strategy is to ship a default, but allow pre-existing policies, or changes to the cluster's policy such that we only apply the default policy if one does not already exist. The pre-existing watcher history cleaner service will be deprecated and removed. (#37443)

  Performance

  We are happy to see that our benchmarks have served as a good safety net performance regressions. This week we found a significant performance regression in the interaction between types removal in the bulk API and the deprecation logger. There is an intermediate fix for this and we have opened a follow-up issue to optimize deduplication in the deprecation logger.

  Speaking of deprecation, We have raised 2 PRs to address some performance issues uncovered in this week’s FixIt meeting whilst discussing this issue.

  Permissions to access restricted indices

  The .security index has long had restricted access; in 2.x no external access was allowed (originally) and in 5.0+ access was restricted to superusers. This was done to avoid having users with permission to read all indices suddenly have permissions to read security information such as password hashes when the native realm was introduced. While thinking about backing up the .security index, it became clear that superuser was too powerful of a role to be using for backups. Albert has embarked on lightening this restriction and we've gone through a few iterations, but have landed on a new field with the indices section of a role called allow_restricted_indices that defaults to false. As of now, the only restricted indices are the security indices but we've kept this generic so that we have freedom to address this in the future.

  TLS

  We continue to work on the custom TLS configuration for reindex by making use of the new library he added last week. Additionally, Tim will be removing TLSv1.0 from the default list of protocols for 7.0as this protocol has known security weaknesses. We have worked on a new TLS tutorial that covers encrypting internode communications. We will be working to update this tutorial to cover the new Zen2 discovery and expand the tutorial to also include setting up HTTPS.

  Total hits accuracy in Elasticsearch 7.0

  Following the response format change for search requests that allowed us to return the total hits that match a query as a lower bound, we opened a PR that will change the default for all search requests to count the total hits accurately up to 10,000. This means that any search request that matches more than 10,000 documents will now return:

  { "hits": { "total": "10000", "relation": "gte" } }

  This will be the behavior by default in 7 and beyond, "track_total_hits" must be set to true explicitly in the request to get the old behavior back.

  Alternative CSS execution

  We opened a PR to handle the merging of search responses coming from different clusters. In the alternative cross cluster search the reduction happens per shards inside each cluster first and then the responses of each cluster (the local one + the remotes) are merged to form the final response. We are also already working on the logical follow up which is the final integration of the alternative mode in the search layer.

  Geo

  We merged the minimal geo objects library into master, and are working on integrating it with the JDBC driver. Igor is also continuing to diagnose the performance problems on the new geoshapes rally test. The current theory is that the slowdown is caused by increasingly slow merging of ever-larger segments.

  Cross Cluster Replication

  We continue work on shard history retention leases, having added persistence to Lucene on commits, and opening a PR to sync retention leases to replicas This has triggered additional discussions about the durability of these leases and the guarantees that we want from the storage and replication system.

  We added a new “CCR follow info" API that reports information about all the follower indices in the cluster together with their parameters and settings from the time where the following got initiated or resumed. We have also adapted the follow stats API to return a 404 instead of 200 with empty body if there are no follow shard tasks for the requested follower indices.

  Following this work, we continued work on the Advanced Settings UI for adding a follower index. After review, the UI was revised to be a simple hide/show toggle for all settings. We also worked on PR that introduces the ability to edit a follower index. Editing refers to changing the follower index's advanced settings only. Since settings can only be changed upon resuming a follower, a warning is surfaced to the user that we will be pausing and resuming the follower in order to update it. We also added remote cluster validation to the create auto-follow pattern form and add/edit follower index forms. This presents a nice UX when

  1. There are no remote clusters
  2. When a selected remote cluster is not connected
  3. When a previously set remote cluster has been removed.

  We are adding timeout handling and rate limiting to the CCR recover from remote functionality. We have also extended the security model for the remote cluster proxy so that the recovery from remote can fetch the Lucene file chunks under the system security context.

  Changes

  Changes in 6.5:

  • SQL: Fix issue with field names containing "." #37364

  Changes in 6.6:

  • Document Seq No powered optimistic concurrency control #37284
  • Add fatal_exception field for ccr stats in monitoring mapping #37563
  • SQL: Describe aliases as views #37496
  • When removing an AutoFollower also mark it as removed. #37402
  • Use executor SAME to handle search related handlers #37427
  • SQL: Make FULL non-reserved keyword in the grammar #37377
  • Fix artifactId in plugin poms #37315
  • Geo: Do not normalize the longitude with value -180 for Lucene shapes #37299

  Changes in 6.7:

  • Create specific exception for when snapshots are in progress #37550
  • Add set_priority action to ILM #37397
  • Propagate Errors in executors to uncaught exception handler #36137
  • Update Put Watch to allow unknown fields #37494
  • Make recovery source send operations non-blocking #37503
  • Allow field types to optimize phrase prefix queries #37436
  • Change file descriptor limit to 65535 #37537
  • SQL: Add protocol tests and remove jdbc_type from drivers response #37516
  • Deprecate requests that have an unconsumed body #37534
  • SQL: Remove slightly used meta commands #37506
  • Packaging: Remove permission editing in postinst #37242
  • Prepare to make send translog of recovery non-blocking #37458
  • Update Delete Watch to allow unknown fields #37435
  • New mapping signature and mapping string source fixed. #37401
  • Restore lost @Inject annotation #37452
  • Make finalize step of recovery source non-blocking
    
    https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2019-01-19
• 

  Brewing in Beats: Adding the add_labels and add_tags processors
  • sorangutan  

  1
  1
  Posts
  81
  Views

  

  Welcome to Brewing in Beats! With these weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

  Did you know that Beats 6.5 is already available? Try it and let us know what you think.

  What's new in Beats

  Escaping HTML disabled

  In the past, we introduced support for disabling HTML escaping in the outputs JSON encoders. With #9914 and the upcoming Beats 7 releases, we will not escape HTML like symbols anymore by default.

  New processors: add_labels, add_tags

  In Beats, users can already add fields and tags to events using the global or per module local   fields, and tags settings. We recently introduced an add_labels, and add_tags processor (#9973) allowing you to add fields and tags via processors as well. Using these processors we can add fields and tags based on conditions and event contents.

  What's new in Stack Monitoring

  Repository: elastic/kibana

  • Addresses issues with filtering tables in the new EUI monitoring tables. #27504
  • Rename the Monitoring app in Kibana to Stack Monitoring #28102
  • Ensure all monitoring charts respond to onBrush [bugfix] #28098
  • Fixing issue with EUI table filtering in monitoring UI [bugfix] #27504

  Repository: elastic/beats

  • Refactoring: Using common.Version instead of strings #9900
  • Refactoring: Elasticsearch metricbeat module #9896
  • Error handling for multiple pipelines with < ES 6.5 #10001
  • Make watcher threadpool stats optional #9909
  • Allow users to convert timezone in elasticsearch module filesets #9761

  What's new in Beats Central Management

  Repository: elastic/kibana

  Changes in master:

  • propigate errors on index creation fail #28541

  What's new with ECS migration

  Packetbeat

  Changes in master:

  • [Packetbeat] Update DNS protocol to use ECS fields #9941

  Filebeat

  Changes in master:

  • Convert Filebeat mongodb.log to ECS #10009
  • Convert Filebeat mysql.* to ECS #10008
  • Convert Filebeat nginx.error to ECS #10007
  • Fix apache fields config for ECS migration #9960
  • Convert Filebeat iis.error to ECS and add IPv6 zone support for IIS access logs #9955
  • Convert Filebeat logstash.* to ECS #9935
  • Update event.dataset to contain module and fileset name #9922
  • Convert Filebeat kibana.log to ECS #9301
  • Fixes parsing of GC entries in elasticsearch server log #9603

  Changes in 6.x:

  • Parameterizing Painless script literals #9770
  • Allow users to convert timezone in elasticsearch module filesets #9761
  • Fixes parsing of GC entries in elasticsearch server log #9603

  Other changes

  Repository: elastic/beats

  Affecting all Beats

  Changes in 6.6:

  • Do not configure aliases in unsupported Elasticsearch versions #9992

  Changes in master:

  • Do not configure aliases in unsupported Elasticsearch versions #9992
  • Fix asset registry slice generation #9986
  • Introduce add_labels and add_tags processors #9973
  • Rename process.exe to process.executable for ECS #9949
  • Rename host.name to host.hostname and add config option for name #9943
  • Pretty print exported dashboards to output #9925
  • Do not escape HTML by default anymore #9914
  • Fix config appender registration #9873
  • Allow multiple object types per field. #9772

  Changes in 6.x:

  • Do not configure aliases in unsupported Elasticsearch versions #9992
  • Fix config appender registration #9873

  Filebeat

  Changes in 6.x:

  • Parameterizing Painless script literals #9770

  Auditbeat

  Changes in 6.6:

  • [Auditbeat] Sort IPv4 addresses before IPv6 #9953
  • [Auditbeat] Add Windows to docs for host dataset #9954

  Changes in master:

  • [Auditbeat] Add Windows to docs for host dataset #9954
  • [Auditbeat] Sort IPv4 addresses before IPv6 #9953

  Changes in 6.x:

  • [Auditbeat] Sort IPv4 addresses before IPv6 #9953
  • [Auditbeat] Add Windows to docs for host dataset #9954

  Testing

  Changes in 6.5:

  • Add system tests for Central Management #8762

  Changes in master:

  • Fix and reenable ml tests #9942

  Changes in 6.x:

  • Minor fixes to error handling #9888

  Documentation

  Changes in master:

  • [Docs] Remove duplicate metricset pages #9940
  • Update kubernetes.asciidoc #9282
  • [Docs] Add topic about running Journalbeat on docker #9386

  Changes in 6.x:

  • Minor grammatical fixes to libbeats configuration file #9511
  • Update kubernetes.asciidoc #9282

  Repository: elastic/go-sysinfo

  Changes in master:

  • Add Process.PID() #37
  • Implement Processes() for Darwin #35

  Repository: elastic/ecs

  Changes in master:

  • Update PyYAML to 4.2b1 #298
  • Generate process.args as []string in Go code #296

  
  
  https://www.elastic.co/blog/brewing-in-beats-adding-the-add-labels-and-add-tags-processorrs
• 

  Logstash Lines: Support for JRuby 9.2
  • sorangutan  

  1
  1
  Posts
  57
  Views

  

  Welcome to Logstash Lines! With these weekly series, we're keeping you up to date with what's new in Logstash, including the latest commits and releases.

  Did you know that Logstash 6.5 is already available? Try it and let us know what you think.

  Logstash support of JRuby 9.2: Update

  On the road to support Java 11, Logstash must be able to run on the latest JRuby release. In the past we were prevented from upgrading from JRuby 9.1.13.0 due to multiple issues, but 9.2.5.0 seems stable enough that efforts have resumed. The JRuby 9.2 ci branch is mostly green now after Ry's work on getting newer Bundler versions to work correctly with Logstash.

  Two manual Jenkins jobs for release management

  Our Logstash CI, now has two jobs to facilitate the manual tasks required for releases. One creates a pull request with a draft for release notes: it was used as the starting point for the 6.6.0 release notes.

  The other job allows the team to update the logstash plugin versions for an upcoming release, respecting major/minor or patch constraints. The result is also a pull request with a modified gem lockfile.

  All changes

  Repository: elastic/Logstash

  • [Doc] Fix broken link in filebeat module #10286
  • Release notes draft for 6.5.5 #10284
  • Add missing ruby shebang to release tools #10283
  • Bump http filter to 1.0.0 #10281 (6.6 branch)
  • Update lock file for 6.6 #10278
  • Properly set thread name in thread context #10269
  • JRuby 9.2 bundler shenanigans #10266
  • Add lockfile bump and release notes draft tools #10265 
  • Ignore some unit tests that are failing on some Windows platforms #10255
  • Abort plugin bump script if base version not found #10254

  Logstash plugin releases:

  logstash-codec-cef - 6.0.0

  • Removed obsolete `sev` and `deprecated_v1_fields` fields

  logstash-filter-cipher - 4.0.0

  • Removed obsolete iv field

  logstash-filter-http - 1.0.0

  • Minor documentation fixes

  logstash-filter-memcached - 0.1.1

  • The plugin common options (e.g., `add_field`, `add_tag`, etc.) are now correctly only invoked when the plugin successfully gets one or more values from, or sets one or more values to memcached (#4)
  • Fix links to argument types in documentation (#3)

  logstash-input-kafka - 9.0.0

  •   Removed obsolete `ssl` option

  logstash-input-tcp - 6.0.0

  • Removed obsolete `data_timeout` and `ssl_cacert` options

  logstash-mixin-rabbitmq_connection - 6.0.0

  • Removed obsolete options `verify_ssl`, `debug`, `tls_certificate_path` and `tls_certificate_password`

  logstash-output-kafka - 8.0.0

  •  Removed obsolete `block_on_buffer_full`, `ssl` and `timeout_ms` options

  logstash-output-redis - 5.0.0

  • Removed obsolete fields `queue` and `name`
  • Changed major version of redis library dependency to 4.x

  logstash-output-sqs - 6.0.0

  •   Removed obsolete fields `batch` and `batch_timeout`
  • Removed workaround to JRuby bug (see more [here](https://github.com/jruby/jruby/issues/3645))

  logstash-output-tcp - 6.0.0

  • Removed obsolete field `message_format`

  Repository: elastic/logstash-docs

  Changes in versioned_plugin_docs:

  • Add http filter to versioned docs #658

  Changes in 6.5:

  • updated docs for 6.5 #654

  
  
  https://www.elastic.co/blog/logstash-lines-support-for-jruby-9-2
• 

  Infrastructure and Logs UI: New ways for Ops to interact with Elasticsearch
  • sorangutan  

  1
  1
  Posts
  94
  Views

  

  With version 6.5 of the Elastic Stack, we released two new ways to interact with your data: the Infrastructure and Logs UIs. These are both in beta in 6.5, but more about that later when I will ask for your input. For each of these, I would like to talk a little about the motivation, the user experience, and the configuration of the UI. Let’s start with the Logs UI.

  Logs UI

  Motivation

  I have heard this so many times: “Just give me the logs, I don’t need anything fancy, everything I need to know is written in the logs, I just want to read them”. We hear you. The choice is yours:

  A souped up tail -f with the most recent at the bottom experience; a visual experience with tables, charts, tag clouds, etc; or a tabular view. You should be able to work the way that is best suited to you, and the openness of the Elastic Stack supports that.

  User Experience

  Using the Logs UI is similar to tailing a log file, but with all of your logs from all of your systems available in one console. The logs stream in, and the bottom of the view is the most recent record, just like a tail -f. By default, the Logs UI shows you all of the records from all of the logs that meet the configuration criteria (more on that in the next section). When you are working an issue and decide that you don’t want all your logs, from all of your services, all together (streaming by faster than anyone could possibly read), then just change the interaction by typing in the search bar up top. For example, if I only want to see 404 errors from the Apache httpd pods with the Kubernetes label tier = “frontend”, I just start typing in the search bar and let the autocomplete help me find the right logs:

  

  Compare this to popping open a terminal, authenticating with your k8s provider, figuring out which pods you want and running kubectl logs -f … | grep 404 (or in the non-container world finding the host names, sshing in, tailing the logs, etc.). We want to make your life easier and give you your data in the way you want to see it.

  Configuration

  The Logs UI docs are in the usual place, but I will highlight some of the configuration options here. Here is the default configuration you can paste these in to config/kibana.yml and then modify from the defaults:

  xpack.infra.sources.default.logAlias: "filebeat-*"
  xpack.infra.sources.default.fields.timestamp: "@timestamp"
  xpack.infra.sources.default.fields.message: ['message', '@message']
  

  Configuration is pretty simple, out of the box the lines that show up in the Logs UI are the message field of every index that matches the filebeat-* alias. So, what if you want to add in all of your Logstash indices? Simply modify the xpack.infra.sources.default.logAlias in config/kibana.yml entry like so and restart Kibana:

  #xpack.infra.sources.default.logAlias: "filebeat-*"
  xpack.infra.sources.default.logAlias: "filebeat-*,logstash-*"
  

  Don’t forget to restart Kibana. Reopen the Logs UI and click on Stream Live and you should see all of your Logstash and Filebeat logs streaming into the Logs UI.

  Note: If you prefer to use Elasticsearch aliases then just set xpack.infra.sources.default.logAlias to the name of the alias and then update the alias as you need to. Here is the equivalent using an alias of logs

  Create the alias:

  curl -X POST "localhost:9200/_aliases" -H 'Content-Type: application/json' -d'
  {
      "actions" : [
          { "add" : { "indices" : ["logstash-*", "filebeat-*"], "alias" : "logs" } }
      ]
  }
  '
  

  Update config/kibana.yml:

  #xpack.infra.sources.default.logAlias: "filebeat-*"
  xpack.infra.sources.default.logAlias: "logs"
  

  Infrastructure UI

  Motivation

  I see three basic maturity levels of managing infrastructure:

  1. Finding out that something is broken, then opening a monitoring system to dig in.
  2. Charting key indicators for all of the systems on a big dashboard and watching that, then digging in to the problems.
  3. Automating Ops with machine learning discovering normal behavior, detecting emerging issues, and alerting Ops.

  The Infrastructure UI is designed to get Ops to the “charting key indicators on a big dashboard” stage. It's safe to say that we all need to get to the “automated with machine learning stage”; you can read more about Elastic machine learning, and if you watch the video at the end of this post you will see a workflow that starts with an alert from a machine learning job and moves along from there through APM and distributed tracing, the Infrastructure, and Logs UIs. Having all of the logs, metrics, APM, and distributed tracing all together in Elasticsearch allows you to use all of the analysis and visualization tools across all of your data.

  Let’s take a look at the Infrastructure UI, and then dig in. In this view we are looking at Kubernetes pods, and I have them grouped by namespace. I am looking at the inbound network traffic. By right clicking on one of the pods I can jump to the logs for that pod, or to a curated metrics dashboard for that pod.

  

  User Experience

  Three types of devices are supported today: hosts, Kubernetes (k8s) pods, and Docker containers. The power of the Infrastructure UI is that you can see the state of key indicators for a very large number of devices. At high volume, you have colored squares with no text until you drill into an interesting subset by using the search bar or clicking on a group. You can also launch into the logs for a device or display a metric dashboard.

  I mentioned “key indicators”; at the moment this is the list, and they all come from Metricbeat:

  Hosts: CPU, Memory, Load, Inbound Traffic, Outbound Traffic, and Log Rate
  Kubernetes: CPU, Memory, Inbound Traffic, Outbound Traffic
  Docker: CPU, Memory, Inbound Traffic, Outbound Traffic

  Grouping enables you to zoom into the list of devices. If you support a particular application that is deployed with Kubernetes then the Namespace group is for you. If you think that an issue is related to a particular Node being overloaded then you might want to subdivide further by grouping by both Namespace and Node. Here is the list of groupings at the moment (and you can use up to two per device type):

  Hosts: CPU, Memory, Load, Inbound Traffic, Outbound Traffic, and Log Rate
  Kubernetes: CPU, Memory, Inbound Traffic, Outbound Traffic
  Docker: CPU, Memory, Inbound Traffic, Outbound Traffic

  If none of the above groupings support the way you want to interact with your data, then use the search bar. Just start typing and the Kibana Query Language (KQL) autocomplete feature will guide you. I provided an example in the GIF of the Logs UI up top, as I started typing kubernetes Kibana offered labels and then tier, and then provided the possible choices based on what was indexed into Elasticsearch.

  Once you group the resources you can drill into a group by clicking on it, this allows you to see more detail on that group. If a particular host, pod, or container interests you, then drill into the logs or metrics for it. Here is a portion of the metric view of a host:

  

  Configuration

  There is very little configuration to be done for the Infrastructure UI. Just deploy Metricbeat and enable the System module. If you are running containers, then also enable the Kubernetes and Docker modules.

  Unless you have modified the default index pattern (metricbeat-*) you are done. If you need to customize things the Infrastructure UI docs will have the details, and I am including the key options here:

  xpack.infra.sources.default.metricAlias: "metricbeat-*"
  xpack.infra.sources.default.fields.host: "beat.hostname"
  xpack.infra.sources.default.fields.container: "docker.container.name"
  xpack.infra.sources.default.fields.pod: "kubernetes.pod.name"
  

  Have your say

  Both of these are beta UIs, and we would love your feedback (and keep the feedback coming even after they become generally available). Please tell us how you prefer to interact with your data. Are the groupings in this beta release of the Infrastructure UI sufficient to support the way that your team is organized? What would be better? Do we have the right metrics driving the colors? Do you need a different way to filter the logs in the Logs UI? When you launch into the metrics view from the Infrastructure UI are you seeing the metrics that you care about?

  Head over to the Kibana Discuss forum and let us know what you like, dislike, need, etc. The developers read and respond to these discussions and love to hear from you. You can also open an issue, submit a PR, or keep an eye on either on GitHub.

  Jump in and give it a try

  Our live demo system has logs and metrics that you can search, visualize, and interact with. Head over and click on the Infrastructure tile to launch the Infrastructure UI:

  

  You can switch to the Kubernetes or Docker views, group the objects using the UI, or narrow the view with the search bar. Left click on a host, pod, or container and launch the Logs UI for that object. While you are at demo.elastic.co, have a look at some of the dashboards and visualizations. You can navigate to different areas from the landing page.

  Watch a demo

  Here is a video that shows the workflow of these UIs (also machine learning and APM with distributed tracing!). Thanks again for reading, and for your feedback!

  
  
  https://www.elastic.co/blog/infrastructure-and-logs-ui-new-ways-for-ops-to-interact-with-elasticsearch
• 

  Using Logstash to Split Data and Send it to Multiple Outputs
  • sorangutan  

  1
  1
  Posts
  1472
  Views

  

  Logstash is an open source, server-side data processing pipeline that ingests data, transforms it, and then sends it to one or more outputs. In this blog, I will present an example that shows how to use Logstash to ingest data from multiple stock markets and to send the data corresponding to each unique stock market to a distinct output. This is accomplished by executing the following steps:

  1. Create copies of each document from a stock market input stream.
  2. Filter each copy to only contain fields that are valid for a given stock market
  3. Add metadata to each copy to indicate which stock market’s data it contains.
  4. Evaluate the metadata in each document to direct the document to the correct output.

  Note that in this blog post, I do not make use of pipeline-to-pipeline communication (beta as of 6.5) which could also likely achieve some of the functionality described here.

  Example input file

  As an input to Logstash, we use a CSV file that contains stock market benchmark values. A few example CSV entries are given below:

  1483230600,1628.75,1678.1,1772.8,2443.6
  1483232400,1613.63,1688.5,1750.5,2460.2
  1483234200,1606.51,1678.6,1718,2448.2
  1483236000,1621.04,1684.1,1708.1,2470.4
  

  The comma separated values represent  “time” and the value of the following stock exchange benchmarks: “DAX”, “SMI”, “CAC”, and “FTSE” . Copy and paste the above lines into a CSV file called “stocks.csv” in order to use this as an input in the example Logstash pipeline.

  Example Logstash pipeline

  Below we will present a Logstash pipeline that does the following:

  1. Read stock market values as CSV-formatted input from a CSV file.
  2. Map each row of the CSV input to a JSON document, where the CSV columns map to the following JSON fields:  “time”, “DAX”, “SMI”, “CAC”, and “FTSE”.
  3. Convert the time field to Unix format.
  4. Use the clone filter plugin to create two copies of each document (these copies are in addition to the original document). The clone filter automatically adds a new “type” field to each new document copy, where the “type” corresponds to the names given in the clones array. We have defined the types to be “clone_for_SMI” or “clone_for_FTSE”, and each clone will ultimately only contain data for either the “SMI” or “FTSE” stock market.
  5. For each clone:
  1. Use the prune filter plugin to remove all fields except those fields which are whitelisted for the specific stock market.
  2. Add metadata to each document corresponding to the “type” that was added by the clone function. This is necessary because we are using the prune function which removes the “type” that was inserted by the clone function, and this information is required in the output stage for directing the document to the correct output.
  6. Use the Elasticsearch output plugin for Logstash to write the documents for each stock market to a different Elasticsearch output, with the output determined by the value defined in the metadata field that we added in Step 5. In order to simplify the code below, each Elasticsearch output writes to a unique index in a local Elasticsearch cluster. If multiple clusters should be used as outputs, then each Elasticsearch output declaration can be easily modified to specify unique Elasticsearch hosts.

  Below is a Logstash pipeline that executes the above steps (with corresponding step numbers added as comments). Copy this pipeline into a file called "clones.conf" for execution:

  ## STEP 1
  input {
    file {
      # make sure to edit the path to use your stocks.csv file
      path => "${HOME}/stocks.csv"
      # The following will ensure re-reading of full input 
      # each time Logstash executes (useful for debugging).
      start_position => "beginning"
      sincedb_path => "/dev/null"
    }
  }
  ## STEP 2
  filter {
     csv {
      columns => ["time","DAX","SMI","CAC","FTSE"]
      separator => ","
      convert => { 
        'DAX' => 'float'
        'SMI' => 'float'
        'CAC' => 'float'
        'FTSE' => 'float'
      }
    }
  ## STEP 3  
  date {
      match => ['time', 'UNIX']
    }
  ## STEP 4
    # The following line will create 2 additional 
    # copies of each document (i.e. including the 
    # original, 3 in total). 
    # Each copy will automatically have a "type" field added 
    # corresponding to the name given in the array.
    clone {
      clones => ['clone_for_SMI', 'clone_for_FTSE']
    }
  ## STEP 5
    if [type] == 'clone_for_SMI' {
      # Remove everything except "SMI"
      prune {
         whitelist_names => [ "SMI"]
      }
      mutate {
        add_field => { "[@metadata][type]" => "only_SMI" } 
      }
    } 
    else if [type] == 'clone_for_FTSE' {
      prune {
         whitelist_names => [ "FTSE"]
      }
      mutate {
        add_field => { "[@metadata][type]" => "only_FTSE" } 
      }
    } 
  }
  ## STEP 6
  output {
    # The following output to stdout is just for debugging 
    # and can be removed
    stdout { 
      codec =>  rubydebug {
        metadata => true
      }
    }
    if [@metadata][type] == 'only_SMI' {
      elasticsearch {
        index => "smi_data"
      }
    }
    else if [@metadata][type] == 'only_FTSE' {
      elasticsearch {
        index => "ftse_data"
      }
    }
    else {
      elasticsearch {
        index => "stocks_original"
      }
    }
  }
  

  Testing the Logstash pipeline

  To test this pipeline with the example CSV data, you could execute the following command, modifying it to ensure that you use paths that are correct for your system. Note that specifying "config.reload.automatic" is optional, but allows us to automatically reload "clones.conf" without restarting Logstash:

  ./logstash -f ./clones.conf --config.reload.automatic
  

  Once Logstash has read the "stocks.csv" file and completed processing, we can view the three resulting indexes called "smi_data", "ftse_data", and "stocks_original".

  Check the SMI index

  GET /smi_data/_search
  

  This will display documents with the following structure. Notice that only “SMI” data appears in the "smi_data" index.

        {
          "_index": "smi_data",
          "_type": "doc",
          "_id": "_QRskWUBsYalOV9y9hGJ",
          "_score": 1,
          "_source": {
            "SMI": 1688.5    
          }
        }
  

  Check the FTSE index

  GET /ftse_data/_search 

  This will display documents with the following structure. Notice that only “FTSE” field appears in documents in the "ftse_data" index.

        {
          "_index": "ftse_data",
          "_type": "doc",
          "_id": "AgRskWUBsYalOV9y9hL0",
          "_score": 1,
          "_source": {
            "FTSE": 2448.2
          }
        }
  

  Check the original documents index

  GET /stocks_originals/_search 

  This will display documents with the following structure. Notice that the original unfiltered version of the documents appears in the "stocks_original" index.

        {
          "_index": "stocks_original",
          "_type": "doc",
          "_id": "-QRskWUBsYalOV9y9hFo",
          "_score": 1,
          "_source": {
            "host": "Alexanders-MBP",
            "@timestamp": "2017-01-01T00:30:00.000Z",
            "SMI": 1678.1,
            "@version": "1",
            "message": "1483230600,1628.75,1678.1,1772.8,2443.6",
            "CAC": 1772.8,
            "DAX": 1628.75,
            "time": "1483230600",
            "path": "/Users/arm/Documents/ES6.3/datasets/stocks_for_clones.csv",
            "FTSE": 2443.6
          }
        }
  

  Conclusion

  In this blog post I have demonstrated a small subset of the capabilities of Logstash. Specifically, I presented an example that showed how to use Logstash to ingest data from multiple stock markets, and to then process and drive that data into distinct outputs. If you are testing Logstash and the Elastic Stack and have questions, feel free to seek help on our public discussion forums.

  
  
  https://www.elastic.co/blog/using-logstash-to-split-data-and-send-it-to-multiple-outputs
• 

  Create, Manage, and Visualize Rollup Data in Kibana
  • sorangutan  

  1
  1
  Posts
  98
  Views

  

  Keeping historical data around for analysis is extremely useful, but storage costs can become a problem as the amount of stored data grows. By "rolling up" the data into a single, summary document in Elasticsearch, the rollup APIs let you summarize the data and store it more compactly. You can then archive or delete the original data to save storage. When you create a rollup, you get to pick all the fields that you’re interested in for future analysis, and a new index is created with just that rolled up data. This new rollup index then lives side by side with the index that it’s being rolled up from. Since rollups are just like any other index — only a lot smaller — you already know how to query and aggregate the data inside.

  Want to learn more about rolled up data? See this video from Elastic{ON} 2018 keynote or check out our documentation on rolling up historical data in Elasticsearch.

  Version 6.5 introduces new features in Kibana for creating, starting, stopping, and deleting rollup jobs, and also provides beta features for visualizing rolled up data in a dashboard.

  Getting started

  To check out the new features in Kibana, go to Management, and under Elasticsearch, click Rollup Jobs. If you don’t have any existing rollup jobs, there won’t be much to see here at first, but never fear! Click Create a rollup job to let Kibana guide you through the steps to get started.

  Create a rollup job

  To create a new rollup, just fill in the name, data flow, and how often you want to roll up the data. Then define a date histogram aggregation for the rollup job and optionally terms, histogram and metrics aggregations.

  Before you save the new job, Kibana will show you a summary of the rollup details so that you can double check the info before you hit save.

  

  Manage existing jobs in the new Rollup jobs view

  Configuration Changes

  Rollup job settings can’t be changed after the job has been created. If you want to select additional fields or redefine any terms, you would need to delete the existing job and create a new one with the updated specifications. Be sure to use a different name for the new rollup job, as reusing the same name could lead to mismatched job configurations. You can read more about logistical details for the rollup job configuration in the Elasticsearch documentation.

  Using rolled up data

  One way to check out these new features in Kibana is to create a rollup to capture log data from sample web logs. If you’d like to follow along with this example, you can load the sample data set by clicking the link next to Add sample data on the Kibana home page. Click Add under Sample web logs, and then go to Management and under Elasticsearch, click Rollup Jobs to create a new rollup.

  For this example, let’s say that we want data that is older than 7 days in the target index pattern (kibana_sample_data_logs) to be rolled up once a day into the index rollup_logstash. The rolled up data can be bucketed on an hourly basis, using 60m for the time bucket configuration to allow for more granular queries, like 2h and 12h.

  In the configuration below, you’ll see that the terms, histogram, and metrics fields reflect the key information we want to retain in the rolled up data, such as where visitors are from (geo.src), what operating system they are using (machine.os.keyword), and how much data is being sent to them (bytes).

  The rolled up data can now be used for analysis at a fraction of the storage cost of the original index. The original data can live side by side with the new rollup index, or be removed or archived manually or via the Index Lifecycle Management API (coming soon).

  Example rollup configuration

  Name: logs_job
  Index pattern: kibana_sample_data_logs
  Rollup index name: rollup_logstash
  Frequency: Every day at midnight
  Page size: 1000
  Delay (latency buffer): 7d
  Date field: @timestamp
  Time bucket size: 60m
  Time zone: UTC
  Terms: geo.src, machine.os.keyword
  Histogram: bytes, memory
  Histogram interval: 1000
  Metrics: bytes (average)
  

  Visualize your rollup data

  Once you’ve created a rollup job, you can create an index pattern and visualize the data in a variety of charts, tables, maps, and more. Most visualizations support rolled up data, with the exception of Timelion, Visual Builder, and Vega visualizations (coming soon).

  You can match an index pattern to rolled up data only, or mix both rolled up and raw data to visualize all data together. An index pattern can match only one rolled up index, but there is no restriction on the number of standard indices that an index pattern can match.

  Combination index patterns use the same notation as other multiple indices in Elasticsearch. To match multiple indices to create a combination index pattern, use a comma to separate the names, with no space after the comma. The notation for wildcards (*) and the ability to "exclude" (-) also apply (for example, test*,-test3). Using the information from the example rollup configuration described above, rollup_logstash would be used to match only the rolled up index pattern, and kibana_sample_data_logs would match only the index pattern for raw data. The notation for a combination index pattern with both raw and rolled up data, would be rollup_logstash,kibana_sample_data_logs.

  

  View side-by-side data visualizations of your rolled up data and live data in a custom Kibana dashboard.

  Wrapping up

  Rollups are like any other Elasticsearch index, only much smaller, so you can query and aggregate them as you would any other index. The features introduced in Kibana 6.5 provide new tools for you to create, manage, and visualize rolled up data.

  You can check out the Elasticsearch documentation to learn more about rolled up data and the rollup API.

  
  
  https://www.elastic.co/blog/how-to-create-manage-and-visualize-elasticsearch-rollup-data-in-kibana
• 

  Hiking the Pacific Crest Trail with the Elastic Stack - Part 3: Mission Complete
  • sorangutan  

  1
  1
  Posts
  41
  Views

  

  This is Part 3 of the Hiking the Pacific Crest Trail with the Elastic Stack blog series. If you haven't read Part 1 or Part 2, I'd recommend you check them out before continuing.

  Mission, complete.

  First and foremost, a couple of quick words directly to my brother:

  Mike, words cannot describe how proud I am of you for completing the Pacific Crest Trail. It’s official, you made it further than Reese Witherspoon in Wild! (Sorry, I had to - it was too easy.) In all seriousness, I am absolutely in awe at what you’ve accomplished. There were a number of times along the trail when it seemed like that stop into town may have been your last. You persevered through injuries, forest fires, bears, freezing rain, mountain lions, below zero (F) temperatures, 30-40 mile days on foot, rattlesnakes, food and water shortages, and on occasion, extreme solitude in the wilderness. It takes a certain type of strength and willpower to walk the 2,650 miles from Mexico to Canada and I have been continually amazed and impressed by your accomplishments this year. I love you and am very proud to call you my brother.

  If you haven’t guessed by now my brother has completed the Pacific Crest Trail. He started his journey on March 29, 2018, baby face and all.

  

  The last few months of Mike’s hike included some of the most treacherous weather, challenging moments and breathtaking views. When we last left you, he was just getting into Oregon and needed to pass through it quickly in order to beat the cold up North. There were days where he was hiking 40+ miles. To put that in perspective that’s about a marathon and half with 50 pounds on your back, every day. He passed through the state in record time, finishing in 14 days.

  

  You may not have noticed, but Mike’s completion date was on October 5, 2018. If you follow Elastic at all, that date might ring a (ze?) bell. That very same day Elastic went public. Given the nature and positive feedback from the Elastic community around my brother’s journey, I don’t think his completion date could have been more perfect. Let’s just say, we did a lot of celebrating in this household that morning.

  The PCT by the Numbers (the fun part)

  Just because my brother’s trek is over doesn’t mean the fun has to stop. For his analytical brother, it means I finally have a complete data set to work with. At the end of the day, we ended up with 2,282 documents for the entire trip. I will totally blame my brother for not turning on his GPS tracker enough for that. In the world of Elasticsearch, that is nothing. But there is still some analysis and summary we can do on the data using some the Elastic Stack.

  Mileage analysis with Elastic machine learning

  Around the 6.4 timeframe, we introduced machine learning nodes to our Elastic Cloud offering. Even though I’m working with a limited data set, because Mike’s check-ins were over an extended period of time I can still gain some insights and detect anomalous behavior in the data set.

  The first thing I was really interested in was if there were any anomalies in his daily miles hiked. Because Elastic machine learning is extremely approachable, I was able to quickly plug in my data and have our algorithms comb through the historic data set to determine the normal upper and lower bounds. There are a few options for spinning up an ML job, but for my sake all I needed was a single metric job.

  

  You’ll notice that a majority of the anomalies detected were during the gap where my brother took time off to heal his achilles tendon. Luckily, there is a configuration option for sparse data that will ignore any empty buckets.

  

  If you look at the anomalies detected, there are days where my brother hiked nearly 70 miles. Mike, that is pretty impressive but let’s just say, I don’t believe you. You definitely got stronger, but you’re not superman.

  There were certainly days where he actually did hike 30 or 40 miles if it was flat enough and the conditions were good, but this is confirming one of the suspicions I’ve had for a while now. I’m not sure how much I trust the Google Distance Matrix API on the trail. For instance, every once in a while the count on my daily report would be a bit off from what my brother actually hiked. Also, his total miles never hit 2,650 and percentage complete was pretty far off from 100%.

  

  For what looks like maybe a mile between points, Google’s API automatically took him the scenic route. So in a day where he actually walked about 20 minutes, we tacked on an extra 45 miles. When creating my ingest script, I opted to use the walking travel mode. The default is driving and given the fact that he didn’t spend much time on the roads (though there was a bit of hitch hiking), I felt the walking mode was our best option.

  I will say, not all of the anomalies here were because of the API. Towards the end of the trip the weather was getting colder and my brothers GPS tracker battery would die more frequently. With less check-ins, the distance between two points would sometimes be a summation of multiple days. You’ll notice more spikes in the amount of miles traveled towards the end of his trip.

  If my brother continued hiking through Canada and up to Alaska (you laugh, but I wouldn’t put it past him) I would have been able to use my historic data to alert on anomalies detected as they occurred. If this hypothetical situation actually happened, I’d be able to configure a custom rule to ignore any mileage over 50 as an anomaly in real time. Mike, maybe you’ll just have to keep going to Alaska to prove my point.

  

  Fun fact: If you want to try out the new navigation coming in Kibana 7, there is a new advanced setting in 6.5 called k7design. Alternatively, you can be part of our Pioneer Program for 7.0. We’d love feedback :-)

  I took the time to build out a workpad that could be used for print some day so my brother to hang in his future home. He’s pretty nomadic at the moment, so I think I have some time to make sure the mileage is correct before I actually print.

  In a lot of Canvas examples we provide, they are usually in a 16:9 ratio, but you have the ability to define any width and height you’d like. My final workpad can be seen in the screenshots below. This workpard consists of three pages and is using the standard letter size and dimensions. I’ll walk through some of the technical details of the more interesting aspects of this workpad.

  

  Speaking of creativity, I really wanted to show off my map to this audience. At the moment, Canvas currently does not support geo, so I had to use a similar trick that I did with the satellite imagery on my dashboard.

  
  
  https://www.elastic.co/blog/hiking-the-pacific-crest-trail-with-the-elastic-stack-part-3-mission-complete

• 

  Aftershock Therapy with Elasticsearch and CSV Data Import
  • sorangutan  

  1
  1
  Posts
  49
  Views

  

  Hey! Before you start reading, I just want to properly set expectations. We (Elastic) have other blogs about using Elasticsearch and Kibana for visualizing earthquake data, and this one is — by far — the most entry-level. For something more in-depth, you may want to check out Earthquake Data with the Elastic Stack blog by Kosho Owa and Mark Walkom. It’s a great piece created by people who know a lot more about using the Elastic Stack than I do. A lot more.

  That said, I hope this blog can help prove you don’t need to be an Elastic expert (like Kosho and Mark) to do some really cool stuff with the Elastic Stack. So if you’re already a master of geo_point visualizations, this may not be the blog for you. But if you’re still a bit of a novice, or stumbled upon this blog via some sort of internet clickhole, enjoy. And if you want, you can skip the intro and jump straight into the tutorial.

  ---

  Elastic is a distributed company, which means I can work from pretty much anywhere with an internet connection. It’s a pretty great setup most of the time. Really, the only time it’s been a problem has been when working from a living room situated less than ten miles from the epicenter of a 7.0 earthquake.

  I live in Chugiak, Alaska, a small town just north of Anchorage. And around 8:30 a.m. on November 30 —  while in the middle of a 1:1 video call with my manager — my whole house started shaking. Not like it was being hit by an earthquake (common here), but like it was being shelled by heavy artillery. Once I realized it was a really big quake, I jumped up from the couch (Bluetooth earbuds still in) and ran down the vibrating hallway as loud cracks and booms echoed throughout the house. Then my girlfriend and I hunkered down for a minute and a half of violent reverberation, hoping the whole time that the house wouldn’t split in two.

  When the shaking had ended, we surveyed the damage. Our cupboards, shelves, and refrigerator had been emptied onto the floor, but luckily, there didn’t appear to be any structural damage or gas leaks. It was a bad day to be a wine glass or ceramic tchotchke, but things could’ve been a whole lot worse. After that I called my boss — who watched the house shake before the internet dropped — to let her know that we were ok. She promptly gave me the day off.

  Just as my nerves started to calm, another quake hit, this time a 4.8. And then a 5.7. And then a 4.0. When the aftershocks slowed, I made some calls and then headed out to check on friends. No one was hurt (not even our friend that was driving 65 on the highway during the 7.0), and no one’s home took any major damage. Sighs of relief all around.

  Once internet was back I hopped onto Slack to touch base with my team. I logged in to find a few dozen messages from people throughout the company, all checking in to see if we were ok and asking if we needed any help. (There were also multiple inquiries into the condition of our cat, who at the time was hiding under a blanket with no intention of ever coming out again.) I’m the only Alaskan Elastician, so I’m not sure what they could’ve done, but just knowing that my Elastic fam was there as a remote safety net started to make me feel a little better. And then the electricity went out.

  Losing power is a tad stressful. Losing power in the winter in Alaska is a tad more stressful. Losing power in the winter in Alaska while aftershocks shake the house every few minutes is really stressful.

  When the power came back a few hours later, I hopped onto internet to survey the damage and see how many aftershocks we’d felt. There had been over 100 aftershocks within the first 8 hours following the quake, ranging from 2.5 to 5.7. And apparently, the aftershocks were coming from two key places. The first was Anchorage, just southwest of us. And the second was Big Lake, just northwest of us. So we were only a few miles east of ping ponging aftershocks, meaning we got to feel all of them. How exciting. That first night after the earthquake was a real miserable night’s sleep. Waking up to a 4.1 the next morning wasn’t that great either.

  In the days directly following, the aftershocks were relentless. It didn’t seem like there was an end in sight, and that uncertainty was steadily driving my stress levels through the roof. But then I realized that there was a way to see the end from where I was standing. If I could just visualize the magnitudes, frequencies, and locations of the quakes, I could (hopefully) understand what was going on beneath me. Enter the Elastic Stack. You know, for sanity.

  Importing CSV earthquake data into Elasticsearch

  As I said at the top of this post, this is a super basic tutorial for analyzing data with the Elastic Stack. No Beats, no Logstash, no ingestion pipelines, no Python scripts, no development background. This is quick and simple approach with a CSV file and a new feature within Elastic machine learning. But I still think it’s cool.

  The first thing I needed to do was get USGS earthquake data as a CSV, and then import it into Elasticsearch. To do this import, I used the data import functionality that was introduced into the File Data Visualizer in Elastic Stack version 6.5.

  Step 1 - Download and scrub USGS earthquake data

  Rather than doing this yourself, download this CSV of Mat-Su Valley (Alaska) earthquake data. Here are the steps I took to make it:

  1. Download all USGS magnitude 2.5+ earthquakes from the past month as a CSV.
  2. Remove all unnecessary columns.
  3. Sort by latitude, and remove any rows outside of 61.1 to 61.9 (Chugiak is at 61.4).
  4. Sort by longitude, and remove any rows outside of -148.1 to -150.9 (Chugiak is at -149.4).
  5. Create a new column named loc that is "latitude,longitude" (Ex: =C2&","&D2).
  6. Save the CSV as quakes_data.csv.

  It’s important to note that #5 is the only step that’s truly necessary for the purposes of this example, and that’s only due to how to the data is presented by the USGS. Elasticsearch can slice and dice data however you want, no matter the size of your data set.

  Step 2 - Start a free Elasticsearch Service trial on Elastic Cloud

  This is not a sales pitch, I promise. This is just the easiest way to do what we’re about to do (File Data Visualizer is a free feature through our Basic subscription). The alternative would be to download the latest versions of Elasticsearch and Kibana and install them yourself (free, as always). That method is also very easy, but the Elasticsearch Service is even easier (and also free for 14 days, no credit card required).

  1. Sign up for a free trial of the Elasticsearch Service.
  2. Log into Elastic Cloud to access your free trial (you’ll receive an email with login instructions when you start your trial).
  3. Click Create Deployment.
  4. Configure a new Memory Optimized deployment (since we don’t need much disk space).
  5. Click Create Deployment at the bottom.
  6. Save a copy of your Username, Password, and Cloud ID while you wait for the deployment process to complete.
  7. Once the instance is running, navigate to Kibana on the left side menu.
  8. Under Endpoints, click Kibana.
  9. Log into Kibana with the username and password from Step 6.
  10. When prompted, choose Explore on my own.

  Step 3 - Import the USGS data from CSV

  Ok, now that we have data to analyze and an Elasticsearch cluster that can do the analyzing, we need to import. We’re going to keep this simple. No index templates, no aliases, no best practices. Just import.

  1. Within Kibana, click on Machine Learning.
  2. In the subnav, click on Data Visualizer.
  3. Under Import Data, click Upload File.
     Note: This feature is new in 6.5. If you find any issues while using it, please let us know in our Elastic machine learning Discuss forum.
  4. Drag and drop the quakes_data.csv file (provided earlier) into the importer.
  5. Scroll through the list of contents. Notice that the loc field is importing as text. We will update this to be geo_point data in the next step. Click Import.
  6. In the Import Data window, switch to Advanced:
     1. Index name: quakes
     2. Create index pattern: unchecked (this is very important, as we’ll be importing an index pattern in the next set of steps)
     3. Index pattern name: blank (and uneditable)
     4. Under Mappings, manually change loc from “keyword” to “geo_point”.
        
        
  7. Click Import. You’ll see that 902 documents have been imported.

  Step 4 - Create Kibana visualizations using quake data

  Kibana offers a variety of ways to slice, dice, and visualize data. This blog is not going to cover creating visualizations and dashboards from scratch. Instead, we’ll walk through importing some visualizations I’ve already built and saved as a downloadable json file.

  1. On the left side navigation, click on Management.
  2. Under Kibana, click on Saved Objects.
  3. Click Import from above the object list.
  4. Click Import and choose quakes_objects.json.
  5. Click the Import button. You’ll see 11 objects were imported.

  At this point, we can start playing with visualizations in Kibana.

  Step 5 - Visualize the ground shaking

  Now the fun part.

  1. On the left side navigation, click on Dashboard.
  2. Click on the Quakeboard dashboard to see:

  

  There are a variety of visualizations within this dashboard. There are two different maps showing quake locations, a Richter scale, a list of the top quakes, and two line graphs of frequencies. Up at the top right, you can see a time picker, and above the first map you can see a magnitude slider. Go ahead and play with both of them to watch the earthquakes come and go.

  To adjust those different visualizations, click on Visualize within the left side navigation. This will display all the different visualizations that are on that dashboard. Explore their configurations, and don’t be afraid to break them. You can always re-import.

  Note: Your Magnitude Timeline visualization looks a little different from the one in the screenshot. This is because I tweaked a setting past its recommended value, which could potentially cause Kibana to run out of available memory. To duplicate my visualization in your non-production environment: On the left side navigation, click on Management. Under Kibana, click on Advanced Settings. Scroll through, and update the Maximum Bars setting from 100 to 10000.

  Finding the end of the quakes with Kibana

  Once I had everything loaded into Elasticsearch and visualized in Kibana, my unreasonable fears that the aftershocks were never going to end started to fade. Using the Time Picker, let’s go through the timeline of aftershocks (2.5 magnitude and higher):

  24 hours later - 420 aftershocks

  Crazy, right? You’d think with that many quakes, my little town would crumble into the Knik Arm (the waterway between Chugiak and the epicenter), but it hasn’t. And it probably won’t. There was a famous 9.2 back in 1964, and Chugiak is still standing. (For perspective, an 8.0 is 10x stronger than a 7.0, and 9.0 is 10x stronger than an 8.0.)

  Digging into the data a bit more, the Quake Map and the Magnitude Timeline did not have a calming effect:

  

  

  But the Quakes per Hour chart was showing positive signs:

  

  Overall, not enough data to help me sleep better at night.

  3 days later - 565 aftershocks

  The maps were still filling in, but the Quakes per Day chart was trending downward quite nicely:

  

  Only 145 new earthquakes over the past two days. That’s only 3 per hour. Way better than 17.5 per hour in the first 24 hours. Relatively speaking, that’s a lot better. Right?

  1 week later - 707 aftershocks

  Everything was still trending download, but even more importantly, they seemed to be losing intensity. On the Top 20 table, only 4 quakes were from after the first 24 hours.

  

  Ah, yes, this is reassuring.

  (Nearly) 1 month later - 900 aftershocks

  All signs are pointing to this eventually ending. The cumulative heat map may look hot, hot, hot:

  

  But looking at each week following the quake shows shows the heat maps getting cold as ice:

  

  Snoozing through anything by the end of the month.

  Still rocking, even if the ground’s still rolling

  And there you have it. While the aftershocks still haven’t completely stopped, watching their frequency steadily decline has definitely helped me sleep easier. So easy, in fact, that I didn’t even stir during the 4.9 that rattled the house at 5:30am on Dec 27th. I think it’s safe to say that my constant fear of aftershocks has finally been quelled. Up next, sharks. (Sike! I’ll never not be scared of sharks! Numbers be damned!)

  If you have interest in learning more about some of the cool stuff you can do with geo data in the Elastic Stack, check out the State of Geo in Elasticsearch talk from Elastic{ON} 2018. And if you’re interested in just learning more about how to use Elasticsearch and Kibana, I’d strongly recommend the Elasticsearch Engineer I and Kibana Data Analysis training courses. You’ll be amazed how at quickly you can go from n00b to proficient. I was, at least.

  
  
  https://www.elastic.co/blog/aftershock-therapy-with-elasticsearch-and-csv-data-import
• 

  A Fast, Satisfying Search Experience with AmericanAncestors.org
  • sorangutan  

  1
  1
  Posts
  60
  Views

  

  This post is a recap of a community talk given at a recent Elastic{ON} Tour event. Interested in seeing more talks like this? Check out the conference archive or find out when the Elastic{ON} Tour is coming to a city near you.
  

  AmericanAncestors.org, the website of the nonprofit New England Historical Genealogical Society, is one of the leading online family history research sites in the United States. They maintain one terabyte of data in Elastic’s Elasticsearch Service that their more than 250,000 members can search in real time, including over 1.4 billion names and an electronic collection of over 40 million pages of centuries-old, handwritten documents.

  The average AmericanAncestors.org member is sixty-five and retired. They have names and locations to research and pages of elegant, faded quill script to decipher. They don’t want to fiddle with a slow, rigid search feature. They want intelligent search that delivers results fast.

  Before switching to Elasticsearch, AmericanAncestors.org had used SQL for their search capability, but that solution didn’t have the flexibility their users needed. When Don LeClair, Associate Director of Database Search and Systems, started at AmericanAncestors.org in 2016, they had been running Elasticsearch 1.3 on five physical servers set up by external contractors. A year later, with those contractors long gone, their clusters were no longer optimized, search was slow, and the environment was unstable. LeClair was learning Elastic on the job and, unfortunately, becoming proficient at full cluster restarts.

  LeClair and AmericanAncestors.org needed to increase availability and boost performance, but they also wanted to add expertise and reduce system administration time and costs, so they purchased a Platinum subscription to the Elasticsearch Service. Additionally, with their new subscription, they were able to implement Elasticsearch security definitions so queries would come through with appropriate authentication. Another major advantage they found in choosing the Elasticsearch Service was they could create a separate cluster to monitor production and visualize the data in Kibana.

  “For the first time we had visibility and facts about how the system was performing rather than opinions.”  Don LeClair, Associate Director of Database Search and Systems | AmericanAncestors.org

  Once the clusters were optimized, LeClair was able to turn his attention to new challenges. The genealogy research industry was trending toward search that offered suggestions. At the time, AmericanAncestors.org’s approach to search was that results must fit search criteria, but that led to many frustrating member searches that delivered no results.

  So they changed to a best fit results approach with prioritized rankings. This change improved the user experience, but also increased the server workload. And that workload increased even more dramatically after Spectre and Meltdown vulnerability fixes were applied to their systems, leading to a performance decline and even an outage. With a new optimization challenge to handle, AmericanAncestors.org decided to work closely with Elastic’s support team to drive improvements.
  

  Through this work, LeClair discovered that Elastic support team’s ethos went deeper than simply working to help mitigate break/fix issues. The Elastic support team helped identify problem queries, performed code reviews on breaking changes between versions, and altered their sharding strategy to increase speed. Support also identified that over 35% of records in their index were logically deleted documents. With this revelation, AmericanAncestry.org changed tools and processes to avoid excessive re-indexing. They also implemented and maintain a full-data test cluster to simplify upgrade testing and enable them to stay current. The test cluster helps during development so AmericanAncestors.org can roll into production with minimal outage and inconvenience.

  In the end, with the help of the Elastic team, they built a highly scalable service with minimal overhead. They reduced disk usage by 60% and CPU by 50%, making search 2-3 times faster. This speed enabled AmericanAncestors.org to offer a major promotion with free signup and search for three days. The system handled a 400% traffic increase with excellent response time. “It was … perfect,” said LeClair. “In fact, it was so smooth on the second day I stopped watching it on the monitor.” Their current solution was dramatically simpler than their previous physical servers, and, even with premium support and a full-data test cluster, it cost 25% less than their old solution.

  Learn about how AmericanAncestors.org is leveraging Elasticsearch Service to drastically improve their service and the lessons they learned along the way implementing by watching Scalability & Support Using the Elasticsearch Service: Elastic @ NEHGS from Elastic{ON} Tour Boston.

  
  

  

  
  
  https://www.elastic.co/blog/a-fast-satisfying-search-experience-with-american-ancestors-org
• 

  The Elastic Advent Calendar 2018, Week 4
  • sorangutan  

  1
  1
  Posts
  142
  Views

  

  Wow, it's finally here! After 25 fantastic articles we've reached the end of the 2018 Elastic Advent series.

  We've covered Kibana - from spaces to maps, Elasticsearch - CCR and relevancy, Logstash enrichment, ingestion with multiple Beats, APM, App Search, Machine Learning and even into new areas around Observability and so much more. In the topics we've spoken in Chinese, Japanese, German, English, French and Portuguese.

  We hope you have enjoyed the posts, topics and information shared in them from our engineers. Here's a recap of the final few days, as well as a summary from all 25 days.
  

  The 25 days of Elastic

  Dec 1 :[FR][Logstash] Enriching Postal Addresses by David Pilato

  Découvrez dans cet article comment transformer des addresses postales en coordonnées géographiques en local à l'aide de la suite Elastic et des données publiques d'Open Street Map.

  Dec 2: [JP][Kibana] Understanding Analyzer with Analyze UI plugin by Jun Ohtani

  Analyzerの動きを理解することが、Elasticsearchの検索をよりよくする、検索にヒットしない理由を解析する近道だったりします。

  素晴らしいことに、Elasticsearchには_analyze APIという便利なAPIが存在します。

  Dec 3: [KR][Logstash] Crawl data from Facebook to Elastic through Logstash by Jongmin Kim

  저는 Elastic 한국 사용자 그룹의 글을 가져오기 위해서 만들었는데 Facebook Group의 글을 가져오기 위해서는 추가 검수가 필요합니다.

  Dec 4: [EN][Machine Learning] Rarity Analysis with Machine Learning by Rich Collier

  Finding items that rarely occur is often very useful. Some example use cases are finding:

  • Rarely occurring log messages
  • Rare running processes for a server
  • Rare connection destinations

  ML's rare function is only available in the Advanced Job Wizard, but the configuration is relatively simple. 

  Dec 5: [EN][App Search] Quick and easy typeahead search implementation with Elastic App Search by Peter Kim

  Elastic App Search is a Search Solution from Elastic that makes it easier for application developers to implement and manage core search functionality in their user-facing applications. There's no need to understand the nuances of text analysis, relevance scoring algorithms, spell correction, etc.

  Dec 6: [JP][Elasticsearch] Dealing with search relevancy by Kosho Owa

  ただ、検索機能をアプリケーションに実装する場合、検索順位を制御したい場合が多々あります。「1000円以上の商品を安い順」「今週入荷した商品を新しい順」「東京都庁から近い順」に並べるといった場合です。それらのうち、複数の条件を組み合わせたい場合もあるでしょう。Elasticsearchを使用して、どのように実装したら良いでしょうか。

  Dec 7: [FR][Kibana] Kibana Canvas by Christelle Kerloc'h

  Avec la sortie de la suite elastic 6.5, Kibana Canvas est là ! Vous pouvez désormais utiliser cette nouvelle application. Venez découvrir l'historie de Canvas et quelques cas d'utilisation pour vous inspirer.

  Dec 8: [EN][Python/Elasticsearch] Getting started with Elasticsearch DSL and Python by Honza Krai

  The purpose of elasticsearch-dsl is to provide an easier and more familiar way to work with Elasticsearch. It focuses on just the most common operations like search and generally working with data...

  Dec 9: [EN][Elastic Stack] Ubiquiti metrics in the Elastic Stack by Mark Walkom

  The Ubiquiti Unifi 15 kit is pretty amazing for its price point. I use it extensively at home to replace a bunch of Google Wifi, power line and wifi extenders and other devices that just never really worked for me. Other than the value for money they provide, it's super simple to setup and manage.

  Dec 10: [EN][Elastic Stack] Correlate and Alert for Security Analytics by Sherry Ger

  Security analytics is a common use case for the Elastic Stack. In this short topic, we will set up a rule based alert that triggers a notification when network or log events correlate with indicators of compromise. We would alert near real time so investigators can start their investigation and remediate as quickly as possible.

  Dec 11: [PT-BR][Elasticsearch] Usando Field Aliases para criar uma visão unificada entre índices com mapeamento distinto by Thiago Souza

  Field Aliases, que apareceu na versão 6.4, é uma poderosa funcionalidade que permite criar uma visão unificada entre índices com mapeamentos distintos.

  Dec 12th: [EN][Elasticsearch] Automatically adding a timestamp to documents by Abdon Pijpelink

  Back in the old days, prior to version 5 of Elasticsearch, documents had a metadata field called _timestamp. When enabled, this _timestamp was automatically added to every document. It would tell you the exact time a document had been indexed.

  Dec 13th: [EN][Elasticsearch] Chaining Ingest Pipelines by Luca Wintergerst

  In the following example we will explore the ability to reference ingest pipelines to remove some of the complexity and duplication. For this we will use the new pipeline processor 1 in 6.5. That allows us to call a pipeline from another pipeline...

  Dec 14th: [EN][Kibana] Canvas: From 0 to dashboard before the morning is out by Aaron Aldrich

  Canvas is probably my favorite new feature of Kibana. Since its first technology previews I was thrilled with the idea of live-data backed slide decks built inside of Kibana and beautiful free-form dashboards. It's only recently, though, that I had the time and inspiration to dig in myself. The Central Pennsylvania Open Source Conference gave me just the excuse (and dataset) I needed to get started.

  Dec 15: [EN][APM] Overhead of the Java agent by Felix Barnsteiner

  In this little blog post, I want to give insight into how much the [java] agent affects the performance of your applications and which configuration settings affect the performance.

  Dec 16: [DE][Elasticsearch] Zusammengesetzte Worte in Elasticsearch by Philipp Krenn

  German, Finnish, Korean and the Scandinavian languages have the characteristic impractical for full-text search that they form compounds, ie combine several words into one. This leads to problems, for example, if you have saved the word "full-text search" and would like to find it by just searching for "search". But how can this request still be implemented?

  Dec 17: [FR][Elastic Stack] Indexer facilement les slow logs d’Elasticsearch avec Filebeat by Lionel Palacin

  Pour détecter les requêtes les plus lentes, Elasticsearch fournit une fonctionnalité appelée Slow logs. Cette fonctionnalité permet d'écrire dans les fichiers de logs les requêtes de recherche ou d'indexation dont le temps d'execution est supérieur à un seuil défini par l'administrateur du cluster Elasticsearch.

  Dec 18: [EN][Elastic Stack] Set up for success - don’t repeat yourself! by Ken Macinnis

  Do you find yourself re-using the same index settings and mappings frequently? Do those mappings have lots of fields with repetitive naming conventions?

  If so, fear not! You aren't alone, and there's relief at hand. Let's talk about Dynamic Templates and Index Templates.

  Dec 19th: [EN][Heartbeat] Configure a TLS expiration notification using Heartbeat by Dov Hershkovitch

  In this blog I would like to cover one of the new features we’ve added to Heartbeat, TLS information. Starting from 6.5, Heartbeat, will retrieve TLS information from the end point it is checking.

  Dec 20th: [EN][Python/Elasticsearch] Python DSL for Elasticsearch by Honza Krai

  In the previous post we learned about elasticsearch-dsl and created an index in Elasticsearch to store all of our git history, now it's time to load in the data and see what we can do with it.

  Dec 21st: [EN][Elasticsearch] Using SQL to query data in Elasticsearch by Aravind Putrevu

  Querying data using Structured Querying Language (SQL) has been introduced in Elasticsearch 6.3 as a plugin. One can write SQL queries on the data stored in any Elasticsearch Index.

  Dec 22nd: [CN][Elasticsearch] 跨集群复制（CCR）的使用 by Medcl Zeng
  

  CCR - Cross Cluster Replication - 跨集群复制是 Elasticsearch v6.5 发布的一个新的特性，这个特性可以让你将一个集群的索引数据同步复制到远程的另外一个集群上面去。
  

  Dec 23rd: [EN][Observability] Querying metrics from Prometheus by Tanya Bragin

  Elastic is increasingly used as a single operational data store for logs, metrics, and trace data. As a result, we get asked whether it is possible to ingest metrics from Prometheus exporters or integrate with Prometheus server.

  Dec 24th: [EN][Kibana] Region maps in Kibana by Bhavya Mandya

  Maps in Kibana are getting awesomer day by day. We have added a ton of features to our existing region and co-ordinate maps.
  

  Dec 25th: [EN][Kibana] Awesome Kibana Spaces by Rashmi Kulkarni

  Kibana Spaces allows you to organize your dashboards, visualizations and other saved objects into meaningful categories based on team, use case, individual or really any way you choose to use it. The beauty of this feature is each space created is independent, so objects that exist in one space don't clutter other spaces.
  

  Thank You!

  All the of the topics will be kept on the Elastic Discuss Forums so you can refer back to them at any time. And, as these are Discuss topics, you can also continue the conversation with the authors and other community members.

  Thanks for following this year's series, we hope it’s provided some useful inspiration for your use of the Elastic Stack. If you’d like us to repeat this, if you have ideas for next year or any other feedback, please let us know via Twitter (@elastic) or feel free to create a topic in our Meta category with your comments.

  We hope 2018 has been an amazing year for you and we look forward to building upon it for 2019!

  
  
  https://www.elastic.co/blog/elastic-advent-calendar-2018-week-four
• 

  This Week in Elasticsearch and Apache Lucene - 2018-12-21
  • sorangutan  

  1
  1
  Posts
  65
  Views

  

  Holiday Hiatus

  A large portion of our engineering team will be spending time with family over the upcoming holiday season. As we expect the coming two weeks to be relatively low-traffic in terms of updates, we will skip this blog for the upcoming two weeks. Expect to see our next update on or around January 11. Whatever your plans are this holiday season, we wish you the very best.

  Elasticsearch Update

  Highlights

  BKD-backed Geo Shapes

  BKD-backed geoshapes have been merged! The new BKD geoshapes will be the new default, and users do not need to configure any of the extra settings associated with quadtrees (precision, etc). Configuring the old settings will re-enable the old quadcells.

  We have been sifting through our backlog of geo issues, and stumbled on an issue where the default quadcells took 4.5 hours indexing a small collection of 10m precision shapes (and ultimately crashed from an error). We re-ran the test on the new BKD geoshapes, and the time came down to ~1s with no crash :)

  For anyone playing along at home, that's four orders of magnitude, or 16,199 times faster. Yes, really!

  Infinite DNS caching, begone!

  We implemented a change to override the JVM DNS cache policy. Per JVM defaults, when a security manager is present, positive hostname lookups will be cached for the life of the JVM. This can be a poor user experience, with users watching a DNS change propagate through their entire infrastructure with the exception of Elasticsearch. This behavior impacts cluster discovery, cross-cluster replication and cross-cluster search, reindex from remote, snapshot repositories, webhooks in Watcher, external authentication mechanisms, and the Elastic Stack Monitoring Service.

  The reason the JVM has this is to guard against DNS cache poisoning attacks. Yet, there is already a defense in the modern world against such attacks: TLS. With proper certificate validation, even if a resolver falls prey to a DNS cache poisoning attack, using TLS would neuter the attack. With Elasticsearch 6.6.0 we will introduce two new system properties for overriding the default JVM DNS cache policy. These properties will ship configured in the default jvm.options file to set the positive DNS cache to sixty seconds, and the negative DNS cache to ten seconds. Note that for users / customers that upgrade from a previous version, these properties will not automatically show up and they need to add them themselves to override the JVM's default behavior.

  Cross-Cluster Replication

  Our cross-cluster replication UI is merged! We are shipping not one, but two apps: first, a UI to configure remote clusters; second, a UI to configure auto-follow patterns. Recent work on the UI includes adding a detail panel to the auto-follow patterns table. In addition to giving the user more info about the auto-follow pattern, this panel also provides the user with a convenient link to view the follower indices in Index Management.We also added the UI for deleting auto-follow patterns and implemented numerous UX improvements, for example validating that an auto-follow pattern name doesn't collide with an existing one when creating a new auto-follow pattern. Check it out:

  

  In addition to the UI work, we added a rolling upgrade test for CCR, which uncovered two issues: a translog seralization bug, and that the CCR stats api didn't filter by follower indices. We also added more information to the CCR stats endpoints about the remote clusters being auto followed, such as time since last checking metadata and the corresponding metadata version.

  We fixed two problems uncovered by the CCR UI team. The first one was a stack overflow error when removing remote cluster connections while auto followers are still active using that remote cluster. We worked on making CCR resilient against missing remote cluster connections. Shard follow tasks and auto followers will now retry instead of failing with an unhandled error. The second one was an issue in the remote cluster infrastructure where the remote cluster info endpoint would fail if any seed for any remote cluster failed to resolve. We changed the remote connection info to display the configured seed nodes, avoiding a hostname resolution.

  We implemented the session management on the leader side for the recovery from remote feature, which will be used to keep a context of files to sync open during the file transfer. He also prototyped a modification to the remote client that allows direct connection to a specific remote node in order to avoid extra hops during the file-based recovery.

  Typeless or bust!

  We have been working through more type deprecations. This week we deprecated _type as field names in queries, inside a reindex, doc create endpoint, get_source and exists_source API, and implemented a cleanup so that search-related endpoints don't throw duplicate deprecations (due to internal optimizations). The next batch of deprecations will tackle the index creation and mappings APIs, which also happen to be the last of the deprecations.

  SQL

  We added now functions, moved SQL off from TimeZone to ZoneId and fixed translation of like/rlike statement. We also worked on preparing a new 32-bit odbc driver artifact into our release system, and concluded the work on intervals support for parameters and result sets in odbc. We marked SQL as experimental for 6.5 and beta for 6.6, as well as fixing a variety of bugs.

  Zen2

  Zen2 will ship as the default in our 7.0-alpha-2 release. We’ve also merged the documentation PR for Zen2. Many thanks to everyone who commented and iterated with us on the docs. If you want to learn how to use Zen2 in the (about-to-be-released) 7.0-alpha-2 release, take a peek at the updated documentation!

  SSL Settings

  The configuration for TLS/SSL within Elasticsearch has had some form of fallback since the first official release. For instance, when configuring an LDAP realm, the value of ssl.cipher_suites would default to the value of xpack.ssl.cipher_suites. Unfortunately, this behavior is often more confusing than helpful, as a change in a fallback setting will cascade to other components. In 6.7.0, we'll deprecate this fallback of SSL configuration and remove it for 7.0. In 7.0, users will need to configure each SSL configuration explicitly.

  Ingest Node

  This week we had a discussion about shipping the ingest-geoip and ingest-user-agent by default. Several Filebeat modules rely on the geoip and user_agent ingest processors being available, and users have a poor experience managing these plugins. Given the importance and prevalence of these use-cases, we are choosing to package these plugins as modules with all distributions.

  In addition, we fixed/added support for newer ingest node features including: support for default pipelines when indexing through an alias (#36231), support for default pipelines when doing bulk upserts (#36618), and support for the drop processor via a on_failure fork of a pipeline (#36686).

  Performance Regression

  This week our nightly benchmarks detected a major performance regression in indexing throughput. This regression is due to our ongoing work to enable indexing nanosecond timestamps, predicated by migrating from Joda time to the Java 8 java.time API. Joda time has within the possibility to define multiple parsers for a given date time formatter, but the Java 8 java.time API does not have this possibility. To handle this, our first implementation would try one format after another, catching exceptions as we go until we find a parser that succeeds. Throwing exceptions is expensive though, and it ended up being impactful enough to cause this regression. We’ve partially reverted to the old behavior while we continue to work on what will be our long-term fix.

  Changes

  Changes in Elasticsearch

  Changes in 7.0:

  • Deprecate the document create endpoint. 36863
  • Avoid duplicate types deprecation messages in search-related APIs. 36802
  • Fix cluster state persistence for single-node discovery 36825
  • Add script filter to intervals 36776
  • [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default geo_shape indexing approach 36751
  • Deprecate types in index API 36575
  • Allow worddelimitergraph_filter to not adjust internal offsets 36699
  • Add typless endpoints for getsource and existsource 36426

  Changes in 6.7:

  • Enable IPv6 URIs in reindex from remote 36874
  • SQL: Make sure now() always uses milliseconds precision 36877
  • Use index-prefix fields for terms of length min_chars - 1 36703
  • Core: Deprecate negative epoch timestamps 36793
  • SQL: Enhance Verifier to prevent aggregate or grouping functions from 36799
  • Add X-Forwarded-For to the logfile audit 36427

  Changes in 6.6:

  • SQL: Fix bug regarding histograms usage in scripting 36866
  • [CCR] Report error if auto follower tries auto follow a leader index with soft deletes disabled 36886
  • Remote cluster license checker and no license info. 36837
  • Remove indexing_complete when removing policy 36620
  • SQL: protocol returns ISO 8601 String formatted dates instead of Long for JDBC/ODBC requests 36800
  • [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default geo_shape indexing approach 36855
  • QueryRescorer should keep the window size when rewriting 36836
  • Core: Revert back to joda's multi date formatters 36814
  • [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default geo_shape indexing approach 36743
  • Use SearchRequest copy constructor in ExpandSearchPhase 36772
  • [GEO] Fork Lucene's LatLonShape Classes to local lucene package 36794
  • BREAKING: Enhance Invalidate Token API 35388
  • Invalidate Token API enhancements - HLRC 36362
  • Make the ingest-geoip databases even lazier to load 36679
  • ingest: fix on_failure with Drop processor 36686
  • ingest: support default pipelines + bulk upserts 36618
  • [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default geo_shape indexing approach 35320
  • Core: Deprecate use of scientific notation in epoch time parsing 36691
  • Deprecation check for classic similarity 36577
  • Deprecation check for renamed bulk threadpool settings 36662
  • Add raw sort values to SearchSortValues transport serialization 36617
  • Fix rollup search statistics 36674
  • Watcher deprecate notification service settings 36403
  • Rename seq# powered optimistic concurrency control parameters to ifSeqNo/ifPrimaryTerm 36757
  • SQL: Extend the ODBC metric by differentiating between 32 and 64bit platforms 36753
  • Expose Sequence Number based Optimistic Concurrency Control in the rest layer 36721
  • Deprecation check for discovery configuration 36666
  • Fix duplicate phrase in shrink/split error message 36734
  • [Painless] Add boxed type to boxed type casts for method/return 36571
  • Do not resolve addresses in remote connection info 36671
  • Deprecation check for audit log prefix settings 36661
  • SQL: Fix translation of LIKE/RLIKE keywords 36672
  • Add doc's sequence number + primary term to GetResult and use it for updates 36680
  • [CCR] Add time since last auto follow fetch to auto follow stats 36542
  • Watcher accounts constructed lazily 36656
  • SQL: Concat should be always not nullable 36601
  • SQL: Scripting support for casting functions CAST and CONVERT 36640
  • Add seq no powered optimistic locking support to the index and delete transport actions 36619

  Changes in 6.5:

  • SQL: Fix issue with always false filter involving functions 36830
  • Bump 6.5 branch to version 6.5.5 36857
  • Fix Rollup's metadata parser 36791
  • Synchronize WriteReplicaResult callbacks 36770
  • SQL: Fix wrong appliance of StackOverflow limit for IN 36724
  • SQL: Fix issue with complex HAVING and GROUP BY ordinal 36594

  Changes in Elasticsearch Management UI

  Changes in 6.6:

  • Remove errant slash from Index Management detail panel. 27605
  • [CCR] Refine warning copy in Remote Clusters. 27620
  • [CCR] Remote Clusters and Cross-cluster Replication apps 26777
  • [Rollups] Fix rollup job wizard bug: coerce histogram interval to Number for validation 27413
  • adding new specs for security endpoints without _xpack prefix 27057
  • Fix search profiler 27326
  • adding loading spinner for index management table 27204

  Changes in Elasticsearch SQL ODBC Driver

  Changes in 6.6:

  • Fix clean target by removing MSI files and leave .gitignore file intact 83
  • Interval type support 80
  • ODBC request mode only added for catalog queries. New client_id request param. 81
  • Server version check 84

  Lucene Update

  Lucene 7.6

  Lucene 7.6 was released and we upgraded Elasticsearch 6.x to this new release so that it will be used in the upcoming Elasticsearch 6.6. This Lucene release most notably features a fix for a potential corruption when index sorting is enabled, and selective indexing for points: the ability to only index a subset of dimensions.

  Faster, smaller geo shapes

  LatLonShape tessellates shapes into a triangular mesh. One interesting question is how to index those triangles. Until now the approach was to index the minimum bounding rectangle of those triangles as a 4D point and then to additionally record the 6 coordinates of the triangle. Then searching triangles is a two-step process that first uses the index to efficiently find all minimum bounding rectangles that intersect with the query, and then checks whether the triangle actually matches. This encoding was redundant due to the fact that 4 out of the 6 coordinates of each triangle are also coordinates of the minimum bounding rectangle of the triangle. And there are only 6 ways that triangles can share coordinates with their minimum bounding rectangle (MBR):

  

  Either two opposite vertices of the MBR are also vertices of the triangle (triangles 1 and 2), or 1 vertex of the MBR is also a vertex of the triangles and the two other vertices of the triangle are on the opposite edges of the MBR (triangles 3, 4, 5 and 6). So we changed the encoding so that instead of recording 6 coordinates for the triangle, we now record the two coordinates that are not shared with the MBR of the triangle and a number that identifies how to reconstruct the triangle from the MBR and those two coordinates. Nightly benchmarks reported 37% faster indexing and a 40% less disk usage when indexing 61M points from OpenStreetMap.

  Other

  • We reported a bad overflow when indexing too many tokens in a field.
  • We’re iterating on improving sparse doc-value iterators.
  • We are adding the ability to enforce a gap between intervals and to filter intervals based on their internal gaps.
  • It will soon be possible to index custom attributes in a field's info.
  • We made a change to make it possible to get consistent values of maxDoc and numDocs from IndexWriter.

  
  
  https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2018-12-21
• 

  Elastic Advent Calendar 2018, Week 3
  • sorangutan  

  1
  1
  Posts
  59
  Views

  

  We are in the final stretch for the 2018 version of the Elastic Advent Calendar! This is the last weekly wrap up, the next one will be a recap of the full 25 days.

  This week we have some great insights into APM. SQL for Elasticsearch, and multiple handy Beats applications.

  Let's dive into Week 3

  Dec 15: [EN][APM] Overhead of the Java agent by Felix Barnsteiner

  In this little blog post, I want to give insight into how much the [java] agent affects the performance of your applications and which configuration settings affect the performance.

  Dec 16: [DE][Elasticsearch] Zusammengesetzte Worte in Elasticsearch by Philipp Krenn

  German, Finnish, Korean and the Scandinavian languages have the characteristic impractical for full-text search that they form compounds, ie combine several words into one. This leads to problems, for example, if you have saved the word "full-text search" and would like to find it by just searching for "search". But how can this request still be implemented?

  Dec 17: [FR][Elastic Stack] Indexer facilement les slow logs d’Elasticsearch avec Filebeat by Lionel Palacin

  Pour détecter les requêtes les plus lentes, Elasticsearch fournit une fonctionnalité appelée Slow logs. Cette fonctionnalité permet d'écrire dans les fichiers de logs les requêtes de recherche ou d'indexation dont le temps d'execution est supérieur à un seuil défini par l'administrateur du cluster Elasticsearch.

  Dec 18: [EN][Elastic Stack] Set up for success - don’t repeat yourself! by Ken Macinnis

  Do you find yourself re-using the same index settings and mappings frequently? Do those mappings have lots of fields with repetitive naming conventions?

  If so, fear not! You aren't alone, and there's relief at hand. Let's talk about Dynamic Templates and Index Templates.

  Dec 19th: [EN][Heartbeat] Configure a TLS expiration notification using Heartbeat by Dov Hershkovitch

  In this blog I would like to cover one of the new features we’ve added to Heartbeat, TLS information. Starting from 6.5, Heartbeat, will retrieve TLS information from the end point it is checking.

  Dec 20th: [EN][Python/Elasticsearch] Python DSL for Elasticsearch by Honza Krai

  In the previous post we learned about elasticsearch-dsl and created an index in Elasticsearch to store all of our git history, now it's time to load in the data and see what we can do with it.

  Dec 21th: [EN][Elasticsearch] Using SQL to query data in Elasticsearch by Arvaind Putrevu

  Querying data using Structured Querying Language (SQL) has been introduced in Elasticsearch 6.3 as a plugin. One can write SQL queries on the data stored in any Elasticsearch Index.

  Not long now

  Only 4 more posts of handy howto's and tips from the engineers behind the projects, and that'll be our 2018 series over! Make sure you subscribe to the Advent category on our forums for the latest posts, and follow @elastic for tweets when new posts are published.

  
  

  
  
  https://www.elastic.co/blog/elastic-advent-calendar-2018-week-three
• 

  Functionbeat: Serverless Ingestion for Elasticsearch
  • sorangutan  

  1
  1
  Posts
  485
  Views

  

  Serverless computing, or Function-as-a-Service (FaaS), has been a trending architecture pattern in many cloud deployments, often as a complement to existing container, VM, and cloud infrastructures. It enables applications to be run as functions on demand, further simplifying deployments when long-running infrastructure isn’t required, and potentially reducing operational cost with users only being charged when functions are being executed.

  With more workloads moving into the cloud, it has become increasingly important to harness the ability to easily ingest cloud data in a scalable, reliable, and cost efficient fashion. Serverless compute services like AWS Lambda, Azure Functions, and GCP Functions enable users to take the “Ops” out of DevOps, and the introduction of Functionbeat now further takes the “Dev” part out as well, leaving you with a notably simple experience for ingesting cloud data into Elasticsearch or Elastic Cloud. The Elasticsearch Service in Elastic Cloud is the only hosted Elasticsearch offering that is built and run by Elastic, the company behind Elasticsearch.

  Introducing Functionbeat

  Functionbeat is a new addition to the Beats product suite that can be easily deployed as a function in serverless compute platforms, providing everything you need to configure, deploy, and transform events like logs and metrics coming from your cloud infrastructure. Available as of 6.5 as a beta, it currently supports AWS Lambda and a subset of data sources, namely AWS Cloudwatch Logs and Simple Queue Service (SQS). We have plans to expand support across more cloud providers and data sources as we move forward. Functionbeat is offered under the Elastic Basic license which means its freely available for all end users.

  

  Ingest from Cloud Data Sources

  With flexibility in mind, Functionbeat opens up endless opportunities to effectively ingest, transform, enrich, and ship your cloud data to Elasticsearch. Here are two use cases in which you may leverage Functionbeat to accomplish:

  • Monitor AWS with the Elastic Stack - Functionbeat can receive events from a Cloudwatch Log group, extract relevant fields with the dissect processor to structure the event, and apply filtering prior to shipping to an Ingest Node pipeline or directly to Elasticsearch.
  • Analyze Application Data from SQS with the Elastic Stack - for applications sending JSON encoded events to an SQS queue, Functionbeat can be used to listen, ingest, and decode the JSON event prior to shipping to Elasticsearch.

  Define Processing Logic without Code

  At the core, Functionbeat operates like any other Beat, using the same underlying framework as Filebeat and Metricbeat, and exposing similar functionality that you’ve already come to know and love. This means that you can add fields to your events, use conditionals to drop or route events, or use any processors to manipulate or enrich your data before sending it to Elasticsearch. What’s great is that all of this can be achieved through a configuration style experience without the need to actually write and compile code!

  Ingesting Cloudwatch Logs in the Real World

  Let’s say we want to send a subset of log messages from Cloudwatch Logs to our Elasticsearch cluster, and to keep the event count reasonable, we are only interested in messages containing the ERROR keyword. We edit the functionbeat.yml and add the following function configuration for a mylogs Lambda function that will be triggered when new events are generated from the Cloudwatch Logs group /aws/lambda/test matching the ERROR pattern:

  functionbeat.provider.aws:
    deploy_bucket: myappdeploybucket
    functions:
      - name: mylogs
        enabled: true
        type: cloudwatch_logs
        concurrency: 5
        description: "A really special lambda"
        fields:
          level: "error"
        triggers:
          - log_group_name: /aws/lambda/test
            filter_pattern: "ERROR"
  output.elasticsearch:
    enabled: true
    hosts: ["localhost:9200"]
    username: "elastic"
    password: "changeme"
  

  We can now deploy that function to AWS Lambda, but before proceeding we should make sure the AWS environment variables are correctly configured. The function can be pushed into Lambda using the deploy subcommand:

  ./functionbeat deploy mylogs
  

  

  
  
  https://www.elastic.co/blog/functionbeat-serverless-ingestion-for-elasticsearch

• 

  Kibana Local File Inclusion Flaw CVE-2018-17246
  • sorangutan  

  1
  1
  Posts
  65
  Views

  

  In the past few days, a number of tweets and news stories have emerged about CVE-2018-17246, a bug in Kibana that could result in including any arbitrary file from the filesystem. This issue was fixed over a month ago in 6.4.3 and 5.6.13, and while we generally don’t publish additional information describing security bugs that have already been fixed, there has been some confusion surrounding this one, so we want to make sure accurate information about the problem is out there.

  The Bug

  The bug in question is known as a Local File Inclusion bug, or LFI. The short explanation is a remote attacker can make Kibana load and run a file from the local disk. In the case of the Kibana bug, it will try to load the content of the file it loads as JavaScript code. In the examples we’re seeing, the file /etc/passwd is being included. Note that this proof of concept simply crashes Kibana since the contents of /etc/passwd are not JavaScript code.

  In any running system, including libraries from external files is a very normal operation. What’s not normal is when an attacker is able to trick a running system into loading a file of their choosing from the disk. A bug like this could be quite dangerous if an attacker is able to first upload a file somewhere on disk, then tell Kibana to load and run it. This is an example of why the practice of segmenting services is generally a good idea.

  The Impact

  Customers running the Elasticsearch Service on Elastic Cloud can be assured that their clusters are already updated to fix this vulnerability. If you are running on another Elasticsearch service (such as Amazon ES) or are managing your own Elasticsearch deployment, you'll need to make sure you take one of the steps listed below to update or workaround this issue.

  The Exploit

  A Cyberark advisory contained an example exploit — the same exploit making its way around Twitter. Using this simple exploit code, anyone can request a Kibana URL and essentially crash Kibana. What this example can’t do, though, is view contents of arbitrary files. While the demo shown uses /etc/passwd, the contents of the file don’t end up sent to the user. Some details are logged to the Kibana log file as part of the crash report, but just the first few lines.

  Our Response

  The Cyberark advisory contains a timeline at the bottom. They alerted Elastic to the issue on Oct. 23, we analyzed it, and promptly fixed it with a new release of the Elastic Stack on Nov 6. We then updated all Elastic Cloud systems immediately.

  We assigned this ESA-2018-18, which is one of our Elastic Security Advisory IDs, as well as CVE-2018-17246. If you’re using Kibana versions 6.4.3 or 5.6.13 (or later), you have nothing to worry about.

  The best way to deal with this issue is to apply timely security updates. But if you can’t apply updates right away, we also have a workaround available. You can disable the Kibana Console plugin which prevents the codepath in question from being executed. The Console plugin can be disabled by setting console.enabled: false in the kibana.yml file.

  It Takes a Village

  If you ever find yourself with interesting information about a potential security flaw affecting the Elastic Stack, such as Kibana in this instance, feel free to let us know about it. Elastic closely watches the [email protected] address, we take reports seriously, reply promptly, and do our best to fix the problem as fast as possible.

  Finally, if you’re not comfortable taking these and other security measures yourself, please consider using our Elasticsearch Service on Elastic Cloud, where security is always enabled.
  
  https://www.elastic.co/blog/kibana-local-file-inclusion-flaw-cve-2018-17246
• 

  Elastic Stack 6.5.4 Released
  • sorangutan  

  1
  1
  Posts
  69
  Views

  

  Version 6.5.4 of the Elastic Stack has been released today. It contains fixes for a couple of notable issues in Beats and Kibana.

  Notable bug fixes:

  • In Kibana, the Elasticsearch timeout setting of 0 is now respected as “no timeout” as documented (#27209). This was a regression that caused visualizations to return partial data or errors starting in the 6.5.3 release.
  • Also in Kibana, the “Show partial buckets” option is now again available in the Visualize app. (#25521).
  • In Filebeat, the Go routine memory leak that was previously fixed  is now included in 6.5. (#9592).
  • A fix for the potential CPU denial of service in certificate validation (CVE-2018-16875). (#9563)

  We recommend that you  upgrade to this latest version. For a full list of changes for each of the Elastic Stack’s products, please refer to the release notes for Elasticsearch, Kibana, Beats, and Logstash.

  
  
  https://www.elastic.co/blog/elastic-stack-6-5-4-released