Elastic News

• 

  Benchmarking Elasticsearch cross-cluster replication
  • sorangutan  

  1
  1
  Posts
  171
  Views

  

  One of the highly anticipated features of Elasticsearch 6.7 and 7.0 was [cross-cluster replication](https://www.elastic.co/guide/en/elastic-stack-overview/current/xpack-ccr.html) (CCR), or the ability to replicate indices across clusters, e.g., for disaster recovery planning or geographically distributed deployments. Since this feature introduces a lot of changes to the Elasticsearch code base, we need our users to be confident about its performance, resilience and stability. While our standard Elasticsearch benchmarking tool, [Rally](https://esrally.readthedocs.io/en/stable/), provided a lot of capabilities that we needed, it was necessary to expand it in several areas: - Ability to communicate with [more than one cluster](https://esrally.readthedocs.io/en/stable/command_line_reference.html#id2) - Collecting metrics (this is performed using [telemetry devices](https://esrally.readthedocs.io/en/latest/telemetry.html#recovery-stats) in Rally) from the new Elasticsearch [ccr-stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-get-stats.html) and [recovery](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-recovery.html) API endpoints. With these features added, we defined a topology involving a leader cluster in a geographical region (Europe) far from the following cluster (North America), ensuring a network latency of at least 100ms: ![Benchmarking Topology][1] The benchmarking scenarios used to evaluate different aspects of CCR were: ## Ensure follower is always able to catch up with different types of load We picked three completely different load scenarios representing [very small](https://github.com/elastic/rally-tracks/tree/master/geopoint#example-document), [medium](https://github.com/elastic/rally-tracks/tree/master/http_logs#example-document) and [very large](https://github.com/elastic/rally-tracks/tree/master/pmc#example-document) document sizes. With these were able to [tune default values](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-put-follow.html#_default_values_2) for following indices — for example, to avoid long garbage collection (GC) pauses with large doc sizes we’ve changed the default value of the CCR parameter [max_read_request_size](https://github.com/elastic/elasticsearch/commit/ddda2d419cb49c8df0df70b8ce8d4c8b7322ef02) from unlimited to 32MB. ## Ensure remote recovery is performing optimally Similarly, we wanted to be sure that bootstrapping a follower index with [remote recovery](https://www.elastic.co/guide/en/elastic-stack-overview/current/remote-recovery.html) is performing well for most cases with the default configuration. With the same topology, we used the [medium doc size dataset](https://github.com/elastic/rally-tracks/tree/master/http_logs) in different scenarios, including indexing it entirely first and then enabling CCR on the following cluster, as well as indexing it up to a certain percentage followed by enabling CCR while indexing continued. Immediately from the first scenario we observed that the time taken to recover was too long, so we added the ability to [fetch chunks from different files in parallel](https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2019-02-08) and optimized the remote recovery settings [default values](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-settings.html). ### … but what about network compression? ![data compression picture][3] Remote clusters can be optionally configured to [compress requests](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-remote-clusters.html#configuring-remote-clusters) on the transport layer. We also evaluated whether transport compression can reduce the time taken for remote recovery and [realized that](https://gist.github.com/dliappis/2e61295744d7e2f95a63394d771a86d4#gistcomment-2848138) without tweaking a number of settings to saturate the network, enabling compression actually ends up making the recovery process slower due to increased CPU usage. ## Stability Another aspect we looked at is CCR stability over a period of 10 days in different scenarios. Keeping the same network topology we used [data](https://github.com/elastic/rally-eventdata-track#elasticlogs_bulk_source) modelled on the Filebeat nginx module format. One scenario concerned leader indices managed using [index lifecycle management (ILM)](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html) with [CCR auto-follow](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-put-auto-follow-pattern.html) configured: a total of 17bn docs got indexed, the index rolled over 341 times, each rolled over index containing ~52 million docs and 20GB worth of data (not including replicas). We also conducted the same ten-day scenario with random restarts of the following cluster — with a five-minute pause period before starting them again — to ensure all data gets recovered as expected. ## Try benchmarking CCR with Rally yourself! We’ve prepared [an easy to use recipe](https://esrally.readthedocs.io/en/latest/recipes.html#testing-rally-against-ccr-clusters-using-a-remote-metric-store) based on Docker to spin up a local CCR environment for tests pulling metrics and shipping them to an Elasticsearch metrics store. Enjoy! [1]: https://images.contentstack.io/v3https://www.elastic.co/assets/bltefdd0b53724fa2ce/blt880747bfe374506d/5cc054f943283e8d640eaa4e/CrossRegionLink.png [2]: https://images.contentstack.io/v3https://www.elastic.co/assets/bltefdd0b53724fa2ce/blt85676a8c9b87d6c9/5cbdb114f8d0b9d818246e7e/CCR_Regions_Transparent.png [3]: https://images.contentstack.io/v3https://www.elastic.co/assets/bltefdd0b53724fa2ce/blt1021763a1731757b/5cbeff5dfcad0dd81e21b835/press-1332506_1920.jpg
  
  https://www.elastic.co/blog/benchmarking-elasticsearch-cross-cluster-replication
• 

  Elastic expands data analyst training to cover new Kibana features
  • sorangutan  

  1
  1
  Posts
  65
  Views

  

  We recently released an expanded version of our Kibana training course in response to increased demand and new features. The instructor-led course prepares technical or non-technical professionals, data analysts, security analysts, operations analysts, DevOps, and others to harness Kibana to visualize, manage, and analyze data in Elasticsearch.

  The revised course has doubled in length to include fresh Kibana content themes like Canvas, query bar, spaces, advanced settings, and much more.

  
  

  Starting with the fundamentals, Elastic instructors will guide participants through the core concepts of data analysis using Kibana — from simple aggregation-based charts to complex time-series visualizations. No prior knowledge of Kibana or the Elastic Stack is required.

  By the end of these lectures, labs, and Q&A sessions, Kibana course participants come away with the knowledge to easily find answers and anomalies within their Elasticsearch data.

  
  

  Participants learn how to create visualizations and dashboards across a variety of data sets, and will become equipped to manage Kibana by creating users, roles, and spaces. Students can keep course materials for continued independent learning or future review.

  
  

  The Kibana Data Analyst course, redesigned from the ground up and now available, includes 16 hours of live, instructor-led training in a virtual setting. Corporate teams can now also request a private Kibana training course taught by an Elastic instructor at the convenience of their worksite — eliminating the time and expense associated with travel.

  
  

  Elastic is constantly refreshing and adding new courses to our portfolio of training offerings. These include a series on becoming an “Elastic Certified Engineer,” in addition to highly focused and specialized training courses on “I didn’t know it could do that,” and so much more.
  
  https://www.elastic.co/blog/elastic-expands-data-analyst-training-to-cover-new-kibana-features
• 

  Indexing documents with the NEST Elasticsearch .NET client
  • sorangutan  

  1
  1
  Posts
  1671
  Views

  

  Introduction

  There are a number of ways you can index documents into Elasticsearch using the NEST Elasticsearch .NET client.

  This blog post will demonstrate some of the simple methods, from indexing a single document at a time, to more advanced methods using the BulkObservable helper.

  Single documents

  Within NEST, a document is modelled as POCO (plain old CLR object), an example is given below:

  public class Person
       {
     public int Id { get; set; }
     public string FirstName { get; set; }
     public string LastName { get; set; }
       }
  

  An instance of this object, which represents a single document in Elasticsearch, can then be indexed using a few different methods. Let's use the following instance as an example:

  var person = new Person
    {
     Id = 1,
     FirstName = "Martijn",
     LastName = "Laarman"
    };
  

  The IndexDocument<T> and IndexDocumentAsync<T> methods provide a simple way to index a single document of type T, using default parameters. The result of this method call can be inspected to determine if the indexing operation was successful.

  // synchronous method that returns an IIndexResponse object
    var indexResponse = client.IndexDocument(person);
    //asynchronous method that returns a Task<IIndexResponse> that can be awaited
    var indexResponseAsync = await client.IndexDocumentAsync(person);
    // Inspect the result of the synchronous operation
    if (!indexResponse.IsValid)
    {
     // If the request isn't valid, we can take action here
    }
  

  The IsValid property can be used to check if a response is functionally valid or not. This is a NEST abstraction to have a single point to check whether something wrong happened with the request.

  If you need to set additional parameters when indexing a document, you can use the fluent or object initializer syntax. This will give you finer control over the indexing process. In the example below we will index the document into an index named “people”.

  // fluent syntax
   var fluentIndexResponse = client.Index(person, i => i.Index("people"));
    // object initializer syntax
   var initializerIndexResponse = client.Index(new IndexRequest<Person>(person, "people"));
  

  A naïve approach to indexing multiple documents would be to simply create a loop to index a single document on each iteration; however, this is a very inefficient approach that will not scale well for large document collections.

  Multiple documents

  The bulk API can be used for indexing multiple documents. First, let’s create a collection of documents to index:

  var people = new []
    {
     new Person
     {
        Id = 1,
        FirstName = "Martijn",
        LastName = "Laarman"
     },
     new Person
     {
        Id = 2,
        FirstName = "Stuart",
        LastName = "Cam"
     },
     new Person
     {
        Id = 3,
        FirstName = "Russ",
        LastName = "Cam"
     }
     // snip
    };
  

  Multiple documents can be indexed using the IndexMany and IndexManyAsync methods, either synchronously or asynchronously, respectively. These methods are specific to the NEST client and wrap calls to the client's Bulk method and bulk API, providing a convenient shortcut for indexing many documents.

  Note that these methods index all documents in a single HTTP request, so for very large document collections, you need to partition the collection into many smaller batches and issue multiple Bulk calls. When you find yourself needing to do this, consider using the BulkAllObservable<T> helper instead, described later in the post.

  // synchronous method that returns an `IBulkResponse`
    var indexManyResponse = client.IndexMany(people);
    if (indexManyResponse.Errors)
    {
      // the response can be inspected for errors
       foreach (var itemWithError in indexManyResponse.ItemsWithErrors)
     {
        // if there are errors, they can be enumerated and inspected
        Console.WriteLine("Failed to index document {0}: {1}",
                          itemWithError.Id, itemWithError.Error);
     }
    }
    // alternatively, documents can be indexed asynchronously
    var indexManyAsyncResponse = await client.IndexManyAsync(people);
  

  If you require finer-grained control over indexing many documents you can use the Bulk and BulkAsync methods and use the descriptors to customise the bulk calls.

  As with the IndexMany methods above, documents are sent to the _bulk endpoint in a single HTTP request. This does mean that consideration will need to be given to the overall size of the HTTP request. For indexing large numbers of documents you'll likely want to use the BulkAllObservable<T> helper.

   // returns an `IBulkResponse` that can be inspected for errors
    var bulkIndexResponse = client.Bulk(b => b
     .Index("people")
     .IndexMany(people));
    // asynchronous version
    var asyncBulkIndexResponse = await client.BulkAsync(b => b
     .Index("people")
     .IndexMany(people));
  

  BulkAllObservable<T> helper

  Using the BulkAllObservable<T> helper allows you to focus on the overall objective of indexing a collection of documents, without having to concern yourself with retry, backoff or batching mechanics.

  Multiple documents can be indexed using the BulkAll method and BlockingSubscribeExtensions Wait() extension method. This helper exposes functionality to automatically retry / backoff in the event of an indexing failure, and to control the number of documents indexed in a single HTTP request.

  In the following example, each request indexes 1000 documents, batched from the original input. In the event of a large number of documents this could result in many HTTP requests, each containing 1000 documents (the last request may contain less, depending on the total number).

  The helper lazily enumerates an IEnumerable<T> collection, allowing you to index a large number of documents easily, such as documents materialized from paginated database records.

  var bulkAllObservable = client.BulkAll(people, b => b
     .Index("people")
     .BackOffTime("30s") // how long to wait between retries
     .BackOffRetries(2) // how many retries are attempted if a failure occurs
     .RefreshOnCompleted()
     .MaxDegreeOfParallelism(Environment.ProcessorCount)
     .Size(1000) // number of items per bulk request
    )
    .Wait(TimeSpan.FromMinutes(15), next => // perform the indexing and wait up to 15 minutes, whilst the BulkAll calls are asynchronous this is a blocking operation
    {
     // do something e.g. write number of pages to console
    });
  

  The BulkAllObservable<T> helper exposes a number of advanced features.

  1. BufferToBulk allows for the customisation of individual operations within the bulk request before it is dispatched to the server.
  2. RetryDocumentPredicate enables fine control on deciding if a document that failed to be indexed should be retried.
  3. DroppedDocumentCallback in the event a document is not indexed, even after retrying, this delegate is called.

  client.BulkAll(people, b => b
       .BufferToBulk((descriptor, list) =>
       {
          // customise the individual operations in the bulk
          // request before it is dispatched
          foreach (var item in list)
          {
             // index each document into either even-index or odd-index
             descriptor.Index<Person>(bi => bi
                .Index(item.Id % 2 == 0 ? "even-index" : "odd-index")
                .Document(item)
             );
          }
       })
       .RetryDocumentPredicate((item, person) =>
       {
          // decide if a document should be retried in the event of a failure
          if (item.Error.Index == "even-index"
              && person.FirstName == "Martijn")
          {
             return true;
          }
          return false;
       })
       .DroppedDocumentCallback((item, person) =>
       {
          // if a document cannot be indexed this delegate is called
          Console.WriteLine($"Unable to index: {item} {person}");
       }));
  

  Ingest nodes

  Since Elasticsearch will automatically reroute ingest requests to ingest nodes, you don't have to specify or configure any routing information. However, if you're doing heavy ingestion and have dedicated ingest nodes, it makes sense to send index requests to these nodes directly, to avoid any extra hops in the cluster.

  The simplest way to achieve this is to create a dedicated "indexing" client instance, and use it for indexing requests.

  // list of ingest nodes
    var pool = new StaticConnectionPool(new []
    {
     new Uri("http://ingestnode1:9200"),
     new Uri("http://ingestnode2:9200"),
     new Uri("http://ingestnode3:9200")
    });
    var settings = new ConnectionSettings(pool);
    var indexingClient = new ElasticClient(settings);
  

  In complex cluster configurations it can be easier to use a sniffing connection pool along with a node predicate to filter out the nodes that have ingest capabilities. This allows you to customise the cluster and not have to reconfigure the client.

   // list of cluster nodes
    var pool = new SniffingConnectionPool(new []
    {
     new Uri("http://node1:9200"),
     new Uri("http://node2:9200"),
     new Uri("http://node3:9200")
    });
    // predicate to select only nodes with ingest capabilities
    var settings = new ConnectionSettings(pool).NodePredicate(n => n.IngestEnabled);
    var indexingClient = new ElasticClient(settings);
  

  Ingest pipelines

  Let’s modify our Person type to include some additional information:

  public class Person
    {
     public int Id { get; set; }
     public string FirstName { get; set; }
     public string LastName { get; set; }
     public string IpAddress { get; set; }
     public GeoIp GeoIp { get; set; }
    }
    public class GeoIp
    {
     public string CityName { get; set; }
     public string ContinentName { get; set; }
     public string CountryIsoCode { get; set; }
     public GeoLocation Location { get; set; }
     public string RegionName { get; set; }
    }
  

  We can create an ingestion pipeline that manipulates the incoming values before they are indexed. Let’s assume that our application always expects surnames to be capitalised, and for initials to be indexed into their own field. We also have an IP address that we'd like to convert into human-readable location.

  We could achieve this requirement by creating a custom mapping and creating an ingest pipeline. The new Person type can then be used without making any further changes.

  First, we’ll create the index and custom mapping:

  client.CreateIndex("people", c => c
     .Mappings(ms => ms
        .Map<Person>(p => p
           .AutoMap() //automatically create the mapping from the type
           .Properties(props => props
                 .Keyword(t => t.Name("initials")) // create an additional field to store the initials
                 .Ip(t => t.Name(dv => dv.IpAddress)) //map field as IP Address type
                 .Object<GeoIp>(t => t.Name(dv => dv.GeoIp)) // map GeoIp as object
           )
        )
     )
    );
  

  Next we’ll create an ingest pipeline, taking advantage of the bundled ingest-geoip plugin, now bundled in version 6.7.

  client.PutPipeline("person-pipeline", p => p
     .Processors(ps => ps
        .Uppercase<Person>(s => s
           .Field(t => t.LastName) //uppercase the lastname
        )
        .Script(s => s
           .Lang("painless") // use a painless script to populate the new field
           .Source("ctx.initials = ctx.firstName.substring(0,1) + ctx.lastName.substring(0,1)")
        )
        .GeoIp<Person>(s => s // use ingest-geoip plugin to enrich the GeoIp object from the supplied IP Address
           .Field(i => i.IpAddress)
           .TargetField(i => i.GeoIp)
        )
     )
    );
  

  Now let’s index a Person instance using this new index and ingest pipeline.

  var person = new Person
    {
     Id = 1,
     FirstName = "Martijn",
     LastName = "Laarman",
     IpAddress = "139.130.4.5"
    };
    // index the document using the created pipeline
    var indexResponse = client.Index(person, p => p.Index("people")
                                                   .Pipeline("person-pipeline"));
  

  Searching now shows the indexed document with the enriched values.

  {
    "took": 5,
    "timed_out": false,
    "_shards": {
      "total": 1,
      "successful": 1,
      "skipped": 0,
      "failed": 0
    },
    "hits": {
      "total": 1,
      "max_score": 1,
      "hits": [
        {
          "_index": "people",
          "_type": "person",
          "_id": "1",
          "_score": 1,
          "_source": {
            "firstName": "Martijn",
            "lastName": "LAARMAN",
            "initials": "ML",
            "geoIp": {
              "continent_name": "Oceania",
              "region_iso_code": "AU-NSW",
              "city_name": "Sydney",
              "country_iso_code": "AU",
              "region_name": "New South Wales",
              "location": {
                "lon": 151.2167,
                "lat": -33.7333
              }
            },
            "ipAddress": "139.130.4.5",
            "id": 1
          }
        }
      ]
    }
  }
  

  When a pipeline is specified, there will be the added overhead of document enrichment when indexing; in the example given above, the execution of the uppercasing and the Painless script.

  For large bulk requests, it could be prudent to increase the default indexing timeout to avoid exceptions.

  client.Bulk(b => b
     .Index("people")
     .Pipeline("person-pipeline")
     .Timeout("5m") //increases the server-side bulk timeout
     .Index<Person>(/*snip*/)
     .Index<Person>(/*snip*/)
     .Index<Person>(/*snip*/)
     .RequestConfiguration(rc => rc
         .RequestTimeout(TimeSpan.FromMinutes(5)) // increases the HTTP request timeout
     )
    );
  

  In summary

  In this blog post we have covered the simple case of indexing a single document, through to bulk indexing multiple documents with ingest pipelines.

  Give it a try in your own cluster, or spin up a 14-day free trial of the Elasticsearch Service on Elastic Cloud. And if you run into any problems or have any questions, reach out on the Discuss forums.

  For the full documentation of indexing using the NEST Elasticsearch .NET client please refer to our docs.

  
  
  https://www.elastic.co/blog/indexing-documents-with-the-nest-elasticsearch-net-client
• 

  Helper Node Series | Sarah Hersh: Tutoring Students with Reading Partners
  • sorangutan  

  1
  1
  Posts
  61
  Views

  

  Elastic believes in giving back, both by supporting the communities in which we live and work, and recognizing that participating in these activities can inspire and enrich our lives in unexpected ways. That’s why we offer our Elasticians volunteer time off (VTO) to foster a culture of caring.

  The Helper Node blog series shines a light on the inspiring ways our Elasticians use their volunteer hours to help their community and the world at large. In this edition of the Helper Node series we meet Sarah Hersh, who volunteers her time tutoring students to improve their reading skills.

  ---

  Can you tell us a little bit about yourself, what do you do at Elastic?

  Sarah Hersh: Hi! My name is Sarah Hersh, and I live in Washington, DC. I joined Elastic last fall and I work as a technical writer on the Kibana team.

  How do you use your volunteer time off (VTO)?

  Sarah: When I joined Elastic, I was really excited to learn about the VTO program. I wanted to find a volunteer opportunity in my local community. I was a middle school teacher for a few years, so I hoped to find something related to education. I did some research and found a program that seemed like a perfect fit: Reading Partners. They’re a national organization, but they partner directly with local schools. The program is designed for students from kindergarten through 4th grade who need extra support with their reading skills. Their mission is, “to help children become lifelong readers and learners by empowering communities to provide individualized instruction with measurable results.”

  

  Sarah Hersh in her library.

  
  

  What is your biggest challenge or roadblock to being involved?

  Sarah: The biggest challenge was probably just figuring out the logistics up front: doing research into different local programs, making phone calls, attending trainings, and figuring out the DC public schools security clearance process. But once I worked through those initial roadblocks, the actual volunteering hours have been pretty easy to work into my schedule.

  It’s really a luxury to have the volunteer time off that Elastic provides; my manager was really supportive and I signed up for a time slot at the beginning of the day, so it doesn’t really impact my work week too much. It might sound like taking an hour away from work each week would be disruptive, but once you get started, it just becomes part of your routine.

  Why did you end up choosing this organization?

  Sarah: During my volunteer orientation, I learned that only about 30% of fourth graders in the US can read proficiently. In Washington, DC, that statistic is even lower, and only about 15% of students are reading at grade level by the time they are in 4th grade (4th grade reading proficiency has been shown to be a strong predictor of future educational outcomes). Those statistics were really motivating, and when I read more about the program's impact, I felt confident that volunteering with them would be time well spent.

  

  Reading Partners has locations all across the country.

  
  

  What does your day look like when volunteering?

  Sarah: Once a week I volunteer for an hour at an elementary school in southeast DC. I’m paired with a six-year-old who has a lot of energy! We start the hour by reading together, and then we work on vocabulary and other reading comprehension skills. Volunteering with Reading Partners is great, because they provide a full curriculum with evidence-based practices for each lesson. Today we had a little time left at the end of the lesson, so we played a game of tic-tac-toe with her sight words.

  What’s your one take away from this experience?

  Sarah: My main takeaway is that one-on-one attention really matters to kids. I can see how the lessons that we follow each week help the student that I work with feel more confident as she sees herself gaining skills, and mastering new sounds and words. I’ve met volunteers of all ages and backgrounds in the reading center, but to me, the common thread is that anyone who makes the commitment to show up, follow the curriculum, and really pay attention to a child’s effort and progress can make a big impact.

  The other important takeaway is that it’s also just a lot of fun to get to hang out with a six-year-old! It’s a great change from my normal routine and provides good perspective on everything else I might have going on during the week.

  Any last thoughts?

  Sarah: I’d encourage anyone who is interested in literacy tutoring to learn more about Reading Partners. They have locations across the country, so they might have partnerships with schools near you! Beyond that, I’d really encourage people to think about finding a local volunteer opportunity. This experience takes me out of my normal routine once a week and helps me feel more connected to my community; it’s great to feel like one hour a week can really make a difference.

  Interested in joining Elastic? Learn more about our teams and career opportunities in the US (and distributed) and apply today.

  
  
  https://www.elastic.co/blog/helper-node-series-sarah-hersh-tutoring-students-with-reading-partners
• 

  This Week in Elasticsearch and Apache Lucene - 2019-04-26
  • sorangutan  

  1
  1
  Posts
  72
  Views

  

  ### Elasticsearch #### Watcher UI There is a PR in code review which will [add support for threshold alert actions](https://github.com/elastic/kibana/pull/35175). Another has been merged a PR that [updates the threshold alert visualization with a chart from the elastic-charts library](https://github.com/elastic/kibana/pull/35404) and one more to add [support for three new comparison types when build a threshold alert](https://github.com/elastic/kibana/pull/35614). We have also [refined the UI of the watch history detail panel](https://github.com/elastic/kibana/pull/35462) and [added 403 and 404 error feedback](https://github.com/elastic/kibana/pull/35444). ![Create Threshold Alert UI][1] #### Enrich Processor We have continued to work on the [enrich processor](https://github.com/elastic/elasticsearch/issues/32789). The [groundwork](https://github.com/elastic/elasticsearch/pull/41010) that allow enrich processor to access a locally allocated shard has been merged in the feature branch. A [new PR](https://github.com/elastic/elasticsearch/pull/41521) has been opened that adds an enrich processor to decorate documents based on exact lookups. This PR also adds a special field mapper that allows the values being decorated to be retrieved in a fast manner. We have some changes in the works to add [get](https://github.com/elastic/elasticsearch/pull/41384) and [delete](https://github.com/elastic/elasticsearch/pull/41495) enrich policy APIs and we have completed the put policy API. We also [simplified the EnrichStore](https://github.com/elastic/elasticsearch/pull/41421) (internal helper used to store and access enrich policies), so that it can be used more easily in the various places where it is needed. #### Consistent Settings We are working on [a feature](https://github.com/elastic/elasticsearch/pull/40416) that will allow us to ensure certain secure (keystore) settings are consistent across the whole cluster. #### API Key privileges there are now [some additional cluster privileges](https://github.com/elastic/elasticsearch/pull/41488) that will allow API keys to be useful for users that do not have superuser/manage_security privileges in the cluster. #### Block-Max Score We have [disabled max-score optimization](https://github.com/elastic/elasticsearch/pull/41361) on queries that contain a mandatory scoring clause with unbounded max score. Lucene 8 has the ability to skip blocks of non-competitive documents. However some queries don't track their maximum score (script_score, span, ...) so they always return Float.POSITIVE_INFINITY as maximum score. This can slow down some boolean queries if other clauses have bounded max scores. #### Field Capabilities The Field Capabilities API will now report fields that are [missing in some indices](https://github.com/elastic/elasticsearch/pull/34071) in an "unmapped" section. Following this change we have [enhanced the index resolution in SQL](https://github.com/elastic/elasticsearch/pull/41377) to use field_caps not just for merging but also for individual table discovery (used inside metadata). #### Querying frozen indices in SQL [SQL now supports frozen indices](https://github.com/elastic/elasticsearch/pull/41558). In SQL you can now indicate that you want index resolution to include frozen indices in two ways: 1. Through a dedicated `FROZEN` grammer extension (e.g. `SELECT field FROM FROZEN index`) 2. Through a configuration parameter on the drivers `index.include.frozen: true` #### Conditional logic in SQL SELECTs We finished up with implementation of [CASE](https://github.com/elastic/elasticsearch/pull/41349). CASE is a powerful ANSI SQL expression which implements the IF.. THEN.. ELSE.. logic of programming languages. It can be used in the SELECT, WHERE, GROUP BY, ORDER BY & HAVING clauses. Here is an example: ``` SELECT count(*) AS count, CASE WHEN NVL(languages, 0) = 0 THEN 'zero' WHEN languages = 1 THEN 'one' WHEN languages = 2 THEN 'bilingual' WHEN languages = 3 THEN 'trilingual' ELSE 'multilingual' END as lang_skills FROM test_emp GROUP BY lang_skills ORDER BY 2; ``` #### Snapshot resiliency The snapshot repository plugin testing for Azure is now on par with the testing we have for S3 and GCS. We have worked on adding [third-party tests for Azure](https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+third-party-tests-azure/), i.e., CI jobs that run some of our snapshot/restore related tests against the actual Azure service instead of just mocks as we do in local test runs. This also completes the work to add CI capabilities that run basic snapshot/restore tests against the real infrastructure of the three major Cloud providers. Based on the [recent work](https://github.com/elastic/elasticsearch/pull/40322) that allows snapshot repositories to implement deletes more efficiently using bulk operations, we are [switching](https://github.com/elastic/elasticsearch/pull/41368) the GCS repository to this new feature by making use of GCS's capability to batch deletes, which will significantly speed up snapshot deletions on a GCS repository. The same functionality was [recently](https://github.com/elastic/elasticsearch/pull/40322) added for S3 as well. #### Cluster coordination We worked with an external contributor to [fix the sample configuration files in our docker-compose docs](https://github.com/elastic/elasticsearch/issues/41394), which were missing a vital discovery.seed_nodes setting for all nodes, resulting in a cluster to form fine only the first time but then fail to properly restart half of the time. We have also adapted the logging output of the functionality where we periodically log a warning message with detailed info when a node cannot discover a master or elect a master, making sure to [filter out master-ineligible nodes](https://github.com/elastic/elasticsearch/issues/41344) from the cluster state, as these nodes are just adding noise and confusion to the log message. ### Lucene #### Backward offsets Users are [complaining](https://issues.apache.org/jira/browse/LUCENE-8776) that the fact that Lucene started [rejecting backward offsets](https://issues.apache.org/jira/browse/LUCENE-7626) in 7.0 breaks their use-case since they no longer can index compound terms as a synonym of their sub terms while preserving offsets. This is unfortunate but the ongoing discussion suggests that Lucene will keep rejecting backward offsets since it allows to implement some algorithms without backtracking and allows to encode offsets more efficiently. #### Performance regressions with Java 11 Since Lucene master now requires Java 11, the benchmarks were upgraded to Java 11 as well, but [it was found that it triggered a slight slowdown](https://markmail.org/message/d2pklgnmkpr4fk2i). We suspect it is due to the change of the default garbage collector from ParallelGC to G1. The benchmarks will temporarily force the ParallelGC collector to isolate whether this is actually due to the garbage collector. #### Other * Work is ongoing to [align the Luwak codebase with Lucene](https://issues.apache.org/jira/browse/LUCENE-8766) to prepare the donation of Luwak to Lucene. * We have javadocs for analysis components, but we generally don't [document the name of these components](https://issues.apache.org/jira/browse/LUCENE-8778), making them hard to used. * An Elasticsearch user reported that [MinHashFilter generates illegal unicode](https://issues.apache.org/jira/browse/LUCENE-8779). * [Corner cases in the Tessellator were fixed](https://issues.apache.org/jira/browse/LUCENE-8775). *How should Lucene [map segments to threads](https://issues.apache.org/jira/browse/LUCENE-8757) when configured to parallelize query execution? ### Changes #### Changes in Elasticsearch Changes in 8.0: * Make 0 as invalid value for `min_children` in `has_child` query [#41347](https://github.com/elastic/elasticsearch/pull/41347) Changes in 7.1: * Disable max score optimization for queries with unbounded max scores [#41361](https://github.com/elastic/elasticsearch/pull/41361) * Deprecate support for first line empty in msearch API [#41442](https://github.com/elastic/elasticsearch/pull/41442) * Improve accuracy for Geo Centroid Aggregation [#41033](https://github.com/elastic/elasticsearch/pull/41033) * Disallow null/empty or duplicate composite sources [#41359](https://github.com/elastic/elasticsearch/pull/41359) * Peer recovery should not indefinitely retry on mapping error [#41099](https://github.com/elastic/elasticsearch/pull/41099) * SSLDriver can transition to CLOSED in handshake [#41458](https://github.com/elastic/elasticsearch/pull/41458) * Introduce aliases version [#41397](https://github.com/elastic/elasticsearch/pull/41397) * fix #35262 define deprecations of API's as a whole and urls [#39063](https://github.com/elastic/elasticsearch/pull/39063) * SQL: Implement IIF(, , ) [#41420](https://github.com/elastic/elasticsearch/pull/41420) * SQL: Use field caps inside DESCRIBE TABLE as well [#41377](https://github.com/elastic/elasticsearch/pull/41377) * SQL: Implement CASE... WHEN... THEN... ELSE... END [#41349](https://github.com/elastic/elasticsearch/pull/41349) * Add ignore_above in ICUCollationKeywordFieldMapper [#40414](https://github.com/elastic/elasticsearch/pull/40414) * Move keystore-cli to its own tools project [#40787](https://github.com/elastic/elasticsearch/pull/40787) * Omit non-masters in ClusterFormationFailureHelper [#41344](https://github.com/elastic/elasticsearch/pull/41344) * Handle unmapped fields in _field_caps API [#34071](https://github.com/elastic/elasticsearch/pull/34071) Changes in 6.7: * Fix Has Privilege API check on restricted indices [#41226](https://github.com/elastic/elasticsearch/pull/41226) * Fix role mapping DN field wildcards for users with NULL DNs [#41343](https://github.com/elastic/elasticsearch/pull/41343) * Reduce security permissions in CCR plugin [#41391](https://github.com/elastic/elasticsearch/pull/41391) * SQL: Fix bug with optimization of null related conditionals [#41355](https://github.com/elastic/elasticsearch/pull/41355) #### Changes in Elasticsearch Management UI Changes in 6.7: * [CCR] Retrieve paused state of follower index from ES instead of depending upon the client to provide it [#35342](https://github.com/elastic/kibana/pull/35342) * [CCR] Allow user to use CCR when security is not enabled. [#35333](https://github.com/elastic/kibana/pull/35333) #### Changes in Rally Tracks * Update target throughput [#74](https://github.com/elastic/rally-tracks/pull/74) [1]: https://images.contentstack.io/v3https://www.elastic.co/assets/bltefdd0b53724fa2ce/blta7f81974851746a4/5cc31327ee4bf29d69c58d00/threshold-alert-ui.png
  
  https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2019-04-26
• 

  This Week in Elasticsearch and Apache Lucene - 2019-05-05
  • sorangutan  

  1
  1
  Posts
  97
  Views

  

  ### Elasticsearch #### Nightly Benchmarks We've created a new benchmarking environment serving default distribution benchmarks and migrated [https://elasticsearch-benchmarks.elastic.co](https://elasticsearch-benchmarks.elastic.co/). You'll notice that for each of the datasets, you can select either `OSS` or `Default` to follow benchmarking results for each distribution of Elasticsearch. #### Documentation Improvements After helping with a couple support tickets related to configuring synonyms, we've [added documentation](https://github.com/elastic/elasticsearch/pull/41645) around how to best use synonyms with token filters such as `word_delimiter` that produce stacked tokens. We are also [reworking the docs for the discovery-ec2 plugin](https://github.com/elastic/elasticsearch/issues/41630). #### Geo_line Aggregation We recently opened the [geo_line PR](https://github.com/elastic/elasticsearch/pull/41612).  This is an aggregation which consumes a series of points and a sort value (e.g. time) and sorts those points into a linestring.  An example use-case is GPS coordinates logged periodically by taxis, container ships, etc.  These individual points are much more useful when arranged chronologically in a line. Here's what that looks like: ![Geo Lines][1] #### Community PR's We love it when our community submits pull requests to Elasticsearch! Thank you to all of our contributors, past and present. Here are a few recent community pull requests: We [reviewed a community PR](https://github.com/elastic/elasticsearch/pull/41404) to reject port ranges in `discovery.seed_hosts`, which were previously accepted but silently ignored by Elasticsearch. We're [reviewing a community PR](https://github.com/elastic/elasticsearch/pull/41489) that is adding the index name to cluster block exceptions. These exceptions are triggered for example when trying to write to an index with a read-only cluster block, which is automatically added by the system when nodes holding this index are running low on disk space. We're [reviewing a community PR](https://github.com/elastic/elasticsearch/pull/41050) that allows running `_cluster/reroute` commands even if the maximum number of retries limit for failed shard allocations has been reached. #### Data Replication Resiliency We've [updated](https://github.com/elastic/elasticsearch/pull/41522) the [resiliency status page](https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html), closing off two important issues that were addressed by a multi-year effort, starting with the sequence numbers project in the 5.x and 6.x series, and culminating in the release of 7.0 with the new cluster coordination subsystem: - *Documents indexed during a network partition cannot be uniquely identified*: We have switched [optimistic concurrency control](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/optimistic-concurrency-control.html) (OCC) from the `_version` field to the new `_seq_no` (sequence number) and `_primary_term` (primary term) fields, which do uniquely identify each operation. To be clear, the `_version` field continues to not uniquely identify a particular version of a document; if you need to do this then you should move to using the `_seq_no` and `_primary_term` fields. All internal consumers that are making use of OCC (e.g. reindex, update-by-query, ...) have been switched to these new fields as well. - *Replicas can fall out of sync when a primary shard fails*: After a primary failover, the new primary now realigns the replicas with itself by rolling back the replicas to a safe point in the history and sending it the missing operations. We've recently also worked on more extensive testing in these areas. We added tests for our new optimistic concurrency control structures (`if_seq_no` and `if_primary_term`), checking [linearizability of compare-and-set operations](https://github.com/elastic/elasticsearch/pull/38561). Based on this work, we learned quite a bit about the guarantees that the system provides under certain failure conditions, and will look at strengthening some of those guarantees. This work also helped in uncovering bugs. We also fixed a problem where a shard that was being [closed during a replica rollback](https://github.com/elastic/elasticsearch/pull/41584) was keeping an active index writer around, causing various follow-up checks to fail. We've added [stronger consistency checks](https://github.com/elastic/elasticsearch/pull/41614) to our disruption test suite to verify that, beside the existing checks on `_seq_no` and `_primary_term` fields, also `_source` and `_version` fields were fully aligned across all shard copies to make sure that at the end of each disruption test all shard copies would contain exactly the same set of documents. #### Token Service Our long running [change to move security tokens into their own index](https://github.com/elastic/elasticsearch/pull/40742) (and out of the main security index) has been merged. This has been a pretty big effort as we want the change to happen automatically when you upgrade the cluster to a new (7.2+) version, but not happen during a mixed cluster (when some nodes are on <7.2), and for it to seamlessly accept any active tokens that were created on the old cluster version and are stored in the main security index. This is part of our long running effort to make it easier and safer to backup & restore the security index. We've made a range of changes to the token service for 7.2, some of which forced us to change the format of the tokens that we provide to client and how we stored them in the index. We are tackling the last part of those changes which is to [change the way we store token strings in the index](https://github.com/elastic/elasticsearch/issues/40765) so that getting read access to the tokens index does not allow you to authenticate with someone else's token. (All previously released versions of ES had this protection, but the way that was implemented didn't fit with the changes we were making in 7.2, so we removed it during this development cycle and are implementing a different solution before feature freeze for 7.2). #### Enrich Processor The Enrich Processor (formerly referred to as the "Lookup Processor") will allow users to define ingest pipelines that _enrich_ the ingested document with data from another index on the cluster (subject to some limitations). There is still a lot of work left to do, but the core pieces are are coming together, which include data-local Lucene queries, a policy runner, and the REST APIs. If you'd like to follow our progress or provide feedback, feel free to check out the [meta issue](https://github.com/elastic/elasticsearch/issues/32789). We've [merged](https://github.com/elastic/elasticsearch/pull/41532) the first iteration of the enrich processor to the feature branch. We've also [merged](https://github.com/elastic/elasticsearch/pull/41707) the first iteration of the enrich policy runner. The enrich policy runner is the background task that reads the enrich policy and synchronizes the source index to the specialized .enrich index. Under the covers this is implemented with the re-index API and will eventually get support for a cron scheduler. Finally, we [added an API](https://github.com/elastic/elasticsearch/pull/41553) to list enrich policies and have started the work on the `_execute` API to allow a user to manually run a policy. #### Snapshot Lifecycle Management The first cut of the [SLM documentation](https://github.com/elastic/elasticsearch/pull/41510) and an API path change from `/ilm/` to `/slm/` have now been merged to the [feature branch](https://github.com/elastic/elasticsearch/tree/snapshot-lifecycle-management). We've also [introduced](https://github.com/elastic/elasticsearch/pull/41607) two new roles: `manage_slm` and `read_slm` to allow configuration for more fine-grained permissions. Finally, we've [started](https://github.com/elastic/elasticsearch/pull/41707) the work to store the results from SLM's snapshot creation to a dedicated history index. This will allow us to set up alerts and have a history of failed/successful snapshots. ### Lucene #### Apache Lucene / Solr 8.1 The release branch for Apache Lucene / Solr 8.1 has been cut and the release process has started. We await the first RC later this or early next week. For Lucene in particular this will bring: * a new BKD tree strategy for segment merging providing significant performance boost for high dimensions * the new [Luke](https://issues.apache.org/jira/browse/LUCENE-2562) module * new query visitor [API](https://github.com/apache/lucene-solr/pull/581) allowing to traverse a query tree efficiently * [read time attributes](https://github.com/apache/lucene-solr/pull/640) that allow to control codec level functionality on a per reader basis for instance to load FSTs per field off-heap. #### Other * We're [working on the Luwak codebase with Lucene](https://issues.apache.org/jira/browse/LUCENE-8766) to prepare the donation of Luwak to Lucene. * Can we [improve](https://issues.apache.org/jira/browse/LUCENE-8788) search performance by sorting the segments by an estimated number of hits? * Some of our JDK 11 builds are hitting a [JVM Bug](https://bugs.openjdk.java.net/browse/JDK-8205399) that's fixed in 12 but not in 11. * JDK 12 doesn't seem to be bug free either - lucene is [hitting](https://issues.apache.org/jira/browse/LUCENE-8668) [this](https://bugs.openjdk.java.net/browse/JDK-8219448) bug frequently. * We are still [discussing](https://issues.apache.org/jira/browse/LUCENE-8757) how we can slice up segments better for parallel search * One persons [bug](https://issues.apache.org/jira/browse/LUCENE-8776) is another persons feature... * Spooky failures are actually [bugs](https://issues.apache.org/jira/browse/LUCENE-8785) sometimes. ### Changes #### Changes in Elasticsearch Changes in 8.0: * Update TLS ciphers and protocols for JDK 11 [#41385](https://github.com/elastic/elasticsearch/pull/41385) * BREAKING: Parse empty first line in msearch request body as action metadata [#41011](https://github.com/elastic/elasticsearch/pull/41011) * Suppress illegal access in plugin install [#41620](https://github.com/elastic/elasticsearch/pull/41620) * Fix: added missing skip [#41492](https://github.com/elastic/elasticsearch/pull/41492) Changes in 7.2: * Improve error message for ln/log with negative results in function score [#41609](https://github.com/elastic/elasticsearch/pull/41609) * Add details to BulkShardRequest#getDescription() [#41711](https://github.com/elastic/elasticsearch/pull/41711) * Amend `prepareIndexIfNeededThenExecute` for security token refresh [#41697](https://github.com/elastic/elasticsearch/pull/41697) * Implement Bulk Deletes for GCS Repository [#41368](https://github.com/elastic/elasticsearch/pull/41368) * Security Tokens moved to a new separate index [#40742](https://github.com/elastic/elasticsearch/pull/40742) * Simplify initialization of max_seq_no of updates [#41161](https://github.com/elastic/elasticsearch/pull/41161) * Handle WRAP ops during SSL read [#41611](https://github.com/elastic/elasticsearch/pull/41611) * Upgrade to Netty 4.1.35 [#41499](https://github.com/elastic/elasticsearch/pull/41499) * Close and acquire commit during reset engine fix [#41584](https://github.com/elastic/elasticsearch/pull/41584) Changes in 7.0: * Run packaging tests on RHEL 8 [#41662](https://github.com/elastic/elasticsearch/pull/41662) * Fix multi-node parsing in voting config exclusions REST API [#41588](https://github.com/elastic/elasticsearch/pull/41588) Changes in 6.8: * Fix for full cluster restart tests [#41723](https://github.com/elastic/elasticsearch/pull/41723) * Drop distinction in entries for keystore [#41701](https://github.com/elastic/elasticsearch/pull/41701) * Fix Watcher deadlock that can cause in-abilty to index documents. [#41418](https://github.com/elastic/elasticsearch/pull/41418) Changes in 6.7: * Bump the bundled JDK to 12.0.1 [#41627](https://github.com/elastic/elasticsearch/pull/41627) * Change JDK distribution source [#41626](https://github.com/elastic/elasticsearch/pull/41626) #### Changes in Elasticsearch Management UI Changes in 7.0: * [ILM] Surface shrink action in edit form if it's already been set on the policy [#35987](https://github.com/elastic/kibana/pull/35987) #### Changes in Elasticsearch SQL ODBC Driver Changes in 7.2: * Add JDBC's protocol tests as integration tests [#149](https://github.com/elastic/elasticsearch-sql-odbc/pull/149) Changes in 6.7: * Consider interval's precision. Allow non-aligned period values as interval encoding [#148](https://github.com/elastic/elasticsearch-sql-odbc/pull/148) [1]: https://images.contentstack.io/v3https://www.elastic.co/assets/bltefdd0b53724fa2ce/blt5d470f15432be392/5cd04b9ae8ec6ef265db8946/geo_line.png
  
  https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2019-05-05
• 

  Share your #ElasticStories
  • sorangutan  

  1
  1
  Posts
  63
  Views

  

  We’re constantly amazed by the innovation and imagination our community displays with the Elastic Stack. We’ve seen use cases that we couldn’t have imagined. We showcase them on our blog, customer page, and at our Elastic{ON} Tour and community events.

  We do this because sharing these stories exposes new and unimaginable possibilities of what can be accomplished with the Elastic Stack. It’s true, knowledge is power.

  But a ton of awesome work engineered with the Elastic Stack by our wonderful community of users is going unnoticed by us and our community at large.

  That’s about to change — one tweet at a time.

  Introducing #ElasticStories

  Today, we’re launching the #ElasticStories hashtag for the benefit of the Elastic community. It’s an unfiltered venue to share Elastic Stack stories regardless of simplicity or complexity.

  Got a machine learning, Logstash or Canvas story? What about an application search or security use case? Wanna share how you've implemented maps, cross-cluster replication, or APM? Tell everybody about it in #ElasticStories. Don’t forget to include links or any artwork or screenshots you might have to illustrate how you leverage our products, solutions and services.

  Your story could involve an entire use case, a feature that made your life easier, a challenge you overcame, or a milestone you reached — you name it.

  Community Engagement

  The Elastic community, in addition to our products and services, is always evolving thanks to productive feedback from members of the community sharing advice and stories. The #ElasticStories hashtag on Twitter is another forum for the community’s voice to be aired and heard.
  

  But there’s much more in store for the #ElasticStories hashtag.

  We will retweet some of these astounding #ElasticStories tweets on our @Elastic account. What’s more, we plan on featuring select #ElasticStories tweets on the Elastic blog and in our Community newsletter. If we find a story that is super inspiring, we might post a writeup — by us or perhaps by you — on our website.

  Over and again we find ourselves both humbled and amazed by the Elastic community’s awe-inspiring uses of our products and services. We expect that state of mind to continue when we learn more from you at #ElasticStories.

  Let’s hear your Elastic stories!

  
  
  https://www.elastic.co/blog/share-your-elasticstories
• 

  Monitoring Microsoft SQL Server using Metricbeat and Elasticsearch
  • sorangutan  

  1
  1
  Posts
  840
  Views

  

  We are pleased to introduce support for Microsoft SQL Server as part of our generally available Metricbeat modules to capture database metrics. Microsoft SQL Server has been a popular and successful SQL database over the years, and we are really excited about this new addition to the Metricbeat family.

  Two metricsets are launching as beta with Metricbeat 7.0: performance and transaction_log. In addition, a curated dashboard was also developed to help navigate Microsoft SQL Server metrics in Kibana.

  Performance metric set

  The first thing that any user will want to look at is, as you have probably guessed, performance. It is very important is track the number of transactions, connected users, and cache hits, and all of this is included in this metricset. These metrics are retrieved from Performance Counters in Microsoft SQL Server

  • user_connections: Total number of user connections
  • logins_per_sec: Total number of logins started per second
  • connections_reset_per_sec: Total number of logins started from the connection pool
  • transactions: Total number of transactions
  • batch_requests_per_sec: Number of Transact-SQL command batches received per second
  • buffer.cache_hit.pct: The ratio of total number of cache hits divided by the total number of cache lookups over the last few thousand page accesses. After a long period of time, this ratio moves very little. Because reading from the cache is much less expensive than reading from disk, you’ll want this ratio to be high.
  • compilations_per_sec: Number of SQL compilations per second. This indicates the number of times the compile code path is entered. This includes compiles caused by statement-level recompilations in SQL Server. After SQL Server user activity is stable, this value reaches a steady state.

  Transaction Log metric set

  This is another key metricset to include in your monitoring, because it helps you understand the shape and size of the server log. Some of the key metrics included here are:

  • space_usage: This shows space usage information about the transaction log, a key metric to understand the performance of a Microsoft SQL Server. It also shows the time passed since the last backup of the log, useful for disaster recovery
  • stats: This returns summary level attributes and information on transaction log files of databases. Use this information for monitoring and diagnostics of transaction log health. Here you can see active size, total size, or backup time for example.

  Getting started with SQL Server monitoring using Metricbeat

  The easiest way to configure this module can be found on the "Add Data UI" screen in Kibana. This UI is a Basic feature, which means it's part of the free, default distribution of Kibana.

  Here you will find instructions to set up the module as shown on the following screen:

  

  Following the instructions, you will see how easy it is to activate the Microsoft SQL Server Metricbeat module by setting up the addresses of the SQL Servers, filling username and password, and enabling the module. Once enabled, you can start Metricbeat and set up the dashboards:

  ./metricbeat modules enable aws
  ./metricbeat -E setup.dashboards.enabled=true -e
  

  Microsoft SQL Server dashboards

  Two curated dashboards are included with the module — one for the Transaction Log, and the other for the Performance metricset. In the following screenshot you can see the "Transaction Log" dashboard.

  

  As shown in the screenshot, you can also select a specific database in the server to see a detailed view of it as well as comparing them in a single View.

  Try it out today

  Deploy a cluster on our Elasticsearch Service or download the stack, spin up the new Metricbeat module (you can also add more modules from the list of supported modules), open the curated dashboards in Kibana and start monitoring your Microsoft SQL Server today!

  
  
  https://www.elastic.co/blog/monitoring-microsoft-sql-server-using-metricbeat-and-elasticsearch
• 

  Dictionary update behavior for CJK language analyzers
  • sorangutan  

  1
  1
  Posts
  130
  Views

  

  Every year, many visitors come to Japan to see [sakura (Cherry blossoms)](https://en.wikipedia.org/wiki/Cherry_blossom) in spring. If you plan to visit Japan, you may want to search for “cherry blossom” in Japanese, or "さくら". When you search for "さくら" in Elasticsearch, you are using full-text search. Full-text search requires analyzers. Per this [blog](https://www.elastic.co/blog/how-to-search-ch-jp-kr-part-1), we can see the default standard analyzer is not the best for CJK (Chinese, Japanese, and Korean) text, and we need to use language analyzers. Moreover, for language analyzers, dictionaries play an important role in deciding how a tokenizer will split a word into tokens. For example, in the Japanese analyzer Kuromoji, by default, the word `東京スカイツリー` will be divided into three parts: `東京`, `スカイ` and `ツリー`. However, `スカイツリー`, which means `Skytree`, is a meaningful word referring to the [Tokyo Skytree broadcasting tower](https://en.wikipedia.org/wiki/Tokyo_Skytree) in Japan. In this case, `sky` and `tree` should not be split into separate tokens. To achieve meaningful tokenization (word breaking), we can use the user dictionary provided in the [Kuromoji tokenizer](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji-tokenizer.html). Besides the user dictionary, there are also synonym dictionaries that can be applied for fine-grained tuning. Recently we have received questions related to the behavior of dictionary updates in Elasticsearch indices. In this blog, we’d like to go over some general concepts to help users gain a better understanding of dictionary updates and improve their search experience. This article is primarily oriented towards CJK analyzers, but the basic concept is the same for other languages as well. Background knowledge --- Analyzers work by applying, in order, a character filter, a tokenizer, and then a token filter. By default, analyzer settings are applied at both [indexing](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/analysis.html#_index_time_analysis) time and [search](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/analysis.html#_search_time_analysis) time. The target of index time analysis is the source data, and the target of search time analysis is the query string. Thus, it's important to be aware that changes to the dictionary also affect both indexing and searching. Dictionary update behavior --- In Elasticsearch, the analyzer reads the dictionary when opening/loading the index, and in general will not read the settings again after loading. Thus, in order to reflect a dictionary update, you need to restart the node that has the dictionary files, or you can simply use the `_close` and then `_open` APIs of the target index. However, it should be noted here that the dictionary update will not be applied to documents that have already been indexed. This is because these documents were indexed using the analyzer before the dictionary was updated. For details on how to reflect the change in existing indexed documents, see _"How to reflect dictionary updates in existing indices"_ below. Updates will take effect on documents that are indexed after the change, and on search queries as well. The search text will be newly tokenized according to the updated content defined in the dictionary. Impact on search results --- Depending on the situation, a dictionary update may or may not have an impact on search results (e.g., a no hit result). #### The good side: there is no impact on search results Suppose we add the following new definition to a dictionary to ensure that the terms `japan` and `sushi` are kept together as a single token: `japan, sushi` In documents indexed before the update, `japan` and `sushi` would be registered as separate tokens because these terms did not previously exist. But when a user searches for either of the terms `japan` or `sushi`, the search term will be looked up in a synonym dictionary, returning results for both terms. #### The flip side: there is an impact on search results As mentioned at the top of this article, some CJK language analyzers (e.g., [Japanese Kuromoji analyzer](https://www.elastic.co/guide/en/elasticsearch/plugins/7.0/analysis-kuromoji-tokenizer.html), [Korean Nori analyzer](https://www.elastic.co/guide/en/elasticsearch/plugins/7.0/analysis-nori-tokenizer.html), [Chinese IK analyzer](https://github.com/medcl/elasticsearch-analysis-ik#dictionary-configuration)) have a feature called a user dictionary (or a similar term), which can be used for tuning tokenization. For instance, in the Japanese Kuromoji analyzer, the user can update the dictionary to prevent `東京大学` from being divided into the two tokens `東京` and `大学` (the default behavior) and keep `東京大学` as one token. In this scenario, in existing documents indexed before the update, `東京大学` would be stored as two tokens, `東京` and `大学`, and there would be no single `東京大学` token. Thus, after the dictionary update, when the user searches for `東京大学`, the query will be performed with the single token `東京大学`, and will return no hits. How to reflect dictionary updates in existing indices --- In Elasticsearch, when updating an existing indexed document, the document is deleted and re-created internally, at which time the language analyzer processes the document again. Therefore, we can update the existing document to let the dictionary update take effect. A handy way to do this is to use the [`Update By Query`](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/docs-update-by-query.html) API to update whole indices or specific parts of an index. The document will then be analyzed again based on the updated dictionary. I hope this brief walkthrough was helpful in understanding how to create and update a dictionary. You can read our document/manual for more details regarding the settings of different analyzers ([Japanese Kuromoji analyzer](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji.html), [Korean Nori analyzer](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-nori.html), [Chinese IK analyzer](https://github.com/medcl/elasticsearch-analysis-ik)). If you have any concerns or questions, please feel free to raise them via our [discuss forum](https://discuss.elastic.co/c/elasticsearch). Our engineers are happy to help you any time.
  
  https://www.elastic.co/blog/dictionary-update-behavior-for-elasticsearch-cjk-language-analyzers
• 

  The Best Thing About Working at Elastic
  • sorangutan  

  1
  1
  Posts
  50
  Views

  

  What's best thing about working at Elastic? IT, Depends on who you ask.

  We decided to ask a few of our Elasticians to share their thoughts on what they love best about working at Elastic.
  
  

  "The best thing about working for Elastic is the people."

  
  
  

  "Elastic makes it really easy to find your place."

  
  
  

  "No one’s foreign and everyone is foreign."

  
  
  

  "We’re passionate about the work we do."

  
  
  

  Learn more by visiting our Source Code. 

  
  
  

  If you're interested in becoming an Elastician yourself, visit our teams page to see if an opportunity speaks to you. 

  
  
  https://www.elastic.co/blog/the-best-thing-about-working-at-elastic
• 

  Monitoring MySQL, Percona Server, and MariaDB with the Elastic Stack
  • sorangutan  

  1
  1
  Posts
  113
  Views

  

  Databases are a fundamental keystone of most applications, and there are many options for many different use cases. Even a single database can be used in a variety of ways, but regardless of the database type, implementation, or use case, the need for a good solution for monitoring them remains a common denominator. Databases keep the most precious information of any company, and any hiccup can be a reason for alarm.

  The majority of the most popular databases have a place in our solutions for logs and metrics, and we are continuously increasing our coverage by supporting more services and more versions. The first implementation of Metricbeat already included support for MySQL and Redis, and the recent 7.0 release came with a new module for Microsoft SQL Server.

  MySQL has a long history of success. Being included in the popular LAMP stack made it incredibly popular, first between enthusiasts of new technologies wanting to use it for their websites and blogs, and then in the biggest companies that saw in it as a free solution that offered a new opportunity to scale with any kind of hardware.

  MySQL modules for Metricbeat and Filebeat

  Metricbeat and Filebeat have MySQL well covered from the metrics and logs monitoring perspective. Let’s look at Filebeat first.

  Filebeat for MySQL

  The Filebeat module for MySQL includes two filesets, one for the database error log and another one for the slow query log.

  The MySQL error log contains a record of messages about the server startup and stop, and about some events that may happen while the service is running. This log is useful to diagnostic problems, it describes for example the reasons why some options cannot be applied, the causes of a replication failure, and it also informs when a table needs repair and if it could be automatically repaired.

  The error fileset parses messages from the MySQL error log to extract the severity of the message and the thread that caused the problem. This information is helpful when monitoring several MySQL nodes, so you can focus on the more problematic messages or you can check how often certain problems occur.
  

  

  The slowlog fileset is very useful for identifying activities that are consuming a high amount of database resources. The slow query log is a very well known feature for MySQL DBAs, and it includes a lot of information about the most expensive queries, like the query itself, its duration, some basic metrics about rows affected and data transferred, and also detailed information about the optimizations that were applied. This feature can be configured to be more or less detailed and to include different sets of fields. Most of these fields are parsed by the slowlog fileset, giving the possibility of performing advanced analytics on this information, allowing you to detect and take early decisions on specific queries that can be problematic for your applications and your databases.

  

  In the preconfigured dashboard, both filesets are combined to give an overview of the messages being reported by all the MySQL instances:

  

  Metricbeat for MySQL

  The Metricbeat module for MySQL collects information from the SHOW GLOBAL STATUS; query. This query is the most important provider of information about the status of a server, it includes many metrics about connections, threads, network, replication binlog, types of queries executed, etc. The most important of these metrics are collected by Metricbeat, so they can be used to monitor the status of your servers or take capacity decisions.

  It also includes a dashboard that offers a view on time of the most important fields:

  

  MySQL friends: Percona and MariaDB

  As mentioned before, MySQL has a story of success. From the original open source project, a whole community and ecosystem of several companies has grown up.

  Percona is one of the best known companies in the MySQL ecosystem. It is the developer of many solutions based on MySQL, including their own distribution of MySQL, curated for performance and security. Percona Server for MySQL includes backports of new features and bug fixes, as well as specific features to improve instrumentation and many other things. It is intended to be a drop-in replacement of MySQL server.

  MySQL AB, the company behind MySQL was acquired in 2008 by Sun Microsystems, which was later acquired by Oracle Corporation. These movements created some concerns in the community, prompting the creation of MariaDB as a fork of the original project. MariaDB is intended to remain free and open source under the GPL license, and is currently maintained by the MariaDB Foundation. Even if it is still in part compatible with MySQL server, it has completely different implementations of some features and it has its own release cycles. At the moment it is the default MySQL distribution in some Linux distributions and is offered by many cloud providers.

  In Metricbeat and Filebeat 6.7, we started to run our tests also on Percona and MariaDB, which allowed us to identify new sources of information. Since then we have increased the number of fields collected by the slowlog fileset, allowing even more insightful analysis of the slow query log.

  Metricbeat and Filebeat 7.0 are compatible with MySQL, Percona Server, and MariaDB.

  And last but not least, Metricbeat also has experimental support for the collection of Galera-specific metrics. Galera is a solution for multi-master clustering of MySQL servers. Our current implementation, the galera_status metricset, is heavily based on the status metricset but is not very well tested at the moment. Contributions and feedback are very welcome.

  MySQL 8 and beyond

  Before 6.7 we were only testing the MySQL modules with MySQL 5.7 and with some log cases of MySQL 5.5, since then we have increased the coverage to support also MySQL 8.0, and the same versions of Percona Server. We now include tests on MariaDB 10.2 and 10.3.

  Recent versions of MySQL and Percona include stronger authentication methods, so we have updated our MySQL clients to support them.

  In the future we have plans to continue updating our modules to follow the latest developments of all these products.

  If you want to get more visibility into your MySQL servers with the Elastic Stack also take a look at:

  • Elastic APM  - Allows you to instrument your applications to know, for example, how much time they spend querying your database servers
  • Packetbeat - Ingests MySQL information from network connections

  Getting started is easy, simply download an install Metricbeat and Filebeat, enable the appropriate module, and start sending your data to your local cluster, or to your 14-day free trial of Elasticsearch Service on Elastic Cloud. If you have any questions or ideas about how we can improve, reach out on the Discuss forums. Enjoy!

  
  
  https://www.elastic.co/blog/monitoring-mysql-percona-server-and-mariadb-with-the-elastic-stack
• 

  Scaling Elasticsearch on a hyper-converged infrastructure at IHG
  • sorangutan  

  1
  1
  Posts
  18
  Views

  

  This post is a recap of a community talk given at a recent Elastic{ON} Tour event. Interested in seeing more talks like this? Check out the conference archive or find out when the Elastic{ON} Tour is coming to a city near you.

  InterContinental Hotels Group, or IHG, is a multinational hospitality company that operates over 5,000 hotels in nearly 100 countries around the globe. IHG’s global technology organization, based in Atlanta, GA, has about 13,000 corporate employees, many of whom are using Elastic to do their jobs. With the success and continued adoption of IHG’s Elastic platform across the company, IHG has explored interesting ways of scaling its infrastructure, most notably the shift to hyper-converged. “We needed a cost-effective way to expand our ELK presence, which is what led us down the path to hyper-converged,” says IHG Systems Engineer Benjie Godfrey.

  Hyper-converged infrastructure is an IT framework that aims to increase scalability by combining compute, network, and storage functions in a single box. Converged infrastructure, in contrast, packages compute, network, and storage together as discrete elements within a validated, standardized platform. IHG was one of the first companies to run Elastic on a hyper-converged infrastructure.

  IHG first came to Elastic for log aggregation. Initially, IHG was running thousands of Linux virtual machines but wasn't collecting syslog data from those machines. To solve this problem, IHG turned to the Elastic Stack, using Filebeat and Winlogbeat to ship logs from application virtual machines and Windows into Logstash. The data was then fed into a messaging queue before being indexed in Elasticsearch. This made it easy to pull up logs in a single window in Kibana rather than dozens of terminal windows. IHG’s adoption of the Elastic Stack signaled a paradigm shift in how the company thought about data — now, there was an expectation that data could be collected in real time and translated into meaningful, actionable dashboards.

  IHG’s Elastic platform became popular among the company’s application developer teams, and the amount of data ingested each day increased rapidly. “With that growth came issues,” says Godfrey. In particular, it was clear to Godfrey’s team that IHG’s Elastic platform needed to be expanded, so they began to experiment with their infrastructure to find a solution. “We had some hardware lying around ... converged hardware,” recalls Godfrey. So the team utilized that existing hardware to put together a proof of concept on their corporate cloud. The proof of concept was successful, and they secured funds to make the switch to a hyper-converged infrastructure. They selected Nutanix as their hyper-converged vendor, and worked with Elastic Support to optimize their Elastic deployments on Nutanix nodes.
  

  Now, IHG’s Elastic platform, running on a hyper-converged infrastructure, is managed by a single part-time engineer and can ingest up to 4TB of data per day at both of their deployments, with a maximum index rate of 120,000 messages per second. For IHG, running the Elastic Stack on a hyper-converged infrastructure has been a huge success. “It’s performing. It’s rock solid,” says Godfrey. “It never ever goes down.”

  Watch Benjie Godfrey’s full Elastic{ON} Atlanta presentation to learn more about why hyper-converged worked for IHG and the lessons they learned along the way.

  

  
  
  https://www.elastic.coundefinedblog/scaling-elasticsearch-on-a-hyper-converged-infrastructure-at-ihg
• 

  Scaling Elasticsearch on a hyper-converged infrastructure at IHG
  • sorangutan  

  1
  1
  Posts
  12
  Views

  

  This post is a recap of a community talk given at a recent Elastic{ON} Tour event. Interested in seeing more talks like this? Check out the conference archive or find out when the Elastic{ON} Tour is coming to a city near you.

  InterContinental Hotels Group, or IHG, is a multinational hospitality company that operates over 5,000 hotels in nearly 100 countries around the globe. IHG’s global technology organization, based in Atlanta, GA, has about 13,000 corporate employees, many of whom are using Elastic to do their jobs. With the success and continued adoption of IHG’s Elastic platform across the company, IHG has explored interesting ways of scaling its infrastructure, most notably the shift to hyper-converged. “We needed a cost-effective way to expand our ELK presence, which is what led us down the path to hyper-converged,” says IHG Systems Engineer Benjie Godfrey.

  Hyper-converged infrastructure is an IT framework that aims to increase scalability by combining compute, network, and storage functions in a single box. Converged infrastructure, in contrast, packages compute, network, and storage together as discrete elements within a validated, standardized platform. IHG was one of the first companies to run Elastic on a hyper-converged infrastructure.

  IHG first came to Elastic for log aggregation. Initially, IHG was running thousands of Linux virtual machines but wasn't collecting syslog data from those machines. To solve this problem, IHG turned to the Elastic Stack, using Filebeat and Winlogbeat to ship logs from application virtual machines and Windows into Logstash. The data was then fed into a messaging queue before being indexed in Elasticsearch. This made it easy to pull up logs in a single window in Kibana rather than dozens of terminal windows. IHG’s adoption of the Elastic Stack signaled a paradigm shift in how the company thought about data — now, there was an expectation that data could be collected in real time and translated into meaningful, actionable dashboards.

  IHG’s Elastic platform became popular among the company’s application developer teams, and the amount of data ingested each day increased rapidly. “With that growth came issues,” says Godfrey. In particular, it was clear to Godfrey’s team that IHG’s Elastic platform needed to be expanded, so they began to experiment with their infrastructure to find a solution. “We had some hardware lying around ... converged hardware,” recalls Godfrey. So the team utilized that existing hardware to put together a proof of concept on their corporate cloud. The proof of concept was successful, and they secured funds to make the switch to a hyper-converged infrastructure. They selected Nutanix as their hyper-converged vendor, and worked with Elastic Support to optimize their Elastic deployments on Nutanix nodes.
  

  Now, IHG’s Elastic platform, running on a hyper-converged infrastructure, is managed by a single part-time engineer and can ingest up to 4TB of data per day at both of their deployments, with a maximum index rate of 120,000 messages per second. For IHG, running the Elastic Stack on a hyper-converged infrastructure has been a huge success. “It’s performing. It’s rock solid,” says Godfrey. “It never ever goes down.”

  Watch Benjie Godfrey’s full Elastic{ON} Atlanta presentation to learn more about why hyper-converged worked for IHG and the lessons they learned along the way.

  

  
  
  https://www.elastic.co/blog/scaling-elasticsearch-on-a-hyper-converged-infrastructure-at-ihg
• 

  This Week in Elasticsearch and Apache Lucene - 2019-04-21
  • sorangutan  

  1
  1
  Posts
  20
  Views

  

  Elasticsearch Highlights
  

  Rollups

  We have started working on saving pre-aggregated data structures directly in a document. This would allow clients, exporters, APM agents, etc to pre-aggregate data locally using agreed-upon algorithms (stuff like HyperLogLog++ for cardinality, HDRHistogram for percentiles) and then index the sketch rather than sending all the raw data. At query time the sketches are loaded and merged, instead of querying raw data.

  Geo

  We have been investigating ways to enhance existing field-types in mappings to include custom fields introduced from plugins. We're still chewing through the options, but it looks like we may be using multi-fields as a way to enhance the mapping from a plugin.

  Cluster privileges for API keys

  We have started working on fine grained cluster privileges for API keys. this enhancement is all about providing finer grained control on API key creation and validation.

  Admins can use a cluster privilege, manage_api_key, to control who should create API keys, but also should not invalidate those keys, and also allow users to restrict API key invalidation to the keys both owned by the user and having the privilege.

  We have discussed two options: one is to add a separate REST endpoint for invalidation of API keys or retrieval of API keys for authenticate user, the second option was more on the lines of Object level security. We have decided to go with adding a separate endpoint.

  Enrich documents at ingest time (#32789)

  An enrich policy defines how, what, and how often to synchronize fields from a source index (a normal index) to a .enrich index (a specialized index used for enrichments). The first iteration of the enrich policy has been merged (#41003).

  The policy runner reads the enrich policy and uses re-indexing and aliases to implement the synchronization of data. (#41088)

  Enrichment processors will be implemented as plugins. We are working on the changes necessary to expose the .enrich index to a plugin for optimal performance. The enrich processor(s) will leverage Lucene level interactions against data local single segment shards. (#41010)

  Snapshot and Restore UI

  We merged snapshots table and detail panel and also merged some nice UX improvements to repositories UI. We worked on adding client-side validation to repository form and added link to filtered snapshots from a repository's details.

  Networking

  We finished up the unification of our network settings in #36652. The settings have been completely removed from 8.0 and deprecated in 7.x with appropriate breaking changes documentation.

  We also opened a pull request to remove the dedicated TLS write buffer in our transport-nio TLS implementation in #41283. The fundamental problem here is that we allocate a single network write buffer for each channel. When we have data to send to the network, we fill the write buffer with the encrypted data and then flush. This requires constantly context switching back and forth between flushing and encrypting. It is the approach that Kafka takes for their TLS implementation, but is inferior to the approach that our existing Netty-provided implementation takes (encrypting all at once into multiple buffers and then flushing). All of the encrypted network buffers are allocated in this PR. The next steps after this PR is merged is to support buffer recycling and removing the single read buffer.

  We worked on #40866 to move the initial handling of bulk requests off the transport thread and onto a WRITE thread. The problem here was that bulk requests can be large enough that even parsing them on the transport thread is too expensive: it could take ~100ms to parse and dispatch the resulting tasks to each shard, which introduces significant and unwanted latency into network request handling. This was discovered during an investigation into other slow activities taking place on transport threads in #39658 which were leading to unstable tests.

  Apache Lucene

  Lucene master/9.0 requires Java 11

  Lucene 9.0 is not expected to go out in less than one year, which makes requiring Java 11 a conservative non-controversial choice. Upcoming 8.x releases will keep Java 8 as a minimum version requirement. We can now start using features of Java 9, 10 or 11 for Lucene 9, which we started doing by simplifying the initialization of some static maps using Java 10's Map#copyOf. Likewise, we could now enable Javadoc search support.

  Lucene 8.1

  We started a thread about releasing Lucene 8.1. Even though Lucene 8.0 was released on March 14th which is pretty recent, the feature freeze for 8.0 started on January 29th so we accumulated a lot of good changes:

  • much faster merging of KD trees,
  • Luke has become a Lucene module,
  • better defaults regarding when to load the terms dictionary off-heap,
  • flattening of disjunctions in disjunctions, which helps WAND do a better job at speeding up queries.

  The proposed timeline is to cut the branch on April 30th and cut the first release candidate a couple days later.

  Follow-ups to block-max WAND

  Even though we already knew that our implementation of block-max WAND still had room for improvement, the results of the release benchmarks of some tracks made us want to have a closer look. For instance if you look at the ones for NOAA, some queries like the conjunction of a term query and a range query that both match lots of documents became slower, in spite of now being able to skip counting hits. There is a different case on the Lucene benchmarks when running disjunctions within conjunctions. We spent some time collecting information about things that need to be improved:

  • Flattening of boolean queries, ie. rewtiting (A OR (B OR C)) as (A OR B OR C). We just implemented it for disjunctions which triggered speedups in the 10-80% range for the queries we tested. The same could be done for conjunctions.
  • Two-phase iteration support. The new block-max (W)AND scorers don't leverage two-phase iterators, so they are inefficient when applied to queries that produce two-phase iterators such as script queries or phrase queries.
  • More accurate estimation of minimum scores. Say your query is (A AND B), the minimum score for the query is 4 and the maximum score for A is 3. This means that B should produce scores that are greater than or equal to 1 right? Floating-point arithmetic makes it a bit more complicated than that : if you required that B produces scores that are greater than or equal to 1, you could miss hits. The actual minimum score that B must produce is 0.9999999 (single-precision float) because 0.9999999+3 is equal to 4 as well. We had done simplifications until now to avoid dealing with this issue, but it seems to be what is hurting the NOAA track, so we will have to dig.
  • Specialization of postings readers. Postings allow you to optionally retrieve documents, frequencies, positions, offsets and payloads. Being able to optimize for the case that only documents and frequencies are requested helps run queries faster. However this is something that we are only doing in the case that counts are needed today, maybe we should start looking into specializing the codec for top-k retrieval too. This is also related to a change that we have been working on, lazy loading of term frequencies.
  • Reconsider the encoding of postings. Our current implementation uses FOR (Frame-of-reference) based on experiments that we made years ago. Given that WAND doesn't stress postings the same way and that a lot of time has passed since last time we experimented, it might be worth exploring again alternatives.

  Luwak donation

  Luwak authors are donating Luwak to Lucene. Luwak allows to find matching queries given a document, similarly to Elasticsearch's percolator.

  Other

  We suggested that we have a query that can run a conjunction or disjunction of multiple ranges on the same field.

  There is discussion on how to deal with points that fall on the edge of a polygon and whether they should be considered inside of outside of the polygon.

  Changes

  Elasticsearch

  Changes in 8.0:

  • BREAKING: Reindex from Remote encoding #41007
  • Upgrade to lucene 8.1.0-snapshot-e460356abe #40952

  Changes in 7.1:

  • Fix unmapped field handling in the composite aggregation #41280
  • more_like_this query to throw an error if the like fields is not provided #40632
  • Handle Bulk Requests on Write Threadpool #40866
  • [Rollup] Validate timezones based on rules not string comparison #36237
  • Clean up Node#close. #39317
  • fix the packer cache script #41183
  • Fix range query edge cases #41160
  • Expand beats_system role privileges #40876
  • Better error messages when pipelines reference incompatible aggs #40068
  • [ML DataFrame] Data Frame stop all #41156
  • Clarify some ToXContent implementations behaviour #41000
  • Add deprecation check for ILM poll interval <1s #41096
  • Improve error message when polygons contains twice the same point in no-consecutive position #41051

  Changes in 7.0:

  • Mark searcher as accessed in acquireSearcher #41335
  • Fix issue with subproject test task dependencies #41321
  • Fix error applying ignore_malformed to boolean values #41261
  • ProfileScorer should propagate setMinCompetitiveScore. #40958
  • Validate cluster UUID when joining Zen1 cluster #41063
  • BlendedTermQuery should ignore fields that don't exist in the index #41125
  • Use environment settings instead of state settings for Watcher config #41087

  Changes in 6.7:

  • Unified highlighter should ignore terms that target the _id field #41275
  • SQL: Predicate diff takes into account all values #41346
  • Fix Broken Index Shard Snapshot File Preventing Snapshot Creation #41310
  • Always check for archiving broken index settings #41209
  • Do not create missing directories in readonly repo #41249
  • SQL: Allow current_date/time/timestamp to be also used as a function escape pattern #41254
  • put mapping authorization for alias with write-index and multiple read indices #40834
  • SQL: Fix LIMIT bug in agg sorting #41258
  • SQL: Translate MIN/MAX on keyword fields as FIRST/LAST #41247
  • SQL: Tweak pattern matching in SYS TABLES #41243
  • Unified highlighter should respect no_match_size with number_of_fragments set to 0 #41069
  • Use alias name from rollover request to query indices stats #40774
  • Full text queries should not always ignore unmapped fields #41062

  Changes in Elasticsearch Management UI

  Changes in 7.1:

  • adding spec to console utility as Kibana script #35232
  • adding data_frame autocomplete rules to dev console #35086

  Changes in Elasticsearch SQL ODBC Driver

  Changes in 7.1:

  • Introduce support for the TIME data type conversions #144
  • DST fixes unit test #145
  • Apply DST #142
  • Introduce auto-escaping of pattern arguments in catalog functions #143

  Changes in Rally

  Changes in 1.1.0:

  • Store mean for response-related metrics #683
  • Use single node discovery type if suitable #681

  Changes in Rally Tracks

  • Allow to override index settings in all cases #72
  • Update target throughput #73
  • Allow to override number of shards for updates #71

  
  
  https://www.elastic.coundefinedblog/this-week-in-elasticsearch-and-apache-lucene-2019-04-21
• 

  This Week in Elasticsearch and Apache Lucene - 2019-04-21
  • sorangutan  

  1
  1
  Posts
  38
  Views

  

  Elasticsearch Highlights
  

  Rollups

  We have started working on saving pre-aggregated data structures directly in a document. This would allow clients, exporters, APM agents, etc to pre-aggregate data locally using agreed-upon algorithms (stuff like HyperLogLog++ for cardinality, HDRHistogram for percentiles) and then index the sketch rather than sending all the raw data. At query time the sketches are loaded and merged, instead of querying raw data.

  Geo

  We have been investigating ways to enhance existing field-types in mappings to include custom fields introduced from plugins. We're still chewing through the options, but it looks like we may be using multi-fields as a way to enhance the mapping from a plugin.

  Cluster privileges for API keys

  We have started working on fine grained cluster privileges for API keys. this enhancement is all about providing finer grained control on API key creation and validation.

  Admins can use a cluster privilege, manage_api_key, to control who should create API keys, but also should not invalidate those keys, and also allow users to restrict API key invalidation to the keys both owned by the user and having the privilege.

  We have discussed two options: one is to add a separate REST endpoint for invalidation of API keys or retrieval of API keys for authenticate user, the second option was more on the lines of Object level security. We have decided to go with adding a separate endpoint.

  Enrich documents at ingest time (#32789)

  An enrich policy defines how, what, and how often to synchronize fields from a source index (a normal index) to a .enrich index (a specialized index used for enrichments). The first iteration of the enrich policy has been merged (#41003).

  The policy runner reads the enrich policy and uses re-indexing and aliases to implement the synchronization of data. (#41088)

  Enrichment processors will be implemented as plugins. We are working on the changes necessary to expose the .enrich index to a plugin for optimal performance. The enrich processor(s) will leverage Lucene level interactions against data local single segment shards. (#41010)

  Snapshot and Restore UI

  We merged snapshots table and detail panel and also merged some nice UX improvements to repositories UI. We worked on adding client-side validation to repository form and added link to filtered snapshots from a repository's details.

  Networking

  We finished up the unification of our network settings in #36652. The settings have been completely removed from 8.0 and deprecated in 7.x with appropriate breaking changes documentation.

  We also opened a pull request to remove the dedicated TLS write buffer in our transport-nio TLS implementation in #41283. The fundamental problem here is that we allocate a single network write buffer for each channel. When we have data to send to the network, we fill the write buffer with the encrypted data and then flush. This requires constantly context switching back and forth between flushing and encrypting. It is the approach that Kafka takes for their TLS implementation, but is inferior to the approach that our existing Netty-provided implementation takes (encrypting all at once into multiple buffers and then flushing). All of the encrypted network buffers are allocated in this PR. The next steps after this PR is merged is to support buffer recycling and removing the single read buffer.

  We worked on #40866 to move the initial handling of bulk requests off the transport thread and onto a WRITE thread. The problem here was that bulk requests can be large enough that even parsing them on the transport thread is too expensive: it could take ~100ms to parse and dispatch the resulting tasks to each shard, which introduces significant and unwanted latency into network request handling. This was discovered during an investigation into other slow activities taking place on transport threads in #39658 which were leading to unstable tests.

  Apache Lucene

  Lucene master/9.0 requires Java 11

  Lucene 9.0 is not expected to go out in less than one year, which makes requiring Java 11 a conservative non-controversial choice. Upcoming 8.x releases will keep Java 8 as a minimum version requirement. We can now start using features of Java 9, 10 or 11 for Lucene 9, which we started doing by simplifying the initialization of some static maps using Java 10's Map#copyOf. Likewise, we could now enable Javadoc search support.

  Lucene 8.1

  We started a thread about releasing Lucene 8.1. Even though Lucene 8.0 was released on March 14th which is pretty recent, the feature freeze for 8.0 started on January 29th so we accumulated a lot of good changes:

  • much faster merging of KD trees,
  • Luke has become a Lucene module,
  • better defaults regarding when to load the terms dictionary off-heap,
  • flattening of disjunctions in disjunctions, which helps WAND do a better job at speeding up queries.

  The proposed timeline is to cut the branch on April 30th and cut the first release candidate a couple days later.

  Follow-ups to block-max WAND

  Even though we already knew that our implementation of block-max WAND still had room for improvement, the results of the release benchmarks of some tracks made us want to have a closer look. For instance if you look at the ones for NOAA, some queries like the conjunction of a term query and a range query that both match lots of documents became slower, in spite of now being able to skip counting hits. There is a different case on the Lucene benchmarks when running disjunctions within conjunctions. We spent some time collecting information about things that need to be improved:

  • Flattening of boolean queries, ie. rewtiting (A OR (B OR C)) as (A OR B OR C). We just implemented it for disjunctions which triggered speedups in the 10-80% range for the queries we tested. The same could be done for conjunctions.
  • Two-phase iteration support. The new block-max (W)AND scorers don't leverage two-phase iterators, so they are inefficient when applied to queries that produce two-phase iterators such as script queries or phrase queries.
  • More accurate estimation of minimum scores. Say your query is (A AND B), the minimum score for the query is 4 and the maximum score for A is 3. This means that B should produce scores that are greater than or equal to 1 right? Floating-point arithmetic makes it a bit more complicated than that : if you required that B produces scores that are greater than or equal to 1, you could miss hits. The actual minimum score that B must produce is 0.9999999 (single-precision float) because 0.9999999+3 is equal to 4 as well. We had done simplifications until now to avoid dealing with this issue, but it seems to be what is hurting the NOAA track, so we will have to dig.
  • Specialization of postings readers. Postings allow you to optionally retrieve documents, frequencies, positions, offsets and payloads. Being able to optimize for the case that only documents and frequencies are requested helps run queries faster. However this is something that we are only doing in the case that counts are needed today, maybe we should start looking into specializing the codec for top-k retrieval too. This is also related to a change that we have been working on, lazy loading of term frequencies.
  • Reconsider the encoding of postings. Our current implementation uses FOR (Frame-of-reference) based on experiments that we made years ago. Given that WAND doesn't stress postings the same way and that a lot of time has passed since last time we experimented, it might be worth exploring again alternatives.

  Luwak donation

  Luwak authors are donating Luwak to Lucene. Luwak allows to find matching queries given a document, similarly to Elasticsearch's percolator.

  Other

  We suggested that we have a query that can run a conjunction or disjunction of multiple ranges on the same field.

  There is discussion on how to deal with points that fall on the edge of a polygon and whether they should be considered inside of outside of the polygon.

  Changes

  Elasticsearch

  Changes in 8.0:

  • BREAKING: Reindex from Remote encoding #41007
  • Upgrade to lucene 8.1.0-snapshot-e460356abe #40952

  Changes in 7.1:

  • Fix unmapped field handling in the composite aggregation #41280
  • more_like_this query to throw an error if the like fields is not provided #40632
  • Handle Bulk Requests on Write Threadpool #40866
  • [Rollup] Validate timezones based on rules not string comparison #36237
  • Clean up Node#close. #39317
  • fix the packer cache script #41183
  • Fix range query edge cases #41160
  • Expand beats_system role privileges #40876
  • Better error messages when pipelines reference incompatible aggs #40068
  • [ML DataFrame] Data Frame stop all #41156
  • Clarify some ToXContent implementations behaviour #41000
  • Add deprecation check for ILM poll interval <1s #41096
  • Improve error message when polygons contains twice the same point in no-consecutive position #41051

  Changes in 7.0:

  • Mark searcher as accessed in acquireSearcher #41335
  • Fix issue with subproject test task dependencies #41321
  • Fix error applying ignore_malformed to boolean values #41261
  • ProfileScorer should propagate setMinCompetitiveScore. #40958
  • Validate cluster UUID when joining Zen1 cluster #41063
  • BlendedTermQuery should ignore fields that don't exist in the index #41125
  • Use environment settings instead of state settings for Watcher config #41087

  Changes in 6.7:

  • Unified highlighter should ignore terms that target the _id field #41275
  • SQL: Predicate diff takes into account all values #41346
  • Fix Broken Index Shard Snapshot File Preventing Snapshot Creation #41310
  • Always check for archiving broken index settings #41209
  • Do not create missing directories in readonly repo #41249
  • SQL: Allow current_date/time/timestamp to be also used as a function escape pattern #41254
  • put mapping authorization for alias with write-index and multiple read indices #40834
  • SQL: Fix LIMIT bug in agg sorting #41258
  • SQL: Translate MIN/MAX on keyword fields as FIRST/LAST #41247
  • SQL: Tweak pattern matching in SYS TABLES #41243
  • Unified highlighter should respect no_match_size with number_of_fragments set to 0 #41069
  • Use alias name from rollover request to query indices stats #40774
  • Full text queries should not always ignore unmapped fields #41062

  Changes in Elasticsearch Management UI

  Changes in 7.1:

  • adding spec to console utility as Kibana script #35232
  • adding data_frame autocomplete rules to dev console #35086

  Changes in Elasticsearch SQL ODBC Driver

  Changes in 7.1:

  • Introduce support for the TIME data type conversions #144
  • DST fixes unit test #145
  • Apply DST #142
  • Introduce auto-escaping of pattern arguments in catalog functions #143

  Changes in Rally

  Changes in 1.1.0:

  • Store mean for response-related metrics #683
  • Use single node discovery type if suitable #681

  Changes in Rally Tracks

  • Allow to override index settings in all cases #72
  • Update target throughput #73
  • Allow to override number of shards for updates #71

  
  
  https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2019-04-21
• 

  Monitoring a NATS messaging system with Beats
  • sorangutan  

  1
  1
  Posts
  132
  Views

  

  In a world where stateless applications are optimized to run blazingly fast, message exchanging cannot be allowed to affect their performance. Having the ability to process millions of messages per second, NATS is the sprinter of messaging systems.

  While benchmarks are good indicators for choosing a tool, there is no way to confirm its value without monitoring its performance in production. As a team working together in the telecommunications sector, we're building NFV solutions on top of Kubernetes microservices, and we're using NATS as a messaging system to ensure the resiliency of the critical paths. These include the communication between the software components running on the clients’ data centers and our centralized platforms. It has allowed us to achieve high rates of message exchanging with great reliability. In this regard, we need to assure its health by providing actionable visibility in real-time. Given the fact that we use EFK to monitor our Kubernetes stack, monitoring of NATS had to be achieved by the exact same way.

  Everything started with a goal: Ship NATS monitoring data to Elasticsearch. 

  What we achieved: Extending Beats, by providing NATS modules for both Metricbeat and Filebeat shippers. Get on board to find out how!

  Motivated by an internal hackathon

  We wanted results and we wanted them fast, but our internal backlog was full of requirements forcing our time to be very limited. Luckily, our company was organizing its annual hackathon. Without hesitation, we grabbed the opportunity to work on creating our own NATS Beat by leveraging the Beats extension mechanism. After two days of effort our NATS Beat succeeded into becoming part of the official list of community Beats. And that was only the beginning. Intrigued by the idea, the Elastic Beats team encouraged us to put our code into the core Beats project. With their support, we managed to develop a Metricbeat module for metrics collection, and a Filebeat module, for log aggregation. Today, NATS monitoring is deeply integrated into the core of Beats upstream.

  How to monitor NATS

  NATS is very helpful when it comes to providing monitoring data. If requested, NATS server can serve such data in JSON format by exposing four different HTTP endpoints:

  • /varz: reports general statistics such as CPU utilization, memory consumption, etc.
  • /connz: reports detailed information on client connections
  • /routez: reports information on routes between servers of the NATS cluster
  • /subz: reports detailed information about current subscriptions and the routing data structure

  By querying these endpoints in a periodic way, user can understand a lot about the running state of his messaging queue. And so we did with the brand new NATS module of Metricbeat.

  But NATS can provide more. Another interesting place to find useful information regarding monitoring is the server’s logs. If requested, NATS server can provide TRACE level logs for every message received or transmitted. These log entries can be proven really valuable if handled properly. Our NATS module for Filebeat did exactly that. Every log is parsed and all its meaningful bits are squeezed out of it.

  NATS as a Metricbeat module

  As it happens with any Metricbeat module, NATS needs to be activated and configured accordingly in order to fetch the data from the monitoring endpoints. Upon configured properly, it fetches data from the endpoints, curates them accordingly and ships them to Elasticsearch.

  Configuration

  The module can be configured as simple as this:

  metricbeat.modules:
  - module: nats
    metricsets: ["connections", "routes", "stats", "subscriptions"]
    period: 10s
    hosts: ["localhost:8222"]
    stats.metrics_path: "/varz"
    connections.metrics_path: "/connz"
    routes.metrics_path: "/routez"
    subscriptions.metrics_path: "/subsz"
    enabled: true
  

  The above configuration provides the location of the NATS server(s) we want to monitor, the query interval for metrics, which metrics to fetch and the URIs under which they can be accessed.

  Let the metrics speak

  Fetching the data is one thing. Sending them to Elasticsearch is another. But what is the value of storing them if you cannot visualize them? With that in mind, this module ships with a pre-built dashboard which can be loaded automatically from Beats and give a good starting point to the user.

  Let’s take a sneak peak:

  

  Users can detect slow consumers which may affect NATS performance, monitor how NATS cache is performing, or even realize the utilization of resources made by our NATS servers. Additionally, users can correlate the stored data according to their needs and extend the dashboard. It is in their hands!

  NATS as a Filebeat module

  After making Metricbeat module a reality, a brilliant idea was put on the table: Is there something we could do for NATS server logs? The answer was standing in front of us… Filebeat!

  And so we did. Filebeat is an agent dedicated to fetching logs, parsing them with a wide range of text processors, and creating docs out of them in an Elasticsearch index. But to go far you need to start simple. Our very first version of NATS module was just extracting log timestamp, level, and message, and putting them into the datastore. The second version though is taking it to the next level. After investigating all log patterns we processed the log messages even more. The outcome is far more interesting, it can be visualized and it can offer a much deeper understanding of the NATS cluster’s state

  Configuration

  Filebeat module can be configured as the Metricbeat module by adding the following lines to the configuration file:

  filebeat.modules:
  - module: nats  
    log:
      enabled: true
      var.paths:
        - /var/log/nats/nats.log*
  

  As it can be easily observed, apart from enabling the module, the only configurable field a user has to set is a list of log path regular expressions. Given the location of the logs, NATS module has everything it needs to start scraping data from them.

  Logs as a Dashboard (LAAD)

  As the title implies, NATS module gives the opportunity to visualize the logs that the user's NATS cluster is producing. With the pre-built dashboard accompanying the module and the log trace option enabled to NATS server, user can come up with a dashboard as interesting as the following:

  

  Filebeat dashboard can provide various forms of data and statistics. Users can observe the distribution of messages by its type, the heat received by the NATS subjects, and the type of logs produced in a timeline. Furthermore, users have a good indicator on messages’ traffic, both incoming and outgoing, as well as the IP range distribution of the clients. The most interesting part: All of the above can happen with a click of a button!

  At the end of the day

  All in all, time invested on this effort was definitely worth it. Not only did we solve our problem of NATS monitoring, which could be achieved with only the community Beat, but we also gained a lot of experience on the way. We had the chance to work with excellent engineers from Elastic and we felt the joy of giving back to the community. So it goes without saying that we enjoyed that and we would suggest such kind of contributions to everyone who faces similar situations.

  Are you a NATS user?  Download Metricbeat 7.0 and build Filebeat master branch, enable the NATS modules, and start looking under the covers to see what you can discover about your NATS environment. 

  ---

  Stamatis Katsaounis is a software engineer, focused mainly on the Cloud. He is an active member of the OPNFV community with contributions to various OPNFV projects. He experiments with OpenStack and Kubernetes deployments and is a Linux enthusiast. He speaks C, Bash and Python. Lately, he is a frequent visitor of Go playground.

  Michael Katsoulis is a cloud software engineer, backed with working knowledge in telecommunications systems, currently focusing in SDN/NFV area. Leveraging Linux power and open source tools he always tries to combine work in internal projects with open source contributions.

  Christos Markou is a systems and software engineer that's passionate about distributed systems and cloud technologies. He has hands-on experience in web services, cloud computing and large system architecture, and is an open source enthusiast with various contributions in high class projects.

  
  
  https://www.elastic.co/blog/monitoring-nats-messaging-system-with-elastic-beats
• 

  Monitoring a NATS messaging system with Beats
  • sorangutan  

  1
  1
  Posts
  25
  Views

  

  In a world where stateless applications are optimized to run blazingly fast, message exchanging cannot be allowed to affect their performance. Having the ability to process millions of messages per second, NATS is the sprinter of messaging systems.

  While benchmarks are good indicators for choosing a tool, there is no way to confirm its value without monitoring its performance in production. As a team working together in the telecommunications sector, we're building NFV solutions on top of Kubernetes microservices, and we're using NATS as a messaging system to ensure the resiliency of the critical paths. These include the communication between the software components running on the clients’ data centers and our centralized platforms. It has allowed us to achieve high rates of message exchanging with great reliability. In this regard, we need to assure its health by providing actionable visibility in real-time. Given the fact that we use EFK to monitor our Kubernetes stack, monitoring of NATS had to be achieved by the exact same way.

  Everything started with a goal: Ship NATS monitoring data to Elasticsearch. 

  What we achieved: Extending Beats, by providing NATS modules for both Metricbeat and Filebeat shippers. Get on board to find out how!

  Motivated by an internal hackathon

  We wanted results and we wanted them fast, but our internal backlog was full of requirements forcing our time to be very limited. Luckily, our company was organizing its annual hackathon. Without hesitation, we grabbed the opportunity to work on creating our own NATS Beat by leveraging the Beats extension mechanism. After two days of effort our NATS Beat succeeded into becoming part of the official list of community Beats. And that was only the beginning. Intrigued by the idea, the Elastic Beats team encouraged us to put our code into the core Beats project. With their support, we managed to develop a Metricbeat module for metrics collection, and a Filebeat module, for log aggregation. Today, NATS monitoring is deeply integrated into the core of Beats upstream.

  How to monitor NATS

  NATS is very helpful when it comes to providing monitoring data. If requested, NATS server can serve such data in JSON format by exposing four different HTTP endpoints:

  • /varz: reports general statistics such as CPU utilization, memory consumption, etc.
  • /connz: reports detailed information on client connections
  • /routez: reports information on routes between servers of the NATS cluster
  • /subz: reports detailed information about current subscriptions and the routing data structure

  By querying these endpoints in a periodic way, user can understand a lot about the running state of his messaging queue. And so we did with the brand new NATS module of Metricbeat.

  But NATS can provide more. Another interesting place to find useful information regarding monitoring is the server’s logs. If requested, NATS server can provide TRACE level logs for every message received or transmitted. These log entries can be proven really valuable if handled properly. Our NATS module for Filebeat did exactly that. Every log is parsed and all its meaningful bits are squeezed out of it.

  NATS as a Metricbeat module

  As it happens with any Metricbeat module, NATS needs to be activated and configured accordingly in order to fetch the data from the monitoring endpoints. Upon configured properly, it fetches data from the endpoints, curates them accordingly and ships them to Elasticsearch.

  Configuration

  The module can be configured as simple as this:

  metricbeat.modules:
  - module: nats
    metricsets: ["connections", "routes", "stats", "subscriptions"]
    period: 10s
    hosts: ["localhost:8222"]
    stats.metrics_path: "/varz"
    connections.metrics_path: "/connz"
    routes.metrics_path: "/routez"
    subscriptions.metrics_path: "/subsz"
    enabled: true
  

  The above configuration provides the location of the NATS server(s) we want to monitor, the query interval for metrics, which metrics to fetch and the URIs under which they can be accessed.

  Let the metrics speak

  Fetching the data is one thing. Sending them to Elasticsearch is another. But what is the value of storing them if you cannot visualize them? With that in mind, this module ships with a pre-built dashboard which can be loaded automatically from Beats and give a good starting point to the user.

  Let’s take a sneak peak:

  

  Users can detect slow consumers which may affect NATS performance, monitor how NATS cache is performing, or even realize the utilization of resources made by our NATS servers. Additionally, users can correlate the stored data according to their needs and extend the dashboard. It is in their hands!

  NATS as a Filebeat module

  After making Metricbeat module a reality, a brilliant idea was put on the table: Is there something we could do for NATS server logs? The answer was standing in front of us… Filebeat!

  And so we did. Filebeat is an agent dedicated to fetching logs, parsing them with a wide range of text processors, and creating docs out of them in an Elasticsearch index. But to go far you need to start simple. Our very first version of NATS module was just extracting log timestamp, level, and message, and putting them into the datastore. The second version though is taking it to the next level. After investigating all log patterns we processed the log messages even more. The outcome is far more interesting, it can be visualized and it can offer a much deeper understanding of the NATS cluster’s state

  Configuration

  Filebeat module can be configured as the Metricbeat module by adding the following lines to the configuration file:

  filebeat.modules:
  - module: nats  
    log:
      enabled: true
      var.paths:
        - /var/log/nats/nats.log*
  

  As it can be easily observed, apart from enabling the module, the only configurable field a user has to set is a list of log path regular expressions. Given the location of the logs, NATS module has everything it needs to start scraping data from them.

  Logs as a Dashboard (LAAD)

  As the title implies, NATS module gives the opportunity to visualize the logs that the user's NATS cluster is producing. With the pre-built dashboard accompanying the module and the log trace option enabled to NATS server, user can come up with a dashboard as interesting as the following:

  

  Filebeat dashboard can provide various forms of data and statistics. Users can observe the distribution of messages by its type, the heat received by the NATS subjects, and the type of logs produced in a timeline. Furthermore, users have a good indicator on messages’ traffic, both incoming and outgoing, as well as the IP range distribution of the clients. The most interesting part: All of the above can happen with a click of a button!

  At the end of the day

  All in all, time invested on this effort was definitely worth it. Not only did we solve our problem of NATS monitoring, which could be achieved with only the community Beat, but we also gained a lot of experience on the way. We had the chance to work with excellent engineers from Elastic and we felt the joy of giving back to the community. So it goes without saying that we enjoyed that and we would suggest such kind of contributions to everyone who faces similar situations.

  Are you a NATS user?  Download Metricbeat 7.0 and build Filebeat master branch, enable the NATS modules, and start looking under the covers to see what you can discover about your NATS environment. 

  ---

  Stamatis Katsaounis is a software engineer, focused mainly on the Cloud. He is an active member of the OPNFV community with contributions to various OPNFV projects. He experiments with OpenStack and Kubernetes deployments and is a Linux enthusiast. He speaks C, Bash and Python. Lately, he is a frequent visitor of Go playground.

  Michael Katsoulis is a cloud software engineer, backed with working knowledge in telecommunications systems, currently focusing in SDN/NFV area. Leveraging Linux power and open source tools he always tries to combine work in internal projects with open source contributions.

  Christos Markou is a systems and software engineer that's passionate about distributed systems and cloud technologies. He has hands-on experience in web services, cloud computing and large system architecture, and is an open source enthusiast with various contributions in high class projects.

  
  
  https://www.elastic.coundefinedblog/monitoring-nats-messaging-system-with-elastic-beats
• 

  What's new in Lucene 8
  • sorangutan  

  1
  1
  Posts
  22
  Views

  

  Lucene 8 was released a few weeks ago with lots of exciting new features and improvements. Here's a look at some of the highlights:

  Query shortcuts

  When executing a search in Lucene 7, the scoring code will visit every document that matches the query, yielding both the top k highest scoring hits and an accurate count of the number of documents that matched. In many circumstances, an accurate count is unnecessary, and for queries that match a large number of documents, significant time is spent counting and scoring documents that will not end up in the top hits. Lucene 8 introduces a new API that allows you to opt out of this counting, returning instead a lower bound of the number of documents that match. This allows the introduction of a number of shortcuts, speeding up query execution.

  Indexing impacts

  The idea that kicked off all these query speedups was first proposed back in 2012, and involves adding new information to the index, making it possible to calculate maximum scores for blocks of documents.

  In general, the values that contribute to a document's score for any given query can be split into global factors (things like the total term frequency or average document length), and per-document per-term factors, known as impacts. These take the form of a pair of numbers, the length of the document (compressed down into a single byte, known as a ‘norm'), and the frequency of the term in that document.

  Lucene already divides indexing information for any given term into blocks, and builds a parallel structure called a skip list to allow queries to efficiently jump over documents that we know won't match a query. By adding a summary of the highest impacts in a block to that skip list, it's possible to calculate the largest score that could be produced by that block, and to skip over it entirely if the score is not competitive. Skip lists are much smaller and more efficient to decode than the postings lists they refer to, so the ability to avoid reading blocks altogether can yield enormous speedups for queries that touch a lot of documents. More details can be found in this blog post about faster retrieval of top hits.

  Faster custom scores

  The standard indexing chain stores term frequencies in the impacts list, but an impact is just a pair of numbers, and we can put any information we like in there. Lucene 8 provides a new field type called a FeatureField that uses term frequencies to encode numerical data, and exposes special queries that use this information for scoring. These queries can then implement the same skipping shortcuts as described above, resulting in very efficient custom-scoring queries. Elasticsearch makes these available via the rank_feature and rank_features fields, as described in this relevance tuning blog.

  As well as simple boosts, you can also score by recency or proximity using distance feature queries. These skip non-competitive hits in a slightly different way: we can convert a minimum competitive score into a bounding box that excludes documents which are too far from the origin to make it into the top k hits. Elasticsearch will provide these via the new distance feature query, due to be included in version 7.1.

  Jump tables for doc values

  Lucene provides a data structure called a docvalue that allows efficient per-document lookup, used for things like sorting or faceting. When this was first added to Lucene back in version 4.0, it was implemented as a straight look-up table with a fixed size for every entry, which allows for very fast access but has a large footprint on disk. It also makes it difficult to distinguish between documents that have an empty value and documents that have no value at all. To deal with this, Lucene 7 changed the doc values API to use iterators, which allows fields where few documents have values to be stored in much less space. The trade-off here, however, is that access to a particular document's value can be slower, as it's no longer a simple case of looking for the value at a particular address.

  To help improve this access speed, Lucene 8 adds jump tables to the default codec, similar to the skip lists described above. This makes looking up a value for a particular document much faster, as rather than decoding blocks of values in sequence until the correct document is reached, we can now jump directly to the correct block.

  Moving the terms dictionary off-heap

  Reading large index files can use a lot of memory. Lucene tries to delegate as much of this memory management to the operating system as it can by loading index data using memory mapping. Frequently, queried data will naturally stay longer in the page cache, while freeing up Java heap memory to be used by your actual application.

  In addition to postings lists and docvalues tables, Lucene 8 now allows the index of terms dictionaries to be loaded off-heap as well. The trade-off here is that in exchange for more heap memory, terms dictionary lookups are slightly slower. In most cases that won't matter, because looking up a term is typically a very small part of a query, but it can have a big effect on primary key lookups, which are used frequently when indexing updates. The default codec mitigates this by keeping these indexes on-heap if each entry only has a single document associated with it. It also only enables these changes if index data is being read via a memory-mapped directory.

  Looking ahead

  Work on Lucene doesn't stop just because a new version has been released. Plenty of improvements have already been committed for a future version 8.1, including more speed-ups for constant score and geo-queries and improvements to memory management. For more information about updates, or to report bugs, you can subscribe to the lucene-dev mailing list.

  
  
  https://www.elastic.co/blog/whats-new-in-lucene-8
• 

  What's new in Lucene 8
  • sorangutan  

  1
  1
  Posts
  20
  Views

  

  Apache Lucene 8 was released a few weeks ago with lots of exciting new features and improvements. Here's a look at some of the highlights:

  Query shortcuts

  When executing a search in Lucene 7, the scoring code will visit every document that matches the query, yielding both the top k highest scoring hits and an accurate count of the number of documents that matched. In many circumstances, an accurate count is unnecessary, and for queries that match a large number of documents, significant time is spent counting and scoring documents that will not end up in the top hits. Lucene 8 introduces a new API that allows you to opt out of this counting, returning instead a lower bound of the number of documents that match. This allows the introduction of a number of shortcuts, speeding up query execution.

  Indexing impacts

  The idea that kicked off all these query speedups was first proposed back in 2012, and involves adding new information to the index, making it possible to calculate maximum scores for blocks of documents.

  In general, the values that contribute to a document's score for any given query can be split into global factors (things like the total term frequency or average document length), and per-document per-term factors, known as impacts. These take the form of a pair of numbers, the length of the document (compressed down into a single byte, known as a ‘norm'), and the frequency of the term in that document.

  Lucene already divides indexing information for any given term into blocks, and builds a parallel structure called a skip list to allow queries to efficiently jump over documents that we know won't match a query. By adding a summary of the highest impacts in a block to that skip list, it's possible to calculate the largest score that could be produced by that block, and to skip over it entirely if the score is not competitive. Skip lists are much smaller and more efficient to decode than the postings lists they refer to, so the ability to avoid reading blocks altogether can yield enormous speedups for queries that touch a lot of documents. More details can be found in this blog post about faster retrieval of top hits.

  Faster custom scores

  The standard indexing chain stores term frequencies in the impacts list, but an impact is just a pair of numbers, and we can put any information we like in there. Lucene 8 provides a new field type called a FeatureField that uses term frequencies to encode numerical data, and exposes special queries that use this information for scoring. These queries can then implement the same skipping shortcuts as described above, resulting in very efficient custom-scoring queries. Elasticsearch makes these available via the rank_feature and rank_features fields, as described in this relevance tuning blog.

  As well as simple boosts, you can also score by recency or proximity using distance feature queries. These skip non-competitive hits in a slightly different way: we can convert a minimum competitive score into a bounding box that excludes documents which are too far from the origin to make it into the top k hits. Elasticsearch will provide these via the new distance feature query, due to be included in version 7.1.

  Jump tables for doc values

  Lucene provides a data structure called a docvalue that allows efficient per-document lookup, used for things like sorting or faceting. When this was first added to Lucene back in version 4.0, it was implemented as a straight look-up table with a fixed size for every entry, which allows for very fast access but has a large footprint on disk. It also makes it difficult to distinguish between documents that have an empty value and documents that have no value at all. To deal with this, Lucene 7 changed the doc values API to use iterators, which allows fields where few documents have values to be stored in much less space. The trade-off here, however, is that access to a particular document's value can be slower, as it's no longer a simple case of looking for the value at a particular address.

  To help improve this access speed, Lucene 8 adds jump tables to the default codec, similar to the skip lists described above. This makes looking up a value for a particular document much faster, as rather than decoding blocks of values in sequence until the correct document is reached, we can now jump directly to the correct block.

  Moving the terms dictionary off-heap

  Reading large index files can use a lot of memory. Lucene tries to delegate as much of this memory management to the operating system as it can by loading index data using memory mapping. Frequently, queried data will naturally stay longer in the page cache, while freeing up Java heap memory to be used by your actual application.

  In addition to postings lists and docvalues tables, Lucene 8 now allows the index of terms dictionaries to be loaded off-heap as well. The trade-off here is that in exchange for more heap memory, terms dictionary lookups are slightly slower. In most cases that won't matter, because looking up a term is typically a very small part of a query, but it can have a big effect on primary key lookups, which are used frequently when indexing updates. The default codec mitigates this by keeping these indexes on-heap if each entry only has a single document associated with it. It also only enables these changes if index data is being read via a memory-mapped directory.

  Looking ahead

  Work on Lucene doesn't stop just because a new version has been released. Plenty of improvements have already been committed for a future version 8.1, including more speed-ups for constant score and geo-queries and improvements to memory management. For more information about updates, or to report bugs, you can subscribe to the lucene-dev mailing list.

  
  
  https://www.elastic.coundefinedblog/whats-new-in-lucene-8
• 

  Improving speed and reliability with the Elastic Stack and Elastic Cloud Enterprise
  • sorangutan  

  1
  1
  Posts
  15
  Views

  

  CSG International is a multinational corporation headquartered in Greenwood Village, Colorado. Although historically a cable and satellite billing company, in recent years CSG has sought to expand into the digital services realm. In 2015, CSG launched Ascendon, a cloud-based platform that helps companies offer digital services — from live-streamed sporting events and concerts to seamless shopping experiences — to consumers.

  At Elastic{ON} Tour Denver in October 2018, CSG Executive Director of Operations Lori Kevin highlighted how moving first to the Elastic Stack and then to Elastic Cloud Enterprise (ECE) improved scalability, reliability, speed, and flexibility for Ascendon.

  Initially, Ascendon’s multitenant platform was based on a private data center with production nodes in North America and Europe, where different business units (i.e., customers) resided together. All the data they collected was stored in relational SQL databases. But as Ascendon’s user base expanded and data volumes grew, problems with the system emerged. Querying the millions of data points in the SQL databases was painfully slow, forcing the Ascendon Operations team to spend hours optimizing queries. Without a single dashboard showing key metrics side by side, it was also difficult for the Operations team to understand target performance thresholds. And issues such as downtime during upgrades and inflexible reporting formats were causing customer complaints due to poor end-user experience. The Operations team realized that many of these problems associated with monitoring, maintaining, securing, and reporting on all of their customers’ data were, fundamentally, search problems. This led them to the Elastic Stack.

  On the monitoring side, pulling their data out of the SQL databases and indexing it in Elasticsearch instantly improved the Ascendon Operations team’s ability to monitor activity across the platform. With Elastic, they were able to review API response times, identify usage trends, and analyze poor performance for greater insight into solutions. They began using Metricbeat to collect CPU, memory, and other health data, which they then visualized in Kibana dashboards. Visualizing this information not only allowed them to assess specific stats, but it provided a unified perspective, ensuring everyone involved — from operations folks to developers to QA testers to service team members — was looking at the same information. These dashboards proved so valuable that they now function as organization-wide resources for solving problems on the platform.

  “[I]f a ticket gets entered by a member of our services team, the first question that is asked by a member of operations is, ‘Where’s the Kibana link that shows me the errors that you received at the time?’ So it’s really been a game-changer for us on the monitoring side.” Lori Kevin, Executive Director of Operations | CSG

  Ascendon also used Elastic to streamline the maintenance of their data platform to ensure it was always available and optimized for the best customer experience possible. To keep their customers happy, Ascendon needs APIs that never die — what they affectionately refer to as “zombie architecture.” Previously, mandatory quarterly upgrades to the Ascendon platform required hours of downtime, as transactional data was stored in databases and system activity needed to be limited while schema changes were instituted behind the scenes. By moving most of the schema requirements out of SQL and into Elastic, the Operations team was able to reduce downtime to just minutes — short enough that many users wouldn’t even notice an interruption. In the future, the Operations team hopes to leverage Elastic to do away with the need for downtime during upgrades altogether.

  Improved security was another benefit of moving to Elastic. The Ascendon Operations team is constantly working to ensure that threats or potential threats to the platform are quickly identified and eliminated. Before adopting the Elastic Stack, the Operations team mostly relied on reactive and manual processes for identifying and blocking threats. With Elastic, they were able to make this process more proactive and automated. They scored an early win by using the alerting features of the Elastic Stack to set up automated alerting on specific types of events. However, because each of Ascendon’s customers has unique traffic patterns, a one-size-fits-all approach wasn’t ideal. That’s where Elastic machine learning features came in. Machine learning enhanced the Operations team’s ability to quickly detect different types of fraudulent activity and take action to block it.

  Ascendon was able to expand their reporting capabilities with Elastic, too. This was critical for the company, as their customers need to report on and analyze their business performance data. Reporting using the previous relational database solution was time-consuming and inflexible. This all changed with the Elastic Stack. By removing relevant data from the relational databases and putting it into Elastic, Ascendon Operations has been able to quickly retrieve and visualize relevant information. “It’s been a huge win for us, and I can’t say enough about [Kibana] because it’s been something that’s really been influential in how we’re continuing to evolve our own solutions within our reporting tools,” Kevin said.

  By 2018, Ascendon’s infrastructure had expanded. Their Elastic deployments were a huge success, but Kevin’s team had an eye for improving efficiency even further — how could they more quickly and efficiently provision and configure their Elasticsearch clusters? They found a perfect solution in Elastic Cloud Enterprise, which has freed up their teams to focus on evolving their platform.

  To learn more about why Ascendon chose ECE and how easy it was to implement, watch Lori Kevin’s full Elastic{ON} Tour Denver talk.

  

  
  
  https://www.elastic.co/blog/elastic-stack-elastic-cloud-enterprise-for-improved-speed-reliability-csg