Elastic News

• 

  This Week in Elasticsearch and Apache Lucene - 2019-05-24
  • sorangutan  

  1
  1
  Posts
  105
  Views

  

  Elasticsearch

  Highlights

  Deprecation info API

  As part of an effort to keep the deprecation info API up to date as we go (as opposed to a last minute push),we opened an issue to track breaking changes in 8.0 that need additions to this API (#42404).

  Enrich

  We have merged the PR that adds a force merge step to the enrich policy execution (#41969).

  Work has also started on integrating security with the enrich feature as is described here.

  We've started looking into how processor/pipeline can indicate that it needs to run on a different node. This is needed, because in the case when a write request with a pipeline is sent to a node that doesn't have an enrich index shard or that shard isn't ready yet. This should work similarly to how we forward write requests with a pipeline from a non ingest node to an ingest node.

  Watcher UI

  Work has continued on making improvements to Watcher. Recently this focused on the watch list view) and threshold alert.

  

  Snapshot Repositories UI

  We have instrumented Snapshot Repositories with UI metrics, and also worked on adding logic to warn users and disable delete for a cloud-managed repository.

  

  Bye Bye Transport Client

  From 7.0 the high level REST Client has feature parity with the transport client, so we have started on the daunting task to remove the transport client. This will also move us closer to the removal of the Guice library. As part of this, the migrate tool was deprecated in 7.x and removed in master. This tool was originally created to allow early users of security to move from the file realm to the native realm when it was introduced. At this point, the tool does not add much value and its removal simplified removing the transport client from x-pack. We have started by removing the TestXPackTransportClient and then moved on to removing transport client usage in x-pack.

  Performance

  We merged Rally#688 resolving a major usability issue (Rally#478) where wrong user specified track parameters, e.g. due to a typo, don't show a warning and may end up running a wrong benchmark. We took the extra mile and display a list of the wrong parameters, the entire list of user configurable parameters as well as possible correct alternatives using Levenshtein style approximate string matching.

  Types Removal

  Work has begun on types removal for 8.0. We have started to remove typed URL paths in the REST layer and have also done some cleanup to completely remove internal references to types in the search APIs.

  Multi-fields

  We have deprecated+removed support for chained multi-fields. As of 8.0 it won't be possible to define a multi-field within a multi-field. This closes a long standing issue that was opened to limit multi-fields to relevant settings only.

  Search low-level cancellation

  A PR was merged that activates low-level cancellation for search by default. The change is minimal but it required a lot of benchmarking in order to ensure that it does not impact search performance. Low-level cancellation is required to improve the responsiveness of search cancellation for long running queries and will benefit to systems (Kibana, …) that cannot change the default settings of an Elasticsearch cluster.

  Removal of node.maxlocalstorage_nodes setting

  We have deprecated and removed the node.max_local_storage_nodes setting. This setting, which prior to Elasticsearch 5 was enabled by default and caused all kinds of confusion, has since been disabled by default and is not recommended for production use. The prefered way going forward is for users to explicitly specify separate data folders for each started node to ensure that each node is consistently assigned to the same data path.

  The main step in getting rid of this setting was to move the test infrastructure away. This also changes the behavior of ESIntegTestCases so that starting up nodes will not automatically reuse data folders of previously stopped nodes.

  Finally, this also opens up the possibility to remove the "nodes/$nodeOrdinal" folder prefix from the data path in 8.0.

  Geo

  We've merged the initial implementation of the major serialized tree structure for storing polygons as doc-values (#40810 and #42331).

  We revisited the tessellation logic after a user reported a failure in a valid polygon in the discuss forum. This new issue showed clearly that the tessellator logic was failing when a hole had a shared vertex with the polygon.  We decided to change the logic when eliminating holes to detect if a hole has a shared vertices and use that vertex as the hole bridge. In order to do that efficiently the algorithm uses the bounding box of the hole to filter out potential candidates. This changed allowed us to remove most of the complex logic we had previously added.

  Lucene

  Lucene 8.1.1

  A 8.1.1 release has been called in order to fix a NPE when a Solr collection is called via an alias. This release has no changes on the Lucene side however.

  How to enforce a maximum clause count?

  Currently Lucene enforces that boolean queries have at most 1024 clauses. Some users have been working around this limit by using multiple levels of boolean queries so that they can have more than 1024 clauses in total but each boolean query has no more than 1024 clauses so Lucene doesn't complain about the clause count. However we recently introduced a new rewrite rule that flattens disjunctions within disjunctions, effectively defeating this workaround, which is surprising some of our users. We are considering deactivating this rewrite rule when it would hit the clause count limit, but this also triggered a separate discussion about whether we could count clauses across the entire query tree as opposed to on a per-BooleanQuery basis.

  Other

  • There is now a way to sort based on the value of a FeatureField. This means that in the future Elasticsearch users will be able to sort on a rank_feature field without having to add a float field in a multi-field as well.
  • There is a lot of work ongoing around getting a working Gradle build.
  • A bug was identified around concurrent rollbacks and reopens causing leakage of segment readers.
  • Javadocs of our analysis components are getting improved.
  • Range fields are getting doc-value support. Elasticsearch has its own, slightly different implementation of doc-value support for range fields due to the fact that it needs to support multiple ranges per document, which the Lucene implementation doesn't support.
  • StoredFieldsVisitor#stringField will now take a String instead of the UTF8 representation of the string as a byte[].
  • The conjunction scorer that leverages maximum per-clause scores was enhanced so that it works better when sub-clauses have two-phase iterators, such as phrase queries.
  • There are ongoing discussions regarding the introduction of a Collector-based rescorer.
  • Can we make codecs more efficiently handle boolean doc values?

  Changes

  Changes in Elasticsearch

  Changes in 8.0:

  • Remove transport client from xpack #42202
  • BREAKING: Remove deprecated search.remote settings #42381
  • BREAKING: Remove node.maxlocalstorage_nodes #42428
  • Remove deprecated Repository methods #42359
  • Fix TopHitsAggregationBuilder adding duplicate _score sort clauses #42179
  • add 7.1.1 and 6.8.1 versions #42253
  • BREAKING: Remove the migrate tool #42174
  • Add ChaCha20 TLS ciphers on Java 12+ #42155

  Changes in 7.3:

  • Bug fix to allow access to top level params in reduce script #42096
  • Avoid HashMap construction on Grok non-match #42444
  • Improve Close Index Response #39687
  • Allow fields to be set to * #42301
  • Bulk processor concurrent requests #41451
  • Rename SearchRequest#crossClusterSearch #42363
  • Fix alpha build error message when generate version object from version string #40406
  • Search - enable lowlevelcancellation by default. #42291
  • Deprecate support for chained multi-fields. #41926

  Changes in 7.2:

  • Fix settings prefix for realm truststore password #42336
  • Implement XContentParser.genericMap and XContentParser.genericMapOrdered methods #42059
  • Upgrade to Lucene 8.1.0 #42214
  • Add .code_internal-* index pattern to kibana user #42247
  • Remove IndexShard dependency from Repository #42213
  • Merge claims from userinfo and ID Token correctly #42277
  • Peer recovery should flush at the end #41660
  • Use translog to estimate number of operations in recovery #42211
  • [ML Data Frame] Persist and restore checkpoint and position #41942
  • Add experimental and warnings to vector functions #42205
  • Prevent in-place downgrades and invalid upgrades #41731
  • Validate non-secure settings are not in keystore #42209
  • Update to joda time 2.10.2 #42199
  • Fix FiltersAggregation NPE when filters is empty #41459
  • Update ciphers for TLSv1.3 and JDK11 if available #42082
  • Hash token values for storage #41792
  • Do not refresh realm cache unless required #42169
  • Protect logged exec spooling from no output #42177
  • Use local outputstream reference #42180
  • Deprecate the native realm migration tool #42142
  • Hide bwc build output on success #42102
  • Cacheability improvements for thirdparty audit task #42085
  • SQL: Add initial geo support #42031
  • Don't create tempdir for cli scripts #41913
  • Cleanup plugin bin directories #41907
  • Prevent order being lost for _nodes API filters #42045
  • [ML] Complete the Data Frame task on stop #41752

  Changes in 7.1:

  • Move the FIPS configuration back to the build plugin #41989
  • 7.1.0 release notes initial commit #42208
  • Implement factory methods for ValidationException #41993

  Changes in 7.0:

  • Remove deprecated sourceexclude and sourceinclude from get API spec #42188
  • Build local year inside DateFormat lambda #42120

  Changes in 6.8:

  • Avoid bubbling up failures from a shard that is recovering #42287
  • Recovery with syncId should verify seqno infos #41265
  • Execute actions under permit in primary mode only #42241
  • Avoid unnecessary persistence of retention leases #42299
  • 6.8 release notes #41802
  • Fix max boundary for rollup jobs that use a delay #42158
  • SQL: Fix issue regarding INTERVAL * number #42014
  • Enforce transport TLS on Basic with Security #42150

  Changes in Elasticsearch Hadoop Plugin

  Changes in 7.3:

  • [DOCS] Updates version attribute file #1296

  Changes in 7.1:

  • Fix the mapping client call on ES 5.x and earlier. #1284

  Changes in 6.8:

  • Fix incorrect scroll termination for ES 2.x #1287

  Changes in Elasticsearch Management UI

  Changes in 7.2:

  • Snapshot Repositories UI #34407

  Changes in Elasticsearch SQL ODBC Driver

  Changes in 7.2:

  • Fix: add pyodbc as instance member to testing class #156
  • DSN parameter: support for "indexincludefrozen" #151
  • Introduce geo support #154

  Changes in Rally

  Changes in 1.2.0:

  • Assume UTC timezone if not specified #693
  • Measure execution time of bulk ingest pipeline #694
  • Fail Rally early if there are unused variables in track-params #688

  
  
  https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2019-05-24
• 

  CreatorIQ: Powering the search for social media influencers with Elasticsearch
  • sorangutan  

  1
  1
  Posts
  77
  Views

  

  This post is a recap of a community talk given at a recent Elastic{ON} Tour event. Interested in seeing more talks like this? Check out the conference archive or find out when the Elastic{ON} Tour is coming to a city near you.

  CreatorIQ’s SaaS marketing platform enables users to sift through millions of social media influencers to find a handful of endorsers that can best drive their brand forward. Say a company wants to broaden their social media presence on a particular marketing campaign. Maybe they’re looking for a Midwestern mom turned blogger and baby product endorser. Or a twenty-something musician with an affinity for adult beverages (good luck narrowing that down). This type of brand advocacy has seen huge growth, but how does CreatorIQ help companies pluck out these influencers? Ones that align with desired target demos and that can promote products and services via social media in a way that traditional ads just can’t? Easy — with highly relevant search.

  Built on top of the Elasticsearch Service on Elastic Cloud, the CreatorIQ Discovery app allows users to pore over millions of popular influencer social media accounts — athletes, actors, comedians, gamers, models, vloggers, and so on — representing over 25 million different social media accounts and billions of data points (posts, audience, demographics, and many others) to find the perfect influencer accounts for the user’s needs. “The [influencer marketing] industry was exploding… and the user story starts with search — finding these influencers,” as CreatorIQ Product Manager Henry Pray put it.

  CreatorIQ not only needed fast, relevant search, but also a robustness to it that went well beyond basic searches of vanity metrics such as followers and likes. But like a lot of lean tech startups, CreatorIQ’s resources were tight. The early versions of Discovery were built on top of a SQL database, which was easy to deploy and manage, but full-text search across a lot of data sources was too slow. They knew they needed faster search, and they needed that system to stay fast as they grew. “We [didn’t want to] reinvent the wheel on search — we’re not search experts. We needed a bigger and better tool to take care of our search functionality, and naturally Elasticsearch was an immediate go-to for us,” said Pray.

  But with just a few engineers on staff, a very limited budget, and an old version of Elasticsearch (2.x), CreatorIQ was spending time they didn’t have managing and optimizing their deployment. They knew they needed a better solution, one that enabled them to concentrate on what they did best. “We wanted to move away from managing [search] ourselves, and [instead] focus on what we do well: analyzing social media and building a workflow management tool,” said Pray. Because CreatorIQ was already running on AWS, their initial hunch was to switch over to AWS Elasticsearch Service. But they realized rather quickly that the Elasticsearch Service on Elastic Cloud was the superior solution for their needs.

  “These are the guys that built Elasticsearch. This is their bread and butter. This is what they do,” said Pray. “They offer all this support… plus machine learning and security — all these things that we didn’t want to have to deal with developing and Amazon doesn’t offer.” Henry Pray, Product Manager | CreatorIQ

  The support that comes from Elastic ended up being the game changer. Dealing with the ever-changing APIs of the social media giants — and the accompanying reindexing challenges — was magnifying CreatorIQ’s technical difficulties and pulling valuable time and resources away from product development. On top of that, CreatorIQ didn’t have a proper staging environment, which left them vulnerable to downtime as they implemented changes. When a cluster failed the day before a big client demo, they immediately called Elastic support for help. Even though CreatorIQ only had access to Elastic Cloud’s standard support offerings, the Elastic Support team jumped in right away to help solve their problem (and later helped them set up a staging environment for appropriate testing). CreatorIQ upgraded to a Gold subscription that day, which gave them guaranteed resources and response times from Elastic, and helped ensure that CreatorIQ could meet service level agreements with their customers going forward. “We should have [gotten a subscription] in the first place, before launch… it’s been really huge for us,” said Pray.

  Watch the full CreatorIQ Elastic{ON} Anaheim presentation to find out more about how CreatorIQ’s Discovery app has evolved, and their plans to try out the Elastic APM and App Search products.  

  

  Are you considering hosted Elasticsearch Service on AWS? Not all offerings are created equal. Find out more about why the fully loaded one from us, Elastic, is right for you. See AWS Elasticsearch comparison: Amazon and Elastic for the complete tale of the tape.
  

  
  
  https://www.elastic.co/blog/creatoriq-powering-the-search-for-social-media-influencers-with-elasticsearch
• 

  Tencent Cloud Elasticsearch Service: A new partnership in China and beyond
  • sorangutan  

  1
  1
  Posts
  68
  Views

  

  Hello from Kunming, China!

  We are pleased to announce a global partnership with Tencent Cloud that brings the best of our Elasticsearch Service to the Tencent Cloud Elasticsearch Service. This partnership will benefit our users and customers in China and across the globe by providing them access to the full breadth of our products and solutions, including free and paid proprietary Elastic Stack features.

  We’re humbled by the opportunity to work with Tencent, one of the world’s leading internet technology companies with products like WeChat, QQ, and Tencent Games. This new relationship exhibits the best attributes of partnerships Elastic enthusiastically pursues. We want to give our users the best options to be successful with our products and solutions. With a rapidly growing Elasticsearch community in China, this partnership ensures that every user and customer can leverage every aspect of our engineering efforts across the entire Elastic Stack, including our free and paid proprietary features, and our solutions on the Tencent Cloud. We’re also excited to collaborate with Tencent Cloud’s engineering team and provide localized support. Such efforts are only made available to partners like Tencent Cloud.

  Available by early June, the new Tencent Cloud Elasticsearch Service service offering will give users access to the full breadth of Elastic features they’ve been asking for. This includes critical free security features like TLS for encrypted communications, role-based access control, file and native authentication, multi-tenancy for Kibana, and the ability to turn on paid security features like single sign-on, active directory/LDAP authentication, and field- and document-level security. In addition, Tencent Cloud will be able to leverage Elastic’s Chinese localization for Kibana and other proprietary features like alerting, monitoring, Canvas, Elasticsearch SQL, Index Lifecycle Management (ILM), Elastic Maps, and more.

  

  Importantly, the Tencent Elasticsearch Cloud Service will be consistent with the Elasticsearch Service available on Elastic's own Elastic Cloud offering — as well as those on other platforms that partner with Elastic — giving users the freedom to work seamlessly across Cloud services while maintaining access to the full functionality of the Elastic Stack and Elastic’s growing set of solutions such as application search, site search, enterprise search, logging, APM, metrics, security, business analytics, and more.

  The partnership is another step forward for Elastic to align with global cloud providers like Tencent Cloud to offer a first-class Elasticsearch Service experience for our users and customers. This also furthers our commitment to China, as we’ve always believed in building a strong local foundation to support our users, customers, and partners. Today, our presence in China has grown to include a strong partner ecosystem and Elastic employees across engineering, support, community, sales, business development, and marketing. We look forward to leveraging our growing China team to make this partnership successful and to help Tencent Cloud build a wonderful Elasticsearch Service.

  
  
  https://www.elastic.co/blog/tencent-cloud-elasticsearch-service-a-new-partnership-in-china-and-beyond
• 

  Security for Elasticsearch is now free
  • sorangutan  

  1
  1
  Posts
  73
  Views

  

  We are thrilled to announce that the core security features of the Elastic Stack are now free. This means that users can now encrypt network traffic, create and manage users, define roles that protect index and cluster level access, and fully secure Kibana with Spaces. This is an exciting next step for our community. We opened the code of these features (and many more) last year and by making them free today, everyone can now run a fully secure cluster, hassle free.
  

  Security is free, starting in versions 6.8.0 and 7.1.0

  For a change this important, we wanted to make sure that it was available to as many people as possible, so today we are releasing versions 6.8.0 and 7.1.0 of the Elastic Stack. These versions do not contain new features; they simply make the following core security features free in the default distribution of the Elastic Stack:

  • TLS for encrypted communications
  • File and native realm for creating and managing users
  • Role-based access control for controlling user access to cluster APIs and indexes; also allows multi-tenancy for Kibana with security for Kibana Spaces

  Previously, these core security features required a paid Gold subscription. Now they are free as a part of the Basic tier. Note that our advanced security features — from single sign-on and Active Directory/LDAP authentication to field- and document-level security — remain paid features. See the full feature matrix for details.

  As always, these releases are available immediately on Elasticsearch Service on Elastic Cloud, the official hosted Elasticsearch.

  But wait, there’s more … heya Kubernetes

  We are announcing this change in conjunction with the announcement and alpha release of Elastic Cloud on Kubernetes (ECK), the official Kubernetes Operator for Elasticsearch and Kibana. ECK is designed to automate and simplify how Elasticsearch is deployed and operated in Kubernetes.

  Security is an integral part of cluster operations, especially in shared and multi-tenant environments like Kubernetes. By moving the core security features into the default distribution of Elastic Stack, we can ensure that all clusters launched and managed by ECK are secured by default at creation time, without any additional burden on the admins. This experience also lines up well with the secure-by-default experience that users have always had on our Elasticsearch Service on Elastic Cloud.

  Upgrade and get started

  To get started, download and install the latest version of the Elastic Stack, or upgrade your clusters to either 6.8 or 7.1.

  To simplify your getting started journey, we have some great material to get you started: 

  • A blog about getting started with Elasticsearch security
  • A video walkthrough of getting started with Elasticsearch security

  Or, accelerate your learning and go deeper into Elasticsearch Security with this new on-demand Fundamentals of Securing Elasticsearch course, which is available for free for a limited time only (regularly $200).

  
  
  https://www.elastic.co/blog/security-for-elasticsearch-is-now-free
• 

  Getting started with Elasticsearch security
  • sorangutan  

  1
  1
  Posts
  256
  Views

  

  Starting with Elastic Stack 6.8 and 7.1, security features like TLS encrypted communication, role-based access control (RBAC), and more are available for free within the default distribution. In this blog post, we’re going to cover how to get started with using these features to secure your Elasticsearch clusters.
  

  As a practical example of securing an Elastic Stack implementation, we’re going to create and secure a 2-node Elasticsearch cluster on a local machine. To do so, we’ll first configure TLS communication between the two nodes. Then we'll enable security for the Kibana instance being used to analyze and visualize the data in the cluster. After that, we’ll configure role-based access control within Kibana to make sure that users can only see data they’re authorized to see.

  While there’s a lot more to how security works, for now, we’re just going to cover what you need to get started. After reading this blog, we’d recommend checking out our documentation on securing the Elastic Stack, which includes helpful practical examples. As a quick note, we’ll be doing this all on a local cluster because, while these features became available by default in 6.8 and 7.1, they have always come standard with our Elasticsearch Service.

  Install Elasticsearch and Kibana

  For this example, I’ll be installing on a laptop running Linux. Adjust the steps listed below to your specific environment.

  Step 1: Download Elasticsearch and Kibana

  The first thing you have to do is download the default distribution of Elasticsearch and Kibana version 6.8+ or 7.1+. Security was added to the default distribution in 7.1 and 6.8, so if you’re on an older version, you’ll need to upgrade.

  Step 2: Extract Elasticsearch and Kibana

  For this example, I’ve downloaded elasticsearch-7.1.0-linux-x86_64.tar.gz and kibana-7.1.0-linux-x86_64.tar.gz. Once downloaded, their contents need to be extracted.

  Step 3: Create two Elasticsearch nodes

  To create a 2-node cluster, make two copies of the Elasticsearch directory, one named master one named node. After completing this step, you’ll have the following files and folders:

  elasticsearch-7.1.0                      elasticsearch-7.1.0-node
  elasticsearch-7.1.0-linux-x86_64.tar.gz  kibana-7.1.0-linux-x86_64
  elasticsearch-7.1.0-master               kibana-7.1.0-linux-x86_64.tar.gz
  

  Configure TLS and authentication

  The first thing we have to do is generate some certificates that allow the nodes to communicate securely. You can do this with an enterprise CA, but for this demo, we have a command called elasticsearch-certutil you can run to do this without any of the usual certificate confusion.

  Step 1: TLS on Elasticsearch master

  You can cd into the master directory and then run this command:

  bin/elasticsearch-certutil cert -out config/elastic-certificates.p12 -pass ""
  

  Next, open up the file config/elasticsearch.yaml in your favorite text editor. Paste the lines below at the end of the file.

  xpack.security.enabled: true
  xpack.security.transport.ssl.enabled: true
  xpack.security.transport.ssl.verification_mode: certificate
  xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
  xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
  

  Save the file, and we’re ready to start the master node. Run the command bin/elasticsearch. This executable has to keep running, put this terminal aside for now.

  Step 2: Elasticsearch cluster passwords

  Once the master node is running, it’s time to set the passwords for the cluster. In a new terminal we have to cd into our master node directory. Then we run the command bin/elasticsearch-setup-passwords auto. This will generate random passwords for the various internal stack users. You can alternatively skip the auto parameter to manually define your passwords using the interactive parameter. Keep track of these passwords, we’ll need them again soon.

  Step 3: TLS on Elasticsearch node

  Now let’s open another new terminal and cd into our node directory. In my case, this was named elasticsearch-7.1.0-node.

  The easiest thing we can do is to just copy everything from the master’s config directory into the node’s config directory.

  cp ../elasticsearch-7.1.0-master/config/* config/
  

  You should also add the configuration option node.master: false to the config/elasticsearch.yml file. I won’t get into the details here, you can find additional details in our cluster documentation. Alternative to just copying the config files wholesale, you can simply copy the certificate files and then set the xpack.security.* keys the same as with the master node.

  Then we start the node by running bin/elasticsearch. We’ll see it join the cluster. And if we look at the master node’s terminal window we can see a message there that a node joined the cluster. Now we have a 2-node cluster running.

  Step 4: Security in Kibana

  The last thing we need to do is configure Kibana. Once again open another terminal window (this is the last one, I promise). We’ll need to add the password for the Kibana user. We can grab that from the previous output of the setup-passwords command.

  Let’s cd into the Kibana directory and open up the config/kibana.yml file with your text editor. Find the lines that look like this

  #elasticsearch.username: "user"
  #elasticsearch.password: "pass"
  

  Uncomment the username and password fields by removing the # character at the beginning of the line. Change "user" to "kibana" and then change "pass" to whatever the setup-passwords command tells us the Kibana password is. Save the file then we can start Kibana by running bin/kibana.

  Configure role-based access control (RBAC) within Kibana

  Once Kibana is running we can switch over to our web browser and open http://localhost:5601. It’s going to ask us for a login, we’ll use the elastic superuser with the password from setup-passwords.

  

  Kibana will ask us if we want to load some sample data when it’s a new install such as this.

  

  Let’s load the flight and web sample data for the sake of this example. Once loaded, click on the configuration gear icon.

  

  Now let’s create our roles. Find and click on the roles option.

  

  Click on Create role.

  

  We’re going to name the first role read_logs.

  

  Scroll down to the index privileges field and select the log index. Then we give it the read privilege.

  

  This section about spaces is something you’ll probably want to investigate once you have security working, but for this post we’re going to skip it. To learn more about spaces — which are great for organizing and securing data within Kibana — check out our Kibana spaces documentation or this introductory Kibana spaces blog.

  Now let’s create another role called read_flight.

  

  We assign the flight index the read privilege.

  

  Now we’re going to create two users and assign them these roles. Select the Users link on the left and click the Create new user button.

  

  We’ll name this user flight_user and set the password. We don’t need to set the full name and email address. We need to assign our read_flight role and we also need to assign the kibana_user role as this user will be viewing data in Kibana.

  

  Hit create, and let’s do it again for the log_user. This time we add the read_logs and kibana_user roles.

  

  OK we’re all set with two new users. We can now log out as the elastic user.

  

  Then we can login as log_user. If we click on the dashboards icon we can see there are two dashboards, one for flights and one for logs.

  

  If you click on the Logs dashboard you will see the demo log data. If we go back and try to view the flight dashboard we can see the dashboard, but we can’t see any of the data. This is a place Kibana spaces can be used for security purposes to only allow certain users to see certain dashboards.

  Now we can logout as the log_user and log back in as the flight_user.

  If we open the log dashboard, this time we can’t see any data. We even get an error because Kibana can’t access the log index.

  

  Now when we check the flight dashboard we can see we have access to the flight data this time.

  

  Congratulations! We now have a two node cluster configured and two users with restricted access to index data.

  Wrapping up

  Security in the Elastic Stack is extremely versatile, there’s so much more than it can do we didn’t cover in this post. The purpose of this post was to get you started. In the future we’ll have plenty of content covering many of the security features and how you can use them. In the meantime, we have more documentation and blog posts on how Elastic Stack security works and what else you can accomplish.

  • Fundamentals of Securing Elasticsearch training
  • Elasticsearch security documentation
  • Configuring TLS blog post
  • Kibana Spaces blog post

  You can also ask for help on our public forum.

  Happy searching!

  
  
  https://www.elastic.co/blog/getting-started-with-elasticsearch-security
• 

  Elasticsearch on Kubernetes: A new chapter begins
  • sorangutan  

  1
  1
  Posts
  64
  Views

  

  We are excited to announce Elastic Cloud on Kubernetes (ECK), a new orchestration product based on the Kubernetes Operator pattern that lets users provision, manage, and operate Elasticsearch clusters on Kubernetes.
  

  Over the past few years, Kubernetes has emerged as the de facto standard for orchestrating containers and applications running in them. The trend is no different in the Elasticsearch community. Elastic Cloud on Kubernetes delivers on our promise to be where our users are, and provide them with the best possible solutions to deploy and operate Elastic products on their platform of choice.

  From releasing official Docker images for Elasticsearch and Kibana to modifying Beats to collect logs and metrics from the ephemeral pods and daemonsets, our journey with Kubernetes goes way, way back. Last December, we doubled down on our commitment by joining the CNCF and launching Helm Charts. ECK is a natural next step — albeit a big one — in our commitment to make it easier for our users to deploy and operate our products and solutions in Kubernetes environments.

  An Elasticsearch Operator, but so much more

  Built using the Kubernetes Operator pattern, ECK installs into your Kubernetes cluster and goes beyond just simplifying the task of deploying Elasticsearch and Kibana on Kubernetes. It focuses on streamlining all those crtoperations, such as:

  • Managing and monitoring multiple clusters
  • Upgrading to new stack versions with ease
  • Scaling cluster capacity up and down
  • Changing cluster configuration
  • Dynamically scaling local storage (includes Elastic Local Volume, a local storage driver)
  • Scheduling backups

  But ECK is much more than a Kubernetes Operator. In addition to automating all the operational and cluster administration tasks, it focuses on streamlining the entire Elastic Stack on Kubernetes experience. The vision for ECK is to provide an official way to orchestrate Elasticsearch on Kubernetes and provide a SaaS-like experience for Elastic products and solutions on Kubernetes.

  Secure by default

  All Elasticsearch clusters launched on ECK are secure by default, which means they have encryption enabled and are protected with a strong default password right at creation time. This experience aligns with what users get when using our Elasticsearch Service on Elastic Cloud.

  Providing this secure-by-default experience in a seamless way was no easy feat. Read about the latest security changes we made on the Elastic Stack side to make this possible. TL;DR: Starting with version 6.8 and 7.1, core Elasticsearch security features — TLS encryption, role-based access control, and file and native authentication — are now free.

  More free features: Canvas, Maps, Uptime, oh my

  All clusters deployed via ECK include powerful free features and tier capabilities such as frozen indices for dense storage, Kibana Spaces, Canvas, Elastic Maps, and more. You can even monitor Kubernetes logs and infrastructure using Elastic Logs and Elastic Infrastructure apps. It’s the fully featured Elastic Stack experience on Kubernetes that you will only get from Elastic.

  Deploy hot-warm-cold and custom topologies

  Hot-warm-cold is a powerful Elasticsearch cluster topology for logging, metrics, and other time series use cases. It’s a common architectural pattern to balance long-term storage and performance needs without breaking the bank. With ECK, users can deploy hot-warm-cold clusters on Kubernetes, and then easily configure data lifecycle policies using index lifecycle management (ILM) to move data between node tiers as it ages.

  There’s no compression algorithm for experience

  When it comes to deploying software, day 1 is easy. Day 2 not so much. A lot goes into streamlining the operation of a stateful system like Elasticsearch in a dynamic orchestration framework like Kubernetes. How do you provide persistent local storage that can dynamically scale? We are building Elastic Local Volume, an integrated storage driver for Kubernetes, right into ECK. Best practices like draining a node prior to scaling down, rebalancing shards as you scale up, and many more are baked right in.

  Elastic Cloud on Kubernetes builds on our years of operational knowledge as the creators of Elasticsearch and Elastic Cloud Enterprise, and operators of our Elasticsearch Service, Elastic's SaaS service available on AWS and GCP . We have encoded all that operational experience into how Elasticsearch and Kibana deployments are orchestrated and operated within Kubernetes.

  From ensuring no data loss on configuration changes to zero downtime when scaling, we left no operational stone unturned when building ECK.

  Forever free tier

  Since the early days of Elasticsearch, our goal has always been to provide new users with a rich getting started experience by packing a lot of powerful features into the free, default distribution. ECK is no different.

  The default distribution of ECK is forever free. In the spirit of openness and transparency, we have also made all the source code for ECK publicly viewable, licensed under the Elastic License, in the Elastic cloud-on-k8s GitHub repository.

  An Enterprise (paid) subscription enables additional features, including the ability to deploy clusters with advanced features such as field- and document-level access control, machine learning, graph analytics, and more. In the future, the Enterprise subscription will also unlock additional advanced orchestration features.

  Official GKE support, with more on the way

  In this initial alpha launch of ECK, we support Google Kubernetes Engine (GKE) and vanilla Kubernetes version 1.11 and above. Future versions will continue to expand support to other flavors of Kubernetes. Many customers also want to take advantage of Elasticsearch on managed Kubernetes across both cloud and on-premises environments such as GKE. ECK takes care of this with a reliable and consistent way to run and manage Elasticsearch clusters at scale

  Get started today

  Try this out today by running the following on your Kubernetes cluster, and visit the quickstart page for more information.

  
  
  https://www.elastic.co/blog/introducing-elastic-cloud-on-kubernetes-the-elasticsearch-operator-and-beyond
• 

  Why Observability loves the Elastic Common Schema
  • sorangutan  

  1
  1
  Posts
  282
  Views

  

  For the Elastic Stack 7.0 release, ingestion (namely Beats) went through a big change by adopting the Elastic Common Schema (ECS). One of the reasons for this is to make the correlation of events from logs, metrics and traces easier. This helps to make systems more “observable”. If you want to learn all about observability, check out the blog post “Observability with the Elastic Stack”.

  In this blog post I want to cover how the Elastic Common Schema changed the data we ingest, how it makes it possible to better correlate data, how it affects our Infrastructure, Logs, APM and Uptime UIs, and how it will help you to make your system more observable.

  Have a problem? Let’s start with the host...

  When investigating an issue, one of the first questions often asked is: Which host / machine is affected? As soon as the host on which an issue happens is known, it should be simple to get all the data about this host. This data could be:

  • Logs from the system, like the auth and system logs
  • Metrics like CPU, memory, and disk
  • Logs from the services running on the host
  • Metrics from the services running on the host
  • APM data from the services running on the host
  • Heartbeat data pinging the host
  • ...

  Before ECS it was not as simple as querying all the available data on a single field to get all events related to the host. For the hostname, there was beat.hostname, host.name and hostname, depending on which shipper was sending the data. The meaning of these could also have been different, sometimes containing what the hostname command returns or the fully qualified hostname depending on the use case. In addition to that there are also the fields defined by our users which might contain the host data based on their definition. There was no common schema that we could share with our users and recommend them to use.

  Hostname wasn’t the only field that had this kind of problem, as many other fields like IP addresses or information about services did too. To solve this problem we decided to introduce ECS to not only have a standard that is used internally, but can also be adopted by our users.

  As an example, with ECS we now have host.name and source.ip. Every data point ingested into these fields now has the same meaning. With the release of Elastic Stack 7.0, all shippers —  (all Beats and all APM agents — have moved to ECS. This means that all events, whether they be logs, metrics, or APM traces shipped from a host, or Heartbeat data about the uptime of host or service, they will contain the field and have it populated with the same meaning. This allows you to easily jump from logs to metrics to APM traces and compare the data.

  ECS also allows to answer other questions. For example, all information about MySQL services can be retrieved by running a query service.type: mysql. Combine this with a query on the host fields, and all information about a MySQL instance on a single host is available.

  Field changes on the ingest side

  ECS not only defines the host.* fields to store information about a host, it also contains many other fields which are important to filter and group on. An example is the service fields like service.type which stores which service the event data was pulled. Or in the case of web server logs like Apache or Nginx, source.* fields like source.ip can be used to identify the source of an event. This had implications on a large portion of fields shipped by Beats.

  Not only were the generic meta information that each Beat ships adapted to ECS, but also all the Filebeat, Metricbeat and Auditbeat modules. This has the effect, especially on the logging side, that the number of different fields was heavily reduced. Access logs shipped by an Apache or Nginx server are now ingested in almost the same format. If you use both services in your infrastructure, you can now correlated the data and build a single machine learning job on top of it.

  As many fields changed, if you built your own dashboards on top of the old field structure in 6.x, check out this breaking change list to see all the fields that were changed. Also check the Upgrading Beats from 6.x to 7.x guide. If you are only using our Infra or Logs UI, make sure that all your data you want to see in the 7.0 UI was created with versions 6.6 or 6.7 of Filebeat and Metricbeat. In 6.6, some ECS alias fields were introduced for compatibility reasons.

  Many more changes happened on the Filebeat module side than on the Metricbeat modules. The reason for this is that metrics are specific and unique. A metric about the size of a table in MySQL cannot be correlated with the index size of an Elasticsearch index. Most adjustments that happened to metrics fields were around host.name and IP fields which are meta information about the metrics itself.

  Elastic Common Schema in the UI

  The Logging and Infrastructure UI in 7.0 heavily depend on ECS fields. This means if shippers other than Beats follow ECS, the basic functionality will keep working. An example here are the container or the cloud fields. If an event contains the metadata around which cloud instance an event is coming from, the Infra UI can automatically group by it:

  

  The Logs UI depends on ECS on the message and the log fields. If the message field exists in an event, the Logs UI can automatically visualise the data correctly. Our UIs will rely more and more on ECS, this means if you ship data based on ECS, our UI’s will be able to make sense of it too.

  How to use the Elastic Common Schema

  Now that most of the data in the Elastic Stack is following ECS, there are a variety of ways to use it. If you have your logs, metrics and traces all stored in the same Elasticsearch cluster, you can use a search request across multiple indices. Most APIs in Elasticsearch by default allow specify multiple indices or a pattern that matches multiple indices.

  In case you have a cluster for logs, one for metrics and one for APM, Elasticsearch supports this use case with cross-cluster search (CCS). CCS allows you to connect multiple clusters and then specify similar to the above multi index search, to search across multiple clusters.

  Cross-cluster search also works for use cases where you might have local logging and metrics clusters in different parts of the world. To get all the data for a specific host.ip across all your cluster, it’s just one search and you will have all the information together in one place.

  What means ECS for the future of Observability?

  ECS just got started and will further evolve. The more standardised fields there are, the better the UI, and the easier it is for users can make sense of their data. It will make it easier for the user to better understand what the meaning of each value is. ECS can be used to correlate different data types based on the ECS fields. The capability of Elasticsearch having all the data in one or multiple clusters to make it available combined with ECS to correlate all the data makes this combination a great tool for your Observability needs. The more Elastic Stack data that is mapped to ECS, the more powerful the combination becomes.

  Not all data is shipped by Beats, so you often have also your own data. We encourage you to map your own data to ECS. This will not only allow you to correlate your data with all the other data but directly gives some context to each of your fields through the definition of ECS. Others looking at your data will understand what each field means. It also guarantees that as long as the same major version of ECS is used, no breaking changes happen. If your fields are mapped to ECS, they will not conflict with other fields shipped by Beats. 

  To get started with mapping your events to ECS, best go to the ECS docs pages. And if you want to see your ECS-normalized data within the curated APM, Infrastructure, Logs, and Uptime UIs, spin up a 14-day free trial of the Elasticsearch Service. As always, if you have any questions, reach out on Discuss.

  
  
  https://www.elastic.co/blog/easier-observability-with-the-elastic-common-schema
• 

  Helper Node Series | Tanya Bragin: Supporting Kids and Educators at MPCSD
  • sorangutan  

  1
  1
  Posts
  61
  Views

  

  Elastic believes in giving back, both by supporting the communities in which we live and work, and recognizing that participating in these activities can inspire and enrich our lives in unexpected ways. That’s why we offer our Elasticians volunteer time off (VTO) to foster a culture of caring.

  The Helper Node blog series shines a light on the inspiring ways our Elasticians use their volunteer hours to help their community and the world at large. In this edition of the Helper Node series we meet Tanya, who volunteers her time at an elementary school in Menlo Park City School District (MPCSD).
  

  ---

  Tell us a bit about yourself and what do you do at Elastic?

  Tanya Bragin: I joined the company in 2014 as one of the first product managers. Currently, I am a senior director of product management and a product lead for Observability, a set of use cases that include Logging, Metrics, and APM. At Elastic, a product lead works side-by-side with team and tech leads to define product and go-to-market strategy.

  This requires immersing yourself deeply in what’s happening in a specific market, listening attentively to customers and users, thinking about how we can bring our technology to solve the problems we see in new ways, and then working collaboratively and cross-functionally to make it happen. Day-to-day, this can result in a dizzying schedule of internal status meetings, customer calls, and travel to conferences, user events, and regional customer trips.

  

  Tanya in front of Tower Bridge in London, UK.
  

  How did you arrive at the idea of volunteering your time?

  Tanya: Volunteering consistently at the same organization has not been something I’ve done before. I joined many one-time events organized by others to further great causes, and have always derived a sense that I’m doing something good, but somehow every time it felt like an excursion. I would immerse myself in a worthy cause for a day, but the next day I would be back in my own life, often not seeing the long-term effect of my actions (or lack thereof -- I was never quite sure).

  When joining Elastic, I knew that the product I would work on, which is open source and free at its core, is adopted by many organizations and projects to further social good. That made me feel happy. However, there was a gnawing feeling that in my own life, I was not doing enough to concretely help contribute to efforts I personally believed in. I struggled with that realization, it made me feel incomplete.

  After having my daughter, I became viscerally aware just how vulnerable children can be. I would tear up when reading news stories that reminded me how many kids need our help, encouragement, and protection in so many ways; from war zones to economic instability to family challenges -- children often suffer the most.

  Looking around, I also saw that you do not have to travel to far-flung places to be a caring adult that can serve as a positive role model and help children. When my daughter started going to preschool, I witnessed firsthand the difference educators make in the development of a young mind. I also knew that in many communities educators can face challenges due to understaffing and a lack of resources. I realized that getting involved in the school system in my own community would be a really concrete way to contribute to a cause I really cared about.

  I wanted to volunteer as a parent aid in a classroom directly supporting educators and kids. However, there was one catch. I already spent considerable energy balancing professional life, family life, and motherhood. And I don’t think that I’m alone in that, for better or worse. I viewed stepping away from work on a regular basis during the work week a breach in the work ethic I’ve adopted over the years.

  

  Tanya in front of MPCSD building.

  What role did the VTO program play in your decision to volunteer on a weekly basis?

  Tanya: Right off the bat, Elastic proved to be a flexible workplace when I arrived here in 2014. Even before the VTO program was established, due to the distributed nature of the company, there was less pressure to work traditional hours and be physically in the office. Since our collaborators were often on different time zones and holiday schedules, we all worked the hours that made us productive, from the locations that made sense, and we were responsible for our results, rather than clocking in and out at a particular office.

  Even so, I found it challenging to step away consistently during business hours to volunteer. Mostly, this was because I felt guilty declining meetings others worked hard to set up while working with my already way too busy schedule. I simply did not know how to explain the trade-off I was making to volunteer consistently during that time to either myself or my peers.

  When Elastic announced the VTO program, I was ecstatic. After looking at the details, I quickly realized it was exactly what I needed to make my weekly volunteering commitment official, stable, and non-negotiable. I finally took the plunge, and signed up every Friday morning to help kindergarten school teachers at a local public school as a parent aid during a math and art class.

  What does your volunteer time look like?

  Tanya: The school does a good job organizationally to make it easy for busy parents like myself to get involved in all kinds of volunteer opportunities. They have an online system where I can go and check for opportunities to volunteer and sign up. This is how I found out about the ability to volunteer on Fridays at the art room. Given my frequent out-of-town trips early in the week, Fridays was an ideal day for me to volunteer consistently.

  My duties are seemingly quite basic. During math class, I facilitate a progressively more challenging set of exercises that help students master the early skills in counting, addition, and subtraction. During art, the teacher needs help preparing the various activities in the classroom for students. Once the students pile in, she gives them an overview of activities available and they plunge right in. The majority of my time is spent answering questions to make sure the kids understood the goals and occasionally helping them with a difficult task.

  

  A kindergarten student working on an art project.

  What do you take out of the experience?

  Tanya: It is gratifying to watch students become more independent and patient with themselves over the course of the year; to not only complete the tasks, but to work together and encourage each other. For my part, I look out for anyone that needs extra advice or encouragement, and let them know I am proud of them as they try new things, ask inquisitive questions, and add creative touches to their projects.

  In some cases, my contribution feels deeper than simply helping with the mechanics.

  In one instance during math class, I noticed that a girl seemed unenthusiastic about the exercise. She was quite a bit behind the peers at her table by the time I joined them as a facilitator. The other kids were gently teasing her. I tried to engender empathy and camaraderie among them, reminding them that everyone learns at their own pace, but still the teasing continued. At some point during class, rushing, I handed the girl a more advanced exercise by mistake, and she suddenly blazed right through it. I realized then that she was unenthusiastic because she was bored! I was so happy to be there to offer individual attention and help her get ahead.

  In another instance, I saw that it was hard for kids to understand why a peer, who recently moved from another country and was still learning the language, acted out in ways they did not understand. I shared with them that I had the same jarring experience of moving to America as an awkward teenager, and encouraged them to be patient, engage with their peer, and incorporate him into their activities. When later in the year, the student, who by then mastered the language, thanked me for being kind to him, my heart simply melted.

  It is during moments like these that I know I am in the right place.

  ---

  Does distributed work sound like the right fit for you? Why not check out our teams and see if there's a fit for you! You’ll be glad you did.

  
  
  https://www.elastic.co/blog/helper-node-series-tanya-bragin-supporting-kids-and-educators-at-mpcsd
• 

  Introducing Field Aliases in Elasticsearch
  • sorangutan  

  1
  1
  Posts
  171
  Views

  

  In Elasticsearch 6.4, we introduced support for a new mapping type called alias. This data type allows users to define an alternate name for a field in an index, which can then be used in place of the target field in searches. Field aliases can be helpful in a few different situations.

  Renaming fields

  Changing the name of a field typically requires all documents containing the field to be reindexed, a potentially expensive undertaking. If you happen to be using time-based indexing, you can use field aliases to rename the field without having to reindex old data. The process works as follows:

  1. On each existing index, we add a field alias mapping the new field name to the old one.
  2. For newly-created indices, we start to use the new field name exclusively.
  3. Eventually the time-based indices containing the old name are dropped or archived, and the rename is complete.

  With the field aliases in place, it’s possible to search across all the indices using the new field name while the rename is in progress.

  Uniting indices under a common schema

  There are many benefits to using a common schema like ECS for all your data: you can easily run searches that span multiple indices, make use of predefined dashboards and machine learning jobs, and more. Migrating existing indices to conform to the common schema can be challenging, however, as they may contain fields with custom names and formats. Field aliases can help in this process: given an index with non-standard field names, we can add field aliases with the names of standard schema fields. We can then search across all indices using the new names, allowing us to use shared dashboards and other tools.

  Now that you have a sense of why field aliases are useful, let’s dive into an example to better understand how they work.

  Field aliases in action

  Imagine we’re part of a group of engineers who help maintain the international space station. We’re developing an application that collects telemetry data from the station in order to monitor and analyze its daily function. The data is stored in Elasticsearch indices that look something like this:

  PUT station_telemetry_2019-04-15
  {
    "mappings": {
      "properties": {
        “active_batteries”: {
          "type": "integer"
        },
        "cabin_temperature": {
          "type": "float"
        }
      }
    }
  }
  

  Because of the large number of readings, the data is organized into time-based indices with a new index per day. Indexes are dropped when they grow older than one month.

  We get word that an American analyst will be joining the project, and it dawns on us that none of the field names specify units! Remembering how dangerous mismatched units can be, we decide to rename the ambiguous fields. We want to ensure that newly-created indices use updated field names like cabin_temperature_celsius. However, our analysis regularly extends to the past week of data, which will also include indices with the original field name.

  As we saw in the section on 'renaming fields', this is a situation in which field aliases can be helpful. On each existing index, we add a new alias field to the mapping called cabin_temperature_celsius that refers to the original field cabin_temperature:

  PUT station_telemetry_2019-04-15/_mapping
  {
    "properties": {
      "cabin_temperature_celsius": {
        "type": "alias",
        "path": "cabin_temperature"
      }
    }
  }
  

  We also make sure that going forward, new indices use the name cabin_temperature_celsius from the start. Now we can search across indices using the new name:

  GET station_telemetry_*/_search
  {
    "query": {
      "range": {
        "active_batteries": {
          "gte": 4
        }
      }
    },
    "aggs": {
      "average_temp": {
        "avg": {
          "field": "cabin_temperature_celsius"
        }
      }
    }
  }
  

  Calling the field capabilities API shows that cabin_temperature_celsius can be searched as a float across all indices. This is true even though it is mapped as a field alias in older indices, and as a concrete field the newly-created ones:

  GET station_telemetry_*/_field_caps?fields=cabin_temperature_celsius
  
  {
    "fields" : {
      "cabin_temperature_celsius" : {
        "float" : {
          "type" : "float",
          "searchable" : true,
          "aggregatable" : true
        }
      }
    }
  }
  

  We are now able to update our dashboards to use the new field names that include units, helping avoid confusion for our new colleague. After enough time, the indices containing the original field will be dropped, leaving only the new field names.

  Searching vs. Indexing

  When processing a search request, Elasticsearch checks each field to see if it matches the name of an alias. If it does, then the field is resolved to its target before executing the search request. Aliases can be used in almost all parts of the search request, including queries, aggregations, and sort fields, as well as docvalue fields and scripts. As we saw above, the field capabilities API supports aliases, which can be helpful in figuring out how to search across different indices.

  It’s important to note that field aliases cannot be used when indexing documents: a document’s source can only contain concrete fields. For this reason, field aliases don’t appear in the result of APIs like GET, which return the document source untouched.

  If the new field names need to be used when ingesting documents or loading their source, then field aliases may not be a good fit. However, when it’s most important to be able to search using these new names, the alias mapping type can provide a lighter-weight option to rewriting the data.

  Learn more

  The field alias documentation contains more detail on this new data type, including important information about its limitations. We hope you get the chance to try out field aliases with Elasticsearch either online or by downloading it, and look forward to your feedback on GitHub and the Discuss forums!

  
  
  https://www.elastic.co/blog/introducing-field-aliases-in-elasticsearch
• 

  Infrastructure Monitoring at King Abdullah University of Science and Technology
  • sorangutan  

  1
  1
  Posts
  60
  Views

  

  King Abdullah University of Science and Technology is a graduate research university located in Thuwal, Saudi Arabia. Founded in 2009, it provides research and graduate training programs and is the kingdom’s first mixed-gender university.

  We met recently with Stanislav Flachs, Manager of the Systems, Storage, Automation & Workflow teams, and Gianluca Castellani, lead for the Automation & Workflows team. The two analyse time-consuming tasks and processes to find ways of improving them. They are both actively managing the Elasticsearch clusters of the university. Flachs and Castellani started using the Elastic Stack so they could connect the data with the visualisation and alerting systems they needed.

  ---

  Prior to Elastic, what infrastructure issues were the university encountering?

  Gianluca Castellani: In the past, we were using different tools to monitor our infrastructure. There was always an issue that persisted. These different systems were not meant to work together. Our previous solutions were scattered and not flexible. We always had to come up with creative ways of connecting the data with the visualization and the alerting we needed. It is crucial for us to have visibility on our systems. Not having a holistic tool to monitor and analyze our operational data was just cumbersome. If you don’t monitor properly, that can have catastrophic consequences on your system. We didn’t have a tool to look efficiently at what we were doing.

  Why did you choose the Elastic Stack to help solve your infrastructure issues?

  Stanislav Flachs: We considered a few options — looking at commercial and open-source solutions. We finally decided to go for the Elastic Stack, partially because of the licensing model. This was not the only reason, however. The community spirit so characteristic of Elastic really attracted us. We wanted to be part of it and it just resonated with us better. We are all programmers. We are keen to contribute and work on this. Also, our researchers were already familiar with the solution. Thinking further ahead about providing resources internally to our research and business communities, it just made more sense to use something they already adopted. The learning curve with the Elastic Stack was also quite smooth, making the adoption easier.

  Castellani: We embraced Elasticsearch for many reasons, such as scalability, easy set up and management, its large user community, flexible and robust data ingestion, and its intuitive data visualization.

  How has the university benefited from the Elastic Stack?

  Flachs: We are using Elastic to store heterogeneous data and feed other tools already used in the organization. We store end-user communication by writing ad hoc Elasticsearch plugins for Slack and regular emails. Then we can provide dashboards, reports and statistics to the upper management so they can have KPIs about our operational data and our systems. The Elastic Stack is at the heart of our IT infrastructure monitoring, and it is a key tool for our daily activities. We can check alerts, CPU and memory utilization, workflow characterization, and so much more. This allows us to improve our capacity planning, so we can now better plan for our infrastructure upgrades and ultimately we are saving time and money.

  What are the next steps with the Elastic Stack?

  Castellani: Since we started using it, the Elastic Stack has proven to be an easy-to-use and flexible platform that can host any type of data whilst gathering everything into one place, and it accommodates multiple use cases. We are now working on implementing the security feature to manage the access to the resources at a more granular level. We are also planning on using machine learning to analyze logs from different sources and to find anomalies in our systems. Our goal is to do intrusion detection and foresee any hardware failure.

  Today, after more than 1 year from our first cluster, we are still discovering new functionalities, creating new applications and getting new insights from our data. To conclude, we want simply to say that Elasticsearch has changed the way we approach data. And the old saying: ‘When you have a hammer, everything looks like a nail’, has become: ‘When you have Elasticsearch, everything looks like JSON.’

  ---

  
  

  

  Stanislav Flachs  s a battle-tested veteran of IT operations, passionate about DevOps culture and technological innovation. In his role as Systems & Storage Manager at KAUST, Stanislav is responsible for implementing and operating compute and storage solutions that scale and adapt to evolving user needs. When he is not behind the keyboard, you’ll find Stanislav underwater exploring the blue depths of the Red Sea.

  
  

  
  

  

  Gianluca Castellani is a Research Computing “jack of all trades, master of many”. His main area of interest is making large computing systems robust, resilient and efficient using testing, monitoring and automation. His true passions are creating solutions and solving extrinsic problems based on the needs of his colleagues and himself.

  
  
  https://www.elastic.co/blog/infrastructure-monitoring-at-king-abdullah-university-of-science-and-technology-with-the-elastic-stack
• 

  Indexing Elasticsearch Audit Logs with Filebeat
  • sorangutan  

  1
  1
  Posts
  130
  Views

  

  When security features are enabled in Elasticsearch, it is possible to enable an audit log as well. This log contains events that occur in the Elasticsearch node that might be interesting from a security perspective. To analyze these events easily, they can be indexed back into Elasticsearch, preferably into a separate cluster from the one that's generating the events. This blog post explains how this indexing feature has changed in recent times.

  Until version 6.7.0 of Elasticsearch, the recommended way of indexing audit logs back into Elasticsearch for easy analysis was to use the index output type when configuring the audit log settings. This output type was deprecated in version 6.7.0. In the same version, we introduced the audit fileset of the elasticsearch module in Filebeat, which performs the same role as the index output type of the Elasticsearch audit log.

  In version 7.0.0 of Elasticsearch, we removed the index output of the Elasticsearch audit log completely. Configuring Filebeat is henceforth the only alternative for easy analysis of the audit log.

  Why has the index output type been retired?

  Although using the index output type was convenient from a configuration standpoint, there are strong technical reasons why this option is not suitable in production environments and so it has been retired. The main drawback is that the audit event throughput is related to the node’s request load. Nodes with higher traffic generate more audit events. The indexing throughput, however, is limited by the destination cluster’s capacity. If the destination cluster can’t keep up with the volume, then we eventually need to drop some of the audit events or selectively drop client requests. Moreover, the queueing buffer for audit events was in-memory with no disk persistence, because disk persistence was assured by the other dedicated logging output type logfile. All this can lead to audit data loss in a multitude of complex failure scenarios.

  In short, this functionality was hard to maintain and provided little gains. Instead, by “outsourcing” the indexing task to Filebeat, Elasticsearch’s audit trail only has to focus on one task: storing events on the persistent storage and with no queueing. The audit capacity eats at a resource (disk I/O) that the administrators already have to plan for. Filebeat, being a separate process that runs alongside the Elasticsearch node, can take the indexing load off the Elasticsearch node's shoulders. And since Filebeat works off the audit log files generated by the Elasticsearch node, these files act as a natural queue. If the Elasticsearch cluster receiving the audit log documents for indexing is under load, Filebeat will wait until the load lessens and then continue indexing from the last entry it ingested from the audit log file.

  Finally, using Filebeat provides you with additional options that weren’t available with the old index output, such as event filter (see examples below) or sending events to Logstash for enrichment or to Kafka for queueing before further routing.

  Using Filebeat to ingest Elasticsearch audit logs

  Let's see what steps are involved in using Filebeat to ingest Elasticsearch audit logs into an Elasticsearch cluster.

  On every node in your Elasticsearch cluster, do the following:

  1. Enable the audit log for that node by setting xpack.security.audit.enabled: true inside the elasticsearch.yml config file. See Auditing Security Events for more details.
  2. Start the Elasticsearch node.
  3. Install Filebeat alongside the Elasticsearch node. This could mean installing Filebeat on the same host or VM as the Elasticsearch node or in a sidecar container.
  4. Filebeat ships with a variety of modules, each catering to a well-known source of log files. In this case, we want to use the Elasticsearch module, so enable it.
  5. Once the Elasticsearch Filebeat module is enabled, it will look in well-known paths for Elasticsearch log files and it will naturally follow them as they are rotated. If you want to customize these paths, edit the elasticsearch.yml file under the modules.d folder and make the necessary configuration changes. The modules.d folder is located in the same folder that contains filebeat.yml, which is the main Filebeat configuration file.
  6. To specify which Elasticsearch cluster you want to index your audit log events into, as well as other settings related to the Elasticsearch output (e.g., authentication settings), edit the output.elasticsearch section in filebeat.yml.
  7. Finally, start Filebeat. Your Elasticsearch audit logs will be indexed into filebeat-* indices (by default) on the Elasticsearch cluster you configured in the previous step.

  

  Common considerations

  Indexing a subset of audit log events

  By default every event that appears in the audit log file will be picked up by Filebeat and indexed into Elasticsearch. However, you may want to index only certain audit log events. This can be achieved by using Filebeat's drop_event processor with the appropriate conditions.

  For instance, you may want to only index audit log events involving the elastic user. To achieve this, edit the modules.d/elasticsearch.yml file and under the audit fileset section, add the following:

  input:
    processors:
      - decode_json_fields:
          fields: [ "message" ]
          target: temp
      - drop_event:
          when:
            not:
              contains:
                temp.user.name: elastic
      - drop_fields:
          fields: [ "temp" ]
  

  You can view the list of available audit event fields in the documentation.

  Indexing audit log events in multiple Elasticsearch clusters

  A single Filebeat instance can only send events to a single Elasticsearch cluster. This means that if you would like to send your audit log events to multiple Elasticsearch clusters that you have a couple of options:

  1. Run multiple Filebeat instances. Configure the output.elasticsearch section, particularly the hosts setting, of each instance to send to a different Elasticsearch cluster.
     

  2. Run a single Filebeat instance and send your audit logs to Logstash by configuring the output.logstash section in your filebeat.yml. Then configure your Logstash pipeline to with multiple elasticsearch outputs to send to multiple Elasticsearch clusters.
     

  3. Run a single Filebeat instance and send your audit logs to one Elasticsearch cluster. Then use cross-cluster replication to replicate the audit logs index over to another Elasticsearch cluster.
     

  Analyzing audit logs in Elasticsearch

  Once your audit logs have been indexed into filebeat-* indices, you can start running queries against them and creating visualizations in Kibana.

  For instance, here's a sample query to count the different types of Elasticsearch transport-level actions that were performed every hour:

  POST filebeat-*/_search
  {
    "size": 0, 
    "query": {
      "bool": {
        "filter": {
          "term": {
            "event.dataset": "elasticsearch.audit"
          }
        }
      }
    },
    "aggs": {
      "by_hour": {
        "date_histogram": {
          "field": "@timestamp",
          "interval": "1h"
        },
        "aggs": {
          "by_event_type": {
            "terms": {
              "field": "elasticsearch.audit.action"
            }
          }
        }
      }
    }
  }
  

  But wait, there's more!

  Not only can Filebeat ingest Elasticsearch audit logs but it can also ingest other types of Elasticsearch logs — the main server log, GC log, slow log and deprecation log. Think of all the amazing correlation and other analysis you could do in Kibana about the all the events happening in your Elasticsearch cluster!

  And if you're looking for an Elasticsearch cluster to send your audit — and other — logs to try taking our Elasticsearch Service for a 14-day free trial!

  As always, questions are welcome (and encouraged) on our Discuss forums.

  
  
  https://www.elastic.co/blog/indexing-elasticsearch-audit-logs-with-filebeat
• 

  This Week in Elasticsearch and Apache Lucene - 2019-05-10
  • sorangutan  

  1
  1
  Posts
  101
  Views

  

  Elasticsearch Highlights
  

  Queryable Object Fields

  We completed some benchmarks for embedded_json fields. The summary is that they look quite good, the performance is not too different from the baseline of mapping each individual field separately. This should conclude the work on this new field and the branch should be merged into master soon now that we have a better idea of the performance.

  Performance

  We released Rally 1.1.0! This release contains quite a few enhancements, including added support for ES 7.x as a metrics store and updates to support 7.x APIs plus some bug fixes and docs for the ccr-stats telemetry device.

  We also worked on making Rally safe by opening a PR to fail Rally fast if a user attempts to override track variables that don’t exist. The existing behavior can be particularly dangerous e.g. when instead of bulk_size a user, due to a typo, specifies --track-params=”bulk-size:1000” thus eventually benchmarking a wrong scenario.

  OpenID Connect Realm

  The configuration guide doc is ready and the kibana PR that adds an OpenID Connect authentication provider in Kibana is undergoing review, so all the pieces for OpenID Connect support in our stack are falling into place!

  Java Time

  We are working on improving the Upgrade Assistant and Deprecation Info API to include guidance for migrating from joda to java.time patterns.

  Say no to downgrades

  Downgrading an Elasticsearch node to an earlier version is unsupported, because we do not make any attempt to guarantee that a node can read any of the on-disk data and metadata written by a future version. Yet today we do not actively prevent downgrades, and sometimes users will attempt to roll back a failed upgrade with an in-place downgrade and get into an unrecoverable state. David is adding checks that will prevent in-place downgrades, with an escape hatch that requires manual intervention.

  Reindex

  We are renaming the top-level reindex parameter size for limiting the maximum number of documents processed to max_docs, since size caused confusion (scroll/batch size vs hit size).

  Frozen SQL

  We worked on a PR to support frozen indices in SQL. This work revealed a small bug in Elasticsearch that we fixed, and we adapted the code in ODBC to accommodate the SQL work with frozen indices.

  GeoSQL

  We opened a PR for merging geosql branch to master! We spent the week working on GeoSQL finishing touches, merging the PR for switching server side of GeoSQL from ShapeBuilder to libs/geo classes, added the ST_Z function. We also fixed the reported display length for geo types and improved error messages and documented current limitations for geo in SQL (notably, geoshapes don't have doc values yet, geopoints are indexed with a potential loss of precision, and that altitude is accepted but not indexed).

  Hadoop

  We addressed some backward compatibility issues, while these issues are technically beyond our supported compatibility; the fixes were relatively straightforward and we want our users to have the best possible experience. (#1287) (#1284) (#1283).

  Snapshot Lifecycle Management

  We finished up the PR to persist SLM history. This will allow users to view the history of snapshots that have taken place and help troubleshoot failures. This also opens the door to create watches to notify someone if the snapshot fails. (#41707)

  Watcher UI

  We are working on bug fixes that will get us over the finish line on the angular to react migration.

  Apache Lucene

  Lucene 8.1

  A second release candidate of Lucene 8.1 is under vote. It includes several changes we are interested in, notably:

  •  LUCENE-7673 Much faster merging of BKD trees with high numbers of dimensions like shapes. Points and ranges benefit a bit from it too but to a much lesser extent.
  •  LUCENE-3041 A query visiting API.
  •  LUCENE-8652 Synonyms can be assigned different weights.
  •  LUCENE-8671 Easier configuration of whether to load the terms index on or off- heap.

  
  

  Lucene 7.7.2

  We plan to start working on releasing 7.7.2 once 8.1 is out. It already has 2 bug fixes that we are interested in for Elasticsearch:

  •   LUCENE-8735 makes FilterDirectory propagate pending deletions, we currently have a workaround for it in Elasticsearch that we could remove in the future like we did on master.
  •   LUCENE-8688 is a bug that makes `_forcemerge` merge too much when `max_num_segments` is not 1.

  Java 11 slowdowns

  If you remember the update from 2 weeks ago, we got curious about slowdowns after we switched to Java 11 and our gut feeling was that it was due to the change of garbage collector. A contributor fixed nightly benchmarks to set the garbage collector back to the same that was used previously instead of relying on defaults, which resolved most slowdowns.

  Other

  •  A bug was reported, which seems to be a bug in IBM J9.
  •  We are working on improving the way that working units are computed when IndexSearcher is configured to search with multiple threads.
  •  We are exploring switching IntArrayDocIdSet#advance from binary search to exponential search, which is expected to better perform in worst-case scenarios.
  •  IndexWriter#deleteAll has concurrency issues due to the fact that it tries to reset the schema. Note: Elasticsearch doesn't use this API.
  •  A contributor is exploring encoding FST arcs densely so that looking up a label would consist of an array lookup rather than a binary search.
  •  The community is suggesting adding a new rescorer implementation that is based on a collector rather than a query.

  Changes in Elasticsearch

  Changes in 8.0:

  • Throw exception if legacy interval cannot be parsed in DateIntervalWrapper #41972
  • BREAKING: Removes typed endpoint from search and related APIs #41640
  • Update TLS ciphers and protocols for JDK 11 #41808
  • BREAKING: Cleanup versioned deprecations in analysis #41560

  Changes in 7.2:

  • Add painless string split function (splitOnToken) #39772
  • SQL: Add support for FROZEN indices #41558
  • fix unlikely bug that can prevent Watcher from restarting #42030
  • Fix node close stopwatch usage #41918
  • Limit max direct memory size to half of heap size #42006
  • Fix IAE on cross_fields query introduced in 7.0.1 #41938
  • Add HTML strip processor #41888
  • Remove manual parsing of JVM options #41962
  • Fix assertion error when caching the result of a search in a read-only index #41900
  • Cleanup exceptions thrown during RollupSearch #41272
  • Update lintian overrides #41561
  • Add gradle plugin for downloading jdk #41461
  • Make ISO8601 date parser accept timezone when time does not have seconds #41896
  • Allow IDEA test runner to control number of test iterations #41653
  • Always set terminated_early if terminate_after is set in the search request #40839
  • Upgrade SDK and test discovery-ec2 credential providers #41732
  • Allow unknown task time in QueueResizingEsTPE #41810
  • BREAKING: Reject port ranges in discovery.seed_hosts #41404
  • Update error message for allowed characters in aggregation names #41573
  • Force selection of calendar or fixed intervals in date histo agg #33727
  • Skip explain phase when only suggestions are requested #41739
  • Add index name to cluster block exception #41489
  • StackOverflowError when calling BulkRequest#add #41672
  • Noop peer recoveries on closed index #41400
  • Cut over ClusterSearchShardsGroup to Writeable #41788
  • Adapt low-level REST client to java 8 #41537

  Changes in 7.1:

  • Fix fractional seconds for strict_date_optional_time #41871

  Changes in 7.0:

  • Handle serialization exceptions during publication #41781

  Changes in 6.8:

  • fix org.elasticsearch.xpack.watcher.test.integration.RejectedExecutionTests #41777
  • Allow reindexing into write alias #41677
  • SQL: Remove CircuitBreaker from parser #41835
  • Remove Harmful Exists Check from BlobStoreFormat #41898
  • Add tasks to build Docker build context artifacts #41819
  • Log warning when unlicensed realms are skipped #41778
  • Enforce that Maven/Ivy repositories use https #41812

  Changes in Elasticsearch Management UI

  Changes in 7.2:

  • Reenable Rollup Jobs API test that was failing due to interval change in ES #36310

  Changes in 6.8:

  • Update license management copy related to security #36204

  Changes in Elasticsearch SQL ODBC Driver

  Changes in 7.2:

  • Fix integration testing app to run on non-Win platforms. Update ES setup sequence #153

  
  
  https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2019-05-10
• 

  Advanced tuning: finding and fixing slow Elasticsearch queries
  • sorangutan  

  1
  1
  Posts
  1626
  Views

  

  Elasticsearch is a very flexible and feature-rich application that provides many different ways to query your data. But have you ever experienced query speeds that were less than you hoped for? With a distributed system like Elasticsearch, there can be various possible factors contributing to query performance, including external factors such as load-balancer settings, network latency (bandwidth, NIC cards/drivers), etc.

  Within this blog, I will be discussing what can cause slow queries and how to identify them within the context of Elasticsearch. This article stems from some of our common troubleshooting methods that may require one to be fairly intimate with how Elasticsearch works.

  Common causes of slow Elasticsearch queries

  Before we look into some of the trickier cases, let's start off with some of the most common causes of slow queries and their solutions.

  Symptom: High resource utilization when inactive

  Every shard consumes resources (CPU/Memory). Even with zero indexing/searching requests, the existence of shards consumes cluster overheads.

  Problem

  The cluster has too many shards to a point where any query may seem slow to execute. A good rule of thumb is to ensure you keep the number of non-frozen shards per node below 20 per GB heap configured.

  Solution

  Reduce shard count, freeze indices, and/or add additional nodes to cope with the load. Consider hot/warm architecture (works very well for time-based indices) together with the rollover/shrink feature in Elasticsearch to efficiently manage shard count. A good start to any deployment would be to perform proper capacity planning to help determine the optimal number of shards for each search use case.

  Symptom: High thread pool rejected count

  The search thread pool is showing an on-going increase in "rejected" count which is cumulative based on last cluster restart.

  GET /_cat/thread_pool/search?v&h=node_name,name,active,rejected,completed
  

  The response looks like this:

  node_name             name   active rejected completed
  instance-0000000001   search      0       10         0
  instance-0000000002   search      0       20         0
  instance-0000000003   search      0       30         0
  

  Problem

  Queries are targeting too many shards, exceeding the number of cores in the cluster. This creates queued tasks in the search thread pool, leading to search rejections. Another common cause is slow disk I/O causing searches to queue up or in some cases completely saturated cpu. 

  Solution

  Adopt a 1 Primary : 1 Replica model when creating indices. Index template is a good way to roll out this setting at index time. (1P:1R will be default in Elasticsearch 7.0 or later). Elasticsearch 5.1 or later supports search task cancellation, which can be useful when the slow query shows up in the Task Management API. To improve disk I/O, check out our storage recommendations and be sure to use recommended hardware for maximum performance.

  Symptom: High CPU and indexing latency

  Metrics correlation shows high CPU utilization and indexing latency when cluster is overwhelmed.

  Problem

  Cluster is heavily indexing, which affects search performance.

  Solution

  Increasing index.refresh_ interval (amount of time between when a document gets indexed and when it becomes visible) to a value like 30s generally helps improve indexing performance. Actual mileage may vary, so testing is key. This makes sure the shards don’t have to work too hard by creating a new segment every 1 second (default).

  For a heavy indexing use case, checkout our index tuning recommendations to optimise both index and search performance.

  Symptom: Increased latency with more replica shards

  Query latency can be observed after increasing replica shards count (e.g., from 1 to 2). If more data is present, then cached data will get evicted sooner leading to an increase in operating system page faults.

  Problem

  The filesystem cache doesn't have enough memory to cache frequently queried parts of the index. Elasticsearch's query cache implements an LRU eviction policy: when the cache becomes full, the least recently used data is evicted to make way for new data.

  Solution

  Set aside at least 50% of the physical RAM for the filesystem cache. The more memory, the more can be cached especially if the cluster is experiencing I/O issues. Assuming heap size has been properly configured, any remaining physical RAM left that can be used for the filesystem cache goes a long way to speed up search performance.

  For example, a 128GB RAM server has 30GB set aside for heap and remaining memory for filesystem cache (sometimes called OS cache), which is a way that the operating system caches recently accessed 4KB blocks of data, so that if you read the same files again and again then you won't need to go to disk most of the time, the read request will be served directly from memory.

  Besides filesystem cache, Elasticsearch uses query cache and request cache to speed up searches. All these caches can be optimised using search-request-preference to route certain search requests to the same set of shards every time instead of alternating between the different copies available. This will make better use of the request cache, node query cache and filesystem cache.

  Symptom: High utilization when sharing resources

  Operating system shows consistently high CPU / disk I/O usage. Performance gain can be seen after stopping third-party applications.

  Problem

  There is resource (CPU and/or Disk I/O) contention between other processes (e.g., Logstash) and Elasticsearch itself.

  Solution

  Avoid running Elasticsearch together with other resource-intensive applications on shared hardware.

  Symptom: High heap usage aggregating highly unique fields

  Poor performance when querying aggregated fields that contain highly unique values (e.g., IDs, usernames, email addresses, etc). During heap dump analysis, you should see Java objects with terms like "search", "buckets", "aggregation", etc., consuming much of the heap.

  Problem

  Aggregations are running on high-cardinality fields requiring lots of resources to fetch many buckets. There can also be nested aggregations involving nested fields and/or join fields.

  Solution

  To improve performance of high-cardinality terms aggregations, have a read at this blog post by my colleague from the consulting team: https://www.elastic.co/blog/improving-the-performance-of-high-cardinality-terms-aggregations 

  For further tuning, check out our recommendations on nested fields and join fields to better improve your aggregation performance.

  Occasional slow queries

  Generally speaking, occasional or intermittent slow queries can benefit from some of the tuning for index/tuning for search recommendations. Occasional slow queries should be closely correlated with one or more of these monitoring metrics:

  • CPU load
  • Indexing throughput
  • Search throughput
  • Garbage collection (GC) activity
  • Search thread pool queue sizes

  Elasticsearch has another useful feature called adaptive replica selection (ARS) that allows the coordinating node to be aware of the load on data nodes and allows it to choose the best shard copies for executing a search, thus improving search throughput as well as latency. ARS can be a big help for occasional slowdowns by more evenly spreading the load during query time. In Elasticsearch 7.0 and later, ARS will be turned on by default.

  Consistent slow queries

  For consistent slow queries, we can try removing features from the query one by one and check whether the query is still slow. Finding the simplest query that reproduces the performance issue helps to isolate and identify the problem:

  • Is it still slow without highlighting?
  • Is it still slow without aggregations?
  • Is it still slow if size is set to 0? (when size is set to 0, Elasticsearch caches the results of search requests to make search faster)

  Can it benefit from some of the "tuning for search" recommendations?

  During troubleshooting, it is often useful to:

  • Get a query response with profile turned on.
  • Gather nodes hot threads output with the query running in a while(true) loop. This will help understand where CPU time is spent.
  • Profile the query using this user-friendlier version of the profile API.

  If a query comes from a Kibana visualization, use the visualization spy panel (Kibana version 6.3 and earlier) or dashboard inspect panel (Kibana version 6.4 and later) to view and export the actual query request and import it into the profile API for further analysis.

  Catching slow or expensive queries

  Sometimes it can be hard to catch slow or expensive queries with different requests/threads being processed concurrently within a distributed application like Elasticsearch. Things get more complicated when there is no control over users running expensive queries that slow down the cluster performance (e.g., long garbage collection (GC) cycles) or worse, an out of memory (OOM) situation.

  In Elasticsearch version 7.0, we introduce a new circuit-breaking strategy that measures real heap memory usage at the time when memory is being reserved. This new strategy improves node resiliency against expensive queries causing cluster to be overloaded and is turned on by default and can be controlled with the new cluster setting indices.breaker.total.use_real_memory.

  However, we should note that these are best efforts; for scenarios that are not covered by these, it would be useful to collect heap dump after OOM crash or heap dump from a running JVM to better understand the root cause.

  Elasticsearch has another protection setting (max bucket soft limit) to safeguard the cluster against OOM. This max bucket aggregation setting stops execution and fails the search request when the number of buckets (defaults to 10,000 in version 7.0) is exceeded (e.g., when running multiple layers of aggregations).

  To further identify potential expensive queries, we can set circuit breaker setting (indices.breaker.request.limit) starting with a low threshold to isolate queries and gradually move up the threshold to narrow down to specific ones.

  Slowlogs

  Slow-running queries can also be identified by turning on slowlogs in Elasticsearch. Slowlogs works specifically on the shard level, which means only data node applies. Coordinating-only/client nodes are excluded as they do not hold data (indices/shards).

  Slowlogs help to answer questions like:

  • How long did the query take?
  • What was the content of the query request body?

  Sample slowlog output:

  [2019-02-11T16:47:39,882][TRACE][index.search.slowlog.query] [2g1yKIZ] [logstash-20190211][4] took[10.4s], took_millis[10459], total_hits[16160], types[], stats[], 
  search_type[QUERY_THEN_FETCH], total_shards[10], source[{"size":0,"query":{"bool":{"must":[{"range":{"timestamp":{"from":1549266459837,"to":1549871259837,"include_lower":true,
  "include_upper":true,"format":"epoch_millis","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"_source":{"includes":[],"excludes":[]},"stored_fields":"*","docvalue_fields":
  [{"field":"timestamp","format":"date_time"},{"field":"utc_time","format":"date_time"}],"script_fields":{"hour_of_day":{"script":{"source":"doc['timestamp'].value.getHourOfDay()",
  "lang":"painless"},"ignore_failure":false}},"aggregations":{"maxAgg":{"max":{"field":"bytes"}},"minAgg":{"min":{"field":"bytes"}}}}], id[]],
  

  Slowlog message breakdown:

  Item	Description
  [2019-02-11T16:47:39,882]	Query date
  [TRACE]	Log level
  [index.search.slowlog.query]	Query phase of search slow log
  [2g1yKIZ]	Node name
  [logstash-20190211]	Index name
  [4]	Shard number on the query executed
  took[10.4s]	Processing time taken on shard [4]. Note: when looking at slowlogs, we want to refrain from adding up all the times from different shards, as each shard may be executing in parallel.
  took_millis[10459]	Time taken in milliseconds
  total_hits[16160]	Total hits
  search_type[QUERY_THEN_FETCH]	Search type
  total_shards[10]	Total shards of the index
  source[]	The query body that was executed

  Audit logs

  Customers with a Gold or Platinum subscription, which includes Elastic security features, can turn on audit logs to capture more details about the query. (Users can start a 30-day trial to test drive Elastic security features.) Audit logging helps to answer questions like:

  • When did the query happen?
  • Who executed the query?
  • What’s the content of the query?

  We need to tune the audit settings, as the default ones are fairly chatty:

  1. Enable security audit log: Set xpack.security.audit.enabled: true in your elasticsearch.yml.
  2. Enable log or index in security audit outputs: Set xpack.security.audit.outputs:[logfile, index] in your elasticsearch.yml.
     
     Notes:
     • The xpack.security.audit.outputs setting only applies to version 6.0-6.2 and 5.x. Version 7.0 no longer accepts this setting and defaults to json output (<clustername>_audit.json) when xpack.security.audit.enabled is set to true.
     • For the purpose of troubleshooting, we recommend choosing logfile over index as the verbosity of audit logging may introduce unwanted stress to the cluster performance where the security index grows beyond its intended size.
     • Audit mode can be very verbose, so consider turning it off once troubleshooting is done.
  3. Include authentication_success access in your events list: Set xpack.security.audit.logfile.events.include: authentication_success in your elasticsearch.yml.
     
     Notes:
     • This setting is not included in the default events. Setting this will overwrite the default ones.
     • If you need to add one more event (not replace), please write the existing default event list first and then add the above setting after the last entry.
     • Output the request body in the audit events: Set xpack.security.audit.logfile.events.emit_request_body: true in your elasticsearch.yml.

  With these settings, you can monitor the user query like this.

  • User: louisong
  • Querying time: 2019-05-01T19:26:53,554 (UTC)
  • Query endpoint: _msearch (usually means it is a query issued from Kibana Visualization/Dashboard)
  • Query body: start from "request.body": in the following log:
    
    

    {"@timestamp":"2019-05-01T19:26:53,554", "node.id":"Z1z_64sIRcy4dW2eqyqzMw", "event.type":"rest", "event.action":"authentication_success", "user.name":"louisong", "origin.type":"rest", "origin.address":"127.0.0.1:51426", "realm":"default_native", "url.path":"/_msearch", "url.query":"rest_total_hits_as_int=true&ignore_throttled=true", "request.method":"POST", "request.body":"{\"index\":\"*\",\"ignore_unavailable\":true,\"preference\":1556709820160}\n{\"aggs\":{\"2\":{\"terms\":{\"field\":\"actions\",\"size\":5,\"order\":{\"_count\":\"desc\"},\"missing\":\"__missing__\"}}},\"size\":0,\"_source\":{\"excludes\":[]},\"stored_fields\":[\"*\"],\"script_fields\":{},\"docvalue_fields\":[{\"field\":\"access_token.user_token.expiration_time\",\"format\":\"date_time\"},{\"field\":\"canvas-workpad.@created\",\"format\":\"date_time\"},{\"field\":\"canvas-workpad.@timestamp\",\"format\":\"date_time\"},{\"field\":\"creation_time\",\"format\":\"date_time\"},{\"field\":\"expiration_time\",\"format\":\"date_time\"},{\"field\":\"maps-telemetry.timeCaptured\",\"format\":\"date_time\"},{\"field\":\"task.runAt\",\"format\":\"date_time\"},{\"field\":\"task.scheduledAt\",\"format\":\"date_time\"},{\"field\":\"updated_at\",\"format\":\"date_time\"},{\"field\":\"url.accessDate\",\"format\":\"date_time\"},{\"field\":\"url.createDate\",\"format\":\"date_time\"}],\"query\":{\"bool\":{\"must\":[{\"range\":{\"canvas-workpad.@timestamp\":{\"format\":\"strict_date_optional_time\",\"gte\":\"2019-05-01T11:11:53.498Z\",\"lte\":\"2019-05-01T11:26:53.498Z\"}}}],\"filter\":[{\"match_all\":{}},{\"match_all\":{}}],\"should\":[],\"must_not\":[]}},\"timeout\":\"30000ms\"}\n", "request.id":"qrktsPxyST2nVh29GG7tog"}
        

  
  

  Conclusion

  In this article, we talked about the common causes of slow queries and solutions to address them. We also discussed the different methods to identify consistent and occasional slow queries. It is common to see slow queries being a symptom of a wider cluster performance issue that needs to be addressed.

  At Elasticsearch, we are constantly seeking to improve query times and working to bring faster query performance with our future version releases. I hope you find this article useful when dealing with slow queries. Feel free to post questions in our Elasticsearch discuss forum for further discussion. Happy searching!

  
  
  https://www.elastic.co/blog/advanced-tuning-finding-and-fixing-slow-elasticsearch-queries
• 

  Improving the performance of high-cardinality terms aggregations
  • sorangutan  

  1
  1
  Posts
  590
  Views

  

  Introduction

  An Elasticsearch terms aggregation is used to create buckets corresponding to unique values of a given field. For example, a terms aggregation on a field containing country names would create a bucket for USA, a bucket for Canada, a bucket for Spain, and so on. Under normal circumstances, terms aggregations are very fast, however in some exceptional cases they may be slow. One reason for slow terms aggregations may be a mis-configured cluster. Another reason for poor performance may be high-cardinality values on the field which a terms aggregation is executed on.

  In this blog post, I’ll first give a brief overview of general instructions that should be followed to ensure the best performance of an Elasticsearch cluster. This is then followed by several sections that present background material that will help to understand the underlying mechanics of terms aggregations, including (1) a definition of high cardinality, (2) a description of the refresh interval, and (3) a description of global ordinals. Next, I’ll show how to view the impact of building global ordinals on terms aggregation performance. Finally, I’ll present several techniques to improve the performance of high-cardinality terms aggregations, including (1) time-based indices, (2) eager global ordinals, and (3) techniques to prevent Elasticsearch from building global ordinals.

  In one instance, the techniques documented in this blog post were able to reduce the execution time of a high-cardinality terms aggregation that was running at a very large retail bank, from 15 seconds to below 15 milliseconds.

  General suggestions

  Tuning a cluster can have a large impact on overall cluster performance. The following Elasticsearch documentation provides details on configuring and tuning an Elasticsearch cluster, and should be followed:

  • Important system configuration
  • Tune for search speed
  • Tune for indexing speed
  • Tune for disk usage

  Additional information on tuning slow queries can be found in this blog about advanced Elasticsearch tuning.

  In addition to the above, having too many shards is a common cause of performance problems. This blog about sharding gives good rules of thumb to follow.

  The remainder of this blog focuses specifically on understanding and tuning terms aggregations.
  

  Cardinality

  The performance of terms aggregations can be greatly impacted by the cardinality of the field that is being aggregated. Cardinality refers to the uniqueness of values stored in a particular field. High cardinality means that a field contains a large percentage of unique values. Low cardinality means that a field contains a lot of repeated values. For example, a field storing country names will be relatively low cardinality since there are less than two hundred countries in the world. Alternatively, a field storing IBAN numbers or email addresses is high cardinality since there may be millions of unique values stored.

  When discussing high cardinality in this blog post, we are referring to fields with hundreds of thousands or millions of unique values.

  Elasticsearch refresh interval

  In order to understand the remainder of this blog, we must have a general understanding of the refresh interval.

  As documents are inserted into Elasticsearch, they are written into a buffer and then periodically flushed from that buffer into segments. This flush operation is known as a refresh, and newly inserted documents are only searchable after a refresh. By default refreshes occur every second, however the refresh interval is configurable.

  The refresh interval is relevant to performance because in the background, Elasticsearch merges small segments into larger segments, and those larger segments are merged into even larger segments, and so on. Therefore, by enabling frequent refreshes, Elasticsearch needs to do more background work merging small segments than it would need to do with less frequent refreshes which would create larger segments.

  While frequent refreshes are necessary if near real-time search functionality is required for newly inserted data, such frequent refreshes may not be necessary in other use cases. If an application can wait longer for recent data to appear in its search results, then the refresh interval may be increased in order to improve the efficiency of data ingestion, which in turn should free up resources to help overall cluster performance.

  Global Ordinals

  Terms aggregations rely on global ordinals to improve efficiency. Global ordinals is a data-structure that maintains an incremental numbering for each unique term for a given field. By default, a terms aggregation relies purely on global ordinals to perform the aggregation at the shard level, then converts global ordinals to the real term only for the final reduce phase, which combines results from different shards.

  The performance of terms aggregations on high-cardinality fields may be slow and unpredictable in-part because (1) the time taken to build global ordinals will increase as the cardinality of the field increases, and (2) by default global ordinals are lazily built on the first aggregation that occurs since the previous refresh. Furthermore, a combination of frequent document insertions, frequent refreshes, and frequently executed terms aggregations would cause global ordinals to be frequently recomputed. 

  Additional details on global ordinal performance can be found in this GitHub issue.

  How to view the impact of global ordinals on terms aggregations

  Elasticsearch log files are very helpful for detecting performance issues. Set log levels to appropriate values, keeping in mind that excessive logging may increase disk IO and could negatively impact performance. Monitor the log file called elasticsearch_index_search_slowlog.log. If slow terms aggregations appear in the slowlog file, then their poor performance may be due to the building of global ordinals, which can be checked as follows:

  • Continue to insert data into Elasticsearch to ensure that global ordinals will be rebuilt when we execute a terms aggregation. Copy the slow terms aggregation from the slowlog file and manually execute and profile it, to get a feeling for where execution time is spent.
  • Continue to insert data into Elasticsearch. Copy the terms aggregation from the slowlog file and manually execute it. While the terms aggregation is executing, simultaneously execute the hot_threads api. If hot threads generally returns results that include references to GlobalOrdinalsBuilder, then the code may be spending significant time building global ordinals.
  • Temporarily enable logging of global ordinals information by executing the following command:
    
    

    PUT _cluster/settings
    {
        "transient": {
            "logger.org.elasticsearch.index.fielddata": "TRACE"
        }
    }
        

    This will write information about time spent building global ordinals to the elasticsearch.log file, such as the following:
    
    

    global-ordinals [<field_name>][1014089] took [592.3ms]
        

    Be sure to set the logging back to the default value after completing the above, as performance may be impacted while logging is set to TRACE.
  • The size of the global ordinals data structure for a given field can be seen by viewing memory_size_in_bytes when executing the indices stats command as follows:
    
    

    GET <index_name>/_stats/fielddata?fielddata_fields=<field_name>
        

  The above commands should give an idea if building global ordinals is consuming significant resources. The remainder of this blog focuses on steps to mitigate the impact of building global ordinals on terms aggregations.

  Use time-based indices

  Global ordinals only need to be re-created on a shard if that shard has been modified since the last computation of its global ordinals. If a shard is unmodified since the last computation of its global ordinals, then previously calculated global ordinals will continue to be used. For time-series data, implementing time-based indices is a good way to ensure that the majority of indices/shards remain unmodified, which will reduce the size of the global ordinals that need to be recomputed after a refresh operation.

  For example, if two years of data is stored in monthly indices instead of in one large index, then each monthly index is 1/24th the size that one large index would be. Since we are considering time-series data, we know that only the most recent monthly index will have new documents inserted. This means that only one of the 24 indices is actively written into. Since global ordinals are only rebuilt on shards that have been modified, the shards in 23 of the 24 monthly indices will continue to use previously computed global ordinals. This would reduce the work required to build global ordinals by a factor of up to 24 times when compared to storing two years worth of data in one large index.
  

  Enable eager global ordinals

  The performance of high-cardinality terms aggregations can be improved by eager building of global ordinals. Enabling eager building of global ordinals will create the global ordinals data structure when segments are refreshed, as opposed to the first query after each refresh. However, the trade-off is that eager building of global ordinals will potentially negatively impact ingest performance because new global ordinals will be computed on every refresh, even if they might not be used. To minimize the additional workload caused by frequently building global ordinals due to frequent refreshes, the refresh interval should be increased.

  Do not build global ordinals

  Another approach to improving the performance of high-cardinality terms aggregations is to avoid the building of global ordinals entirely, and therefore eliminate the execution time that would normally be spent computing global ordinals.

  In Elasticsearch 6.7 or newer this can be accomplished by specifying "execution_hint": "map" which tells Elasticsearch to aggregate field values directly without leveraging global ordinals, or in older versions this can be accomplished by using the alternate technique of executing a script inside a terms aggregation, which is described below.

  Be aware that if global ordinals are not built for a terms aggregation, the trade-off is that the terms aggregation itself (i.e. not counting the time that would have been used for building of global ordinals) will be less efficient. Additionally, the terms aggregation will use more memory per request because Elasticsearch will need to keep a map of all of the unique terms that appear in the result set. This could potentially trigger an internal circuit breaker to prevent memory overuse if a large number of unique terms are to be aggregated, which could result in a failed aggregation.

  Therefore, this approach should only be applied if a terms aggregation is expected to execute on a relatively small number of documents. For example, this would likely be the case if the aggregation is defined along with a selective bool query executing in filter context. Run an experiment with real data to determine if this approach improves the performance of a specific use case.

  By default a terms aggregation will return buckets for the top ten terms ordered by the number of documents in each bucket. Below is an example terms aggregation for top IBAN identifiers. By default this terms aggregation will rebuild global ordinals on the first execution after the last refresh.

  "aggregations": {
      "top-ibans": {
          "terms": {
              "field": "IBAN_keyword"
          }
      }
  }
  

  This could be re-written to avoid using global ordinals in Elasticsearch version 6.7 or newer as follows.

  "aggregations": {
      "top-ibans": {
          "terms": {
              "field": "IBAN_keyword",
              "execution_hint": "map"
          }
      }
  }
  

  In versions of Elasticsearch prior to 6.7, there was a bug that caused global ordinals to be computed even if "execution_hint": "map" was specified. This was fixed in this github pull request. For older versions, the following technique can be applied to prevent global ordinals from being built for the above terms aggregation. 

  "aggregations" : {
      "top-ibans" : {
           "terms" : {
               "script": {
                   "source" : "doc['IBAN_keyword'].value",
                   "lang" : "painless"
               }
           }
       }
  }
  

  Conclusion

  In this blog post I first presented a brief overview of documentation that should be followed to ensure the best overall performance of an Elasticsearch cluster. This was followed by a deep dive into terms aggregations including an overview of how they work and several options to improve their performance.

  If you have any questions about terms aggregations, Elasticsearch performance, or any other Elasticsearch-related topics, have a look at our Discuss forums for valuable discussion, insights, and information.

  
  
  https://www.elastic.co/blog/improving-the-performance-of-high-cardinality-terms-aggregations-in-elasticsearch
• 

  Life@ Elastic | Lessons I've Learned From a Distributed Workplace
  • sorangutan  

  1
  1
  Posts
  53
  Views

  

  I've worked distributed full time for about 1.6 years, but worked partly distributed a few years prior to that.

  I recently wrote a blog post on my personal website to share some of the things I like about working distributed. Given the success of that post, I decided to expand some of my thoughts on the benefits of working distributed with Elastic, and some of the ways to make the distributed experience work as smoothly as possible.

  First, let’s look at some of the things that I think make working distributed so wonderful.

  Number one: I love to travel.

  Anyone who knows me knows that I like to pack my bags and go.

  Naturally, my favorite kind of travel is going to see the people I love. You know how some people put pictures on their desks of their loved ones to inspire them? Well, I don’t need the picture as much because I get to see the real-life version by going and working with them in the same room.
  

  In fact, just this year alone, I spent over 100 hours in a plane or car, and have traveled over 17,500 miles. That might not be impressive to someone who regularly travels around the world for work, but these trips are purely domestic and mostly non-work related. Since I can work distributed, Elastic supports that I take flights in the middle of the day, spend my day in a coffee shop, or (my favorite) work from my best friend's couch in Hoboken, NJ.

  

  Me (on the right) and my sister in a coffee shop in Louisville.
  
  

  Number two: Distributed working lets me find when I'm most productive.

  When you're with a distributed company, you're working with a lot of time zones, and not all of them fit nicely in the 9-5 where you're based. While some may see this as a problem, I see it as a huge benefit because my work doesn’t have to squeeze into a specific time box. I can simply work when I'm most productive.

  You know that afternoon slump we all get from time to time (or every day)? Being distributed, I could sit at my computer and pretend to be productive. Instead, I can take a break when I actually need a break. I can take a walk outside, go grab a coffee to give my mind a break, or even take a nap so I can come back refreshed.

  Does flexible time mean I sometimes join meetings at 8pm at night? Yes, occasionally, but I don't mind because with great flexibility also comes great responsibility.

  

  My cat's paw prints can be seen in the newest Kibana build.

  Lastly: Distributed work is like getting a raise.

  Working from home lets me save money because I don't have to spend money on gas. In fact, I don’t actually need a car because I can work without driving anywhere (a privilege that a lot of people in the world don't have). I don't have to buy lunch during the day or worry about packing leftovers, because my kitchen is about a three-step commute from my desk.

  Doesn’t that all sound great? So, we've established why working distributed is amazing for me, but what does it mean for Elastic? Does me working distributed help them as a company? I think so, and here are a few things about working distributed that I believe are beneficial to Elastic ...

  Managers get to spend more time making strategic decisions and less time "baby-sitting" employees.

  Instead of having to monitor whether I'm sitting at my desk, my manager helps me strategically solve the problems and roadblocks in front of me. There's no expectation of making sure I'm "doing my work", because the results I'm producing, and not just the hours I'm sitting in front of my computer, speaks for itself.

  We communicate better at Elastic.

  Because we're distributed and often working with teammates in different parts of the world, we have a strong online communication culture. Calls take place on zoom, not around someone's desk. (And those calls are often recorded for anyone who missed it.) We chat constantly on Slack, and everyone is diligent about making sure that stakeholders are all on the same page.

  Being distributed is a competitive advantage over other employers.

  Elastic hires people at the top of their field, and these people are often getting offers from other, larger companies. Allowing employees to work distributed is a perk that many other employers haven't yet embraced, even though, as you've seen already, it's has so many benefits.

  Doing it right is hard work.

  Even though distributed work is fantastic for the employee and the employer, it doesn't mean that it's easy (or for everyone). It also doesn't mean that you're a good distributed company just because you let people work from anywhere.

  So how do you do it right, exactly? I’ve got some thoughts. First, I think it's critical to create an atmosphere where distributed workers can thrive, which is exactly what we do at Elastic. But I also think that what makes Elastic work so well relies so much on what our Elasticians bring to the table. So, here are some of the pro tips I’ve learned along the way for working well within a distributed culture: 

  1. Know yourself. How do you work best? Experiment a little to understand when you do your best work. For example, I'm better at my more creative tasks in the afternoon, as right now I'm sitting on an airplane typing this post in the middle of the day. Mornings I'm much better at more "manual" tasks that require less brainpower. I'm more analytical in the late afternoon/early evening hours.
  2. Communicate often. Don't work in a silo. Distributed work requires you to be more intentional with communication because it doesn't just happen automatically like when you're in the same room as all your coworkers. Make time to communicate. Understand others’ communication styles and cater to them. And for goodness sake, talk about more than just work. In an office, you never have 100% work focused conversations, so don't expect that online either. Make sure to discuss non-work things with your coworkers. You know, like the most recent Game of Thrones episode, the model plane you just built, or that new book you just read about power stances.
  3. Go see people. One of the things I love about Elastic is that pretty much anywhere I travel to, I can find another Elastician there too. I love getting face time with others in the company, whether it's the people I work with every day, or ones I don’t work with closely but we’re still contributing to the same goal - helping people learn about Elastic and use our products and solutions to solve their problems. Nothing can replace being in person with people, so with the money your company saves by not paying for an office, they should be happy to spend a small portion of that to help you spend time with other coworkers. Teams at Elastic have regular offsites, and we also meet up every year at our Global All Hands.
  4. Don't assume malice. This is part of our Source Code, and one of the biggest pieces to making distributed work ... well ... work! It's easy for meanings and intentions to get lost in digital communication. For example, someone says something on Slack that rubs you the wrong way, or someone jumps up with a question on Zoom during your presentation that feels confrontational. You don't have to assume the world is full of butterflies and everyone is out to make your life better, but it will sure be easier if you decide to not assume malice.
  5. Be respectful. Keep time zones in mind when scheduling calls and do your best to keep meetings within a person's work day. Try to stay out of noisy environments when you know you've got a day full of meetings you'll be talking in. Think about what it's like to be on the other end, and you'll make sure to provide the best experience for everyone.

  

  Here I am out and about with my fellow Elasticians.
  
  

  Does distributed work sound like the right fit for you? Why not check out our teams and see if there's a fit for you! You’ll be glad you did.

  
  
  https://www.elastic.co/blog/life-elastic-lessons-ive-learned--from-a-distributed-workplace
• 

  New Elasticsearch JavaScript client released
  • sorangutan  

  1
  1
  Posts
  72
  Views

  

  After a month of active development, gathering feedback and introducing new features, we are thrilled to announce the GA release of the new JavaScript client!

  Wait, a new JavaScript client? Yes! We announced the RC1 a few weeks back, and now we’re production-ready.
  

  Try out the Elasticsearch JavaScript client

  First, install the client and run Elasticsearch. You can install Elasticsearch locally with our docker image.

  npm install @elastic/elasticsearch
  

  Then create a JavaScript file (TypeScript is supported as well!) and paste inside the following snippet:

  'use strict'
  const { Client } = require('@elastic/elasticsearch')
  const client = new Client({ node: 'http://localhost:9200' })
  async function run () {
    // Let's start by indexing some data
    await client.index({
      index: 'game-of-thrones',
      body: {
        character: 'Ned Stark',
        quote: 'Winter is coming.'
      }
    })
    await client.index({
      index: 'game-of-thrones',
      body: {
        character: 'Daenerys Targaryen',
        quote: 'I am the mother of dragons.'
      }
    })
    await client.index({
      index: 'game-of-thrones',
      // here we are forcing an index refresh,
      // otherwise we will not get any result
      // in the consequent search
      refresh: true,
      body: {
        character: 'Tyrion Lannister',
        quote: 'A mind needs books like a sword needs a whetstone.'
      }
    })
    // Let's search!
    const { body } = await client.search({
      index: 'game-of-thrones',
      body: {
        query: {
          match: {
            quote: 'winter'
          }
        }
      }
    })
    console.log(body.hits.hits)
  }
  run().catch(console.log)
  

  What’s new?

  In addition to all the features added with the new client, in the past month we have listened to your feedback and added new cool features. Some recently-added features are observability, support for sniffing hostnames, improved the type definitions, and the support for a custom HTTP agent. We also vastly improved our JS client documentation and decreased the size of the library.

  Observability

  Thanks to the new observability features now it will be easier to connect the dots between events. In every event, you will find the id of the request that generated the event, as well as the name of the client (which will be very useful if you are working with child clients).

  Below you can see all the data that is exposed by our observability events:

  body: any;
  statusCode: number | null;
  headers: anyObject | null;
  warnings: string[] | null;
  meta: {
    context: any;
    name: string;
    request: {
      params: TransportRequestParams;
      options: TransportRequestOptions;
      id: any;
    };
    connection: Connection;
    attempts: number;
    aborted: boolean;
    sniff?: {
      hosts: any[];
      reason: string;
    };
  };
  

  Want to know more? Check out our observability documentation!

  Type definitions

  The client offers first-class support for TypeScript, it offers type definitions for its entire API and the parameter of every method. The type definition of the request and response bodies are not yet supported, but we are working on that!

  In the meantime, we have vastly improved the developer experience by using generics for all the body definitions.

  import { RequestParams } from '@elastic/elasticsearch'
  interface SearchBody {
    query: {
      match: { foo: string }
    }
  }
  const searchParams: RequestParams.Search<SearchBody> = {
    index: 'test',
    body: {
      query: {
        match: { foo: 'bar' }
      }
    }
  }
  // This is valid as well
  const searchParams: RequestParams.Search = {
    index: 'test',
    body: {
      query: {
        match: { foo: 'bar' }
      }
    }
  }
  

  You can find a complete example in our TypeScript documentation.

  Conclusion

  We’re excited about the new JavaScript client, and we hope you are too. Try it out today, locally or on the Elasticsearch Service, and let us know what you think. If you want to know more, you can open an issue in the client repository, post a question in Discuss, or ping @delvedor on Twitter. Happy coding!

  
  
  https://www.elastic.co/blog/new-elasticsearch-javascript-client-released
• 

  Distance Feature Query: Time and Geo in Elasticsearch Result Ranking
  • sorangutan  

  1
  1
  Posts
  105
  Views

  

  In Elasticsearch 7.2, you will be able to use proximity (geographical or time) as part of your ranking score. The new way for combining proximity within the ranking score is easy to configure, performs well, and will likely yield superior ranking in many scenarios. This blog explains why and how.

  Why use geographical distance and time for ranking?

  Consider a search for a restaurant. One typically uses a criteria, including type, price range, rating, as well as physical proximity. All other things kept equal, you would prefer the nearest restaurant. This is not unique — it would be the same in countless other relevance ranking scenarios e.g. when searching for a job, a date, a plumber, a dog-walker, a babysitter, a tourist attraction. In fact, relevance ranking for anything in which physical location matters — which is almost anything that exists in the physical world — would benefit from relating to geographical distance.

  A similar situation occurs with time related search, e.g. when searching for an academic article, a newspaper article, a blogpost or a product review, one will typically prefer recent records over less recent records (assuming all other aspects are equal). As with physical proximity in search for things that have physical location in the real world, recency would be the typical preference in searches for information. In other words, almost any relevance ranking criteria would benefit from the inclusion of proximity, either date proximity or geographic proximity.

  We have frequently seen date and geographical proximity used in either of the following ways:

  1. As a filter, i.e. dichotomic criteria, e.g. retrieve only records nearer than x or within the last y days, and rank these records by other search criteria (e.g. using BM25 on text)
  2. As the only ranking criteria, e.g. sort all the records that answer a dichotomic criteria by their geographical or date proximity

  In other words, proximity is either used as a retrieval criterion, rather than for relevance ranking, or is used as a single criterion for relevance ranking.

  The problem is that the user is often willing to walk a longer distance for a better ranked restaurant, wait a little longer for a better ranked babysitter or read the most cited article even if it is not the most recent. In other words our actual search criteria as searchers combines proximity (space and time) with other criteria (such as word frequency related criteria) into a single relevance score.

  Can’t you do it with Function Score or Painless script?

  Picking our other hat, as the people designing, tuning and programming search algorithms, we frequently avoided such a combined proximity and word frequency (or various other numeric) ranking criteria because it was technically challenging. There are two main challenges:

  1. Normalization - The proximity score has to be normalized so that it will have the right impact when summed together with other relevance scores, such as BM25. The default in Lucene is to simply sum the various scores into one number and in most cases it makes sense, as long as the different scores have been normalized first. The normalization itself, however, is not trivial.
  2. Performance - Re-ranking with proximity took a significant performance toll. To deal with it, the more sophisticated Elastic users frequently formed a criteria that did not include proximity to select a very small subset of most relevant records, and then reranked only these records after taking proximity into account. This compromise means the algorithm will not consider results that could become the most relevant ones after proximity is taken into account.

  We want to assist our users with non-trivial algorithmic challenges, so we sprung into action. In version 7.2 we will enable users to rank all records with a combination of the ‘normal’ ranking score and a normalized proximity (time or space) score.

  So can you add proximity to the ranking algorithm using Function Score or Script Score? It depends. It is possible to relate to proximity and to normalize it with function score and Painless script even prior to 7.2, but only on a small subset of the result set, and with a significant performance toll. Sometimes these quantitative differences sum up to a qualitative difference: with 7.2, ranking by proximity in combination with other relevance ranking algorithms, becomes the reasonable principle ranking mechanism in many scenarios.

  Saturation Normalization Function

  For the normalization, we used the concept of the Saturation normalization function. The concept behind the Saturation normalization function is similar to the one used in BM25 to relate to the length of the text in the document in setting the ranking score for that document. A record’s score will be computed using the following formula: $\mathrm{Score}=\mathrm{Boost}*\frac{\mathrm{Pivot}}{\mathrm{Pivot}+\mathrm{Distance}}$. Assuming $\mathrm{Pivot}>0$, the function $\frac{\mathrm{Pivot}}{\mathrm{Pivot}+\mathrm{Distance}}$ is a nonlinear strictly decreasing function starting from $1$ when $\mathrm{Distance}=0$ (assuming $\mathrm{Distance}>=0$) with ${\lim }_{\mathrm{Distance}\to \infty }\frac{\mathrm{Pivot}}{\mathrm{Pivot}+\mathrm{Distance}}=0$. One way to think of the $\mathrm{Pivot}$ parameter is that it defines the distance at which $\frac{\mathrm{Pivot}}{\mathrm{Pivot}+\mathrm{Distance}}=\frac{1}{2}$but another way to think of it is that it defines how quickly the contribution of distance to the score diminishes. The boost parameter is an optional parameter that by default is set to $1$, and allows to set the influence of distance on the overall score in a way that is reduced linearly to the distance, i.e. it is a way to indicate how important will distance be in the overall score, regardless of the specific area of the scale a record is in.

  

  The above is the function plot for the function $\mathrm{Score}=\frac{\mathrm{Pivot}}{\mathrm{Pivot}+\mathrm{Distance}}$ with $\mathrm{Pivot}$ set to $2$ and $\mathrm{Distance}$ plotted on the X axis.

  The bottom line is that by providing one mandatory parameter ($\mathrm{Pivot}$) the user gets a normalized score, ranging from $1$ to $0$ (given that our computers are discrete and have a ‘highest number’ that functions as infinity in this case). This score allows distance to impact more when the distance is small and less when the distance is large, which is how people typically want distance to impact.

  Better performance enabling better ranking

  In theory, all of this could be done prior to 7.2 using re-ranking through Painless script. The addition of Distance Feature Query simplifies it considerably and opens the door for those who are less proficient with Painless, which is important, but not the main benefit. The main issue is that in practice the performance toll frequently meant that the proximity impact on the ranking score could only be calculated for a tiny fraction of the records. The implication was that it was quite likely that the most relevant records after the introduction of the proximity score would not be in the set for which this impact would be calculated.

  The Distance Feature Query performs better than the Painless script, but that’s not the big news. The really big news is that the distance feature query was designed to leverage the performance improvements resulting from the faster top-k queries (see Magic WAND: Faster Retrieval of Top Hits in Elasticsearch blog post). Those who can afford to get a lower boundary for number of results instead of an exact count can enjoy a potentially unbounded improvement in performance because the algorithm will disqualify at an early stage all records that cannot be one of the top K candidates.

  Examples

  Let’s plan an application for hotel reviews. A core functionality is to allow searching for reviews about a hotel. The user will want to search for the specific hotel, and to get reviews about aspects that are of interest to the user in the hotel as well as being comprehensive and as recent as possible.

  Say our record structure is as follows:

  PUT reviews
  {
    "mappings": {
      "properties": {
        "hotel": {
          "type": "text"
        },
        "review": {
          "type": "text"
        },
        "date": {
          "type": "date"
        }
      }
    }
  }
  PUT reviews/_doc/1
  {
    "hotel": "Fawlty Towers",
    "review": "The hotel would have been managed in a perfect way, if it wasn't for the guests",
    "date" : "2017-12-27"
  }
  PUT reviews/_doc/2
  {
    "hotel": "Fawlty Towers",
    "review": "The restaurant is, well, interesting",
    "date": "2018-07-07"
  }
  PUT reviews/_doc/3
  {
    "hotel": "Fawlty Towers",
    "review": "We had an interesting vacation there. The service at the restaurant is a great chance to brush the dust off your Spanish. Mr. Fawlty himself had recently moved to The Life of Brian.",
    "date": "2019-03-25"
  }
  GET reviews/_search
  {
      "query": {
          "bool" : {
              "must" : {
                  "match" : {"hotel": "Fawlty Towers"}
              },
              "should" : [
                  {"distance_feature" : {"field" : "date","pivot" : "30d","origin": "now"}},
                  {"match" : {"review": "restaurant"}}    
              ]
          }
      }
  }
  Results:
  {
    "took": 5,
    "timed_out": false,
    "_shards": {
      "total": 1,
      "successful": 1,
      "skipped": 0,
      "failed": 0
    },
    "hits": {
      "total": {
        "value": 3,
        "relation": "eq"
      },
      "max_score": 1.5571662,
      "hits": [
        {
          "_index": "reviews",
          "_type": "_doc",
          "_id": "3",
          "_score": 1.5571662,
          "_source": {
            "hotel": "Fawlty Towers",
            "review": "We had an interesting vacation there. The service at the restaurant is a great chance to brush the dust off your Spanish. Mr. Fawlty himself had recently moved to The Life of Brian.",
            "date": "2019-03-25"
          }
        },
        {
          "_index": "reviews",
          "_type": "_doc",
          "_id": "2",
          "_score": 1.0365787,
          "_source": {
            "hotel": "Fawlty Towers",
            "review": "The restaurant is, well, interesting",
            "date": "2018-07-07"
          }
        },
        {
          "_index": "reviews",
          "_type": "_doc",
          "_id": "1",
          "_score": 0.32892755,
          "_source": {
            "hotel": "Fawlty Towers",
            "review": "The hotel would have been managed in a perfect way, if it wasn't for the guests",
            "date": "2017-12-27"
          }
        }
      ]
    }
  }
  

  The query filters to show only records in which the hotel is “Fawlty Towers”. Within these records, the ranking is based on the word restaurant, which two records contain. By BM25 only, record 2 would be ranked higher than record 1, just because both records include the word “restaurant” once and the review in the second record is longer than in the 3rd one (the first record does not include the word restaurant and so would be ranked lowest). However, due to the time distance query, the 3rd record would show up first, and the user will get the latest update about Fawlty Towers.

  The distance feature query skips non-competitive records and through that achieved better performance, which allows running the combined relevance criteria that includes both time proximity and BM25 on natural language.

  Conclusion

  Proximity in either geographical space or time is an important aspect of almost every relevance ranking use case. Adding distance ranking to the relevance ranking, together with the normalization capability, will for sure prove useful in real life scenarios, and represents our long term strategic investment in relevance ranking.

  Just as important is the full story of how this capability came about. It’s all started with a user from the community that came up with a suggestion to incorporate the MAXSCORE algorithm into Lucene. That original suggestion, although it was not accepted to Lucene in its original form, later led to our contribution of the block-max WAND algorithm to Lucene, and more generally introduced APIs that help compute top hits without evaluating all matches. This in-turn led to the development of Lucene's FeatureField for static features — exposed in Elasticsearch via the rank_feature and rank_features fields — and now feature queries for dynamically computed distances. This cycle of contributing and exposing the code we produce so that the community can come up with feedback and ideas, helps steer us towards new, useful and innovative directions.

  We’re excited about this new feature, and we hope you are too. So for now, read more about the distance_feature query, and once the release candidate of 7.2 is available, download it and take it for a spin. Once you’ve had a chance to test this new geo-ranking feature out, give us your feedback and comments about how you want to use distance feature query. Your input will help direct us in our future development.

  
  
  https://www.elastic.co/blog/distance-feature-query-time-and-geo-in-elasticsearch-result-ranking
• 

  Introducing the Elastic Enterprise Search Beta: Search Everything, Anywhere
  • sorangutan  

  1
  1
  Posts
  76
  Views

  

  Today we are pleased to announce Elastic Enterprise Search, a new product that allows teams and organizations to search all the data scattered across the many tools that power the productivity of their workforce.

  Across all industries, team members now enjoy access to an increasing number of purpose-built, cloud-based tools that are more focused and easier to use than ever before. This enables new forms of collaboration, faster execution and an explosion of productivity for teams both large and small. This proliferation of content has created tremendous value, and it has also created a new opportunity: the chance to unify all information in one place and make it easily searchable from a single search box. Elastic Enterprise Search makes that possible, and it is now easier than ever before.

  

  Keeping the Simple, Simple

  Elastic Enterprise Search is a consumer grade enterprise search solution. It provides easy to use, powerful search across all the tools used throughout your organization. It's a fast, scalable, and relevant search bar for your everyday work life.

  Building on Elasticsearch's proven relevance and scalability, we've designed a search experience that simply requires you to search the way you'd speak to a colleague, rapidly getting you to the document or answer you need. No need to configure any elaborate applications, traverse through complicated interfaces, or learn any strange conventions: Elastic Enterprise Search understands your intention and lets you remain focused on the task at hand.

  

  Advanced keyword detection

  Say farewell to the anxious days where multiple browser windows held hundreds of open tabs. Search all of your documents, pull requests, issues, tickets, contracts, spreadsheets — whatever it is, wherever they are, you have a single, organized gateway to getting things done.

  Well Connected

  Elastic Enterprise Search comes stock with cloud application connectors that get you started in just a few minutes. For this first beta release, we've focused on the cloud applications that are most commonly used in today's modern organizations:

  • Google Drive: search over all of your docs, sheets, slides and stored files
  • GitHub: ingest all of your issues and pull requests
  • Salesforce: sync accounts, contacts, leads and opportunities
  • Dropbox:  index all of your images, static files and documents
  • Custom Connector Framework: sync data from any source, like an internal application, analytics engine, private communication channel or knowledge base

  

  Relevance, Together

  Enterprise Search provides privacy groups and tunable search relevance so that the right things are found by the right people. Assign each team member to a group: Engineering, Sales, Marketing, Design, Product, Leadership — whatever best fits your organization — and calibrate their sources' relevance, based on their specific needs:

  

  Adjust the significance of GitHub for your Engineering team, Salesforce for your sales team, Dropbox for your design team; whatever fits. With granular access controls and group assignments, you can securely guide what will be found by individuals and groups. And what will not be found…

  

  Elastic Enterprise Search packs all the greatness you've come to expect from an Elastic solution: simple data indexing, seamless scalability, and blazing speeds coupled with highly tuned relevance algorithms, search history, autocomplete, language analysis, typo-tolerance and much more. All that backed by the well documented, community-focused expert support of Elastic.

  Ready to get started?

  Read on to join the beta or sign up for the webinar to learn more.

  Run, Search

  Elastic Enterprise Search requires...

  • Java 8 or Java 11.
  • An Elasticsearch 6.7.1 cluster.
  • An Elasticsearch premium license.
    • Generate a 30 day trial premium license during the set-up process.

  Download Enterprise Search

  Decompress the package, then enter the directory.

  Adjust the configuration files of either Elastic Enterprise Search or Elasticsearch to permit automatic index creation. You can do that in one of two ways -- choose whichever feels best:

  1. Enterprise Search: Open config/enterprise_search.yml, add the line: allow_es_settings_modification: true.
  2. Elasticsearch: Open config/elasticsearch.yml or within your Elasticsearch cluster settings, add the line: action.auto_create_index: ".ent-search-*-logs-*,-.ent-search-*,+*"

  And then run the binary ~

  bin/enterprise-search
  

  Last step: access http://localhost:3002, login using the default credentials, change them, then weave together all of your organizational data. Add your team members, tune group relevance, and then watch your workflows transform through search.

  Note: Beta1 is an early sneak peek. It's unencrypted by default and not fit for production usage. 

  We'd Love Your Beta Feedback

  We'll need your help to make Elastic Enterprise Search the best product it can be. If you run into any bugs, have any comments or questions, please reach us at: [email protected].

  
  
  https://www.elastic.co/blog/elastic-enterprise-search-beta1-released
• 

  Keeping up with Kibana: This week in Kibana for May 3rd, 2019
  • sorangutan  

  1
  1
  Posts
  108
  Views

  

  Releases

  We released some nice bug fixes in 6.7.2 and 7.0.1.  Read about them in the release notes Kibana 6.7.2 and Kibana 7.0.1.

  Hiring
  

  • Director / Sr Director Product Management - Kibana
  • Kibana - UI/Front End Team Lead
  • Kibana Platform - Principal JavaScript Engineer
  • Kibana - Senior JavaScript Engineer
  • Kibana - Senior Security Developer (JavaScript)
  • Kibana - Platform Javascript Engineer (Node.js)
  • Kibana - Software Engineer - Operations
  • Kibana - Visualisations & Vega Engineer
  • Kibana - Senior Product Designer

  Security
  

  Feature Controls

  We’ve continued working on the documentation for feature controls, and have started converting to asciidoc. We’re also introducing a saved object management feature, as opposed to implicitly granting access to saved object management based on your existing privileges to saved objects. This will allow us to specify which users have access to import/export all saved objects, and hide this screen for those who shouldn’t have access.

  Secret Service

  The PR introducing encrypted saved object attributes completed review this week and merged to master!

  Kerberos

  Work on Kerberos resumed this week, stay tuned for more exciting updates!

  Platform

  New Platform

  Work continues on building support for the security plugin and application service.

  The `start` lifecycle hook has been added to the server-side core and plugins to match client-side (#35793) which was a blocker for the security plugin migration. Modifications to the BasePathService are pending review (#35951) to unblock Spaces as well.

  The URL encoding fix for Angular has been merged (#34300) which paves the way to building the Application Service router in the new platform. Re-working how UI Capabilities are fetched by the client is nearing final review (#35194), this unblocks supporting feature controls in the new platform. The new platform replacement for kfetch, HttpService, is in review (#35486).

  Cleanup efforts are nearing completion, with a review pending to cleanup interface consistency for core services (#35733).

  Alerting Service

  This week we continued the development of the connector and action services. An API has been created to do CRUD operations on actions and an API to list connectors. We started developing a slack and email connector that will be available by default.

  Telemetry

  A blocker for the 6.7.2 and 7.0.1 releases was merged (#34609) that fixed how frequently Kibana was polling Elasticsearch for telemetry data when monitoring was also enabled.

  Work continues on pulling telemetry out of xpack_main into the new platform (#35403) as well as encrypting telemetry data before sending to Elastic’s telemetry cluster (#33121).

  Operations

  In #35586 we are now bootstrapping the native real accounts and using the ‘kibana’ user by default in development. This should hopefully eliminate permission based issues. If you would like to change the password, this can be done through kbn-es, use `--help` for more information.

  Add exception to transpile normalize-url mode to es5 when bundled into the DLL #35804

  [kbn-es] Set password for native realm accounts #35586

  Upgrade jQuery to 3.4.1 #35908

  Bump elasticsearch-js to 15.5.0 #35910

  Geo-Maps App

  Geojson upload [IN PROGRESS]

  • POC is ready. UX under review (#34748)
  • We are extracting the file-upload backend in a new plugin, and this will be merged separately and blocks the geojon upload feature. This is going through review now as well (#32945)

  Dashboard embeddability [COMPLETED]

  Filtering [IN PROGRESS]

  • Shape filtering PR in review #33686

  

  Misc

  • Usable legend coming, likely 7.2. This adds a host of improvements to legends, including readouts of dynamic values for all styling properties, and better preview icons (#35417). This included some preliminary refactoring work (#35763, #35893)

  

  • In progress:
  • Investigated performance issue with large files.
  • Improvements to Elasticsearch-result decoding (#36062) and to how tooltips load content (#36059) will significantly speed up loading of large data-files. Reviews in progress.
  • Merged bug fixes
  • Improve speed of layer selection and UX Toc control (#35905, #35751)
  • Clear tooltips when mouse exits map (#35825)
  • Do not allow mouse-scroll on dashboards (#35811)

  Geo-Elastic Maps Service

  Additional basemap styles [in progress]

  This week we updated the tile service manifest to include additional styles. [ems-file-service/#99]. However this has been since bumped to another release. Work continues on updating the EMS client in Kibana to handle the new manifest and styles.

  App Architecture

  New Platform Migration

  We finished moving the search and query bars into the data plugin. We're currently working through the filter bar, including an ambitious project to remove (most of) Angular from it. We're looking at some of our commonly-used registries like VisTypes, and working out how we can prepare these for migration. And also working to migrate key pieces of infrastructure from ui/vis to the new visualizations plugin.

  Embeddable API Phase 1 PR updates:

  • It’s in PR form!  Not in review yet though.

  • Tests converted from snapshot matching to being more explicit since snapshots ended up adding 100k LOC! Due to internationalization, rxjs observables, and embeddable -> parent relationships totally blowing up the size of the props passed down.
  • New functional tests added.
  • Code coverage
  • Went down slightly for embeddable_api, but we did add some more tests to cover actual bugs we encountered, they just didn’t increase the %
  • 74.22% Statements 380/512
  • 60.16% Branches 148/246
  • 73.66% Functions 137/186
  • 74.21% Lines 374/504
  • Went up for dashboard_embeddable!
  • 81.97% Statements 150/183
  • 66.3% Branches 61/92
  • 75.41% Functions 46/61
  • 81.25% Lines 143/176

  Kibana App (Visualizations, Sharing, Discover, Reporting)

  Lens (formerly New Visual Editor)

  This week we decided on a name for the editor, which is Lens. We worked through open questions about the architecture, specifically focused on the contract between Datasource and Visualization authors. Initial commit. We also fixed remaining UX issues on the branch that we had developed for user testing. The cleanup work is nearing an end, so we will soon focus the entire team on writing high-quality code for the various parts.

  

  Elastic Charts

  This week we continued to support teams with integration of elastic-charts.  Some bug fixes from this week in response to issues filed by consumers of the library include:

  • use bisect to handle invertWithStep (PR #200) [this fixes issues with crosshairs & tooltips not rendering correctly for discontinuous series data]
  • improve ticks for time domains spanning a DST switch (PR #204)

  We also introduced a couple of features:

  • add paddings between bars (PR #190)
  • add option to toggle value labels on bar charts (PR #182)
  • add RectAnnotation type (PR #180)

  We also met this week with team members from SIEM and Uptime to better understand their charting needs.  We are also working on replacing the chart in Discover with elastic-charts.

  Canvas

  Postcards from the airport

  Our friends at Crimson Macaw gave us a little tech preview of what they are working on over at the technically complex, complex which is the Manchester Airport at Stansted.  They have created one workbook which is designed to be bulletproof against accidental changes that has ~1500 lines of Canvas Expression code. They use Atom externally to edit, and then paste back into Canvas, but never fear we are planning some Expression Editor (and SQL Editor) updates soon.  Any suggestions or thoughts?

  Here is what they just launched

  

  “One two one two, this is just a test”  -- Beastie Boys

  The Design team created a first go of Stand out with Canvas, a video which gets users started using Canvas, and is designed to compliment our other efforts with Getting Started videos, and showing new users how to get moving fast with our Elastic solutions.  Take a peek, and let us know what you think.

  

  Expanding Content for Users

  A PR for element templating, which is a big step towards giving users the freedom to create their own templates easily, and opens the door to even more creative possibilities within Canvas at an even faster rate, was checked in.

  Pages, and pages, and pages, and...

  Today on a call with Crimson Macaw, we spoke of features they wanted to see, and discussed the soon to be completed and oft requested -  Kiosk Mode (KM). One of the slick features we plan on for KM is being able to rotate through the pages of a Canvas automagically, and we plan on highlighting the competitive advantage of a multi page dashboard in many locations in the coming short while.  Hint: Stay tuned to your local office displays.

  

  Design

  Color Picker refactor coming to EUI

  We have a PR up for a ground up rebuild of our colorpicker. The previous version used a popular npm solution that was extremely bulky and not accessible. The new one is written from scratch, accessible and very lightweight. We’re hoping to get this one in soon and close out a handful of accessibility issues. At the moment it looks like it should be a non-breaking change and require no changes to downstream apps.

  

  Selectable cards now available in EUI / Kibana

  We merged a PR that allows cards to show a selectable state.

  

  Kibana Maps cleanup

  This weeks cleanup had us in the maps app trying to work on a slight redesign of the maps legend display. The newer version gives some better indication of the content of each layer.

  

  EUI TypeScript conversion progress

  EUI is making good progress converting to TypeScript due to some help from the community. Contributor Theodesp has merged 15 PRs towards that goal over the last month.

  Kibana guides / new homepage progress

  We’re continuing to make strides on the Kibana guides / new homepage proof of concept. We’re up to 4 videos total with the newest being an introduction to Canvas. We hope to have 5 before soon with the hope that the new homepage can debut in an upcoming minor release.

  

  
  

  
  
  https://www.elastic.co/blog/keeping-up-with-kibana-2019-05-03
• 

  Happy to be Here: Elastic in Top Three of Highest-Rated Cloud Companies to Work for in 2019
  • sorangutan  

  1
  1
  Posts
  66
  Views

  

  Elastic was just named the number three public company by Battery Ventures in their annual Highest-Rated Cloud Companies to Work For list. We are proud and humbled to be highlighted with other private and public B2B-focused cloud companies. Battery uses Glassdoor data to compile these rankings based on company ratings submitted by employees, so many thanks to our Elasticians all around the world!

  

  From our Equal Pay Initiative to all the great things about working at Elastic, we’re continuously innovating and working to build a company that draws the best talent from around the world and keeps them here. We will continue putting in the time and effort to not only have a distributed company that’s culturally diverse and upholds our Source Code, but also acknowledges that this type of recognition and success comes directly from our amazing Elasticians!

  Interested in becoming an Elastician yourself? Jump over to our teams page and learn more about which might be the right fit for you!

  

  
  
  https://www.elastic.co/blog/happy-to-be-here-elastic-in-top-three-highest-rated-cloud-companies-to-work-for-in-2019