This Week in Elasticsearch and Apache Lucene - 2019-01-19

sorangutan

Elasticsearch Highlights

Leniency in request handling

We are trying to remove leniency throughout Elasticsearch. It makes for a more predictable experience for our users, where Elasticsearch does what the user thinks they are trying to do, or otherwise tells them when something seems amiss. An example we have recently decided to address is HTTP requests with a body, where the endpoint doesn’t actually consume the body. We have recently merged a change to reject HTTP requests with a body when the endpoint doesn’t use the body. This is a breaking change that will ship in 7.0.

Watcher

We added a whitelist to the HttpClient that is used by Watcher. This allows admins to allow only specific URL's to be called from Watcher. For passivity this defaults to all, but in future versions will require this to be explicitly set. (#36817)

Index management

We worked on a couple of PRs adding support for various index states to the index management UI. One added marking indices with a "Rollup" badge when they are created by a rollup job. We also added support for frozen indexes to index management, including freeze/unfreeze actions in the context menu and a "Frozen" badge in the index list.

Index Lifecycle Management

We submitted PR for the first system level index, .watch-history, to be managed by ILM. This lays the groundwork for a future system level index to be managed by ILM. The general strategy is to ship a default, but allow pre-existing policies, or changes to the cluster's policy such that we only apply the default policy if one does not already exist. The pre-existing watcher history cleaner service will be deprecated and removed. (#37443)

Performance

We are happy to see that our benchmarks have served as a good safety net performance regressions. This week we found a significant performance regression in the interaction between types removal in the bulk API and the deprecation logger. There is an intermediate fix for this and we have opened a follow-up issue to optimize deduplication in the deprecation logger.

Speaking of deprecation, We have raised 2 PRs to address some performance issues uncovered in this week’s FixIt meeting whilst discussing this issue.

Permissions to access restricted indices

The .security index has long had restricted access; in 2.x no external access was allowed (originally) and in 5.0+ access was restricted to superusers. This was done to avoid having users with permission to read all indices suddenly have permissions to read security information such as password hashes when the native realm was introduced. While thinking about backing up the .security index, it became clear that superuser was too powerful of a role to be using for backups. Albert has embarked on lightening this restriction and we've gone through a few iterations, but have landed on a new field with the indices section of a role called allow_restricted_indices that defaults to false. As of now, the only restricted indices are the security indices but we've kept this generic so that we have freedom to address this in the future.

TLS

We continue to work on the custom TLS configuration for reindex by making use of the new library he added last week. Additionally, Tim will be removing TLSv1.0 from the default list of protocols for 7.0as this protocol has known security weaknesses. We have worked on a new TLS tutorial that covers encrypting internode communications. We will be working to update this tutorial to cover the new Zen2 discovery and expand the tutorial to also include setting up HTTPS.

Total hits accuracy in Elasticsearch 7.0

Following the response format change for search requests that allowed us to return the total hits that match a query as a lower bound, we opened a PR that will change the default for all search requests to count the total hits accurately up to 10,000. This means that any search request that matches more than 10,000 documents will now return:

{ "hits": { "total": "10000", "relation": "gte" } }

This will be the behavior by default in 7 and beyond, "track_total_hits" must be set to true explicitly in the request to get the old behavior back.

Alternative CSS execution

We opened a PR to handle the merging of search responses coming from different clusters. In the alternative cross cluster search the reduction happens per shards inside each cluster first and then the responses of each cluster (the local one + the remotes) are merged to form the final response. We are also already working on the logical follow up which is the final integration of the alternative mode in the search layer.

Geo

We merged the minimal geo objects library into master, and are working on integrating it with the JDBC driver. Igor is also continuing to diagnose the performance problems on the new geoshapes rally test. The current theory is that the slowdown is caused by increasingly slow merging of ever-larger segments.

Cross Cluster Replication

We continue work on shard history retention leases, having added persistence to Lucene on commits, and opening a PR to sync retention leases to replicas This has triggered additional discussions about the durability of these leases and the guarantees that we want from the storage and replication system.

We added a new “CCR follow info" API that reports information about all the follower indices in the cluster together with their parameters and settings from the time where the following got initiated or resumed. We have also adapted the follow stats API to return a 404 instead of 200 with empty body if there are no follow shard tasks for the requested follower indices.

Following this work, we continued work on the Advanced Settings UI for adding a follower index. After review, the UI was revised to be a simple hide/show toggle for all settings. We also worked on PR that introduces the ability to edit a follower index. Editing refers to changing the follower index's advanced settings only. Since settings can only be changed upon resuming a follower, a warning is surfaced to the user that we will be pausing and resuming the follower in order to update it. We also added remote cluster validation to the create auto-follow pattern form and add/edit follower index forms. This presents a nice UX when

There are no remote clusters
When a selected remote cluster is not connected
When a previously set remote cluster has been removed.

We are adding timeout handling and rate limiting to the CCR recover from remote functionality. We have also extended the security model for the remote cluster proxy so that the recovery from remote can fetch the Lucene file chunks under the system security context.

Changes

Changes in 6.5:

SQL: Fix issue with field names containing "." #37364

Changes in 6.6:

Document Seq No powered optimistic concurrency control #37284
Add fatal_exception field for ccr stats in monitoring mapping #37563
SQL: Describe aliases as views #37496
When removing an AutoFollower also mark it as removed. #37402
Use executor SAME to handle search related handlers #37427
SQL: Make FULL non-reserved keyword in the grammar #37377
Fix artifactId in plugin poms #37315
Geo: Do not normalize the longitude with value -180 for Lucene shapes #37299

Changes in 6.7:

Create specific exception for when snapshots are in progress #37550
Add set_priority action to ILM #37397
Propagate Errors in executors to uncaught exception handler #36137
Update Put Watch to allow unknown fields #37494
Make recovery source send operations non-blocking #37503
Allow field types to optimize phrase prefix queries #37436
Change file descriptor limit to 65535 #37537
SQL: Add protocol tests and remove jdbc_type from drivers response #37516
Deprecate requests that have an unconsumed body #37534
SQL: Remove slightly used meta commands #37506
Packaging: Remove permission editing in postinst #37242
Prepare to make send translog of recovery non-blocking #37458
Update Delete Watch to allow unknown fields #37435
New mapping signature and mapping string source fixed. #37401
Restore lost @Inject annotation #37452
Make finalize step of recovery source non-blocking

https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2019-01-19