6. ECE 2.0: Supercharge Your Use Case

ECE 2.0: Supercharge Your Use Case


  • We recently released Elastic Cloud Enterprise (ECE) 2.0 which includes a number of features that give you and your team additional flexibility in managing and scaling your Elastic fleet. This blog post highlights three use cases that take advantage of these new capabilities.

    Before you begin

    This blog post assumes you are already familiar with the new concepts introduced in ECE 2.0, which were covered in the release blog post. If you are unfamiliar with the features described below, we encourage you to spend a few minutes reading about them in our user guide before moving on.

    1. Allocator tags: key-value pairs that allow you to classify your hardware and later use this classification with instance configurations to match a specific component to the underlying hardware.
    2. Instance configurations: custom configuration that matches each Elastic component to an underlying set of allocators. For example: your Kibana instance, machine learning node, Elasticsearch data node, and so on.
    3. Deployment templates: a set of instance configurations that will be used to provision a new deployment when a specific template is selected.

    Using the right hardware for the right use case

    The Elastic Stack can be used for search, logging, metrics, security analytics, and much more. In order to give you maximum performance it is important to use the right hardware for the specific use case. Thankfully you can use a combination of allocator tagging and instance configurations to help define RAM:Disk ratios and ensure you use the best hardware available for each use case when using ECE. If you are using Amazon Web Services or Google Compute Engine be sure to check out the configurations we use for those respective platforms with our Elasticsearch Service.

    Logging

    When building for logging use cases, there are several hardware types that will help you optimize costs without slowing down your ingestion rate or your searches for recent data. This is because we now allow you to configure a hot-warm architecture with a hot tier for ingestion of new data (which is searched more frequently) and a warm tier to retain data, that is less frequently queried, for longer retention period.

    For the hot tier, we recommend using fast local storage with a smaller RAM:Disk ratio used in High IO hardware profile. In our Elasticsearch Service we use local NVMe SSD storage as well as a RAM:Disk ratio of 1:30. For the warm tier, we use a larger RAM:Disk ratio of around 1:100 with slower spinning disks that are normally used in a hardware profile optimized for storage.

    In order to support this setup you can tag the relevant hosts meant to host the hot tier with tags like type:highio and hot:true, while the hosts dedicated for the warm nodes can be tagged with type:highstorage and warm:true.

    Node Types

    Using different hardware for Kibana, dedicated master nodes, and ingest nodes

    One of the most powerful features of ECE is the ability to configure different instance configurations based on different roles of your Elastic Stack deployment. One of the top tricks is to use servers that have a high capacity of memory for Kibana and dedicated master nodes, without having too much local storage. This means you can also use much smaller RAM:Disk ratios such as 1:4, or 1:8 for Kibana and dedicated master nodes, as neither use much storage.

    On the ingest side of the deployment, you may want to prioritize for CPU depending on the number of pipelines and operations you have. You can once again have a low amount of storage for hosts running these instance configurations. CPU time is shared between deployments based on their relative RAM. For example, in a 256GB RAM and 16 Cores machine, a node with 32GB of RAM with get the same CPU ratio. In this case that’s 32 / 256 which is 12.5% of the CPU time shares.

    In these example you can tag your allocators with something like the following:

    • Allocators used for Kibana and master nodes: type:highmem, kibana:true, master:true
    • Allocators used for Ingest nodes: type:highcpu, ingest:true

    Your teams deserve the best

    Often times companies choose ECE as their preferred weapon of choice to manage thousands of clusters in order to allow internal users to spin up and manage their deployments with ease while optimizing resource utilization since, by default, ECE will always attempt to fill allocators while respecting the allocator search criteria. A common ask from our users is to isolate deployments on different allocators based on the team that will use those deployments.

    The reasons to separate teams can vary. One example is a need to use different budgets to pay for the underlying hardware, which is a very common request for ECE installations where a centralized IT team will manage ECE in order to provide Elastic Stack as a Service, allowing their internal teams to leverage the zero downtime upgrades, ease of scaling a deployment with a single REST API request, etc.

    Another example is when teams have different needs and different SLAs. A certain team might use Elastic as an internal logging solution while another is highly dependent on their Elasticsearch cluster as part of their production tool set. Now, with the ability to build these requirements into deployment templates, you don’t need to worry about constantly checking how a deployment is running, and you can rest easy knowing that each deployment is running on the correct host. This also helps teams better plan for redundancy and disaster recovery for each deployment. To do so, simply create a deployment template for each team with the relevant configuration “baked in” and you’re good to go!



    https://www.elastic.co/blog/elastic-cloud-enterprise-ece-2-0-supercharge-your-use-case

Log in to reply
 

© Lightnetics 2024