Elastic News

• 

  Life@ Elastic | Black History Month
  • sorangutan  

  1
  1
  Posts
  33
  Views

  

  February is Black History Month. Hear from two of our Elasticians — Alyssa Hester and A.J. Angus — to find out what Black History Month means to them, who has had an impact on their lives, and why sharing who they are with colleagues is important.  

  Life@ Elastic: Hi Alyssa and A.J., thanks for joining us! Please tell us a little bit about yourselves and what you both do here at Elastic? 

  Alyssa Hester: I'm a consulting engineer on the Professional Services team and have been at Elastic for about a year now. Basically, that involves going to work with different clients, and helping them gain success with Elastic products by giving them architectural guidance, best practices, or even doing code implementations. Sometimes I also help out the Elastic Education team with training delivery. And that's always fun, like when I get to go to Elastic{ON} and help with the trainings there before moving over to the AMA to help people dive deeper into their questions.

  

  A.J. Angus: Hi everyone, I’m A.J. I've been with Elastic for three years. For my first two years I was dedicated to our CRM Engineering teams, which included building up our Salesforce implementation for our customers and employees. This past November though, I made a change into product management within IT -- where I think I've always wanted to be. 

  

  I've never really been sure if I'm a business person who knows about technology or a technology person that knows about business, and product management is kind of at that nexus. So now I'm getting the opportunity to do it, managing a new system that's still a work in progress. I've been learning a lot and it's been great to be a product manager and start to build something from scratch here at Elastic.

  Tell us a little bit about what motivates you to get up and “go to” work? Or how you get your days started off on the “right” foot? 

  Alyssa: I love solving problems and helping people solve problems. And to get my days started, a good cup of coffee, a yogurt, and some sunshine.

  A.J.: I have two dogs so everyday, at 7AM sharp, I’m up and we go on a 1.5 mile walk. Rain or shine! Scout is two years old and Shadow is nine months. If we don’t get in our walk, they are crazy! And after the walk, it’s time for meditation. I lead our daily “e-Pause” session for our U.S. Central & Eastern time zones. ePause is a virtual meditation that we host around the globe, four sessions a day, so that Elasticians in all time zones can participate. We do this together using Zoom video conferencing and the Headspace app. 

  What do you both enjoy most about the culture here at Elastic? 

  A.J.: I can be my authentic self at Elastic. I feel like we do a good job of letting people be who they are, kind of because we have to since we have people all around the United States and the globe. There's not one dominant culture or experience. We have to be inclusive of more than one monolith, otherwise we wouldn't be able to operate as a company.

  Alyssa: It's just easy to be who you are because like AJ said,  we're so distributed. And since there's no one predominant culture, it's actually so much easier to find other people you have things in common with. It's more freeing, more liberating.

  What would you like to share as African Americans to others of color in the tech sector? Or to a young person of color heading into the workplace?   
  

  Alyssa: I guess really what it comes down to for me is, when you see all these narratives about black people and when you see all the stuff in popular culture, it can really weigh you down. It can weigh you down and make you diminish yourself and your own ability. But don't let other people's opinions of you influence what you are going to do. 

  It's easier said than done -- and it's a constant struggle -- but keep pressing forward, and know that you're not alone. Rely on your foundations, and the people who love you to help get you through those tough moments of self-doubt that you might have. Don't hold yourself back because of what other people might think. You can't control other people, but you can always control your own actions and how you feel about a situation.

  A.J.: I think for me it's kind of escaping some things that Alyssa was alluding to, like me escaping the feelings of imposter syndrome. I know things, and I know that I know things, but then I work with PhD computer scientists, and I forget that I know. But I need to remind myself that I know what the hell I'm talking about! And you're going to have to do that too.

  I think that if you look at people in the black community, not as many go to the name brand and Ivy League schools that most tech companies, historically, hire from. Remember that you have a foundation of knowledge through your education and through your life. You just need to get to that place where you have confidence in yourself when you talk to people. It's easier than you think. Get out of that imposter syndrome. You know what the hell you're talking about!

  

  So, what does Black History Month mean to each of you?
  

  Alyssa: For me, Black History Month is about recognizing all of the fantastic work that black people have done, as well as raising awareness about black issues continuing today. Without understanding the past, we can’t hope to understand the present or improve the future. You can't really move forward without a clear grasp of what's happened in the past. 

  Today, some of the narratives persist, even just comments that people make, come from a lack of knowledge around black history. Positive African American representation is rare in the mainstream, so it is important for people -- and especially kids -- to understand the rich legacy they come from, and to know that there have been people who look like them who have done incredible things across all industries.

  It's also important for illuminating the disparities of both the past and the present. For example, understanding health disparities in African Americans becomes a lot clearer when put into the historical context of past medical experimentation on African Americans and knowing that African Americans have been treated as having less pain than others and still are by some doctors.

  A.J.: And for me, Black History Month means honoring and celebrating the people who paved the way for me to live my life today. As a mixed-race person who's father is from Jamaica (Scottish/West-African) and mother is from Illinois (Irish/Scottish), it's taken me years to see how I fit in with what it means to be black in America. I've recently gone back in history and read the autobiographies of historical black figures like Malcolm X, W.E.B. Dubois, Booker T. Washington, and Sojourner Truth. Reading about their lives now as an adult has really imprinted their struggle for basic rights that I surely take for granted. It's set me on a course for how I can make a difference for the people who will come after me. 

  Who are the people from black history whom you admire or who’s made an impact on you?
  

  A.J.: Malcolm X. Reading his biography really changed my life. There is a perception of him as a radical with a lot of hatred. Although I can understand that viewpoint, I think Malcolm was simply reacting to a world he was living that was unfriendly to his skin tone. You can see a clear change in his viewpoint after completing his pilgrimage and meeting people abroad who treated him positively in a way he never saw in America.

  Aside from the Autobiography of Malcolm X, some other books I'd highly recommend are The Mis-Education of the Negro, The Talented Tenth, Autobiography of an Ex-Colored Man, Narrative of the Life of Frederick Douglass, and Souls of Black Folk. Let me know when you get done with those and I'll give you more.

  Alyssa: I really identify with the books of James Baldwin a lot. He wasn't just a black man, but he was also a gay black man. And to speak about it and write openly about it at that time was incredible revolutionary. I identified with his books a lot as a member of both the black and  LGBT+ communities. Reading The Fire Next Time, reading Notes of a Native Son, reading Go Tell It on the Mountain, reading Giovanni's Room ... it was cathartic. Just knowing that someone has walked a similar walk as you before is a relief. 

  Are there people making Black History today in your opinion?
  

  Alyssa: Arlan Hamilton. It takes an incredible amount of courage to boldly step into an industry where you seem like an outsider from many angles and disrupt it. 

  Also, anyone who’s doing the best they can and also giving back to their community and working to raise others up.

  A.J.: Anyone and everyone working on reforming the criminal justice system in the United States. The current system disproportionately affects African Americans and people of color. Books like The New Jim Crow and reporting like Season 3 of Serial put a spotlight in this area and offer commentary on the current situation and what some reforms could look like. 

  Thank you both for your insights! If there is one more thing you’d like to share, what would it be?
  

  Alyssa: Being a minority can be tough and confusing as you seek to be yourself while navigating the culture around you, especially in industries where you’re underrepresented. Sometimes it’s confusing if you’re not sure that the way you’re feeling is normal, or whether you’re being true to yourself as you find your way in the world as an African American.

  By speaking up about Black History month, and by celebrating loudly (sometimes Black History Month is glossed over), we hope to encourage a younger generation and other people of color that yes, you can be heard as a minority.

  We hope this interview will be a fresher, more personal way to show that these feelings and struggles are normal. Pursue your passions and what’s important to you, surround yourself with people who understand your struggles and joys, but also share with colleagues who want to learn more about who you are. #ShareBlackStories

  
  

  ---

  Distributed by intention. Strength in diversity.Learn more about Elastic and why a diverse company is a resilient company.

  
  
  https://www.elastic.co/blog/life-at-elastic-black-history-month-2019
• 

  Streaming POST Requests to Elastic APM in Ruby with http.rb and IO.pipe
  • sorangutan  

  1
  1
  Posts
  37
  Views

  

  To keep the memory footprint of our Elastic APM agents small, the agents stream events as they happen via long-running POST requests to APM Server The server accepts ndjson which is JSON objects separated with newline \n characters. Can’t get simpler than that.

  Streaming in Ruby can be tricky — Ruby is synchronous, so with regular HTTP requests, the program will block and hold everything else back until the request is finished. This is of course not what we want in this case, as our app should continue to serve our users while it is sending its events one by one to APM Server.

  We instead start the request in a thread and pipe the events from our main program thread to the request thread using Ruby’s built-in IO.pipe The wonderful Rubygem http.rb supports IO objects so we’ll depend on that.

  require 'bundler/inline'
  
  gemfile do
    source 'https://rubygems.org'
    gem 'http'
  end
  
  read_pipe, write_pipe = IO.pipe
  
  # Unfortunately for our case Http.rb calls `rewind` on the IO
  # object after the request ends but pipes don't have such a method.
  # We'll add an method stub to circumvent this:
  read_pipe.define_singleton_method(:rewind) { nil }
  
  # First we make an HTTP client and set the Transfer-Encoding
  # header to let the server know that we'll be sending in chunks
  client = HTTP.headers({ 'Transfer-Encoding' => 'chunked' }) 
  
  # In a thread we open the request and give it the read pipe
  request_thread = Thread.new do
    # This thread is blocked on the request below while the main
    # program thread continues on
    client.post('https://example.com', body: read_pipe).flush
  end
  
  # Next let's simulate 10 events with a bit of pause between them
  1.upto(10).each do |number|
    data = { number: number }
    write_pipe.write("#{data.to_json}\r\n")
    sleep 0.2
  end
  
  # Then when we're done sending our data, we'll close the pipe
  # which will also end the request
  write_pipe.close
  
  # When the request's body is closed the request thread will reach
  # its end. We'll wait for it before we end the script.
  request_thread.join
  

  This is a constructed example script of course. In a real-world instance of the Ruby APM agent we have timeouts and byte size limits for the requests that, when reached, ends the current request and starts a new one.

  We also have GZip – let’s add that to our example:

  require 'bundler/inline'
  
  gemfile do
    source 'https://rubygems.org'
    gem 'http'
  end
  
  # We'll need the built-in zlib library
  require 'zlib'
  
  read_pipe, write_pipe = IO.pipe
  
  read_pipe.define_singleton_method(:rewind) { nil }
  
  # Toggle binary mode for the write pipe
  write_pipe.binmode
  client = HTTP.headers({
    'Transfer-Encoding' => 'chunked',
    # Add Content-Encoding header to let the server know we are
    # gzip'ing now
    'Content-Encoding' => 'gzip'
  }) 
  request_thread = Thread.new do
    client.post('https://example.com/', body: read_pipe).flush
  end
  
  # Initialize a new GZip pipe and put it in front of our existing
  # write pipe
  gzip_pipe = Zlib::GzipWriter.new(write_pipe)
  (1..10).each do |number|
    data = { number: number }
    # Writing looks the same but uses the gzip pipe instead
    gzip_pipe.write("#{data.to_json}\r\n")
    sleep 0.2
  end
  
  # Close the gzip pipe instead
  gzip_pipe.close
  
  request_thread.join
  

  There you have it
  

  Streaming requests in Ruby might require a bit more effort than in, say, JavaScript with its evented nature, but this isn’t too bad, is it? Doing this, the agent can release its objects as soon as they are sent. The Ruby APM agent uses this approach from 2.0 and onwards.

  Adding Elastic APM to your Ruby app is as easy as ever, requiring as little as adding the gem to your Gemfile. Learn how and get the agent streaming metrics for your app today. As always, if you have any questions or feedback, start a new thread on our APM forum.

  
  
  https://www.elastic.co/blog/streaming-post-requests-to-elastic-apm-in-ruby-with-http-rb-and-io-pipe
• 

  CenturyLink: Building a Network Management Framework with Open Source
  • sorangutan  

  1
  1
  Posts
  44
  Views

  

  This post is a recap of a community talk given at a recent Elastic{ON} Tour event. Interested in seeing more talks like this? Check out the conference archive or find out when the Elastic{ON} Tour is coming to a city near you.

  CenturyLink is the third-largest telecommunications company in the United States and has created one of the largest networks in the world, spanning more than 450,000 route miles — long enough to go around the earth 18 times. At any given moment, they’re dealing with serious volumes of data from a plethora of different network devices, and those devices continue to evolve year after year. Networks that once supported landlines and pagers have had to adapt to deal with requests from smartphones, smart speakers, and tablets, all of which generate new and different types of data — and more of it. And CenturyLink itself has evolved, too, growing substantially over the years through mergers and acquisitions as well as organic growth. Today, CenturyLink’s environment includes over 300 thousand devices that generate over 1 million alerts daily.

  CenturyLink’s rapid growth, in addition to the complex types and volumes of data that it is handling across its massive network, have posed challenges for the company’s logging and metrics analytics strategy. On the front lines is CenturyLink’s Service Assurance team, who is responsible for ensuring that network operations centers (NOCs) have the tools they need to ensure that the network is optimal at all times. The key here is the system logs and metrics that trigger alerts informing Operations when, for example, a fiber line has been cut or traffic is being rerouted.

  When CenturyLink was relatively small and was juggling only a few network management systems (NMSs), a basic architecture — an adapter on top of the NMSs that fed into an Oracle database, which in turn fed into a display — was enough to keep those alerts flowing. But as CenturyLink acquired new partners that used different alarm types and disparate network management structures, database queries started to get complicated. Struggling database performance translated into repeated outages and downtime for their NMS app, which meant Operations couldn’t see what was going on in the network. It quickly became clear that this structure wouldn’t be able to scale as CenturyLink grew.

  CenturyLink began mapping out their entire system with one goal in mind: identifying a simpler, more robust architecture that would be able to scale with the company’s future growth. They scored an early victory by implementing a CQRS (Command Query Responsibility Segregation) architecture that separated the transaction manager and query cache into different components, with a REST service on top. Then they began working to integrate these homegrown elements with a number of other pieces that had been built over the years, with the goal of simplifying the overall system. On the query side, they tried implementing H2, distributed MySQL databases, and even a HashMap distributed across multiple applications, and they began to see some improvements — reduced downtime and a happier Operations center. But the complexity of the queries and the volume of the alarms made it difficult to continue to support this in a relational database management system.

  They found their answer in Elasticsearch. By incorporating Elasticsearch into the query side of their CQRS architecture, they were able to query more data than ever, faster than ever. Queries that had previously taken 1 or 2 minutes now took only seconds. Not only was Elasticsearch easy to set up — they were able to spin up a cluster instantly — but it also scaled effortlessly. Gone were the days of bracing for the next acquisition announcement. With this simpler network architecture, which also included Redis and Kafka and a custom UI working in concert with Elasticsearch, new network management systems and devices could be smoothly brought into the fold. Today, CenturyLink is supporting 1,000 users on this application, with greatly reduced outages and zero production defects.

  Working with open source solutions like Elasticsearch has made the Service Assurance team realize that it’s possible to do even more with their data. In addition to the fault and performance data that they are already feeding into their systems, they are also collecting information about network utilization, weather, and even how technicians solved problems in the past — and they want to bring more of this data into their environment so that they can make more informed decisions. To make that a reality, the Service Assurance team envisions a large framework in which various types of data can be put through predictive algorithms and then operationalized in real time in order to produce actionable insights for the Operations team. To build this framework, CenturyLink has continued to leverage the flexibility of the Elastic Stack, with custom Logstash plugins and Elasticsearch playing key roles. And they plan on developing custom Kibana plugins in the future.

  Want more details on what CenturyLink is doing to get better use out of their data? Watch The Elastic Evolution of CenturyLink’s Network Management System from Elastic{ON} Tour Denver.

  
  
  https://www.elastic.co/blog/centurylink-building-a-network-management-framework-with-open-source

• 

  Building a Real-World Alarm with Elasticsearch and Raspberry Pi
  • sorangutan  

  1
  1
  Posts
  30
  Views

  

  Do you remember the last time when you saw a fire alarm or the police turning on a patrol car's flashing lights? That kind of warning light or loud sound simply notifies you that something is very important, or gives some some sense of danger, and that you must pay attention to it.

  Sometimes, you may also want to do a similar thing with your data, like when your server is down. That’s a big thing and you should call your admin as soon as possible to get it fixed. If your admin is nearby, you could even directly shout out to him/her.

  “Aha!” you might say, let’s use Elasticsearch to monitor our infrastructure and use the alerting feature to notify our admin via email or Slack. Wow, that’s great, but it can be better. Sometimes, the admin’s mobile phone maybe needs to be charged or it’s been left on the table, or the admin may be having a conversation and they may not pay attention to the message, so how can we bring it up?

  What if we used a real world warning light, like what we see during a fire alarm, and even add a loud sound that will be interesting and noticeable, so no one will miss it?

  In this post I will explain how to wire a Raspberry Pi with an alarm light and integrate it with the Elastic Stack to make this happen.

  Intro to Raspberry Pi

  Raspberry Pi is a very small, but fully featured computer on a single board. You may plug in a monitor with an HDMI cable and attach a keyboard, mouse, camera, or even speakers via USB ports. It can also connect to your network through Wi-Fi or an ethernet port.

  

  The GPIO pins can be used with a variety of alternative functions, some are available on all pins, others on specific pins. This list below details the functions:

  • PWM (pulse-width modulation)
    • Software PWM available on all pins
    • Hardware PWM available on GPIO12, GPIO13, GPIO18, GPIO19
  • SPI
    • SPI0: MOSI (GPIO10); MISO (GPIO9); SCLK (GPIO11); CE0 (GPIO8), CE1 (GPIO7)
    • SPI1: MOSI (GPIO20); MISO (GPIO19); SCLK (GPIO21); CE0 (GPIO18); CE1 (GPIO17); CE2 (GPIO16)
  • I2C
    • Data: (GPIO2); Clock (GPIO3)
    • EEPROM Data: (GPIO0); EEPROM Clock (GPIO1)
  • Serial
    • TX (GPIO14); RX (GPIO15)

  Basically, GPIO lets us connect to physical electronic device like alarm lights and also allows us to control these easily.

  How it works

  So how it does this all work? The idea of this blog post is to use Raspberry Pi to control a relay module and light up a warning light. There are many alarm lights available to buy and they are very simple. You plug it in, it starts to flash, you unplug it, it stops flashing. More than simple enough for us to use. But wait! They usually are powered with higher voltages, like 12 V, 36 V, or even 220 V, and the Pi can only support 5 V or 3.3 V, which means we can’t control it directly with the Pi, but with a relay we can make it happen. A relay is an electrically operated switch. It uses a very low power electromagnet to mechanically operate a switch, and that switch can be the bridge to connect the high power power supply and our warning light.

  Then we will build a webhook service to send the command to the relay, tell the relay to switch on or off, and control the alarm light.

  And finally, we will use Elasticsearch’s alerting feature to trigger the webhook when we find some interesting events in Elasticsearch.

  Are you interested? If so, let’s get started!

  Materials required

  So besides the Raspberry Pi, what materials are needed?, We also need some other electronics items, all of them are available at most electronic stores. Here is the list:

  Name	Unit	Comments
  Raspberry Pi 3 model B+	1	You know, for Pi
  16GB MicroSD card + reader	1	Storage for Raspberry Pi
  5V USB power supply + USB cable	1	Power for Raspberry Pi
  Warning light with 12 V power	1	The light to do the alarm
  5 V Relay module	1	Control the lights
  Breadboard	1	Connect all the wires
  GPIO Breakout Expansion Board + Ribbon Cable	1	Connect breadboard and Pi
  3.3 V Active Piezo Buzzer Module	1	Let’s make some noise
  Female – Female jumper cable	~5	Wire stuff together
  Male – Female jumper cable	~5	Wire stuff together

  Hardware setup

  Below is the diagram of how to connect them together. As you can see, we are using a ribbon cable to connect the Pi and the breadboard, also make sure the GPIO breakout aligns with the breadboard.

  

  Software setup

  Now the hardware is ready, let’s jump to the software part. The Raspberry Pi is based on the ARM CPU architecture and supports the Linux operating system. I will not teach you how to install a OS in Pi, because you can get very detailed instructions from Raspberry Pi’s official site.

  I have chosen RASPBIAN OS instead.

  And now you should be able to use your favourite terminal tool to log into the Pi.

  Controlling the Alarm with Python

  OK, let’s have some Python scripting fun by creating a script like this:

  #!/usr/bin/env python
  import RPi.GPIO as GPIO
  PIN_RELAY =  22
  GPIO.setmode(GPIO.BCM)
  GPIO.setwarnings(False)
  GPIO.setup(PIN_RELAY, GPIO.OUT)
  GPIO.output(PIN_RELAY, GPIO.LOW)
  

  At first, we imported a Python library RPi.GPIO, which is a neat Python package allowing you to operate the GPIO easily.

  Then, we set which GPIO pin we are going to operate. Our relay is connected on pin 22 so we set the variable PIN_RELAY to 22. We then setup the pin to GPIO.OUT mode, which means we are going to write out some data to this pin and we set a value GPIO.LOW to this pin. GPIO.LOW is the new state value of the pin, there is the other state GPIO.HIGH which we will use to control the relay, and relay will be switched on after you set the state to GPIO.LOW. If you set it back to GPIO.HIGH, the relay will be switched off. There is also a jumper on the relay to set the default state.

  Note that if you set it to GPIO.LOW before you set it to GPIO.HIGH, the relay will keep this state, which means that the alarm light keeps flashing. You can turn it off after a while, like this:

  time.sleep(2)
  GPIO.output(PIN_RELAY, GPIO.HIGH)
  

  Now let’s see how to make a sound. It is pretty similar to the relay, the only difference is the pin code, since the sound buzzer is connected to pin 12. Let’s see the code below:

  PIN_AUDIO = 12  #gpio 12
  GPIO.setup(PIN_AUDIO, GPIO.OUT)
  GPIO.output(PIN_AUDIO, GPIO.LOW)
  time.sleep(0.5)
  GPIO.output(PIN_AUDIO, GPIO.HIGH)
  

  An alerting webhook service

  Building a webhook service is easy. We need to make sure that we call the webhook and it then calls the relay and buzzer. Let’s put it together using the code we built above, and a few extra sections:

  import os
  import time
  import RPi.GPIO as GPIO
  from socket import *
  PIN_AUDIO = 12  #gpio 12
  PIN_RELAY = 22  #gpio 22
  def setup():
    GPIO.setmode(GPIO.BCM)      
    GPIO.setwarnings(False)
    GPIO.setup(PIN_RELAY, GPIO.OUT)
    GPIO.setup(PIN_AUDIO, GPIO.OUT)
  def alarm():
    GPIO.output(PIN_AUDIO, GPIO.LOW)
    GPIO.output(PIN_RELAY, GPIO.LOW)
    time.sleep(2)
    GPIO.output(PIN_RELAY, GPIO.HIGH)
    GPIO.output(PIN_AUDIO, GPIO.HIGH)
  def createServer():
      serversocket = socket(AF_INET, SOCK_STREAM)
      serversocket.bind(('0.0.0.0',9000))
      serversocket.listen(3)
      while(1):
          (clientsocket, address) = serversocket.accept()
          alarm()
          clientsocket.send("HTTP/1.1 200 OK\n"
           +"Content-Type:application/json\n"
           +"\n" # Important!
           + '{"success":true}'
           +"\n")
          clientsocket.shutdown(SHUT_WR)
          clientsocket.close()
      serversocket.close()
  def destroy():
    GPIO.output(PIN_RELAY, GPIO.HIGH)
    GPIO.output(PIN_RELAY, GPIO.HIGH)
    GPIO.cleanup() 
  if __name__ == '__main__':   
    setup()
    try:
      createServer()
    except KeyboardInterrupt:  
      destroy()
  

  Yay! We’ve just created a simple web server. You can start the webhook easily, just run: python web.py, it listens on port 9000, each time we access this port it will return a success message. Can’t be easier. But for our alerting usage this is enough.

  Integrate with alerting

  We have a small Elasticsearch cluster running on Pi, but it can be on Elastic Cloud or on your own hardware. We also have Heartbeat to monitor several services, and we will create a Watch in Elasticsearch to watch the Heartbeat indices, and will be using our new webhook endpoint (192.168.1.200:9000) in the alerting action. Here is the full Watch:

  PUT _xpack/watcher/watch/server_is_down
  {
   "trigger": {
     "schedule": {
       "interval": "1s"
     }
   },
   "input": {
     "search": {
       "request": {
         "search_type": "query_then_fetch",
         "indices": [
           "heartbeat-*"
         ],
         "types": [],
         "body": {
           "size": 0,
           "query": {
             "bool": {
               "must": [
                 {
                   "range": {
                     "@timestamp": {
                       "gte": "now-3s"
                     }
                   }
                 },
                 {
                   "match": {
                     "monitor.status": "down"
                   }
                 }
               ]
             }
           }
         }
       }
     }
   },
   "condition": {
     "compare": {
       "ctx.payload.hits.total": {
         "gte": 1
       }
     }
   },
   "actions": {
     "alarm_webhook": {
       "webhook": {
         "scheme": "http",
         "host": "192.168.1.200",
         "port": 9000,
         "method": "post",
         "params": {},
         "headers": {},
         "body": "SOS, server is down!"
       }
     }
   },
   "throttle_period_in_millis": 5000
  }
  

  As you can see the request above, the Watch will check the Heartbeat index every second. If it finds that at least one service is down in the most recent 3 seconds, the Watch will trigger a webhook action, and also we set the throttle period to 5 seconds to reduce repeat alerts.

  When the Watch is created, it will watch your cluster 24x7 without rest, and also you can view the execution history details from Watcher’s UI:

  
  
  https://www.elastic.co/blog/building-a-real-world-alarm-with-elasticsearch-and-raspberry-pi

• 

  Keeping up with Kibana: This week in Kibana for Feb 22, 2018
  • sorangutan  

  1
  1
  Posts
  56
  Views

  

  Hiring
  

  • Senior Product Manager, Kibana (Platform & Security)
  • Kibana - UI/Front End Team Lead
  • Kibana Platform - Principal JavaScript Engineer
  • Kibana - Senior JavaScript Engineer
  • Kibana - Senior Security Developer (JavaScript)
  • Kibana - Platform Javascript Engineer (Node.js)
  • Kibana - Visualisations & Vega Engineer

  Roadmap themes updates

  New platform

  This week two new hires joined the new platform project and we had an official new platform team kick-off meeting. For the next two to three weeks, the team will be focused on making  modifications of the already completed platform services/components. Plugin migration plan is WIP. We also met with Kibana app team to discuss some migration tech details.

  Security

  The security team continued working on integrating feature controls into the saved objects management, and the infra applications. Additionally, some small modifications were made to the CSP which will be implemented in Kibana 6.7 to allow Edge to continue functioning properly.

  Middleware/Alerting/Notifications/Scheduling/Secrets

  With the changes to the saved objects service merged and backported the Secret Service is all ready for review. https://github.com/elastic/kibana/pull/28894

  This week was spent working on the design of the actions service and vetting more details about how it will work. Some work has been done on a couple of main use cases. An RFC proposal is currently being created with a more detailed design. Will send out a follow-up when it is in a PR for review.

  Saved Object

  This week we finalized the development of the export API and continued working on the new import API.

  Geo

  Maps

  Towards GA:

  • Started work on embeddability for Maps into Dashboards. Some of the groundwork and internal Kibana changes have already been completed. Please track progress here  #31137 (screenshot below displays a map in a dashboard).

  • Preliminary discussions on GeoJson upload started. #28765

  Polish for 6.7/7.0 Maps Beta continues:

  • More bug fixes and UI polish as we’re getting closer to release the Maps Beta
  • Localization of code-base of the Maps-app has started. This will likely not make 6.7.0/7.0.0, but will be released with patch. #30881

  Elastic Maps Service

  Data production

  • Infra started work on data production of tile-data. Maintaining tile-data is a new responsibility, and will be a long term effort.
  • New layers in the pipeline. These will be published to EMS next when reviewed and validated.
    • Belgium, Belarus, Croatia, Estonia, France, Hungary, Luxembourg, Poland, Slovakia, Slovenia
    • India, Brazil
    • 

  7.0 Stack Upgrade

  Working through resolving an issue with the upgrade assistant re-indexing of security indices (elasticsearch#39284). These indices are whitelisted, so the decision has been made to stick with the naming convention created in the 5.6 upgrade assistant (.security-X).

  Browser testing coverage

  • The WebDriver change was backed out while some stability issues are resolved

  Release Manager: Elastic Stack Testing

  • Automated Stack Upgrades
    • Wrote test plan for phase 1
    • One script variation implemented and in testing

  Team summaries

  Canvas

  The Canvas team continues to work on performance and stability in the march towards GA. We’ve long suspected that React was thrashing and killing in-browser performance, and that’s where a lot of the perceived slowness was coming from. This week we confirmed and fixed one of the largest issues, where the entire workpad, all pages, and all elements, would be re-rendered for all sorts of crazy reasons, like even moving your mouse. You can see React thrashing in this flame graph

  

  With a little bit of code to make things smarter about re-renders, that same graph now looks like this:

  We’ve also been seeing EventEmitter memory leak message recently, and that’s now been fixed. Here’s a before and after of the memory profile:

  But it hasn’t been all bug fixes. For example, the essql function got time_zone support this week:

  

  Demodata now functions with the time filter, and the data was updated so you always have current data to play with and includes a @timestamp field so adding a time filter works from the start:

  We added content to the help menu for 7.0+, including a hotkey guide:

  

  Simplifying the code around the Layout Engine: Canvas is a presentation editor like Google Slides or Powerpoint. It supports lots of stateful interactions like rotation/move/resize, snap, grouping etc. going beyond regular UI, things like align/distribute and 3D planned.

  Processing user interactions and rendering their results must be done fluidly, 30-60 times per second. Therefore the layout engine now updates a separate, fast, plain state object, not the Canvas Redux state with ES indices etc. attached. But maintaining and synchronizing this with the central Canvas state is error prone, hindering feature development, fixes and perf improvements. Now a silo, it's being converted to stuff all developers use, eg. React component props/state and Canvas state.

  The WIP work simplifies the state management (we're targeting this for 6.7 though it's not a prerequisite):

  • no per-page layout state living in a Map outside React/Redux, having to sync with page ops
  • no longer maintaining two separate, overlapping state objects from an action stream; most layout middleware code gone and preparing for single-source of truth
  • avoids explicit rerendering triggers (setState trick)
  • simplified redux-reselect-like selectors (and selectReduce is gone)
  • helps complete the TypeScript conversion
  • adding `typings-tester` to our code base, without which TS types can degrade over time
  • due to simplifications, data-driven layout updates and an eventual fusion of tiled (Dashboard) and freeform (Canvas) editors will be easier

  It becomes even more compact around 10kB - the somewhat comparable react-beautiful-dnd which would cover a small fraction of what’s needed (and things Canvas doesn’t need) is 108kB minified.

  Design

  The design team is finally recovering from the 7.0 sprint.  7.0 isn’t yet over for us yet though, and we’re still working through some dark mode cleanup and polish work for the Uptime app and global datepicker component. On the EUI front we finally closed our some pesky issues around focus traps and portals within modals. We’re also working towards our final CSP solution and have a bunch of fixes / additions for the range slider, filter group, and icon components. Last but not least we’re slowly making progress on converting EUI to Typescript. Phew… there’s a lot going on!

  Working on some new homepage designs that lead people towards guides

  The ACE editor now works in dark mode

  Uptime app getting some cleanup

  Operations

  Our priority has been on resolving blockers with the upgrade process as they arise.

  PRs

  • Improve dll plugin relation with webpackshims (#30129)
  • [logging] Corrects intercept of ECONNRESET #31742
  • [UA] Protects against Cloud plugin being disabled #31637

  QA

  The team focused on testing 6.7.0 release

  KibanaApp (Visualizations, Sharing, Discover)

  The team is still stabilizing the 7.0 release by fixing several bugs, converting remaining UI to the new design (e.g. the create new visualization dialog) and implementing minor enhancements (like copy buttons for request/responses in the Inspector or better human readable labels in the Advanced Settings).

  PRs

  • Update embeddables metadata to support multiple index patterns (#31538)
  • Surface vis loader errors in the UI. (#30594)
  • tests typescript migration (#31234)
  • Add referenced pipeline aggs to every level of query (#31121)
  • Fix Visual Builder wrong color rules on multiple metric (#31455)
  • Dark overrides for Ace/Brace TM (textmate) theme (#31628)
  • Don't send request if it already failed during query parsing (#31219)
  • Add placeholder text indicating the filters agg takes only lucene and Query DSL (#31296)
  • Add visual indication of selected language in query bar (#30899)
  • TSVB error message needs to be formatted (#31569)
  • moved vis table to use EUI (#30610)
  • Add i18n context to inspector panel (#31467)
  • Use default index pattern when one is not present in visualize (#30574)
  • Remove use of 's' regex flag in KQL (#31292)
  • New visualization search selection (#30196)
  • Renamed the kuery setting to KQL (#29587)
  • Add documentation links to Visualize/Dashboard (#31406)
  • Remove geotile_grid aggregation from non-geospatial visualizations (#31354)
  • Add copy button to inspector request/response panel (#31408)
  • Add labels for sort order and request preference setting (#31236)
  • Fix bottom tooltips flickering on dashboard (#31245)
  • typescript search embedable (#30933)
  • Filter colours are not overriding label colours in Visual Builder (#30822)

  Elastic Chart PRs

  • test(chart_state): fill in missing coverage (elastic/elastic-charts#74)
  • test(theme): add tests for `mergeWithDefaultTheme` (elastic/elastic-charts#73)
  • docs: add codecov badge to README (elastic/elastic-charts#62)
  • docs: add feature request and pull request templates (elastic/elastic-charts#60)
  • ci(codecov): add coverage with codecov (elastic/elastic-charts#61)
  • feat: add dark theme (elastic/elastic-charts#44)

  Cheers,
  Kibana

  
  
  https://www.elastic.co/blog/keeping-up-with-kibana-2019-02-22
• 

  The Department of Work and Pensions: Proactive Monitoring with the Elastic Stack
  • sorangutan  

  1
  1
  Posts
  44
  Views

  

  The Department for Work and Pensions (DWP) is responsible for welfare, pensions, and child maintenance policy. As the UK’s biggest public service department it administers the State Pension and a range of working age, disability, and ill health benefits to around 20 million claimants and customers. The Child Maintenance Service is a department within the DWP that handles financial support for separated households and determines how parents handle costs for their shared responsibility in bringing up a child.

  The Child Maintenance Service (CMS) currently uses Siebel to handle caseloads and manage citizen requests. A new Digital Experience Monitoring (DXM) project was set up to help provide greater insight into the performance of the Siebel system and develop monitoring that would help the Live Service Support team be more proactive in ensuring system performance, provide a targeted view of core system performance, and improve capabilities to analyse system data both historically and in near real time.

  Finding the Right Solution

  An internal squad was formed in April, 2017 to help build this new monitoring capability. During the discovery phase of the project, the Elastic Stack was chosen for its ability to parse logs and extract meaningful information for visualization. The Elastic Stack was easy to get started using the open source version of the Elastic Stack for testing purposes, it also proved to be easy to use and scale once in production, and was also within cost constraints — ensuring efficiency and value for money.

  So, how did the Elastic Stack help shape Digital Experience Monitoring?

  Developing the Architecture

  An Agile squad were formed and, working in two-week sprints, set up delivery milestones to get the monitoring up and running as quickly as possible.

  Here is an overview of what the development team decided the ideal architecture might look like:

  

  Four phases were planned for the development of this setup:

  • Discovery: This was a short phase dedicated to researching user needs, what features should be built into the system, and any constraints the team would be working within to deliver a viable product.
  • Alpha: This was another short phase for prototyping solutions and determining whether they met user needs. This was a great opportunity for the team to test functionality with a small group of users and stakeholders, and to receive early feedback on the design and functionality of the service before moving towards formal development. In this phase, the team was impressed with the speed and ease in which they could create visualisations that gave their Live Service Support colleagues insight into the issues at hand and helped them ask more intricate, problem solving questions.
  • Beta: In this phase, the solution was tested in a production environment to discover the limits of scaling, and where the system might break. It also introduced broadening and thickening of outputs bringing in more logs where the business had key interfaces e.g. HMRC, and creating more business specific visualisations. Several Beta versions were released before the go-live.
  • Live: DXM is now widely available, with regular improvements and ongoing enhancements.

  In Production

  Today, the DXM team use several components of the Elastic Stack to run their Digital Monitoring System:

  • Logstash to ship, parse, and load data from a variety of sources.
  • Elasticsearch to store, search, and analyse the data. Every event created is stored on Elasticsearch as an individual document under a defined index.
  • Kibana dashboards makes deep dive analysis possible (such as multi-level aggregation, percentile analysis, baseline comparison, active users, site statistics, etc.).
  • Elastic security features to keep all this information secure.

  Here’s what the DXM setup looks like in practice:

  

  The data processed by DXM comes from a wide array of sources: Siebel Application response measurement, Siebel communication logs, BPM system logs, BPM application logs, and database queries. They use Logstash Forwarder Java, an open source shipper, to move data from application server to Logstash Server with the help of the lumberjack plugin.

  Data is extracted every 10 minutes from the Siebel Application and shipped to Logstash to capture critical performance data and enrich reporting on specific areas of the business.

  Seeing Results

  It was easy to see, even during the Discovery phase, how effective the Elastic Stack would be at uncovering patterns in daily component performance. However, it was during the Alpha phase, when visualisations and business-specific analytics were built, that Elastic’s value became truly evident.

  DXM built visualizations that looked at:

  • The average response time from Siebel
  • Total number of calls made to the system
  • Failed events, such as unexpected behaviors or patterns when processing application, or system logs

  Using this proactive method of monitoring, DXM was able to flag a variety of performance concerns and subsequently find solutions for them before they impacted end users. For example, at the early stages of development, DXM created dashboards that showed views of system performance pre- and post-fixes. One example dashboard (see below) highlights average response time to queries in seconds (left) and number of calls made to execute a Siebel operation (right).

  

  The top graph enabled DXM to do a pre- and post view of the world. For example: What was the average response time before the fix went in compared to what it was afterwards.

  Looking at the second graph the system fix that had been applied brought about system automation i.e. something that was time consuming for caseworkers was now automatic.

  Dashboard users are wide ranging, including the Live Service Support team who proactively monitor the Siebel system performance, the Child Maintenance Director and his team, and users working on specific projects when customised dashboards are created. Splitting the graphs like this enabled DXM to see ‘at a glance’ the impact of changes in case worker response times.

  Scaling for the Future

  The success of DXM was widely recognised and brought much praise from Senior Leaders who can now start to see and better understand Siebel system performance.

  DXM have shared their work with others across DWP and where appropriate are supporting others in adopting a similar monitoring solution. DXM have also upgraded the current Elastic Stack version being used from 5.4.2 to 6.3 where they will look to explore dedicated machine learning enabling them to uncover and predict anomalies within the system automatically and unsupervised.

  ---

  Suzy Robertson is a data engineer at the Department for Work and Pensions.

  
  
  https://www.elastic.co/blog/the-department-of-work-and-pensions-proactive-monitoring-with-the-elastic-stack
• 

  Mayr-Melnhof: Visualizing Cardboard Production Data with the Elastic Stack
  • sorangutan  

  1
  1
  Posts
  154
  Views

  

  This post is a recap of a deeper dive into how Mayr-Melnhof Group harnesses the Elastic Stack. Check out our Customers page for more in-depth Elastic Stack customer stories.

  When producing more than 1.7 million tons of cardboard a year, and processing billions of consumer goods packages annually — as the Mayr-Melnhof Group does — any mistake in the complex production process can be costly.

  The amount of moisture, fiber, and expensive materials in the manufacturing process required for every batch of cardboard needs to be precise.

  That’s why the Vienna, Austria-based multinational has deployed more than 25,000 sensors to capture production process data on everything from pressure and temperature to machine speed and power consumption.

  Enter the Elastic Stack

  Mayr-Melnhof didn’t have the right tools to quickly acquire all the meaningful intelligence from the 2 TB of data these sensors were generating. Belatedly discovering a deviation in the amount of virgin fiber in the manufacturing process, for example, could be a financial headache, and even halt production.

  In response, the company harnessed the Elastic Stack — specifically Elasticsearch, Kibana, and Logstash — to connect and visualize cardboard production data and usage of all relevant materials in real time. Four months later, the company lowered consumption of high-cost production materials by 20%.

  Bridging Information Technology and Operational Technology

  "The Elastic Stack makes sense of otherwise disparate data, bringing in a new real-time perspective on production processes," said Stephan Hampe, technologist for Mayr-Melnhof Karton, a unit of the Mayr-Melnhof Group. "With one feature set, we're able to bridge the two worlds of IT and OT, and optimize the process."

  Watch their talk given at a recent Elastic{ON} Tour Frankfurt detailing the company’s industrial production visualization process. Check out the conference archive or find out when the Elastic{ON} Tour is coming to a city near you.

  

  
  
  https://www.elastic.co/blog/mayr-melnhof-visualizing-cardboard-production-data-with-the-elastic-stack
• 

  This Week in Elasticsearch and Apache Lucene - 2019-02-22
  • sorangutan  

  1
  1
  Posts
  100
  Views

  

  Elasticsearch Highlights

  Zen2 at scale

  We have been investigating Zen2's behaviour with large cluster states and/or large clusters, pushing past the boundaries of what we might consider to be a reasonable deployment and into the territory of the OOM killer. Pleasingly, we found that master election behaves reasonably well even with tens of master-eligible nodes. We fixed one memory-consuming issue ( #39179) and continue to look for other ways to bound the memory needed to publish 100+MB of cluster state to 50+ nodes all at once.

  Performance

  We have released Rally 1.0.4. We did the release mainly to ensure users can still benchmark in a world of typeless APIs as this will be the default in Elasticsearch 7.0.0.

  After switching the default store type from mmapfs to hybridfs we have seen reduced performance in our nightly benchmarks (up to 20% less indexing throughput and significantly increased latency for some queries). The goal of that change was to avoid page cache thrashing for very large indices but for practical purposes (all benchmarks need to finish within a day) our nightly benchmarks use only rather "small" indices of up to 75GB. It turns out that Lucene uses a combined segment file format (.cfs) for smaller segments to save file handles which we accessed via NIO instead of memory-mapping them and that lead to significantly worse performance for the "small" indices we use in our nightly benchmarks. After discussions with the Lucene team and more benchmarking, we have now added cfs files to the list of files to memory-map. This restored original performance for small indices while retaining the performance benefit for large indices.

  ILM (and CCR)

  ILM is now fully integrated with CCR (#34648). The final commit to ensure the ILM and CCR work correctly together has been merged (#38529).

  In order to replicate across clusters efficiently and correctly, CCR needs to keep a history of the operations performed on a leader shard - indexing, deletes, etc. When those operations are replicated to the follower shard, they'll be applied in a way that keeps everything consistent. If CCR doesn't have that history of operations, it has to fall back to file-based recovery, which can be much less efficient. During this time, there would not be any available shard copy on the follower side, making this a situation that we want to avoid.

  Shard history retention leases allow the follower to be able to mark in the changes stream where it is, allowing the leader to keep all shard history after the operation specified in the lease to ensure that it doesn't have to be rebuilt from scratch via a file-based recovery.

  ILM now pays attention to these leases and waits to perform operations that would necessarily destroy shard history, specifically the Shrink and Delete actions, if there are any leases put in place by followers, ILM will wait until those leases are released by the follower or the leases time out. This makes sure we keep all the shard history around so that it can be replicated to followers - if we didn't do this, we would risk losing the history of shard operations while a follower is still trying to replicate operations from the leader.

  Search - Intervals

  We exposed some additional Interval filters to the elasticsearch query DSL. These new operators overlapping, before and after add new ways to match intervals within documents.

  DFS query

  We worked on an improvement to the way DFSPhase builds distributed term statistics. This is part of a longer-term plan to possibly remove Weight.extractTerms() from Lucene.

  ES Management UI

  We also worked on refactoring our server logic to a common library and avoid loads of code duplication across our apps. This work inspired us to work on refactoring our License checker logic for our application. This is the first step towards improving our UX to guide our users when they don't have the correct license to access a plugin.

  ODBC

  We implemented the functionality to enable the new DATE data type conversions. With it, an application that queries a column of type DATE will have the data returned as native ODBC DATE data structure, with all components broken down, thus enabling faster operations on it. This PR also updates the currently advertised available standards scalars.

  Apache Lucene

  Query Visiting

  Our team members are known for bringing up decade old issues after we learned a lot more what needs to be done, this time we are looking into Query Visitors to add a generic and flexible way to traverse Query trees. It's puzzling why these issues are originally assigned or opened by the same person.

  Last Minute

  We got in some last minute API breakage into 8.0 that makes delegation of TermsEnum much less trappy. This caused many issues in the past including enormous memory usage and slow retrieval.

  Also coming in last minute we added On-Disk term dictionaries. FSTs are Lucene's wonder weapon when it gets to fast term lookups. This data-structure is loaded entirely into heap memory until 8.0. Now in 8.0 FST are read from disk if the file is memory mapped. It tries to detect if mmap directory is used and reads off-disk if the term statistics imply that the field is not a ID field. The performance numbers are on-par with in-memory for non-ID fields while saving significant amounts of memory.

  Geo Land

  We pushed another performance optimization on the BKD tree by making the heap objects more efficient. This change has a side effect on other indexing strategies (e.g BKD tree on 1 dimensional points) as the tree will always create one of those heap objects regardless the data that is going to work on. We opened another issue to create such objects only when they are needed. We also refactored the tests for LatLonShape as a step to facilitate the implementation of CONTAINS.

  Changes in Elasticsearch

  Changes in 8.0:

  • Fix the OS sensing code in ClusterFormationTasks 38457
  • Remove setting index.optimize_auto_generated_id (#27583) 27600

  Changes in 7.1:

  • Distance measures for dense and sparse vectors 37947
  • Don't swallow IOExceptions in InternalTestCluster. 39068
  • BREAKING: Enforce Completion Context Limit 38675
  • Add overlapping, before, after filters to intervals query 38999
  • Tie break search shard iterator comparisons on cluster alias 38853

  Changes in 7.0:

  • Remove nGram and edgeNGram token filter names (#38911) 39070
  • Extend nextDoc to delegate to the wrapped doc-value iterator for date_nanos 39176
  • Do not create the missing index when invoking getRole 39039
  • Don't close caches while there might still be in-flight requests. 38958
  • Blob store compression fix 39073
  • Fix libs:ssl-config project setup 39074
  • Fix #38623 remove xpack namespace REST API 38625
  • Also mmap cfs files for hybridfs 38940
  • Recover peers from translog, ignoring soft deletes 38904
  • Fix NPE on Stale Index in IndicesService 38891

  Changes in 6.7:

  • ReadOnlyEngine should update translog recovery state information 39238
  • Align generated release notes with doc standards 39234
  • Rebuild remote connections on profile changes 37678
  • minor updates for user-agent ecs for 6.7 39213
  • Only create MatrixStatsResults on final reduction 38130
  • Link to 7.0 documentation in deprecation checks 39194
  • Ensure global test seed is used for all random testing tasks 38991
  • Bump jackson-databind version for AWS SDK 39183
  • Reduce refresh when lookup term in FollowingEngine 39184
  • Deprecate fallback to java on PATH 37990
  • Deprecate Hipchat Watcher actions 39160
  • Bump jackson-databind version for ingest-geoip 39182
  • Remove retention leases when unfollowing 39088
  • Resolve concurrency with watcher trigger service 39092
  • Allow retention lease operations under blocks 39089
  • Fix DateFormatters.parseMillis when no timezone is given 39100
  • Fix shard follow task startup error handling 39053
  • Specify include_type_name in HTTP monitoring. 38927
  • Introduce retention lease state file 39004
  • Generate mvn pom for ssl-config library 39019
  • Integrate retention leases to recovery from remote 38829
  • ShardBulkAction ignore primary response on primary 38901

  Changes in 6.6:

  • SQL: add "validate.properties" property to JDBC's allowed list of settings 39050
  • SQL: enforce JDBC driver - ES server version parity 38972
  • Fix simple query string serialization conditional 38960
  • Advance max_seq_no before add operation to Lucene 38879

  Changes in 6.5:

  • Build: Fix issue with test status logging 38799

  Changes in Elasticsearch Management UI

  Changes in 7.1:

  • [Rollup] Add unit tests for Job table 31561
  • [CCR] Add data-test-subj to forms and buttons 30325
  • [CCR] i18n feedback 30028

  Changes in Elasticsearch SQL ODBC Driver

  Changes in 7.1:

  • Enable DATE conversions 115

  Changes in 6.7:

  • Test: driver uninstallation done now with wmic only 116

  Changes in Rally

  Changes in 1.0.4:

  • Make types optional 647

  
  
  https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2019-02-22
• 

  Elastic Support Alert: Elastic Cloud Enterprise and the runc Vulnerability
  • sorangutan  

  1
  1
  Posts
  64
  Views

  

  Elastic is aware of the runc vulnerability in older versions of Docker, and its potential impact on Elastic Cloud Enterprise (ECE) customers. This blog explains the risks and recommends actions for mitigating impact on affected environments.

  Please note that while our Elasticsearch Service is based on ECE, it is continuously patched and monitored by our cloud team. No action is required from any Elasticsearch Service customers.

  Background

  For hosts running older versions of Docker engine, the runc vulnerability allows gaining administrative privileges on the host through an attacker controlled image or a compromised container. For more information and a list of impacted versions, please read this Docker Security Update.

  Potential Impact on ECE

  ECE does not allow running arbitrary docker images. Further, it incorporates a number of controls to prevent privilege escalation within a container. These include running the Elastic Stack as a non-root user, and removing suid flag in the suid binaries (like “sudo”) for images shipped with ECE that are not EOL.

  Since ECE is a software that allows managing Elasticsearch clusters in customer-owned infrastructure or cloud, ECE admins can naturally access and modify all deployments managed by ECE.

  The users of deployments managed by ECE (e.g. an Elasticsearch user) cannot modify containers nor the images in ECE (prerequisites for a runc attack). With the protections in place as described above, the risk of privilege escalation is further reduced.

  Recommended Actions for ECE

  Elastic recommends upgrading Docker engine and monitoring runc binaries on the hosts.

  Customers running ECE 2.x should upgrade to the latest version of Docker Engine. Customers running ECE 1.x will need to upgrade to ECE 2.1, which is required to run on the latest version of Docker engine. Customers can perform host maintenance prior to initiating the upgrade. Customers should consider monitoring runc binary for any changes by running Auditbeat on hosts.

  In general, it is important to keep up with the ECE upgrades to mitigate against these types of vulnerabilities. Please note that ECE 2.1 is fully backwards compatible with earlier versions and all of existing deployments and API usage will continue to work without changes.

  Further Assistance

  As always, our support team is available to answer questions you have regarding runc vulnerability. Please use your standard support channel to ask those questions. Elastic Consulting is also available to help. Our standard consulting packages, such as our Upgrade Strategy (8 Consulting Days) or ECE Jumpstart (16 Consulting Days), are a great fit for upgrade assistance. Reach out with any consulting questions you may have.
  
  https://www.elastic.co/blog/elastic-support-alert-elastic-cloud-enterprise-and-the-runc-vulnerability
• 

  From ELK Stack to Elastic Cloud Enterprise: Scaling Up Capabilities at John Deere
  • sorangutan  

  1
  1
  Posts
  62
  Views

  

  When it comes to farming, most people’s conceptions are outdated. “[T]hey think of some guy in bib overalls driving a tractor,” said Tim Arp, Senior Operations Engineer in the Intelligent Solutions Group (ISG) at John Deere. “But really, it’s all about the data now.” And that data — whether it’s GPS information powering self-driving equipment, metrics about machine performance, or even data about how crops are being planted — represents an immense amount of information pouring into John Deere’s systems.

  While many people associate the company with its ubiquitous “green machines,” John Deere’s data-based applications are just as crucial to agricultural professionals. Today, John Deere supports farming operations with machine and agronomic telematics applications that help farmers maximize yield and manage costs.

  At Elastic{ON} Tour Chicago, Arp shared how the company began this work with JDLink, their first mobile application. Originally built on 25 servers, JDLink provided farmers with an easy way of collecting data — including geographic location and machine health metrics — on farm equipment such as combines and harvesters. But as the application grew, requiring the deployment of more and more servers, the ISG began to encounter difficulty scaling, ultimately resulting in outages. It quickly became clear that to ensure that the application worked for the farmers who relied on it, the ISG needed a way to aggregate logs and consolidate data so they could visualize the system holistically.

  In 2013, the ISG found a solution: the Elastic Stack. Arp and a colleague began experimenting with Elasticsearch, putting together a working demo using version 0.98. Before long, they were able to build their first production cluster on version 1.4, using 4 nodes.

  “We started pulling in all kinds of logs: web server logs, application logs, system logs, load balancer logs. What we soon realized was all the analytics that we could get with this. No longer did we look at servers individually, but we looked at the system.” Tim Arp, Senior Operations Engineer, Intelligent Solutions Group (ISG), John Deere

  Arp and team also used Kibana to create useful dashboards for monitoring errors and Apache web logs, visualizing response codes and hits by server.

  Over the years, John Deere’s application platform has expanded to help farmers in a range of crucial areas. No longer focused solely on data management, the company offers applications to support remote management, guidance, variable rate application, and field and water management. As its offerings have grown, so has the organization’s use of Elasticsearch.  

  Their current environment, Logcentral@Deere, scaled up from 4 nodes to 25 to provide a logging framework for all of the ISG application development teams. Running Elastic Stack 6.3, the system handles 18 billion documents across 11 terabytes of storage. Roughly 14 different applications log to it at any one time, amounting to 20,000 events being streamed in every second. John Deere has also expanded Logcentral into the cloud, with 8 Elasticsearch clusters. The largest cluster, which has more than 100 applications logging to it, contains 51 nodes.

  Today, John Deere relies on Elastic Cloud Enterprise (ECE) for features such as support, integrated authentication and authorization, monitoring and alerts through the Elastic Stack’s alerting features (Watcher), cross cluster search, seamless upgrades, and machine learning. In the future, John Deere looks forward to making use of new capabilities such as rollups and data aggregations to maximize results for the farmers who count on them.

  Watch Tim Arp’s full session at Elastic{ON} Tour to learn more about how John Deere is using Elasticsearch to put critical data in the hands of agricultural professionals across the world.

  

  
  
  https://www.elastic.co/blog/from-elk-stack-to-elastic-cloud-enterprise-scaling-up-capabilities-at-john-deere
• 

  Easier Relevance Tuning in Elasticsearch 7.0
  • sorangutan  

  1
  1
  Posts
  300
  Views

  

  Improving relevance is hard, it really is. There are often a variety of parameters and ranking characteristics to potentially change: how much boost should the title field get vs the body?  Should you boost a category field, and if so, how much? Each of these are parameters you could manipulate to potentially get better (or worse!) results. Ideally you would employ a number of people to tag result sets for a large number of queries, and then optimize parameters that affect relevance to improve relevance metrics. The most common obstacle to this way of improving relevance is the creation of a training set, because it requires a lot of human time. Smaller organizations often end up guessing a scoring formula based on their knowledge of the domain that they are operating in. In any case, at some point you will need to combine textual relevance with other relevance signals such as popularity or authority. Elasticsearch 7.0 comes with some new tools that make this task easier.

  Most text ranking schemes — including BM25 (the default ranking scheme in Elasticsearch) — merge the score contributions of multiple terms via a sum. It makes it natural to incorporate more score contributions via a sum as well. However these linear combinations only work well if features follow comparable distributions. For this reason, users have been asking for ways to get normalized scores with Elasticsearch, which has a number of drawbacks and can't be done without collecting all matches and scores.

  A 2005 paper, Relevance Weighting for Query Independent Evidence, takes the opposite approach and engineers features in such a way that they look more like BM25 scores. In particular, one suggested function is called satu, which stands for "saturation", and is the function that is used to incorporate term frequency into BM25 scores. This paper has been influential to Elasticsearch 7.0 and lead to the development of two new fields: rank_feature and rank_features, as well as a new query called rank_feature that operates on these fields.

  New rank_feature and rank_features fields

  The rank_feature field works similarly to a regular float field from the outside, but indexes data in a way that allows Elasticsearch to run queries efficiently when it is used for ranking. rank_feature is typically used for measures of the relevance of a document such as popularity or authority, or irrelevance such as URL length in web search. rank_features works very similarly but is better suited for sparse features, such as a set of weighted tags or categories

  As often, an example is worth a thousand words, so here is what you would do when indexing web pages in order for scores to be computed as score = bm25_score + satu(pagerank) thanks to the rank_feature query:

  PUT test
  {
    "mappings": {
      "properties": {
        "pagerank": {
          "type": "rank_feature"
        },
        "content": {
          "type": "text"
        },
        "url": {
          "type": "keyword"
        }
      }
    }
  }
  
  PUT test/_doc/1
  {
    "url": "http://en.wikipedia.org/wiki/2016_Summer_Olympics",
    "content": "Rio 2016",
    "pagerank": 50
  }
  
  PUT test/_doc/2
  {
    "url": "http://en.wikipedia.org/wiki/2016_Brazilian_Grand_Prix",
    "content": "Formula One motor race held on 13 November 2016 at the Autódromo José Carlos Pace in São Paulo, Brazil",
    "pagerank": 20
  }
  
  PUT test/_doc/3
  { 
    "url": "http://en.wikipedia.org/wiki/Deadpool_(film)",
    "content": "Deadpool is a 2016 American superhero film",
    "pagerank": 35
  }
  
  POST test/_refresh
  
  GET test/_search
  {
    "query": {
      "bool": {
        "must": [
          {
            "match": {
              "content": "2016"
            }
          }
        ],
        "should": [
          {
            "rank_feature": {
              "field": "pagerank"
            }
          }
        ]
      }
    }
  }
  

  Note that these fields can neither be searched or aggregated. If you want to be able to search or aggregate them as well, you would need to use multi-fields to map these fields both as a float and a rank_feature field. For instance the below index creation call maps the pagerank field as a float and pagerank.feature as a rank_feature field:

  PUT test
  {
    "mappings": {
      "properties": {
        "pagerank": {
          "type": "float",
          "fields": {
            "feature": {
              "type": "rank_feature"
            }
          }
        }
      }
    }
  }
  

  The rank_feature query gives some options to change the function that is used to turn these features into scores: 3 different functions are available, which all have some tuning parameters. Unless you have a training set that you can use to tune these parameters, we recommend that you only use query boosts to tune the weight of your features compared to your main query and otherwise stick to the defaults which are the saturation function with parameters that are computed automatically from index statistics.

  These fields provide little flexibility but come with a great benefit: they work with top-k retrieval optimizations that we also brought to Elasticsearch 7.0, which often helps retrieve the top matches of a query while only collecting a small subset of matches. As a consequence, you don't need to sacrifice performance entirely in order to apply one or two features on the whole result set. On the other hand if you need more flexibility but sacrificing performance is acceptable, a better way is to go with scripting.

  Full flexibility with script_score

  Script Score Query, designed to replace Function Score Query, allows you to define a scoring formula for your documents through Painless scripting. Script Score Query comes with some predefined functions that can be used in combining relevance signals. Two of these functions are saturation and sigmoid. For example, the previous example with rank_feature can be approximated with the following example using a script_score query:

  GET /_search
  {
      "query" : {
          "script_score" : {
              "query" : {
                  "match": { "content": "2016"}
              },
              "script" : {
                  "source" : "_score + saturation(doc['pagerank'].value, 50))"
              }
          }
       }
  }
  

  Here, we define a document rank as a combination of the document’s textual score with its pagerank value. The pagerank value is transformed beforehand through a saturation function to give as a value from 0 to 1.

  The decay functions of the script_score query for numeric, geo or date fields may be used for the transformation of relevance signals of numeric, geo or date types. If your relevance signals have a form of dense or sparse vector fields, soon it will be possible to use these fields in scoring as well as we are adding vector functions to the script_score query.

  Using scripts you can also completely define your own scoring formula appropriate for your domain. For example, this blog about better result sorting presents a script for ranking products based on users’ ratings for an online retail industry.

  Conclusion

  While most of the work is still on the user side, we hope that these new tools are going to make improving relevance easier. We are especially excited that retrieving top matches doesn't require to collect all matches all the time, including when the score contains non-textual relevance signals, as this can result in significantly faster query results. We hope to bring similar tools in the near future for dynamic features that need to be computed on the fly such as recency or geo-distance.

  Download the 7.0.0 beta and give rank_feature(s) and script_score a go, and let us know what you think. Become an Elastic Pioneer and you’ll also have a chance to win some cool Elastic swag.

  
  
  https://www.elastic.co/blog/easier-relevance-tuning-elasticsearch-7-0
• 

  Elastic Stack 6.6.1 and 5.6.15 Released
  • sorangutan  

  1
  1
  Posts
  55
  Views

  

  Versions 5.6.15 and 6.6.1 of the Elastic Stack were released today. We recommend you upgrade to these latest versions.

  Each includes fixes for a number of security issues in Kibana, Elasticsearch, and Logstash.

  • Resolved a cross-site scripting (XSS) vulnerability in Kibana that could allow an attacker to obtain sensitive information or perform destructive actions.
  • Fixed an issue in the Timelion application in Kibana that could allow an attacker to attempt to execute javascript code.
  • Fixed an issue with Kibana that could allow an attacked to attempt to execute javascript code when audit logging was enabled.
  • Fixed an issue in Elasticsearch that would give an attacker additional permissions against a restricted index when using the _aliases, _shrink, or _split endpoints.
  • Fixed an issue with Logstash where it would inadvertently log credentials as part of an error message.

  For a detailed explanation of these issues, and details on how to solve or mitigate these issues, please visit the security advisory page .

  The 6.6.1 patch contains fixes and small enhancements for the stack. Notable bug fixes in Beats include:

  • Packetbeat no longer crashes on Linux when the TPACKET_V3_af_packet_interface is used. (#10477)
  • Correctly stop all modules when they were started by Kubernetes autodiscover.( #10476)

  For a full list of changes for each product, please refer to the release notes:

  6.6.1 Release Notes

  • Kibana
  • Elasticsearch
  • Beats
  • Logstash

  5.6.15 Release Notes

  • Kibana
  • Elasticsearch
  • Beats
  • Logstash

  
  
  https://www.elastic.co/blog/elastic-stack-6-6-1-and-5-6-15-released
• 

  Distributed Tracing, OpenTracing and Elastic APM
  • sorangutan  

  1
  1
  Posts
  278
  Views

  

  The World of Microservices

  Enterprises are increasingly adopting microservice architectures. They are developing and deploying more microservices everyday. Often, these services are developed in different programming languages, deployed into separate runtime containers, and managed by different teams and organizations. Large enterprises like Twitter can have tens of thousands of microservices, all working together to achieve their business goals. As they discussed in this Twitter blog post, visibility into the health and performance of the diverse service topology is extremely important for them to be able to quickly determine the root cause of issues, as well as increasing Twitter’s overall reliability and fficiency.

  

  This is where Distributed Tracing can really help. Distributed Tracing helps with two fundamental challenges faced by microservices:

  1. Latency tracking
     One user request or transaction can travel through many different services in different runtime environments. Understanding the latency of each of these services for a particular request is critical to the understanding of the overall performance characteristics of the system as a whole, and provides valuable insight for possible improvements.
  2. Root cause analysis
     Root cause analysis is even more challenging for applications that build on top of large ecosystems of microservices. Anything can go wrong with any of the services at any time. Distributed tracing is of crucial importance when debugging issues in such a system.

  Take a step back, tracing is only one piece of the puzzles of the Three Pillars of Observability - Logging, Metrics and Tracing. As we will discuss briefly, Elastic Stack is a unified platform for all three pillars of observability. When logs, metrics, and APM data are stored in the same repository, analyzed, and correlated together, you gain the most context-rich insight into your business applications and systems. In this blog, we will solely focus on the tracing aspect.

  

  Distributed Tracing with Elastic APM

  Elastic APM is an application performance monitoring system built on the Elastic Stack. It allows you to monitor software services and applications in real time, collecting detailed performance information on response time for incoming requests, database queries, calls to caches, external HTTP requests, etc. Elastic APM agents offer rich auto-instrumentation out of the box (e.g. timing db queries, etc.) for supported frameworks and technologies. You can also use custom instrumentation for custom purposes. This makes it much easier to pinpoint and fix performance problems quickly.

  Elastic APM supports distributed tracing and is OpenTracing compliant. It enables you to analyze performance throughout your microservice architecture all in one view. Elastic APM accomplishes this by tracing all of the requests, from the initial web request to your front-end service, to queries made to your back-end services. This makes finding possible bottlenecks throughout your application much easier and faster. The Timeline visualization in APM UI shows a waterfall view of all of the transactions from individual services that are connected in a trace:

  

  Elastic Stack is also a great platform for log aggregation and metrics analytics. Having logs, metrics, and APM traces all stored and indexed in Elasticsearch is super powerful. Being able to quickly correlate data sources like infrastructure metrics, logs and traces enables you to debug the root cause much faster. In the APM UI, when looking at a trace, you can quickly jump to the host or container metrics and logs by clicking the Actions menu, if these metrics and logs are also collected.

  

  It would be wonderful if everybody was using Elastic APM to instrument their applications and services. However, Elastic APM is not the only distributed tracing solution available today. There are other popular open source tracers like Zipkin and Jaeger. Concepts like polyglot programming and polyglot persistence are well known and well accepted in the world of microservices. Similarly, “polyglot tracing” is going to be more common than not. Because of the independent and decoupled nature of microservices, people responsible for different services will likely use different tracing systems.

  Challenges for Developers

  With many different tracing systems available, developers are faced with real challenges. At the end of the day, tracers live inside the application code. Some common challenges are:

  1. Which tracing system to use?
  2. What if I want to change my tracer? I don’t want to change my entire source code.
  3. What do I do with shared libraries that might be using different tracers?
  4. What if my third-party services use different tracers?

  Not surprisingly, we need standardization to address these concerns. Before discussing where we are with the standardization, let’s take a step back and look at distributed tracing from an architectural perspective in a holistic manner and understand what’s required to achieve the distributed tracing “nirvana”.

  Architectural Components of Distributed Tracing

  Modern software systems can be broken down into a few high-level components, typically designed and developed by different organizations and run in different runtime environments.

  • Your own application code and services
  • Shared libraries and services
  • External services

  To monitor such a system in a holistic and integrated fashion with distributed tracing, we need four architectural components:

  

  1. Standardized distributed tracing API. A standardized vendor-neutral tracing API allows developers to instrument their code in a standardized way, does not matter what tracer they might choose to use later during the runtime. This is the first step towards anything.
  2. Standardized tracing context definition and propagation. For a trace to go across from one runtime to another, the tracing context has to be understood by both parties, and there has to be a standard way of propagating that context. At the minimum, the context carries a trace ID.
  3. Standardized tracing data definition. For trace data from one tracer to be understood and consumed by another tracer there has to be a standardized and extensible format for it.
  4. Interoperable tracers. Finally, to achieve 100% runtime compatibility, different tracers have to provide mechanisms for them to both export and import trace data from other tracers in an open way. Ideally, a shared library or service instrumented by a tracer like Jaeger should be able to have it’s tracing data sent directly to Elastic APM or another tracer via the Jaeger agent through a configuration change.

  Now, enter OpenTracing.

  The OpenTracing Specification

  The OpenTracing specification defines an open, vendor-neutral API for distributed tracing. It enables users to avoid vendor lock-in by allowing them to switch the OpenTracing implementer at any time. It also enables developers of frameworks and shared libraries to provide tracing functionality out of the box, in a standard fashion to enable better insights into the frameworks and libraries. Web-scale companies like Uber and Yelp are using OpenTracing to get deeper visibility into their highly distributed and dynamic applications.

  The OpenTracing Data Model

  Basic concepts of OpenTracing and the fundamental data model came from Google’s Dapper paper. Key concepts include trace and span.

  

  1. A trace represents a transaction as it moves through a distributed system. It can be thought of as a directed acyclic graph of Spans.
  2. A span represents a logical unit of work that has a name, start time, and duration. Spans may be nested and ordered to model relationships. Spans accept key:value tags as well as fine-grained, time-stamped, structured logs attached to the particular span instance.
  3. Trace context is the trace information that accompanies the distributed transaction, including when it passes the service to service over the network or through a message bus. The context contains the trace identifier, span identifier, and any other data that the tracing system needs to propagate to the downstream service.

  How does it all fit in?

  Ideally, with standardization, tracing information from custom application code, shared libraries, and shared services developed and run by different organizations are exchangeable and runtime compatible, does not matter what tracer each of these components chose to use.

  

  However, OpenTracing only addresses the first of the four architectural components we discussed before. So, where are we today with other components and what the future holds for us?

  Where are We Today?

  As we discussed, OpenTracing defines a standard set of tracing APIs for different tracers to implement, which is a great start and very encouraging. However, we still need tracing context standardization and tracing data standardization for them to be compatible and exchangeable with each other.

  1. OpenTracing API provides a standard set of APIs. This is pretty much the only standardization we have as of today. There is limitation to the specification too. For example, it does not cover all the programming languages. Nevertheless, it’s a wonderful effort and gaining great traction.
  2. No standardized tracing context definition yet. The W3C Distributed Tracing Work Group is in the process of standardizing the tracing context definition - the W3C Trace Context specification. The specification defines a unified approach to context and event correlation within distributed systems, and will enable end-to-end transaction tracing within distributed applications across different monitoring tools. Elastic APM supports the W3C Trace Context working group's effort to standardize the HTTP header format for distributed tracing. Our agent implementations closely follow the Trace Context draft specification, and we intend to fully support the final specification.
     
     As an example of the incompatibility of the tracing context today, here is an example of the HTTP header used by Elastic APM and Jaeger for trace ID. As you can see, both the name and encoding of the ID are different. When different tracing headers are used, traces will break when they cross the boundaries of the respective tracing tools.
     
     Jaeger:
     uber-trace-id: 118c6c15301b9b3b3:56e66177e6e55a91:18c6c15301b9b3b3:1
     
     Elastic APM:
     elastic-apm-traceparent: 00-f109f092a7d869fb4615784bacefcfd7-5bf936f4fcde3af0-01
     
     There are other challenges too, other than the definition itself. For example, not all HTTP headers are automatically forwarded by service infrastructure and routers etc. Whenever headers are dropped, the trace will break.
  3. No standardized tracing data definition yet. As stated by the W3C Distributed Tracing Work Group, the second piece of the puzzle for trace interoperability is “a standardized and extensible format to share trace data -- full traces or fragments of traces -- across tools for further interpretation”. As you can imagine, with many open source and commercial players involved, agreeing on a standard format is not an easy thing. Hopefully we will get there soon.
  4. Tracers are not runtime-compatible. Because of everything we discussed above, plus mixed motivation of making their system open and compatible with the rest of the world, tracers are simply not compatible with each other during the runtime today. I can confidently say that it will probably be that way for the foreseeable future.

  How Elastic APM Works with Other Tracers Today

  Even though we are not even close to 100% compatibility among tracers yet today, there is no need to be discouraged. Elastic Stack can still work with other tracers in a couple of different ways.

  1. Elasticsearch as the scalable backend data store for other tracers.
     
     Not surprisingly, Elasticsearch has been used as the backend data store for other tracers like Zipkin and Jaeger, due to its massive scalability and rich analytics capabilities. Shipping Zipkin or Jaeger tracing data into Elasticsearch is a simple configuration for both of them. Once the tracing data is inside Elasticsearch, you can use the powerful analytic and visualization capability of Kibana to analyze your tracing information and create eye-catching visualizations that provide deep insight into your application performance.
  2. Elastic OpenTracing Bridge
     
     The Elastic APM OpenTracing bridge allows you to create Elastic APM Transactions and Spans, using the OpenTracing API. In other words, it translates the calls to the OpenTracing API to Elastic APM and thus allows for reusing existing instrumentation. For example, an existing instrumentation done by Jaeger can be simply replaced with Elastic APM by changing a couple of lines of code.
     
     Original instrumentation by Jaeger:
     
     

     import io.opentracing.Scope;
     import io.opentracing.Tracer;
     import io.jaegertracing.Configuration;
     import io.jaegertracing.internal.JaegerTracer;
     ...
     private void sayHello(String helloTo) {
         Configuration config = ...
         Tracer tracer = config.getTracer();
         try (Scope scope = tracer.buildSpan("say-hello").startActive(true)) {
             scope.span().setTag("hello-to", helloTo);
         }
         ...
     }
         

     Replace Jaeger with Elastic OpenTracing bridge:
     
     

     import io.opentracing.Scope;
     import io.opentracing.Tracer;
     import co.elastic.apm.opentracing.ElasticApmTracer;
     ...
     private void sayHello(String helloTo) {
         Tracer tracer = new ElasticApmTracer();
         try (Scope scope = tracer.buildSpan("say-hello").startActive(true)) {
             scope.span().setTag("hello-to", helloTo);
         }
         ...
     }
         

     
     
     With this simple change, the tracing data will be happily flowing into Elastic APM, without you having to modify other tracing code. That’s the power of OpenTracing!

  Elastic APM Real User Monitoring

  While we mostly focus on the backend services when discussing tracing and context propagation etc., there is great value to start the trace on the client side in the browser. When doing so, you get trace information the moment a user clicks on something in the browser. That trace information represents the “real user experience” of your applications from performance aspect. Unfortunately again, there is no standardized way of forwarding that information today. The W3C group does intend to extend the trace context all the way to the browser in the future.

  Elastic APM Real User Monitoring (RUM) provides exactly that functionality today. The RUM JS agent monitors the real user experience within your client-side application. You will be able to measure metrics such as "Time to First Byte", domInteractive, and domComplete which helps you discover performance issues within your client-side application as well as issues that relate to the latency of your server-side application. Our RUM JS agent is framework-agnostic which means that it can be used with any JavaScript-based frontend application.

  
  
  https://www.elastic.co/blog/distributed-tracing-opentracing-and-elastic-apm

• 

  KPN Harnesses the Elastic Stack to Thwart Security Data Overload
  • sorangutan  

  1
  1
  Posts
  56
  Views

  

  This post is a recap of a community talk given at a recent Elastic{ON} Tour event. Interested in seeing more talks like this? Check out the conference archive or find out when the Elastic{ON} Tour is coming to a city near you.

  KPN Security Services is a SOC/SIEM security incident and threat-event management company based in the Netherlands. Having the philosophy that everyone should be able to connect online free of exploits and vulnerabilities, the company’s security offerings include preventing, detecting, responding to, and monitoring for threats.

  Han Pieterse, a KPN senior solution architect, and Marius Iversen, who at the time was a KPN senior platform engineer, presented at Elastic{ON) Tour Amsterdam and discussed how the Elastic Stack was leveraged to power security operations for KPN and their global customers.

  The problem facing KPN was that while its customer base was growing and scaling, the amount of data it had to manage was endlessly multiplying as well. Knowing that data volumes would never cease growing and would become chaotic to manage, the company embraced the Elastic Stack as the solution.

  Visibility gaps

  Among other reasons, the move to the Elastic Stack was to prevent “visibility gaps in our day-to-day business” and to enable the 400-member KPN security team to stick to the core function of preventing and thwarting attacks on its customers instead of dealing with data management and data collection headaches, Pieterse says.

  Pieterse says that an issue the company was confronting boiled down to “security data overload.” That data glut could bring “risk in our business” such as misinterpretation of threats and mismanagement of collected machine data, in addition to delays in threat detection, he says.

  “To be secure, to be also operational in the future, we have to handle that volume,” he says.

  The Elastic Stack is assisting KPN Security Solutions toward attaining several goals:

  • Controlling “data chaos,” complexity, and costs
  • Gaining more complete, and accurate, security analysis and visibility
  • Managing data sources, streams, and destinations with flexibility
  • Using security resources more efficiently

  Pre-Elastic Stack

  The high-level view of the company, before adopting the Elastic Stack, was essentially a setup that was single-tenant, manually configured, had a basic API, and normalized data only.

  A key limitation that would eventually throttle growth was KPN’s single-tenant implementation for each customer, Iversen says.

  “As we get more and more customers, it's going to get really hard to manage because you always have to grow your own team of engineers at the same rate as you get customers because you get so many more clusters — there's so much manual configuration. And that’s not going to work in the long run,” Iversen says.

  KPN’s deployment supports non-CEF, harnesses the ArcSight Logstash plugin, TLS, and seizes on the benefits of having multiple pipelines funneling through Kibana dashboards.Data ingestion with the ArcSight Logstash plugin, Iversen says, was “a lot easier than we anticipated.”

  “We only really needed to flip the switch. We really didn't need to do anything else. We just set up Logstash, installed the plugin, and it already works,” he says. “We’re only going to scale this larger, and larger.”

  Machine Learning as a Service

  Iversen says KPN is going to build interesting use cases with the Elastic Stack machine learning capabilities — for both non-security and security purposes alike.

  “It's going to be big for us on security breaches,” he says, noting that machine learning can monitor hundreds or thousands of servers simultaneously “for new processes, file changes, and so on.”

  And if there is a breach, he says, “We can much faster come to the conclusion of how far the breach went.”

  Machine learning is also good for security monitoring in finance by looking for irregularities in money transfers. And it can also look for irregularities in employee behavior, from access to buildings via monitoring security badges to finding anomalies in worker data usage, he says.

  When it comes to machine learning outside the security context, Iversen says, “We want to use machine learning to run forecasts, depending on both the performance and resource requirements for our platform.”

  One key issue KPN wants to head off is a nightmare for any organization: running out of memory space.

  As part of KPN’s machine learning, Iversen says, they’ll ask the program: “How much memory do you think we’re going to need for the next 6 months?”

  Watch and listen to Iversen and Pieterse as they get into more of the details on indexing, sharding, censoring data, storage over time, benchmarking, multi-tenancy implementation, encryption, naming conventions, third-party integrations, event brokering and alerting, and Kibana pipeline management.

  

  
  
  https://www.elastic.co/blog/kpn-harnesses-elastic-stack-to-thwart-security-data-overload
• 

  Keeping up with Kibana: Welcome to 2019! (catch up edition)
  • sorangutan  

  1
  1
  Posts
  78
  Views

  

  Welcome to Kibana weekly update!

  Happy 2019! 

  Lots of news to share from the last month while Kibana Weekly blog was on vacation. But first, yes, we are still hiring...

  Openings

  • Senior Product Manager, Kibana (Platform & Security)
  • Kibana UI/Front end Team Lead
  • Kibana Platform - Principal JavaScript Engineer
  • Kibana - Senior JavaScript Engineer
  • Kibana - Senior Security Developer (JavaScript)
  • Kibana - Visualisations & Vega Engineer

  7.0 is coming!

  7.0 Beta is coming soon. Here is a sneak preview of the new chrome and the light/dark themes that will be part of that release:

  

  Introducing Elastic Charts

  We've published the first version of elastic-charts on npm.  This library will replace the existing EUI component (as well as other chart library implementations that we currently use) and will include new features and functionality in the next phase.

  Feel free to check out the various styles and features here.

  Security

  Kibana now has a content security policy! 

  The new Role Management UI is in! We also began work on removing deprecated functionality for 7.0, including the ability to rely upon direct index access to Kibana and some legacy ssl settings. 

  Progress continues on integrating the various applications with Feature Controls. After some deliberation, the method in which APM, Monitoring and Machine Learning will be exposed using feature controls has been decided, and the PR is going through review. We’ve enabled feature controls in Maps, Advanced Settings, Maps; and we’re actively working on integrating them into the remaining application. 

  Localized Kibana

  We have been working on wrapping up localization and Chinese translations for 7.0.

  We also worked on how to better support/enable users to translate in their preferred languages while waiting for Elastic supported and maintained translation.

  Middleware/Alerting/Notifications/Scheduling/Secrets

  With the help of the infamous kibana security team the changes to the saved object service for hiding saved objects from the default repository are coming along but still in review. The secret store has been renamed to the secret service and is ready for early review and feedback #28894 . FYI, a more thorough definition of the phases in Alerting meta issue is currently a work in progress.

  Browser testing coverage

  Have had a successful build with WebDriver (replacing Leadfoot library) and most tests also pass on Firefox #26477

  Platform
  

  New platform: The initial review of the new Elasticsearch service is done, and now work turns toward polish and test coverage. This will be the new definitive way to interact with Elasticsearch, but it'll seamlessly power the legacy elasticsearch plugin capabilities. The first existing plugin to start migrating functionality over to the new platform will be canvas! The canvas team had rolled their own plugin system awhile back in order to be more flexible with their expression model, but their functions are relatively decoupled from the legacy bits of Kibana, so they're a great candidate to move to our new plugin system.

  The saved object service now supports first-class object relationships. This is a breaking change in 7.0 that will allow us to implement much more robust object interactions and sophisticated import/export functionality.

  You can now specify whether you want spaces to act as OR or AND when searching with the saved object client.

  An extensible help menu was merged for 7.0 onward. This links off to Kibana documentation by default and allows individual apps to register their own help content.

  All previously deprecated elasticsearch.* settings have been removed for 7.0 onward.

  Visual Testing  

  • Evaluating visual test tools: Percy and Applitools: Kibana Slides.  
    • Percy highlights: DOM Snapshotting and rendering across browsers
    • Applitools highlights: Nice way to omit verification from parts of the screen and allowing multiple matching options.  
  • Working with Percy support to get canvas elements to render in their tool and re-doing examples with Kibana.  
  • Evaluating Cypress with use with Percy for visual testing.  
  • Walter gave a nice demo of his work.

  Team summaries

  Design

  The design team is mostly in polish and feedback mode now against 7.0. We took in a lot of feedback on the nav changes and made some subtle adjustments there with timings that should make those interactions more pleasant, but there’s still work to do there. Thanks to Matt and Nathan, the EUI datepicker is now is use across the discover, dashboard and visualize applications. Now that the core theming / sass system is complete we’re able to go through each app and do lots of cleanup work. Last week we focused on the Dasbhboard and Discovery apps, but have plans to do similar cleanup for the others as well.

  On the EUI front, the dual range slider was merged and available in Kibana master. We’ve locked the 6.7 and 7.0 branches at [email protected] and will only be releasing patch releases there if we can manage. We got an unexpected and much loved PR from Liza to provide some i18n deugging tools which we hope to get to master soon. As part of our cleanup around general security and dependency improvements in Kibana we went ahead and updated lodash to version 4. This will not go in with 7.0 and we’ll monitor for any changes downstream it might cause.

  PRs

  • Make map gradients more subtle #30345
  • Slight tweak with dark bg color #1530
  • K7 Discover app cleanup #30037
  • Homepage style updates #30341
  • EUI 6.10.0 & Background color fixes #30136
  • EUI-ify the empty prompt view #4
  • [K7] Side nav interaction adjustments #29978
  • [K7 navigation] Reposition Discover tooltip and hide chrome on welcome #29769
  • Feature/more i18n tokens #1504Move pagination react keys onto wrapping elements #1514
  • Upgrade lodash to v4 #1534
  • EuiDualRange component #1485
  • Pause `EuiOutsideClickDetector` in `EuiPopover` and `EuiColorPicker` #1527

  

  
  

  
  

  
  

  Geo

  Maps App

  • Polish for beta in 6.7/7.0 continued, notably:
    • better error handling and more useful warning and error messages
    • upgraded mapbox dependency for smoother line visualizations
    • improve sample data for all three sample data sets
    • Integrate geotile_grid aggregation as default clustering mechanism in 7.0. This improves the look and feel of heatmaps. The app can now preserve the same aspect ratio of the grid at each scale.

  
  

  
  

  • Kickstart effort on getting embeddability of maps in dashboards. This is a major next feature.

  Data Production

  • Work on deprecating France and US zip code layers and replace them with newer ones

  Operations

  • Upgrade assistant telemetry (#28878)
  • Support for generating docker image on kibana build (#28380)
  • Upgrade thread-loader (#27518)

  KibanaApp (Visualizations, Sharing, Discover)
  

  The team focused on and will focus in the next weeks on creating a very solid 7.0 release, meaning a lot of bug fixes incoming, stabilizing features, finishing the last remaining EUI changes and refactoring (for better maintenance).

  Made significant design improvements on Dashboard (including responsive layout for dashboards!). Also we're busy moving the saved object client (browser side code) to TypeScript, since it's one of the core APIs used in a lot of places.
  

  PRs

  • Enable query bar datepicker in core apps where appropriate (#30171)
  • Fix/reporting vis preserve (#30456)
  • Don't trigger autorefresh when autofetch false (#30405)
  • Remove filterbar for markdown visualizations (#30354)
  • fixes gauge filters (#30057)
  • Correct label for OS vis in logs sample data (#30249)
  • Add autocomplete to TSVB (#30158)
  • Fix filter context menu overlapping with left side bar (#30020)
  • Makes KQL the default query language for new searches (#27092)
  • Fix legend overflow bug on dashboard vis (#29948)
  • Hide dashboard edit mode on small screen sizes (#30103)
  • Replace angular timepicker with EuiSuperDatePicker (#29204)
  • Use Elastic Maps Service v7.0 (#30006)
  • Toggle between Expression Pipeline & Visualize Data loader (#29874)
  • Enable orderBy and orderAgg in vis editor for rollup visualizations (#29894)
  • Timelion/hide timelion (#30131)
  • [OSS-Telemetry] Make the plugin have the right config prefix (#30005)
  • Fix TSVB flot chart render after index pattern change (#29949)
  • Microsoft SQL Server Kibana Tutorial (#29987)
  • Prepare vis data loader cutover (#29873)
  • Introduce `I18nService` core service and `I18nContext`. (#29379)
  • Fix fieldformatter calls in pipeline helper (#29842)

  Elastic-Chart PRs:

  • fix(offscreen canvas): set negative position to move offscreen (elastic/elastic-charts#50)
  • fix(axis labels): offset previous space correctly (elastic/elastic-charts#45)
  • ci(travis): fix storybook build (elastic/elastic-charts#49)
  • build(storybook): fix ci storybook deploy (elastic/elastic-charts#48)
  • ci(travis): fix deploy phase description (elastic/elastic-charts#47)
  • build: prepare build for semantic-release (elastic/elastic-charts#46)
  • fix(axis): scale tick labels to fix text truncation on chrome (elastic/elastic-charts#38)
  • refactor(axis/grid): render grid lines in separate layer from axis (elastic/elastic-charts#30)
  • fix: add missing min interval compare function (elastic/elastic-charts#37)
  • fix(axis): add axisTitleHeight to axis increments (elastic/elastic-charts#29)
  • fix: legend titles respects series ids or groups (elastic/elastic-charts#28)

  Cheers,Kibana

  
  

  
  
  https://www.elastic.co/blog/keeping-up-with-kibana-2019-02-15
• 

  This Week in Elasticsearch and Apache Lucene - 2019-02-15
  • sorangutan  

  1
  1
  Posts
  63
  Views

  

  Elasticsearch Highlights

  API Keys

  We added the ability to get the enabled status for both the token service and API key service to the xpack usage API.

  Zen2

  Zen2 is ready for release and has met all goals for 7.0. Follow-up work focuses on extending the documentation, clean-ups, benchmarks, and promotional and educational material. We are investigating how Zen2 behaves in large clusters up to 100 nodes. This work is based on the Stack/Terraform project, which has recently seen a lot of use for Cross Cluster Replication as well. Scalability tests focus on master elections, general cluster formation and cluster state publishing performance. Typical setups (3 dedicated master nodes) show sub-second elections, and even larger setups (e.g. 51 master-eligible nodes, which is definitely not a setup we would recommend) still show that elections are possible, even though they're slower. Additional testing for typical setups (e.g. 3 dedicated master nodes and 100 data nodes) as well as large cluster states is in the works.

  Build

  • We have added Fedora 29 and removed Fedora 27 from our packaging tests.
  • We have a bwcTestSnapshots task now that runs all the bwc tests that gradle check would without running any other tests.

  Deprecations

  We deprecated in 6.7.0 and removed support in 7.0.0 for using the low-level REST client on JDK 7. JDK 7 has been EOL since April 2015, and JDK 8 was marked as EOL in January 2019. Removing support for JDK 7 enables us to modern language features when developing and maintaining the client.

  Ingest node

  We added support for the ecs flag in the user_agent processor in 7.0. This flag allow the user_agent processor to emit Elastic Common Schema compatible output. This flag was introduced in 6.7 and was originally scheduled to be removed in 7.0. However, to help Beats consumers with the migration from 6.7 to 7.0, this flag is now preserved in 7.0, but will now default to true (use ECS output). The flag will be removed in 8.0.

  Geo Performance

  We have been working on a new radix partitioning for BKD segment merging, which gave great improvements to indexing throughput and temp disk usage. We have continued to optimize the algorithm by re-using objects when working on-heap, netting an additional 12-17% improvement in indexing throughput.

  We merged geopoints-as-geoshape benchmark into rally.

  We have been working on creating a new nightly geo_shape benchmarks specifically for comparing performance of different shape implementations. Lucene's nightly geo benchmarks uses a corpus of 60.8 million points from the Planet OSM data set to measure index and query performance. While its good practice to measure incremental performance improvements as code is committed or changed, comparing the performance between LatLonPoint / Geo3D, and LatLonShape on the same graph is misleading and an apples and oranges comparison since LatLonShape is a higher dimension index targeted for indexing both shapes and points.

  What we really need are performance comparisons between the different shape indexing and search implementations (BKD shapes, vs PrefixTree approaches: geohash, quadtree, composite). Preliminary results for 60M shapes are extremely promising:

  • BKD shapes are 45454.55 X more accurate (0.11 cm vs 50 m)
  • BKD shape index has 2.4 X smaller index (9.94 GB vs 24.13 GB)
  • BKD shape indexes 2.4 X faster (1hr 35 mins vs 3hrs 48 mins)
  • BKD shape queries are 50% faster (6.85 QPS vs 4.5 QPS)

  Types Removal

  We adjusted the deprecation behavior in 6.7 to be more lenient. It will be very common to set include_type_name=true in 6.7 so we've decided to emit a deprecation warning only if include_type_name is not set at all. The warning serves as an important note to users that the request and response format of these APIs will change in a breaking way in 7.0.

  Watcher UI

  We fixed a lot of bugs in master on the watcher UI this week, disabling unsupported actions for threshold alerts, fixing watch deletion, and adding action types

  Changes

  Changes in Elasticsearch

  Changes in 8.0:

  • Enable JsonThrowablePatternConverterTests with more debug 38721

  Changes in 7.1:

  • Add enabled status for token and api key service 38687
  • _cat/indices with Security, hide names when wildcard 38824
  • SQL: Implement :: cast operator 38774
  • Fix failing bwc test against 6.3 38770
  • Clean up ShardSearchLocalRequest 38574
  • mute Failing tests related to logging and joda-java migration 38704
  • Mute RetentionLeastIT.testRetentionLeasesSyncOnRecovery on 7x 38597

  Changes in 7.0:

  • [Monitoring] Remove include_type_name parameter from GET _template request 38818
  • Fix line separators in JSON logging tests 38771
  • Make 7.0 like 6.7 user agent ecs 38757
  • [Monitoring] Remove _type usages in _search requests 38819
  • Format Watcher.status.lastChecked and lastMetCondition 38626
  • Mute failing test 20_mix_typless_typefull 38781
  • Enable IndexActionTests and WatcherIndexingListenerTests 38738
  • Tie break on cluster alias when merging shard search failures 38715
  • Restore date aggregation performance in UTC case 38221
  • Move testToUtc test to DateFormattersTests 38610
  • Reject setting index.optimize_auto_generated_id after version 7.0.0 28895
  • BREAKING: Drop support for the low-level REST client on JDK 7 38540
  • Mute failing ApiKeyIntegTests 38614
  • Mute RetentionLeastIT.testRetentionLeasesSyncOnRecovery on 7.0 38600

  Changes in 6.7:

  • Remove immediate operation retry after mapping update 38873
  • Filter out upgraded version index settings when starting index following 38838
  • Look up connection using the right cluster alias when releasing contexts 38570
  • Handle the fact that ShardStats instance may have no commit or seqno stats 38782
  • Fix PreConfiguredTokenFilters getSynonymFilter() implementations 38858
  • Ensure that maxConcurrentShardRequests is never defaulted to 0 38734
  • [TEST] address testCollectNodes rare failure 38559
  • Fix LocalIndexFollowingIT#testRemoveRemoteConnection() test 38709
  • Only issue a deprecation warning if include_type_name is not set. 38825
  • Copy retention leases when trim unsafe commits 37995
  • Fix PreConfiguredTokenFilters getSynonymFilter() implementations 38839
  • Suppress error message when /proc/sys/vm/max_map_count is not exists. 35933
  • Add dedicated retention lease exceptions 38754
  • Introduce retention lease actions 38756
  • Enable removal of retention leases 38751
  • Prefix java formatter patterns with '8' 38712
  • Mute RetentionLeastIT#testRetentionLeaseSyncedOnRemove 38765
  • Specialize pre-closing checks for engine implementations 38702
  • Concurrent file chunk fetching for CCR restore 38495
  • Prevent possible assertion failure in IndicesQueryCache.close 38714
  • Deprecate the low-level REST client on JDK 7 38542
  • Mute IndexFollowingIT.testIndexFallBehind 38618
  • Mute RemoveCorruptedShardDataCommandIT 38616
  • Mute RetentionLeastIT.testRetentionLeasesSyncOnRecovery on 6.7 (#38597) 38601

  Changes in 6.6:

  • Skip BWC tests in checkPart1 and checkPart2 38730
  • Use consistent view of realms for authentication 38815
  • Only flush Watcher's bulk processor if Watcher is enabled 38803
  • Fix synchronization in LocalCheckpointTracker#contains 38755
  • Enhance parsing of StatusCode in SAML Responses 38628
  • Fix GeoHash PrefixTree BWC 38584
  • SQL: Relax StackOverflow circuit breaker for constants 38572
  • SQL: Prevent grouping over grouping functions 38649
  • Enable Dockerfile from artifacts.elastic.co 38552

  Changes in 6.5:

  • SQL: fall back to using the field name for column label 38842

  Changes in Elasticsearch Management UI

  Changes in 7.1:

  • refactoring license management server routes 30845
  • refactoring server routes in index management to use common code 30299
  • [Watcher] Add Index, HipChat, PagerDuty, and Jira Action types on the client 30043

  Changes in 7.0:

  • Fix width of Watcher table. 30311
  • Fix ILM dependency upon xpack.index_management.enabled setting. 30592
  • [CCR] Put back API integration test for follower indices 30260
  • Rollup job Functional UI test 30280

  Changes in 6.7:

  • disabling actions for threshold alerts that have default actionType 31129
  • [CCR] Fix plugin order collision 30596

  Changes in 6.6:

  • fixing watch delete 30989

  Changes in Elasticsearch SQL ODBC Driver

  Changes in 7.1:

  • ODBC Installer API for system info DSN access 110

  Changes in 7.0:

  • Integration testing: uninstall any old driver before attempting a new install 114
  • Integration testing: print installer log on failure 113
  • Integration tests fixes for current ES 112

  Changes in 6.6:

  • Init all record fields 111

  Changes in Rally

  Changes in 1.0.4:

  • Add node_name in node-stats docs for ... 646
  • Allow collection of jvm gc section in node-stats telemetry device 644

  Changes in Rally Tracks

  • Adds a track that indices geopoints as geoshapes (backport to 6) 65
  • Adds a track that indices geopoints as geoshapes 62

  Apache Lucene

  Lucene 7.7

  Lucene 7.7 has been released on February 11th, and Elasticsearch 6.7 now depends on this new version.

  Lucene 8.0

  We built a first release candidate of 8.0. Unfortunately a Solr bug has been discovered at about the same time that will likely require us to do a 7.7.1 release and postpone 8.0 a bit.

  Other

  • We are optimizing how Lucene builds BKD trees. It's exciting that indexing points in a shape field is now as fast as indexing points in a points field in 7.x.
  • There seem to be code paths where IndexWriter fails to consider exceptions as tragic.
  • We revived an old feature request that allows visiting query trees.
  • forcemerge merges more than necessary.
  • We are exploring leveraging payloads in intervals.

  
  
  https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2019-02-15
• 

  Life@ Spotlight | Nicole LaJeunesse: Running with Heart
  • sorangutan  

  1
  1
  Posts
  54
  Views

  

  Want to know what Life at Elastic is like? You've come to the right place. Hear from our Elasticians first hand as we highlight their stories and photos from around the world. In this Spotlight, Nicole LaJeunesse shares her journey from suffering a sudden cardiac arrest at age 22, to recovery on the road as a runner tackling marathons with renewed strength.

  ---

  Life@ Elastic: Tell us about yourself and what you do here at Elastic?

  Nicole LaJeunesse: Hi! I’m Nicole and I work in recruiting operations at Elastic. I strive to provide exceptional experiences for both our candidates and hiring teams throughout our hiring process. Some of my responsibilities include coordinating interviews across multiple time zones, being onsite to greet candidates here at HQ in Mountain View, and sending satisfaction surveys to our candidates after the interview process to learn how their overall experience interviewing at Elastic went, whether they were hired or not.

  What do you most enjoy about your role at Elastic?

  Nicole: I love being part of people’s journey into Elastic. I may not be their recruiter, their manager, or even on their direct team, but I’m with them the whole way throughout the process. I really enjoy helping them have a great experience, regardless of whether they join us. I also enjoy the flexibility at Elastic — not having to be in the office every day, or having to work to a certain schedule, for example. It enables me to have control over my life and my calendar. I don't think I would've been able to train for my last marathon without the flexibility to schedule my training runs in and around my work. Also, the “as you are” part of our Source Code is another favorite of mine, because it’s true — we do all come with different interests and skills.

  What inspired you to start running?

  Nicole: I grew up going to the finish lines of my mom’s races, who’s been a runner my whole life. That gave me a familiarity with running. But I started running when I started college to get in a quick workout that avoided the gym. I actually swore I'd never be crazy enough to run like my mom. Sure enough, less than a year after I started my "recreational" runs, I crossed the finish line of my first half marathon with her and I was hooked.

  

  Nicole and her mom
  

  Can you share more about going into cardiac arrest? Was it related to running marathons?

  Nicole: We’ll never know for sure. It was January, 2017, and I was on a vacation in South Lake Tahoe with my parents and my boyfriend, Josh. On the morning of January 3rd, I woke up, grabbed my phone from the wall where it was charging, jumped back onto the bed, and suddenly went into cardiac arrest. I am fortunate my mom is CPR certified and told Josh how to perform CPR on me while she and my dad contacted 911. I was intubated and taken to Barton Hospital in South Lake Tahoe, then airlifted to Renown Hospital in Reno. It was really an experience being seen by multiple doctors. They ran several tests. I was released three days later with a small implanted heart monitor. No one had a clear understanding why my heart had stopped and I was given an otherwise relatively clean bill of health.

  Did you have to see any other doctors?

  Nicole: Yes, I had to start seeing a Cardiac Electrophysiologist, a type of cardiologist who treats rhythm problems of the heart. He is amazing! Causes of sudden cardiac arrest and death in young people are rare and vary. He really educated me and showed me what different heartbeats and heart waves look like. It was scary! But also informative and empowering. He wanted to make sure I understood what was going on with me. A lot of times, causes of deaths in young people cannot be determined because the cause is never found after someone goes into cardiac arrest. In March of 2017, I had surgery to place a Subcutaneous Implantable Defibrillator in my left side, right under my arm. It ensures my heart keeps beating properly.

  What does a Subcutaneous Implantable Defibrillator do for you exactly?

  Nicole: It’s a device that monitors my heart rhythm and will detect sudden cardiac arrest. It will administer a strong shock if it detects a dangerously irregular heartbeat. I haven’t experienced a shock, and I may never experience one. But it’s comforting to know that my device is there to protect me just in case I need it. The battery lasts a while, but I will eventually have to have surgery to replace the pack when the battery runs low. Depending on if I receive a shock or not, it lasts between 7-10 years.

  When did you start running again?

  Nicole: Actually, almost immediately after post-op. I was hesitant, but my doctor said I should. I started out slow with two miles a week. I was sore but grateful to be up and running again. Six months later, in June 2017, I ran a half marathon and set a personal record time. I worked my way up to run my first marathon after post cardiac arrest. Another part of our Source Code that really ties in for me is perfection is not a destination, it’s about progress and that’s what I’m striving to do with my running these days.

  

  Do you have a favorite race?

  Nicole: I think the Napa Valley Marathon is my favorite as far as emotions go. Just because it was my first trace back. I trained really hard for it. I'd never been able to run one all the way through before, I took breaks and walked some of the way but for this one, I ran the entire time and set a personal marathon record. That was a really big accomplishment for me.

  

  Tell us a bit about running the New York Marathon.

  Nicole: Crossing the finish line I felt really strong. I felt like I had my life back again. To me, finishing made me feel like, "Okay, I'm back. I can keep doing what I was doing before, and be even better."

  Has Elastic been supportive during your training and fundraising for the marathon? If so, how?

  Nicole: Working from home and having flexible hours helps a lot to train. I use running as stress relief, or I'll use it as like a break in my day. When I work from home, I'll generally take my lunch as a run. When I go out for like a lunch run, I’ll just put a running emoji on Slack and post "Lunch run," for my status and off I go. If I have something that's on my mind, or I'm stressed about something, I like to use my runs to think about it and work out a solution.

  And through our Elastic Cares program, the donation matching was tremendous for my fundraising. I wouldn’t have been able to hit the financial goals that I set for myself without it.

  You raised nearly $4000 dollars for the American Heart Association! What does running the New York City marathon and the fundraising for AHA mean to you?

  Nicole: It's been amazing! Especially the support of my family, friends, and coworkers. I’d really like to highlight our program Elastic Cares. I know a lot of other millennials like myself are interested in working for companies that are going to make them feel like they're making a difference. And Elastic does just that by supporting a cause that means so much to me like the American Heart Association. I love that I work for a company that enables me to make a difference in the way I choose, and do good in ways that mean the most to me.

  

  ---

  Nicole was able to raise nearly $4,000 for the American Heart Association with the matching component of our Elastic Cares program, which matches donations up to 1% of an Elastician’s respective base salary per calendar year. Learn more about our Source Code and discover whether Elastic is a place you’d like to call your own.

  
  
  https://www.elastic.co/blog/life-at-elastic-spotlight-on-nicole-lajeunesse-running-with-heart
• 

  Elastic Stack 7.0.0 Beta1 Released
  • sorangutan  

  1
  1
  Posts
  53
  Views

  

  7.0.0 Beta 1 is here! This is the third in a series of 7.0.0 preview releases intended to give you an early look at the new features and changes coming in our next major version. And we want your feedback — become an Elastic Pioneer!
  

  Elasticsearch

  Faster Search

  When it comes to search, query performance is a key feature; some would say the key feature. We have achieved a significant performance improvement in Elasticsearch 7.0 for situations in which the exact hit count is not needed. For example, if your users typically just look at the first page of results and don’t need a precise count of the hits, you may be able to show “more than 10,000 hits” and then provide them with paginated results. This improvement allows us to skip the scoring/ranking process for documents that are so low-ranking that they will not make it into the top 10,000. The number of top results that are scored is configurable, and it’s important to note that this optimization won’t apply to aggregations queries. You can read more about this powerful algorithmic development in our blog post Magic WAND: Faster Retrieval of Top Hits in Elasticsearch.

  Improved Query Language

  7.0.0 beta-1 brings a number of new queries and improvements.

  There are many use cases, like patent or legal search, where users want to find records in which words or phrases are within a certain distance of one another. The new intervals query introduces a simple but powerful syntax for specifying these queries. Compared to the span query, which was historically the only way to meet these needs, the intervals query is easier to define and also use the analyzer, so the terms in the query are analyzed the same way as your documents.

  Relevance is one of the key attributes of good search. We are introducing a new way to do custom result scoring — the script_score query. It provides a simpler and more flexible way to generate a ranking score per record. The script_score query is constructed of a set of functions, including arithmetic and distance functions, which the user can mix and match to construct arbitrary function score calculations. The modular structure is simpler to use and will open this important functionality to additional users. Try it out and let us know what you think!

  And there is so much more! From a new geotile_grid aggregation (shown in Kibana below), nanosecond precision in timestamps, TLS 1.3, the Java High Level REST Client completion, to further improvements in the Zen2, the cluster coordination layer. Read the details in the Elasticsearch release blog!

  Kibana

  New Design and Navigation

  Since the introduction of the Elastic UI library in 6.2, Kibana has slowly been shifting towards a design revamp. 7.0.0 Beta 1 is the first release to show it off in its final form. The most prominent change is a switch to a new global navigation which introduces a constant header to manage spaces and breadcrumbs. We're also now defaulting the side navigation to a collapsed view so that individual applications have more horizontal space to work with.

  Beyond the new design, all the hard work behind the scenes also let us deliver on a much-requested feature: Dark Mode for all of Kibana!

  
  

  Autocomplete on by default

  In Kibana 6.3, we introduced autocomplete features to our query bar along with the Kibana query language. In 7.0.0 Beta 1 that feature is now turned on by default. For users that prefer the Lucene syntax that option is still available and can be toggled on or off as you need in advanced settings.

  

  Maps app for geospatial analysis

  In the mobile and distributed world we live in today, geo data is everywhere. Kibana 7.0.0 Beta 1 will include our new Maps application (beta) for geospatial and location based analytics. This new tool will allow you to visualize your geo data, at scale, with speed and in real time. With features like multiple layers and indices in a map, plotting raw documents, dynamic client side styling and the ability to search globally across multiple layers you will truly be able to understand and monitor your data with ease.

  With Maps, you will immediately get the complete value of our Elastic Maps Service, providing out of the box vector layers and baselayer maps. If for some reason we don't have what you're looking for, you'll also have the option to add your own custom vector shapes, tile map services or web map services as layers so you can truly customize your map to your liking. In 7.0, the Maps application will also be taking full advantage of the new geotile_grid aggregation for a more precise and seamless experience when interacting with the UI.

  
  

  And there’s more Kibana goodness! A new more unified query and filter bar, completely new timepicker, changes to the saved object APIs, and more. See all the details in the Kibana release blog!

  Beats

  New Supported Metric Data Types!

  Demand for turnkey metrics collection using the Elastic Stack continues to grow. To support this, we recently added several new Metricbeat modules, including community-contributed modules for CouchDB (shout out to berfinsari) and NATS (thank you to ChrsMark, skatsaounis, MichaelKatsoulis). We also added support for MariaDB and Percona as part of our MySQL database module. On top of that, 7.0.0 Beta 1 also brings long-requested support for the MS SQL database. and debuts the AWS EC2 module for fetching metrics directly from AWS Cloudwatch via Metricbeat.

  We have also improved our existing integration with Prometheus to avoid the need for user defined mappings. Metrics will be stored using fields with the same names that users see in Prometheus, and mappings will be derived automatically. For instance, all metric fields will use the double type, which is the internal store type in Prometheus, and all labels will be stored under prometheus.labels.* and mapped as keyword.

  We’re also hard at work to bring many of the existing Metricbeat modules out to general availability. Among these are a number of modules from the Kubernetes ecosystem -- Traefik, etcd, Envoy Proxy -- as well as modules for Windows, RabbitMQ, Couchbase, golang, php_fpm, Memcached, and ceph.

  Elastic Common Schema

  Going forward, all of our Metricbeat and Filebeat modules will be compatible with Elastic Common Schema (ECS), which defines a common set of fields for ingesting data into Elasticsearch. Having a common schema allows you correlate data from sources like logs and metrics for IT operations analytics and security analytics

  Try it Now!

  Happy downloading. Go forth, and explore.

  • Elasticsearch download | Release Notes
  • Kibana download | Release Notes
  • Logstash download | Release Notes
  • Beats downloads | Release Notes
  • ES-Hadoop download | Release Notes
  • APM Server download | Release Notes

  
  

  
  
  https://www.elastic.co/blog/elastic-stack-7-0-0-beta1-released
• 

  Elasticsearch 7.0.0 Beta 1 Released
  • sorangutan  

  1
  1
  Posts
  66
  Views

  

  Today we are pleased to announce the release of Elasticsearch 7.0.0-beta1, based on Lucene 8.0.0-SNAPSHOT.  This is the third in a series of pre-7.0.0 releases designed to let you test out your application with the features and changes coming in 7.0.0, and to give us feedback about any problems that you encounter. Open a bug report today and become an Elastic Pioneer!

  IMPORTANT: This is a beta release and is intended for testing purposes only. Indices created in this version may not be compatible with Elasticsearch 7.0.0 GA. Upgrading 7.0.0-beta1 to any other version is not supported. DO NOT DEPLOY IN PRODUCTION

  • Download Elasticsearch 7.0.0-beta1
  • Elasticsearch 7.0.0-beta1 release notes
  • Elasticsearch 7.0 breaking changes

  You can read about all the changes in the release notes linked above, but there are a few changes which are worth highlighting:

  Faster Queries for Top K Hits

  When it comes to search, query performance is a key feature; some would say the key feature. We have achieved a significant improvement to search performance in Elasticsearch 7.0 for situations in which the exact hit count is not needed and it is sufficient to set a lower boundary to the number of results. For example, if your users typically just look at the first page of results on your site and don’t care about exactly how many documents matched, you may be able to show them “more than 10,000 hits” and then provide them with paginated results. It’s quite common to have users enter frequently-occurring terms like “the” and “a” in their queries, which has historically forced Elasticsearch to score a lot of documents even when those frequent terms couldn’t possibly add much to the score.

  In these conditions (which are the typical conditions for many search usage scenarios) Elasticsearch can now skip calculating scores for records that are identified at an early stage as records that will not be ranked at the top of the result set.  This can significantly improve the query speed. The actual number of top results that are scored is configurable, but the default is 10,000. The behavior of queries that have a result set that is smaller than this threshold will not change - i.e. the results count is accurate but there is no performance improvement for queries that match a small number of documents. Because the improvement is based on skipping low ranking records, it does not apply to aggregations. You can read more about this powerful algorithmic development in our blog post Magic WAND: Faster Retrieval of Top Hits in Elasticsearch, or better still, download 7.0.0 beta-1 and try it on your data and queries to see how much your query performance improves!

  Intervals Query - The Next-Gen Span Query for Legal and Patent Search

  Sometimes our users want to find records in which words or phrases are within a certain distance from each other. In areas like patent and legal search, this is the main way in which experts find documents.  It used to be that the only way to do that was span queries, but now we are introducing a brand new way to construct such queries: interval queries. While span queries are a good tool, they are not always easy to use. Span queries do not use the analyzer, so the person performing the query has to be aware of the analyzer’s logic and perform actions like stemming. Since analyzers can be sophisticated, writing span query logic can be just as sophisticated (and complicated).

  The new intervals query are not just easier to define, they also use the analyzer, so the person writing them does not have to be familiar with the transformations performed by the analyzer. In addition, intervals queries are based on sound mathematical research, published in the article Efficient Optimally Lazy Algorithms for Minimal-Interval Semantics.  This allowed us to accurately deal with a number of edge cases that were not accurately handled with span queries. If you are making queries that relate to the distance between words or phrases, we believe a quick look at the simplicity of the intervals query DSL will have you quickly downloading 7.0.0 beta-1 and trying them out.

  Script Score Query (a.k.a. Function Score 2.0)

  If you haven’t already gathered, 7.0 is loaded with features for search use cases, and there’s another to add to the list: with 7.0.0-beta1, we are introducing the next generation of our function score capability.  This new script_score query provides a new, simpler, and more flexible way to generate a ranking score per record. The script_score query is constructed of a set of functions, including arithmetic and distance functions, which the user can mix and match to construct arbitrary function score calculations. The modular structure is simpler to use and will open this important functionality to additional users. Try it out and let us know what you think!

  Smooth Zoom on Maps with Geotile Grid

  A new aggregation has been introduced to handle (geo) map tiles in a way that allows a user to zoom in and out on the map without any change to the fringes of the shape. The new geotile_grid aggregation groups geo_points into buckets that represent cells in a grid, with each cell correlating with a tile in a map. Prior to this change, the fringes of the shape could slightly change with the change in resolution (a.k.a the zoom level), because the rectangle tiles would change orientation at different zoom levels. You can try it with Kibana 7.0.0-beta 1 or your own mapping application that uses Elasticsearch 7.0 beta-1.

  Nanosecond Precision Support

  Up until now Elasticsearch could only store timestamps with millisecond precision., If you want to process events that occur at a higher rate, for example if you want to store and analyze tracing or network packet data in Elasticsearch, you may want higher precision. Historically, we have used the Joda time library to handle dates and times, and Joda lacked support for such high precision timestamps.

  With JDK 8, an official Java time API has been introduced which can also handle nanosecond precision timestamps and over the past year, we’ve been working to migrate our Joda time usage to the native Java time while trying to maintain backwards compatibility.  As of 7.0.0-beta1, you can now make use of these nanosecond timestamps via a dedicated date_nanos field mapper.  Note that aggregations are still on a millisecond resolution with this field to avoid having an explosion of buckets.

  Better Integration with Stack Features

  We’re taking a look at a variety of Elasticsearch features and how they can have better defaults that really make things more seamless across the stack.  In 7.0.0-beta1, we’ve made 4 such improvements.

  First, our GeoIP ingest lookup processor is now shipped by default with Elasticsearch.  This popular processor takes an IP address and resolves the latitude, longitude, and other structured location information and is widely used by data shipped from our ingest products -- Beats and Logstash.  We’re likewise shipping the user_agent processor -- another popular processor -- by default now and in addition, user_agent ingest processor now uses the Elastic Common Schema for its field names.  All of this should help to make typical logging and metrics use cases using Elasticsearch much less error prone and more compatible with common visualizations and dashboards users typically set up for these use cases.

  Second, for those of you that have used Watcher, our alerting functionality, you may be aware of the cleaner service, which can delete old watch history indices after a period of time.  Following on the heels of the initial release of Index Lifecycle Management (ILM), we’ve now integrated that cleaner service with ILM.  This way, you can define the lifecycle/retention of these history indices the same way you work with other time-based data.

  Finally, JSON logging is now enabled in Elasticsearch in addition to plaintext logs. Starting in 7.0.0-beta1, you will find new files with .json extension in your log directory. This means you can now use filtering tools like jq to pretty print and process your logs in a much more structured manner. You can also expect finding additional information like node.id, cluster.uuid, type (and more) in each log line. The “type” field per each JSON log line will let you to distinguish log streams when running on docker.

  We’re working towards using these JSON logs to enable Beats to ingest Elasticsearch logs and provide more details to our Monitoring UI. For backwards compatibility reasons, Elasticsearch 7.0.0 will emit logs in both the old format and in JSON. You can read more on configuring your logging in our reference and migration guides.

  TLS 1.3

  Elasticsearch has supported encrypted communications for a long time, however, we recently started supporting JDK 11.  JDK 11 now has TLSv1.3 support so starting with 7.0, we’re now supporting TLSv1.3 within Elasticsearch for those of you running JDK 11.  In order to help new users from inadvertently running with low security, we’ve also dropped TLSv1.0 from our defaults. For those running older versions of Java, we have default options of TLSv1.2 and TLSv1.1.  Have a look at our TLS setup instructions if you need help getting started.

  High Level REST Client

  If you’ve been following our blog or our GitHub repository, you may be aware of a task we’ve been working on for quite a while now: creating a next-generation Java client for accessing an Elasticsearch cluster.  We started off by working on the most commonly-used features like search and aggregations, and have been working our way through administrative and monitoring APIs.  Many of you that use Java are already using this new client, but for those that are still using the TransportClient, now is a great time to upgrade to our High Level REST Client, or HLRC.

  As of 7.0.0-beta1, the HLRC now has all the API checkboxes checked to call it “complete” so those of you still using the TransportClient should be able to migrate.  We’ll of course continue to develop our REST APIs and will add them to this client as we go. For a list of all of the APIs that are available, have a look at our HLRC documentation.  To get started, have a look at the getting started with the HLRC section of our docs and if you need help migrating from the TransportClient, have a look at our migration guide.

  Cluster Coordination Layer Improvements

  In 7.0 Alpha 2, we released our new cluster coordination layer in Elasticsearch. With 7.0 Beta 1, the Elasticsearch cluster coordination layer received several improvements, including enhancements to rolling upgrade capabilities and faster cluster state publishing. Most importantly, users should be aware of the changes in Elasticsearch 7.0 to how a cluster performs discovery and is initially bootstrapped. A complete overview of the new cluster coordination layer in Elasticsearch 7.0 can be found in the Elasticsearch documentation. We love receiving feedback from our users, please don’t hesitate to create discuss topics or Github issues as you begin working with the new cluster coordination layer in Elasticsearch 7.0!

  Soft-deletes by default: new indices automatically eligible for replication to other Elasticsearch clusters

  In Elasticsearch 6.5, we released Cross Cluster Replication (CCR) as a beta feature. CCR requires any replicated index to maintain a history of document changes (when a document is updated or deleted) through the soft_deletes index setting on the leader index at index creation time. By retaining these soft deletes, a history can be maintained on the leader shards and replayed for replicating index changes to other Elasticsearch clusters. The soft_deletes index setting is required for CCR. Soft deletes will also be valuable for future Elasticsearch data replication improvements outside of CCR. Any newly created index has the soft_deletes setting enabled by default.

  Conclusion

  Please download Elasticsearch 7.0.0-beta1, try it out, and let us know what you think on Twitter (@elastic) or in our forum. You can report any problems on the GitHub issues page.

  
  
  https://www.elastic.co/blog/elasticsearch-7-0-0-beta1-released
• 

  Kibana 7.0.0 Beta 1 Released
  • sorangutan  

  1
  1
  Posts
  55
  Views

  

  Kibana 7.0.0 Beta 1 is now available for download. This is the third in a series of beta previews aimed to let you try out the newest version of Kibana and provide feedback before it releases. Outside of the paired changes to Elasticsearch, changes to Kibana are mostly around design.

  Dark mode everywhere

  Kbana now supports a dark mode theme across the entire application, not just dashboards. This setting is saved at the space level and can be found within the advanced settings page under management. Simply flip the switch, save, and reload your page to try it out. During the beta, some of the deeper pages (like Machine Learning) still might show some light backgrounds, but these will be fixed before the final release. Note that because theming is now a Space concern, you will no longer be able to set dark or light mode individually per dashboard.

  

  New design and navigation

  Since the introduction of the Elastic UI library in 6.2 Kibana has slowly been shifting towards a design revamp. 7.0.0 Beta 1 is the first release to show it off in its final form. The most prominent change is a switch to a new global navigation which introduces a constant header to manage Spaces and breadcrumbs. We're also now defaulting the side navigation to a collapsed view so that individual applications have more horizontal space to work with. If you're an old Kibana user you're used to the navigation coloring changing with each release. This time we decided to focus first on the content, and everything should have a lighter, more minimal feel that gets out of the way. Lastly on the design front, we did some much needed touch ups to our Dashboards and other applications with a big change to our default fonts.

  Saved objects are now aware of each other through references

  One challenge in Kibana has aways been around how we reference saved objects in relation to each other. In 7.0.0 Beta 1 saved objects no longer contain ids of other saved objects under attributes. They have been moved under references with a named reference in attributes. This will make it easier for us during the 7.x release cycle to better handle how we deal with imports and exports of saved objects.

  Example structure before

  {
    ...
    "attributes": {
      ...
      "panelsJSON": [{
        ...
        "type": "visualization",
        "id": "dbf71bb0-ffad-11e8-acfd-d359b3d9069"
      }]
    }
  }
  

  Example structure now

  {
    ...
    "attributes": {
      ...
      "panelsJSON": [{
        ...
        "panelRefName": "panel_0"
      }]
    },
    "references": [{
      "name": "panel_0",
      "type": "visualization",
      "id": "dbf71bb0-ffad-11e8-acfd-d359b3d9069"
    }] 
  }
  

  Autocomplete queries on by default

  In Kibana 6.3 we introduced autocomplete features to our query bar along with the Kibana Query Language. In 7.0.0 Beta 1 that feature is now turned on by default. For users that prefer the Lucene syntax that option is still available and can be toggled on or off as you need in advanced settings.

  

  A new filter bar and date-picker

  Alongside that query bar comes a totally rebuilt date picker and filter bar. The date picker is now completely accessible and no longer pushes the content of the page down (or forces a query reload) as you interact with it. We've also made it easier than ever to deal with complex date queries like mixing relative times with absolute ones. Like before, you can save your own presets or jump to quick dates at any time.

  Kibana 7 also introduces a redesigned filter bar. Gone are the indistinguishable rows of icons now that each filter "pill" comes with its own menu to clearly label its actions. We've made it easier than ever to deal with these filters in bulk and apply them from within your dashboards and saved searches.

  

  

  Maps for geospatial analysis

  In the mobile and distributed world we live in today, geo data is everywhere, regardless of use case. Kibana 7.0.0 Beta 1 will include our new Maps application (beta) for geospatial analysis and location based analytics. This new tool will allow you to parse through your geographical data, at scale, with speed and in real time. With features like multiple layers and indices in a map, plotting raw documents, dynamic client side styling and the ability to search globally across multiple layers you will truly be able to understand and monitor your data with ease. 

  With Maps, you will immediately get the complete value of our Elastic Maps Service, providing out of the box vector layers and baselayer maps. If for some reason we don't have what you're looking for, you'll also have the option to add your own custom vector shapes, tile map services or web map services as layers so you can truly customize your map to your liking. In 7.0, the Maps application will also be taking full advantage of the new geotile_grid aggregation for a more precise and seamless experience when interacting with the UI.

  As this is a beta feature, we'd especially love your feedback! Feel free to drop in questions and comments in the Kibana discuss forums.
  

  

  
  
  https://www.elastic.co/blog/kibana-7-0-0-beta1-released