Monitoring AWS EC2 using Metricbeat

[sorangutan]

Starting with 7.0, we introduced a new module in Metricbeat to monitor Amazon Web Services. At the time of the initial release in 7.0.0-beta1, this AWS module provides ability to collect AWS EC2 monitoring metrics from Cloudwatch using Metricbeat and stores them in a metric set called ec2. These metrics provide visibility into all EC2 instances across regions without the need to install Metricbeat directly onto the instances.

Background

Amazon Elastic Compute Cloud (EC2) instances are virtual servers that are created on the AWS platform to provide users with secure and resizable compute capacity in the public cloud. EC2 is arguably the most critical service out of hundreds provided by AWS -- virtually everyone needs access to raw compute resources.

Monitoring EC2 instances centrally to help users detect and debug issues is challenging. Metricbeat can already collect detailed resource metrics using the system module, but this requires installing an instance of Metricbeat on each of your EC2 instances. While the metrics it collects are very detailed and granular, sometimes you just want a quick way to get visibility into resource usage of all your EC2 instances at a high level.

This is where AWS Cloudwatch comes in. Cloudwatch automatically collects and centralizes basic resource utilization metrics from all your EC2 instances. This data is collected at a 5-minute granularity by default, and you can sign up for detailed monitoring to get higher granularity and more advanced metrics.

Configuring Metricbeat AWS

Before getting started with Metricbeat, you will need a running Elastic Stack, which includes Elasticsearch for storing and indexing the data and Kibana for data exploration. For this you can either create a cluster in Elasticsearch Service on Elastic Cloud or set up the Elastic Stack on your local machine.

Once you loaded Kibana, you can navigate to the Add Data UI from the home screen and follow the AWS module setup instructions from there.

With this dashboard, you can detect issues and make sure every instance is optimized for usage. For example, if you are seeing continuously high CPU utilization and performance issues from the same instance, this possibly indicates that CPU is the bottleneck and this instance needs more CPU power. Also, if you see an instance with low CPU utilization for a long time, this means the instance is oversized on CPU, and possibly can be scaled down or consolidated with another instance.

What’s next

Having a centralized location to visualize monitoring metrics from different AWS services and regions allows you to optimize resources usage, detect and fix issues faster. Give it a try, let us know what you think, give us feedback, and tell us how you are using this in our Beats forum.

The EC2 metricset in the AWS module is just the beginning! We are already hard at work on adding more metric sets to monitor other commonly used services, such as Amazon S3, SQS, RDS, and so on. If you have an opinion on what we should tackle next or want to contribute, please do not hesitate to engage in the Beats Github repository!

https://www.elastic.co/blog/monitoring-aws-ec2-using-metricbeat-and-the-elastic-stack