Grafana Labs News

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you may not be able to execute some actions.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Grafana Labs News

A Look at the Latest Cloud Data Source Plugins in Grafana
• sorangutan

1

1
Posts

43
Views

The engineers at Grafana Labs have their heads in the clouds.

“This is a new world: We have hybrid clouds and multiclouds,” Daniel Lee told the crowd gathered at GrafanaCon 2019 in Los Angeles. And the advantage clients have when using Grafana’s hosted services is that “they can deploy them on any cloud,” said Lee.

“With new data sources and new forms of data that you can get into Grafana, you can visualize, analyze, and understand all your systems,” said Lee. “We keep moving with the times.”

In addition to AWS Cloudwatch, which was added to Grafana in 2015, Grafana Labs recently released Azure Monitor as well as Google Stackdriver plugins in Grafana v6.0.

“Grafana’s all about avoiding vendor lock-in,” said Lee. “It fits with our core vision of democratizing metrics.”

Here is a look at the latest cloud data source plugins:

Azure

Microsoft’s Azure Monitor is no stranger to the Grafana community, having had a plugin for nearly two years.

The plugin supports four different Azure services: infrastructure metrics, application metrics and insights, and log-based metrics, in addition to monitoring. “You have support for three different types of metrics in one data source,” said Lee. “As a bonus, you will get Grafana Alerting as well.”

But if it’s already a plugin, why move it into Grafana’s core offerings? “We’re basically committing to a higher level of support,” explained Lee. “With so many plugins, often support for them is sporadic. But now with Azure Monitor, we are committing to keeping it in tip-top shape, improving it, and fixing any bugs that come up.”

So if any of the growing number of clients who use Azure Monitor reports an issue, “it will be resolved in one day,” said Lee.

Like Grafana Labs, Microsoft wanted Azure to be flexible across different services. “One of the nice things is the ability to bring together a visualization that joins data from multiple sources, from something like CPU percentages to application insights,” said Brendan Burns, Distinguished Engineer at Microsoft.

“With metrics, people want consistency,” said Burns. While Azure supports government clouds as well as public clouds, “it’s great to be across all of these clouds, but when you come to something like monitoring, you want it to look and feel the same no matter where you are.”

In addition to being able to “mix and match” different data sources, “it’s great to be able to provide Grafana to people who are using a SaaS-based monitoring inside of Azure,” said Burns.

Other features of the Azure Monitor plugin that Burns highlighted included the ability to edit queries, even within the in-line Grafana experience.

Also, with its log analytics service, Azure can help manage a Kubernetes cluster, log straight to the standard output stream, get the data into log analytics, and visualize the information in Grafana. “That’s a really great ability for users,” said Burns.

And capabilities such as dimension filtering, templating and alerting are “a nice value-add for a lot of people,” said Burns.

In other words, said Lee, the Azure Monitor plugin “is a bit of a beast.”

Stackdriver

Google provides a broad suite of products through Stackdriver, which helps improve the development experience on GCP as well as other cloud environments.

Services included on Stackdriver are monitoring (for platform system applications, your metrics, and custom metrics), logging (for log-based metrics and alerting), APM (Trace and Profiler are provided; Debug is in production), and IRM (for command and control systems for instances and instance repositories). Instance response management was also introduced last year.

But there’s one area where GCP couldn’t get ahead of the game. “We’ve got lots of user feedback saying we, as a GCP user, like how Stackdriver improved over the years, but in terms of UI visualization, we still like the way that Grafana does things,” said Joy Wang, Product Manager on Google Stackdriver.

So in early 2018, the GCP team reached out to Grafana Labs to collaborate on building a plugin together. Or as Wang put it: “I made some requirements, Daniel did all the work.”

“It’s been great to work with Google, helping us with all our questions,” said Lee of the development process. “They’re sort of guinea pigs in a way, because it’s the first data source written in React in Grafana.”

By October 2018, the beta version of the Stackdriver plugin was announced at Google Next, and a few months later, the feature advanced to GA on Grafana v6.0.

“At Google, lots of projects are open source, and we believe that as a GCP user, you benefit from open source solutions like Grafana [because] you get to choose what monitoring solutions you want for all the infrastructures that you have,” said Wang.

Wang also gave GrafanaCon attendees a preview of the notable updates to Stackdriver coming this year. “Basically, we’re doing a lot of things on the UI to allow users to put SRE best practices in place and also share those best practices with your coworkers,” she said.

On the monitoring front end, Stackdriver will introduce more widgets as well as allowing users to apply group-bys and filters on a dashboard level and save views of one dashboard. “You can slice and dice your metrics in a different way,” said Wang. “That will help users do in-context, ad-hoc analysis much faster.”

Stackdriver is also introducing Kubernetes monitoring, and SLO monitoring is another key concept for Google SRE. “We’re working on getting the service level monitoring and SLO monitoring out-of-box for users,” said Wang.

Finally, there will be “APIs for everything,” said Wang. “Later this year, we’re also announcing account API dashboards that allow users to set up their monitoring from end to end automatically. We’re also bringing up metric granularity and retention.”

“That’s only a fraction of the features that we’re working on,” said Wang. “There’s a lot more to come.”

Oracle

The Oracle Cloud Infrastructure (OCI) allows users to launch any type of instance from a VM to bare metal servers to GPU shapes, all for the same API and console. But recently “we’ve been focusing a lot on supporting container-native workloads,” said Mies Hernandez van Leuffen, VP of Solution Development at Oracle Cloud Native Labs.

So in December 2018, Oracle launched the Oracle Cloud Native Framework, which consists of Oracle’s managed Kubernetes solution (OKE) as well as a container registry that is completely Docker v2 compatible.

“The managed Kubernetes solution runs plain vanilla Kubernetes, nothing forked,” said Van Leuffen.

“At the same time, we’ve been working a lot with other companies rooted in open source to build up these ecosystem partnerships,” said Van Leuffen. “To that end, we’re pretty excited to launch our Grafana plugin.”

The Oracle plugin can be installed either via local environment, through Kubernetes, or on a VM. It allows users to surface and graph metrics from the OCI monitoring service. “The current metrics that we support are the Compute Agent, Block Store, Load Balancer, and Virtual Cloud Networks,” said Van Leuffen.

From the beginning, it hasn’t been a hard sell for companies to see the value in the Oracle plugin. “We have a pretty cool launching customer that is already using this in production,” said Van Leuffen. “It’s called Booster Fuels, and they effectively bring the gas station to your car, as opposed to the other way around.”

Added Van Leuffen: “Our mission is effectively to build customer deployable cloud native and container-centric solutions that bridge the gap between where OCI is today and what customers would like to use in terms of open source projects that we all know and love.”

Check out more sessions from GrafanaCon2019 on YouTube.

https://grafana.com/blog/2019/04/04/a-look-at-the-latest-cloud-data-source-plugins-in-grafana/
Grafana v6.1 Released
• sorangutan

1

1
Posts

37
Views
v6.1 Stable released!

A few weeks have passed since the excitement of the major Grafana 6.0 release during GrafanaCon, which means it’s time for a new Grafana release. Grafana 6.1 iterates on the permissions system to allow for teams to be more self-organizing. It also includes a feature for Prometheus that enables a more exploratory workflow for dashboards.

What’s New in Grafana v6.1
- Ad hoc filtering for Prometheus - A new query-focused workflow for ad hoc data exploration and troubleshooting.
- Permissions: Editors can own dashboards, folders, and teams they create - a new option to allow teams to self-organize more.
- List and revoke user auth tokens in the API - first step in building a new security feature in Grafana.
Download Grafana 6.1 Now

Ad hoc filtering for Prometheus

The ad hoc filter feature allows you to create new key/value filters on the fly with autocomplete for both keys and values. The filter condition is then automatically applied to all queries on the dashboard. This makes it easier to explore your data in a dashboard without changing queries and without having to add new template variables.

Other timeseries databases with label-based query languages have had this feature for a while. Prometheus recently added support for fetching label names from its API, and thanks to Mitsuhiro Tanda’s work implementing it in Grafana, the Prometheus datasource finally supports ad hoc filtering.

Support for fetching label names was released in Prometheus v2.6.0, so that is a requirement for this feature to work in Grafana.

Editors can own dashboards, folders, and teams they create

When the dashboard folders feature and permissions system were released in Grafana 5.0, users with the editor role were not allowed to administer dashboards, folders, or teams. In the 6.1 release, we have added a config option that can change the default permissions so that editors are admins for any dashboard, folder, or team they create.

This feature also adds a new team permission that can be assigned to any user with the editor or viewer role and enables that user to add other users to the team.

We believe that this is more in line with the Grafana philosophy, as it will allow teams to be more self-organizing. This option will be made permanent if it gets positive feedback from the community, so let us know what you think in the issue on GitHub.

To turn this feature on, add the following config option to your Grafana ini file in the users section, and then restart the Grafana server:
```
[users]
editors_can_admin = true
```
List and revoke user auth tokens in the API

As the first step toward a feature that would enable you to list a user’s signed-in devices/sessions and to log out those devices from the Grafana UI, support has been added to the API to list and revoke user authentication tokens.

Minor Features and Fixes

This release contains a lot of small features and fixes:
- A new keyboard shortcut d l toggles all graph legends in a dashboard.
- A small bug fix for Elasticsearch - template variables in the alias field now work properly.
- Some new capabilities have been added for datasource plugins that will be of interest to plugin authors:
  
  There’s a new oauth pass-through option.
  
  It’s now possible to add user details to requests sent to the dataproxy.
- Heatmap and Explore fixes.
- The Prometheus range query alignment was moved down by one interval. If you have added an offset to your queries to compensate for alignment issues, you can now safely remove it.
Changelog

Check out the CHANGELOG.md file for a complete list of new features, changes, and bug fixes.

Download

Head to the download page for download links & instructions.

Thanks

A big thanks to all the Grafana users who contribute by submitting PRs, bug reports, and feedback!

https://grafana.com/blog/2019/04/03/grafana-v6.1-released/
Grafana Plugin Tutorial: Polystat Panel (Part 1)
• sorangutan

1

1
Posts

187
Views
Polystat

The grafana-polystat-panel plugin was created to provide a way to roll up multiple metrics and implement flexible drilldowns to other dashboards.

This example will focus on creating a panel for Cassandra using real data from Prometheus collected from our Kubernetes clusters. We’ll focus on the basic metrics for CPU/Memory/Disk coming from cAdvisor, but a well-instrumented service will have many metrics that indicate overall health, such as requests per second, error rates, and more.

This panel allows you to group these metrics together into an overall health status, which can be used to drill down to more detailed dashboards. For this Cassandra example, the end result will look like this:

The Basics

Getting CPU, memory, and disk utilization will give enough metrics to demonstrate the idea behind compositing metrics and displaying them in Grafana. The PromQL queries below are simple and can be adapted with template variables to make the panel more “general purpose.” To get started, some simple queries will be used, then later modified.

CPU
```
container_cpu_usage_seconds_total{namespace="metrictank", pod_name=~"cassandra-sfs-.*", container_name="cassandra"}
```
The above query with polystat will show a large number of polygons (one per metric):

There are quite a number of pods displayed (we have multiple Cassandra clusters), so we will narrow this down to just a single cluster:
```
container_cpu_usage_seconds_total{namespace="metrictank", pod_name=~"cassandra-sfs-.*", container_name="cassandra", cluster="ops-tools1"}
```
The names still don’t show up since they are very long (hint: tooltips will show them). Adding {{pod_name}} to the Legend field will result in a better display:

Result:

The query needs a little more work – the metric is a counter – so we’ll use irate to get instantaneous per-second values.
```
irate(container_cpu_usage_seconds_total{namespace="metrictank", pod_name=~"cassandra-sfs-.*", container_name="cassandra", cluster="ops-tools1"}[1m])
```
Disk

While CPU is interesting, disk space in cassandra is usually what tends to run out, so we’ll add this query to show disk usage:
```
container_fs_usage_bytes{namespace="metrictank", pod_name=~"cassandra-sfs-.*", container_name="cassandra", cluster="ops-tools1"}
```
```
container_fs_limit_bytes{namespace="metrictank", pod_name=~"cassandra-sfs-.*", container_name="cassandra", cluster="ops-tools1"}
```
```
container_fs_limit_bytes{namespace="metrictank", pod_name=~"cassandra-sfs-.*", container_name="cassandra", cluster="ops-tools1"} - container_fs_usage_bytes{namespace="metrictank", pod_name=~"cassandra-sfs-.*", container_name="cassandra", cluster="ops-tools1"}
```
Memory

To complete our stats, add this memory query:
```
container_memory_usage_bytes{namespace="metrictank", pod_name=~"cassandra-sfs-.*", container_name="cassandra", cluster="ops-tools1"}
```
We can now see the result:

Formatting

The stats themselves have “short” as the value type in Grafana. Switching to the options of polystat, we can adjust them to something more meaningful:

Thresholding

The next step is to create thresholds for each of the metrics. In the thesholds section, add a new threshold, set the name to match a metric, and configure as needed. This example sets a 60% warning and 80% critical for CPU utilization.

The panel will now look like this:

Composites

Now that we have basic metrics and thresholds, we can create composites. (NOTE: The composite being created here is for a single node to keep everything simple, but the final result will have all nodes displayed.)

Composites allow you to group multiple metrics together and display a single item with the threshold state reflected. The polygon is given the color of the “worst” state. The tooltip will show individual states, sorted by worst to best.

To create a composite, click Add in the Composites section:

This will create a new composite named “Cassandra” and will include all metrics that match CPU/Memory/Disk.

The result of the composite will change the polystat to show a single polygon that represents three different metrics, and will animate to show the value for each metric.

Clickthroughs

There are three levels of clickthroughs provided by this panel.
1. Default clickthrough
2. Override clickthrough
3. Composite clickthrough
The order of precedence is most-specific to least-specific (3, 2, 1).

Default Clickthrough

You can set a clickthrough to be used globally when there are no override or composite clickthroughs defined for a polygon.

In this example, the clickthrough is set to:
```
dashboard/db/cassandra
```
Clicking on the polygon will take you to the Cassandra dashboard, in the same Grafana server. The clickthrough can be any valid url.

The plugin also includes parameters that can be passed to other dashboards.
```
dashboard/db/cassandra?var-environment=$Cluster&var-instance=All
```
Additional variables can be passed; see this for details: https://github.com/grafana/grafana-polystat-panel#single-metric-variables.

Override Clickthrough

In the overrides section, you can specify a clickthrough that applies for that specific override. This is mainly used when not using composites.

Setting the clickthrough for CPU to be…
```
dashboard/cpu?var-node=${__cell_name}
```
…will take you to a dashboard named “CPU” and pass the value of the clicked polygon.

Composite Clickthrough

The third type of clickthrough is used to specify where to go when a composited polygon is clicked. The implementation is the same as above.

Composites have another set of variables that can be passed to clickthroughs. See: https://github.com/grafana/grafana-polystat-panel#composite-metric-variables.

Templating

To keep the example above simple, the names are hardcoded. Leveraging Grafana template variables will make the dashboard more flexible.

The queries use “namespace” and “cluster,” so let’s create those.

Add a template variable to allow selection of different clusters:

Almost there

The dashboard will look like this, showing a single node with three different metrics displayed.

Wrapping Up

To complete the panel, just modify the composites to match regex per-node.

After changing the composites, the end result will look like this:

About Part 2

Part 2 will detail more composite options and advanced features to make them even easier to create.

If you have created some dashboards already with polystat, we’d love to see them!

https://grafana.com/blog/2019/04/02/grafana-plugin-tutorial-polystat-panel-part-1/
How We're Using Prometheus Subqueries at Grafana Labs.
• sorangutan

1

1
Posts

35
Views
In the Prometheus 2.7 release, Ganesh Vernekar added a feature called “Subqueries”. Ganesh published an explanation of how to use subqueries over on the Prometheus blog. In this post we’ll share a couple of real-life examples of how we use them at Grafana Labs.

Subqueries make it possible to do a certain class of queries against Prometheus in an ad hoc fashion. Previously Prometheus wouldn’t allow you to take a range vector of the output of a function; you could only take a range vector of a timeseries selector. We mainly justified this for two reasons: (1) We argued performance of this would be poor, and (2) it conveniently stopped you from taking the rate of a sum, something you should never do in Prometheus.

But a lot has changed in the past few years. Prometheus 2.0 introduced a new storage engine, and many improvements have been made to query performance. So justification (1) is no longer valid – and at the post-PromCon dev summit back in August, we decided the time was right to add subqueries.

Billing

The first example of how we use subqueries at Grafana Labs is for billing. We do usage-based billing for Grafana Cloud, partly based on your P95 datapoints per minute. Before Prometheus 2.7 and subqueries, we needed to use a recording rule to calculate the ingestion rate:
```
record: id_namespace:cortex_distributor_received_samples_total:sum_rate
expr: sum by (id, namespace) (rate(cortex_distributor_received_samples_total[5m]))
```
…and then a separate query to calculate the P95 of that using the quantile_over_time function:
```
quantile_over_time(0.95, id_namespace:cortex_distributor_received_samples_total:sum_rate[30d])
```
With subqueries, we can roll this all into one:
```
quantile_over_time(0.95, sum by (id, namespace) (rate(cortex_distributor_received_samples_total[5m]))[30d:])
```
The biggest benefit here is that we no longer need to use recording rules. They have to be declared up front, and changes made to them are not retroactively applied, making it very hard to experiment with ad hoc queries that require a range vector of a function’s result.

Capacity Planning

The second example of how we use subqueries at Grafana Labs is for capacity planning. We run Grafana Cloud on a set of Kubernetes clusters around the world. We need to ensure our jobs have the resources they need to serve traffic within our latency targets, but at the same time we do not want to be throwing money away on underutilized machines. As such we keep a close eye on our pods’ CPU utilization, and ensure we size the containers’ CPU requests appropriately.

To achieve good utilization, we size the containers’ CPU requests at P95 of their CPU usage. Per-container CPU usage is exported by cAdvisor as a counter. To calculate the per-second CPU usage, you have to apply the rate function:
```
record: container_name_pod_name_namespace:container_cpu_usage_seconds_total:sum_rate_5m
expr: sum by (container_name, pod_name, namespace) (rate(container_cpu_usage_seconds_total{namespace="cortex-ops", container_name!="POD"}[5m]))
```
And to see the P95 CPU usage over the last 7 days:
```
quantile_over_time(0.95, container_name_pod_name_namespace:container_cpu_usage_seconds_total:sum_rate_5m[7d])
```
With subqueries, we can now combine this into a single query:
```
quantile_over_time(0.95,
  sum by (container_name, pod_name, namespace) (
    rate(container_cpu_usage_seconds_total{namespace="cortex-ops", container_name!="POD"}[5m])
  )[7d:]
)
```
And as a bonus, you can use kube-state-metrics to export the configured CPU requests as a metric (some jiggery-pokery is need to make the labels consistent with cAdvisor):
```
sum by (container_name, pod_name, namespace) (
  label_join(
    label_join(
      kube_pod_container_resource_requests_cpu_cores,
      "pod_name", "", "pod"
    ),
    "container_name", "", "container"
  )
)
```
And combine the two queries to find underprovisioned pods:
```
  quantile_over_time(0.95,
    sum by (container_name, pod_name, namespace) (
      rate(container_cpu_usage_seconds_total{namespace="cortex-ops", container_name!="POD"}[5m])
    )[7d:]
  )
>
  sum by (container_name, pod_name, namespace) (
    label_join(
      label_join(
        kube_pod_container_resource_requests_cpu_cores,
        "pod_name", "", "pod"
      ),
      "container_name", "", "container"
    )
  )
```
Gotchas

As with “normal” queries, with subqueries you must never do a sum before a rate. For example, instead of rate((success+failure)[1m]) you must do rate(success[1m]) + rate(failure[1m]). This is because the rate function requires special handling for counter resets, which it cannot do if you have aggregated the counter with another. Avoid the temptation to simplify!

https://grafana.com/blog/2019/04/01/how-were-using-prometheus-subqueries-at-grafana-labs./
timeShift(GrafanaBuzz, 1w) Issue 83
• sorangutan

1

1
Posts

55
Views
Welcome to TimeShift

This week we have updates and articles from the Grafana Labs team, some initial impressions on our Prometheus-inspired log aggregation project Loki, and lots more. Plus learn how to make your own air quality monitor.

See an article we missed? Contact us.

Latest Beta Release: Grafana v6.1.0-beta1

Major New Features
- Prometheus: Ad-hoc Filtering makes it easier to explore data in a Grafana dashboard.
- Permissions: A new option so that Editors can own dashboards, folders and teams they create. This makes it easier for teams to self-organize when using Grafana.
Check out all the features and bug fixes in the latest beta release.
Download Grafana v6.1.0-beta1 Now

From the Blogosphere

Everything You Need to Know About the OSS Licensing War, Part 2.: Part 2 of our OSS licensing war series picks up where we left off: In 2015, AWS had taken the Elasticsearch software and launched their own cloud offering, and Elastic N.V. doubled down on an ‘open core strategy.’

Grafana Logging using Loki: Julien dives into Loki, its architecture, configuration, and some various use-cases.

Tinder & Grafana: A Love Story in Metrics and Monitoring: Two years ago, when it was time for the L.A.-based company to find and implement a perfect metrics monitoring partner, the process proved to be more slow-burn love affair versus whirlwind romance.

Scaling Graphite to Millions of Metrics: The folks at Klaviyo discuss the challenges they overcame to scale their Graphite stack to reliably handle over a million active metric keys at any given time across 17 million total metric keys.

Dynamic Configuration Discovery in Grafana: John provides an introduction to how dynamic configuration discovery works in Grafana for data sources and dashboards.

Build an air quality monitor with InfluxDB, Grafana, and Docker on a Raspberry Pi: Learn how to build your own air quality monitor. Collect metrics for temperature, humidity, barometric pressure, and air quality, and visualize all your data in Grafana.

Timeseries and Timeseries Again (pt. 2): Leon describes his initial impressions of using our Prometheus-inspired log aggregation project Loki in part 2 of his series focused on the metrics stack he uses at work. Check out part 1 to get up to speed.

Writing React Plugins for Grafana: In Grafana 6.0 we started the migration to using React in Grafana. This post will walk you through how to create plugins for Grafana using ReactJS.

How to deploy Telegraf, InfluxDB, and Grafana with Puppet Bolt: Check out a demo to configure and deploy monitoring stack via Puppet Modules.

What’s New in Prometheus 2.8: WAL-Based Remote Write: Learn about Prometheus’ new Write-Ahead Logging (WAL) for the remote_write API, which was included in the Prometheus 2.8 release. It’s a change intended to safeguard client metrics in the face of any network issues.

Grafana Plugin Update

Three plugin updates to share this week. Update or install any plugin on your on-prem Grafana via the grafana-cli tool, or update with one-click on Hosted Grafana.

UPDATED PLUGIN
WorldPing App - WorldPing v1.2.5 has been released which adds support for Grafana v6.0 and includes a few minor bug fixes.

Install

UPDATED PLUGIN
Polystat Panel - Polystat panel v1.0.16 has been released and includes a fix for variable encoding in clickthrough urls.

Install

UPDATED PLUGIN
Instana Data Source - The latest update of the Instana data source has lots of bug fixes as well as some new features:
- Application metrics have been added
- The datasource connection check has been improved
- Support added for beacon.meta grouping
Install

Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

DevOps Days Vancouver 2019 | Vancouver BC, Canada - 03.29.19-03.30.19:

Callum Styan: Grafana Loki - Log Aggregation for Incident Investigations - Get an inside look at Grafana Labs' latest open source log aggregation project Loki, and learn how to better investigate issues using Grafana's new Explore UI.
Register Now

KubeCon + CloudNativeCon Europe 2019 | Barcelona, Spain - 05.20.19-05.23.19:

May 21 - Tom Wilkie, Intro: Cortex
May 22 - Tom Wilkie, Deep Dive: Cortex
Cortex provides horizontally scalable, highly available, multi-tenant, long term storage for Prometheus metrics, and a horizontally scalable, Prometheus-compatible query API. Cortex allows users to deploy a centralized, globally aggregated view of all their Prometheus instances, storing data indefinitely. In this talk we will discuss the benefits of, and how to deploy, a fully disaggregated, microservice oriented Cortex architecture. We'll also discuss some of the challenges operating Cortex at scale, and what the future holds for Cortex. Cortex is a CNCF sandbox project.

May 23 - Tom Wilkie, Grafana Loki: Like Prometheus, But for logs.
Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be cost effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream.

Loki initially targets Kubernetes logging, using Prometheus service discovery to gather labels for log streams. As such, Loki enables you to easily switch between metrics and logs, streamlining the incident response process - a workflow we have built into the latest version of Grafana.

In this talk we will discuss the motivation behind Loki, its design and architecture, and what the future holds. Its early days after the launch at KubeCon Seattle, but so far the response to the project has been overwhelming, with more the 4.5k GitHub stars and over 12hrs at the top spot on Hacker News.

May 23 - David Kaltschmidt, Fool-Proof Kubernetes Dashboards for Sleep-Deprived Oncalls
Software running on Kubernetes can fail in various, but surprisingly well-defined ways. In this intermediate-level talk David Kaltschmidt shows how structuring dashboards in a particular way can be a helpful guide when you get paged in the middle of the night. Reducing cognitive load makes oncall more effective. When dashboards are organized hierarchically on both the service and the resource level, troubleshooting becomes an exercise of divide and conquer. The oncall person can quickly eliminate whole areas of problems and zone in on the real issue. At that point a single service or instance should have been identified, for which more detailed debugging can take place.
Register Now

Percona Live 2019 | Austin, TX - 05.28.19-05.30.19:

Tom Wilkie: Grafana Loki - Grafana Loki: Like Prometheus, But for logs. - Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be cost effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream.
Learn More
Monitorama PDX 2019 | Portland, OR - 06.03.19-06.05.19:

Tom Wilkie: Grafana Loki - Prometheus-inspired open source logging - Imagine if you had Prometheus for log files. In this talk we'll discuss Grafana Loki, our attempt at creating just that.
Learn More
InfluxDays London 2019 | London, United Kingdom - 06.13.19-06.14.19:

David Kaltschmidt - Mixing metrics and logs with Grafana + Influx - Imagine if you had Prometheus for log files. In this talk we'll discuss Grafana Loki, our attempt at creating just that.
Learn More

We're Hiring

Have fun solving real world problems building the next generation of open source tools from anywhere in the world. Check out all of our current opportunities on our careers page.
View All our Open Positions

How are we doing?

We’re always looking to make TimeShift better. If you have feedback, please let us know! Email or send us a tweet, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

https://grafana.com/blog/2019/03/29/timeshiftgrafanabuzz-1w-issue-83/
Everything You Need to Know About the OSS Licensing War, Part 2.
• sorangutan

1

1
Posts

37
Views

Where we left off: AWS had taken the Elasticsearch software and launched their own cloud offering in 2015, and Elastic N.V. had doubled down on an “open core strategy.”

Once AWS decides to offer a project like Elasticsearch, it immediately becomes a truly formidable competitor to anyone trying to do the same, even the company behind the software itself. AWS has huge scale, operational expertise, and various network effects that really compound.

Over time, however, AWS struggled to keep up with all the new versions of Elasticsearch, and all the innovation coming out of Elastic N.V. The version they had originally taken had to be heavily customized, and it fell behind the latest and greatest from Elastic N.V. and the open source community. It became “vintage” – not a good look for software.

To me, this is almost unconscionable on the part of AWS, given the huge revenues brought in by the service. The engineering effort required by AWS would have been minimal. It should have been an easy decision for them to invest in some upkeep and maintenance and at least rebase their code.

Despite these problems, it’s rumored that the revenue of the AWS Elasticsearch service has grown to eclipse the entire revenue of Elastic N.V. AWS had captured more value selling Elasticsearch than the company that had created it. It was a testament to the power of AWS.

An Escalation of Hostilities

Last year, a whole slew of open source companies – including Elastic N.V., MongoDB Inc., Confluent (the company behind Apache Kafka), and Redis Labs (the company behind Redis) – made pretty drastic and sudden changes to their licenses.

Elastic N.V. evolved their “open core” strategy, further blurring the lines between open source and commercial. They started to make some of its commercial software available for “free” to its users, and even allowed them to see the source code of that software. But the company carefully added restrictions to its license so that public cloud providers couldn’t do the same.

Some of this code was “open” but not open source. Elasticsearch was walking a very fine line deliberately designed to protect itself from the likes of AWS. Some thought they weren’t being clear enough about what was open and what was not. Had they gone too far?

MongoDB Inc. took a more direct approach, releasing MongoDB in its entirety under a new license, the SSPL. Its main purpose was to prevent public clouds like AWS from using the software to offer a MongoDB service. Was MongoDB even open source anymore? Did the company care? Did the community? The Internet was abuzz.

MongoDB Inc. had previously disclosed to investors that it soon expected 50 percent of its revenue to come from delivering MongoDB as a service. The world was trending toward consuming software as a service, and the license change would prevent anyone else from competing with the company’s ability to offer that. It seemed like a winning strategy.

These commercial open source companies were fighting back – and fighting for their valuations.

They’re faced with an existential question, particularly when they’re VC-funded and burning money: Can they monetize fast enough to command their eye-popping valuations? Can they capture enough of the value that their open source projects create?

The Nuclear Option

The war took on a more sinister and existential tone in early March, when AWS announced its “open distribution for Elasticsearch” (ODE) project, which strives to provide a “truly open source” (Apache2 licensed) distribution of Elasticsearch.

AWS had gone from taking open source to forking it.

The new fork of Elasticsearch will not only power Amazon’s hosted offerings, but it also has the potential to split or shift the center of mass of the open source community away from Elastic N.V.’s own offering.

AWS downplays the fact that this is a fork in their blog:

“Our intention is not to fork Elasticsearch, and we will be making contributions back to the Apache 2.0-licensed Elasticsearch upstream project as we develop add-on enhancements to the base open source software.”

I think that it is a fork, and that AWS is being disingenuous. It’s a fork because of the intentions and the messaging, not because they say their “intention” isn’t to fork. It’s a fork because attempts to merge changes back will be half-hearted at best on both sides.

So the codebases will diverge, and the community has the potential to split. But that’s obviously the implicit threat being made by AWS with this shiny new “distribution.”

The ability to fork is what gives open source its power. It’s an indictment against the leadership and governance of the open source project, a call to arms to the community to choose sides. Indeed, after waxing on how awesome Elasticsearch is, and how it has “played a key role in democratizing analytics of machine-generated data,” AWS spoke out pretty aggressively against Elastic N.V. in the blog:

“Unfortunately, since June 2018, we have witnessed significant intermingling of proprietary code into the code base. While an Apache 2.0 licensed download is still available, there is an extreme lack of clarity as to what customers who care about open source are getting and what they can depend on.”

Elastic N.V. did play a bit fast and loose with how it reframed “open” and how it made the default distribution of Elasticsearch veer more into shareware territory than open source. But the company also went out of its way to make pure open source versions of its software available and to communicate what it was doing. Fundamentally, Elasticsearch is still very much a liberally licensed open source project, and Elastic N.V. has invested significantly to improve it over the years.

And AWS adds insult to injury:

“The maintainers of open source projects have the responsibility of keeping the source distribution open to everyone and not changing the rules midstream.”

So a company that makes most of its money selling a closed source cloud that has lock-in as a goal – and capturing a huge amount of the value created by open source – is preaching to open source companies that they have to stay purely open source? This makes me laugh.

AWS doesn’t “support OSS” as it claims. It just wants to commoditize popular open source software so it can rent out its high-margin computers. AWS simply doesn’t have the political capital in the open source world to make that kind of moral judgement.

But could this be a new dawn at AWS? Are things changing? The company currently has a poor reputation, even compared to other public clouds, when it comes to participating in open source communities. Can AWS get good at running open source projects and actually build a real community of users, not just customers? It’s certainly within the realm of possibility.

What about Elastic? Where do they go from here?

And what about Grafana Labs? How has watching this unfold changed our perspective? Will we start playing the same licensing games used by companies like Elastic and MongoDB? Are we worried about our business?

Stay tuned for the third and final part of the blog, for my own opinions on these topics.

https://grafana.com/blog/2019/03/28/everything-you-need-to-know-about-the-oss-licensing-war-part-2./
Tinder & Grafana: A Love Story in Metrics and Monitoring
• sorangutan

1

1
Posts

31
Views

Tinder is the world’s most popular dating app, with more than 26 million matches made each day. But two years ago, when it was time for the L.A.-based company to find and implement a perfect metrics monitoring partner, the process proved to be more slow-burn love affair versus whirlwind romance.

“They say Rome wasn’t built in a day. Same could be said for Tinder’s Grafana infrastructure,” Tinder Observability Software Engineer Wenting Gong told the audience at Grafanacon 2019 in L.A.

Since it was founded in 2012, Tinder has grown to more than 320 employees and utilizes Grafana to monitor hundreds of microservices as well as the containers inside its Kubernetes environment.

Here’s how Tinder’s relationship with Grafana evolved.

First Impressions: Swiping Right on Grafana

In 2017, Tinder had about 50 microservices running on Amazon EC2 instances, supported with CloudWatch monitoring and Elastic load balancers. The Cloud Infrastructure Team came together with the Observability Team to start building an internal infrastructure that monitored the overall health of all of Tinder’s services.

“Because everything was running on AWS, AWS CloudWatch could provide those helpful metrics,” said Gong. “Grafana is a very popular open source tool for the data analytics and visualization for applications and infrastructure, so they decided to enable our CloudWatch on those instances and on the Elastic load balancers and the auto-scaling groups as well.”

By pulling metrics from the CloudWatch data source (which is native to Grafana), Grafana allowed engineers to view and check their service metrics directly and created a centralized location to access real-time metrics. In short, Tinder liked Grafana’s profile, swiped right, and enjoyed its early interactions with the monitoring system.

Meeting the Family: Introducing Prometheus

It didn’t take long for Tinder’s backend engineers to decide to get to know Grafana better too. They wanted to use it for monitoring the efficacy of their services, creating dashboards to track P95 latencies or request status updates.

When looking at the scope of time series database options, the Cloud Infrastructure Team investigated OpenTSDB (“It requires Hadoop, which didn’t fit our scenario,” said Gong), Graphite (“It’s hard to scale for an open-source solution”), InfluxDB (“Its clustering feature is available as a paid version”), and Nagios (“It’s one generation old”).

“Prometheus comes without any of those outstanding drawbacks, and it provides great client libraries so developers could use this to expose their application metrics from their code directly,” Gong said. “So Prometheus was the best option for us.”

A single Prometheus server was launched to pull metrics for all of Tinder’s services, and Grafana was used to monitor service-detailed market metrics as well as CloudWatch house metrics. But the honeymoon phase didn’t last long.

Within a few months, backend engineers were finding that data was being dropped from time to time as Tinder’s business continued to grow at a rapid rate. One Prometheus server was just not enough to carry the load of the dating app’s booming services.

The solution was to create a more scalable infrastructure that involved assigning individual Prometheus servers to each service within Tinder’s suite and creating a separate dashboard on Grafana for individual services. Not only did this solution make it easy to scale an individual Prometheus server based on a service load, but “with different Prometheus servers and recording rules enabled, this infrastructure also enhanced the current performance for those frequently used expressions and those computationally expensive expressions,” said Gong.

Getting Serious: Committing to Long-Term Data

As monitoring proved to be of value to Tinder’s business, the Backend Team wanted access to metrics for longer periods of time.

“We were only keeping metrics for several days for each individual service,” explained Gong. “But [engineers] still wanted to check some historical metrics for important data.”

There were two options to solve this problem. “One was to increase the retention period for all of Tinder’s Prometheus servers, which is super easy,” said Gong. But because not all data was needed for long-term retention, “the money and resources would be wasted on unnecessary metrics.”

The Observability Team then suggested launching a separate Prometheus server strictly for archiving key metrics, which engineers could readily access as needed.

The newly created server would pull metrics from all the individual Prometheus servers and expose itself as a separate data source on Grafana. “Also we have the data auto-discovery service running inside this archive server so that whenever a new service came up or one got removed, the data source, targets and endpoints will always be updated on the archive server side as well as the Grafana side,” said Gong.

“Even though this way requires the extra setup and the module owners need to understand and update the existing configuration,” Gong added, “this helps us use the resources and the budget in a more efficient way.”

Moving in Together: Migrating to Kubernetes

As Tinder’s microservices started to add up to the hundreds, the Cloud Infrastructure Team decided it was time to move to a Kubernetes environment, which would help engineers deploy and manage applications at scale. “This way, it will help give our developers more velocity, efficiency, and agility, and it will also help us save some money,” said Gong.

As a result, the Observability Team had to figure out how to execute and support the current monitoring infrastructure within Kubernetes, which involved different services running in different namespaces for different clusters.

Because developers are already familiar with using Prometheus to expose application metrics, it didn’t make sense to implement a new system. Instead, Tinder launched the Prometheus Operator inside its monitoring namespace which manages, creates and configures Prometheus instances within the Kubernetes environment. “The service monitor will help select the targets we’re trying to monitor, and the Prometheus service itself will pull the metrics from those service monitors,” said Gong. “This Prometheus service endpoint is also exposed as a separate Prometheus data source to Grafana.”

Within the monitoring namespace, there is also a separate sidecar service set up specifically for the archived metrics to collect the Prometheus data and put them into a separate MySQL database. “The archive server, which is in a traditional EC2 instance, will go and check those database results, pull those target metrics in, and expose itself as a separate Prometheus archive data source,” said Gong.

And what if there are any emergency issues in the Kubernetes environment? The Observability Team set up a monitoring system for the Kube cluster itself. “We launched some node exporters, a kube-state-metric exporter in that monitoring namespace and used a separate Prometheus cadence to pull those Kube-related metrics in,” said Gong. “This Kube-related cadence is also a separate data source for Grafana to plot the graphs related to the Kube states.”

Tinder also uses Prometheus exporters to monitor other infrastructure components such as Elasticsearch and Kafka.

With Tinder’s infrastructure settled into its Kubernetes environment, engineers can now easily monitor test environments. “But it’s a headache for our engineers to duplicate those dashboards for different environments for the similar modules,” Gong said.

To reduce the manual creation work, Tinder uses many of the APIs supported by Grafana. Said Gong: “We went ahead and developed customized dashboard automation utilities, with the help of GrafanaLib, in Python using an HTTP client-wrapper that will help when duplicating similar dashboards for different clusters and for different environments.”

In short, it’s a relationship built to last.

Check out more sessions from GrafanaCon2019 on YouTube.

https://grafana.com/blog/2019/03/26/tinder--grafana-a-love-story-in-metrics-and-monitoring/
Writing React Plugins
• sorangutan

1

1
Posts

39
Views
In this blog post we will go through how you can create plugins for Grafana using ReactJS. This presumes you have some basic knowledge about writing components in React.

(complete code for the example used in this post can be found here).

In Grafana 6.0 we started the migration to using React in Grafana. This allows you to write plugins using React instead of AngularJS. We are making it easier to write React plugins by releasing a Grafana component library - the new @grafana/ui npm package. The new npm package is still in Alpha and we are making breaking changes to the react plugin framework. But we want to encourage people to test it and give us early feedback.

Let’s take a look at how you can build your own plugin, using React and TypeScript.

Setup

There are a few things to consider when writing a new plugin. With Grafana 6.0, we need to move our plugins directory outside of the Grafana project directory. Feel free to put your plugins directory where you usually store code on your computer. Next, we need to tell Grafana where it should look for plugins. Grafana comes with a defaults.ini file in grafana/conf/, and we can overwrite this by creating and modifying a custom.ini. So put yourself in the grafana/conf directory and cp defaults.ini custom.ini.

Open custom.ini with your file editor of choice and search for this phrase:

Directory where grafana will automatically scan and look for plugins

Modify the line under that to:

plugins = <path to your plugins directory>

Restart your grafana-server after this.

Now we’re ready to move on!

The Structure

Grafana needs some basic project structure in your plugin. Grafana will look for a plugin.json located in a src directory. The plugin.json should contain some information about your plugin; you can read more about it here.

Also within the src directory we need a module.tsx file. In this file, we will introduce the first magic from our newly-released @grafana/ui package.
```
import { ReactPanelPlugin } from '@grafana/ui';

import { RssPanel } from './components/RssPanel';
import { RssPanelEditor } from './components/RssPanelEditor';

import { defaults, RssOptions } from './types';

export const reactPanel = new ReactPanelPlugin<RssOptions>(RssPanel);

reactPanel.setEditor(RssPanelEditor);
reactPanel.setDefaults(defaults);
```
Let’s go through this and figure out what this file does:
- First off, we’re creating a new instance of a ReactPanelPlugin, which is a class imported from @grafana/ui. We’re sending in our option type (in this case RssOptions, which we’ll get to later).
- Next up we’re setting the editor component for our plugin with the setEditor() function.
- Lastly we’re setting any default options that we might have.
That’s it!

The Panel

Now we’re at the fun part. This is where you can let your creativity flow. In this example we’re building an Rss-panel, and what we’re going to need is some kind of table to display our result. We’re going to use an interface exported by @grafana/ui called PanelProps. This will provide us with the props we need, such as height and width. I won’t go into any specifics about writing React components, but I will highlight some things that we do to make our panels written in React work.

Basic setup of a panel class:
```
interface Props extends PanelProps<RssOptions> {}
interface State {}

export class RssPanel extends PureComponent<Props, State> {}
```
It’s important to use React’s life cycle methods to make sure your component updates when the props change. We do this by invoking componentDidUpdate in our Rss-panel example. So when our user updates the url to the rss feed, we will update the panel to fetch an rss feed from the new url. In this example we’re using a library called rss-to-json to fetch and transform the rss feed to javascript objects.

The Panel editor

For adding options to Plugins, we’re using a concept called Editors. In this example we’ll create a component called <RssPanelEditor />. We have an interface for Editors in @grafana/ui as well, called PanelEditorProps. If we provide our options type to this interface, we will have the onChange method available for updating our panel when we change the options.
```
export class RssPanelEditor extends PureComponent<PanelEditorProps<RssOptions> {
  onUpdatePanel = () => this.props.onChange({
    ...this.props.options,
    feedUrl: 'this new rss feed url'
  });
}
```
Types

We strongly encourage you to use types in your panel. This makes it easier for you and others to spot potential bugs. In this example we’ve added some types for RssFeed, RssFeedItem, and RssOptions. These are located in src/types.ts.

Building

To be able to load the plugin, Grafana expects the code to be in plain JavaScript. We’re using webpack for the build step to transpile TypeScript to JavaScript in our RSS-plugin example.

Testing

Start your grafana-server again, and make sure that your plugin is registered.

Registering plugin logger=plugins name="Rss Panel"

Add a new panel to a dashboard, and locate your new panel in the visualization picker.

https://grafana.com/blog/2019/03/26/writing-react-plugins/
Writing React Plugins
• sorangutan

1

1
Posts

48
Views
In this blog post we will go through how you can create plugins for Grafana using ReactJS. This presumes you have some basic knowledge about writing components in React.

(complete code for the example used in this post can be found here).

In Grafana 6.0 we started the migration to using React in Grafana. This allows you to write plugins using React instead of AngularJS. We are making it easier to write React plugins by releasing a Grafana component library - the new @grafana/ui npm package.

Let’s take a look at how you can build your own plugin, using React and TypeScript.

Setup

There are a few things to consider when writing a new plugin. With Grafana 6.0, we need to move our plugins directory outside of the Grafana project directory. Feel free to put your plugins directory where you usually store code on your computer. Next, we need to tell Grafana where it should look for plugins. Grafana comes with a defaults.ini file in grafana/conf/, and we can overwrite this by creating and modifying a custom.ini. So put yourself in the grafana/conf directory and cp defaults.ini custom.ini.

Open custom.ini with your file editor of choice and search for this phrase:

Directory where grafana will automatically scan and look for plugins

Modify the line under that to:

plugins = <path to your plugins directory>

Restart your grafana-server after this.

Now we’re ready to move on!

The Structure

Grafana needs some basic project structure in your plugin. Grafana will look for a plugin.json located in a src directory. The plugin.json should contain some information about your plugin; you can read more about it here.

Also within the src directory we need a module.tsx file. In this file, we will introduce the first magic from our newly-released @grafana/ui package.
```
import { ReactPanelPlugin } from '@grafana/ui';

import { RssPanel } from './components/RssPanel';
import { RssPanelEditor } from './components/RssPanelEditor';

import { defaults, RssOptions } from './types';

export const reactPanel = new ReactPanelPlugin<RssOptions>(RssPanel);

reactPanel.setEditor(RssPanelEditor);
reactPanel.setDefaults(defaults);
```
Let’s go through this and figure out what this file does:
- First off, we’re creating a new instance of a ReactPanelPlugin, which is a class imported from @grafana/ui. We’re sending in our option type (in this case RssOptions, which we’ll get to later).
- Next up we’re setting the editor component for our plugin with the setEditor() function.
- Lastly we’re setting any default options that we might have.
That’s it!

The Panel

Now we’re at the fun part. This is where you can let your creativity flow. In this example we’re building an Rss-panel, and what we’re going to need is some kind of table to display our result. We’re going to use an interface exported by @grafana/ui called PanelProps. This will provide us with the props we need, such as height and width. I won’t go into any specifics about writing React components, but I will highlight some things that we do to make our panels written in React work.

Basic setup of a panel class:
```
interface Props extends PanelProps<RssOptions> {}
interface State {}

export class RssPanel extends PureComponent<Props, State> {}
```
It’s important to use React’s life cycle methods to make sure your component updates when the props change. We do this by invoking componentDidUpdate in our Rss-panel example. So when our user updates the url to the rss feed, we will update the panel to fetch an rss feed from the new url. In this example we’re using a library called rss-to-json to fetch and transform the rss feed to javascript objects.

The Panel editor

For adding options to Plugins, we’re using a concept called Editors. In this example we’ll create a component called <RssPanelEditor />. We have an interface for Editors in @grafana/ui as well, called PanelEditorProps. If we provide our options type to this interface, we will have the onChange method available for updating our panel when we change the options.
```
export class RssPanelEditor extends PureComponent<PanelEditorProps<RssOptions> {
  onUpdatePanel = () => this.props.onChange({
    ...this.props.options,
    feedUrl: 'this new rss feed url'
  });
}
```
Types

We strongly encourage you to use types in your panel. This makes it easier for you and others to spot potential bugs. In this example we’ve added some types for RssFeed, RssFeedItem, and RssOptions. These are located in src/types.ts.

Building

To be able to load the plugin, Grafana expects the code to be in plain JavaScript. We’re using webpack for the build step to transpile TypeScript to JavaScript in our RSS-plugin example.

Testing

Start your grafana-server again, and make sure that your plugin is registered.

Registering plugin logger=plugins name="Rss Panel"

Add a new panel to a dashboard, and locate your new panel in the visualization picker.

/blog/2019/03/26/writing-react-plugins/
What’s New in Prometheus 2.8: WAL-Based Remote Write
• sorangutan

1

1
Posts

41
Views

Six months in the making, Write-Ahead Logging (WAL) for the remote_write API was one of the enhancements we included in the Prometheus 2.8 release on March 12. It’s a change intended to safeguard client metrics in the face of any network issues.

The remote_write API allows you to send data from Prometheus to other monitoring systems, including Grafana Cloud. The previous implementation hooked into Prometheus’s metric scraping and was given copies of all the samples Prometheus scraped, sending them out to configured remote write endpoints.

If the remote endpoint was down or Prometheus was unable to reach it for any reason, there was only a small in-memory buffer in place - which proved problematic for two reasons: the data could back up, end up using too much memory, and causing Prometheus to OOM. Or if we got to the maximum size of the buffer defined by the existing configuration, we would start dropping data.

WAL-based remote_write

In the latest 2.8 Prometheus release, instead of buffering the data in memory, the remote_write now reads the write-ahead log. Before committing data to long-term storage, Prometheus writes all of the transactions that are occurring - samples that have been scraped and metadata about new time series - to a write-ahead log.

So if the endpoint is having an issue, we simply stop where we are in the write-ahead log and attempt to resend the failed batch of samples. It won’t drop data or cause memory issues because it won’t continue reading the write-ahead log until it successfully sends the data. The 2.8 update effectively uses a constant amount of memory, and the buffer is virtually indefinite, depending only on the size of your disk.

Another key enhancement we made involves cases with a single batch of data: If that send fails, the system no longer encodes that data every time we attempt to resend it. We simply encode it once and then keep sending it until it succeeds.

Corner cases

Initially, this was pitched as a two- to four-week project. But what happened along the way was that there were a lot of little edge cases that came up. The new approach broke assumptions in the existing code around locking, parallelization and concurrency that needed addressing.

For example, the existing code within Prometheus assumed that all write-ahead log files were fully written. So one of the major things we had to do was write code for the new remote write to read the write-ahead log files as they were being written.

Also, the existing reader silently accepted files with certain errors that the code would eventually repair. We developed the new reader on the assumption that there weren’t any corruptions in the files. In reality the WAL can become corrupted in many ways, so we had to add techniques to deal with this.

But the work was worth it. Thus far, the update has gotten a positive response - CPU usage is down and memory usage is much more predictable. This being a pretty big change, I expected there could be some issues with it. So far there have only been a few reported issues related to WAL corruptions, which we have ideas on how to address. If you have feedback, please feel free to open issues on GitHub.

/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/
What’s New in Prometheus 2.8: WAL-Based Remote Write
• sorangutan

1

1
Posts

35
Views

Six months in the making, Write-Ahead Logging (WAL) for the remote_write API was one of the enhancements we included in the Prometheus 2.8 release on March 12. It’s a change intended to safeguard client metrics in the face of any network issues.

The remote_write API allows you to send data from Prometheus to other monitoring systems, including Grafana Cloud. The previous implementation hooked into Prometheus’s metric scraping and was given copies of all the samples Prometheus scraped, sending them out to configured remote write endpoints.

If the remote endpoint was down or Prometheus was unable to reach it for any reason, there was only a small in-memory buffer in place - which proved problematic for two reasons: the data could back up, end up using too much memory, and causing Prometheus to OOM. Or if we got to the maximum size of the buffer defined by the existing configuration, we would start dropping data.

WAL-based remote_write

In the latest 2.8 Prometheus release, instead of buffering the data in memory, the remote_write now reads the write-ahead log. Before committing data to long-term storage, Prometheus writes all of the transactions that are occurring - samples that have been scraped and metadata about new time series - to a write-ahead log.

So if the endpoint is having an issue, we simply stop where we are in the write-ahead log and attempt to resend the failed batch of samples. It won’t drop data or cause memory issues because it won’t continue reading the write-ahead log until it successfully sends the data. The 2.8 update effectively uses a constant amount of memory, and the buffer is virtually indefinite, depending only on the size of your disk.

Another key enhancement we made involves cases with a single batch of data: If that send fails, the system no longer encodes that data every time we attempt to resend it. We simply encode it once and then keep sending it until it succeeds.

Corner cases

Initially, this was pitched as a two- to four-week project. But what happened along the way was that there were a lot of little edge cases that came up. The new approach broke assumptions in the existing code around locking, parallelization and concurrency that needed addressing.

For example, the existing code within Prometheus assumed that all write-ahead log files were fully written. So one of the major things we had to do was write code for the new remote write to read the write-ahead log files as they were being written.

Also, the existing reader silently accepted files with certain errors that the code would eventually repair. We developed the new reader on the assumption that there weren’t any corruptions in the files. In reality the WAL can become corrupted in many ways, so we had to add techniques to deal with this.

But the work was worth it. Thus far, the update has gotten a positive response - CPU usage is down and memory usage is much more predictable. This being a pretty big change, I expected there could be some issues with it. So far there have only been a few reported issues related to WAL corruptions, which we have ideas on how to address. If you have feedback, please feel free to open issues on GitHub.

https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/
timeShift(GrafanaBuzz, 1w) Issue 82
• sorangutan

1

1
Posts

40
Views
Welcome to TimeShift

This week we have a number of articles from the Grafana Labs blog we’d like to share, plus a new panel plugin and other updates. Also, take a look at our upcoming events. If you’re going to be attending any of them, please come and say hello!

See an article we missed? Contact us.

Latest Stable Release: Grafana v6.0.2

Bug Fixes
- Alerting: Fixed issue with AlertList panel links resulting in panel not found errors. #15975, @torkelo
- Dashboard: Improved error handling when rendering dashboard panels. #15970, @torkelo
- LDAP: Fix allow anonymous server bind for ldap search. #15872, @marefr
- Discord: Fix discord notifier so it doesn’t crash when there are no image generated. #15833, @marefr
- Panel Edit: Prevent search in VizPicker from stealing focus. #15802, @peterholmberg
- Datasource admin: Fixed url of back button in datasource edit page, when root_url configured. #15759, @dprokop
Check out all the new features and enhancements in v6.0
Download Grafana v6.0.2 Now

From the Blogosphere

Everything You Need to Know About the OSS Licensing War, Part 1.: Grafana Labs Co-founder and CEO Raj Dutt describes how a new breed of commercial open source company and dominance of the cloud has set off a licensing war that calls into question the very meaning of open source, and has the potential to change how software is developed, funded, and delivered. As a primer, check out the spirited panel discussion on the same subject from GrafanaCon.

GrafanaCon L.A. Recap: Grafana 6.0, LGTM, and More!: If you didn’t get a chance to join us last month at GrafanaCon LA we wanted to put together a recap of what you missed, plus link to all the videos of the talks to check out.

Pro Tips: How Booking.com Handles Millions of Metrics Per Second with Graphite: This talk from GrafanaCon EU 2018 discusses how Booking.com has scaled up its Graphite monitoring stack to handle over 10 million unique points per second and the challenges of handling metrics at scale.

Getting Started with Application Monitoring with Prometheus on VMware Enterprise PKS: With the migration from monolith applications to microservices and Kubernetes, the way you monitor your apps is changing. But where do you start? Xiao shows you how to get started painlessly with Prometheus and Grafana.

Visualize your Azure Sentinel data with Grafana: Learn how to use Grafana and the Log Analytics connector that Microsoft provides for Grafana, to visualize your Azure Sentinel data.

Setup Prometheus/Grafana Monitoring On Azure Kubernetes Cluster (AKS): This walkthrough provides some background info on Grafana and Prometheus and takes you step by step to install and configure everything on AKS.

Grafana Plugin Update

We have a new panel plugin and a few plugin updates to share this week. To update or install any plugin on your on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana, update with one-click.

NEW PLUGIN
Flowcharting Panel - Create flowcharts, diagrams, floorplans and more with this new panel.

Install

UPDATED PLUGIN
JSON Data Source - Version 0.1.3 allows you to use variables in the Additional JSON Data field.

Install

UPDATED PLUGIN
Statusmap Panel - New enhancements include Grafana 6.0 support, InfluxDB and MySQL support, initial annotation support and more.

Install

UPDATED PLUGIN
Sun and Moon Data Source - 0.1.4 of the Sun and Moon Data Source adds support for datasource provisioning, and annotations for noon and midnight.

Install

UPDATED PLUGIN
TrackMap Panel - This latest update fixes issues with Grafana v6.X and adds support for Snapshots.

Install

Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

Webinar: Why Open Source Works for DevOps Monitoring | 03.26.19, 1PM EDT:

Learn how to use open source tools for your performance monitoring of your stacks, systems, and sensors in a way that is faster, easier, and to scale.

In this webinar, Jacob Lisi from Grafana, and Chris Churilo from InfluxData will provide you with step by step instruction on how to use InfluxDB with Grafana, two popular open source projects, to capture and analyze untapped data from virtual and physical assets, giving you new visibility to customer experience and business growth.
Register Now

DevOps Days Vancouver 2019 | Vancouver BC, Canada - 03.29.19-03.30.19:

Callum Styan: Grafana Loki - Log Aggregation for Incident Investigations - Get an inside look at Grafana Labs' latest open source log aggregation project Loki, and learn how to better investigate issues using Grafana's new Explore UI.
Register Now

KubeCon + CloudNativeCon Europe 2019 | Barcelona, Spain - 05.20.19-05.23.19:

May 21 - Tom Wilkie, Intro: Cortex
May 22 - Tom Wilkie, Deep Dive: Cortex
Cortex provides horizontally scalable, highly available, multi-tenant, long term storage for Prometheus metrics, and a horizontally scalable, Prometheus-compatible query API. Cortex allows users to deploy a centralized, globally aggregated view of all their Prometheus instances, storing data indefinitely. In this talk we will discuss the benefits of, and how to deploy, a fully disaggregated, microservice oriented Cortex architecture. We'll also discuss some of the challenges operating Cortex at scale, and what the future holds for Cortex. Cortex is a CNCF sandbox project.

May 23 - Tom Wilkie, Grafana Loki: Like Prometheus, But for logs.
Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be cost effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream.

Loki initially targets Kubernetes logging, using Prometheus service discovery to gather labels for log streams. As such, Loki enables you to easily switch between metrics and logs, streamlining the incident response process - a workflow we have built into the latest version of Grafana.

In this talk we will discuss the motivation behind Loki, its design and architecture, and what the future holds. Its early days after the launch at KubeCon Seattle, but so far the response to the project has been overwhelming, with more the 4.5k GitHub stars and over 12hrs at the top spot on Hacker News.

May 23 - David Kaltschmidt, Fool-Proof Kubernetes Dashboards for Sleep-Deprived Oncalls
Software running on Kubernetes can fail in various, but surprisingly well-defined ways. In this intermediate-level talk David Kaltschmidt shows how structuring dashboards in a particular way can be a helpful guide when you get paged in the middle of the night. Reducing cognitive load makes oncall more effective. When dashboards are organized hierarchically on both the service and the resource level, troubleshooting becomes an exercise of divide and conquer. The oncall person can quickly eliminate whole areas of problems and zone in on the real issue. At that point a single service or instance should have been identified, for which more detailed debugging can take place.
Register Now

Monitorama PDX 2019 | Portland, OR - 06.03.19-06.05.19:

Tom Wilkie: Grafana Loki - Prometheus-inspired open source logging - Imagine if you had Prometheus for log files. In this talk we'll discuss Grafana Loki, our attempt at creating just that.
Learn More

We're Hiring

Have fun solving real world problems building the next generation of open source tools from anywhere in the world. Check out all of our current opportunities on our careers page.
View All our Open Positions

Tweet of the Week
We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove
We are able to display #grafana dashboards in the office. You see some average KPI's on the image. The dashboards are switching from time to time. Everyone in the team can see what is going on. #developers 🙌🎉 It is a small #angular application. 😉 pic.twitter.com/kwkrAFXocj
— вנöяи (@BrocksiNet) March 18, 2019

Awesome! Having dashboards up for the whole team to see is beneficial for all, and helps democratize metrics.

How are we doing?

We’re always looking to make TimeShift better. If you have feedback, please let us know! Email or send us a tweet, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

/blog/2019/03/22/timeshiftgrafanabuzz-1w-issue-82/
timeShift(GrafanaBuzz, 1w) Issue 82
• sorangutan

1

1
Posts

46
Views
Welcome to TimeShift

This week we have a number of articles from the Grafana Labs blog we’d like to share, plus a new panel plugin and other updates. Also, take a look at our upcoming events. If you’re going to be attending any of them, please come and say hello!

See an article we missed? Contact us.

Latest Stable Release: Grafana v6.0.2

Bug Fixes
- Alerting: Fixed issue with AlertList panel links resulting in panel not found errors. #15975, @torkelo
- Dashboard: Improved error handling when rendering dashboard panels. #15970, @torkelo
- LDAP: Fix allow anonymous server bind for ldap search. #15872, @marefr
- Discord: Fix discord notifier so it doesn’t crash when there are no image generated. #15833, @marefr
- Panel Edit: Prevent search in VizPicker from stealing focus. #15802, @peterholmberg
- Datasource admin: Fixed url of back button in datasource edit page, when root_url configured. #15759, @dprokop
Check out all the new features and enhancements in v6.0
Download Grafana v6.0.2 Now

From the Blogosphere

Everything You Need to Know About the OSS Licensing War, Part 1.: Grafana Labs Co-founder and CEO Raj Dutt describes how a new breed of commercial open source company and dominance of the cloud has set off a licensing war that calls into question the very meaning of open source, and has the potential to change how software is developed, funded, and delivered. As a primer, check out the spirited panel discussion on the same subject from GrafanaCon.

GrafanaCon L.A. Recap: Grafana 6.0, LGTM, and More!: If you didn’t get a chance to join us last month at GrafanaCon LA we wanted to put together a recap of what you missed, plus link to all the videos of the talks to check out.

Pro Tips: How Booking.com Handles Millions of Metrics Per Second with Graphite: This talk from GrafanaCon EU 2018 discusses how Booking.com has scaled up its Graphite monitoring stack to handle over 10 million unique points per second and the challenges of handling metrics at scale.

Getting Started with Application Monitoring with Prometheus on VMware Enterprise PKS: With the migration from monolith applications to microservices and Kubernetes, the way you monitor your apps is changing. But where do you start? Xiao shows you how to get started painlessly with Prometheus and Grafana.

Visualize your Azure Sentinel data with Grafana: Learn how to use Grafana and the Log Analytics connector that Microsoft provides for Grafana, to visualize your Azure Sentinel data.

Setup Prometheus/Grafana Monitoring On Azure Kubernetes Cluster (AKS): This walkthrough provides some background info on Grafana and Prometheus and takes you step by step to install and configure everything on AKS.

Grafana Plugin Update

We have a new panel plugin and a few plugin updates to share this week. To update or install any plugin on your on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana, update with one-click.

NEW PLUGIN
Flowcharting Panel - Create flowcharts, diagrams, floorplans and more with this new panel.

Install

UPDATED PLUGIN
JSON Data Source - Version 0.1.3 allows you to use variables in the Additional JSON Data field.

Install

UPDATED PLUGIN
Statusmap Panel - New enhancements include Grafana 6.0 support, InfluxDB and MySQL support, initial annotation support and more.

Install

UPDATED PLUGIN
Sun and Moon Data Source - 0.1.4 of the Sun and Moon Data Source adds support for datasource provisioning, and annotations for noon and midnight.

Install

UPDATED PLUGIN
TrackMap Panel - This latest update fixes issues with Grafana v6.X and adds support for Snapshots.

Install

Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

Webinar: Why Open Source Works for DevOps Monitoring | 03.26.19, 1PM EDT:

Learn how to use open source tools for your performance monitoring of your stacks, systems, and sensors in a way that is faster, easier, and to scale.

In this webinar, Jacob Lisi from Grafana, and Chris Churilo from InfluxData will provide you with step by step instruction on how to use InfluxDB with Grafana, two popular open source projects, to capture and analyze untapped data from virtual and physical assets, giving you new visibility to customer experience and business growth.
Register Now

DevOps Days Vancouver 2019 | Vancouver BC, Canada - 03.29.19-03.30.19:

Callum Styan: Grafana Loki - Log Aggregation for Incident Investigations - Get an inside look at Grafana Labs' latest open source log aggregation project Loki, and learn how to better investigate issues using Grafana's new Explore UI.
Register Now

KubeCon + CloudNativeCon Europe 2019 | Barcelona, Spain - 05.20.19-05.23.19:

May 21 - Tom Wilkie, Intro: Cortex
May 22 - Tom Wilkie, Deep Dive: Cortex
Cortex provides horizontally scalable, highly available, multi-tenant, long term storage for Prometheus metrics, and a horizontally scalable, Prometheus-compatible query API. Cortex allows users to deploy a centralized, globally aggregated view of all their Prometheus instances, storing data indefinitely. In this talk we will discuss the benefits of, and how to deploy, a fully disaggregated, microservice oriented Cortex architecture. We'll also discuss some of the challenges operating Cortex at scale, and what the future holds for Cortex. Cortex is a CNCF sandbox project.

May 23 - Tom Wilkie, Grafana Loki: Like Prometheus, But for logs.
Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be cost effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream.

Loki initially targets Kubernetes logging, using Prometheus service discovery to gather labels for log streams. As such, Loki enables you to easily switch between metrics and logs, streamlining the incident response process - a workflow we have built into the latest version of Grafana.

In this talk we will discuss the motivation behind Loki, its design and architecture, and what the future holds. Its early days after the launch at KubeCon Seattle, but so far the response to the project has been overwhelming, with more the 4.5k GitHub stars and over 12hrs at the top spot on Hacker News.

May 23 - David Kaltschmidt, Fool-Proof Kubernetes Dashboards for Sleep-Deprived Oncalls
Software running on Kubernetes can fail in various, but surprisingly well-defined ways. In this intermediate-level talk David Kaltschmidt shows how structuring dashboards in a particular way can be a helpful guide when you get paged in the middle of the night. Reducing cognitive load makes oncall more effective. When dashboards are organized hierarchically on both the service and the resource level, troubleshooting becomes an exercise of divide and conquer. The oncall person can quickly eliminate whole areas of problems and zone in on the real issue. At that point a single service or instance should have been identified, for which more detailed debugging can take place.
Register Now

Monitorama PDX 2019 | Portland, OR - 06.03.19-06.05.19:

Tom Wilkie: Grafana Loki - Prometheus-inspired open source logging - Imagine if you had Prometheus for log files. In this talk we'll discuss Grafana Loki, our attempt at creating just that.
Learn More

We're Hiring

Have fun solving real world problems building the next generation of open source tools from anywhere in the world. Check out all of our current opportunities on our careers page.
View All our Open Positions

Tweet of the Week
We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove
We are able to display #grafana dashboards in the office. You see some average KPI's on the image. The dashboards are switching from time to time. Everyone in the team can see what is going on. #developers 🙌🎉 It is a small #angular application. 😉 pic.twitter.com/kwkrAFXocj
— вנöяи (@BrocksiNet) March 18, 2019

Awesome! Having dashboards up for the whole team to see is beneficial for all, and helps democratize metrics.

How are we doing?

We’re always looking to make TimeShift better. If you have feedback, please let us know! Email or send us a tweet, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

https://grafana.com/blog/2019/03/22/timeshiftgrafanabuzz-1w-issue-82/
Pro Tips: How Booking.com Handles Millions of Metrics Per Second with Graphite
• sorangutan

1

1
Posts

40
Views

More than 1.55 million room nights are reserved on the Booking.com platform every day. It’s a staggering amount of traffic, and not surprisingly, the Amsterdam-based travel e-commerce company has a lot of knowledge to share about handling metrics at scale.

At GrafanaCon EU 2018, Booking.com System Administrator Vladimir Smirnov gave a talk about why the company started to use Graphite almost five years ago, and how he and his team scaled it to handle millions of metrics per second. “According to the graphiteapp.org, Graphite does three things really well: It kicks ass, chews bubblegum,” he quipped, “and it makes it easy to store and graph metrics.” The ease of use, via an HTTP API, and its modularity were big draws for Booking.com.

Vladimir Smirnov from GrafanaCo EU 2018
The most common use cases for Graphite within the organization were capacity planning and troubleshooting. “For capacity planning, we really need to store data for quite some time, ideally forever,” he said. “Also, we wanted to use it as source of data for troubleshooting and postmortems. We decided to store and visualize the business metrics, so whatever developers think is needed about the application, they also send it there.”

The Challenges of Graphite at Scale

Over the years, Booking.com’s Graphite grew to consist of hundreds of servers. “It’s hundreds of terabytes of data, hundreds of millions of unique metrics,” Smirnov said. “It ingests more than 10 million unique points per second, and we have tens of thousands of requests per second on both frontend and on backend of the stack that’s being monitored. All those requests fetch 10 million points every second out of our storages from disk. This does not include the stuff that’s fetched from cache.”

The original stack that was built in 2014 required several data centers with exactly the same setup. “You’ll end up having Graphite web and a bunch of servers or containers or virtual machines that talk to storage servers,” Smirnov said. “You’ll have some sort of load balancer on ingestion flow where servers and devices will send their metrics.”

There were problems inherent to this: “Carbon-relay was really a single point of failure for us,” he said. “The second problem was that for our cases, the backend does not really scale well. The carbon cache used to use really a lot of CPU. We do not like the disk usage patterns. Also, there was no easy way to sync our data if we have some storage failures. For example, one storage dies, and a second one leaves, then the second one comes back, and users keep getting the different results every time they refresh, depending on who wants it first. So the render request was really slow for us.”

Rewriting the Relay

To solve the problem of having a single point of failure, a Graphite team at Booking.com rewrote the relay in C. “We decided to place a relay as close as possible to the producers of metrics, so every server has a small relay that knows about several bigger endpoints—basically several relays in each data center,” he said. The central relay knows about the storages and how to balance the metrics, and has rules to get rid of some things or to rewrite them into something else.

The relay provides redundancy and speed. “It can consume more than a million data points per second using only two CPU cores, which was, well, a really desirable feature for us at that moment,” Smirnov said.

Using the Graphite line protocol, the relay reads metrics, validates them, and decides where to send them. Importantly, it can buffer the data for some time if there’s a network or node failure, and there is no data loss in most cases of outages.

But the company’s Graphite journey didn’t end there. “The more metrics we got, the more strange stuff we started to notice,” Smirnov said. “First of all, we saw that the metric distribution was not quite good with the bulk stream consistent hashing algorithm. In the worst cases, the difference was about 20%, so basically when one server ran out of disk space or ran out of capacity, other servers were fine.”

The team found a white paper from Google about the jump hashing algorithm, and implemented it in the relay to achieve an almost even distribution. The difference with the new algorithm was less than 1%.

More recently, they realized that this implementation wasn’t easy to maintain, so they started to rewrite the relay in Go. They also tried to switch from the traditional plaintext carbon protocol input and output to start using Kafka.

Dealing with Differences between Metrics

In the original 2014 stack, “when one server dies, and it comes up after a while, people will start seeing a difference.” He added, “Every time they refresh the graph they will see a different result because, well, two servers got a different amount of data.”

Their approach to solve this problem was simple: They wrote a small proxy that sits in front of the front-ends that emulates the behavior of graphite-web that fetches the data from all known backends that contain this metric. “If there are any gaps inside, it will try to do its best to fill up the gap and present the user the full set of data,” he said. This fix reduced the problems almost to zero. “There are still rare cases when it can happen but, well, almost not noticeable by normal people,” he said.

They also decided to replace the carbon cache, which wasn’t easy to maintain, with an open source project called Go Carbon. “We contributed some code to it to allow to read the metrics and present them in the way the Graphite web was expecting them,” Smirnov said. “That also solved some of the problems with the speed of the queries, because in our test, the Graphite web query carbon cache allowed us to do something like 80 requests per second, and after rewriting the stuff in Go, it was almost 3,000 requests per second, per backend.”

Enabling Grafana

Smirnov’s team installed Grafana, and “people really liked it because it’s so much nicer than Graphite web,” said Smirnov. “By using it, they started to do more and more complex queries, more complex dashboards, and that again became a problem from the perspective of speed and other stuff.” They decided to write some parts of the Graphite web in Go. “That led to a quite significant decrease of the average response time from the front-ends,” he reports. “Basically, this was the moment when we rolled out Carbon API as a main data source for Grafana. Average response time reduced from something like 15 seconds to less than one.”

All the work that Booking.com did on functions and parsing features is available as a Go library.

Improving the Backend

One resilient and failure-safe approach to storing data for a backend is Replication Factor 2. However, the backend tools the team was using to do the operational work on Graphite didn’t work with Replication Factor 2. They experimented with using Replication Factor 1, sending it twice to split the server fleet manually into two equal parts and sending it out to different parts.

In order to choose which approach to use, they created a replication factor test to calculate the potential for data loss in case of server failure. For a group of eight servers, the team found that with Replication Factor 2, you lose a smaller amount of data than with Replication Factor 1. But when two servers fail with Replication Factor 2, there will always be a small percentage of data that is definitely not available. With Replication Factor 1, the probability that data is lost when two servers fail is only 15%. The team opted for using Replication Factor 1 in two different sets of servers to reduce the probability of losing data.

More Issues to Resolve

With hundreds of servers and people, Booking.com has some particular challenges. “Sometimes you get something like, alert: disk space usage is showing you have 10% left, please provide new servers,” said Smirnov. “But the problem is that, well, you checked it five minutes ago, and it was like 20% left, so who the hell created all this new strange stuff?”

The truth is, “Because it’s really easy to send data to Graphite, it’s really easy to abuse it,” Smirnov said. “For example, instead of the variable name you can by accident get the address of the variable and send it instead. Every instance of your application will create a bunch of strangely named metrics. You never know until you actually encounter something like that.”

To find a solution, the team began collecting all possible stats about the backend, all the meta information about the file: when was it last accessed, when was it created, what’s the size. Using a library found on Github, the team presented the disk space in flame graphs.

“When you have something that can be considered as a good data point, it’s really easy to switch to the next one and see the difference, so that you can easily spot where it happened, and tell if a metric is a legit one or if someone created it by accident,” said Smirnov. “This also allowed us to do a lot of analytical queries across all the data set: how people use our stack, what metrics are the most commonly requested, and so on. Around 99% of metrics nobody cares about, but the last 1% is something really hot, and we can do some optimizations based on that, for example.”

What’s Next

The team has a substantial to-do list to keep improving Booking.com’s Graphite stack. 1. Find a replacement for whisper because it doesn’t have any compression. “It’s 12 bytes per point, so it takes really a lot of disk space, and most of our vacants are actually disk space bounded, not CPU, memory or network bound.” 2. Replace the Graphite line protocol between the components. “We are looking forward to migrating to streaming gRPC, but in terms of ingestion it’s still line protocol across all the stack. After the first step you don’t really need it. You can use something that’s compressed and even binary not to spend the CPU time on the backends.” 3. Make the flame graphs more or less production ready for other use cases, perhaps even having something like differential flame graphs. 4. Start collecting the stack traces from the Graphite stack in production to get some insights about what’s happening under the hood when we face problems.

TL/DR

While the team considers the Graphite stack still a work in progress, Smirnov can recommend the basic setup that they have come to after five years of experimenting and troubleshooting. “It was really surprising for us that not only we use our stack, but some part of our stack is used by eBay Classified and Slack,” he said.

To sum up: “Every frontend has exactly three pieces of software - Grafana, Carbon API, and Carbon Zipper (which is included in Carbon API)—and whenever the frontend has a problem, we just deploy a new server that contains these three pieces,” said Smirnov. “For the backend side, it’s always Go Carbon with carbonserver enabled inside. It’s always a single process, and whenever we run out of disk space or like in rare cases when we hit the limits in terms of capacity, like amount of metrics per second we can receive from the storage side, we deploy a bunch of new servers and do the rebalancing with buckytools.”

Smirnov, who has now left Booking.com, maintains the original Carbon API project that he started working on when he was there. Booking’s current stack can be found at its github account. For more details, check out the slides from his presentation.

Video: How Booking.com Handles Millions of Metrics Per Second with Graphite

Missed the conference? You can watch the talks here.

/blog/2019/03/21/pro-tips-how-booking.com-handles-millions-of-metrics-per-second-with-graphite/
Pro Tips: How Booking.com Handles Millions of Metrics Per Second with Graphite
• sorangutan

1

1
Posts

35
Views

More than 1.55 million room nights are reserved on the Booking.com platform every day. It’s a staggering amount of traffic, and not surprisingly, the Amsterdam-based travel e-commerce company has a lot of knowledge to share about handling metrics at scale.

At GrafanaCon EU 2018, Booking.com System Administrator Vladimir Smirnov gave a talk about why the company started to use Graphite almost five years ago, and how he and his team scaled it to handle millions of metrics per second. “According to the graphiteapp.org, Graphite does three things really well: It kicks ass, chews bubblegum,” he quipped, “and it makes it easy to store and graph metrics.” The ease of use, via an HTTP API, and its modularity were big draws for Booking.com.

Vladimir Smirnov from GrafanaCo EU 2018
The most common use cases for Graphite within the organization were capacity planning and troubleshooting. “For capacity planning, we really need to store data for quite some time, ideally forever,” he said. “Also, we wanted to use it as source of data for troubleshooting and postmortems. We decided to store and visualize the business metrics, so whatever developers think is needed about the application, they also send it there.”

The Challenges of Graphite at Scale

Over the years, Booking.com’s Graphite grew to consist of hundreds of servers. “It’s hundreds of terabytes of data, hundreds of millions of unique metrics,” Smirnov said. “It ingests more than 10 million unique points per second, and we have tens of thousands of requests per second on both frontend and on backend of the stack that’s being monitored. All those requests fetch 10 million points every second out of our storages from disk. This does not include the stuff that’s fetched from cache.”

The original stack that was built in 2014 required several data centers with exactly the same setup. “You’ll end up having Graphite web and a bunch of servers or containers or virtual machines that talk to storage servers,” Smirnov said. “You’ll have some sort of load balancer on ingestion flow where servers and devices will send their metrics.”

There were problems inherent to this: “Carbon-relay was really a single point of failure for us,” he said. “The second problem was that for our cases, the backend does not really scale well. The carbon cache used to use really a lot of CPU. We do not like the disk usage patterns. Also, there was no easy way to sync our data if we have some storage failures. For example, one storage dies, and a second one leaves, then the second one comes back, and users keep getting the different results every time they refresh, depending on who wants it first. So the render request was really slow for us.”

Rewriting the Relay

To solve the problem of having a single point of failure, a Graphite team at Booking.com rewrote the relay in C. “We decided to place a relay as close as possible to the producers of metrics, so every server has a small relay that knows about several bigger endpoints—basically several relays in each data center,” he said. The central relay knows about the storages and how to balance the metrics, and has rules to get rid of some things or to rewrite them into something else.

The relay provides redundancy and speed. “It can consume more than a million data points per second using only two CPU cores, which was, well, a really desirable feature for us at that moment,” Smirnov said.

Using the Graphite line protocol, the relay reads metrics, validates them, and decides where to send them. Importantly, it can buffer the data for some time if there’s a network or node failure, and there is no data loss in most cases of outages.

But the company’s Graphite journey didn’t end there. “The more metrics we got, the more strange stuff we started to notice,” Smirnov said. “First of all, we saw that the metric distribution was not quite good with the bulk stream consistent hashing algorithm. In the worst cases, the difference was about 20%, so basically when one server ran out of disk space or ran out of capacity, other servers were fine.”

The team found a white paper from Google about the jump hashing algorithm, and implemented it in the relay to achieve an almost even distribution. The difference with the new algorithm was less than 1%.

More recently, they realized that this implementation wasn’t easy to maintain, so they started to rewrite the relay in Go. They also tried to switch from the traditional plaintext carbon protocol input and output to start using Kafka.

Dealing with Differences between Metrics

In the original 2014 stack, “when one server dies, and it comes up after a while, people will start seeing a difference.” He added, “Every time they refresh the graph they will see a different result because, well, two servers got a different amount of data.”

Their approach to solve this problem was simple: They wrote a small proxy that sits in front of the front-ends that emulates the behavior of graphite-web that fetches the data from all known backends that contain this metric. “If there are any gaps inside, it will try to do its best to fill up the gap and present the user the full set of data,” he said. This fix reduced the problems almost to zero. “There are still rare cases when it can happen but, well, almost not noticeable by normal people,” he said.

They also decided to replace the carbon cache, which wasn’t easy to maintain, with an open source project called Go Carbon. “We contributed some code to it to allow to read the metrics and present them in the way the Graphite web was expecting them,” Smirnov said. “That also solved some of the problems with the speed of the queries, because in our test, the Graphite web query carbon cache allowed us to do something like 80 requests per second, and after rewriting the stuff in Go, it was almost 3,000 requests per second, per backend.”

Enabling Grafana

Smirnov’s team installed Grafana, and “people really liked it because it’s so much nicer than Graphite web,” said Smirnov. “By using it, they started to do more and more complex queries, more complex dashboards, and that again became a problem from the perspective of speed and other stuff.” They decided to write some parts of the Graphite web in Go. “That led to a quite significant decrease of the average response time from the front-ends,” he reports. “Basically, this was the moment when we rolled out Carbon API as a main data source for Grafana. Average response time reduced from something like 15 seconds to less than one.”

All the work that Booking.com did on functions and parsing features is available as a Go library.

Improving the Backend

One resilient and failure-safe approach to storing data for a backend is Replication Factor 2. However, the backend tools the team was using to do the operational work on Graphite didn’t work with Replication Factor 2. They experimented with using Replication Factor 1, sending it twice to split the server fleet manually into two equal parts and sending it out to different parts.

In order to choose which approach to use, they created a replication factor test to calculate the potential for data loss in case of server failure. For a group of eight servers, the team found that with Replication Factor 2, you lose a smaller amount of data than with Replication Factor 1. But when two servers fail with Replication Factor 2, there will always be a small percentage of data that is definitely not available. With Replication Factor 1, the probability that data is lost when two servers fail is only 15%. The team opted for using Replication Factor 1 in two different sets of servers to reduce the probability of losing data.

More Issues to Resolve

With hundreds of servers and people, Booking.com has some particular challenges. “Sometimes you get something like, alert: disk space usage is showing you have 10% left, please provide new servers,” said Smirnov. “But the problem is that, well, you checked it five minutes ago, and it was like 20% left, so who the hell created all this new strange stuff?”

The truth is, “Because it’s really easy to send data to Graphite, it’s really easy to abuse it,” Smirnov said. “For example, instead of the variable name you can by accident get the address of the variable and send it instead. Every instance of your application will create a bunch of strangely named metrics. You never know until you actually encounter something like that.”

To find a solution, the team began collecting all possible stats about the backend, all the meta information about the file: when was it last accessed, when was it created, what’s the size. Using a library found on Github, the team presented the disk space in flame graphs.

“When you have something that can be considered as a good data point, it’s really easy to switch to the next one and see the difference, so that you can easily spot where it happened, and tell if a metric is a legit one or if someone created it by accident,” said Smirnov. “This also allowed us to do a lot of analytical queries across all the data set: how people use our stack, what metrics are the most commonly requested, and so on. Around 99% of metrics nobody cares about, but the last 1% is something really hot, and we can do some optimizations based on that, for example.”

What’s Next

The team has a substantial to-do list to keep improving Booking.com’s Graphite stack. 1. Find a replacement for whisper because it doesn’t have any compression. “It’s 12 bytes per point, so it takes really a lot of disk space, and most of our vacants are actually disk space bounded, not CPU, memory or network bound.” 2. Replace the Graphite line protocol between the components. “We are looking forward to migrating to streaming gRPC, but in terms of ingestion it’s still line protocol across all the stack. After the first step you don’t really need it. You can use something that’s compressed and even binary not to spend the CPU time on the backends.” 3. Make the flame graphs more or less production ready for other use cases, perhaps even having something like differential flame graphs. 4. Start collecting the stack traces from the Graphite stack in production to get some insights about what’s happening under the hood when we face problems.

TL/DR

While the team considers the Graphite stack still a work in progress, Smirnov can recommend the basic setup that they have come to after five years of experimenting and troubleshooting. “It was really surprising for us that not only we use our stack, but some part of our stack is used by eBay Classified and Slack,” he said.

To sum up: “Every frontend has exactly three pieces of software - Grafana, Carbon API, and Carbon Zipper (which is included in Carbon API)—and whenever the frontend has a problem, we just deploy a new server that contains these three pieces,” said Smirnov. “For the backend side, it’s always Go Carbon with carbonserver enabled inside. It’s always a single process, and whenever we run out of disk space or like in rare cases when we hit the limits in terms of capacity, like amount of metrics per second we can receive from the storage side, we deploy a bunch of new servers and do the rebalancing with buckytools.”

Smirnov, who has now left Booking.com, maintains the original Carbon API project that he started working on when he was there. Booking’s current stack can be found at its github account. For more details, check out the slides from his presentation.

Video: How Booking.com Handles Millions of Metrics Per Second with Graphite

Missed the conference? You can watch the talks here.

https://grafana.com/blog/2019/03/21/pro-tips-how-booking.com-handles-millions-of-metrics-per-second-with-graphite/
Everything You Need to Know About the OSS Licensing War, Part 1.
• sorangutan

1

1
Posts

31
Views

The emergence of a new breed of commercial open source company, challenging the dominance of public cloud, has set off a licensing war that calls into question the very meaning of open source. We debated this topic at last month’s GrafanaCon Los Angeles, where I participated in a spirited panel.

Since then, the battle lines have been redrawn. Last week, Amazon announced its Open Distribution for Elasticsearch. And MongoDB Inc. abandoned OSI approval of its new SSPL license.

We’re getting into uncharted territory. This has the potential to change how software is developed, funded, and delivered. What is going on? How did we get to this point? And where do we go from here? Let’s look back at the history of commercial open source, so we can understand why there is a fundamental tension growing between that very concept and the public cloud vendors.

The Birth of Commercial Open Source

You can’t discuss commercial open source without talking about Red Hat Inc. They were the pioneers, going public way back in 2002, and shortly thereafter achieving the honor of becoming the world’s first $1B commercial open source company. This was a big deal. Back then, believing wholesale in the fundamental advantages of open source made you something of a heretic. Red Hat’s success making a business out of what had basically been a religion was intoxicating for a generation of future open source entrepreneurs.

Initially focused on Linux, Red Hat diversified into becoming the trusted purveyor of a wide variety of open source infrastructure software. Over the years, Linux and the open source stacks added insane value to the world. It underpinned empires. The open source community watched it topple Sun and Microsoft in the datacenter. Power the NYSE. Go to Mars. Give birth to Android.

The rise of Linux and Red Hat was great for users, moved the entire software industry forward, and trickled down into all sorts of new infrastructure software and companies. This happened because the world was free to use, meld, and otherwise co-opt the open source Linux code to scratch their own itches. This was built into the license. It was part of the ethos.

A few months ago, IBM announced that it would purchase Red Hat for $36B, bringing things full circle. It was a public market validation of what everyone could already see: Open source had truly arrived.

The most sophisticated companies wanted their most mission critical software open. Red Hat rode that first wave, and had built either a great or a pathetic business—people couldn’t seem to decide—by capturing a relatively small part of that value.

As Grafana Labs became a company in 2015, I keenly watched a new wave of commercial open source companies like Elastic NV (creators of Elasticsearch), and MongoDB Inc. (creators of MongoDB) grow their communities and their companies. There are many more such companies on the horizon. My friend Joseph Jacks (one of the most un-VC-like VCs in the business) is tracking over 40 such commercial open source companies with annual revenue worth >$100M. Open source, and the infrastructure software space in general, is a much bigger market than many of us thought it would be.

Software Becomes A Service

Red Hat got their start selling CD-ROMs of Linux through physical distributors like book stores. I still have mine from 1994. The shift from physical media to digital downloads was something they navigated beautifully. They started continually delivering updates and fixes over a new network called the Internet.

Their primary offering — access to their support and this continual, curated update stream — evolved into a “subscription.” It would reflect the start of another shift in how customers were thinking about their software in general. They wanted to treat it like a utility: consume what they needed, and pay for what they used. They didn’t want to have to deal with the muck and hassle of installing, maintaining, and scaling their software themselves. They didn’t want to buy the bits anymore. They wanted to rent the service.

Software as a Service (SaaS), and the cloud in general, would reshape the entire value chain in how software was consumed. At first, software services were built upon a diverse ecosystem of data centers run by telecom companies, web hosters, and co-location providers. More recently, we’ve seen the rise of the super datacenters: the large public cloud players like Amazon Web Services (AWS), Microsoft Azure, and Google Cloud.

AWS in particular deserves special attention. It is the apex predator of the public clouds. The reason why Amazon is the most valuable company in the world is because of AWS, not the e-commerce side project that it runs. If AWS were a company, it’d be worth more than all the companies on Joseph’s list combined. AWS dominates its competition.

Open source may be winning infrastructure software, but the public cloud, and AWS specifically, is winning where that software is run and how it is consumed by customers.

This sets the stage for our conflict.

AWS: Friend or Foe?

The actual “thing” that AWS is ultimately sells is computer capacity. AWS wants to commoditize the complement of what uses this capacity: software. Since open source software is starting to become the most coveted infrastructure software, it’s the ideal software for AWS to try and commoditize, and sell as a service to its customers.

Due to the freedoms that open source provides, it’s also the easiest to just “take”.

I suspect AWS finds this extremely convenient. The cornerstone, original AWS service that it launched with in 2006 was Linux itself (EC2), rented by the minute. AWS continiues to take the most popular open source software, to sell as a service, without even asking let alone paying the creators of that software. It’s like taking candy from a baby.

Except in the case of software like Elasticsearch and MongoDB, the baby is a multi-billion-dollar open source company that is very concerned about meeting its sales targets. And the candy was paid for by the VCs.

On October 1, 2015, AWS took the candy, and launched the Amazon Elasticsearch service. It was likely a dark day at Elastic NV. They were late to the cloud party, having just bought another startup offering Elasticsearch as a service. Now they’d been significantly disintermediated by the apex predator of cloud.

But it was a clear sign of how far their Elasticsearch project had come.

Open Source Is Not a VC Business Model

Elastic NV buckled down and started focusing on writing more commercial software to complement its popular open source project — and to build a moat around its business.

Unlike Red Hat, most of the new-generation commercial open source companies were not, strictly speaking, purely open source software companies. They were interested in value creation, but they were very interested in value capture. Most of them had a hybrid business model to optimize for that fact, and the high expectations of their VCs.

Such an “open core” strategy, where a company like Elastic NV maintains an open source project (like Elasticsearch) but also provides additional enhancements as commercial software (like X-Pack), would be widely adopted by many of the companies on Joseph’s list. This commercial software would only be available to paying customers. Thus even customers who used the open source Elasticsearch software via the AWS offering would still have plenty of reasons to deal with — and pay — Elastic NV directly.

Elastic NV also enhanced its own Elastic as a service cloud offering. The company innovated to improve both its open source core and its commercial add-ons. It got better at being able to deliver software as a service. It competed. The company went public in October 2018, and is currently valued at more than $6B. Not bad for an open source project that Elasticsearch B.V. CEO Shay Bannon started in 2010.

But the war would be far from over. Both AWS and Elastic NV would escalate hostilities over the coming months.

Coming soon: Part 2 - The licensing war escalates.

/blog/2019/03/20/everything-you-need-to-know-about-the-oss-licensing-war-part-1./
GrafanaCon L.A. Recap: Grafana 6.0, LGTM, and More!
• sorangutan

1

1
Posts

42
Views
The rest of the city may still have been in a post-Oscars haze, but over 350 monitoring mavens gathered in downtown L.A. bright and early on Feb. 25 to kick off GrafanaCon 2019. The next two days were filled with 40+ talks, including Grafana end user stories from companies like Bloomberg and Tinder. The second day’s schedule allowed attendees to choose from multiple deeper-dive tracks such as cloud native observability, real-time analytics in IoT, and Grafana itself (getting the most out of it and extending it).

Here are some highlights and takeaways from the conference:
1. Grafana v6.0 was launched. The release includes Explore, a new query-focused workflow for ad-hoc data exploration and troubleshooting; integration with Grafana Loki, the new open source log aggregation system; React plugin support to make building plugins easier; and support for all of the major public clouds’ monitoring stacks (Azure Monitor, Google Stackdriver, Amazon CloudWatch, Oracle Cloud monitoring). “The release has been a hit with users that love the new Explore mode and the new ability to switch visualizations,” says Grafana Labs CGO/Cofounder Torkel Ödegaard. “Personally, I am really excited about the path we are on with so much progress having been made on improving Grafana as a modern platform and how we will capitalize on that foundational work with an increased rate of new features and visualizations in the coming months.”
2. LGTM looks good to us. The Grafana Labs team unveiled their vision for LGTM, a full, composable observability stack: “It stands for Loki, Grafana, Tracing, and Metrics,” says Tom Wilkie, Grafana Labs VP Product, “but it’s also logging, graphing, tracing, and metrics, in any combination, as long as the G is for Grafana. As a company, we want to support all combinations of this to help our users to find the right technologies to fit.”
3. Interest in Kubernetes and Prometheus was sky-high. The cloud native observability track on the second day—which included sessions on visualizing chaos engineering experiments and treating dashboards as code—was “super well-attended, with really good quality questions, lots of real interest, and a great amount of energy,” reports Wilkie.
4. There was plenty of love for Graphite and MetricTank. Impressed by the high demand for Graphite and MetricTank content among conference attendees, Wilkie emphasizes that the project continues to be a large focus for Grafana Labs. “We’re still investing in that team and hiring new people to work on that team,” he says, “and we’ve got some really exciting stuff coming!”
5. Networking goes well with beer. The v6.0 launch party on the first night featured lots of craft beer and conversations late into the night. Above all, connecting with fellow travelers on the observability journey was a true highlight of the conference. See you next year!
Missed the conference? You can watch the talks here.

/blog/2019/03/19/grafanacon-l.a.-recap-grafana-6.0-lgtm-and-more/
timeShift(GrafanaBuzz, 1w) Issue 81
• sorangutan

1

1
Posts

47
Views
Welcome to TimeShift

TimeShift is back after a few weeks away with a lot of updates to share. Nearly all of the Videos and presentations from GrafanaCon LA are available, so please check them out and let us know what you think. Also, if you hadn’t heard, Grafana v6.0 stable was recently released which has lots of new features and enhancements. Download the latest version v6.0.1 and read about the highlights below.

See an article we missed? Contact us.

Latest Stable Release: Grafana v6.0.1

Bug Fixes
- Metrics: Fixes broken usagestats metrics for /metrics #15651
- Dashboard: Fixes kiosk mode should have &kiosk appended to the url #15765
- Dashboard: Fixes kiosk=tv mode when autofitpanels should respect header #15650
- Image rendering: Fixed image rendering issue for dashboards with auto refresh, . #15818, @torkelo
- Dashboard: Fix only users that can edit a dashboard should be able to update panel json. #15805, @marefr
- LDAP: Fix allow anonymous initial bind for LDAP search. #15803, @marefr
- UX: Fixed scrollbar not visible initially (only after manual scroll). #15798, @torkelo
- Datasource Admin: TestData #15793, @hugohaggmark
- Dashboard: Fixed scrolling issue that caused scroll to be locked to bottom. #15792, @torkelo
- Explore: Viewers with viewers_can_edit should be able to access /explore. #15787, @jschill
- Security: Fixes limit access to org admin and alerting pages. #15761, @marefr
- Panel Edit: MinInterval changes did not persist #15757, @hugohaggmark
- Teams: Fixes bug when getting teams for user. #15595, @hugohaggmark
- Stackdriver: Fix for float64 bounds for distribution metrics #14509
- Stackdriver: No reducers available for distribution type #15179
Check out all the new features and enhancements in v6.0
Download Grafana v6.0.1 Now

From the Blogosphere

The Linux Foundation Launches New CommunityBridge Platform to Help Sustain Open Source Communities: The Linux Foundation recently announced CommunityBridge – a new platform created to empower open source developers and the individuals and organizations who support them – and Grafana Labs is thrilled to be part of the platform!

Grafana | DigitalOcean Marketplace: Grafana is one of the first 1-click Apps on the new DigitalOcean Marketplace.

Why You Should Be Using Grafana With OCI: We recently announced the release of Oracle Cloud Infrastructure as a Data Source for Grafana. In a follow up post on the Oracle Blog, Todd Sharp describes the immediate benefits he realized when he created his first OCI dashboard in Grafana and why you should be using it.

How New City Energy Is Supporting Sustainability with Grafana): In this talk from GrafanaCon EU 2018, Andrew Rodgers shows how Grafana is enabling Washington D.C. to reduce energy costs as well as its carbon footprint.

Remote monitoring and alerting for IoT: Learn how to set up remote monitoring and alerting for IoT devices using Google Cloud Platform, Prometheus, and Grafana.

Container resources’ request/limit/usage metrics: This article discusses the Container Resource Exporter (CRE) for Prometheus which captures containers’ resource request quantity, limit quantity and current usage in real-time and triggers a proactive alert when a certain threshold of our choice is crossed. These metrics are printed in the Prometheus metric text format for easy visualization in Grafana.

Prometheus: RTFM blog monitoring set up with Ansible – Grafana, Loki, and promtail: In this post, setevoy describes the Prometheus, node_exporter, Grafana, Loki, and promtail set up process. He also includes some of the minor issues he encountered during the process and troubleshooting he conducted to help make things a little easier for others.

[VIDEO] Monitoring Open Liberty with Prometheus & Grafana: Learn how to add monitoring to Open Liberty instances using Prometheus and Grafana. In this video, learn how to configure Open Liberty and how to run the instance together with a database and Prometheus in a Kubernetes cluster. Then, take a look at an example Grafana dashboard you can easily import to visualize that data.

How to Get Started Using CrateDB and Grafana to Visualize Time-Series Data: This is a quick walkthrough of how to get up and running with CrateDB and Grafana. In only a few minutes you’ll learn how to install both CrateDB and Grafana, generate some data, add a PostgreSQL data source, and visualize the data you generated on a new Grafana dashboard.

3D Printer OctoPrint Monitoring with Prometheus and Grafana: Dr. Scott M. Baker wrote a Prometheus plugin in order to keep track of stats of his 3D Printer; things like power usage, temperature, extrusion, filament usage, etc. This video and accompanying blog article takes you through some of the specifics and how you can install and start using the plugin.

Why You Want Easy-to-Setup Grafana Dashboards: Find out what you have to gain by setting up Grafana dashboards and the easy steps involved.

Phenomenal Outages: This Tumblr gallery is a great example of “data as art.” As its tagline states: outages don’t have to be inelegant.

Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

Go Meetup at Grafana Labs | Stockholm, Sweden - 03.20.19, 5:30 PM to 9:00 PM CET:

Join us at the Grafana Labs Stockholm office for an evening of hacking on code, talking about Go, or just having a good time meeting other Gophers. If you want to write some code then bring a computer. Maybe you have a side project that you want to present or get input on?

Agenda:
- 17:30 - 18:00 Meet and greet
- 18:00 - 18:10 Welcome
- 18:10 - 21:00 Hacking/Talking
NOTE: This event has reached capacity, but you can still join the waiting list.
Learn More

Webinar: Why Open Source Works for DevOps Monitoring | 03.26.19, 1PM EDT:

Learn how to use open source tools for your performance monitoring of your stacks, systems, and sensors in a way that is faster, easier, and to scale.

In this webinar, Jacob Lisi from Grafana, and Chris Churilo from InfluxData will provide you with step by step instruction on how to use InfluxDB with Grafana, two popular open source projects, to capture and analyze untapped data from virtual and physical assets, giving you new visibility to customer experience and business growth.
Register Now

DevOps Days Vancouver 2019 | Vancouver BC, Canada - 03.29.19-03.30.19:

Callum Styan: Grafana Loki - Log Aggregation for Incident Investigations - Get an inside look at Grafana Labs' latest open source log aggregation project Loki, and learn how to better investigate issues using Grafana's new Explore UI.
Register Now

KubeCon + CloudNativeCon Europe 2019 | Barcelona, Spain - 05.20.19-05.23.19:

May 21 - Tom Wilkie, Intro: Cortex
May 22 - Tom Wilkie, Deep Dive: Cortex
Cortex provides horizontally scalable, highly available, multi-tenant, long term storage for Prometheus metrics, and a horizontally scalable, Prometheus-compatible query API. Cortex allows users to deploy a centralized, globally aggregated view of all their Prometheus instances, storing data indefinitely. In this talk we will discuss the benefits of, and how to deploy, a fully disaggregated, microservice oriented Cortex architecture. We'll also discuss some of the challenges operating Cortex at scale, and what the future holds for Cortex. Cortex is a CNCF sandbox project.

May 23 - Tom Wilkie, Grafana Loki: Like Prometheus, But for logs.
Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be cost effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream.

Loki initially targets Kubernetes logging, using Prometheus service discovery to gather labels for log streams. As such, Loki enables you to easily switch between metrics and logs, streamlining the incident response process - a workflow we have built into the latest version of Grafana.

In this talk we will discuss the motivation behind Loki, its design and architecture, and what the future holds. Its early days after the launch at KubeCon Seattle, but so far the response to the project has been overwhelming, with more the 4.5k GitHub stars and over 12hrs at the top spot on Hacker News.

May 23 - David Kaltschmidt, Fool-Proof Kubernetes Dashboards for Sleep-Deprived Oncalls
Software running on Kubernetes can fail in various, but surprisingly well-defined ways. In this intermediate-level talk David Kaltschmidt shows how structuring dashboards in a particular way can be a helpful guide when you get paged in the middle of the night. Reducing cognitive load makes oncall more effective. When dashboards are organized hierarchically on both the service and the resource level, troubleshooting becomes an exercise of divide and conquer. The oncall person can quickly eliminate whole areas of problems and zone in on the real issue. At that point a single service or instance should have been identified, for which more detailed debugging can take place.
Register Now

Monitorama PDX 2019 | Portland, OR - 06.03.19-06.05.19:

Tom Wilkie: Grafana Loki - Prometheus-inspired open source logging - Imagine if you had Prometheus for log files. In this talk we'll discuss Grafana Loki, our attempt at creating just that.
Learn More

We're Hiring

Have fun solving real world problems building the next generation of open source tools from anywhere in the world. Check out all of our current opportunities on our careers page.
View All our Open Positions

Tweet of the Week
We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove
Working on an upcoming Bar Gauge visualization in @Grafana, will have different looks, LCD , gradient, translucent etc :) Starting to come together, just need to add the series name in repeat mode. #monitoringlove #datavisualization pic.twitter.com/7t0JjQb7kn
— Torkel Ödegaard (@torkelo) March 15, 2019

We don't normally share our own tweets, but this one was too beautiful to pass up.

How are we doing?

We’re always looking to make TimeShift better. If you have feedback, please let us know! Email or send us a tweet, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

/blog/2019/03/15/timeshiftgrafanabuzz-1w-issue-81/
How New City Energy Is Supporting Sustainability with Grafana
• sorangutan

1

1
Posts

55
Views

Andrew Rodgers - New City Energy

Grafana isn’t just being used for monitoring applications. Andrew Rodgers, Senior Systems Architect at New City Energy, spoke in Amsterdam at GrafanaCon EU 2018 about how his organization is using Grafana to measure energy usage in municipal buildings in Washington D.C. for the Department of General Services (DGS).

DGS manages infrastructure facilities for 28 million square feet of real estate in the District of Columbia. Of its half a billion dollars in operating costs of all these buildings, $100 million is spent on energy per year.

In 2012, the city government established a division of addressing sustainability and energy, with a goal of carbon neutrality by 2050. A consultancy called NCE was brought in to help, and the first thing it did was build a platform called BuildSmart DC, which launched in the summer of 2013.

Making Energy Data Public

“It’s a publicly accessible portal that aggregates the energy consumption data for the entire D.C. municipal building portfolio for anyone to see,” said Rodgers. “It was groundbreaking because utilities in the U.S. don’t tend to share the data like you would hope. By putting that data in the public view, it also established a responsibility to the public from the D.C. government to address buildings that were, for example, using four times the national average for energy per square foot. It really provided a political cover for spending money on sustainability and efficiency.”

The portal is integrated with VOLTTRON, a U.S. Department of Energy-sponsored open source platform for transactional energy. “They envision this future where every time a device needs energy for something, it negotiates with the other devices in your house, and your house may negotiate with the other houses in your neighborhood to establish where that energy should come from,” said Rodgers. “How does it give you the best cost for the energy possible? How does it have the least environmental impact? You can build your market that makes these decisions around whatever metrics you want.”

The platform provides a framework for distributed sensing and control, with “sane security and trust defaults that allow you to build trusted distributed systems that can communicate with each other and negotiate these transactions or move data around,” Rodgers added. “It was a system built by researchers for researchers.”

Improving a Transactional Energy Platform

As such, Rodgers said, a lot of work was needed to make the platform more suitable for commercial adoption. Rodgers started working on the project in 2016, focusing on drilling down from the energy level data of a building to what each piece of equipment in the building is doing. The next step was retro commissioning, which is figuring out the difference between what the system is doing and what it should be doing, and restoring it to the desired performance.

Systems need to respond to constantly changing conditions and demands for room temperature, for instance, as people enter or leave a room.

“The focus on energy efficiency for mechanical engineers designing these systems has driven some really incredible innovations over the last few decades,” Rodgers said. “But they keep getting more and more complex, and if you don’t have that sort of continuous path into the facilities’ operators and all that complexity, suddenly you’ve designed something at one efficiency level, and the actual achieved is half of that.”

So one of the goals for the Built Environment portfolio for the city of D.C. is to define the metrics. “What you measure is what’s going to drive behavior,” Rodgers said. Knowing these metrics enables systems to reduce energy consumption as well as move it around in order to reduce the carbon footprint; for example, use wind energy when a lot of that is available.

Enter Grafana

The platform is now using Grafana to visualize these key metrics. Aside from reducing energy usage, an important goal of this initiative is to provide a comfortable and healthy workspace for the people in those municipal buildings.

Rodgers has been working on disaggregating the energy and operational data into multiple temperature zones within a single room. “We want to address not just efficiency; we are not trying to run at the bare minimum,” said Rodgers. “We want to make sure that we’re delivering the amount of oxygen people need, that the CO2 levels are regulated correctly. Ventilation is important.” The room temperatures, the airflow, the temperature of the air flow, the number of watts going into cooling each area of the room all need to be measured, and then delivered in a usable way.

“Data qualIty is very important in this space,” said Rodgers. “So our approach is to address all that at the front end, make sure we’re doing the right kind of tagging. We want to be able to use this data at multiple levels of the organization. We want the facilities guy that’s onsite here to be able to pull up his phone and answer questions that can help him solve the day-to-day problems. We want the technicians who are solving deeper technical problems with the equipment to be able to dig in and troubleshoot individual points. And then we wanted to deliver this energy cost efficiency to the executive level people who are making decisions about which buildings to sell or buy. And then we also want to deliver this data to the public as well because there is this value in establishing this sort of political accountability to how these systems operate.”

Building equipment has multiple standards, but Rodgers and his team figured out ways to aggregate them into one standard. The IoT gateway running the VOLTTRON platform interrogates using that standard, pushes the data up the message bus into a CrateDB cluster on AWS. They then visualize out of Crate into Grafana.

This system also uses Prometheus, Logstash, and Elasticsearch.

The platform now monitors 46 buildings, pulling about 30,000 data points out of those buildings. In total, it collects about 2.7 million samples per day, with a corpus of almost a billion records.

Technicians Building Dashboards

“Our dashboards are not beautiful,” Rodgers admitted, “but I am really proud of these because I didn’t build them.” Staying true to the goal of giving every level of the organization access to the metrics, they use dashboards used by actual HVAC technicians.

Rodgers showed a number of them during his presentation. “Yes, there’s too much data up there, but the fact that I have nontechnical users using Grafana and doing the data discovery process is huge for me,” he said. “I think that’s where the value of these systems over the vendor-driven systems really shines in this particular segment. So this just allows a technician to pick a location or a building, then a type of equipment, and see all the data points associated with that. If someone’s calling about a particular room, they can dig in to the room level and see what’s going on in that room.”

Pretty or not, this platform has saved the District of Columbia $4 million to date, Rodgers said. “We’re sitting at about a million dollars a year in annual recurring savings on the energy side, 20 percent reduction at the sites that we’ve really focused on,” he said. “And we’ve established the retro commissioning process that is using this data. We’re on track for this year to deliver an additional half million dollars a year in recurring savings.”

Looking Ahead

With such stellar results, what’s next for the platform? “We want to continue to develop the business processes,” Rodgers said. “The data doesn’t really matter if you’re not using it and making good decisions. We think that by really focusing on the right policies, especially at a municipal level, we’d love to influence commercial portfolio owners to adopt some of these practices to reduce the carbon footprint of the city at large.”

Constant commissioning is a top goal: “to use the data we’re collecting to find each little problem as it pops up, and solve it while it’s still addressable with the existing operations budget, before we end up needing a whole new $2 million mechanical system at this building,” he said.

Distributed energy resource is another. “The next transactional energy paradigm is the ability to deliver demand reduction across a large scale using whatever assets you have,” Rodgers said. “Whether you have electric vehicles plugged into your building, a photovoltaic system on the roof, actual mechanical systems that you can shut down or slow down, the goal is to, across a large portfolio, achieve megawatt or larger cuts in your demand.

And on the front lines of facilities management, Rodgers said he wanted to give everyone the tools they need to take ownership of the challenge of sustainability in a building. “Maybe we have to build some simplified things,” he said, “but eventually we want to get them using Grafana.”

While this system is now taking root in D.C., Rodgers said he has a greater mission: “We want to build the platform that enables this not just for D.C. but everywhere.”

Check out the full video of Andrew’s talk below and download his presentation slides.

Video: Bringing Observability to the Built Environment at City Scale

/blog/2019/03/14/how-new-city-energy-is-supporting-sustainability-with-grafana/
Grafana v6.0 Released
• sorangutan

1

1
Posts

50
Views
v6.0 Stable released!

GrafanaCon LA is underway! Check out the live stream on the Grafana youtube channel. I am proud to announce that we are releasing Grafana v6.0 stable today. Before every major release I feel super excited about the progress that has been made and with this release, this is especially true.

Grafana v6.0 brings many new and exciting features and enhancements and at the same time includes a new React platform and architecture for panels & plugins that will fundamentally change Grafana’s future as a platform.

What’s New in Grafana v6.0
- Explore - A new query focused workflow for ad-hoc data exploration and troubleshooting.
- Grafana Loki - Integration with the new open source log aggregation system from Grafana Labs.
- Gauge Panel - A new standalone panel for gauges.
- New Panel Editor UX improves panel editing and enables easy switching between different visualizations.
- Google Stackdriver Datasource is out of beta and is officially released.
- Azure Monitor plugin is ported from being an external plugin to being a core datasource
- React Plugin support enables an easier way to build plugins.
- Named Colors in our new improved color picker.
- Removal of user session storage makes Grafana easier to deploy & improves security.
Download Grafana 6.0 Now

Explore

Grafana’s dashboard UI is all about building dashboards for visualization. Explore strips away all the dashboard and panel options so that you can focus on the query & metric exploration. Iterate until you have a working query and then think about building a dashboard. You can also jump from a dashboard panel into Explore and from there do some ad-hoc query exporation with the panel queries as a starting point.

For infrastructure monitoring and incident response, you no longer need to switch to other tools to debug what went wrong. Explore allows you to dig deeper into your metrics and logs to find the cause. Grafana’s new logging datasource, Loki is tightly integrated into Explore and allows you to correlate metrics and logs by viewing them side-by-side.

Explore is a new paradigm for Grafana. It creates a new interactive debugging workflow that integrates two pillars of observability - metrics and logs. Explore works with every datasource but for Prometheus we have customized the query editor and the experience to provide the best possible exploration UX.

Explore and Prometheus

Explore features a new Prometheus query editor. This new editor has improved autocomplete, metric tree selector, integrations with the Explore table view for easy label filtering and useful query hints that can automatically apply functions to your query. There is also integration between Prometheus and Grafana Loki (see more about Loki below) that enabled jumping between metrics query and logs query with preserved label filters.

Explore splits

Explore supports splitting the view so you can compare different queries, different datasources and metrics & logs side by side!

Explore and Grafana Loki

The log exploration & visualization features in Explore are available to any data source but are currently only implemented by the new open source log aggregation system from Grafana Lab called Grafana Loki.

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective, as it does not index the contents of the logs, but rather a set of labels for each log stream. The logs from Loki are queried in a similar way to querying with label selectors in Prometheus. It uses labels to group log streams which can be made to match up with your Prometheus labels.

Read more about Grafana Loki here or Grafana Labs hosted Loki.

The Explore feature allows you to query logs and features a new log panel. In the near future, we will be adding support for other log sources to Explore and the next planned integration is Elasticsearch.
Your browser does not support the video tag.

New Panel Editor

Grafana v6.0 has a completely redesigned UX around editing panels. You can now resize the visualization area if you want more space for queries & options and vice versa. You can now also change visualization (panel type) from within the new panel edit mode. No need to add a new panel to try out different visualizations! Checkout the video below to see the new Panel Editor in action.
Your browser does not support the video tag.

Gauge Panel

We have created a new separate Gauge panel as we felt having this visualization be a hidden option in the Singlestat panel was not ideal. When it supports 100% of the Singlestat Gauge features we plan to add a migration so all singlestats that use it become Gauge panels instead. This new panel contains a new Threshold editor that we will continue to refine and start using in other panels.

React Panels & Query Editors

A major part of all the work that has gone into Grafana v6.0 has been on the migration to React. This investment is part of the future proofing of Grafana and it’s code base and ecosystem. Starting in v6.0 Panels and Data source plugins can be written in React using our published @grafana/ui sdk library. More information on this will be shared closer to or just after release.

Google Stackdriver Datasource

Built-in support for Google Stackdriver is officially released in Grafana 6.0. Beta support was added in Grafana 5.3 and we have added lots of improvements since then.

To get started read the guide: Using Google Stackdriver in Grafana.

Azure Monitor Datasource

One of the goals of the Grafana v6.0 release is to add support for the three major clouds. Amazon Cloudwatch has been a core datasource for years and Google Stackdriver is also now supported. We developed an external plugin for Azure Monitor last year and for this release the plugin is being moved into Grafana to be one of the built-in datasources. For users of the external plugin, Grafana will automatically start using the built-in version. As a core datasource, the Azure Monitor datasource will get alerting support for the official 6.0 release.

The Azure Monitor datasource integrates four Azure services with Grafana - Azure Monitor, Azure Log Analytics, Azure Application Insights and Azure Application Insights Analytics.

Provisioning support for alert notifiers

Grafana now added support for provisioning alert notifiers from configuration files. Allowing operators to provision notifiers without using the UI or the API. A new field called uid has been introduced which is a string identifier that the administrator can set themselves. Same kind of identifier used for dashboards since v5.0. This feature makes it possible to use the same notifier configuration in multiple environments and refer to notifiers in dashboard json by a string identifier instead of the numeric id which depends on insert order and how many notifiers that exists in the instance.

Easier to deploy & improved security

Grafana 6.0 removes the need of configuring and setup of additional storage for user sessions. This should make it easier to deploy and operate Grafana in a high availability setup and/or if you’re using a stateless user session storage like Redis, Memcache, Postgres or MySQL.

Instead of user sessions a solution based on short-lived tokens that are rotated frequently have been implemented. This also replaces the old “remember me cookie” solution, which allowed a user to be logged in between browser sessions, and which have been subject to several security holes throughout the years. Read more about the short-lived token solution and how to configure it here.

Please note that due to these changes, all users will be required to login upon next visit after upgrade.

Besides these changes we have also made security improvements regarding Cross-Site Request Forgery (CSRF) and Cross-site Scripting (XSS) vulnerabilities:
- Cookies are per default using the SameSite attribute to protect against CSRF attacks
- Script tags in text panels are per default disabled to protect against XSS attacks
If you’re using Auth Proxy Authentication you still need to have user sessions setup and configured but our goal is to remove this requirements in a near future.

Named Colors

We have updated the color picker to show named colors and primary colors. We hope this will improve accessibility and helps making colors more consistent across dashboards. We hope to do more in this color picker in the future, like show colors used in the dashboard.

Named colors also enables Grafana to adapt colors to the current theme.

Other features
- The ElasticSearch datasource now supports bucket script pipeline aggregations. This gives the ability to do per bucket computations like the difference or ratio between two metrics.
- Support for Google Hangouts Chat alert notifications
- New built in template variables for the current time range in $__from and $__to
Upgrading

See upgrade notes.

Changelog

Checkout the CHANGELOG.md file for a complete list of new features, changes, and bug fixes.

Download

Head to download page for download links & instructions.

Thanks

A big thanks to all the Grafana users who contribute by submitting PRs, bug reports & feedback!

http://blog.grafana.com.s3-website-us-west-2.amazonaws.com/2019/02/25/grafana-v6.0-released/

3 / 5

A Look at the Latest Cloud Data Source Plugins in Grafana • sorangutan

Azure

Stackdriver

Oracle

Grafana v6.1 Released • sorangutan

v6.1 Stable released!

What’s New in Grafana v6.1

Ad hoc filtering for Prometheus

Editors can own dashboards, folders, and teams they create

List and revoke user auth tokens in the API

Minor Features and Fixes

Changelog

Download

Thanks

Grafana Plugin Tutorial: Polystat Panel (Part 1) • sorangutan

Polystat

The Basics

CPU

Disk

Memory

Formatting

Thresholding

Composites

Clickthroughs

Default Clickthrough

Override Clickthrough

Composite Clickthrough

Templating

Almost there

Wrapping Up

About Part 2

How We're Using Prometheus Subqueries at Grafana Labs. • sorangutan

Billing

Capacity Planning

Gotchas

timeShift(GrafanaBuzz, 1w) Issue 83 • sorangutan

Welcome to TimeShift

Latest Beta Release: Grafana v6.1.0-beta1

Major New Features

From the Blogosphere

Grafana Plugin Update

Upcoming Events

We're Hiring

How are we doing?

Everything You Need to Know About the OSS Licensing War, Part 2. • sorangutan

An Escalation of Hostilities

The Nuclear Option

Tinder & Grafana: A Love Story in Metrics and Monitoring • sorangutan

First Impressions: Swiping Right on Grafana

Meeting the Family: Introducing Prometheus

Getting Serious: Committing to Long-Term Data

Moving in Together: Migrating to Kubernetes

Writing React Plugins • sorangutan

Setup

The Structure

The Panel

The Panel editor

Types

Building

Testing

Writing React Plugins • sorangutan

Setup

The Structure

The Panel

The Panel editor

Types

Building

Testing

What’s New in Prometheus 2.8: WAL-Based Remote Write • sorangutan

What’s New in Prometheus 2.8: WAL-Based Remote Write • sorangutan

timeShift(GrafanaBuzz, 1w) Issue 82 • sorangutan

Welcome to TimeShift

Latest Stable Release: Grafana v6.0.2

Bug Fixes

From the Blogosphere

Grafana Plugin Update

Upcoming Events

We're Hiring

Tweet of the Week

How are we doing?

A Look at the Latest Cloud Data Source Plugins in Grafana
• sorangutan

Grafana v6.1 Released
• sorangutan

Grafana Plugin Tutorial: Polystat Panel (Part 1)
• sorangutan

How We're Using Prometheus Subqueries at Grafana Labs.
• sorangutan

timeShift(GrafanaBuzz, 1w) Issue 83
• sorangutan

Everything You Need to Know About the OSS Licensing War, Part 2.
• sorangutan

Tinder & Grafana: A Love Story in Metrics and Monitoring
• sorangutan

Writing React Plugins
• sorangutan

Writing React Plugins
• sorangutan

What’s New in Prometheus 2.8: WAL-Based Remote Write
• sorangutan

What’s New in Prometheus 2.8: WAL-Based Remote Write
• sorangutan

timeShift(GrafanaBuzz, 1w) Issue 82
• sorangutan

timeShift(GrafanaBuzz, 1w) Issue 82
• sorangutan

Pro Tips: How Booking.com Handles Millions of Metrics Per Second with Graphite
• sorangutan

Pro Tips: How Booking.com Handles Millions of Metrics Per Second with Graphite
• sorangutan

Everything You Need to Know About the OSS Licensing War, Part 1.
• sorangutan

GrafanaCon L.A. Recap: Grafana 6.0, LGTM, and More!
• sorangutan

timeShift(GrafanaBuzz, 1w) Issue 81
• sorangutan

How New City Energy Is Supporting Sustainability with Grafana
• sorangutan

Grafana v6.0 Released
• sorangutan