What are OpenTelemetry Metrics and Exporters

Last time we ended by musing that Prometheus, while an open source project, still creates a case of vendor lock-in for its users. Getting metrics IN to Prometheus is easy. Getting metrics out is less easy. Changing-to or trying-out a new metric system is even less easy.

Today we’re going to take a look at the OpenTelemetry project. OpenTelemetry is an open source project whose aim is to create a common, OpenTelemetry platform that’s capable of sending your telemetry data to any monitoring platform. Instrument things once with OpenTelemetry, and you can send your data to any system that’s capable of importing it.

Today we’ll take a look at creating a counting metric using the OpenTelemetry metrics library. Then we’ll show you how you can export this metric data to any platform — including Prometheus!

To get the most out of this tutorial you’ll want to be familiar with some starting Prometheus concepts we covered in the first two articles in this series.

Instrumenting a Service

Let’s start with the same small NodeJS program from our first article.

// File: index.js
const express = require('express')

const app = express()
const port = 3000

app.get('/stuff', function (req, res) {
res.type('json')
  res.send(JSON.stringify({hello:"world"}))
})

app.listen(
  port,
  function() {
    console.log(`Example app listening at http://localhost:${port}`)
  }
)

We’re going to do the same thing we did in part one: Create a metric that counts how many times the /stuff endpoint gets called. However, this time we’ll use the OpenTelemetry metrics system instead of Prometheus.

To start, we’ll want to install the @opentelemetry/metrics package into our project.

$ npm install @opentelemetry/metrics

Then, we’ll use this library to instrument our program.

const express = require('express')
// NEW: The `MeterProvider` class creates factory-ish objets that
//      allows us to create `Meter` objects.  `Meter` objects are
//      a sort of global-ish singleton-ish object that we'll use to create
//      all our metrics.
const { MeterProvider } = require('@opentelemetry/metrics');

// NEW: Here's where we instantiate the `MeterProvider`.  The `my-meter`
//      string that uniquely identifies this meter.  The `getMeter` method
//      will fetch a meter if it already exists, or create a new one
//      if it does not.
const meter = new MeterProvider().getMeter('my-meter');

// NEW: Here we use our `Meter` object to create a counter metric.
//      This is similar to `require('prom-client').Counter(...)`
//      from the prom-client library
const counter = meter.createCounter('my-open-telemetry-counter', {
  description: 'A simple counter',
});

const app = express()
const port = 3000

app.get('/stuff', function (req, res) {
  // NEW: Here we increment our counter by 1.  This is similiar to
  //      the counter.inc() in prom-client, except that we're passing
  //      in the amount to increment a counter by.

  // NOTE: Ideally we should be able to invoke this via
  //       `counter.add(1)` once this Issue lands
  //
  //       https://github.com/open-telemetry/opentelemetry-js/issues/1031
  //
  //       If this issue hasn't landed, you'll need to pass an empty
  //       object in as the second paramter
  counter.add(1, {})
  res.type('json')
  res.send(JSON.stringify({hello:"world"}))
})

app.listen(
  port,
  function() {
    console.log(`Example app listening at http://localhost:${port}`)
  }
)

Here we see patterns that are similar patterns to the Prometheus instrumentation library. We create a counter metric

const counter = meter.createCounter('my-open-telemetry-counter', {
  description: 'A simple counter',
});

and then we increment that metric.

// counter.add(1, {})
counter.add(1, {})

What’s a little different here is the MeterProvier and Meter objects

const { MeterProvider } = require('@opentelemetry/metrics');

const meter = new MeterProvider().getMeter('my-meter');

A Meter object (created with a MeterProvider object) is what we use to create our metrics. This is just a case of the OpenTelemetry project being a little more abstracted than the prom-client library we used with Prometheus. This will be important later, but for now let’s keep our focus on the metrics themselves.

An Exporter by any Other Name

So far this all seems similar to what we saw with Prometheus. OpenTelemetry’s big differentiation is how we, or other systems, are able to view these metrics. There is not, at the time of this writing, a client for scraping OpenTelemetry metrics or a UI for viewing them. Instead, we need to rely on OpenTelemetry Exporters to move our data into other systems.

These OpenTelemetry exporters are not Prometheus exporters. In Prometheus, an exporter allows you to “export” metrics from a system that isn’t instrumented by Prometheus.

An OpenTelemetry Exporter has a different job. In OpenTelemetry, an exporter allows you to export metrics from an instrumented application to another third party system.

There is an OpenTelemetry exporter capable of exporting OpenTelemetry metrics in a way where the Prometheus client can consume them. Put another, even more confusing way, there’s an an OpenTelemetry Prometheus Exporter that is not a Prometheus Exporter, but is an OpenTelemetry Exporter.

Yes, naming things remains hard. Hopefully some code will clear things up.

Remember that Meter object we created? Let’s change its instantiation so it looks like this

// NEW: pulls in the `PrometheusExporter` class from the
//      '@opentelemetry/exporter-prometheus library
const { PrometheusExporter } = require('@opentelemetry/exporter-prometheus');

// NEW: Instantiates a `PrometheusExporter` object.
const exporter = new PrometheusExporter(
  {startServer: true,},
  () => {
    console.log('prometheus scrape endpoint: http://localhost:9464/metrics');
  },
);

// NEW: When creating our meter object
const meter = new MeterProvider({
  exporter:exporter,
  interval: 1000,
}).getMeter('my-meter');

This new code uses a class from the @opentelemetry/exporter-prometheus package, so we’ll want to pull that into our project

$ npm install @opentelemetry/exporter-prometheus

Run your program,

$ node index.js

send some traffic to /stuff,

$ curl http://localhost:3000/stuff

and then hit the /metrics URL exposed by the OpenTelemetry Prometheus exporter

$ curl http://localhost:9464/metrics
# HELP my_open_telemetry_counter A simple counter
# TYPE my_open_telemetry_counter counter
my_open_telemetry_counter 3 1589046070557

Congratulations! You just exported your OpenTelemetry metrics. From here all you’d need to do is point your Prometheus client at this URL (as we did in part one of this series), and your OpenTelemetry metrics would be imported into Prometheus

What just Happened?

A Meter object is aware of every metric that’s created and, in turn, aware of the current value of every metric. When you instantiate a Meter, you can provide it with an exporter object.

const meter = new MeterProvider({
  exporter:exporter,
  interval: 1000,
}).getMeter('my-meter');

When you configure an exporter, the Meter object will send that exporter a list of every metric object on a timed interval. That’s what the interval configuration field is for. The Meter will send the exporter every metric in the system once every 1000 milliseconds (i.e. once a second). It’s the job of an exporter to examine these records, and then take steps to make that data available to other systems. What these steps are will depend entirely on the system you’re trying to export your metrics to.

In the case of the Prometheus exporter, those steps are exposing an HTTP endpoint that a Prometheus client can read, and printing out the metric data in a form that Prometheus understands. If you wanted to export your metrics to another system, you’d probably take different steps (POSTing those metric values to an HTTP endpoint, streaming them over GRPC, writing them out to disk for consumption by a cron job, etc.)

For example, here’s “The World’s Simplest Exporter™”: It sends metric data to the console of the running application.

const simpleExporter = {
  export: function(metrics, resultCallback) {
    for(const [,metric] of metrics.entries()) {
      console.log(metric)
    }

    // you'll need to npm install @opentelemetry/base to get
    // access to the status code expected by
    resultCallback(
      require("@opentelemetry/base").ExportResult.SUCCESS
    )
  },
  shutdown: function(shutdownCallback) {
    console.log('do any work needed to shutdown your exporter')
    // call shutdownCallback to indicate exporter is shutdown
    shutdownCallback()
  }
}

All we need to do is add this simpleExporter object to the configuration when you’re instantiating your Meter object

const meter = new MeterProvider({
  exporter: simpleExporter,
  interval: 1000,
}).getMeter('my-meter');

Run your program with this exporter in place, send some traffic to /stuff, and you’ll see your metric printed out to the console about once a second.

$ node index.js
Example app listening at http://localhost:3000
{
  descriptor: {
    name: 'my-open-telemetry-counter',
    description: 'A simple counter',
    unit: '1',
    metricKind: 0,
    valueType: 1,
    labelKeys: [],
    monotonic: true
  },
  labels: {},
  aggregator: CounterSumAggregator {
    _current: 1,
    _lastUpdateTime: [ 1589046826, 890210944 ]
  }
}

OpenTelemetry is built around the idea that there should be a central way to create instrumentation data, but that anyone should be able to export and use that data. While it would be easy to dismiss OpenTelemetry as just one more standard, the OpenTelemetry team is doing everything they can to make this system compatible with your existing solution.

Wrap Up

There’s a lot more to OpenTelmetry than what we discussed in these three articles. We only scratched the surface on Metrics, and we didn’t even get into Traces/Spans. If you want to learn more the specification overview document, while dense, is as good a place as any to start. From there exploring the code and engaging with the community via their mailing lists and Gitter Chat are the best ways to get started.

I suspect I’ll have more to say on the subject as time goes on.