- What is a Zipkin Trace?
- Instrumenting Traces with Zipkin
- Tracing NodeJS Services with Open Telemetry
- OpenTelemetry’s NPM Packages
Last time we ran a small, three service pre-instrumented system that reported spans into Zipkin. In this article we’re going to take a look at how we instrumented that system using Zipkin’s free and open source libraries.
This article assumes you have a version of NodeJS on your computer, a Zipkin instance running on port 9411
, and have a copy of our sample services checked-out/cloned on your computer. If you’re not sure what we’re talking about, checkout part one of this series.
What is Zipkin?
Zipkin is a system designed to create, store, and display distributed traces. Each span in Zipkin represents a single request to a service. This service could be HTTP based, but it doesn’t need to be.
In order to instrument a service, Zipkin will need to
- Do some work whenever code makes a request to a service
- Do some work when a service starts processing a request
Whenever we make a service request, we need to either create a new trace ID or determine what the current trace ID is. We also need to pass that trace ID along with the service request. This trace ID allows Zipkin to connect all the traces together.
We need to pass that trace ID along so that when a service starts processing a request it knows what trace the request is a part of.
Our Service
Let’s take a look at one of our un-instrumented sample services from our previous article.
// File: uninstrumented/src/service-main.js
const express = require('express')
const fetch = require('node-fetch')
const {getUrlContents} = require('./util')
const setupMainProgram = function() {
const app = express()
const port = 3000
app.get('/main', async function (req, res) {
// fetch data from second service running on port 3001
const resultString = await getUrlContents(
'http://localhost:3001/hello', fetch)
const resultJson = JSON.parse(resultString)
res.type('json')
res.send(JSON.stringify({main:resultJson.message}))
})
app.listen(
port,
function() {
console.log(`Example app listening at http://localhost:${port}`)
}
)
}
setupMainProgram()
As programs go it’s not too complex. If there’s concepts in here that you haven’t encountered before, the Mozilla Developers Network express tutorials are a great place to get up to speed on express
services.
The two parts of this program we’re interested in are the code that sets up our /main
route/URL, and the code that “makes a service call” — i.e. the code that fetches the contents of http://localhost:3001/hello
. You can see the route setup here.
// File: uninstrumented/src/service-main.js
const fetch = require('node-fetch')
/* ... */
app.get('/main', async function (req, res) {
// fetch data from second service running on port 3001
const resultString = await getUrlContents(
'http://localhost:3001/hello', fetch)
const resultJson = JSON.parse(resultString)
res.type('json')
res.send(JSON.stringify({main:resultJson.message}))
})
The function passed as the second argument to app.get
is the function that express will call when handling a request to http://localhost:3001/main
The getUrlContents
function is what fetches a URL using the node-fetch
library. We can see the source of getUrlContents
here
//File: uninstrumented/src/util.js
const getUrlContents = function(url, fetch) {
return new Promise((resolve, reject)=>{
fetch(url)
.then(res => res.text())
.then(body => resolve(body));
})
}
module.exports = {
getUrlContents
}
This function uses the passed in node-fetch
library (the fetch
variable above) to retrieve the URL’s contents, and then the route handler uses those results as part of its own response.
Instrumenting with Zipkin
Earlier, we said that in order to instrument a service with Zipkin, we need to
either create a new trace ID or determine what the current trace ID is. We also need to pass that trace ID along with the service request.
For this particular service, that means we’ll need to
- Add a Zipkin middleware to our
express
application (this creates-or-determines the trace ID) - Wrap the
node-fetch
library with a Zipkin wrapper (this passes trace ID along to the next service)
We can see the instrumented version of this service here.
Instrumenting Express
Adding the express middleware is the more straight forward of the two tasks — we can see the code required in this partial fragment
// File: instrumented/src/util.js
const createTracer = (localServiceName) => {
const tracer = new Tracer({
ctxImpl: new CLSContext('zipkin', true),
recorder: new BatchRecorder({
logger: new HttpLogger({
endpoint: 'http://localhost:9411/api/v2/spans',
jsonEncoder: JSON_V2
})
}),
localServiceName: localServiceName // name of this application
});
return tracer;
}
// File: instrumented/src/service-main.js
const tracer = createTracer('service-main')
/* ... */
app.use(createZipkinMiddleware({tracer}));
app.get('/main', async function (req, res) {
// fetch data from second service running on port 3001
const zipkinFetch = createFetcher('service-hello', tracer)
const resultString = await getUrlContents(
'http://localhost:3001/hello', zipkinFetch)
const resultJson = JSON.parse(resultString)
res.type('json')
res.send(JSON.stringify({main:resultJson.message}))
})
This code creates a tracer object, which includes the name we want to use for the service (service-main
)
// File: instrumented/src/service-main.js
const tracer = createTracer('service-main')
Then we load the Zipkin library for instrumenting express
// File: instrumented/src/service-main.js
const createZipkinMiddleware = require('zipkin-instrumentation-express').expressMiddleware;
and then we use the returned createZipkinMiddleware
function to create a middleware. Finally the code adds that express middleware to the current application.
// File: instrumented/src/service-main.js
app.use(createZipkinMiddleware({tracer}));
Important: Be sure to add this app.use
line before your call to app.get
. Route-less Express middleware will not fire if they’re added after another middleware that calls res.end()
.
Instrumenting node-fetch
Next — we’ll take a look at the important bits for instrumenting node-fetch
// File: instrumented/src/util.js
const createFetcher = (remoteServiceName, tracer) => {
const wrapFetch = require('zipkin-instrumentation-fetch');
return wrapFetch(fetch,
{
tracer:tracer,
remoteServiceName:remoteServiceName
}
);
}
// File: instrumented/src/service-main.js
const fetch = require('node-fetch')
/* ... */
const zipkinFetch = createFetcher('service-hello', tracer)
const resultString = await getUrlContents(
'http://localhost:3001/hello', zipkinFetch)
Unlike express, node-fetch
doesn’t have the concept a middleware or plugin system. In order to instrument the node-fetch
module, Zipkin needs to wrap the node-fetch
module. That is, the wrapFetch
function provided by the zipkin-instrumentation-fetch
module will take an object that implements the fetch api, and returns a version with its methods changed such that they perform the necessary steps to add the trace ID to the outgoing request.
// File: instrumented/src/util.js
const zipkinFetch = createFetcher('service-hello', tracer)
// File: instrumented/src/util.js
const createFetcher = (remoteServiceName, tracer) => {
const wrapFetch = require('zipkin-instrumentation-fetch');
return wrapFetch(fetch,
{
tracer:tracer,
remoteServiceName:remoteServiceName
}
);
}
The wrapFetch
function requires us to provide our previously instantiated tracer, as well as provide the name of the remote service.
With this new version of fetch in hand (stored in the zipkinFetch
variable below)
// File: TO DO find file, use up to date code
const fetch = require('node-fetch')
/* ... */
const zipkinFetch = createFetcher('service-hello', tracer)
we can then use it in our getUrlContents
function call
// File: TO DO find file, use up to date code
const resultString = await getUrlContents(
'http://localhost:3001/hello', zipkinFetch)
What is the Tracer
You probably noticed that both the express and fetch instrumentation required an instantiated Tracer object, created via our createTrace
function.
// File: instrumented/src/util.js
const {
Tracer,
BatchRecorder,
jsonEncoder: {JSON_V2}
} = require('zipkin');
const CLSContext = require('zipkin-context-cls');
const {HttpLogger} = require('zipkin-transport-http');
const createTracer = (localServiceName) => {
const tracer = new Tracer({
ctxImpl: new CLSContext('zipkin', true),
recorder: new BatchRecorder({
logger: new HttpLogger({
endpoint: 'http://localhost:9411/api/v2/spans',
jsonEncoder: JSON_V2
})
}),
localServiceName: localServiceName // name of this application
});
return tracer;
}
The tracer is a single instance object that contains the basic configuration that other instrumentation modules will need to trace the application.
The localServiceName
parameter is our Zipkin name for the service we’re instrumenting.
The recorder
parameter is an object we want to use to send data to our actual Zipkin system. You’ll see we’ve used a BatchRecorder
object configured with a HttpLogger
object that’s pointed at our http://localhost:9411
Zipkin URLs. We won’t get too into how these object work, but it’s worth investigating if you’re planning on using Zipkin to trace your systems.
The ctxImpl
parameter is an object that implements Zipkin’s (implicit) context interface. It’s beyond the scope of this article to explain tracing context in greater detail, but in Zipkin it’s the thing that makes sure that a trace ID stays consistent between the asynchronous callbacks in Node’s network handling code.
Zipkin Challenges
So that’s how you instrument an express service using Zipkin.
If you weren’t using express (or an express compatible framework) you’d need to find a Zipkin library for that framework. For example, there’s also a library for instrumenting applications built using the Koa framework.
Similarly — if your application uses a library other than node-fetch
for making HTTP requests, you’d need to find another Zipkin library. For example, if you were using the got
framework to make HTTP requests, there’s a zipkin-instrumentation-got
library.
If you can’t find a library for the package you’re using, you’re of out of luck. While it’s certainly possible to implement your own instrumentation library, there’s not a well documented path forward for doing so. Even if you’re willing to figure out how these Zipkin libraries work, the Zipkin project itself doesn’t (appear to?) offer a formal, supported API for writing instrumentations.
And then — even if you do invest the effort into writing custom instrumentation — those instrumentations are Zipkin specific. You have no easy way to switch to a different tracing library. Similar to what we saw with Prometheus, even though Zipkin is an open system based on open source code, its complexity has created a situation where you can still end up locked into a specific open source ecosystem.
What to do? In out next article, we’ll take a look at the OpenTelemetry project’s tracing code, and how it can offer you a path out of this open-source-vendor-lock-in.