How to Write a Metrics Exporter

As the nature of monitoring/observability has changed over the years, the exercise I've given engineers in training has been the same: monitor the weather in Boston, fire an alert when it gets hot. I did this with Nagios plugins in Icinga, and I do it with Prometheus now. I'll likely keep doing this whenever the next cool thing comes out.

Today, I'm going to give a brief overview on how to write an exporter in Go from scratch. We're not writing textfile collector scripts for node-exporter, we're just diving right in.

First, here are some resources worth checking out.

• Writing Exporters (official Prometheus documentation)
• Prometheus Client Libraries (in case you don't want to use Go)
• client_golang documentation
• Default port allocations for Prometheus-related items
• The source code from the last time I ran this as a demo

Our input source: weather.gov

The National Weather Service provides a free API that doesn't require credentials or a token. This is fantastic for training exercises such as this. Search for your zip code and you'll find the code name of the nearest weather station. I'm going to be using Boston (KBOS) for my example code.

Replace KBOS in the following URL with your local weather station, or just use Boston.

https://api.weather.gov/stations/KBOS/observations/latest

Look at that wonderful, free JSON payload full of useful information we can extract! This will be key to our project.

Initializing Your Go Project

Learning Go is a good book. If you don't know Go but know something else, you can grab the documentation for the client library for your preferred language and adapt the idea here. I'm not going to be too verbose in explaining Go concepts, but I'll at least get you moving.

Install Go with dnf or apt or brew or pacman or snap or flatpak or just get it from them directly. Create a folder for your project, initialize your module, and then update your go.modfile.

% go mod init training-exercise
go: creating new go.mod: module training-exercise
% go get github.com/prometheus/client_golang@v1.17.0

# update go.mod with the following
module training-exercise

go 1.21

require github.com/prometheus/client_golang v1.17.0

Although editor wars are as old as time itself (I'm still team Emacs), every Go developer I know uses Microsoft Visual Studio Code because the Go integration for it is just that good. I recommend setting that up and enabling all the Go extensions if you're going to do this at length.

What We're Building

Prometheus client libraries for different programming languages typically have everything you need to build an exporter. They define, they increment, they parse, they even serve. Go's client library uses the built in net/http library to serve the metrics. This means we can crank out this example at about 100 lines of code or less.

Our goal is the following:

• Provide Boston's temperature in Celsius (gauge).
• Report how any times we've scraped the endpoint (counter).

The basic example from the Go doc for this client library is relatively close to what we're going to do. It doesn't really do anything though; we're going to write something that actually works.

Writing the Exporter

Start writing main.go. First, let's import our needed modules and set a couple of constants.

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

const Namespace = "helloworld"
const WeatherStation = "KBOS" // Boston Logan Airport

Our Namespace constant is worth noting. Prometheus metrics follow a convention that is roughly exportername_metricname_unitofmeasurement. You should definitely read the document on this before starting your own exporter. In this case, our metrics will start with helloworld_.

We're importing some standard things. We need the json and http modules to get the weather data, the Prometheus module for formatting metrics, and the promhttp one for magically making the endpoint appear.

We need to define a couple types.

type WeatherResponse struct {
	Properties struct {
		Temperature struct {
			Value float64 `json:"value"`
		} `json:"temperature"`
	} `json:"properties"`
}

type metrics struct {
	temperature *prometheus.GaugeVec
	queries     *prometheus.CounterVec
}

WeatherResponse is pretty straightforward; that's where the temperature is hiding in that giant payload and we need to extract it. The metrics struct on the other hand is different. Here we're defining the two metrics we want, one being a gauge, one being a counter.

You'll notice they are appended with Vec, you're more often than not going to define them this way than plainly. All this means is that you can add labels to the metrics.

Alright, let's make a function to get the weather.

func getWeather() (float64, error) {
	url := fmt.Sprintf("https://api.weather.gov/stations/%s/observations/latest", WeatherStation)
	httpClient := &http.Client{Timeout: time.Second * 30}
	res, err := httpClient.Get(url)

	if err != nil {
		log.Printf("ERROR: Could not fetch %s", url)
		return 0, err
	}

	defer res.Body.Close()
	body, err := io.ReadAll(res.Body)

	var weather WeatherResponse
	err = json.Unmarshal(body, &weather)

	if err != nil {
		log.Println("ERROR: Could not unmarshal json")
		return 0, err
	}

	temperature := weather.Properties.Temperature.Value
	return temperature, nil
}

First thing to discuss here is the type we chose. The Prometheus client library for Go expects all metric values to be of type float64. Other than that, this is the same basic unmarshal json into a struct strategy that you'll see in any other document about interacting with APIs in Go.

We have to register our metrics before we can start assigning values to them.

func RegisterMetrics(reg prometheus.Registerer) *metrics {
	m := &metrics{
		temperature: prometheus.NewGaugeVec(
			prometheus.GaugeOpts{
				Namespace: Namespace,
				Name:      "outdoor_temperature_celsius",
				Help:      "Outdoor temperature reported by the NWS.",
			},
			[]string{"station"},
		),
		queries: prometheus.NewCounterVec(
			prometheus.CounterOpts{
				Namespace: Namespace,
				Name:      "nws_query_attempts_total",
				Help:      "Number of times we've queried the NWS API.",
			},
			[]string{"station"},
		),
	}
	reg.MustRegister(m.temperature)
	reg.MustRegister(m.queries)
	return m
}

MustRegister must be run against all metrics before they're used. Just think of it as any other kind of initialization.

You'll also notice the Help values. If you've ever looked at the output of a Prometheus exporter, then you know these are expected and need to be defined.

The last argument here is a []string which presently contains the word "station". This is where you define the names of all the labels, their assignment happens later.

Okay, now let's prepare to actually run the thing.

func main() {

	reg := prometheus.NewRegistry()
	m := RegisterMetrics(reg)

	go func() {
		for {
			temperature, err := getWeather()
			m.queries.With(prometheus.Labels{"station": WeatherStation}).Inc()
			if err != nil {
				log.Print(err)
				log.Println("Not updating metrics on this run.")
				time.Sleep(time.Minute * 1)
				continue
			}
			m.temperature.With(prometheus.Labels{"station": WeatherStation}).Set(temperature)
			time.Sleep(time.Minute * 5)
		}
	}()

	http.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{Registry: reg}))
	log.Fatal(http.ListenAndServe(":23456", nil))
}

First thing we did was define a new registry and assign these new metrics to it. There is a default registry full of some standard Go metrics that we don't really need in this case.

We're launching a goroutine to infinitely loop over the getWeather() function with a 5 minute wait. If Prometheus scrapes on average every 10-60 seconds, why bother with a 5 minute wait? Well, weather.gov updates kind of randomly, but in my experience it's an average of once an hour. It's also free. Spamming a free API that doesn't update often is just rude. This is a good thing to consider when writing this sort of program; is my data actually changing frequently, and how intensive is it to determine?

Our gauge is having a value explicitly set, while our counter is using .Inc(). Counters only go up until the exporter restarts, at which point it resets to zero. As well, we're making sure the "station" label equals KBOS (or whatever you set it to).

Finally, we're serving an http endpoint on port 23456. Honestly, pick any port that nothing on this system is using nor that is used by another exporter.

Let's put this together.

% go mod tidy
go: downloading github.com/prometheus/procfs v0.11.1
go: downloading github.com/prometheus/client_model v0.4.1-0.20230718164431-9a2bf3000d16
go: downloading github.com/prometheus/common v0.44.0
go: downloading github.com/cespare/xxhash/v2 v2.2.0
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.4
go: downloading github.com/golang/protobuf v1.5.3
% go build .
% ./training-exercise

Running Your Exporter

If everything compiled without error, we should be able to curl our endpoint and see actual metrics.

% curl localhost:23456/metrics
# HELP helloworld_nws_query_attempts_total Number of times we've queried the NWS API.
# TYPE helloworld_nws_query_attempts_total counter
helloworld_nws_query_attempts_total{station="KBOS"} 1
# HELP helloworld_outdoor_temperature_celsius Outdoor temperature reported by the NWS.
# TYPE helloworld_outdoor_temperature_celsius gauge
helloworld_outdoor_temperature_celsius{station="KBOS"} -0.6
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0

In Conclusion

It really is that easy. You can monitor pretty much anything, which should be reassuring to old guys like me who had written hundreds of Nagios plugins in the past. Now go instrument your applications and delete those old cron jobs.