An Open Source Tutorial – InApps 2022

Recently adopted by the Cloud Native Computing Foundation, Prometheus is an open-source systems monitoring and alerting toolkit, focused on supporting the operation of microservices and containers. Like any open source project, it can be augmented with additional capabilities.

Contributing to Prometheus is no different than most other open source endeavors, which, like many projects, welcomes community contributions. Let’s gain better familiarity with the process by augmenting Prometheus’ Alert Manager with a new “history” view.

Tip: See this guide for a more complete approach to contributing to an open source project in general.

The first step, naturally, is to check out the contributing guidelines for the specific repository (in this case, Alert Manager’s).

When electing to contribute to any open source project, you’ll want to ensure that you are capable of wielding the technologies used with the project — in this case, those are Go, AngularJS, SQL, etc.

The AlertManager component handles alerts sent by client applications such as the Prometheus server, carefully de-duplicating, correlating, and routing their notifications to their appropriate receiver (e.g. email, webhook, etc.). Current behavior of this component is only to display actively firing alerts.

Prometheus AlertManager — no alert history

What if you’d like to get a historical perspective of alerts that have been notified through AlertManager? Enter our user story:

Story:

As an Operator, I would like to not only see a list of firing alerts, but also a list of all transpired alerts, so that I may have additional context as the thresholding behavior for a given defined alert.

Limitations:

1. AlertManager database (SQLite) is not intended to provide long-term storage.

Acceptance Criteria:

1. Once fired, whether actively firing or not, alerts will be displayed on the History page.
2. Optionally, fired alerts will be notified to a Slack channel.

With this limited scope, we’re able to concentrate on a subsection of the full Prometheus architecture, the Alert Manager, which runs as a separate binary apart from Prometheus itself:

Prerequisites

With architecture in hand and user story in mind, let’s create our development environment. Our user story does not call for changes to the Prometheus server itself. So, we only need to interact with this component as a pre-compiled binary. Follow the getting started instructions to download, configure and run the Prometheus server. Be sure to create and run the random sample targets and point it at your soon-to-be AlertManager:

We’ll want to ensure that one or more easily triggered alert rules are defined. Here’s a simple alert rule that will fire when any given target is unreachable for longer than 5 seconds:

Save this alert rule to another YAML file (e.g. alert.rules) and reference the alert.rules file the Prometheus server configuration:

/prometheus.yml

Tip: AlertManager configurations can be visualized by using the routing tree editor.

Next, check out the AlertManager project code:

Given that our user story includes making front-end changes to AlertManager, ensure that you install a small utility to generate Go code from any file. In this case, the Prometheus project uses the utility for embedding static web files (HTML, CSS, AngularJS, etc.) into the alert manager binary. Execute:

Optional Prerequisite

Next (optionally) create an alert notification receiver. Of the supported AlertManager receivers, let’s opt for integrating Slack.

Create an incoming webhook for your app, choose an existing channel or create a new one and choose “Add Incoming Webhook.” Copy your new webhook URL (e.g. https://hooks.slack.com/services/asdfXX/asdfXX) and use it along with your channel name to create a file (e.g. notification.yml) with the following YAML configuration:

Build, Run, Test

Next, verify you have a functional development environment by building and running the project:

These steps will become familiar as you’ll use this same three-step build, run, test cycle throughout the process of creating new project functionality.

Verify

If you choose to setup a Slack channel, you should now see new alerts firing as and when your random targets go up and down.

Irrespective of the optional Slack integration, you should see new alerts in the AlertManager user interface.

Coding

Now that our development environment is setup let’s start our contribution.

Prometheus AlertManager – no alert history

A good principle of any modern software is to ensure extensibility and programmatic integration by way of solid APIs. Any functionality displayed in the user interface should be provided by an underlying REST API. Let’s open the /api.go file to add this new API endpoint.

/api.go

All UI functionality should be addressable via API. Let’s register a new /history API endpoint:
r.Get("/history", ihf("history", api.listAllAlerts))

With our /api/v1/history endpoint a newly addressable API endpoint, we’ll need to build a function to handle requests made to it. The api.listAllAlerts function will handle inbound HTTP requests made to the new endpoint.

/ui/app/js/app.js

With the new /api/vi/history endpoint and handler defined, we’ll need to register and define this new backend endpoint in the front-end user interface. By modifying /ui/app/js/app.js, we’ll provide a way of invoking this endpoint, and subsequently, navigate to a History view. There are a number of updates to the Angular application to be done.

Start by adding an item within NavCtrl for the History menu item:

Insert a new History directive:

as well as a new History service:

and a new History controller:

Finally, we’ll need a page in which to view the transpired alerts. So, create a new file, history.html, under /ui/app/partials. History.html will simply format the display a tabular recordset. A new recordset will be retrieved from our data provider.

/provider

With API endpoint and front-end changes in place, let’s turn our attention to the backend for collecting the right recordset from our data provider:

1. Add a new AlertIterator (e.g. GetAll() AlertIterator) to /provider/provider.go
2. Add a new AlertProvider and SQL query to /provider/sqlite/sqlite.go
3. Add a new AlertIterator and AlertProvider to /provider/boltmem/boltmem.go

Test

Assuming you’ve gone through a few build, run, test cycles, you should see a new History menu item with a list of all fired alerts, whether actively firing or not.

Summary

While the AlertManager is not intended to provide a history (the roadmap calls for a form of logging that other tools can hook into), this example enhancement provides a view of transient history — that of the period that the SQlite database holds. The example enhancement is not intended to be a full-blown historical record.

The Prometheus open source project moves at a fair clip. Be sure to avoid frustration by ensuring that you submit a pull request in advance of creating any significant enhancement. This example enhancement is not well-aligned with the project trajectory given sweeping changes coming to the data provider backend.

For those interested in digging into Prometheus more, join us for a post-DockerCon 2016 meetup in Austin, TX. Prometheus is the subject of the next Microservices and Containers Austin meetup.

Feature art by Dino Reichmuth via Unsplash.