Instrumenting the Application with OpenTelemetry and Prometheus

Our application has already been set up with a logger, but we need traces and metrics to achieve observability. The OpenTelemetry project aims to support all three (logging, traces, and metrics) in the Go SDK, but in version v1.10, only tracing is stable. So, we will leave our logger in place and interact directly with Prometheus for our metrics. We will begin with OpenTelemetry and distributed tracing.

Adding distributed tracing to the application

Getting started with OpenTelemetry is very easy; first, we will need to create a connection to the collector. In our application, we will have one running and available at the default port. The monolith or microservices will use the following environment variables to configure themselves:

The OpenTelemetry SDK we will use will look for specific variables in the environment that all begin with the OTEL prefix, short for OpenTelemetry. The two variables shown in the preceding snippet are the minimum we will need to run our demonstration. Here, OTEL_SERVICE_NAME should be set to a unique name for the application or component. Also, we are setting it to mallbots for the monolith application. For the services, we will use their package names. The SDK defaults to looking for the collector on localhost. This could work, but we have set it to the hostname we have given it in the Docker Compose environment. We will be communicating with the collector using the OpenTelemetry Protocol (OLTP) and will set OTEL_EXPORTER_OTLP_ENDPOINT to our collector host and the OLTP port.

An OpenTelemetry collector is a vendor-agnostic service that provides instrumentation data collection, processing, and exporting functionality. A single collector can replace the need to run, configure, and connect to multiple agents to instrument your application. ...