Explore Centralized Logging Through Papertrail

Papertrail #

The first centralized logging solution we’ll explore is Papertrail. We’ll use it as a representative of a logging-as-a-service solution that can save us from installing and, more importantly, maintaining a self-hosted alternative.

Papertrail features live trailing, filtering by timestamps, powerful search queries, pretty colors, and quite a few other things that might (or might not) be essential when skimming through logs produced inside our clusters.

Register at PaperTrail #

The first thing we need to do is to register or, if this is not the first time you tried Papertrail, log in.

open "https://papertrailapp.com/"

Please follow the instructions to register, or log in if you already have a user in their system.

You will be glad to find out that Papertrail provides a free plan that allows storage of 50 MB of logs searchable for one day, as well as a full year of downloadable archives. That should be more than enough for running the examples we are about to explore. If you have a relatively small cluster, that should keep you going indefinitely. Their prices are reasonable, even if your cluster is bigger and you have more monthly logs than 50 MB. Arguably, they are so cheap that we can say it provides a better return on investment than if we’d run an alternative solution inside our own cluster. After all, nothing is free. Even self-hosted solutions based on open source create costs in maintenance time as well as in computing power.

For now, what matters is that the examples we’ll run with Papertrail will be well within their free plan.

If you have a small operation, Papertrail will work well. But, if you have many applications and a bigger cluster, you might be wondering whether Papertrail scales to suit your needs. Worry not. One of their customers is GitHub, and they are likely bigger than you are. Papertrail can handle (almost) any load. However, whether it is a good solution for you is yet to be discovered. Read on.

Start using PaperTrail #

Let’s go to the Start screen unless you are already there.

open "https://papertrailapp.com/start"

If you were redirected to the welcome screen, you are not authenticated (your session might have expired). Login and repeat the previous command to get to the start screen.

Click the Add systems button.

If you read the instructions, you’ll probably think that the setup is relatively easy. It is. However, Kubernetes is not available as one of the options. If you change the value of the from drop-down list to something else…, you’ll see a fairly big list of log sources that can be plugged into Papertrail. Still, there is no sign of Kubernetes. The closest one on that list is Docker. Even that one will not do. Don’t worry. I prepared instructions for you or, to be more precise, I extracted them from the documentation buried in Papertrail’s site.

Please note the Your logs will go to logsN.papertrailapp.com:NNNNN and appear in Events message at the top of the screen. We’ll need that address soon, so we better store the values in environment variables.

PT_HOST=[...]

PT_PORT=[...]

Please replace the first [...] with the host. It should be something like logsN.papertrailapp.com, where N is the number assigned to you by Papertrail. The second [...] should be replaced with the port from the before mentioned message.

Now that we have the host and the port stored in environment variables, we can explore the mechanism we’ll use to collect and ship the logs to Papertrail.

Mechanism to collect and ship the logs #

Since I already claimed that most vendors adopted Fluentd for collecting and shipping logs to their solutions, it should come as no surprise that Papertrail recommends it as well. Folks from SolarWinds (Papertrail’s parent company) created an image with customized Fluentd that we can use. In turn, I created a YAML file with all the resources we’ll need to run their image.

cat logging/fluentd-papertrail.yml

As you can see, the YAML defines a DaemonSet with ServiceAccount, SolarWind’s Fluentd, and a ConfigMap that uses a few environment variables to specify the host and the port where logs should be shipped.

We’ll have to change the logsN.papertrailapp.com and NNNNN entries in that YAML before we apply it. Also, I prefer running all logs-related resources in logging Namespace, so we’ll need to change that as well.

cat logging/fluentd-papertrail.yml \
    | sed -e \
    "s@logsN.papertrailapp.com@$PT_HOST@g" \
    | sed -e \
    "s@NNNNN@$PT_PORT@g" \
    | kubectl apply -f - --record

kubectl -n logging \
  rollout status ds fluentd-papertrail

Now that we’re running Fluentd in our cluster and that it is configured to forward logs to our Papertrail account, we should turn back to its UI.

Please switch back to Papertrail console in your browser. You should see a green box stating that logs were received. Click the Events link.