Access logs on Node JS with Winston, Morgan, and storage on Elasticsearch
At Packmind, our PaaS provider CleverCloud offers many great features and add-ons to run Web apps. However, there are currently limitations for managing access logs and route them to an external system, such as an ELK stack. CleverCloud allows for draining server logs to Elasticsearch, but there’s no parsing option, and our logs are considered a single string object.
We’re interested in having access logs available to set up monitoring of our API and compute statistics on our traffic.
In this post, we’re going to use Morgan, a request logger middleware for node.js to use in combination with Express, to create access logs, send them to an Elasticsearch database and visualize reports with Kibana. (Good news: with CleverCloud, we can deploy in a few seconds an Elasticsearch & Kibana!)
We’re aware this schema goes against the 11th principle of the 12 factors app, but this is, according to us, a fair trade-off.
All the code of this post is available at the end of this post.
Setup Express and Morgan
Here is a basic setup of a node.js app that will be used for this tutorial. We’re going to install Express, Morgan, and the logger framework Winston.
Let’s first init the modules :
And this is how we can start an Express server with a logger writing on the standard output and use Morgan to log all the incoming requests.
Also, the Morgan (‘combined’) format is the standard Apache combined log output, but we will change it in the following steps.
To make sure everything is ready, if you run your server, and ping http://localhost:3001, you’ll see those logs in your terminal :
Note that Morgan generates the last line because of the incoming query.
Create our custom logger with Morgan
With Morgan, we can create a custom format to log our queries. As we’re going to store our logs into an Elasticsearch database, we want to build our access logs in JSON format.
The documentation of Morgan provides more details on which properties can be retrieved. In this example, we compute a JSON object that will be stringified and containing basic properties for access logs :
If you run now your server and ping it, you’ll get the following log in your terminal :
As you can see, we don’t use our Winston logger yet. This is written only in the standard output. We need to configure morgan for that :
And now we have proper access logs :
Tip #1: User-Agent Parsing
If you have already worked with user agents, you probably know how they’re not easy to understand. Consider this example:
See what I mean? 😉
Fortunately, the npm package ua-parser-js comes to the rescue! Thanks to this library, our complex user agent can be translated into an object with properties that we can easily understand.
In this version, we add four more properties to catch the browser name and version, in addition to the os name, and version :
Tip #2: Sanitize the URL
Depending on how you designed your API, you may use identifiers in URL, like /api/product/:id, where :id is the identifier of your product. In this context, you’re likely to receive calls like this :
However, you would like to gather all these routes under a single one /api/product/:id for a monitoring purpose.
Here is a simple example of how to sanitize your URL :
Tip #3: Get remote ip when running behind a proxy
If your API is called behind an Nginx or any proxy, you may encounter issues with the remote_addr attribute since you will not get the original source IP, but your proxy’s one. Don’t worry, if the configuration of your proxy allows that, you can just use the request header X-Forwarded-For to access this information. That’s why updated the code in this way :
Send logs to Elasticsearch
For this step, another npm module will come to the rescue, winston-elasticsearch. This module provides a Winston transport for Elasticsearch. We need to update our code to include a new transporter for our logger :
Note that the clientOpts will depend on your configuration. You may not need it if you have no authentication, for instance.
The result
That’s it!
After configuring your indexes in Kibana, you can now visualize your logs :
The final version of the code is available publicly on a GitHub repository if you want to take a look!