↓ Skip to main content

Mtail: Turn Your Logs into Prometheus Metrics

·369 words·2 mins
Milad Zangeneh
Author
Milad Zangeneh

mtail is Google’s excellent log parser that extracts metrics from application logs for monitoring. Instead of writing complex log processing scripts or paying for expensive log analytics tools, mtail lets you define simple patterns that automatically convert log entries into Prometheus‑compatible metrics.

In this guide, we’ll walk through installing and configuring mtail manually on a Linux system.

What is mtail?
#

mtail reads log files in real time and applies user‑defined programs to extract metrics. Prometheus can then scrape these metrics. It’s particularly useful for:

  • Converting web server access logs to request metrics
  • Extracting error rates from application logs
  • Monitoring database performance from log files
  • Creating business metrics from custom application logs

Step 1: Create required directories
#

Set up the directory structure mtail needs:

sudo mkdir -p /var/lib/mtail
sudo mkdir -p /etc/mtail/progs
sudo mkdir -p /var/log/mtail
sudo mkdir -p /usr/local/bin

Step 2: Download and install mtail
#

Download the latest release from GitHub:

mkdir /tmp/mtail-download
cd /tmp/mtail-download

# Download mtail (replace 3.0.3 with latest version)
wget https://github.com/google/mtail/releases/download/v3.0.3/mtail_3.0.3_linux_amd64.tar.gz

tar -xzf mtail_3.0.3_linux_amd64.tar.gz
sudo cp mtail /usr/local/bin/
sudo chmod 755 /usr/local/bin/mtail

rm -rf /tmp/mtail-download

Step 3: Create your first mtail program
#

Let’s create a simple program to parse nginx access logs. Open /etc/mtail/progs/nginx.mtail with your favorite editor, then add these configurations:

counter nginx_status_codes by code

/^(\S+) \S+ \S+ \[[^]]+\] "\S+ \S+ \S+" (?P<code>\d{3}) / {
    nginx_status_codes[$code]++
}

Step 4: Create a systemd service
#

Create the systemd service file at /etc/systemd/system/mtail.service:

[Unit]
Description=mtail log parser
Documentation=https://github.com/google/mtail
After=network.target

[Service]
Type=simple
# You can run it as the mtail user, but this user must have access to the log files.
# User=mtail
# Group=mtail
ExecStart=/usr/local/bin/mtail \
  -logs /var/log/nginx/access.log \
  -progs /etc/mtail/progs \
  -port 3903 \
  -log_dir /var/log/mtail \
  -emit_prog_label \
  -emit_metric_timestamp \
  -logtostderr
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=mtail

# Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/mtail

[Install]
WantedBy=multi-user.target

Step 5: Start and enable the service
# 

sudo systemctl daemon-reload
sudo systemctl enable mtail
sudo systemctl start mtail

# Check status
sudo systemctl status mtail

Step 6: Verify it’s working
#

Check that mtail is running and serving metrics:

# Check if mtail is listening on port 3903
sudo netstat -tlnp | grep 3903

# Fetch metrics
curl -s http://localhost:3903/metrics | grep nginx