3 Effective Log File Management Tools for Viewing and Monitoring Logs

Introduction to 3 handy log file management tools for simple but effective monitoring

Photo by Bench Accounting on Unsplash

Log files are essential for identifying issues in our applications. Often, we need to sift through large log files to find relevant information.

While modern solutions like Kibana exist, you might not want to use complex software for smaller projects, or you might be restricted from using such application monitoring tools.

In this tutorial, I’ll introduce you to three tools: swatchdog, glogg, and lnav. These tools are easy to get started with and offer great features for free. We’ll explore their most common uses with straightforward examples.

Let’s dive in!

1. Swatchdog

Swatchdog stands for “simple watch dog.” As its name implies, it is used for monitoring system activities. It requires a configuration file with patterns to search for.

Why is it useful?

Suppose that you are running a third-party application that doesn’t send an email when its process has finished. Instead of checking its log file manually, you might implement your own solution by creating a custom script to monitor the file. Here is where swatchdog comes in handy — it’s much simpler to use.

Also, this tool can be useful if you want to get notified when an error message occurs in your application.

Example configuration

First, let’s install swatchdog.

• For Ubuntu/Debian distributions:

$ sudo apt install swatch

• For Fedora distributions:

$ sudo dnf install swatch

• For CentOS distributions:

$ sudo yum install epel-release && sudo yum install swatch

• Swatchdog requires a configuration file called .swatchrc. Let’s create it:

$ touch /home/user/.swatchrc

Suppose that you want to get notified when the word Finished is found in the program’s log:

• You can have a config like this:

watchfor /Finished/
          echo green
 mail=user, subject="Pattern found"

We’re telling swatchdog to look for Finished as a fixed string. You could also use regular expressions for more flexibility.

When it finds the pattern, it will highlight the text in the terminal in green. You can specify a different color for various kinds of messages.

Swatchdog watches the /var/log/syslog file by default, so we’ll have to modify this setting if we want to monitor our custom log instead.

• To provide a different configuration, let’s create a secure.conf file in a new directory called swatch:

watchfor /Finished/
echo green
exec echo "execute some script here..."
mail=user, subject="Script done"

Note that you can provide multiple search patterns here. You can also configure different email subjects and recipients.

• mail is used to send an email to the specified recipient.

• exec can execute a specified command or a script when the given pattern is found.

Now that our configuration is ready, let’s check out how it works.

• Execute the following command and adjust the location of your log file. Mine is called dummy.log:

$ swatchdog -c ~/swatch/secure.conf -t ~/swatch/dummy.log*** swatchdog version 3.2.4 (pid:21064) started at 2020. dec. 11., Friday, 15:26:41 CET

To overwrite the default configuration file, we’re using the -c option.

The -t flag is tailing the specified log file. It’s like the tail -f command.

Demo

I’ve prepared a very simple Bash script for this demo which simulates some work. It’s called script.sh and looks like this:

#!/bin/bash
for i in {1..100}
do
   echo "Doing something $i" >> dummy.log
         done
   echo "Finished script" >> dummy.log

• Let’s execute the script in another terminal window:

$ bash script.sh

• Go back to swatchdog’s window. You should see an output like this, and the pattern will be highlighted in green:

  Finished script

• Now check your user’s emails:

$ less /var/mail/user

Or if you have the mail utility installed, type mail:

$ mail
"/var/mail/user": 1 message 1 new
>N   1 user             p dec 11 15:26    12/445   Script done

Things to keep in mind

Note that swatchdog runs in the foreground, so if you close the terminal, the process will end. You can run it in the background to prevent it from stopping if you end your session. To achieve this, use the — daemon option. This way, you’ll be able to monitor multiple log files if needed.

Another cool feature is that you can specify a time window for when to execute an action. Just provide the when option:

mail=user@domain.com,when=1–6:8–17

Note that the changes to the config file take effect after you restart swatchdog.

Check out the manual page for more options.

2. Glogg

Sometimes you have to search through complex and long log files. It might be more efficient to visualize the log and search for regex patterns.

Glogg is a multi-platform GUI tool that can open large log files and search for given patterns. The official documentation describes it best: “glogg can be seen as a graphical, interactive combination of grep and less.”

Main advantages

• It opens big files fast.

• It saves your search filters so you don’t have to retype them next time.

• You can mark suspicious lines that you might want to review later.

• You can monitor your log file in real time, like with the tail -f command.

• It supports regex, fixed strings, and wild cards in your search patterns.

Demo

First, let’s install glogg. It is available on both Windows and Unix OS. Follow the installation guide according to your requirements.

I’m using it on Ubuntu:

$ apt-get install glogg

You can invoke it directly by typing glogg in the terminal. By default, it will open the GUI, and you can choose which file to open.

You can also provide the log file’s path as an argument to open it directly:

$ glogg ~/projects/app.log

Screenshot from my PC showing how a log opened in glogg looks like

By clicking Tools -> Filters, you can define what patterns to look for.

For example, I created filters for the word api and anything starting with Exception. After I clicked on Apply, the matched lines were immediately highlighted in the defined color:

Screenshot from my PC showing custom filters

Refer to the documentation for more usages.

3. Lnav

Lnav is a is a powerful, open-source log file viewer designed to help you easily browse and analyze log files. It provides real-time monitoring, advanced filtering, and search capabilities. With its intuitive interface and useful features, lnav makes it simpler to troubleshoot and gain insights from your log data.

Main advantages

Here is how lnav compares to grep, tail, less:

Standard Unix utilities are excellent for processing raw text lines, but they fall short in understanding log messages. For example, while tail can monitor multiple files simultaneously, it doesn't display messages in chronological order and doesn't allow backward scrolling. Grep can find matching lines but doesn't return complete multi-line log messages. Less can only display one file at a time. Additionally, none of these basic tools handle compressed files effectively.

Demo

Installation on Ubuntu:

sudo apt-get install lnav

To start using lnav, simply run the command followed by the log file you want to view:

bash lnav /path/to/your/logfile.log

You can also open multiple log files simultaneously:

lnav /var/log/syslog /var/log/auth.log

lnav allows you to highlight specific patterns in your logs. To highlight all instances of the word "warning":

:highlight  /warning/

Turn to the docs for more commands.

Conclusion

We’ve explored some of the main features of swatchdog, lnav, and glogg. While modern log monitoring tools might provide more features, they require more time to get a handle on their functionalities. It’s worth knowing simple tools like these if you want to get started quickly.

I hope that you’ve learned something new today. Thank you for reading, and happy coding!