Log Exploration from the Command Line

One of the most useful skills a developer or system administrator can master is the command line. It is a powerful productivity accelerator and the instantaneous feedback loop of crafting a beautiful command that does exactly what you need is addictive. Command line skills are absolutely necessary to reach your full potential as a technologist, even if you mainly end up working in front-end or business intelligence.

I plan to write many blogs on this topic as there are fathoms of depth to explore when it comes to the command line. That being said I figured a good one to start out with would be log exploration. This is a skill that you will almost certainly find yourself doing on the command line at some point or another.

caution

There are an absurd number of enterprise-grade log management tools out there. The skills mentioned in this article are for ad-hoc troubleshooting and analysis from the command line. They do not replace a full-stack observability solution.

Mind Map of Command Line Analysis Tools

I figured a good way to start this off would be with a mindmap of command-line analysis tools. These tools come out of the box with most *NIX systems (Mac OS and Linux) but of course your mileage may vary with your particular system and command line. This is certainly not an exhaustive list but it is enough to get us started:

I decided to break things down into three main categories:

Text-Producers

These are usually the starting point for log exploration or analysis. Before we can investigate a log file we usually need to get it's text into the working scope of the shell where we can then do things with it.

Text-Processors

Once we have some text to work with we can process it. Usually this will involve on or more pipes to pass around the i/o from each step.

Text-Output

Once we have ran some analysis on our text. We need to decide what to do with it. In general we are either writing it out to another file to look at later. Or we are just reading it from standard output.

The basic workflow will usually look something like this:

Text Producers

The first step of analyzing a log file (or really any text file) from the command line would be to get the text into standard input. A good example of this would be the cat command. I will use /var/log/messages as my example but really this can apply to any log file that exists on your system. When you cat a file you are reading it's contents into standard input and if you do nothing else, write it to standard output:

cat /var/log/messages

It's also possible that maybe you do not want to look at the contents of a single file, but analyze something about the log directory it self. A good example of this is the ls command, which will list the contents of the current working directory into standard input and if you do nothing else, write it to standard output:

ls /var/log

You may also want to just watch the log file as new lines are added for monitoring purposes during some other job. A good way of doing this would be the tail command.

tail -f /var/log/messages

You could also use this to get the last "n" lines of the log file where "n" is some integer:

tail -n 3 /var/log/messages

These text producing commands will serve as the foundation for log analysis.

Text Analysis

Hooray!

Now comes the fun part!

So once we have produced some text, we can then move on to analysis/exploration. Here are a few questions I always ask myself when I am doing something like this:

• What data am I trying to extract from the log?
• How can I best filter out un-necessary noise from my data?

In order to start addressing some of these questions, I think it is important to go over the grep utility. This is one of those tools that is very simple to get started with, but you also get an extreme amount of flexibility and capability with.

A basic grep will just take in a pattern and then return any lines the text that contain the pattern:

cat /var/log/messages | grep error

note

The | operator takes the output from the command on the left, and passes it to the command on the right. A way to read the above command in plain english would be "Read the contents of /var/log/messages and then print any lines that contain the word 'error'"

Of course, you may want to do something much more complex than just print the lines containing a single pattern. Grep can do much more:

cat /var/log/messages | grep -A 3 "error"

tip

This grep command will return lines that contain "error" and also the three lines after the error. This is useful as you may need more context to know why your error occurred. You could also use "B" instead of "A" in order to get the three lines before the pattern match. Or you can use "C" to get the three lines before and after the pattern match. Fun!

This is great! We know grep is a powerful tool and it could likely be the topic of it's own blog post. You can explore the grep documentation for more information on how to grep through text files.

You may want a metric or counter of how many times your error occurs in the log file. This gives you a different data point that may help you answer a question like "How many times did this http 500 error occur?". We can do this by piping our grep lines to another great utility wc.

cat /var/log/messages | grep error | wc -l

tip

This command will first read the contents of /var/log/messages, then it detect all the lines in the file that have the word "error" in it, then it will count the number of lines and write the output to stdout

The final tool we will quickly go over in this section is awk . This is a powerful text analysis tool that is actually a programming language in it's own right. This is a great tool to perform more complex queries than grep, or when you do not know the exact expression to search for.

For example, lets say we had an fruit.txt file that looked something like this:

lemon citrus yellow
strawberry berry red
lime citrus green

Then we wanted to pull extract all the fruit that are citrus. We can use awk like so:

awk '/citrus/ {print}' fruit.txt

Output

lemon citrus yellow
lime citrus green

This is a great way to remove noise from a log file, in this case we removed all non-citrus fruit.

tip

awk is a deep topic. There are whole books written on the tool.

Text Output

There are two main ways you will usually be outputting your data after log analysis.

• Write to standard output (what we have been doing in our examples).
• Write to a different file for analysis.

Writing to standard output is great for ad-hoc log analysis. But there are times you may need to have that data saved for further utility. For example, lets say there was a problem with your application on November 14th. You are paged about this problem at 3:00 AM and you are very very tired. You probably just want to find the errors, then write them out to a different file to look at the after you get some sleep.

cat november14log.txt | grep error > nov14errors.txt

tip

In this example we search for errors from the november 14th log, then write them out to a file called 'nov14errors.txt'. This output file can then take on further analysis later when you are less tired.

cat nov14errors.txt | grep HTTP500

Maybe at a later time you search the error file to see if there are any lines that represent an HTTP500 error that may have caused problems in your application.

Of course, a popular option is to upload the errors log you created to a cloud provider for storage and futher analysis. Here is an example using CloudWatch logs:

aws logs put-log-events --log-group-name my-logs --log-stream-name 11-14 --log-events file://nov14errors.txt

tip

This command utilizes the aws cli to upload the nov14errors.txt file to a log group called 'my-logs'.

Conclusion

Log analysis from the command line is a powerful tool that every dev or sysadmin should have in their toolbelt. Knowing how to query for the information you want and then display it in a useful way will allow you to detect issues and gain insight into application behavior.