Filtering NGINX logs in Cloudwatch insights
July 26, 2020
July 26, 2020
If you are using the cloudwatch agent to send your Nginx access logs to cloudwatch, Insights can be a pretty powerful tool. Especially, if you are running multiple instances of a particular server and aggregating your logs in cloudwatch.
I've been running the following log format in my Nginx config: log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';
The default query in Cloudwatch Insights doesn’t really parse the logs very well. It just parses a timestamp and then globs the entire log together.
What we want is something a little more useful and easier to search. For that, we’ll use ‘parse’ to format the log how we want. The output of the log format I showed above provides a log that looks like this:
Each piece lines up with the log format:
So, if we want to have Cloudwatch Insights parse this, we just need to tell it what each field is. We do that by setting a wildcard in place of any value we want to be a field. We then set the name we want for that field. For example,
In Cloudwatch Insights, it would look like this: '* - - [*] "* * *" * * "-" "*"' So it's as easy as telling Cloudwatch Insights what to use for each wildcard using parse.
parse @message '* - - [*] "* * *" * * "-" "*"' as remote_addr, timestamp, request_type, location, protocol, response_code, body_bytes_sent
Now you can do things like this:
Make sure you save that query. This way you can use it to search for anything in your logs!
Advanced Data Engineering Platform for Cleansing, Preprocessing and Analytics