How to send application logs to AWS SQS using log4j appender

Prerequisites:

1. AWS SQS queue set up
2. Log4j configuration
3. Maven configuration

Enablement :

AWS Simple Queue Service set up:

Step 1: Once you logged in AWS account, navigate to IAM service from AWS console and check

if any IAM user exists or create a new IAM user

Step 2: click on the user name and go to summary and then Security credentials and Create Access ID. which will create a new Aws- Access ID and secret. Copy both the value or download the CSV file.

Step 3: Now navigate to AWS SQS service and create a new queue. Refer to the below screenshots for new queue creation.

Log4j configuration:

Step 1: Go to the mule application and navigate to log4j.xml under src/main/resources and

configure SQS log4j appender. This appender pushes all the application logs to specified Amazon

SQS Queue.

<Appenders>
<SQS name="sqs_cloud"
awsAccessKey="${sys:awsAccessKey}"
awsRegion="us-east-2"
awsSecretKey="${sys:awsSecretKey}"
maxBatchOpenMs="10000"
maxBatchSize="5"
maxInflightOutboundBatches="5"
queueName="demoqueueservice">
<PatternLayout pattern="%d{ISO8601} %-5p PID[%X{correlationId}][%c{1}] - %m %n"/>
</SQS>
</Appenders>

Step 2: We need to add some package information to support the mule- AWS SQS integration.

<Configuration packages=”com.avioconsulting.log4j”>

Step 3: Also include appender reference in Loggers section and define the log level of AsyncRoot

<Loggers>
<AsyncRoot level=”INFO”>
<AppenderRef ref=”file” />
<AppenderRef ref=”sqs_cloud” />
</AsyncRoot>

Maven configuration:

For the above configuration, you need to add a maven dependency in POM.XML

<dependency>
<groupId>com.avioconsulting</groupId>
<artifactId>log4j2-sqs-appender</artifactId>
<version>1.0.0</version>
</dependency>

Execution:

Refer to the sample application and complete log4j.xml file.

Refer to the sample Log4j.xml file. Here, you need to Provide the AWS Access ID ({sys:awsAccessKey}) and Secret key ({sys:awsSecretKey}) that received initially for an IAM user as runtime argument. Also, you need to specify a queueName and AWSregion.

<?xml version=”1.0" encoding=”utf-8"?>
<Configuration packages=”com.avioconsulting.log4j”>
<Appenders>
<RollingFile name=”file” fileName=”${sys:mule.home}${sys:file.separator}logs${sys:file.separator}xyz.log” filePattern=”${sys:mule.home}${sys:file.separator}logs${sys:file.separator}xyz-%i.log”>
<PatternLayout
pattern=”%-5p %d [%t] [processor: %X{processorPath}; event: %X{correlationId}] %c: %m%n” />
<SizeBasedTriggeringPolicy size=”10 MB” />
<DefaultRolloverStrategy max=”10" />
</RollingFile>
<SQS name=”sqs_cloud” 
awsAccessKey=”${sys:awsAccessKey}”
awsRegion=”us-east-2"
awsSecretKey=”${sys:awsSecretKey}”
maxBatchOpenMs=”10000" maxBatchSize=”5"
maxInflightOutboundBatches=”5"
queueName=”demoqueueservice”>
<PatternLayout
pattern=”%d{ISO8601} %-5p PID[%X{correlationId}][%c{1}] — %m %n” />
</SQS>
</Appenders>
<Loggers>
<AsyncRoot level=”INFO”>
<AppenderRef ref=”file” />
<AppenderRef ref=”sqs_cloud” />
</AsyncRoot>
</Loggers>
</Configuration>

Now while hitting the service the application logs will be sent to specified Amazon SQS Queue.

Console logs :

Queue logs:

Step 1: Click to queue name and navigate to the send and receive a message.

Step 2: Scroll to the receive message section and click poll for messages for getting any new message.

Log Message has 3 parts.

• Details
• Body
• attributes

Details where you can check the AWS sender account ID, timestamp, size of a log message.

The message body will be shown in the Body section

Message attributes data will be shown in the Attribute section if present.

Conclusion:

This is a custom appender for log4j. This appender pushes all the application logs to specified Amazon SQS Queue. This appender is very useful

* When you may want to store the logs somewhere other than in CloudHub

* You can also feed this Queue to any Log analyzers like Splunk, ELK

— — — — — — — Thank you for reading — — — — — —