Application Logging through RabbitMQ and Graylog

Tracking multiple log application logs on multiple servers can be a hassle.

Centralizing your application logging allows for more robust log management and easier issue resolution.

Graylog is an open-source log management tool built upon Elasticsearch and MongoDB that allows you to:

"Store, search & analyze log data from any source." -- www.graylog.org

For this tutorial, we're running everything under a single host under Docker containers. Once you understand the flow and underlying technologies this can easily be scaled out.

Deploy RabbitMQ

Graylog provides many ways to feed logging data into its index; HTTP, TCP, UDP, AMPQ, Kafka, Flume to name a few.

I use AMPQ  for most of our projects because we have already built out large RabbitMQ clusters so it's a technology we use and support for other projects. AMPQ is supported by most of the logging appenders and allows for the mobility that log shipping tools don't and security that TCP and UDP don't easily provide.

So we'll first deploy RabbitMQ container for log data routing. We'll launch it with exposed ports 5672 (AMQP) and 15672 (Admin interface)

$> docker run -d --name rabbitmq \ 
     -p 5672:5672 \ 
     -p 15672:15672 \ 
     rabbitmq

Once our container is up and running you should be able to browse to the RabbitMQ Admin interface at http://hostname_ip:15672 and login with the credentials guest:guest.

We'll need to create an exchange and queue to route the logging data.

Deploy Graylog

Next, we'll deploy the Graylog all-in-one image that contains preconfigured Elasticsearch and MongoDB servers along with the Graylog server and a web console. We'll link in the RabbitMQ container that we started previously, expose Graylog's web component on port 9000, and finally specify an admin username and password.

$> docker run -d --name graylog \ 
     --link rabbitmq:rabbitmq \ 
     -p 9000:9000 \ 
     -e GRAYLOG_USERNAME=admin \ 
     -e GRAYLOG_PASSWORD=somepasswordatleast16chars \ 
     graylog/allinone

See the docker README for more container settings and options.

At this point, you should have a running Graylog stack but no way to feed in log data, so we'll now attach a GELF AMQP input.

Log into the Graylog web console http://hostname_ip:9000 with the username and password you provided when launching the docker container.

Once logged in, select "System / Inputs -> Inputs" from the top menu and then select "GELF AMQP" and click "Launch new input".

Leave the default settings on all fields except the following...

• Title: GELF AMQP or anything you want
• Broker hostname: rabbitmq the alias for the docker link when launching the Graylog container
• Queue: log-messages
• Allow throttling this input: true, check this feature

Configure Application Loggers

There are many ways to send data to Graylog but for this example, I'm just going to focus on Java logging... particularly Log4j.

Simply add a log4j GELF appender library (t0xa/gelfj) to your project:

<dependencies> 
    ... 
    <dependency> 
        <groupId>org.graylog2</groupId> 
        <artifactId>gelfj</artifactId> 
        <version>1.1.10</version> 
        <scope>compile</scope> 
    </dependency> 
    ... 
</dependencies>

Then configure a GELF appender to your log4j.properties:

# Define the graylog2 destination 
log4j.appender.graylog2=org.graylog2.log.GelfAppender 
log4j.appender.graylog2.amqpURI=amqp://hostname_ip:5672 
log4j.appender.graylog2.amqpExchangeName=log-messages 
log4j.appender.graylog2.amqpRoutingKey=log-messages 
log4j.appender.graylog2.amqpMaxRetries=5 
log4j.appender.graylog2.facility=test-application 
log4j.appender.graylog2.layout=org.apache.log4j.PatternLayout 
log4j.appender.graylog2.layout.ConversionPattern=%d{HH:mm:ss,SSS} %-5p [%t] [%c{1}] - %m%n 
log4j.appender.graylog2.additionalFields={'environment': 'DEV', 'application': 'Sample Java App'} 
log4j.appender.graylog2.extractStacktrace=true 
log4j.appender.graylog2.addExtendedInformation=true 
# Send all INFO logs to graylog2 
log4j.rootLogger=INFO, graylog2

Or in a Grails Config.groovy:

log4j.main = { 
    appenders { 
        appender new org.graylog2.log.GelfAppender(
                name: 'graylog', 
                amqpURI: 'amqp://hostname_ip:5672', 
                amqpExchangeName: "log-messages", 
                amqpRoutingKey: "log-messages", 
                amqpMaxRetries: 5, 
                facility: 'test-application', 
                layout: pattern(conversionPattern: "%d{HH:mm:ss,SSS} %-5p [%t] [%c{1}] - %m%n"), 
                additionalFields: "{'applicationEnvironment': '${environment}', 'applicationName': '${applicationName}', 'applicationVersion': '${applicationVersion}'}", 
                extractStacktrace: true, 
                addExtendedInformation: true 
        ) 
    } 
    root { info 'graylog' } 
}

If you're not using Java, there are many other GELF appenders for other languages.

Java

PHP = gelf-php