MongoDB Metrics Storage Engine

This project provides a storage engine based on MongoDB for timeseries event data stream. Metrics (min, max, mean, sum) are calculated periodically and stored separately from the event data stream.

Requirements:

• Apache Maven version 3

• Spring Data MongoDB 1.4.0.RC1

• MongoDB 2.2.x or later

Quickstart:

Installation:

Clone the repository using command

git clone git://github.com/aparnachaudhary/mongodb-metrics-store.git

Building:

To build, issue this from the command line:

mvn clean install

Start Server:

To start the application using Jetty server from command line:

mvn clean install -pl core jetty:run

How it works:

Gauge Event

Event data stream is represented as GaugeEvent. Event can be represented in the form of following JSON structure. Gauge event has a value field stored as double and optionally context data can be stored in the form of map construct. Events are stored in their separate collections. e.g. for event with name 'request'; a collection with name request.event is created in the eventstore database.

Events can be stored using the REST interface.

curl -i -X POST http://localhost:8080/rest/events/ -H "Content-Type: application/json" -d
'{"occuredOn":"2013-12-28T14:19:56Z","eventType":"request","value":11.0,"contextData":{"hostname":"localhost"}}'

JSON Structure of stored Gauge Event:

{
    "_id" : ObjectId("52bede0cc2e6f68c4ad8b7a6"),
    "_class" : "net.arunoday.metric.store.model.GaugeEvent",
    "occuredOn" : ISODate("2013-12-28T14:19:56Z"),
    "eventType" : "request",
    "value" : 11,
    "contextData" : {
        "hostname" : "localhost"
    }
}

Gauge Metric

Metrics (min, max, mean, sum) are calculated based on the event input. Metrics are calculated using hierarchial aggregations.

Following aggregations are supported:

• Minute-wise

• Hourly

• Daily

• Monthly

• Yearly

For each event type; aggregation results are stored in separate collections in metricstore database. e.g. request.minute request.hourly request.daily request.monthly and request.yearly For generating minute wise aggregations, event data is used. But for hourly aggregations, instead of using event data stream; minute wise aggregations are used. Same approach is used for further aggregations of higher granularity.

JSON Structure Gauge Metric:

{
    "_id" : ISODate("2013-12-28T13:56:00Z"),
    "value" : {
        "total" : 7170,
        "count" : 12,
        "min" : 0,
        "max" : 603,
        "mean" : 597.5,
        "ts" : ISODate("2013-12-28T13:57:00Z")
    }
}

REST Support

Following REST operations are supported.

Table 1. Table Gauge Event REST Resources

Resource	Description
POST /events/	Stores gauge events and returns the event identifier.
DELETE /events/eventName	Deletes all events for the given event name.

Table 2. Table Gauge Metric REST Resources

Resource	Description
GET /metric/minute/eventId/year/month/day/hour/minute	Returns a single minute wise metric aggregation for the given minute.
GET /metric/minute/eventId/year/month/day/hour	Returns a collection of minute wise metric aggregations for the given hour.
GET /metric/minute/eventId/year/month/day	Returns a collection of minute wise metric aggregations for the given date.
GET /metric/minute/eventId/year/month	Returns a collection of minute wise metric aggregations for the given month.
GET /metric/hourly/eventId/year/month/day/hour	Returns a single hourly metric aggregation for the given hour.
GET /metric/hourly/eventId/year/month/day	Returns a collection of hourly metric aggregations for the given date.
GET /metric/hourly/eventId/year/month	Returns a collection of hourly metric aggregations for the given month.
GET /metric/daily/eventId/year/month/day	Returns a single daily metric aggregation for the given date.
GET /metric/daily/eventId/year/month	Returns a collection of daily metric aggregations for the given month.
GET /metric/daily/eventId/year	Returns a collection of daily metric aggregations for the given year.
GET /metric/monthly/eventId/year/month	Returns a single monthly metric aggregation for the given month.
GET /metric/monthly/eventId/year	Returns a collection of monthly metric aggregations for the given year.
GET /metric/yearly/eventId/year	Returns a single yearly metric aggregation for the given year.

Feedback:

Please feel free to raise bug reports or feature requests here Issues.

Licensing and Copyright:

The project is licensed under the Apache License, Version 2.0